Build Substrate on Ubuntu: Step-by-Step Guide
Today I’ll share the steps required for me to install the Substrate on an ubuntu 18.04 bionic machine along with the obstacles met on this way and fixes applied to overcome them and move further.
Though errors and fixes described below are for that specific OS, they may be applicable for other Debian-like linux distributives.
Summary
Substrate is a framework from Parity for building your own blockchains from scratch. You can find some design- and vision- specifics of it in my previous blog post.
So when you go to https://docs.substrate.dev/ it just suggests to run a single command to install all modules needed for Substrate. Sounds great! But it turns out things are not so easy, at least when you aren’t a Mac user. Maybe it’s just my fate, but more likely I’m not the only one building this on ubuntu and meeting these problems. So I’d like to publish them here, and I hope it might be useful for fellows who come along this way after me.
Step 0: openssl problem
Go to the docs.substate.dev and run the magic command given there
$ curl https://getsubstrate.io -sSf | bash
Hopefully, after a while (about an hour or so), you’ll get all the stuff installed. At least, this is what is supposed to happen…
But this was not the case for me. The build failed for the first time with the following
--- stderr
thread 'main' panicked at '
Header expansion error:
Error { kind: ToolExecError, message: "Command \"cc\" \"-O0\" \"-ffunction-sections\" \"-fdata-sections\" \"-fPIC\" \"-g\" \"-fno-omit-frame-pointer\" \"-m64\" \"-Wall\" \"-Wextra\" \"-E\" \"build/expando.c\" with args \"cc\" did not execute successfully (status code exit code: 1)." }
Failed to find OpenSSL development headers.
You can try fixing this setting the `OPENSSL_DIR` environment variable
pointing to your OpenSSL installation or installing OpenSSL headers package
specific to your distribution:
# On Ubuntu
sudo apt-get install libssl-dev
# On Arch Linux
sudo pacman -S openssl
# On Fedora
sudo dnf install openssl-devel
See rust-openssl README for more information:
https://github.com/sfackler/rust-openssl#linux
', /home/greez/.cargo/registry/src/github.com-1ecc6299db9ec823/openssl-sys-0.9.43/build/main.rs:126:13
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
Alright. So what we see here is that the rust-openssl cargo failed to build because it cannot find the OpenSSL development headers. My general approach to deal with such errors is to checkout that particular crate separately and try to build it alone several times changing something here and there. Because it would take much much less time to re-run a single cargo build than build all the packages needed for Substrate.
Let’s do just that. Git clone the rust-openssl crate sources
$ git clone https://github.com/sfackler/rust-openssl
then build it
$ cd rust-openssl
$ cargo build
First time we certainly get the same error. Now some brief googling on topic and reading through the openssl crate docs says it requires specific OpenSSL version
OpenSSL versions 1.0.1 through 1.1.1 and LibreSSL versions 2.5 through 2.8 are supported.
Brief check what version I got says it is
$ openssl version
OpenSSL 1.0.2n 7 Dec 2017
Which should be ok (1.0.2n is between 1.0.1 and 1.1.1). But it doesn’t :\ (I have no idea why).
Moreover, the libssl-dev ubuntu package installed as well
$ apt install libssl-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
libssl-dev is already the newest version (1.1.0g-2ubuntu4.3).
0 upgraded, 0 newly installed, 0 to remove and 213 not upgraded
Which doesn’t help anyway. Googling about the topic leads to this Github issue. Although proposed solution didn’t help in my case.
So next try is to get that openssl library sources and build it ourselves. Sounds like a plan.
Follow https://wiki.openssl.org/index.php/Compilation_and_Installation to install the opensll with specified –openssldir path
$ git clone git://git.openssl.org/openssl.git
$ cd openssl
$ ./config --openssldir=/usr/local/ssl
$ make
$ make test
$ sudo make install
Now check what openssl version we’ve got
$ openssl version
OpenSSL 1.1.1b 26 Feb 2019
Version is OpenSSL 1.1.1b. Cool.
Then set the same path to OPENSSL_LIB_DIR env var, and finally build the crate.
$ export OPENSSL_DIR="/usr/local/ssl"
$ cd rust-openssl
$ cargo build
And voila! It builds now. Problem solved.
Step 1: libclang & libz problem
Turn back to the initial installation script. This time is passed further the rust-openssl crate and fails on building the substrate crate with
--- stderr
thread 'main' panicked at 'Unable to find libclang: "the `libclang` shared library at /usr/lib/llvm-6.0/lib/libclang-6.0.0.so could not be opened: libz.so.1: cannot open shared object file: No such file or directory"', src/libcore/result.rs:997:5
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
warning: build failed, waiting for other jobs to finish...
error: failed to compile `substrate v2.0.0 (/tmp/tmp.y6aLaljbz4)`, intermediate artifacts can be found at `/tmp/tmp.y6aLaljbz4/target`
Again, following our general process to resolving such issues: let’s get the substrate package alone and build it (this is a catcha way as we’ll seen below).
$ git clone https://github.com/paritytech/substrate
$ cd substrate/
$ cargo build
Unsurprisingly, we get the same error
--- stderr
thread 'main' panicked at 'Unable to find libclang: "the `libclang` shared library at /usr/lib/llvm-6.0/lib/libclang-6.0.0.so could not be opened: libz.so.1: cannot open shared object file: No such file or directory"', src/libcore/result.rs:997:5
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
warning: build failed, waiting for other jobs to finish...
error: build failed
According to it, we need to have libclang-dev installed.
$ apt list -a libclang-dev
Listing... Done
libclang-dev/bionic-updates,now 1:6.0-41~exp5~ubuntu1 amd64 [installed]
libclang-dev/bionic 1:6.0-41~exp4 amd64
Alright, we have it. Obviously the cargo cannot find it. What about LIBCLANG_PATH env var
$ echo $LIBCLANG_PATH
Isn’t specified… let’s find the lib file
$ sudo find / -name libclang.so
/usr/lib/llvm-6.0/lib/libclang.so
Here it is. Let’s set path to it to the env var
export LIBCLANG_PATH="$LIBCLANG_PATH:/usr/lib/llvm-6.0/lib"
Better put this string into the ~/.basrhc file to autoload this value on bash start.
Rebuild the crate… ooops, the same error appears. Okay then, my hypothesis here is that its looks for libclang-6.0.0.so file while what we have is different named one libclang.so.
Simple hack here is to create a symlink
$ sudo ln -s /usr/lib/llvm-6.0/lib/libclang.so /usr/lib/llvm-6.0/libclang-6.0.0.so
And one more thing - mention the second file in the failed build log: libz.so.1.
So lets just copy it to the path where compiler seeks for it.
$ sudo find / -name libz.so.1
/lib/x86_64-linux-gnu/libz.so.1
$ sudo cp /lib/x86_64-linux-gnu/libz.so.1 /usr/local/lib/
And then rebuild the cargo.
Boom! We solved this error. Got the next one - libtinfo.so.5.
Step 2: libtinfo problem
$ cd substrate/
$ cargo build
error: failed to ruerror: failed to run custom build command for `librocksdb-sys v5.17.2`
process didn't exit successfully: `dev/substrate/substrate/target/debug/build/librocksdb-sys-bd831870f3fc9569/build-script-build` (exit code: 101)
--- stdout
cargo:rerun-if-changed=build.rs
cargo:rerun-if-changed=rocksdb/
cargo:rerun-if-changed=snappy/
cargo:rerun-if-changed=lz4/
cargo:rerun-if-changed=zstd/
cargo:rerun-if-changed=zlib/
cargo:rerun-if-changed=bzip2/
--- stderr
thread 'main' panicked at 'Unable to find libclang: "the `libclang` shared library at /usr/lib/llvm-6.0/libclang-6.0.0.so could not be opened: libtinfo.so.5: cannot open shared object file: No such file or directory"', src/libcore/result.rs:997:5
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
Same fix - again, try to find the file, then if found, copy it to the lib folder
$ sudo find / -name libtinfo.so.5
/lib/x86_64-linux-gnu/libtinfo.so.5
$ sudo cp /lib/x86_64-linux-gnu/libtinfo.so.5 /usr/local/lib/
And try cargo build again. Alright, the last mentioned error disappeared. But this time we’ve got something which looks pretty much other way.
Step 3: wasm issues
$ cd substrate/
$ cargo build
Compiling node-executor v2.0.0 (/home/greez/Documents/dev/substrate/substrate/node/executor)
error: couldn't read node/executor/src/../../runtime/wasm/target/wasm32-unknown-unknown/release/node_runtime.compact.wasm: No such file or directory (os error 2)
--> node/executor/src/lib.rs:26:100
|
26 | native_executor_instance!(pub Executor, node_runtime::api::dispatch, node_runtime::native_version, include_bytes!("../../runtime/wasm/target/wasm32-unknown-unknown/release/node_runtime.compact.wasm"));
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
error: aborting due to previous error
error: Could not compile `node-executor`.
So I suppose just checking out master branch from substrate package github repo and then running cargo build on it is not enough… Let’s look inside the installation script coming with the initial command
$ curl https://getsubstrate.io > install_script.sh
As we see inside the downloaded install_script.sh, it runs ./scripts/init.sh before making the build, which initializes wasm build environment and installs wasm-gc
if [ -z $CI_PROJECT_NAME ] ; then
rustup update nightly
rustup update stable
fi
rustup target add wasm32-unknown-unknown --toolchain nightly
# Install wasm-gc. It's useful for stripping slimming down wasm binaries.
command -v wasm-gc || \
cargo +nightly install --git https://github.com/alexcrichton/wasm-gc --force
So yeah, this time we’re just running the build command incorrectly. Assuming other issues have been solved for now, let’s just start from the beginning again.
Solved!
$ curl https://getsubstrate.io -sSf | bash
Now this command successfully builds everything (though with warnings).
This takes an hour or so… after all, it’s completed! And this command checks the success criteria
$ substrate --version
substrate 2.0.0-563096ee-x86_64-linux-gnu
Nice. Now we can play with Substrate. For instance, check out the Crypto Kitties Tutorial.