Summary

Substrate is a framework from Parity for building your own blockchains from scratch. You can find some design- and vision- specifics of it in my previous blog post.

Today I'll share the steps required for me to install the Substrate on an ubuntu 18.04 bionic machine along with the obsctacles met on this way and fixes applied to overcome them and move further. Though errors and fixes described below are for that specific OS, they may be applicable for other Debian-like linux distros. Or even, taking the general approach to such problem-solving, other systems as well.

So when you go to https://docs.substrate.dev/ it just suggests to run a single command to install all modules needed for Substrate. Sounds great! But it turns out things are not so easy, at least when you aren't a Mac user. Maybe it's just my fate, but more likely I'm not the only one building this for ubuntu and meeting these problems. So I'd like to publish them here, and hope it would be useful for fellows who come along this way after me.

SPOILER: Problems described are not reproducable on a virgin VPS Ubuntu bionic 18.04.1 LTS. Checked this today, build succeeded taking more than 7 hours on a AWS E2 t.small instance. So the reason of troubles was a corrupted env on my laptop. Nevertheless, this info could be useful for folks building on their non-virgin machines as did I (and googling on these particular errors below show some demand on this on GitHub and StackOverlow).

Step 0: openssl problem

Go to the docs.substate.dev and run the magic command given there

$ curl https://getsubstrate.io -sSf | bash

Hopefully, after a while (about an hour or so), you'll get all the stuff installed. At least, this is what is supposed to happen...
But this was not the case for me. The build failed for the first time with the following

--- stderr
thread 'main' panicked at '
Header expansion error:
Error { kind: ToolExecError, message: "Command \"cc\" \"-O0\" \"-ffunction-sections\" \"-fdata-sections\" \"-fPIC\" \"-g\" \"-fno-omit-frame-pointer\" \"-m64\" \"-Wall\" \"-Wextra\" \"-E\" \"build/expando.c\" with args \"cc\" did not execute successfully (status code exit code: 1)." }

Failed to find OpenSSL development headers.

You can try fixing this setting the `OPENSSL_DIR` environment variable
pointing to your OpenSSL installation or installing OpenSSL headers package
specific to your distribution:

    # On Ubuntu
    sudo apt-get install libssl-dev
    # On Arch Linux
    sudo pacman -S openssl
    # On Fedora
    sudo dnf install openssl-devel

See rust-openssl README for more information:

    https://github.com/sfackler/rust-openssl#linux
', /home/greez/.cargo/registry/src/github.com-1ecc6299db9ec823/openssl-sys-0.9.43/build/main.rs:126:13
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

Alright. So what we see here is that the rust-openssl cargo failed to build because it cannot find the OpenSSL development headers. My general approach to deal with such errors is to checkout that particular crate separately and try to build it alone several times changing something here and there. Because it would take much much less time to re-run a single cargo build than build all the packeages needed for Substrate.

Let's do just that. Git clone the rust-openssl crate sources

$ git clone https://github.com/sfackler/rust-openssl

then build it

$ cd rust-openssl
$ cargo build

First time we certainly get the same error. Now some brief googling on topic and reading through the openssl crate docs says it requires specific OpenSSL version

OpenSSL versions 1.0.1 through 1.1.1 and LibreSSL versions 2.5 through 2.8 are supported.

Brief check what version I got says it is

$ openssl version
OpenSSL 1.0.2n  7 Dec 2017

Which should be ok (1.0.2n is between 1.0.1 and 1.1.1). But it doesn't :\ (I have no idea why).
Moreover, the libssl-dev ubuntu package installed as well

$ apt install libssl-dev
Reading package lists... Done
Building dependency tree       
Reading state information... Done
libssl-dev is already the newest version (1.1.0g-2ubuntu4.3).
0 upgraded, 0 newly installed, 0 to remove and 213 not upgraded

Which doesn't help anyway. Googling about the topic leads to this Github issue. Although proposed solution didn't help in my case.

So next try is to get that openssl library sources and build it ourselves. Sounds like a plan.

Follow https://wiki.openssl.org/index.php/Compilation_and_Installation to install the opensll with specified –openssldir path

$ git clone git://git.openssl.org/openssl.git
$ cd openssl
$ ./config --openssldir=/usr/local/ssl
$ make
$ make test
$ sudo make install

Now check what openssl version we've got

$ openssl version
OpenSSL 1.1.1b  26 Feb 2019

Version is OpenSSL 1.1.1b. Cool.
Then set the same path to OPENSSL_LIB_DIR env var, and finally build the crate.

$ export OPENSSL_DIR="/usr/local/ssl"
$ cd rust-openssl
$ cargo build

And voila! It builds now. Problem solved.

Step 1: libclang & libz problem

Turn back to the initial installation script. This time is passed further the rust-openssl crate and fails on building the substrate crate with

--- stderr
thread 'main' panicked at 'Unable to find libclang: "the `libclang` shared library at /usr/lib/llvm-6.0/lib/libclang-6.0.0.so could not be opened: libz.so.1: cannot open shared object file: No such file or directory"', src/libcore/result.rs:997:5
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

warning: build failed, waiting for other jobs to finish...
error: failed to compile `substrate v2.0.0 (/tmp/tmp.y6aLaljbz4)`, intermediate artifacts can be found at `/tmp/tmp.y6aLaljbz4/target`

Again, following our general process to resolving such issues: let's get the substrate package alone and build it (this is a catcha way as we'll seen below).

$ git clone https://github.com/paritytech/substrate
$ cd substrate/
$ cargo build

Unsurprisingly, we get the same error

--- stderr
thread 'main' panicked at 'Unable to find libclang: "the `libclang` shared library at /usr/lib/llvm-6.0/lib/libclang-6.0.0.so could not be opened: libz.so.1: cannot open shared object file: No such file or directory"', src/libcore/result.rs:997:5
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

warning: build failed, waiting for other jobs to finish...
error: build failed

According to it, we need to have libclang-dev installed.

$ apt list -a libclang-dev
Listing... Done
libclang-dev/bionic-updates,now 1:6.0-41~exp5~ubuntu1 amd64 [installed]
libclang-dev/bionic 1:6.0-41~exp4 amd64

Alright, we have it. Obviously the cargo cannot find it. What about LIBCLANG_PATH env var

$ echo $LIBCLANG_PATH


Isn't specified... let's find the lib file

$ sudo find / -name libclang.so
/usr/lib/llvm-6.0/lib/libclang.so

Here it is. Let's set path to it to the env var

export LIBCLANG_PATH="$LIBCLANG_PATH:/usr/lib/llvm-6.0/lib"

Better put this string into the ~/.basrhc file to autoload this value on bash start.

Rebuild the crate... ooops, the same error appears. Okay then, my hypothesis here is that its looks for libclang-6.0.0.so file while what we have is different named one libclang.so.

Simple hack here is to create a symlink

$ sudo ln -s /usr/lib/llvm-6.0/lib/libclang.so /usr/lib/llvm-6.0/libclang-6.0.0.so

And one more thing - mention the second file in the failed build log: libz.so.1.
So lets just copy it to the path where compiler seeks for it.

$ sudo find / -name libz.so.1
/lib/x86_64-linux-gnu/libz.so.1

$ sudo cp /lib/x86_64-linux-gnu/libz.so.1 /usr/local/lib/

And then rebuild the cargo.

Boom! We solved this error. Got the next one - libtinfo.so.5.

Step 2: libtinfo problem

$ cd substrate/
$ cargo build

error: failed to ruerror: failed to run custom build command for `librocksdb-sys v5.17.2`
process didn't exit successfully: `dev/substrate/substrate/target/debug/build/librocksdb-sys-bd831870f3fc9569/build-script-build` (exit code: 101)
--- stdout
cargo:rerun-if-changed=build.rs
cargo:rerun-if-changed=rocksdb/
cargo:rerun-if-changed=snappy/
cargo:rerun-if-changed=lz4/
cargo:rerun-if-changed=zstd/
cargo:rerun-if-changed=zlib/
cargo:rerun-if-changed=bzip2/

--- stderr
thread 'main' panicked at 'Unable to find libclang: "the `libclang` shared library at /usr/lib/llvm-6.0/libclang-6.0.0.so could not be opened: libtinfo.so.5: cannot open shared object file: No such file or directory"', src/libcore/result.rs:997:5
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

Same fix - again, try to find the file, then if found, copy it to the lib folder

$ sudo find / -name libtinfo.so.5
/lib/x86_64-linux-gnu/libtinfo.so.5

$ sudo cp /lib/x86_64-linux-gnu/libtinfo.so.5 /usr/local/lib/

And try cargo build again. Alright, the last mentioned error disappeared. But this time we've got something which looks pretty much other way.

Step 3: wasm issues

$ cd substrate/
$ cargo build

Compiling node-executor v2.0.0 (/home/greez/Documents/dev/substrate/substrate/node/executor)
error: couldn't read node/executor/src/../../runtime/wasm/target/wasm32-unknown-unknown/release/node_runtime.compact.wasm: No such file or directory (os error 2)
  --> node/executor/src/lib.rs:26:100
   |
26 | native_executor_instance!(pub Executor, node_runtime::api::dispatch, node_runtime::native_version, include_bytes!("../../runtime/wasm/target/wasm32-unknown-unknown/release/node_runtime.compact.wasm"));
   |                                                                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: aborting due to previous error

error: Could not compile `node-executor`.

So I suppose just checking out master branch from substrate package github repo and then running cargo build on it is not enough... Let's look inside the installation script coming with the initial command

$ curl https://getsubstrate.io > install_script.sh

As we see inside the donwloaded install_script.sh, it runs ./scripts/init.sh before making the build, which initialises wasm build environment and installs wasm-gc

if [ -z $CI_PROJECT_NAME ] ; then
   rustup update nightly
   rustup update stable
fi

rustup target add wasm32-unknown-unknown --toolchain nightly

# Install wasm-gc. It's useful for stripping slimming down wasm binaries.
command -v wasm-gc || \
        cargo +nightly install --git https://github.com/alexcrichton/wasm-gc --force

So yeah, this time we're just running the build command incorectly. Assuming other issues are solved now, let's just start from the begining again.

Solved!

$ curl https://getsubstrate.io -sSf | bash

Now this command successfully builds everything (though with warnings).
This takes an hour or so… after all, it's completed! And this command checks the success criteria

$ substrate --version
substrate 2.0.0-563096ee-x86_64-linux-gnu

Nice. Now we can play with Substrate. For instance, perform the Crypto Kitties Tutorial. Will share my experience on it in the next post.