chore: compile mobile with nwaku support (android)

[richard-ramos]

Mobile should compile in android while using nwaku. This means all dependencies must be met, and no missing libraries warning should be visible.

• Create custom library with a single HelloWorld function for all android architectures
• Compile status-go including the custom library
• Modify status-mobile to load a custom library in status-mobile
• Run status-mobile in the simulator and in a device, and make sure the "HelloWorld" is displayed correctly in the logs
• Replace custom library for libwaku
• Run status-mobile in the simulator and in a device, and make sure the nwaku node is created with wakuNew from libwaku
• Nix flake for building nwaku and modify status-mobile to use this nwaku flake
• Modify CI build scripts

[jakubgs]

Building status-go with included nim-waku(also stupidly known as nwaku) is not going to be trivial, for at least 3 reasons:

1. Fetching status-go with nim-waku with included submodules = true takes about 4 minutes every time.
2. nim-waku requires a custom build of Nim compiler due to use of nimbus-build-system which pins specific version.
3. nim-waku cannot be built using a Nix derivation because it contains Rust submodules which require network access.

For these reasons it is not feasibly to simply add nim-waku as a submodule. We need a solution which will allow us to cache an already built version of nim-waku, otherwise developers both on status-go and status-mobile side will waste a lot of time re-compiling nim-waku, probably multiple times a day.

To achieve this in a clean manner the best possible solution is probably to develop a Nix flake setup for nim-waku. The same way we did it for nim-codex repository which also includes a Rust circom-compat-ffi submodule, which needed its own Nix flake.

[richard-ramos]

@markoburcul, @jakubgs:

To add a bit more of details of what should the .flake for nwaku support to compile libwaku for mobile:

nwaku:

1. For building RLN, this is done with rust and cross-rs in this script. cross-rs uses docker
2. AndroidNDK should be installed (should be part of the dependencies in nix)
3. An env variable ANDROID_NDK_HOME should be setup pointing to AndroidNDK
4. This is the list of targets we need to execute

make libwaku-android-amd64
make libwaku-android-arm64
# compilation fails for the following targets, but it should still be possible to indicate to nix we want to compile these targets
# make libwaku-android-arm
# make libwaku-android-x86

This will generate the following folders inside ./build/android/. Each one of these folders should contain 2 shared objects: librln.so and libwaku.so

./arm64-v8a
./armeabi-v7a
./x86
./x86_64

[richard-ramos]

@Ivansete-status @gabrielmer @jakubgs @markoburcul

To build status-mobile with libwaku, I'm currently doing this manually. Do note i'm using specific branches of status-go and status-mobile

1. Open Android Studio, and run the emulator

# Clone the repos
git clone https://github.com/status-im/status-go.git
git clone https://github.com/status-im/status-mobile.git

# Build libwaku
cd status-go
export STATUS_GO_SRC_OVERRIDE=$(pwd)
git checkout feature/nwaku-in-status
git submodule init
git submodule update
cd third_party/nwaku
make update
make libwaku-android-amd64
make libwaku-android-arm64

# Create .aar
cd ../../../status-mobile
git checkout nwaku-in-status
TEMP_DIR=$(mktemp -d)
mkdir $TEMP_DIR/jni
cp -R ../status-go/third_party/nwaku/build/android/arm64-v8a $TEMP_DIR/jni/.
cp -R ../status-go/third_party/nwaku/build/android/x86_64 $TEMP_DIR/jni/.
echo '<manifest xmlns:android="http://schemas.android.com/apk/res/android" package="com.example.mylibrary" />' > $TEMP_DIR/AndroidManifest.xml
pushd $TEMP_DIR
zip -r libwaku.aar *
popd $TEMP_DIR
cp $TEMP_DIR/libwaku.aar ./android/app/libs/.
rm -rf $TEMP_DIR

# Build status-mobile
make run-clojure 
# Wait until it says `[:mobile] Build completed. (2034 files, 26 compiled, 0 warnings, 9.73s)` and press Ctrl+C

make run-android

[jakubgs]

Thanks, that's very helpful.

[markoburcul]

I've managed to overcome building the Nim compiler using nimbus-build-system. The issue I've encountered is that nwaku is using Nim 2.0.12 while for nim-codex it is 2.0.14.I will put this PR in draft so whoever reads this can see the progress.

[markoburcul]

@richard-ramos do we want also do this for iOS?

[markoburcul]

After debug session with @jakubgs , we are still getting the error when building libwaku-android-arm64 using Nix:

nwaku> /nix/store/hbllc75qwkqkx8r40814bdsgym34gc6x-nim-unwrapped-2.0.12/nim/config/nim.cfg(63, 3) Hint: added path: '/build/source/vendor/.nimble/pkgs2' [Path]
nwaku>     nimblepath="/opt/nimble/pkgs2/"
nwaku>     ^
nwaku> /build/nim-2.0.12/compiler/nimblecmd.nim(144, 12) compiler msg initiated here [MsgOrigin]
nwaku> /nix/store/hbllc75qwkqkx8r40814bdsgym34gc6x-nim-unwrapped-2.0.12/nim/config/nim.cfg(63, 3) Hint: added path: '/build/source/vendor/.nimble/pkgs' [Path]
nwaku>     nimblepath="/opt/nimble/pkgs2/"
nwaku>     ^
nwaku> /build/nim-2.0.12/compiler/nimblecmd.nim(144, 12) compiler msg initiated here [MsgOrigin]
nwaku> /nix/store/hbllc75qwkqkx8r40814bdsgym34gc6x-nim-unwrapped-2.0.12/nim/config/nim.cfg(64, 3) Hint: added path: '/build/source/vendor/.nimble/pkgs2' [Path]
nwaku>     nimblepath="/opt/nimble/pkgs/"
nwaku>     ^
nwaku> /build/nim-2.0.12/compiler/nimblecmd.nim(144, 12) compiler msg initiated here [MsgOrigin]
nwaku> /nix/store/hbllc75qwkqkx8r40814bdsgym34gc6x-nim-unwrapped-2.0.12/nim/config/nim.cfg(64, 3) Hint: added path: '/build/source/vendor/.nimble/pkgs' [Path]
nwaku>     nimblepath="/opt/nimble/pkgs/"
nwaku>     ^
nwaku> /build/nim-2.0.12/compiler/nimblecmd.nim(144, 12) compiler msg initiated here [MsgOrigin]
nwaku> /nix/store/hbllc75qwkqkx8r40814bdsgym34gc6x-nim-unwrapped-2.0.12/nim/config/nim.cfg(68, 1) Hint: added path: '/build/source/vendor/.nimble/pkgs2' [Path]
nwaku>   nimblepath="$home/.nimble/pkgs2/"
nwaku>   ^
nwaku> /build/nim-2.0.12/compiler/nimblecmd.nim(144, 12) compiler msg initiated here [MsgOrigin]
nwaku> /nix/store/hbllc75qwkqkx8r40814bdsgym34gc6x-nim-unwrapped-2.0.12/nim/config/nim.cfg(68, 1) Hint: added path: '/build/source/vendor/.nimble/pkgs' [Path]
nwaku>   nimblepath="$home/.nimble/pkgs2/"
nwaku>   ^
nwaku> /build/nim-2.0.12/compiler/nimblecmd.nim(144, 12) compiler msg initiated here [MsgOrigin]
nwaku> /nix/store/hbllc75qwkqkx8r40814bdsgym34gc6x-nim-unwrapped-2.0.12/nim/config/nim.cfg(69, 1) Hint: added path: '/build/source/vendor/.nimble/pkgs2' [Path]
nwaku>   nimblepath="$home/.nimble/pkgs/"
nwaku>   ^
nwaku> /build/nim-2.0.12/compiler/nimblecmd.nim(144, 12) compiler msg initiated here [MsgOrigin]
nwaku> /nix/store/hbllc75qwkqkx8r40814bdsgym34gc6x-nim-unwrapped-2.0.12/nim/config/nim.cfg(69, 1) Hint: added path: '/build/source/vendor/.nimble/pkgs' [Path]
nwaku>   nimblepath="$home/.nimble/pkgs/"
nwaku>   ^
nwaku> /build/nim-2.0.12/compiler/nimblecmd.nim(144, 12) compiler msg initiated here [MsgOrigin]
nwaku> /build/source/waku.nimble(1, 2) Error: cannot open file: ""
nwaku> /build/nim-2.0.12/compiler/modulepaths.nim(75, 17) compiler msg initiated here [MsgOrigin]
nwaku> make[1]: *** [Makefile:412: build-libwaku-for-android-arch] Error 1
nwaku> make[1]: Leaving directory '/build/source'
nwaku> make: *** [Makefile:420: libwaku-android-arm64] Error 2

Further debugging with strace to see which files were touched on the system, we noticed it is trying to open a nonexisting file which is an empty string:

2249559 newfstatat(AT_FDCWD, "/nix/store/hbllc75qwkqkx8r40814bdsgym34gc6x-nim-unwrapped-2.0.12/nim/lib/impure/formatfloat.nim", 0x7fffffff99e0, 0) = -1 ENOENT (No such file or directory)
2249559 newfstatat(AT_FDCWD, "/nix/store/hbllc75qwkqkx8r40814bdsgym34gc6x-nim-unwrapped-2.0.12/nim/lib/wrappers/formatfloat.nim", 0x7fffffff99e0, 0) = -1 ENOENT (No such file or directory)
2249559 newfstatat(AT_FDCWD, "/nix/store/hbllc75qwkqkx8r40814bdsgym34gc6x-nim-unwrapped-2.0.12/nim/lib/wrappers/linenoise/formatfloat.nim", 0x7fffffff99e0, 0) = -1 ENOENT (No such file or directory)
2249559 newfstatat(AT_FDCWD, "/nix/store/hbllc75qwkqkx8r40814bdsgym34gc6x-nim-unwrapped-2.0.12/nim/lib/windows/formatfloat.nim", 0x7fffffff99e0, 0) = -1 ENOENT (No such file or directory)
2249559 newfstatat(AT_FDCWD, "/nix/store/hbllc75qwkqkx8r40814bdsgym34gc6x-nim-unwrapped-2.0.12/nim/lib/posix/formatfloat.nim", 0x7fffffff99e0, 0) = -1 ENOENT (No such file or directory)
2249559 newfstatat(AT_FDCWD, "/nix/store/hbllc75qwkqkx8r40814bdsgym34gc6x-nim-unwrapped-2.0.12/nim/lib/js/formatfloat.nim", 0x7fffffff99e0, 0) = -1 ENOENT (No such file or directory)
2249559 newfstatat(AT_FDCWD, "/nix/store/hbllc75qwkqkx8r40814bdsgym34gc6x-nim-unwrapped-2.0.12/nim/lib/deprecated/pure/formatfloat.nim", 0x7fffffff99e0, 0) = -1 ENOENT (No such file or directory)
2249559 newfstatat(AT_FDCWD, "", 0x7fffffff99e0, 0) = -1 ENOENT (No such file or directory)
2249559 newfstatat(AT_FDCWD, "/nix/store/hbllc75qwkqkx8r40814bdsgym34gc6x-nim-unwrapped-2.0.12/nim/lib/pure/std/formatfloat.nim", 0x7fffffff98c0, 0) = -1 ENOENT (No such file or directory)
2249559 newfstatat(AT_FDCWD, "/nix/store/hbllc75qwkqkx8r40814bdsgym34gc6x-nim-unwrapped-2.0.12/nim/lib/core/std/formatfloat.nim", 0x7fffffff98c0, 0) = -1 ENOENT (No such file or directory)
2249559 newfstatat(AT_FDCWD, "/nix/store/hbllc75qwkqkx8r40814bdsgym34gc6x-nim-unwrapped-2.0.12/nim/lib/arch/std/formatfloat.nim", 0x7fffffff98c0, 0) = -1 ENOENT (No such file or directory)
2249559 newfstatat(AT_FDCWD, "/nix/store/hbllc75qwkqkx8r40814bdsgym34gc6x-nim-unwrapped-2.0.12/nim/lib/pure/unidecode/std/formatfloat.nim", 0x7fffffff98c0, 0) = -1 ENOENT (No such file or directory)

We added an export of current environment to this line in the nimbus-build-system env script to compare it with a local non-nix build(which is successful) to see what are the differences in terms of nim related environment variables. I've found out that in our nix build there was no: NIMBLE_DIR, NIM_PATH, NIMC and NIMBUS_ENV_DIR. After setting all of these, I've got the same error.

[markoburcul]

Here are the env files to compare them:

• env.nix.initial.fail.log is the nix build without missing NIM-related variables
• env.nix.final.fail.log is the nix build with missing NIM-related variables
• env.local.success.log is the local build without nix

env.nix.final.fail.log
env.nix.initial.fail.log
env.local.success.log

[richard-ramos]

@markoburcul: do we want also do this for iOS?

yes, altho i have not tried compiling libwaku for ios before so I do not know what issues we could run into.

[jakubgs]

According to Dustin the issue also sometimes appears in nimbus-eth2 and usually is resolved by make update which ensures all submodules are fetched and on the correct version, which in a way confirms that this is a dependency related issue.