macos: dyld: Library not loaded: @rpath/libtest-77ee8c29c330e4a3.dylib

[steveej]

Note from maintainer: Here's an outline of the work required for this. Please dive in if you're interested!

---

we're having issues running cargo nextest in a nix-shell on a macos github actions runner.
we run the same steps on an ubuntu runner in parallel which succeeds, so this seems to be macos specific.

it may or may not have something to do that the tested source code is under /tmp, which on this macos runner has an implicit /private/ prefix.

some references for context:

• a script that runs cargo nextest
• a recent actions log of which the final output is pasted below

(...)
dyld: Library not loaded: @rpath/libtest-77ee8c29c330e4a3.dylib
  Referenced from: /private/tmp/holochain_repo/target/fast-test/deps/hdk_derive-73ec051829ad694a
  Reason: image not found
error: creating test list failed
Caused by:
  for `hdk_derive::proc-macro/hdk_derive`, running command `/private/tmp/holochain_repo/target/fast-test/deps/hdk_derive-73ec051829ad694a --list --format terse` failed
Caused by:
  command ["/private/tmp/holochain_repo/target/fast-test/deps/hdk_derive-73ec051829ad694a", "--list", "--format", "terse"] exited with code <signal 6>
Error: Process completed with exit code 104.

please let me know if i can provide further debug output to help us understand what's happening.

[sunshowers]

Thanks. This looks like an issue with a dynamic library provided by Rust, presumably created at build time, not being found at runtime.

How is cargo installed? I'm guessing a rustup based installation sets particular environment variables that a nix install doesn't.

[sunshowers]

OK, I know what's going on—we need to grab the value of "rustc --print sysroot" and set the sysroot path as part of LD_FALLBACK_LIBRARY_PATH properly. Will fix in the next couple of days or you're welcome to fix it as well.

[sunshowers]

This method needs to be updated to also grab the sysroot:

nextest/nextest-runner/src/list/test_list.rs

Line 385 in 3ab960d

pub(crate) fn create_dylib_path(

[steveej]

thanks for looking into this @sunshowers.

How is cargo installed?

it's provided by a nix-shell derivation which makes use of functions from here: https://github.com/oxalica/rust-overlay/.

I'm guessing a rustup based installation sets particular environment variables that a nix install doesn't.

this is possible and could be easily resolved in the nix-shell setup as well.

i'm puzzled by the difference in linux and macos. the nix-shell input is identical for both environments. are you aware of any rust specific platform differences that influences what cargo-nextest sees?

[sunshowers]

I'm not sure what nix is doing, but it does look like they had to work around a very similar issue on just macOS here: https://github.com/oxalica/rust-overlay/blob/44801306a2aa0e9aaa47588d615ce6df4acf18c6/mk-component-set.nix#L89

[sunshowers]

This is exactly what we would have to fix, except we'll use DYLD_FALLBACK_LIBRARY_PATH rather than adding the rpath to the binary.

[steveej]

looks like there's already a TODO for this in the codebase:

nextest/nextest-runner/src/list/rust_build_meta.rs

Line 88 in e7f1035

// TODO: rustc sysroot library path? is this even possible to get?

[sunshowers]

Yeah that definitely looks like it.

I'm not sure how this would work with reused builds. We currently promise not to require rustc, but we may not be able to uphold that promise when dealing specifically with proc-macro tests.

[steveej]

This is exactly what we would have to fix, except we'll use DYLD_FALLBACK_LIBRARY_PATH rather than adding the rpath to the binary.

it seems like this isn't effective. i ran the following on the github macos runner and it still produced the same error:

Mac-1654809093762:holochain runner$ nix-shell --pure --run "env DYLD_FALLBACK_LIBRARY_PATH=$(rustc --print sysroot) hc-test-standard-nextest"
Using /Users/runner/work/holochain/holochain as target prefix...
(...)
+ cargo nextest run --test-threads=2 --workspace --exclude holochain --lib --tests --cargo-profile fast-test
    Finished fast-test [unoptimized + debuginfo] target(s) in 1.10s
dyld: Library not loaded: @rpath/libtest-77ee8c29c330e4a3.dylib
  Referenced from: /Users/runner/work/holochain/holochain/target/fast-test/deps/hdk_derive-73ec051829ad694a
  Reason: image not found
error: creating test list failed

Caused by:
  for `hdk_derive::proc-macro/hdk_derive`, running command `/Users/runner/work/holochain/holochain/target/fast-test/deps/hdk_d73ec051829ad694a --list --format terse` failed

Caused by:
  command ["/Users/runner/work/holochain/holochain/target/fast-test/deps/hdk_derive-73ec051829ad694a", "--list", "--format", "] exited with code <signal 6>

on the positive side, the path looks healthy:

$ nix-shell --pure --run 'ls -lha $(rustc --print sysroot)/lib/libtest-77ee8c29c330e4a3.dylib'
Using /Users/runner/work/holochain/holochain as target prefix...
lrwxr-xr-x 1 root wheel 111 Jan  1  1970 /nix/store/qbci06vgxhgxzhjyi9rflgiikr27dmhy-rust-default-1.60.0/lib/libtest-77ee8c29c330e4a3.dylib -> /nix/store/88lhjdgxdkdwdichnji1r79lp25rdrwb-rustc-1.60.0-x86_64-apple-darwin/lib/libtest-77ee8c29c330e4a3.dylib

i'll look into whether the nextest-runner does not forward this environment variable to the test binary, which would be an explanation. or maybe we do need to modify the rpath after all.

---

EDIT:

whops, i used " instead of ' in the first command. so the shell expanded the command before entering the nix-shell and setting the env var. i repeated the command properly and it still fails in the same way so the above is still relevant.

[sunshowers]

You want DYLD_FALLBACK_LIBRARY_PATH=$(rustc --print sysroot)/lib

[steveej]

You want DYLD_FALLBACK_LIBRARY_PATH=$(rustc --print sysroot)/lib

with this correct path the workaround described above works indeed, thank you 🙌

now, i think it's worth putting a best-effort solution into cargo-nextest the way you described.

trying to think about how we could solve this without shelling out and thus depending on rustc.
in the special case when the runner is executed using cargo nextest, could query cargo for this info? OTOH, the result would still depend on rustc with its sysroot being installed so the dependency is there regardless.

[sunshowers]

Agreed about supporting it properly by updating the sysroot, though given the complexity and edge case nature here (only occurs for proc-macro tests + not using cargo from rustup) I'm going to prioritize it downwards because a workaround exists for now.

However, if someone would like to send a PR then that would be really appreciated as well!

I'll outline the main things that need to be done, for both someone who would like to contribute a PR and also for my future self:

1. Grab the host and target rust libdirs (turns out a better workaround is $(rustc --print target-libdir)) run with and without the --target option for host and target. Note while running rustc that it should look at $RUSTC first, just like our Cargo executions look at $CARGO first.
2. Add them to the dylib path -- should dedup the dylib paths by using an IndexSet maybe.
3. Add the following to RustBuildMeta and its serialized form RustBuildMetaSummary:
   a. The name of the target
   b. Names of libraries found at these paths (all files other than the ones ending in .rlib) (use just the file name).
4. While creating archives, serialize them to a path e.g. target/nextest/_libdir.
5. While reusing builds, use the libdirs in this order:
   a. First, prefer the libdir at the target/nextest path if the files within it exist. (This will need to obey target-dir remapping.)
   b. Then, if those aren't found, invoke rustc to get the libdir. There are three cases here:
   1. rustc is found. In this case, check to see if the file names exist. If they don't or the names are different, warn that some tests may not be able to be run.
   2. rustc exited with a non-zero status code. This should warn same as above.
   3. rustc is not found. In this case, produce a message with info!() saying that some tests may not be successful at running.

For testing:

• Unit tests: e.g. for deduping paths -- we likely also want to build this in a way that rustc's output is mockable.

• Integration tests: in CI, we basically want to run the fixture tests while bypassing rustup's cargo. The way to do this is essentially to use:

  #!/bin/bash
  set -e -o pipefail
  pushd "$(git rev-parse --show-toplevel)" 
  
  export CARGO="$(rustup which cargo)"
  # Required for dylib linking
  export RUSTFLAGS="-C prefer-dynamic"
  
  target/debug/cargo-nextest nextest run --manifest-path fixtures/nextest-tests/Cargo.toml --workspace
  # https://docs.rs/nextest-metadata/latest/nextest_metadata/enum.NextestExitCode.html#associatedconstant.TEST_RUN_FAILED
  if [[ $? -eq 100 ]]; then
      echo "test run failed as expected"
  else
      echo "expected exit code 100 (test run failed), found $?"
      exit 1
  fi

And a similar thing should be done for reused builds.

This is hard to do within the current nextest suite because it runs under cargo (we want to minimize the things that can add a libdir to the dylib path), so it should should be a script within the scripts directory so it can be tested outside of CI as well.

This is likely 2-3 days of work that I think can be done by a contributor willing to dive into this!

[gakonst]

@DaniPopes if you'd be down to do this, go for it

[NobodyXu]

@sunshowers I've also encountered this when running:

export CARGO='cargo'

cargo-nextest nextest run --workspace  --target x86_64-unknown-linux-gnu --no-default-features --features rustls,fancy-with-backtrace,zstd-thin,log_release_max_level_debug,pkg-config

in our CI.

However, if I run:

export CARGO='cargo'

cargo nextest run --workspace  --target x86_64-unknown-linux-gnu --no-default-features --features rustls,fancy-with-backtrace,zstd-thin,log_release_max_level_debug,pkg-config

then it would work fine, presumably because cargo overwrites environment variable CARGO.

Also, I'm now trying to get cargo-nextest to work with cargo-zigbuild for cross compilation.
When cross compiling, we set export CARGO=cargo-zigbuild to use cargo-zigbuild instead of cargo for building binaries.

[sunshowers]

All right, so as the flurry of activity above suggests, we ended up being forced to fix this due to a rustup behavior change on Windows.

• Most of the work was done by @06393993 in Add rustc folder to dynamic linker search path #1499 -- thanks!
• More work by myself in [nextest-runner] tidy up libdir support #1511 and [nextest-runner] archive and reuse stdlibs #1514.

That should address this class of issues comprehensively, in all situations including archives (we now include libstd in archives -- unconditionally for now, but we could get smarter and/or add config knobs in the future).

After #1514, nextest will no longer depend on rustup. It'll continue to depend on cargo, of course, until #842 is resolved.

[sunshowers]

cargo-nextest 0.9.72 is out with a fix for this. Try it out!