x86_64-unknown-linux-musl links to Glibc With C++ Dependencies

[Alexhuszagh]

Checklist

• I've looked through the issues and pull requests for similar reports

Describe your issue

UPDATE: All targets except x86_64-unknown-linux-musl have been patched as of #905, and the MIPS64 targets in #906.

When compiling for some musl targets, they link to dynamically to glibc rather than musl libc statically. This only occurs with C++ dependencies. A simple test is as follows:

$ git clone https://github.com/cross-rs/rust-cpp-hello-word
$ cd rust-cpp-hello-word
$ cross build --target aarch64-unknown-linux-musl
   Compiling cc v1.0.73
   Compiling hellopp v0.1.0 (/home/ahuszagh/Desktop/cross/rust-cpp-hello-word)
    Finished dev [unoptimized + debuginfo] target(s) in 2.58s

$ file target/aarch64-unknown-linux-musl/debug/hellopp
target/aarch64-unknown-linux-musl/debug/hellopp: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, with debug_info, not stripped

Some targets, such as armv7-unknown-linux-musleabihf link properly, however, while running, they cannot find the correct dynamic library loader.

This is related to #101, although not exactly.

What target(s) are you cross-compiling for?

aarch64-unknown-linux-musl

Which operating system is the host (e.g computer cross is on) running?

• macOS
• Windows
• Linux / BSD
• other OS (specify in description)

What architecture is the host?

• x86_64 / AMD64
• arm32
• arm64 (including Mac M1)

What container engine is cross using?

• docker
• podman
• other container engine (specify in description)

cross version

cross 0.2.2 (latest main)

Example

No response

Additional information / notes

No response

[Alexhuszagh]

So the two targets that fail are:

• aarch64-unknown-linux-musl
• x86_64-unknown-linux-musl

ARM64 musl is actually fine: it just wants to name what it links against as a glibc loader, but it only works with a musl loader. x86_64 just doesn't work, nor will it.

Likely the only real solution long-term, if this target with C++ is desired enough, is to make x86_64-unknown-linux-musl an Alpine image, likely with a sub (so x86_64-unknown-linux-musl.alpine).

[Emilgardis]

should we include this in 0.2.3?

[Alexhuszagh]

It shouldn't be too much work, so I think that's fine. We could push it back thought to 0.3.0, since x86_64 Alpine build systems are not too difficult to setup.

[Alexhuszagh]

It might be worth adding a warning for this target, since it seems to fail with dylib. I might check to ensure it works with static linking C++ dependencies.

[Alexhuszagh]

A little bit more information on the x86_64-unknown-linux-musl image. We can determine that the toolchain itself is built incorrectly, specifically, the C++ standard library.

Issue

Let's try to build and run a Rust executable with a C++ dependency.

$ git clone https://github.com/cross-rs/rust-cpp-hello-word
$ cd rust-cpp-hello-word
$ cross run --target x86_64-unknown-linux-musl
    Finished dev [unoptimized + debuginfo] target(s) in 0.13s
     Running `/qemu-runner x86_64 /target/x86_64-unknown-linux-musl/debug/hellopp`
Segmentation fault (core dumped)

Let's also try to run the binary natively on Alpine:

$ docker run -it --rm -v "$PWD/target/x86_64-unknown-linux-musl":/target alpine:latest sh
$ target/debug/hellopp
Segmentation fault (core dumped)

So the binary is still failing, even when it's run on Alpine.

Basis

First, let's just ensure that we're on the same page: the following program headers and sections are fine:

NOTE: having __gnu_cxx symbols is expected for both, as is having
.gnu.hash for dynamic libraries, as are the following program headers:

• GNU_PROPERTY
• GNU_EH_FRAME
• GNU_STACK
• GNU_RELRO

For static libraries, the following program headers are normal:

• GNU_PROPERTY
• GNU_STACK
• GNU_RELRO
• GNU

So just having some GNU sections, program headers, or symbols is normal.

The Problem

Let's download our x86_64 musl toolchain and try to build a C++ executable from it:

# outside the container
$ docker pull ghcr.io/cross-rs/x86_64-unknown-linux-musl:main
$ docker run -it --rm ghcr.io/cross-rs/x86_64-unknown-linux-musl:main bash

# inside the container
$ echo '#include <cstdio>
#include <iostream>
int main() {
  printf("Hello World!\n");
  std::cout << "Hello World!" << std::endl;
  return 0;
}
' > main.cc

# DYNAMIC
# this links to libstdc++ built using GLIBCXX symbols.
$ $CXX_x86_64_unknown_linux_musl main.cc -o musl_dynamic
$ readelf -a musl_dynamic | grep GLIBCXX | head -n 1
000000601080  000c00000005 R_X86_64_COPY     0000000000601080 _ZSt4cout@GLIBCXX_3.4 + 0
# however, no libc built against glibc symbols are found
$ readelf -a musl_dynamic | grep GLIBC | grep puts | head -n 1

# STATIC
# we probably have the same issue with the static build, but it's
# trickier to determine, since the symbols aren't versioned.
$CXX_x86_64_unknown_linux_musl main.cc -static -o musl_static

This unfortunately seems to be a broader issues than just x86_64:

# outside the container
$ docker pull ghcr.io/cross-rs/aarch64-unknown-linux-musl:main
$ docker run -it --rm ghcr.io/cross-rs/aarch64-unknown-linux-musl:main bash

# inside the container
$ echo '#include <cstdio>
#include <iostream>
int main() {
  printf("Hello World!\n");
  std::cout << "Hello World!" << std::endl;
  return 0;
}
' > main.cc

# DYNAMIC
$ $CXX_aarch64_unknown_linux_musl main.cc -o musl_dynamic
$ readelf -a musl_dynamic | grep GLIBCXX | head -n 1
000000411058  000f00000400 R_AARCH64_COPY    0000000000411058 _ZSt4cout@GLIBCXX_3.4 + 0

This mostly isn't an issue, since the existing libstdc++ is generally a drop-in replacement, however, we've had a few issues running binaries and these resulting binaries don't play nicely on Alpine. From the musl FAQ:

Be aware that, "out of the box", the wrapper only supports C applications, not C++. This is because the C++ libraries and headers are missing from the musl include/library path. The existing libstdc++ is actually compatible with musl in most cases and could be used by copying it into the musl library path, but the C++ header files are usually not compatible. One option may be rebuilding just libstdc++ against musl; however, if C++ support is needed, it's recommended just to build a native toolchain targetting musl.

Summary

In short, we probably need to update our toolchains to build libstdc++ against musl. None of these issues exist with Alpine-proper, which has a libstdc++ built against musl. It mostly works, but that's not necessarily good enough.

[Alexhuszagh]

I've tried deversioning them by compiling the archive to a shared library, and nothing seems to work. It might be worth looking into how Alpine builds their toolchain.

[Alexhuszagh]

I should have a fix for this out today. It turns out, compiling rust-cpp-hello-word on Alpine itself segfaults. That being said, for anything besides C++ dependencies, Rust on Alpine links statically to libc with static-pie linkage (both for Rust-only packages and those with external C dependencies), so our solution is quite simple: use --enable-default-pie and --disable-shared.