STATUS_HEAP_CORRUPTION in rust-lld with a custom target on x86_64-pc-windows-gnu

[ehuss]

The following example has recently started causing rust-lld to crash on x86_64-pc-windows-gnu:

#![no_main]
#![allow(internal_features)]
#![feature(no_core)]
#![feature(lang_items)]
#![no_core]

#[lang = "sized"]
pub trait Sized {
    // Empty.
}
#[lang = "copy"]
pub trait Copy {
    // Empty.
}

with a custom JSON target:

{
    "llvm-target": "x86_64-unknown-none-gnu",
    "data-layout": "e-m:e-i64:64-f80:128-n8:16:32:64-S128",
    "arch": "x86_64",
    "target-endian": "little",
    "target-pointer-width": "64",
    "target-c-int-width": "32",
    "os": "none",
    "linker-flavor": "ld.lld",
    "linker": "rust-lld",
    "executables": true
}

Building with:

rustc --target custom-target.json main.rs

This causes rust-lld to fail about 50% of the time, with errors like:

error: linking with `rust-lld` failed: exit code: 0xc0000374
  |
  = note: "rust-lld" "-flavor" "gnu" "main.main.4fa2520d328b6ba3-cgu.0.rcgu.o" "--as-needed" "-L" "C:\\Users\\eric\\.rustup\\toolchains\\nightly-2023-09-01-x86_64-pc-windows-gnu\\lib\\rustlib\\custom-target\\lib" "-Bdynamic" "--eh-frame-hdr" "-z" "noexecstack" "-L" "C:\\Users\\eric\\.rustup\\toolchains\\nightly-2023-09-01-x86_64-pc-windows-gnu\\lib\\rustlib\\custom-target\\lib" "-o" "main" "--gc-sections"
  = note: rust-lld: warning: cannot find entry symbol _start; not setting start address

error: aborting due to previous error

or

error: linking with `rust-lld` failed: exit code: 0xc0000005
  |
  = note: "rust-lld" "-flavor" "gnu" "main.main.4fa2520d328b6ba3-cgu.0.rcgu.o" "--as-needed" "-L" "C:\\Users\\eric\\.rustup\\toolchains\\nightly-2023-09-01-x86_64-pc-windows-gnu\\lib\\rustlib\\custom-target\\lib" "-Bdynamic" "--eh-frame-hdr" "-z" "noexecstack" "-L" "C:\\Users\\eric\\.rustup\\toolchains\\nightly-2023-09-01-x86_64-pc-windows-gnu\\lib\\rustlib\\custom-target\\lib" "-o" "main" "--gc-sections"
  = note: rust-lld: warning: cannot find entry symbol _start; not setting start address
          PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
          Stack dump:
          0.    Program arguments: rust-lld -flavor gnu main.main.4fa2520d328b6ba3-cgu.0.rcgu.o --as-needed -L C:\\Users\\eric\\.rustup\\toolchains\\nightly-2023-09-01-x86_64-pc-windows-gnu\\lib\\rustlib\\custom-target\\lib -Bdynamic --eh-frame-hdr -z noexecstack -L C:\\Users\\eric\\.rustup\\toolchains\\nightly-2023-09-01-x86_64-pc-windows-gnu\\lib\\rustlib\\custom-target\\lib -o main --gc-sections
          Exception Code: 0xC0000005
          0x00007FFDD755633C, C:\WINDOWS\SYSTEM32\ntdll.dll(0x00007FFDD7530000) + 0x2633C byte(s), RtlGetCurrentServiceSessionId() + 0x1AEC byte(s)
          0x00007FFDD7555B74, C:\WINDOWS\SYSTEM32\ntdll.dll(0x00007FFDD7530000) + 0x25B74 byte(s), RtlGetCurrentServiceSessionId() + 0x1324 byte(s)
          0x00007FFDD75547B1, C:\WINDOWS\SYSTEM32\ntdll.dll(0x00007FFDD7530000) + 0x247B1 byte(s), RtlFreeHeap() + 0x51 byte(s)
          0x00007FFDD5609C9C, C:\WINDOWS\System32\msvcrt.dll(0x00007FFDD55F0000) + 0x19C9C byte(s), free() + 0x1C byte(s)
          0x00007FFDD5609366, C:\WINDOWS\System32\msvcrt.dll(0x00007FFDD55F0000) + 0x19366 byte(s), _aligned_free() + 0x16 byte(s)
          0x00007FF74451890A, C:\Users\eric\.rustup\toolchains\nightly-2023-09-01-x86_64-pc-windows-gnu\lib\rustlib\x86_64-pc-windows-gnu\bin\rust-lld.exe(0x00007FF741000000) + 0x351890A byte(s)
          0x00007FF742807BF9, C:\Users\eric\.rustup\toolchains\nightly-2023-09-01-x86_64-pc-windows-gnu\lib\rustlib\x86_64-pc-windows-gnu\bin\rust-lld.exe(0x00007FF741000000) + 0x1807BF9 byte(s)
          0x00007FF7426D9634, C:\Users\eric\.rustup\toolchains\nightly-2023-09-01-x86_64-pc-windows-gnu\lib\rustlib\x86_64-pc-windows-gnu\bin\rust-lld.exe(0x00007FF741000000) + 0x16D9634 byte(s)
          0x00007FF742256947, C:\Users\eric\.rustup\toolchains\nightly-2023-09-01-x86_64-pc-windows-gnu\lib\rustlib\x86_64-pc-windows-gnu\bin\rust-lld.exe(0x00007FF741000000) + 0x1256947 byte(s)
          0x00007FF74208C943, C:\Users\eric\.rustup\toolchains\nightly-2023-09-01-x86_64-pc-windows-gnu\lib\rustlib\x86_64-pc-windows-gnu\bin\rust-lld.exe(0x00007FF741000000) + 0x108C943 byte(s)
          0x00007FF74106032E, C:\Users\eric\.rustup\toolchains\nightly-2023-09-01-x86_64-pc-windows-gnu\lib\rustlib\x86_64-pc-windows-gnu\bin\rust-lld.exe(0x00007FF741000000) + 0x6032E byte(s)
          0x00007FF74457EB45, C:\Users\eric\.rustup\toolchains\nightly-2023-09-01-x86_64-pc-windows-gnu\lib\rustlib\x86_64-pc-windows-gnu\bin\rust-lld.exe(0x00007FF741000000) + 0x357EB45 byte(s)
          0x00007FF741001395, C:\Users\eric\.rustup\toolchains\nightly-2023-09-01-x86_64-pc-windows-gnu\lib\rustlib\x86_64-pc-windows-gnu\bin\rust-lld.exe(0x00007FF741000000) + 0x1395 byte(s)
          0x00007FF7410014E6, C:\Users\eric\.rustup\toolchains\nightly-2023-09-01-x86_64-pc-windows-gnu\lib\rustlib\x86_64-pc-windows-gnu\bin\rust-lld.exe(0x00007FF741000000) + 0x14E6 byte(s)
          0x00007FFDD6CF7614, C:\WINDOWS\System32\KERNEL32.DLL(0x00007FFDD6CE0000) + 0x17614 byte(s), BaseThreadInitThunk() + 0x14 byte(s)
          0x00007FFDD75826B1, C:\WINDOWS\SYSTEM32\ntdll.dll(0x00007FFDD7530000) + 0x526B1 byte(s), RtlUserThreadStart() + 0x21 byte(s)

This is part of cargo's testsuite, which is causing issues in our CI.

I have bisected the regression to #114048 (starting nightly-2023-08-09). cc @nikic

It doesn't seem to be happening on any other platforms.

Meta

rustc --version --verbose:

rustc 1.74.0-nightly (203c57dbe 2023-09-17)
binary: rustc
commit-hash: 203c57dbe20aee67eaa8f7be45d1e4ef0b274109
commit-date: 2023-09-17
host: x86_64-pc-windows-gnu
release: 1.74.0-nightly
LLVM version: 17.0.0

[cr1901]

Maybe this is another useful data point, but I can duplicate with the following from riscv-rt:

RUSTFLAGS="-C link-arg=-Texamples/device.x" cargo build --example empty --target=riscv32i-unknown-none-elf

Sometimes it succeeds, sometimes it crashes, sometimes it hangs.

I found this line in my global cargo config from a while ago:

[target.riscv32i-unknown-none-elf]
rustflags = [ "-C", "link-arg=--threads=1" ]

Unfortunately, I don't remember where I learned this- there's probably a Zulip thread- but multithreaded lld is (was?) known to be buggy and hang a lot on Windows, and there's not been much interest in fixing it on the LLD side. So I was a bit surprised to see this issue appear after a while not needing lld.

Apparently, setting RUSTFLAGS on the command line for quick testing overrides my config. When I add the link-arg flag back in, the crashes go away for me:

RUSTFLAGS="-C link-arg=--threads=1 -C link-arg=-Texamples/device.x" cargo build --example empty --target=riscv32i-unknown-none-elf

William@DESKTOP-3H1DSBV MINGW64 ~/Projects/rust-embedded/riscv-rt
$ rustc -Vv
rustc 1.73.0 (cc66ad468 2023-10-03)
binary: rustc
commit-hash: cc66ad468955717ab92600c770da8c1601a4ff33
commit-date: 2023-10-03
host: x86_64-pc-windows-gnu
release: 1.73.0
LLVM version: 17.0.2

[mati865]

This shouldn't happen since GCC update in #119229
Is this still an issue?

[mati865]

Answering myself: yes, it still fails on CI.

[jieyouxu]

Has also been crashing on rust-lang/rust windows-gnu CI quite frequently, cf. #137938

[mati865]

Yeah, I meant Rust's CI.

[ChrisDenton]

I think the frequency may have been increasing lately. In the last 24hrs #140356 and #123239 (twice).

[ChrisDenton]

Looking back through the logs it does seem we are indeed hitting this more often in recent days:

February (1):

• Build GCC on CI #136921

March (3):

• Compile run-make-support and run-make tests with the bootstrap compiler #137373
• resolve: Avoid some unstable iteration 2 #138580
• Remove terminating_scopes hash set. #139067

April (10):

• [beta-1.87] Update cargo #139463
• Rollup of 9 pull requests #139912
• Rollup of 4 pull requests #140134
• Autodiff flags #139700
• Rollup of 8 pull requests #140165
• Revert compiletest new-executor, to re-land without download-rustc #140233
• [beta] backports #140269
• Rollup of 8 pull requests #140356
• Implement a lint for implicit autoref of raw pointer dereference - take 2 #123239 (x2)