ICE on extremely large arrays

[3442853561]

When I test this program with cargo test:

struct Bar(Vec<u8>);

impl Bar {
    fn new(values: &[u8]) -> Self {
        Bar(values.into())
    }
    fn get(&self, index: usize) -> u8 {
        self.0[index]
    }
}


#[test]
fn test() {
    let foo = Bar::new(&[0u8; 4294967295]);
    assert_eq!(foo.get(1), 0);
}

fn main() {
    println!("hello world");
}

I got this:

C:\...\test>cargo test
   Compiling foo v0.1.0 (file:///C:/.../test)
error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.23.0-nightly (d6b06c63a 2017-11-09) running on i686-pc-windows-gnu

thread 'rustc' panicked at 'capacity overflow', src\libcore\option.rs:839:4
note: Run with `RUST_BACKTRACE=1` for a backtrace.

error: Could not compile `foo`.

To learn more, run the command again with --verbose.

[sfackler]

That slice is bigger than the 32 bit address space so this isn't going to compile, but we shouldn't panic.

[3442853561]

@sfackler However it actually compiles and panic(s) at runtime

[sfackler]

The error you posted is of the compiler panicking, not your program panicking.

[3442853561]

@sfackler thanks

[estebank]

Oddly enough, if you increase the value significantly and only allocate the slice, there's a non-ICE error that you would expect to see:

error: the type `[u8; 1009999994294967295]` is too big for the current architecture

but with the provided size, the compilation succeeds but panics on execution:

thread 'main' has overflowed its stack
fatal runtime error: stack overflow

[Centril]

This doesn't seem to ICE anymore?

[AdrienChampion]

@Centril I played a bit with this and it does not seem to ICE anymore.

But depending on the actual size, it compiles and then crashes. Here is what @lthls, @codename68 and myself found when we investigated (mostly them TBH).

Let me know if you'd rather have this in a separate issue.

Using this program:

pub const INC: usize = 14;
pub const SIZE: usize = 1 << 32 + INC;

fn main() {
    let big_array: [u8; SIZE] = [0; SIZE];
    for n in 0..SIZE {
        println!("{}", big_array[n])
    }
}

You can compile and crash with signal SIGSEGV (Address boundary error) in debug and release as long as INC < 15:

> cargo run
   Compiling big_array v0.1.0 (...)
    Finished dev [unoptimized + debuginfo] target(s) in 0.28s
     Running `target/debug/big_array`
fish: “cargo run” terminated by signal SIGSEGV (Address boundary error)

When INC ≥ 15, I get the expected error:

error: the type `[u8; 140737488355328]` is too big for the current architecture
 --> src/main.rs:5:9
  |
5 |     let big_array = [0; SIZE];
  |         ^^^^^^^^^

If I just declare big_array, i.e. replace main's body with let _big_array: [u8; SIZE];, it also compiles with INC < 15 but crashes with SIGILL (Illegal instruction). That's only in debug though, since _big_array is optimized away in release.

I seems that, if you only declare the array, you can run code with an erroneous stack pointer, and thus pretty much do anything you want. We will see if we can come up with a small working example, I think @codename68 manages to write in a String he does not own already.

[hanna-kruppe]

Looking over the assembly generated by rustc 1.44.0-nightly (f509b26a7 2020-03-18) on x86_64-unknown-linux-gnu, I see the frame size being truncated to 32 bits when implementing x86_64 stack probing. This means that the stack probing does nothing when the remainder (frame_size as u32) is less than one page. It also messes up the stack pointer adjustment in the prologue (but not in the epilogue).

I can reproduce this behavior independently of Rust with LLVM 9, but not with LLVM 10. So I believe this is an LLVM bug that has been fixed already, and we can either cherry-pick the fix if we can find it, or get this bug fixed automatically when we update to LLVM 10.

[hanna-kruppe]

Note: this explains the reports of runtime crashes, which all appear to assume a 64 bit target. The ICE that was originally reported must have a different cause. Since it occurred on i686-pc-windows-gnu and is obviously related to the size of usize on the target, we should not be sure it has been fixed until someone tries OP's program with a 32 bit target (this notably excludes the playground, which is x86_64).

[saethlin]

we should not be sure it has been fixed until someone tries OP's program with a 32 bit target (this notably excludes the playground, which is x86_64).

OP's program does not compile on a 32-bit target since Rust 1.26: https://godbolt.org/z/1E7MM8sK1

[Dylan-DPC]

Closing this as it's an hard error now as demonstrated in the above link.