switch-to-configuration not interpreted using perl

[eadwu]

Issue description

Running it as an executable returns a bad interpreter error while running it explicitly with perl works fine. Running with bash also replicates the following error.

/nix/store/v0ba874xw18j47f3hisk4130qll3wfrx-nixos-system-nixos-19.03.git.824a927/bin/switch-to-configuration: line 3: use: command not found
/nix/store/v0ba874xw18j47f3hisk4130qll3wfrx-nixos-system-nixos-19.03.git.824a927/bin/switch-to-configuration: line 4: use: command not found
/nix/store/v0ba874xw18j47f3hisk4130qll3wfrx-nixos-system-nixos-19.03.git.824a927/bin/switch-to-configuration: line 5: use: command not found
/nix/store/v0ba874xw18j47f3hisk4130qll3wfrx-nixos-system-nixos-19.03.git.824a927/bin/switch-to-configuration: line 6: use: command not found
/nix/store/v0ba874xw18j47f3hisk4130qll3wfrx-nixos-system-nixos-19.03.git.824a927/bin/switch-to-configuration: line 7: use: command not found
/nix/store/v0ba874xw18j47f3hisk4130qll3wfrx-nixos-system-nixos-19.03.git.824a927/bin/switch-to-configuration: line 8: syntax error near unexpected token `('
/nix/store/v0ba874xw18j47f3hisk4130qll3wfrx-nixos-system-nixos-19.03.git.824a927/bin/switch-to-configuration: line 8: `use Sys::Syslog qw(:standard :macros);'

Steps to reproduce

nixos-rebuild switch

Technical details

Please run nix-shell -p nix-info --run "nix-info -m" and paste the
results.

 - system: `"x86_64-linux"`
 - host os: `Linux 5.0.0-rc1, NixOS, 19.03.git.824a927 (Koi)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.2pre6526_9f99d624`
 - channels(root): `"nixos-19.03pre165037.eebd1a92637"`
 - nixpkgs: `$HOME/Downloads/nixpkgs`

[dtzWill]

I encountered the same when trying out the new 5.0.0-rc1 (!!) kernel, and it appears to have been resolved when reverting.

Does that work for you? If so, then we need to sort out why it's acting this way (config? dunno!)....

[eadwu]

Seems like it is a problem with 5.0.0-rc1, reverted to 4.20.1 and the error doesn't show anymore.

[matthewbauer]

Please report this upstream! I think https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/fs/binfmt_script.c?id=c2315c187fa0d3ab363fdebe22718170b40473e3 is the culprit but don't know enough about the kernel to fix it.

If someone is able to, verify that these cases work:

#!/usr/bin/env perl
use constant;
print 'ok';

and

#! /usr/bin/env perl
use constant;
print 'ok'

[eadwu]

Both don't work if you run them through bash, running them through perl and works though. I'll see if I can compile 5.0-rc2 fast enough before I can goto bed.

Using chmod +x beforehand also seems to make it work.

~/Downloads/tests
➜ ls
test1  test2

~/Downloads/tests
➜ cat test1
#!/usr/bin/env perl
use constant;
print 'ok';

~/Downloads/tests
➜ cat test2
#! /usr/bin/env perl
use constant;
print 'ok'

~/Downloads/tests
➜ bash test1
test1: line 2: use: command not found
test1: line 3: print: command not found

~/Downloads/tests
➜ bash test2
test2: line 2: use: command not found
test2: line 3: print: command not found

~/Downloads/tests
➜ perl test1
ok%

~/Downloads/tests
➜ perl test2
ok%

~/Downloads/tests
➜ chmod +x test1 test2

~/Downloads/tests
➜ ./test1
ok%

~/Downloads/tests
➜ ./test2
ok%

~/Downloads/tests
➜ uname -a
Linux nixos 5.0.0-rc1 #1-NixOS SMP Mon Jan 7 01:08:20 UTC 2019 x86_64 GNU/Linux

Still seems to be a problem on 5.0.0-rc1, using nixos-rebuild switch.

Still occuring on 5.0.0-rc3.

[L-as]

This problem also applies to the command-not-found script. Also, for me running the scripts directly from fish (i.e. /path/to/script and not fish /path/to/script), it will give a more accurate error about the operating system not being able to execute the files. I will post the exact error I get once I can.

[L-as]

This is what happens if I try to run the file directly from a fish shell:

Failed to execute process '/nix/store/8n8cvy1kbqzs13i5b8vnl55q50bjvr4h-nixos-system-las-19.03pre166308.626233eee6e/bin/switch-to-configuration'. Reason:
exec: Exec format error
The file '/nix/store/8n8cvy1kbqzs13i5b8vnl55q50bjvr4h-nixos-system-las-19.03pre166308.626233eee6e/bin/switch-to-configuration' is marked as an executable but could not be run by the operating system.

[dorianr666]

I confirm this bug as well, running linux 5.0.0-rc4.

[dorianr666]

So I noticed that the switch-to-configuration script has very long hashbang line and did some testing. I found out that kernel 5.0.0-rc1 - 5.0.0-rc4 (newest at this moment) starts to fail to execute a script if its hashbang line is longer than exactly 128 bytes (newline included).

Reproduction:
Run any kernel in range 5.0.0-rc1 - 5.0.0-rc4.
Create file script containing:
#!/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa (newline included)
Execute:
$ chmod a+x script
$ ./script
You'll get:

Failed to execute process './script'. Reason:
exec: Exec format error
The file './script' is marked as an executable but could not be run by the operating system.

Now remove one a from the hashbang line and run it again; you'll get the expected:

Failed to execute process './script'. Reason:
The file './script' specified the interpreter '/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', which is not an executable command.

Running both variants of the script on kernel 4.20.5 throws only the second, expected, error. So this looks like userspace-breaking kernel regression, if it affects essential NixOS scripts. Not sure what kernel dev to contact so I'll leave that up to someone else.

PS: I'm using fish shell, error messages in bash may differ but the problem remains.

[dtzWill]

Looks like they're enforcing the limit that was supposed to already be applied?

pypa/pip#1773 (comment)

And indeed looking at our headers I'm seeing:

/* sizeof(linux_binprm->buf) */
#define BINPRM_BUF_SIZE 128

Scrolling to the end it looks like this is getting attention elsewhere recently as well,
likely others encountering same?

I think the commit linked previously is indeed responsible-- or certainly rather relevant (see the commit message!).

Not sure where this leaves us.... :/

[samueldr]

Has anyone contacted upstream yet? Is there a report (maybe from another project) that is relevant to follow?

Fixing it in the nixos kernels is probably not enough; kernel 5.x from other distros may still have issues with Nixpkgs stuff.

[L-as]

@dtzWill: it doesn't matter that it was a bug, kernel policy is nothing breaks userspace. @samueldr I'll try contacting upstream

[L-as]

Well I made a bug report on bugzilla, hopefully I CC'ed the right guy.

[aszlig]

I've bisected this using this VM test and the commit that causes the regression is torvalds/linux@8099b04.

[aszlig]

I also think this could break more than just our use case, especially because Perl seems to expect that the hashbang is truncated by BINPRM_BUF_SIZE. It parses the hashbang and execve()s itself, so that multiple arguments are possible and the length limitation doesn't affect it, so I guess it's expected behavior even on other Unices.

So even if our bug report doesn't get noticed, I'd suspect that this change will get reverted eventually, hopefully before 5.0 gets its final release.

[dtzWill]

4.20.8 maybe has this behavior now too, maybe? Anyway I did the obvious thing and patched `BINPRM_BUF_SIZE` and things are good again--at least as long as I don't use non-NixOS machines or something :P. dtzWill@8dc14c6 and parent (apologies). Not suggesting nixpkgs should adopt this but thought I'd share in case others found things acting strange after a recent update as well ;).

…

On Sat, 02 Feb 2019 15:30:26 -0800, aszlig ***@***.***> wrote: I also think this could break more than just our use case, because while the hashbang is truncated by `BINPRM_BUF_SIZE`. There is a reason why Perl [parses the hashbang and `execve()s` itself](https://perl5.git.perl.org/perl.git/blob/04db542212fdad3a62f13afe741c99028f4bf799:/toke.c#l5524), so that multiple arguments are possible and the length limitation doesn't affect it, so I guess it's expected behavior even on other Unices. So even if our bug report doesn't get noticed, I'd suspect that this change will get reverted eventually, hopefully before 5.0 gets its final release. -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #53672 (comment) part: text/html

[samueldr]

This might be affecting the current LTS (4.19.21) in addition to 4.20.8 (as reported by @dtzWill) and 5.0-rc*.

Assuming the commit tracked down by @aszlig is the right one:

• 4.19.y
• 4.20.y

Found also in

• 4.14.y (v4.14.99)

[grahamc]

Based on the documentation at https://www.kernel.org/doc/linux/REPORTING-BUGS and some extra courage from @mmlb, we've decided @samueldr will write a message to LKML and Linus. I'll send Oleg (author of that patch) an email now, though.

[samueldr]

Mailing list thread:

• https://lkml.org/lkml/2019/2/13/853

[NeQuissimus]

@dtzWill Could we have a PR with the patch? Even if only as a temporary mediation?

[L-as]

@samueldr I think you may want to CC Andrew Morton [email protected], since he signed off on the commit. I'd like to CC him myself, but then I'd have to figure out how to send e-mails to the mailing list first.

[L-as]

Actually, one can just send him an e-mail regarding the matter now that I think about it.
Edit: Well I sent him an e-mail.

[grahamc]

Did any channel bump including these bad kernels?

…

Sent from my iPhone

On Feb 14, 2019, at 08:33, Tim Steinbach ***@***.***> wrote: Oh, I see... Although from what I can tell, Kees' patch is more or less that — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[NeQuissimus]

Only the small channels have seen a bump (just looking at http://howoldis.herokuapp.com/)

[grahamc]

The nixos-unstable-small and nixos-18.09-small channel URLs are being rolled back.

[NeQuissimus]

I can now confirm that #55763 fixes the immediate issue. Built 4.20.8-hardened and booted into it. nixos-rebuild switch --upgrade -I nixpkgs=... still runs fine.

[NeQuissimus]

Does anybody feel strongly about which fix we apply as long as we have something for the moment?

[grahamc]

I think we should do this multi-pronged:

1. fix our perl shebangs to not be massive: not everybody runs nix on nixos
2. apply what looks like the most likely upstream patch

[L-as]

I think the safest choice should be taken, although a proper fix, i.e. this patch is preferable in the end.

[L-as]

good point @grahamc, but how would you make the shebangs smaller?

[grahamc]

Hydra already does this with buildEnv. Nice perk of this is we get a much faster perl startup time: https://github.com/NixOS/hydra/blob/master/release.nix#L47

[NeQuissimus]

Somebody make an executive decision or I merge #55763 :D
I don't think we should just sit on this for too long...

[grahamc]

I'd like for at least one other person to take a look at #55763 -- kernel upstream or otherwise, but it is my preferred patch.

[grahamc]

Should we leave this one open and pinned for a bit, until channels advance?

[grahamc]

FYI: Our revert patch can be deleted, as upstream has released new, fixed kernels:

• https://cdn.kernel.org/pub/linux/kernel/v4.x/ChangeLog-4.9.158
• https://cdn.kernel.org/pub/linux/kernel/v4.x/ChangeLog-4.14.101
• https://cdn.kernel.org/pub/linux/kernel/v4.x/ChangeLog-4.19.23
• https://cdn.kernel.org/pub/linux/kernel/v4.x/ChangeLog-4.20.10

:)

[NeQuissimus]

I am on it!

[samueldr]

Should we leave this one open and pinned for a bit, until channels advance?

I didn't know for sure, and when I saw the title of the issue, once pinned, I got cold feet in re-writing the title and editing the main post to make it more useful for external users. Though, I'm thinking it could be a good idea if done. Though it doesn't necessarily needs to be kept open, only visible?

[NeQuissimus]

master: 50f518c 7954ec0 8c14948 8c14948

release-18.09: f0ce0f3 6e90746 6da8b83 0ba800c

[vcunat]

Let me link the upstream summary of this: https://lwn.net/SubscriberLink/779997/11de2bdc8dbc0d69/ (taken from weekly.nixos.org)

[domenkozar]

I've seen this also with NixOS 20.09 and kernel 5.4.77.

[samueldr]

Many issues could end up causing similar behaviour. It might not be that the kernel again decided to stop interpreting the shebang in its entirety.

I guess you'll need to have a minimal repro, so we can see what's going on. Shouldn't it fail on Hydra too, if it is a reproducible issue? I don't remember if the issue eluded our tests at the time. Though it's likely to be a new issue, rather than linked to a now year old issue. Or else we'll have to have strong words with the kernel, again. Though, check with the most up to date kernel, I'm running 5.4 past .77 and I haven't had this issue.

[domenkozar]

Repro is to deploy a new Hetzner machine with NixOS 20.09.

I have only helped a friend get things going, so I didn't spend much time investigating what's going on this time.