Investigate flaky test-stream-pipe-same-destination-twice on SmartOS

[Trott]

• Version: v8.0.0-pre
• Platform: smartos16-64
• Subsystem: test, stream

From https://ci.nodejs.org/job/node-test-commit-smartos/9202/nodes=smartos16-64/console:

not ok 1006 parallel/test-stream-pipe-same-destination-twice
  ---
  duration_ms: 0.592
  severity: crashed
  stack: |-
    oh no!
    exit code: CRASHED (Signal: 4)

/cc @nodejs/platform-smartos

[misterdjules]

@nodejs/build The process terminating due to receiving SIGILL should have made it generate a core dump. Is it possible for you to grab that core file from the test-joyent-smartos16-x64-1 VM (it should be somewhere in the /home/iojs/cores/ directory) and upload it somewhere?

[refack]

@misterdjules AFAIK the fs state is ephemeral, the next run would have cleared the tree 😞
I started a stress test, might reproduce: https://ci.nodejs.org/job/node-stress-single-test/1233/nodes=smartos15-64/

[misterdjules]

@refack

AFAIK the fs state is ephemeral

What do you mean by "the fs state is ephemeral"? Is /home/iojs/cores/ cleaned up before and/or after every tests run? That was not my understanding.

[refack]

@refack

AFAIK the fs state is ephemeral
What do you mean by "the fs state is ephemeral"? Is /home/iojs/cores/ cleaned up before and/or after every tests run? That was not my understanding.

Yes you are right. It's just the build dir that gets cleaned up. I didn't read the fine print.

[gibfahn]

@misterdjules I'm not seeing that folder on test-joyent-smartos16-x64-1 (the machine that failed):

➜  ~ ❯ s test-joyent-smartos16-x64-1 ls /home/iojs                                                                                                                                                        ~
build
jenkins_diagnostics.txt
jenkins_manifest.xml
node-tmp
slave.jar

[misterdjules]

@gibfahn

I'm not seeing that folder on test-joyent-smartos16-x64-1

In this case core dumps won't be written to that directory. It needs to be created.

[gibfahn]

Okay, I've added the folder to all the test machines. This probably needs something in our ansible scripts to automate it, but this works for now.

➜  ~ ❯ for i in `cat ~/.ssh/config | grep smartos | grep test | awk '{print $2}'`; do echo ">>> $i"; ssh $i ls -al /home/iojs/cores; done
>>> test-joyent-smartos14-x64-1
total 2
drwxr-xr-x  2 iojs iojs   2 May 26 09:18 .
drwxr-xr-x 11 iojs other 26 May 26 09:18 ..
>>> test-joyent-smartos14-x64-2
total 2
drwxr-xr-x  2 iojs iojs   2 May 26 09:18 .
drwxr-xr-x 10 iojs other 22 May 26 09:18 ..
>>> test-joyent-smartos14-x86-1
total 2
drwxr-xr-x  2 iojs iojs   2 May 26 09:19 .
drwxr-xr-x 10 iojs other 26 May 26 09:19 ..
>>> test-joyent-smartos14-x86-2
total 2
drwxr-xr-x  2 iojs iojs   2 May 26 09:19 .
drwxr-xr-x 10 iojs other 22 May 26 09:19 ..
>>> test-joyent-smartos15-x64-1
total 2
drwxr-xr-x  2 iojs iojs   2 May 26 09:19 .
drwxr-xr-x 12 iojs other 26 May 26 09:19 ..
>>> test-joyent-smartos15-x64-2
total 2
drwxr-xr-x  2 iojs iojs   2 May 26 09:20 .
drwxr-xr-x 12 iojs other 26 May 26 09:20 ..
>>> test-joyent-smartos16-x64-1
total 2
drwxr-xr-x  2 iojs iojs   2 May 26 09:10 .
drwxr-xr-x 11 iojs other 25 May 26 09:13 ..
>>> test-joyent-smartos16-x64-2
total 2
drwxr-xr-x  2 iojs iojs  2 May 26 09:20 .
drwxr-xr-x 11 iojs iojs 24 May 26 09:20 ..

If it fails again LMK and I'll get the coredump.

[misterdjules]

@gibfahn Thank you for doing that!

[refack]

After nodejs/build#736 has been addressed, Stressing smartOS16: https://ci.nodejs.org/job/node-stress-single-test/1247/nodes=smartos16-64/

[Trott]

More than six months without a re-occurrence. Going to close this...