Tests for atom-shell startup

[springmeyer]

We learned recently (#1166) that it is possible for all appveyor tests to pass even if the the windows app is completely unable to start. In the case of #1166 I think was happened was the server process started up not the shell (which left the server as a zombie).

To protect against releasing another broken build we need to:

• Dig into what happened in more detail in order to figure out where the missing test coverage is
• Add tests that cover the previous case (for tilemill we used to test app startup like this)
• Try reverting 5a5e753 and ensure the tests catch this fail on windows
• Try creating testcases emulating an app that cannot startup on linux and ensure travis tests can catch this.

As far as finding a fix, so that 5a5e753 can be safely reverted (and shell.log logging restored for windows) we are still empty handed. As far as I know the problem is still a mystery. My hunch is that we are being bitten by nodejs/node-v0.x-archive#3584. If this is the case then node v0.12.x should fix the problem. Or using a 0.10.x version with the blocking io fixes backported (refs mapbox/node-cpp11#18).

/cc @camilleanne @yhahn @bsudekum

[yhahn]

Yes -- we currently don't have any tests around the atom shell startup process (wonder if we can do this headlessly or if that is its own can of worms...)

We'll have to dig into this to see what's possible -- in the meantime we should consider any changes to the atom shell portion of studio something someone must test manually on all 3 platforms.

[springmeyer]

@yhahn we used to test tilemill startup like this. Maybe a good place to start would be to add a cheap test like this to travis and mock ways that the shell might fail to start on unix.

[yhahn]

Sounds good. I bet we can monitor the app log as well to get a feel for if the app is starting (but hanging) or getting to other stages of startup.

[bsudekum]

@springmeyer I do like the look of this a lot:

killall node 2>/dev/null
killall tilemill-tile 2>/dev/null
killall tilemill-ui 2>/dev/null

When you say 'tests' are you talking about tests that run on every start or unit testing?

[springmeyer]

@bsudekum unit tests that run on travis/appveyor every commit.

[springmeyer]

@bsudekum - interested in working on this or have other ideas?