Improve automated testing

[pnasrat]

Currently by the looks of it circle just checks the source can compile and build. To support contributor development of greater OS coverage issue #19 it would be nice if there was a simple test cycle that could be run per OS

I'd imagine a shell based test driver that boots the OS - records console output and does some validation would be a simple first step.

I don't want to implement without discussion so please give me feedback if this sounds of use.

[avsm]

Most definitely useful! We do need to verify if the CI provider exposes the Hypervisor framework into their VMs though -- experiments on CircleCI have given us both answers. We could also set up a bare metal Mac and host it somewhere, which would give us better control over the base OS as well.

[pnasrat]

We can definitely separate the goals - first having repeatable tests manually run (also good!) from CI.

[ijc]

I think the issues with CI not exposing support for the Hypervisor framework are largely historical, it's been a long time since we've seen an issue like that in the CI testing of Docker for Mac for example. I suppose they've e.g. updated the software on their whole fleet since we last saw it.

Some sort of simple "1) boot 2) check it looks ok 3) shutdown" test in CI would be really nice for sure. As I hinted in #23 my preference would be for an Alpine based thing, but really at this point anything would be better than nothing (and in any case I'd expect the scripting for "2)" to be where most of the complexity would be).

[pnasrat]

Do you have any preferences on approach - eg on adding dependencies to make the testing work (eg bash + expect) or writing a custom test "driver" in C or ocaml.

[ijc]

Looks like expect is part of osx (i.e. it's in /usr/bin/expect already on my mac mini) so that seems fine to use (and if it weren't already there I'd see no problem with getting it from brew). ocaml would be fine too.

I'd probably avoid C though for this particular sort of job.

[pnasrat]

So it looks as if tinycorelinux.net is down today - I started to look at moving to alpine which just seems to distribute a cdrom. I'm asking about on IRC around netboot images.

[pnasrat]

Note to self brittle initial test could be

#!/usr/bin/env expect

spawn "./hyperkitrun.sh"
expect "tc@box:~$"
send "sudo halt\r\n";
expect "reboot: System halted"
close

[ijc]

It's a shame isoinfo isn't available on OSX (even via brew) otherwise getting the kernel+initrd out of the ISO would be fairly easy.

FWIW the proposed expect script looks ok to me.