Proposal for nightly code coverage run

[CurryKitten]

We'd like to get code coverage run every night and published to codecoverage.nodejs.org

There has been some great work done already as discussed in:
#31

To produce these coverage results, we believe the steps are as follows:

• Move the patches in https://github.com/addaleax/node-core-coverage to https://github.com/nodejs/testing/coverage
• Create a jenkins job (initially for master) which builds Node.js with code coverage enabled. This job will initially use the patches from https://github.com/nodejs/testing/coverage and will build Node.js and run the core Node.js tests (test-ci target). It will be configurable so that it can be run on any particular branch. (this job takes about 15-20 minutes to run)
• Make the jenkins job push the resulting coverage html files to a directory on the local machine. We will assume the coverage job will be only run for x86 and will run on the Benchmarking machine.
• Setup an hourly job on the website infra which will pull the coverage files from the benchmarking machine and put them in the correct place for them to be viewed from coverage.nodejs.org. The files will be in a directory that matches the date of the run so that coverage can be compared across time.
• In order to avoid ongoing conflicts between the patches and the Node.js source, move as many of the changes applied by the patches into the Node.js build files themselves.
• Configure backups so that historical data is maintained and can be recovered if the benchmarking machine needs to be re-imaged. Each run seems to take about 12MB.
• Decide how many days of data we can afford to retain.
• Consider if we need results for more branches than master and add jobs for those branches as appropriate.
• Look at adding additional tests to what is run in order to get a more complete picture of our coverage (for example running CITGM in addition to the base Node.js tests)
• Add in automatic triggers to send emails to the testing team when coverage drops by more than X%
• Analyse the results and look for gaps in coverage and submit new tests to close any coverage gaps

If we agree this is the right thing to do and we should move forward I'll volunteer to start working on these steps. I've talked to @mhdawson and he can help out with the parts that require creating jenkins jobs.

FYI @addaleax, @Trott

[Trott]

@nodejs/build

[addaleax]

Hi @CurryKitten! :)

I believe @jbergstroem was already working on something in that direction?

[jbergstroem]

Sounds good! @mhdawson: you want to carry this? I was talking to @Trott and @addaleax on how to (optimally) run this as part of a pr review but it's a bit too expensive for us at the moment.

[mhdawson]

I happy to help move this forward, create CI jobs, etc and pull in other build team members when necessary for pieces like pushing the files over to the website.

[jbergstroem]

@mhdawson perhaps slap a label on the same hosts that run v8test (ish) and use those seeing how they have more cpu/ram to spare. test-rackspace-fedora24-x64-1 being a good example. We'd need a new playbook for required packages though.

[mhdawson]

@jbergstroem the reason I was going to suggest the benchmarking machine is because as I understand the website needs to pull fro a known location.

[jbergstroem]

@mhdawson sgtm as long as we can make sure there's no benchmarking going on.

[mhdawson]

Staring to integrate here:

https://ci.nodejs.org/job/node-test-commit-linux-coverage-mdawson/11/nodes=benchmark/consoleFull

Still work to do, adding bit by bit so that I understand, also modifying so that it is the same as our standard build job except for the additions required to generate the code coverage.

So far have added up to the point where it builds and runs the tests.

[mhdawson]

Ok, job is setup to run as documented here:

#37

In the interim I have data being pushed here nodejs.devrus.com:8080 (one of my machines) so that people can see what it will look like.

@addaleax could you submit a PR adding these files to github.com/nodejs/testing/coverage:

gcovr-patches.diff
generate-index-html.py
patches.diff

I'll then submit a PR for my updates to generate-index-html.py.

@jbergstroem, @rvagg can you review #37 and look at adding the pull from the benchmarking machine and serving the files as coverage.nodejs.org ? Just let me know if you need any additional information. The same key used to pull the charts for benchmarking.nodejs.org can be used to do the pull in a similar manner. The content will be at ~/coverage-out/out

[mhdawson]

As an FYI I also renamed the job to remove my name so its now: https://ci.nodejs.org/view/All/job/node-test-commit-linux-coverage/

[rvagg]

done, live @ https://coverage.nodejs.org/, config @ nodejs/build#516

[mhdawson]

Thank @rvagg !

@addaleax I'm thinking we should send out some sort of announcement through the evangelism group. I'm happy to do it but given you did a lot of the initial work I was thinking you might want to do that or we could collaborate to put it together.

[addaleax]

@mhdawson Where are the files from which the https://coverage.nodejs.org/ is currently generated? There are still references to my github project, and idk if that should be fixed before making it something “official”… otherwise, yeah, you can feel free to send out anything you like :)

[rvagg]

oh, btw, they are in coverage-out/out on benchmarking, that's where we are pulling from so if you fix that up let us know and we can adjust, happy to leave that as is if it doesn't bother you tho

[mhdawson]

@addaleax. Some of the files are still coming from your repo, with one having been modified and residing on the benchmarking machine. Once we land your PR I'll commit the changed file there 2.

Once that happens I'll get you to take another look to make sure all of the references are gone and then send something out.

@rvagg once they are fixed up it should automatically update the files in coverage-out/out on the benchmarking data machine.

[mhdawson]

Ok updated the job to clone the testing repo instead and use the files from there. @addaleax are all the references you were worried about gone ?

[addaleax]

https://coverage.nodejs.org/ looks good to me, yep 👍

[mhdawson]

Announced via weekly update. Improved via comments/updates from website group. My next step is to update the documentation. I know @CurryKitten is working on moving more into core in order to avoid possible conflicts in the future.

[Trott]

I think is done via coverage.nodejs.org so closing. Comment or re-open if I'm missing something!