feat: allow citgm job comparison #455

codebytere · 2020-07-10T06:13:34Z

Follow-up to #454.
Closes #352.

Extends ncu-ci cigtm <jobid> to ncu-cu cigtm <jobid> [jobid2], allowing for direct comparisons between two CITGM runs to see if new failures were introduced.

I also did a bit of a refactor here, since there were a million classes in the result parser file and it was getting wildly unwieldy. I tried to do the minimum necessary to make this work but I plan to follow it up with the remainder of the the class split-out work.

Display:

Developer ❯ ncu-ci citgm 2392 2390                                             11:06PM
--------------------------------------------------------------------------------
[1/1] Running CITGM: 2392
--------------------------------------------------------------------------------
✔  Summary data downloaded
✔  Results data downloaded
✔  Summary data downloaded
✔  Results data downloaded
----------------------------------- Summary ------------------------------------
Result     FAILURE
URL        https://ci.nodejs.org/job/citgm-smoker/2392/
Source     https://api.github.com/repos/nodejs/node/git/refs/heads/v12.x
Commit     [feed95cd4c2c] Working on v12.18.1
Date       2020-06-02 20:27:47 +0200
Author     Michaël Zasso <[email protected]>
----------------------------------- Summary ------------------------------------
Result     FAILURE
URL        https://ci.nodejs.org/job/citgm-smoker/2390/
Source     https://github.com/nodejs/node/pull/33811/
Commit     [9a60117875dd] 2020-06-16, Version 12.18.1 'Erbium' (LTS)
Date       2020-06-09 20:23:09 -0700
Author     Shelley Vohr <[email protected]>
----------------------------------- Results ------------------------------------
┌────────────────────────┬───────────────────────────┬────────────────────────────┬─────────────────────────┐
│        (index)         │             0             │             1              │            2            │
├────────────────────────┼───────────────────────────┼────────────────────────────┼─────────────────────────┤
│       debian9-64       │ 'express-session-v1.17.1' │ 'yeoman-generator-v4.10.1' │                         │
│     centos7-ppcle      │    'ember-cli-v3.18.0'    │      'multer-v1.4.2'       │ 'torrent-stream-v1.2.0' │
│      rhel7-s390x       │  'torrent-stream-v1.2.0'  │                            │                         │
│   fedora-latest-x64    │    'spawn-wrap-v2.0.0'    │                            │                         │
│     ubuntu1804-64      │                           │                            │                         │
│ fedora-last-latest-x64 │                           │                            │                         │
│     ubuntu1604-64      │                           │                            │                         │
└────────────────────────┴───────────────────────────┴────────────────────────────┴─────────────────────────┘

Sample JSON Output

[
  {
    "baseBuild": {
      "source": "https://api.github.com/repos/nodejs/node/git/refs/heads/v12.x",
      "upstream": "https://ci.nodejs.org/job/citgm-smoker/2390/"
    },
    "comparisonBuild": {
      "source": "https://github.com/nodejs/node/pull/33811/",
      "upstream": "https://ci.nodejs.org/job/citgm-smoker/2392/"
    },
    "debian9-64": [
      {
        "name": "express-session-v1.17.1",
        "status": "FAILED"
      },
      {
        "name": "yeoman-generator-v4.10.1",
        "status": "FAILED"
      }
    ],
    "centos7-ppcle": [
      {
        "name": "ember-cli-v3.18.0",
        "status": "FAILED"
      },
      {
        "name": "multer-v1.4.2",
        "status": "FAILED"
      },
      {
        "name": "torrent-stream-v1.2.0",
        "status": "FAILED"
      }
    ],
    "rhel7-s390x": [
      {
        "name": "torrent-stream-v1.2.0",
        "status": "FAILED"
      }
    ],
    "fedora-latest-x64": [
      {
        "name": "spawn-wrap-v2.0.0",
        "status": "FAILED"
      }
    ],
    "ubuntu1804-64": [],
    "fedora-last-latest-x64": [],
    "ubuntu1604-64": []
  }
]

Sample Markdown Output

# CITGM Data for [#2392](https://ci.nodejs.org/job/citgm-smoker/2392/) - [#2390](https://ci.nodejs.org/job/citgm-smoker/2390/)

## New Failures in job [#2390](https://ci.nodejs.org/job/citgm-smoker/2390/)

### debian9-64

* express-session-v1.17.1
* yeoman-generator-v4.10.1

### centos7-ppcle

* ember-cli-v3.18.0
* multer-v1.4.2
* torrent-stream-v1.2.0

### rhel7-s390x

* torrent-stream-v1.2.0

### fedora-latest-x64

* spawn-wrap-v2.0.0

### ubuntu1804-64

None.

### fedora-last-latest-x64

None.

### ubuntu1604-64

None.

cc @richardlau (since i can't request your review)

codecov · 2020-07-10T06:14:40Z

Codecov Report

Merging #455 into master will decrease coverage by 0.25%.
The diff coverage is 69.56%.

@@            Coverage Diff             @@
##           master     #455      +/-   ##
==========================================
- Coverage   76.66%   76.41%   -0.26%     
==========================================
  Files          21       27       +6     
  Lines        1603     1717     +114     
==========================================
+ Hits         1229     1312      +83     
- Misses        374      405      +31

Impacted Files	Coverage Δ
lib/ci/build-types/test_build.js	`49.52% <49.52%> (ø)`
lib/ci/build-types/citgm_comparison_build.js	`64.93% <64.93%> (ø)`
lib/ci/build-types/citgm_build.js	`79.26% <79.26%> (ø)`
lib/ci/ci_result_parser.js	`42.54% <79.31%> (-9.14%)`	⬇️
lib/ci/build-types/job.js	`81.63% <81.63%> (ø)`
lib/ci/ci_utils.js	`100.00% <100.00%> (ø)`
lib/ci/jenkins_constants.js	`100.00% <100.00%> (ø)`
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d4b189b...3f4fcb1. Read the comment docs.

MylesBorins · 2020-07-15T18:25:03Z

Just realized that we might want to compare both citgm + citgm-nobuild jobs. Is that possible with this implementation
?

codebytere · 2020-07-15T21:35:12Z

@MylesBorins it does not at present - this pulls citgm-smoker jobs at the API level only so i'd need to extend this by adding the ability to fetch jobs at:

https://ci.nodejs.org/job/citgm-smoker-nobuild/${id}/api/json
https://ci.nodejs.org/job/citgm-smoker-nobuild/${id}/testReport/api/json

I'll make an issue and add that as a followup TODO, if that sounds good to you!

codebytere · 2020-07-16T04:15:17Z

Also - for more effective reviewing, just d22d5e1 is the actual new change introduced in this PR.

lib/ci/build-types/citgm_comparison_build.js

richardlau · 2020-07-20T23:11:07Z

Trying to make some sense of this:

-bash-4.2$ npx -p codebytere/node-core-utils#allow-citgm-comparison ncu-ci citgm 2421 2417
npx: installed 349 in 18.908s
--------------------------------------------------------------------------------
[1/1] Running CITGM: 2421
--------------------------------------------------------------------------------
✔  Summary data downloaded
✔  Results data downloaded
✔  Summary data downloaded
✔  Results data downloaded
----------------------------------- Summary ------------------------------------
Result     FAILURE
URL        https://ci.nodejs.org/job/citgm-smoker/2421/
Source     https://api.github.com/repos/nodejs/node/git/refs/heads/v10.x
Commit     [0bd44b4a88b7] Working on v10.21.1
Date       2020-06-02 18:57:33 +0100
Author     Bethany.Griggs <[email protected]>
----------------------------------- Summary ------------------------------------
Result     FAILURE
URL        https://ci.nodejs.org/job/citgm-smoker/2417/
Source     https://api.github.com/repos/nodejs/node/git/refs/heads/v10.22.0-proposal
Commit     [c5215d0b3da0] 2020-07-21, Version 10.22.0 'Dubnium' (LTS)
Date       2020-07-15 11:27:18 -0400
Author     Richard Lau <[email protected]>
----------------------------------- Results ------------------------------------
┌────────────────────────┬─────────────────────────┬─────────────────────┬───────────────────────────────┬─────────────────────┬─────────────────────┬──────────────────┬──────────────────────┬─────────────────┬──────────────────────┐
│        (index)         │            0            │          1          │               2               │          3          │          4          │        5         │          6           │        7        │          8           │
├────────────────────────┼─────────────────────────┼─────────────────────┼───────────────────────────────┼─────────────────────┼─────────────────────┼──────────────────┼──────────────────────┼─────────────────┼──────────────────────┤
│      aix71-ppc64       │ 'path-to-regexp-v6.1.0' │    'pug-v3.0.0'     │         'uuid-v8.2.0'         │                     │                     │                  │                      │                 │                      │
│        osx1015         │     'acorn-v7.3.1'      │    'ava-v3.10.1'    │        'clinic-v6.0.2'        │ 'ember-cli-v3.19.0' │  'react-v16.13.1'   │ 'semver-v7.3.2'  │ 'underscore-v1.10.2' │                 │                      │
│   fedora-latest-x64    │    'react-v16.13.1'     │                     │                               │                     │                     │                  │                      │                 │                      │
│        osx1014         │     'acorn-v7.3.1'      │    'ava-v3.10.1'    │ 'eslint-plugin-jest-v23.18.0' │   'clinic-v6.0.2'   │ 'ember-cli-v3.19.0' │ 'fastify-v3.0.3' │   'react-v16.13.1'   │ 'semver-v7.3.2' │ 'underscore-v1.10.2' │
│ fedora-last-latest-x64 │    'react-v16.13.1'     │                     │                               │                     │                     │                  │                      │                 │                      │
│     ubuntu1804-64      │    'react-v16.13.1'     │                     │                               │                     │                     │                  │                      │                 │                      │
│     centos7-ppcle      │     'clinic-v6.0.2'     │ 'ember-cli-v3.19.0' │                               │                     │                     │                  │                      │                 │                      │
│     ubuntu1604-64      │                         │                     │                               │                     │                     │                  │                      │                 │                      │
│     ubuntu1404-64      │                         │                     │                               │                     │                     │                  │                      │                 │                      │
│       debian9-64       │    'react-v16.13.1'     │                     │                               │                     │                     │                  │                      │                 │                      │
│      rhel7-s390x       │ 'torrent-stream-v1.2.0' │                     │                               │                     │                     │                  │                      │                 │                      │
└────────────────────────┴─────────────────────────┴─────────────────────┴───────────────────────────────┴─────────────────────┴─────────────────────┴──────────────────┴──────────────────────┴─────────────────┴──────────────────────┘
-bash-4.2$

This shows, for example, 'torrent-stream-v1.2.0' for rhel-s390x but that module appears as failing in both citgm runs.

codebytere · 2020-07-20T23:52:42Z

@richardlau spotted the error - fix incoming!

codebytere · 2020-07-21T01:08:29Z

@richardlau you should see:

node-core-utils on git:allow-citgm-comparison ❯ ncu-ci citgm 2421 2417
--------------------------------------------------------------------------------
[1/1] Running CITGM: 2421
--------------------------------------------------------------------------------
✔  Summary data downloaded
✔  Results data downloaded
✔  Summary data downloaded
✔  Results data downloaded
----------------------------------- Summary ------------------------------------
Result     FAILURE
URL        https://ci.nodejs.org/job/citgm-smoker/2421/
Source     https://api.github.com/repos/nodejs/node/git/refs/heads/v10.x
Commit     [0bd44b4a88b7] Working on v10.21.1
Date       2020-06-02 18:57:33 +0100
Author     Bethany.Griggs <[email protected]>
----------------------------------- Summary ------------------------------------
Result     FAILURE
URL        https://ci.nodejs.org/job/citgm-smoker/2417/
Source     https://api.github.com/repos/nodejs/node/git/refs/heads/v10.22.0-proposal
Commit     [c5215d0b3da0] 2020-07-21, Version 10.22.0 'Dubnium' (LTS)
Date       2020-07-15 11:27:18 -0400
Author     Richard Lau <[email protected]>
----------------------------------- Results ------------------------------------



FAILURE: 9 failures in 2417 not present in 2421


┌────────────────────────┬─────────────────────┬──────────────────┐
│        (index)         │          0          │        1         │
├────────────────────────┼─────────────────────┼──────────────────┤
│      aix71-ppc64       │ 'ember-cli-v3.19.0' │                  │
│     ubuntu1804-64      │                     │                  │
│ fedora-last-latest-x64 │ 'spawn-wrap-v2.0.0' │ 'winston-v3.3.3' │
│        osx1014         │  'fastify-v3.0.3'   │                  │
│     ubuntu1404-64      │                     │                  │
│      rhel7-s390x       │ 'uglify-js-v3.10.0' │                  │
│        osx1015         │                     │                  │
│     centos7-ppcle      │ 'ember-cli-v3.19.0' │ 'multer-v1.4.2'  │
│     ubuntu1604-64      │   'clinic-v6.0.2'   │ 'yargs-v15.4.1'  │
│       debian9-64       │                     │                  │
│   fedora-latest-x64    │                     │                  │
└────────────────────────┴─────────────────────┴──────────────────┘

now which makes far more sense!

MylesBorins · 2020-07-23T16:58:19Z

At a cursory glance it seems like the --no-build flag is a binary flag for the entire tool, not for independent jobs... so there would not be a way to compare a regular CITGM job to a no-build job, is that accurate?

Could we potentially also allow a small grammar here, rather than a flag... perhaps ${jobType}:${jobNumber}

e.g.

1337 ~ citgm job 1337
citgm:1337 ~ citgm job 1337
nobuild:1337 ~ nobuild job 1337

thoughts?

codebytere · 2020-07-23T17:02:30Z

@MylesBorins the way i've written it at present is that for comparison jobs, if --nobuild is passed that would apply to the base job:

https://github.com/nodejs/node-core-utils/pull/455/files#diff-5bdf2b1736f8191ae8dc6098a8f25651R6-R21

so if one passed nci-ci citgm 880 2433 --nobuild that would compare 880, a nobuild job, to 2433, a regular job. However, i do agree that that might be a little confusing 🤔

MylesBorins · 2020-07-23T17:03:24Z

@codebytere as long as it is documented and possible that seems like a good way to move forward... and we could revisit later if we want to improve the dx

codebytere requested review from targos, mmarchini and BridgeAR July 10, 2020 06:13

codebytere force-pushed the allow-citgm-comparison branch 2 times, most recently from dd48ae2 to 3cd93b9 Compare July 10, 2020 06:44

richardlau approved these changes Jul 17, 2020

View reviewed changes

richardlau reviewed Jul 20, 2020

View reviewed changes

lib/ci/build-types/citgm_comparison_build.js Outdated Show resolved Hide resolved

codebytere force-pushed the allow-citgm-comparison branch 2 times, most recently from 8b99bba to 1fc9c06 Compare July 20, 2020 18:33

mmarchini approved these changes Jul 20, 2020

View reviewed changes

codebytere force-pushed the allow-citgm-comparison branch 2 times, most recently from 843b259 to 0dbe375 Compare July 21, 2020 01:11

richardlau mentioned this pull request Jul 21, 2020

v10.22.0 proposal nodejs/node#34170

Merged

4 tasks

codebytere added 6 commits July 22, 2020 11:36

refactor: pull some builds into own files

7d54e44

feat: enable citgm run comparisons

08b01ad

spec: add tests for comparisons

5148297

Fix filter error and simplify logic

063238a

Remove accidental test diff

18109a6

Fixup after nobuild PR

91f107e

codebytere force-pushed the allow-citgm-comparison branch from 0ec2b1f to 91f107e Compare July 22, 2020 19:06

codebytere added 2 commits July 22, 2020 13:26

Add new test for noBuild comparison

8f9dd8e

Add better documentation

3f4fcb1

codebytere merged commit 336dc7f into nodejs:master Jul 23, 2020

codebytere deleted the allow-citgm-comparison branch July 23, 2020 19:19

codebytere mentioned this pull request Aug 4, 2020

chore: split CI builds to new files #464

Merged

mmarchini mentioned this pull request Aug 31, 2020

Add difference to CITGM output nodejs/build#1944

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: allow citgm job comparison #455

feat: allow citgm job comparison #455

codebytere commented Jul 10, 2020 •

edited

Loading

codecov bot commented Jul 10, 2020 •

edited

Loading

MylesBorins commented Jul 15, 2020

codebytere commented Jul 15, 2020 •

edited

Loading

codebytere commented Jul 16, 2020 •

edited

Loading

richardlau commented Jul 20, 2020

codebytere commented Jul 20, 2020

codebytere commented Jul 21, 2020 •

edited

Loading

MylesBorins commented Jul 23, 2020

codebytere commented Jul 23, 2020

MylesBorins commented Jul 23, 2020

feat: allow citgm job comparison #455

feat: allow citgm job comparison #455

Conversation

codebytere commented Jul 10, 2020 • edited Loading

codecov bot commented Jul 10, 2020 • edited Loading

Codecov Report

MylesBorins commented Jul 15, 2020

codebytere commented Jul 15, 2020 • edited Loading

codebytere commented Jul 16, 2020 • edited Loading

richardlau commented Jul 20, 2020

codebytere commented Jul 20, 2020

codebytere commented Jul 21, 2020 • edited Loading

MylesBorins commented Jul 23, 2020

codebytere commented Jul 23, 2020

MylesBorins commented Jul 23, 2020

codebytere commented Jul 10, 2020 •

edited

Loading

codecov bot commented Jul 10, 2020 •

edited

Loading

codebytere commented Jul 15, 2020 •

edited

Loading

codebytere commented Jul 16, 2020 •

edited

Loading

codebytere commented Jul 21, 2020 •

edited

Loading