action: tav

[v1v]

What

Migrate TAV to run as GitHub actions when

• a merge commit in the main branch
• manually triggered

TAV matrix is declared in the GH workflow hence no more .tav files to be maintained.

Checklist

• Implement code

Caveats

We agreed recently that It will not run on PRs for now, if needed we can add that feature in a follow up, maybe using the smarter jenkins implementation if it still works.

[apmmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-03-27T09:36:46.445+0000

• Duration: 24 min 39 sec

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• run module tests for <modules> : Run TAV tests for one or more modules, where <modules> can be either a comma separated list of modules (e.g. memcached,redis) or the string literal ALL to test all modules

• run benchmark tests : Run the benchmark test only.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[trentm]

How can one trigger a TAV run on a PR?

• a full TAV run
• just for a particular module
• just for a particular node version

[trentm]

nit: Maybe call this "module" instead of "framework". This is minor, however.

[v1v]

How can one trigger a TAV run on a PR?
a full TAV run
just for a particular module
just for a particular node version

I probably misunderstood what we discussed online the other day, somehow I thought the smart jenkins stuff was never used hence I kept it simple to run only on the main branch or manually when needed.

If the above is needed, then let's discuss about each individual scenario:

• a full TAV run on a PR basis

  • Use a GitHub command, though we don't have that particular functionality in place yet. It requires some permissions to know who triggered the build through a comment to avoid anyone outside of the Elastic org to run them. Last time when I checked the org permissions were not accessible through the GitHub temporary token, but I can double check.
  • Use a GitHub label, though we used something like that in the APM Agent java, I'm not convinced yet as it required several on: types, see https://github.com/elastic/apm-agent-java/blob/a6be6fc83c802215fb449706cd7bb016558fbfc2/.github/workflows/main.yml#L43-L49

• just for a particular module and just for a particular node version

  • Same as above, either specific labels or using a github command, unless we wanna use the git changeset and compare what modules have been changed.

A few other questions:

1. What's your preference in terms of usability?
2. What's the one of the above options that we can discard?
3. Any other ideas?

Thanks

[trentm]

How can one trigger a TAV run on a PR?
...

I probably misunderstood what we discussed online the other day, somehow I thought the smart jenkins stuff was never used hence I kept it simple to run only on the main branch or manually when needed.

@v1v I think we only need the ability to manually start a TAV test run. I don't think we need any smart automatic triggering. I also don't have any requirements for how we do that -- any of the following would be fine: a GitHub comment, clicking some GitHub checks UI, running some gh workflow ... command. Whatever is cleanest/easiest.

---

Should we be able to see some "TAV / ..." GitHub checks on this PR?

[trentm]

Approving, though I'm still not sure what the suggested way to manually trigger a TAV run is.

[v1v]

Should we be able to see some "TAV / ..." GitHub checks on this PR?

I'm afraid new GH actions coming from forked repositories are not executed. Let me create a branch in the upstream and cherry-pick the changes so you can see how it looks like.

See #3221

[trentm]

I'm afraid new GH actions coming from forked repositories are not executed. Let me create a branch in the upstream and cherry-pick the changes so you can see how it looks like.

Oh, right. Thanks for doing that. This looks very good.

---

Some TAV comparisons between GH Actions and Jenkins:

• " Total duration 1h 19m 9s " (from https://github.com/elastic/apm-agent-nodejs/actions/runs/4540484765?pr=3221) vs. about 1h15m for a Jenkins TAV run (https://apm-ci.elastic.co/job/apm-agent-nodejs/job/apm-agent-nodejs-mbp/job/main/)

• Both have a way to look at all the separate tests run:

• To find TAV tests that were taking very long to run (to know if we should look into reducing the number of versions tested), I had written https://github.com/elastic/apm-agent-nodejs/blob/main/dev-utils/jenkins-build-slow-steps.sh At some point I'll see about writing a replacement for these GH Actions. That definitely is not a requirement for this work.

[trentm]

@v1v Do you know if there will be a way to manually run just a part of the TAV tests on a PR? For example, say I have a PR that is working on the "fastify" module. Can I run the TAV tests for module=fastify for either all node versions, or just a subset of the node versions?

This doesn't work:

% gh workflow run tav.yml --ref test/action-tav -f node=16 -f module=fastify
could not create workflow dispatch event: HTTP 422: Unexpected inputs provided: ["module", "node"] (https://api.github.com/repos/elastic/apm-agent-nodejs/actions/workflows/46437793/dispatches)

because the matrix values are not the same as workflow dispatch inputs. I haven't played with workflow dispatch inputs in GH Actions, so I don't know if what I'm asking is possible.

This isn't a requirement for this PR. We could look into this separately.

[trentm]

I notice this from https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstrategymatrix

A matrix will generate a maximum of 256 jobs per workflow run. This limit applies to both GitHub-hosted and self-hosted runners.

Currently we have 35 modules * 7 node versions = 245 jobs. So we are very close to the maximum currently. :)
When node v20 is released (expected 2023-04-18) and there are initial node v21 releases (I'm not sure when), then we'll be over the limit.
Or, if/when we add two new instrumented modules, then we'll be over the limit.
This is just something to be aware of. We may have a new major version of the APM agent by then that drops support for some of the older ancient node versions. Or we could stop TAV testing for the deprecated node v8. Or stop testing some older, less useful modules like got or bluebird.

[v1v]

@v1v Do you know if there will be a way to manually run just a part of the TAV tests on a PR? For example, say I have a PR that is working on the "fastify" module. Can I run the TAV tests for module=fastify for either all node versions, or just a subset of the node versions?

I'd like to work on this in a follow up, so we can use one of the above proposals (#3127 (comment))

Currently we have 35 modules * 7 node versions = 245 jobs. So we are very close to the maximum currently. :)

We managed to use buckets in the apm-agent-python:

• https://github.com/elastic/apm-agent-python/blob/28ecb8e804f38458d07dfad515d8eb3f828abcec/.github/workflows/test.yml#L35-L81

It's not ideal, otherwise if possible we could create combinations of two modules per runner, though I don't know whether that could cause any other issues (tearing down the environment might be required)

To find TAV tests that were taking very long to run (to know if we should look into reducing the number of versions tested), I had written https://github.com/elastic/apm-agent-nodejs/blob/main/dev-utils/jenkins-build-slow-steps.sh At some point I'll see about writing a replacement for these GH Actions. That definitely is not a requirement for this work.

You can use the existing opentelemetry data :) https://ela.st/oblt-ci-cd-stats. (though it's only accessible for Elasticians and the new TAV GH action won't be available until this PR gets merged)

Our plan is to leverage the existing GitHub metadata and provide insights that are meaningful. We are at the moment with the migration stage, but definitely it's something we will invest effort so we can help to answer those kind of questions.

[trentm]

You can use the existing opentelemetry data :)

That'll be cool to browse around, and yes perhaps we could make a dashboard from that data.

Here is the beginnings of a replacement using the GH API and gh:

#!/bin/bash
branch=main
branch=test/action-tav
latestTavRun=$(gh run list -R elastic/apm-agent-nodejs -b "$branch" -w TAV -L 1 --json databaseId --jq '.[].databaseId')
gh api --paginate repos/elastic/apm-agent-nodejs/actions/runs/$latestTavRun/jobs \
    | json -ga jobs \
    | json -ga -e '
        this.s = (new Date(this.completed_at) - new Date(this.started_at)) / 1000;
        this.minSec = Math.floor(this.s/60) + "m" + (this.s%60).toString().padStart(2,"0") + "s"
        ' s minSec name \
    | sort -n

Running that yields:

...
1118 18m38s test-tav (10, aws-sdk)
1125 18m45s test-tav (12, tedious)
1129 18m49s test-tav (10, fastify)
1199 19m59s test-tav (8, aws-sdk)
1203 20m03s test-tav (12, pg)
1221 20m21s test-tav (8, pg)
1248 20m48s test-tav (10, mongodb-core)
1256 20m56s test-tav (14, next)
1307 21m47s test-tav (12, knex)
1307 21m47s test-tav (14, knex)
1323 22m03s test-tav (10, pg)
1386 23m06s test-tav (12, graphql)
1431 23m51s test-tav (14, tedious)
1496 24m56s test-tav (10, graphql)
1508 25m08s test-tav (8, graphql)
1757 29m17s test-tav (14, graphql)
1794 29m54s test-tav (10, knex)