Automatic Issues For Nightly KIC Test Failures

[shaneutt]

Problem Statement

In the past there have been some nightly test failures that have taken longer for us to resolve than we would have liked. The purpose of this task is to add automatic issue creation whenever this occurs to help increase the noise made from a failure.

Disambiguation: this is about KIC tests that we run nightly; not tests that we run against Kong Gateway nightly builds.

Acceptance Criteria

• issues are now created at severity/high for any nightly test run failure
• issues are deduplicated by failed test name (or some other reasonable identifier); if an existing issue is closed, a failure reopens it with a comment linking to the failure.

[pmalek]

This looks relatively easy to do with e.g. https://github.com/marketplace/actions/create-github-issue

steps:
  - uses: actions/checkout@v3
  - name: Something that might fail
    run: exit 1
  - name: Create Issue on Failed workflow
    if: ${{ failure() }}
    uses: dacbd/create-issue-action@main
    with:
      token: ${{ github.token }}
      title: Action workflow failed.
      body: |
        ### Context
        [Failed Run](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }})
        [Codebase](https://github.com/${{ github.repository }}/tree/${{ github.sha }})
        Workflow name - `${{ github.workflow }}`
        Job -           `${{ github.job }}`
        status -        `${{ job.status }}`
      assignees: SomeUsername,AnotherUsername

[shaneutt]

Sounds good.

[mflendrich]

Blocked until we get rid of frequent causes for test flakiness (and get test flakiness under a "reasonable" threshold).

[mflendrich]

Concern by @rainest: this might end up with numerous reports for the test failing time after time.

Proposed solution: dedup by test case name. The automation would reopen the issue when an issue for this test case name fails (and comment with the run ID).

[mflendrich]

Another alternative proposed by @rainest and @pmalek: use external tooling to visualize trends in failures for specific test cases.