Gitlab Code Quality report

[bhirsz]

Add new report that supports Gitlab code quality artifacts: https://docs.gitlab.com/ee/ci/testing/code_quality.html#implement-a-custom-tool

[mathieugouin]

If it can help, here is how I manually did it:

    # Run robocop linter
    - |
        python -m robocop `
        --configure return_status:quality_gates:E=0:W=-1:I=-1 `
        --reports sarif `
        --configure sarif:report_filename:robocop-report-sarif.json `

Then (I'm on Windows based runner, so you might have to adjust accordingly):

    # Run an inline Python script to format robocop report into gitlab format
    - |
        $pythonCode = @"
        import json
        import hashlib

        # Local functions:
        def read_file_line(filepath: str, line_number: int) -> str:
            try:
                with open(filepath, 'r', encoding='utf-8') as file:
                    for current_line_number, line in enumerate(file, start=1):
                        if current_line_number == line_number:
                            return line.strip('\n\r')
                    # Special case for robocop last line issue
                    if current_line_number + 1 == line_number:
                        return line.strip('\n\r')
                # If the loop completes without finding the line
                raise AssertionError(f'Line {line_number} not found in {filepath}')
            except FileNotFoundError:
                raise FileNotFoundError(f'File not found: {filepath}')
            except Exception as e:
                raise Exception(f'An error occurred: {e}')


        def get_issue_line_content(issue: dict) -> str:
            return read_file_line(
                issue['location']['path'],
                issue['location']['lines']['begin']
                )

        def get_fingerprint(original_issue: dict, line_content: str, differentiator_id: int) -> str:
            issue = {
                'description': original_issue['description'],
                'check_name': original_issue['check_name'],
                # 'fingerprint': original_issue['fingerprint'],  # We don't want this one.
                'severity': original_issue['severity'],
                'location':
                    {
                        'path': original_issue['location']['path'],
                        # 'lines': {'begin': original_issue['location']['lines']['begin']},  # We don't want this one.
                    },
                'line_content': line_content,
                'differentiator_id': differentiator_id,
            }
            return hashlib.sha1(bytes(str(issue), 'utf-8')).hexdigest()

        # Read original sarif report
        report = None
        with open('robocop-report-sarif.json', 'r', encoding='utf-8') as f:
            report = json.load(f)

        fingerprint_list = []
        results = []
        for issue in report['runs'][0]['results']:

            result = {
                'description': issue['message']['text'],
                'check_name': 'robocop:' + issue['ruleId'],
                'severity': issue['level'],
                'location': {
                    'path': issue['locations'][0]['physicalLocation']['artifactLocation']['uri'],
                    'lines': {'begin': issue['locations'][0]['physicalLocation']['region']['startLine']}
                    },
                'fingerprint': 'x',
            }

            line_content = get_issue_line_content(result)
            for i in range(int(1e6)):
                fingerprint = get_fingerprint(result, line_content, i)
                if fingerprint not in fingerprint_list:
                    fingerprint_list.append(fingerprint)
                    result['fingerprint'] = fingerprint  # Overwrite the fingerprint with the one we computed
                    break

            results.append(result)

        with open('robot-gl-codequality.json', 'w', encoding='utf-8') as outfile:
            json.dump(results, outfile, indent=4)
        "@
        python -c $pythonCode

I made it as inline script so I can easily re-use this as a gitlab pipeline template:

include:
  - project: 'xxx/devops/pipeline-templates'
    file: 'robot-lint.yml'
    ref: '1.0.0'  # tag

[bhirsz]

@mathieugouin Thanks, it will be helpfull!

We're nearing Robocop 6.0 release which is the biggest release ever - one of my goal was to also improve integration/result reporting. Most of the tools support Sarif but with some changes.. that's why I will create separate reports:

robocop check --reports gitlab

There will be also option to enable them using -- shorthand:

robocop check --github

Current goal is to include it also in 6.0. But I can't test it directly, I can only build some unit test based on examples for documentation. Would it be possible for you to test it if I release it in dev version? ie 6.0beta . Note that 6.0 is breaking releasing and requires rewriting configuration (thankfully it's mostly covered by robocop migrate command). You would need to migrate anyway if you wish to use new versions in the future, but it will be sooner if you would test it :)

Optionally I could produce example report (based on some example file) which you could try to load in Gitlab job as artifact. This will not require updating to 6.0 at all.

[mathieugouin]

I don't have a huge config. Simply the arguments I pass as shown above, so it should be ok.

Sure I can try your beta version when ready.

Thanks !

[bhirsz]

I have implemented report locally and I have decided to test it first with free Gitlab plan: https://gitlab.com/MuminekM/robocop_test/-/tree/main

I see issues are loaded correctly:

I don't have access to more view (available for Premium / Ultimate etc) but it should also work.

I will let know in this thread when beta version with this report is out.

[bhirsz]

As for implementation details, Robocop uses 3 severity levels while Code Quality has 5 levels. I have decided to map it in following way:

        if diagnostic.severity == RuleSeverity.INFO:
            return "info"
        if diagnostic.severity == RuleSeverity.WARNING:
            return "minor"
        if diagnostic.rule.rule_id.startswith("ERR"):
            return "blocker"
        return "major"

rule_ids with ERR prefix are usually syntax issues, parsing-error etc so I have decided to label is as a blocker.

[mathieugouin]

That seems very good !

The tricky part I had trouble with is getting the fingerprint to work properly. Did you do similarly to what I did ?

[bhirsz]

Yes, I have stolen borrowed your code:

    @staticmethod
    def get_fingerprint(diagnostic: Diagnostic) -> str:
        """Generate unique identifier of the issue based on the rule message, name and location."""
        issue = {
            "description": diagnostic.message,
            "check_name": diagnostic.rule.name,
            "location": {
                "path": str(diagnostic.source),
                "region": {
                    "startLine": diagnostic.range.start.line,
                    "endLine": diagnostic.range.end.line,
                    "startColumn": diagnostic.range.start.character,
                    "endColumn": diagnostic.range.end.character,
                },
            }
        }
        return hashlib.sha1(bytes(str(issue), 'utf-8')).hexdigest()

I have used full range and check_name and description as enough information to have unique identifier of the issue (and 're-productable' issue, so if I run the check again on the same, unmodified files I should get the same fingerprints). I didn't use differentiator_id etc as I didn't see the need for it - but of course I could be wrong, we will see in more tests later :)

[mathieugouin]

I'm glad my code could help.

You have to be careful though. During my tests, I found that if you include the line number in the sha computations, you will notice that any new line added before already found issues will mark them as fixed and new issues will be reported. This is due to the fact that the sha is used in the gitlab merge request to differentiate between new and fixed issues. In my test I wanted to make sure that line shifts do not affect already found issues. This is why I added a differentiator ID in case in the same file more than one line of exactly the same content are reported as an exact same issue. This is also why I included the line content in the sha computation.

Hope this helps!

[bhirsz]

Ah, you're right. Thanks for pointing this out. I had similar issue in the past were I was testing loading Robocop issues into Jenkins new warnings plugin - it uses it's own mechanism for discovering unique issues but in end adding/removing lines marked most of the issues as solved and new respectively. Turns out I will reuse even more of your code then :) (with slightly modified approach though, as I have access to more tooling inside Robocop).

[bhirsz]

I've decided to release 6.0.0a3 version instead of beta to release this report for testing early. I have included your recent suggestions, you should be able to test it now:

> robocop check --gitlab
or
> robocop check --reports gitlab

It will generate robocop-code-quality.json file. Can be configured via output_path:

robocop --configure gitlab.output_path=path/to/file.json --reports gitlab