Support for --machine in very_good test

[davidmartos96]

Description
Support for the --machine option from flutter test. It provides a way to output a summary of the tests that were executed. How many failed, how long they took, etc... This output is useful to then convert it to popular formats such as an XML JUnit report, which can be interpreted by different other tools, like CI/CD platforms or IDEs. One tool to convert this output is https://pub.dev/packages/junitreport

On a dart test, the report output is obtained with --reporter json, not with --machine

[nilsreichardt]

Are there any reasons why very_good does not support passing arguments with the -- operator, like very_good test -- --machine? npm uses the -- operator as well (npm/npm#5518).

cc: @marcossevilla

[felangel]

Are there any reasons why very_good does not support passing arguments with the -- operator, like very_good test -- --machine? npm uses the -- operator as well (npm/npm#5518).

cc: @marcossevilla

The goal is not to expose an API that mirrors the underlying flutter/dart tool APIs. We wanted the CLI to serve as an abstraction above existing tooling so we didn't want to tightly couple them by proxying commands/options directly. Also, the CLI is using package:args which has a standard way of defining CLI flags and options. If we accepted -- and proxied them then they wouldn't be properly documented in the CLI usage (via --help).

Hope that helps 👍

[nilsreichardt]

Ah, thanks for the detailed answer!

[davidmartos96]

Isn't the requested feature valid?
I agree that mirroring with -- would not be ideal, but maybe some equivalent option for the test command would help for this use case?

[felangel]

Yup reopening this to track the lack of support for --machine

[agacemi]

Hi @felangel

any update for this feature. We need to generate also a json file for sonar or other tools. We tried to open a PR to add this option or an equivalent option. We discovered that --reporter=json is already added to flutter test command but the output is mapped to a list of objects.

thanks for your help

[justinschier]

Big +1! It would really help us out to have machine readable output.

[felangel]

Hey all, just want to give an update:

This is in on our radar but we probably won't get to it for a couple of weeks. We're more than happy to review any PRs in the meantime if this is urgent/blocking for folks.

[jackgllghr]

Added a PR here: #473
We require this for our CI pipelines, it's the only thing stopping us from migrating to very_good (it's cutting our test runs to 30% of their original run time!)

[Thelm76]

It would be great to get this option.... I'm looking forward to it, it is mandatory for my project in order to use it in our pipelines

[justinschier]

Any news on this feature by chance?

[DFelten]

This would be a really useful feature for our GitHub pipeline.

We're using very_good with optimization and it's a lot faster than the default tests. But unfortunately it's not possible to use an action like Test Reporter because of the missing json summary file.

[Thelm76]

Hello! I think this is not an issue anymore since VGC test supports "vanilla" parameters using -- (added to documentation with #840 )

Just be careful not to use --machine or --report as they modify stdout used by the cli to know test statuses. You should use --file-report=<reportType>:<path> as so :
very_good test --coverage -- --file-reporter json:test.txt

[DFelten]

I tried the file-reporter parameter. The test therefore takes four times as long. Is that expected?

very_good test --optimization --coverage -> 00:29 +641 ~2: All tests passed!

very_good test --optimization --coverage -- --file-reporter json:test_reports/dd_test_report.json -> 01:53 +641 ~2: All tests passed!

[alestiago]

@DFelten, sorry for the late reply, whenever -- is used the test optimisation is disabled, so it is expected.

I'm moving this to triage (cc: @tomarra ) and sharing with the team to see if it can get prioritised.