.NET 8: Design E2E testing solution for source-build/VMR

[mmitche]

Describe the Problem

.NET does not currently have a unified set of automated scenario tests that are run against the product. We do have testing provided by vendors, we do have a wide variety of unit tests run against product repositories, and we do have SDK tests the closely mimic the product. However, we are missing general scenario tests.

This is not great for the source-built Linux product. Aside from a series of smoke tests, partners do not have a great way to verify that what they build meets a quality bar. The unit tests that are also present in the product repos do not generally work against the end product, and often require resources that our partners are not able to use in their testing environments.

Describe the Solution

.NET should have a series of automated scenario tests that can run against the end product. These scenario tests and their infrastructure should be designed in such a way that:

• They are more 'complex' scenario tests that are cross-cutting rather than repository specific. These are not unit tests.
• They test large surface areas of the products.
• They can be run against an installed product. Installed might mean extracted from an archive, installed from a distribution package manager, or installed from an msi, for example.
• They can be run automatically on each check-in to the VMR/VMR-lite (source-built repo)
• The tests that are run, and potentially the tests themselves, can adapt to the test environment requirements of our partners. For instance, internet may not be available.

This issue covers three aspects:

• Design or identification of an appropriate test harness
• Design of the CI/PR pipeline(s) that will run these tests regularly against the source-built product.
• Identification of sets of tests that should be created or repurposed and placed within the E2E test infra.

[mmitche]

@richaverma1 Is this something you might be interested in helping work on for .NET 8?

[mmitche]

@MichaelSimons @tkapin @crummel

[MichaelSimons]

I view support for running in an offline mode part of this work.

[MichaelSimons]

Linking to the scenario testing design document - https://github.com/dotnet/arcade/blob/main/Documentation/UnifiedBuild/Scenario-Testing.md

@mmitche - what design work remains for this issue?

[MichaelSimons]

@mmitche - Can this be closed? What work remains? TIA