Add a MSTest sample with a program.cs

[Evangelink]

Relates to #4198

[julealgon]

@Evangelink I have a few questions regarding this sample.

First, we still have 3 MSBuild properties that I thought would be controlled by the setup code in Program.cs when that was manually handled:

• EnableMSTestRunner
• TestingPlatformDotnetTestSupport
• TestingPlatformShowTestsFailure

Is there no way to control those aspects without MSBuild settings?

Second, it would be nice if this:

• GenerateTestingPlatformEntryPoint

Was false by default, so that "magic" projects would opt into it, while "handmade" projects wouldn't need to specify it. Any chance that that could happen? I can create a proposal if so.

And lastly, can you elaborate if it is possible to control the DOTNET_CLI_TELEMETRY_OPTOUT or TESTINGPLATFORM_TELEMETRY_OPTOUT variables via code instead of via environment variables when using the manual Program.cs approach?

[nohwnd]

Yes these options or the msbuild support of testing platform is not strictly necessary if you are using only the exe to run your tests. But as soon as you want to start running tests e.g. via VS or through dotnet test, you need to interact with MSBuild to be correctly detected. Because the execution there also works via MSbuild and targets.

EnableMSTestRunner controls a lot of things, among them adding project capabilities:
<ProjectCapability Include="TestContainer" /> which you need to be detected by UI.
<ProjectCapability Include="TestingPlatformServer" /> which is needed for the new experience in VS.

https://github.com/microsoft/testfx/blob/main/src/Platform/Microsoft.Testing.Platform.MSBuild/buildMultiTargeting/Microsoft.Testing.Platform.MSBuild.targets#L12

There are also IsTestProject property and replacement of VSTest target to hook into the dotnet test machinery, to allow running under dotnet test.

---

I don't think there is a way for us to have the GenerateTestingPlatformEntryPoint switch to be off by default, and have the projects opt-in. Unless we want to show the property in the project directly. Which we expect to be less common than a project opting-out and implementing their own program.cs. It has also been the same in VSTest where you opt-out from the autogenerated entrypoint.

If you decide to look into this anyway, please know that there is a small bug in the ordering in the targets which forces us to set all the internal variables on the adapter level:

rather than this default being effective and only customized by the adapters:

---

You can opt out from telemetry by using this option TestApplicationOptions.EnableTelemetry provided to overload of TestApplication.CreateBuilderAsync.

[Evangelink]

To complement @nohwnd message:

Was false by default, so that "magic" projects would opt into it, while "handmade" projects wouldn't need to specify it. Any chance that that could happen? I can create a proposal if so.

As you can see there are different views and we need to consider the migration where people are not used to manually define the main themselves. I would still love to consider this request for v2 of the platform as it is for me a strong strength of the new platform and can result in many "hacks" to be avoided.

And lastly, can you elaborate if it is possible to control the DOTNET_CLI_TELEMETRY_OPTOUT or TESTINGPLATFORM_TELEMETRY_OPTOUT variables via code instead of via environment variables when using the manual Program.cs approach?

You can also simply not register the telemetry provider on line 22 https://github.com/microsoft/testfx/blob/main/samples/public/mstest-runner/MSTestProjectWithExplicitMain/MSTestProjectWithExplicitMain/Program.cs#L22.

As a global rule, here is how the extensibiliy model works:

1. Reference the extension dll (NuGet package, project reference or local code)
2. Register the extension
3. Check for extension activation. Most of the time it's disabled by default and then enabled by a condition (e.g. CLI option) but the telemetry is working in reverse by being on by default and looking for disabling flags.

[julealgon]

@nohwnd

Yes these options or the msbuild support of testing platform is not strictly necessary if you are using only the exe to run your tests. But as soon as you want to start running tests e.g. via VS or through dotnet test, you need to interact with MSBuild to be correctly detected. Because the execution there also works via MSbuild and targets.

EnableMSTestRunner controls a lot of things, among them adding project capabilities: <ProjectCapability Include="TestContainer" /> which you need to be detected by UI. <ProjectCapability Include="TestingPlatformServer" /> which is needed for the new experience in VS.

https://github.com/microsoft/testfx/blob/main/src/Platform/Microsoft.Testing.Platform.MSBuild/buildMultiTargeting/Microsoft.Testing.Platform.MSBuild.targets#L12

There are also IsTestProject property and replacement of VSTest target to hook into the dotnet test machinery, to allow running under dotnet test.

The more such MSBuild properties you mention, the stronger I believe there should be a standard Microsoft.NET.Sdk.Test project SDK that is used for all test projects, similar to how .Web, .Worker, etc SDKs are used for those types of applications.

Yes, I'm aware that the MSTest SDK exists, but to me, that is the wrong approach, because that SDK ties you to the MSTest engine as well: now if I want to use xUnit or nUnit or some other test framework, I cannot use the SDK anymore and have to downgrade my experience by having to control some weird specific MSBuild properties.

Have you guys considered creating a more general-purpose Test.Sdk (instead of MSTest.Sdk) to standardize all test projects?

I don't think there is a way for us to have the GenerateTestingPlatformEntryPoint switch to be off by default, and have the projects opt-in. Unless we want to show the property in the project directly.

I would vote for the emphasized. Have it be part of the "dumbed down" test project template.

Which we expect to be less common than a project opting-out and implementing their own program.cs. It has also been the same in VSTest where you opt-out from the autogenerated entrypoint.

See, this is where we diverge. My take is that you guys should push for the Program.cs-based approach to become the standard, not the other way around. Of course, in the beginning, you would expect the "automatic" projects to be more common, because that's how test projects have been since forever. But every single project in the ecosystem now relies on a much more standard "application building" code in Program.cs, and I think this is a massive opportunity for you guys to re-define this for test projects too.

If all samples and articles would point people to the Program.cs-based approach, that will very quickly become the new standard, and I would love for you guys to do exactly that.

You can opt out from telemetry by using this option TestApplicationOptions.EnableTelemetry provided to overload of TestApplication.CreateBuilderAsync.

Understood. This is exactly what I was looking for, thanks.

The only thing missing is of course adding standard IConfiguration support for the test builder similar to how other application builders work so that one can customize those settings as they see fit using the configuration provider of their choice (Azure AppConfig for the most part on our side, for example).

---

@Evangelink

As you can see there are different views and we need to consider the migration where people are not used to manually define the main themselves. I would still love to consider this request for v2 of the platform as it is for me a strong strength of the new platform and can result in many "hacks" to be avoided.

To be clear, I totally get where you are coming from. You want to avoid a massive breaking change for everybody since the old way test projects were handled was super automatic from the beginning.

As mentioned above, I think this is where you could provide a "legacy" test template, and a "modern" one, using an explicit Program.cs, and then push people to use the modern approach for new projects.

And lastly, can you elaborate if it is possible to control the DOTNET_CLI_TELEMETRY_OPTOUT or TESTINGPLATFORM_TELEMETRY_OPTOUT variables via code instead of via environment variables when using the manual Program.cs approach?

You can also simply not register the telemetry provider on line 22 https://github.com/microsoft/testfx/blob/main/samples/public/mstest-runner/MSTestProjectWithExplicitMain/MSTestProjectWithExplicitMain/Program.cs#L22.

Gotcha. So this is yet another option. Good to know.

[Evangelink]

Have you guys considered creating a more general-purpose Test.Sdk (instead of MSTest.Sdk) to standardize all test projects?

Yes that's WIP or to be more precise, that's being considered. We are trying to find the scope and how to design it not to block test frameworks.

I would vote for the emphasized. Have it be part of the "dumbed down" test project template.

Probably a good idea for the separate template design (see below)

As mentioned above, I think this is where you could provide a "legacy" test template, and a "modern" one, using an explicit Program.cs, and then push people to use the modern approach for new projects.

Yep dotnet/test-templates#448! About the new template, this is definitely what I am exploring at the moment, still having some discussions about the possibility of having "MSTest" following the new recommended pattern (MSTest runner + explicit main) and to have "MSTest legacy" or "MSTest (using VSTest)" as the other but we always hit the "changing user habits" discussion!

See, this is where we diverge. My take is that you guys should push for the Program.cs-based approach to become the standard, not the other way around. Of course, in the beginning, you would expect the "automatic" projects to be more common, because that's how test projects have been since forever. But every single project in the ecosystem now relies on a much more standard "application building" code in Program.cs, and I think this is a massive opportunity for you guys to re-define this for test projects too.

If all samples and articles would point people to the Program.cs-based approach, that will very quickly become the new standard, and I would love for you guys to do exactly that.

I cannot make promises but please know that's exactly my view and position.

Test is sadly not a field where we get enough support from management or users in general. It's still viewed as a cost and things should be working as they were working since forever. We get lot of pressure from management to not disrupt there.

This being said, I will continue to push and maybe one day we will have a good news!

The only thing missing is of course adding standard IConfiguration support for the test builder similar to how other application builders work so that one can customize those settings as they see fit using the configuration provider of their choice (Azure AppConfig for the most part on our side, for example).

We have our own IConfiguration. We already discussed the why something custom and I still have somewhere in the back of my mind to try and build a "bridge" extensions that would allow to plug existing ILogger, IConfiguration....