Skip to content

Commit d783a8c

Browse files
Update library testing docs page to reduce confusion (#54324)
* Add warning on unmaintained testing doc page * Update testing.md Some example text that seems more clear to me, but only offered as a suggestion. Feel free to adjust it or if you want to use it as-is please double check what I wrote is accurate : ) I think the useful elements are: 1. Being explicit about what workflow steps need to happen in total 2. Being explicit about which commands are covering the entire workflow and which ones are only covering a part of it 3. Show the simple "do-it-all" options first before showing more complex partial options. Glancing at the first example and blindly copying it should land in the pit of success. Co-authored-by: Viktor Hofer <[email protected]>
1 parent 0aafceb commit d783a8c

File tree

1 file changed

+41
-24
lines changed

1 file changed

+41
-24
lines changed

docs/workflow/testing/libraries/testing.md

+41-24
Original file line numberDiff line numberDiff line change
@@ -1,54 +1,71 @@
11
# Testing Libraries
22

3-
We use the OSS testing framework [xunit](https://github.com/xunit/xunit).
3+
## Full Build and Test Run
44

5-
To build the tests and run them you can call the libraries build script. For libraries tests to work, you must have built the coreclr or mono runtime for them to run on.
5+
These example commands automate the test run and all pre-requisite build steps in a single command from a clean enlistment.
66

7-
**Examples**
8-
- The following shows how to build only the tests but not run them:
7+
- Run all tests - Builds clr in release, libs+tests in debug:
98
```
10-
build.cmd/sh -subset libs.tests
9+
build.cmd/sh -subset clr+libs+libs.tests -test -rc Release
1110
```
1211

13-
- The following builds and runs all tests using clr:
12+
- Run all tests - Builds Mono in release, libs+tests in debug:
1413
```
15-
build.cmd/sh -subset clr+libs.tests -test
14+
build.cmd/sh -subset mono+libs+libs.tests -test -rc Release
1615
```
1716

18-
- The following builds and runs all tests using mono:
17+
- Run all tests - Build Mono and libs for x86 architecture in debug (choosing debug for runtime will run very slowly):
1918
```
20-
build.cmd/sh -subset mono+libs.tests -test
19+
build.cmd/sh -subset mono+libs+libs.tests -test -arch x86
2120
```
2221

23-
- The following builds and runs all tests in release configuration:
22+
## Partial Build and Test Runs
23+
24+
Doing full build and test runs takes a long time and is very inefficient if you need to iterate on a change.
25+
For greater control and efficiency individual parts of the build + testing workflow can be run in isolation.
26+
See the [Building instructions](../../building/libraries/README.md) for more info on build options.
27+
28+
### Test Run Pre-requisites
29+
Before any tests can run we need a complete build to run them on. This requires building (1) a runtime, and
30+
(2) all the libraries. Examples:
31+
32+
- Build release clr + debug libraries
2433
```
25-
build.cmd/sh -subset libs.tests -test -c Release
34+
build.cmd/sh -subset clr+libs -rc Release
2635
```
2736

28-
- The following builds clr in release, libs in debug and runs all tests:
37+
- Build release mono + debug libraries
2938
```
30-
build.cmd/sh -subset clr+libs+libs.tests -test -rc Release
39+
build.cmd/sh -subset mono+libs -rc Release
3140
```
3241

33-
- The following builds mono and libs for x86 architecture and runs all tests:
42+
Building the `libs` subset or any of individual library projects automatically copies product binaries into the testhost folder
43+
in the bin directory. This is where the tests will load the binaries from during the run. However System.Private.CorLib is an
44+
exception - the build does not automatically copy it to the testhost folder. If you [rebuild System.Private.CoreLib](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md#iterating-on-systemprivatecorelib-changes) you must also build the `libs.pretest` subset to ensure S.P.C is copied before running tests.
45+
46+
### Running tests for all libraries
47+
48+
- Build and run all tests in release configuration.
3449
```
35-
build.cmd/sh -subset mono+libs+libs.tests -test -arch x86
50+
build.cmd/sh -subset libs.tests -test -c Release
3651
```
3752

38-
- The following example shows how to pass extra msbuild properties to ignore tests ignored in CI:
53+
- Build the tests without running them
3954
```
40-
build.cmd/sh -subset libs.tests -test /p:WithoutCategories=IgnoreForCI
55+
build.cmd/sh -subset libs.tests
4156
```
4257

43-
Unless you specifiy `-testnobuild`, test assemblies are implicitly built when invoking the `Test` action.
44-
- The following shows how to only test the libraries without building them
58+
- Run the tests without building them
4559
```
4660
build.cmd/sh -subset libs.tests -test -testnobuild
4761
```
4862

49-
## Running tests on the command line
63+
- The following example shows how to pass extra msbuild properties to ignore tests ignored in CI.
64+
```
65+
build.cmd/sh -subset libs.tests -test /p:WithoutCategories=IgnoreForCI
66+
```
5067

51-
To build tests you need to specify the `test` subset when invoking build.cmd/sh: `build.cmd/sh -subset libs.tests`.
68+
### Running tests for a single library
5269

5370
The easiest (and recommended) way to build and run the tests for a specific library, is to invoke the `Test` target on that library:
5471
```cmd
@@ -68,21 +85,21 @@ dotnet build /t:Test /p:TargetArchitecture=x86
6885

6986
There may be multiple projects in some directories so you may need to specify the path to a specific test project to get it to build and run the tests.
7087

71-
#### Running a single test on the command line
88+
### Running a single test on the command line
7289

7390
To quickly run or debug a single test from the command line, set the XunitMethodName property, e.g.:
7491
```cmd
7592
dotnet build /t:Test /p:XunitMethodName={FullyQualifiedNamespace}.{ClassName}.{MethodName}
7693
```
7794

78-
#### Running outer loop tests
95+
### Running outer loop tests
7996

8097
To run all tests, including "outer loop" tests (which are typically slower and in some test suites less reliable, but which are more comprehensive):
8198
```cmd
8299
dotnet build /t:Test /p:Outerloop=true
83100
```
84101

85-
#### Running tests on a different target framework
102+
### Running tests on a different target framework
86103

87104
Each test project can potentially have multiple target frameworks. There are some tests that might be OS-specific, or might be testing an API that is available only on some target frameworks, so the `TargetFrameworks` property specifies the valid target frameworks. By default we will build and run only the default build target framework which is `net5.0`. The rest of the `TargetFrameworks` will need to be built and ran by specifying the `BuildTargetFramework` option, e.g.:
88105
```cmd

0 commit comments

Comments
 (0)