Skip to content

Commit adaf602

Browse files
authored
test_runner: add initial CLI runner
This commit introduces an initial version of a CLI-based test runner. PR-URL: #42658 Reviewed-By: Antoine du Hamel <[email protected]>
1 parent 24adba6 commit adaf602

23 files changed

+669
-128
lines changed

doc/api/cli.md

+11
Original file line numberDiff line numberDiff line change
@@ -1052,6 +1052,16 @@ minimum allocation from the secure heap. The minimum value is `2`.
10521052
The maximum value is the lesser of `--secure-heap` or `2147483647`.
10531053
The value given must be a power of two.
10541054

1055+
### `--test`
1056+
1057+
<!-- YAML
1058+
added: REPLACEME
1059+
-->
1060+
1061+
Starts the Node.js command line test runner. This flag cannot be combined with
1062+
`--check`, `--eval`, `--interactive`, or the inspector. See the documentation
1063+
on [running tests from the command line][] for more details.
1064+
10551065
### `--test-only`
10561066

10571067
<!-- YAML
@@ -2033,6 +2043,7 @@ $ node --max-old-space-size=1536 index.js
20332043
[jitless]: https://v8.dev/blog/jitless
20342044
[libuv threadpool documentation]: https://docs.libuv.org/en/latest/threadpool.html
20352045
[remote code execution]: https://www.owasp.org/index.php/Code_Injection
2046+
[running tests from the command line]: test.md#running-tests-from-the-command-line
20362047
[security warning]: #warning-binding-inspector-to-a-public-ipport-combination-is-insecure
20372048
[timezone IDs]: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
20382049
[ways that `TZ` is handled in other environments]: https://www.gnu.org/software/libc/manual/html_node/TZ-Variable.html

doc/api/test.md

+63
Original file line numberDiff line numberDiff line change
@@ -219,6 +219,67 @@ test('a test that creates asynchronous activity', (t) => {
219219
});
220220
```
221221

222+
## Running tests from the command line
223+
224+
The Node.js test runner can be invoked from the command line by passing the
225+
[`--test`][] flag:
226+
227+
```bash
228+
node --test
229+
```
230+
231+
By default, Node.js will recursively search the current directory for
232+
JavaScript source files matching a specific naming convention. Matching files
233+
are executed as test files. More information on the expected test file naming
234+
convention and behavior can be found in the [test runner execution model][]
235+
section.
236+
237+
Alternatively, one or more paths can be provided as the final argument(s) to
238+
the Node.js command, as shown below.
239+
240+
```bash
241+
node --test test1.js test2.mjs custom_test_dir/
242+
```
243+
244+
In this example, the test runner will execute the files `test1.js` and
245+
`test2.mjs`. The test runner will also recursively search the
246+
`custom_test_dir/` directory for test files to execute.
247+
248+
### Test runner execution model
249+
250+
When searching for test files to execute, the test runner behaves as follows:
251+
252+
* Any files explicitly provided by the user are executed.
253+
* If the user did not explicitly specify any paths, the current working
254+
directory is recursively searched for files as specified in the following
255+
steps.
256+
* `node_modules` directories are skipped unless explicitly provided by the
257+
user.
258+
* If a directory named `test` is encountered, the test runner will search it
259+
recursively for all all `.js`, `.cjs`, and `.mjs` files. All of these files
260+
are treated as test files, and do not need to match the specific naming
261+
convention detailed below. This is to accommodate projects that place all of
262+
their tests in a single `test` directory.
263+
* In all other directories, `.js`, `.cjs`, and `.mjs` files matching the
264+
following patterns are treated as test files:
265+
* `^test$` - Files whose basename is the string `'test'`. Examples:
266+
`test.js`, `test.cjs`, `test.mjs`.
267+
* `^test-.+` - Files whose basename starts with the string `'test-'`
268+
followed by one or more characters. Examples: `test-example.js`,
269+
`test-another-example.mjs`.
270+
* `.+[\.\-\_]test$` - Files whose basename ends with `.test`, `-test`, or
271+
`_test`, preceded by one or more characters. Examples: `example.test.js`,
272+
`example-test.cjs`, `example_test.mjs`.
273+
* Other file types understood by Node.js such as `.node` and `.json` are not
274+
automatically executed by the test runner, but are supported if explicitly
275+
provided on the command line.
276+
277+
Each matching test file is executed in a separate child process. If the child
278+
process finishes with an exit code of 0, the test is considered passing.
279+
Otherwise, the test is considered to be a failure. Test files must be
280+
executable by Node.js, but are not required to use the `node:test` module
281+
internally.
282+
222283
## `test([name][, options][, fn])`
223284

224285
<!-- YAML
@@ -368,5 +429,7 @@ behaves in the same fashion as the top level [`test()`][] function.
368429

369430
[TAP]: https://testanything.org/
370431
[`--test-only`]: cli.md#--test-only
432+
[`--test`]: cli.md#--test
371433
[`TestContext`]: #class-testcontext
372434
[`test()`]: #testname-options-fn
435+
[test runner execution model]: #test-runner-execution-model

doc/node.1

+3
Original file line numberDiff line numberDiff line change
@@ -381,6 +381,9 @@ the secure heap. The default is 0. The value must be a power of two.
381381
.It Fl -secure-heap-min Ns = Ns Ar n
382382
Specify the minimum allocation from the OpenSSL secure heap. The default is 2. The value must be a power of two.
383383
.
384+
.It Fl -test
385+
Starts the Node.js command line test runner.
386+
.
384387
.It Fl -test-only
385388
Configures the test runner to only execute top level tests that have the `only`
386389
option set.

lib/internal/errors.js

+17-3
Original file line numberDiff line numberDiff line change
@@ -207,6 +207,16 @@ function isErrorStackTraceLimitWritable() {
207207
desc.set !== undefined;
208208
}
209209

210+
function inspectWithNoCustomRetry(obj, options) {
211+
const utilInspect = lazyInternalUtilInspect();
212+
213+
try {
214+
return utilInspect.inspect(obj, options);
215+
} catch {
216+
return utilInspect.inspect(obj, { ...options, customInspect: false });
217+
}
218+
}
219+
210220
// A specialized Error that includes an additional info property with
211221
// additional information about the error condition.
212222
// It has the properties present in a UVException but with a custom error
@@ -862,6 +872,7 @@ module.exports = {
862872
getMessage,
863873
hideInternalStackFrames,
864874
hideStackFrames,
875+
inspectWithNoCustomRetry,
865876
isErrorStackTraceLimitWritable,
866877
isStackOverflowError,
867878
kEnhanceStackBeforeInspector,
@@ -1549,11 +1560,14 @@ E('ERR_TEST_FAILURE', function(error, failureType) {
15491560
assert(typeof failureType === 'string',
15501561
"The 'failureType' argument must be of type string.");
15511562

1552-
const msg = error?.message ?? lazyInternalUtilInspect().inspect(error);
1563+
let msg = error?.message ?? error;
15531564

1554-
this.failureType = error?.failureType ?? failureType;
1555-
this.cause = error;
1565+
if (typeof msg !== 'string') {
1566+
msg = inspectWithNoCustomRetry(msg);
1567+
}
15561568

1569+
this.failureType = failureType;
1570+
this.cause = error;
15571571
return msg;
15581572
}, Error);
15591573
E('ERR_TLS_CERT_ALTNAME_FORMAT', 'Invalid subject alternative name string',

lib/internal/main/test_runner.js

+157
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
'use strict';
2+
const {
3+
ArrayFrom,
4+
ArrayPrototypeFilter,
5+
ArrayPrototypeIncludes,
6+
ArrayPrototypePush,
7+
ArrayPrototypeSlice,
8+
ArrayPrototypeSort,
9+
Promise,
10+
SafeSet,
11+
} = primordials;
12+
const {
13+
prepareMainThreadExecution,
14+
} = require('internal/bootstrap/pre_execution');
15+
const { spawn } = require('child_process');
16+
const { readdirSync, statSync } = require('fs');
17+
const console = require('internal/console/global');
18+
const {
19+
codes: {
20+
ERR_TEST_FAILURE,
21+
},
22+
} = require('internal/errors');
23+
const test = require('internal/test_runner/harness');
24+
const { kSubtestsFailed } = require('internal/test_runner/test');
25+
const {
26+
isSupportedFileType,
27+
doesPathMatchFilter,
28+
} = require('internal/test_runner/utils');
29+
const { basename, join, resolve } = require('path');
30+
const kFilterArgs = ['--test'];
31+
32+
prepareMainThreadExecution(false);
33+
markBootstrapComplete();
34+
35+
// TODO(cjihrig): Replace this with recursive readdir once it lands.
36+
function processPath(path, testFiles, options) {
37+
const stats = statSync(path);
38+
39+
if (stats.isFile()) {
40+
if (options.userSupplied ||
41+
(options.underTestDir && isSupportedFileType(path)) ||
42+
doesPathMatchFilter(path)) {
43+
testFiles.add(path);
44+
}
45+
} else if (stats.isDirectory()) {
46+
const name = basename(path);
47+
48+
if (!options.userSupplied && name === 'node_modules') {
49+
return;
50+
}
51+
52+
// 'test' directories get special treatment. Recursively add all .js,
53+
// .cjs, and .mjs files in the 'test' directory.
54+
const isTestDir = name === 'test';
55+
const { underTestDir } = options;
56+
const entries = readdirSync(path);
57+
58+
if (isTestDir) {
59+
options.underTestDir = true;
60+
}
61+
62+
options.userSupplied = false;
63+
64+
for (let i = 0; i < entries.length; i++) {
65+
processPath(join(path, entries[i]), testFiles, options);
66+
}
67+
68+
options.underTestDir = underTestDir;
69+
}
70+
}
71+
72+
function createTestFileList() {
73+
const cwd = process.cwd();
74+
const hasUserSuppliedPaths = process.argv.length > 1;
75+
const testPaths = hasUserSuppliedPaths ?
76+
ArrayPrototypeSlice(process.argv, 1) : [cwd];
77+
const testFiles = new SafeSet();
78+
79+
try {
80+
for (let i = 0; i < testPaths.length; i++) {
81+
const absolutePath = resolve(testPaths[i]);
82+
83+
processPath(absolutePath, testFiles, { userSupplied: true });
84+
}
85+
} catch (err) {
86+
if (err?.code === 'ENOENT') {
87+
console.error(`Could not find '${err.path}'`);
88+
process.exit(1);
89+
}
90+
91+
throw err;
92+
}
93+
94+
return ArrayPrototypeSort(ArrayFrom(testFiles));
95+
}
96+
97+
function filterExecArgv(arg) {
98+
return !ArrayPrototypeIncludes(kFilterArgs, arg);
99+
}
100+
101+
function runTestFile(path) {
102+
return test(path, () => {
103+
return new Promise((resolve, reject) => {
104+
const args = ArrayPrototypeFilter(process.execArgv, filterExecArgv);
105+
ArrayPrototypePush(args, path);
106+
107+
const child = spawn(process.execPath, args);
108+
// TODO(cjihrig): Implement a TAP parser to read the child's stdout
109+
// instead of just displaying it all if the child fails.
110+
let stdout = '';
111+
let stderr = '';
112+
let err;
113+
114+
child.on('error', (error) => {
115+
err = error;
116+
});
117+
118+
child.stdout.setEncoding('utf8');
119+
child.stderr.setEncoding('utf8');
120+
121+
child.stdout.on('data', (chunk) => {
122+
stdout += chunk;
123+
});
124+
125+
child.stderr.on('data', (chunk) => {
126+
stderr += chunk;
127+
});
128+
129+
child.once('exit', (code, signal) => {
130+
if (code !== 0 || signal !== null) {
131+
if (!err) {
132+
err = new ERR_TEST_FAILURE('test failed', kSubtestsFailed);
133+
err.exitCode = code;
134+
err.signal = signal;
135+
err.stdout = stdout;
136+
err.stderr = stderr;
137+
// The stack will not be useful since the failures came from tests
138+
// in a child process.
139+
err.stack = undefined;
140+
}
141+
142+
return reject(err);
143+
}
144+
145+
resolve();
146+
});
147+
});
148+
});
149+
}
150+
151+
(async function main() {
152+
const testFiles = createTestFileList();
153+
154+
for (let i = 0; i < testFiles.length; i++) {
155+
runTestFile(testFiles[i]);
156+
}
157+
})();

0 commit comments

Comments
 (0)