Skip to content

Commit be14d2f

Browse files
committed
test_runner: add initial CLI runner
This commit introduces an initial version of a CLI-based test runner.
1 parent d835b1f commit be14d2f

23 files changed

+632
-128
lines changed

doc/api/cli.md

+10
Original file line numberDiff line numberDiff line change
@@ -1052,6 +1052,15 @@ minimum allocation from the secure heap. The minimum value is `2`.
10521052
The maximum value is the lesser of `--secure-heap` or `2147483647`.
10531053
The value given must be a power of two.
10541054

1055+
### `--test`
1056+
1057+
<!-- YAML
1058+
added: REPLACEME
1059+
-->
1060+
1061+
Starts the Node.js command line test runner. See the documentation on
1062+
[running tests from the command line][] for more details.
1063+
10551064
### `--test-only`
10561065

10571066
<!-- YAML
@@ -2033,6 +2042,7 @@ $ node --max-old-space-size=1536 index.js
20332042
[jitless]: https://v8.dev/blog/jitless
20342043
[libuv threadpool documentation]: https://docs.libuv.org/en/latest/threadpool.html
20352044
[remote code execution]: https://www.owasp.org/index.php/Code_Injection
2045+
[running tests from the command line]: test.md#running-tests-from-the-command-line
20362046
[security warning]: #warning-binding-inspector-to-a-public-ipport-combination-is-insecure
20372047
[timezone IDs]: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
20382048
[ways that `TZ` is handled in other environments]: https://www.gnu.org/software/libc/manual/html_node/TZ-Variable.html

doc/api/test.md

+60
Original file line numberDiff line numberDiff line change
@@ -219,6 +219,64 @@ test('a test that creates asynchronous activity', (t) => {
219219
});
220220
```
221221

222+
## Running tests from the command line
223+
224+
The Node.js test runner can be invoked from the command line by passing the
225+
[`--test`][] flag:
226+
227+
```bash
228+
node --test
229+
```
230+
231+
By default, Node.js will recursively search the current directory for
232+
JavaScript source files matching a specific naming convention. Matching files
233+
are executed as test files. More information on the expected test file naming
234+
convention and behavior can be found in the [test runner execution model][]
235+
section.
236+
237+
Alternatively, one or more paths can be provided as the final argument(s) to
238+
the Node.js command, as shown below.
239+
240+
```bash
241+
node --test test1.js test2.mjs custom_test_dir/
242+
```
243+
244+
In this example, the test runner will execute the files `test1.js` and
245+
`test2.mjs`. The test runner will also recursively search the
246+
`custom_test_dir/` directory for test files to execute.
247+
248+
### Test runner execution model
249+
250+
When searching for test files to execute, the test runner behaves as follows:
251+
252+
* Any files explicitly provided by the user are executed.
253+
* If the user did not explicitly specify any paths, the current working
254+
directory is recursively searched for files as specified in the following
255+
steps.
256+
* `node_modules` directories are skipped unless explicitly provided by the
257+
user.
258+
* If a directory named `test` is encountered, the test runner will search it
259+
recursively for all all `.js`, `.cjs`, and `.mjs` files. All of these files
260+
are treated as test files, and do not need to match the specific naming
261+
convention detailed below. This is to accommodate projects that place all of
262+
their tests in a single `test` directory.
263+
* In all other directories, `.js`, `.cjs`, and `.mjs` files matching the
264+
following patterns are treated as test files:
265+
* `^test$` - Files whose basename is the string `'test'`. Examples:
266+
`test.js`, `test.cjs`, `test.mjs`.
267+
* `^test-.+` - Files whose basename starts with the string `'test-'`
268+
followed by one or more characters. Examples: `test-example.js`,
269+
`test-another-example.mjs`.
270+
* `.+[\.\-\_]test$` - Files whose basename ends with `.test`, `-test`, or
271+
`_test`, preceded by one or more characters. Examples: `example.test.js`,
272+
`example-test.cjs`, `example_test.mjs`.
273+
274+
Each matching test file is executed in a separate child process. If the child
275+
process finishes with an exit code of 0, the test is considered passing.
276+
Otherwise, the test is considered to be a failure. Test files must be
277+
executable by Node.js, but are not required to use the `node:test` module
278+
internally.
279+
222280
## `test([name][, options][, fn])`
223281

224282
<!-- YAML
@@ -368,5 +426,7 @@ behaves in the same fashion as the top level [`test()`][] function.
368426

369427
[TAP]: https://testanything.org/
370428
[`--test-only`]: cli.md#--test-only
429+
[`--test`]: cli.md#--test
371430
[`TestContext`]: #class-testcontext
372431
[`test()`]: #testname-options-fn
432+
[test runner execution model]: #test-runner-execution-model

doc/node.1

+3
Original file line numberDiff line numberDiff line change
@@ -381,6 +381,9 @@ the secure heap. The default is 0. The value must be a power of two.
381381
.It Fl -secure-heap-min Ns = Ns Ar n
382382
Specify the minimum allocation from the OpenSSL secure heap. The default is 2. The value must be a power of two.
383383
.
384+
.It Fl -test
385+
Starts the Node.js command line test runner.
386+
.
384387
.It Fl -test-only
385388
Configures the test runner to only execute top level tests that have the `only`
386389
option set.

lib/internal/errors.js

+17-3
Original file line numberDiff line numberDiff line change
@@ -207,6 +207,16 @@ function isErrorStackTraceLimitWritable() {
207207
desc.set !== undefined;
208208
}
209209

210+
function inspectWithNoCustomRetry(obj, options) {
211+
const utilInspect = lazyInternalUtilInspect();
212+
213+
try {
214+
return utilInspect.inspect(obj, options);
215+
} catch {
216+
return utilInspect.inspect(obj, { ...options, customInspect: false });
217+
}
218+
}
219+
210220
// A specialized Error that includes an additional info property with
211221
// additional information about the error condition.
212222
// It has the properties present in a UVException but with a custom error
@@ -862,6 +872,7 @@ module.exports = {
862872
getMessage,
863873
hideInternalStackFrames,
864874
hideStackFrames,
875+
inspectWithNoCustomRetry,
865876
isErrorStackTraceLimitWritable,
866877
isStackOverflowError,
867878
kEnhanceStackBeforeInspector,
@@ -1549,11 +1560,14 @@ E('ERR_TEST_FAILURE', function(error, failureType) {
15491560
assert(typeof failureType === 'string',
15501561
"The 'failureType' argument must be of type string.");
15511562

1552-
const msg = error?.message ?? lazyInternalUtilInspect().inspect(error);
1563+
let msg = error?.message ?? error;
15531564

1554-
this.failureType = error?.failureType ?? failureType;
1555-
this.cause = error;
1565+
if (typeof msg !== 'string') {
1566+
msg = inspectWithNoCustomRetry(msg);
1567+
}
15561568

1569+
this.failureType = failureType;
1570+
this.cause = error;
15571571
return msg;
15581572
}, Error);
15591573
E('ERR_TLS_CERT_ALTNAME_FORMAT', 'Invalid subject alternative name string',

lib/internal/main/test_runner.js

+155
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,155 @@
1+
'use strict';
2+
const {
3+
ArrayFrom,
4+
ArrayPrototypeFilter,
5+
ArrayPrototypeIncludes,
6+
ArrayPrototypePush,
7+
ArrayPrototypeSlice,
8+
ArrayPrototypeSort,
9+
Promise,
10+
SafeSet,
11+
} = primordials;
12+
const {
13+
prepareMainThreadExecution,
14+
} = require('internal/bootstrap/pre_execution');
15+
const { spawn } = require('child_process');
16+
const { readdirSync, statSync } = require('fs');
17+
const console = require('internal/console/global');
18+
const {
19+
codes: {
20+
ERR_TEST_FAILURE,
21+
},
22+
} = require('internal/errors');
23+
const test = require('internal/test_runner/harness');
24+
const { kSubtestsFailed } = require('internal/test_runner/test');
25+
const {
26+
isSupportedFileType,
27+
doesPathMatchFilter,
28+
} = require('internal/test_runner/utils');
29+
const { basename, join, resolve } = require('path');
30+
const kFilterArgs = ['--test'];
31+
32+
prepareMainThreadExecution(false);
33+
markBootstrapComplete();
34+
35+
// TODO(cjihrig): Replace this with recursive readdir once it lands.
36+
function processPath(path, testFiles, options) {
37+
const stats = statSync(path);
38+
39+
if (stats.isFile()) {
40+
if (options.userSupplied ||
41+
(options.underTestDir && isSupportedFileType(path)) ||
42+
doesPathMatchFilter(path)) {
43+
testFiles.add(path);
44+
}
45+
} else if (stats.isDirectory()) {
46+
const name = basename(path);
47+
48+
if (!options.userSupplied && name === 'node_modules') {
49+
return;
50+
}
51+
52+
// 'test' directories get special treatment. Recursively add all .js,
53+
// .cjs, and .mjs files in the 'test' directory.
54+
const isTestDir = name === 'test';
55+
const { underTestDir } = options;
56+
const entries = readdirSync(path);
57+
58+
if (isTestDir) {
59+
options.underTestDir = true;
60+
}
61+
62+
options.userSupplied = false;
63+
64+
for (let i = 0; i < entries.length; i++) {
65+
processPath(join(path, entries[i]), testFiles, options);
66+
}
67+
68+
options.underTestDir = underTestDir;
69+
}
70+
}
71+
72+
function createTestFileList() {
73+
const cwd = process.cwd();
74+
const hasUserSuppliedPaths = process.argv.length > 1;
75+
const testPaths = hasUserSuppliedPaths ?
76+
ArrayPrototypeSlice(process.argv, 1) : [cwd];
77+
const testFiles = new SafeSet();
78+
79+
try {
80+
for (let i = 0; i < testPaths.length; i++) {
81+
const absolutePath = resolve(testPaths[i]);
82+
83+
processPath(absolutePath, testFiles, { userSupplied: true });
84+
}
85+
} catch (err) {
86+
if (err?.code === 'ENOENT') {
87+
console.error(`Could not find '${err.path}'`);
88+
process.exit(1);
89+
}
90+
91+
throw err;
92+
}
93+
94+
return ArrayPrototypeSort(ArrayFrom(testFiles));
95+
}
96+
97+
function runTestFile(path) {
98+
return test(path, () => {
99+
return new Promise((resolve, reject) => {
100+
const args = ArrayPrototypeFilter(process.execArgv, (arg) => {
101+
return !ArrayPrototypeIncludes(kFilterArgs, arg);
102+
});
103+
ArrayPrototypePush(args, path);
104+
105+
const child = spawn(process.execPath, args);
106+
// TODO(cjihrig): Implement a TAP parser to read the child's stdout
107+
// instead of just displaying it all if the child fails.
108+
let stdout = '';
109+
let stderr = '';
110+
let err;
111+
112+
child.on('error', (error) => {
113+
err = error;
114+
});
115+
116+
child.stdout.setEncoding('utf8');
117+
child.stderr.setEncoding('utf8');
118+
119+
child.stdout.on('data', (chunk) => {
120+
stdout += chunk;
121+
});
122+
123+
child.stderr.on('data', (chunk) => {
124+
stderr += chunk;
125+
});
126+
127+
child.once('exit', (code, signal) => {
128+
if (code !== 0 || signal !== null) {
129+
if (!err) {
130+
err = new ERR_TEST_FAILURE('test failed', kSubtestsFailed);
131+
err.exitCode = code;
132+
err.signal = signal;
133+
err.stdout = stdout;
134+
err.stderr = stderr;
135+
// The stack will not be useful since the failures came from tests
136+
// in a child process.
137+
err.stack = undefined;
138+
}
139+
140+
return reject(err);
141+
}
142+
143+
resolve();
144+
});
145+
});
146+
});
147+
}
148+
149+
(async function main() {
150+
const testFiles = createTestFileList();
151+
152+
for (let i = 0; i < testFiles.length; i++) {
153+
runTestFile(testFiles[i]);
154+
}
155+
})();

0 commit comments

Comments
 (0)