Skip to content

Commit f8a34dc

Browse files
committed
test_runner: add initial CLI runner
This commit introduces an initial version of a CLI-based test runner.
1 parent 3ac7f86 commit f8a34dc

23 files changed

+625
-128
lines changed

doc/api/cli.md

+10
Original file line numberDiff line numberDiff line change
@@ -1052,6 +1052,15 @@ minimum allocation from the secure heap. The minimum value is `2`.
10521052
The maximum value is the lesser of `--secure-heap` or `2147483647`.
10531053
The value given must be a power of two.
10541054

1055+
### `--test`
1056+
1057+
<!-- YAML
1058+
added: REPLACEME
1059+
-->
1060+
1061+
Starts the Node.js command line test runner. See the documentation on
1062+
[running tests from the command line][] for more details.
1063+
10551064
### `--test-only`
10561065

10571066
<!-- YAML
@@ -2033,6 +2042,7 @@ $ node --max-old-space-size=1536 index.js
20332042
[jitless]: https://v8.dev/blog/jitless
20342043
[libuv threadpool documentation]: https://docs.libuv.org/en/latest/threadpool.html
20352044
[remote code execution]: https://www.owasp.org/index.php/Code_Injection
2045+
[running tests from the command line]: test.md#running-tests-from-the-command-line
20362046
[security warning]: #warning-binding-inspector-to-a-public-ipport-combination-is-insecure
20372047
[timezone IDs]: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
20382048
[ways that `TZ` is handled in other environments]: https://www.gnu.org/software/libc/manual/html_node/TZ-Variable.html

doc/api/test.md

+60
Original file line numberDiff line numberDiff line change
@@ -219,6 +219,64 @@ test('a test that creates asynchronous activity', (t) => {
219219
});
220220
```
221221

222+
## Running tests from the command line
223+
224+
The Node.js test runner can be invoked from the command line by passing the
225+
[`--test`][] flag:
226+
227+
```bash
228+
node --test
229+
```
230+
231+
By default, Node.js will recursively search the current directory for
232+
JavaScript source files matching a specific naming convention. Matching files
233+
are executed as test files. More information on the expected test file naming
234+
convention and behavior can be found in the [test runner execution model][]
235+
section. If the current directory contains a directory named `test`, Node.js
236+
will also recursively search that directory for any file with the extensions
237+
`.js`, `.cjs`, and `.mjs`.
238+
239+
Alternatively, one or more paths can be provided as the final argument(s) to
240+
the Node.js command, as shown below.
241+
242+
```bash
243+
node --test test1.js test2.mjs custom_test_dir/
244+
```
245+
246+
In this example, the test runner will execute the files `test1.js` and
247+
`test2.mjs`. The test runner will also recursively search the
248+
`custom_test_dir/` directory for test files to execute.
249+
250+
### Test runner execution model
251+
252+
When searching for test files to execute, the test runner behaves as follows:
253+
254+
* Any files explicitly provided by the user are executed.
255+
* If the user did not explicitly specify any paths, the current working
256+
directory is recursively searched for files as specified in the following
257+
steps.
258+
* If the user did not explicitly specify any paths and the current working
259+
directory contains a directory named `test`, the `test` directory is
260+
recursively searched for all `.js`, `.cjs`, and `.mjs` files.
261+
* `node_modules` directories are skipped unless explicitly provided by the
262+
user.
263+
* `.js`, `.cjs`, and `.mjs` files matching the following patterns are
264+
treated as test files:
265+
* `^test$` - Files whose basename is the string `'test'`. Examples:
266+
`test.js`, `test.cjs`, `test.mjs`.
267+
* `^test-.+` - Files whose basename starts with the string `'test-'`
268+
followed by one or more characters. Examples: `test-example.js`,
269+
`test-another-example.mjs`.
270+
* `.+[\.\-\_]test$` - Files whose basename ends with `.test`, `-test`, or
271+
`_test`, preceded by one or more characters. Examples: `example.test.js`,
272+
`example-test.cjs`, `example_test.mjs`.
273+
274+
Each matching test file is executed in a separate child process. If the child
275+
process finishes with an exit code of 0, the test is considered passing.
276+
Otherwise, the test is considered to be a failure. Test files must be
277+
executable by Node.js, but are not required to use the `node:test` module
278+
internally.
279+
222280
## `test([name][, options][, fn])`
223281

224282
<!-- YAML
@@ -368,5 +426,7 @@ behaves in the same fashion as the top level [`test()`][] function.
368426

369427
[TAP]: https://testanything.org/
370428
[`--test-only`]: cli.md#--test-only
429+
[`--test`]: cli.md#--test
371430
[`TestContext`]: #class-testcontext
372431
[`test()`]: #testname-options-fn
432+
[test runner execution model]: #test-runner-execution-model

doc/node.1

+3
Original file line numberDiff line numberDiff line change
@@ -381,6 +381,9 @@ the secure heap. The default is 0. The value must be a power of two.
381381
.It Fl -secure-heap-min Ns = Ns Ar n
382382
Specify the minimum allocation from the OpenSSL secure heap. The default is 2. The value must be a power of two.
383383
.
384+
.It Fl -test
385+
Starts the Node.js command line test runner.
386+
.
384387
.It Fl -test-only
385388
Configures the test runner to only execute top level tests that have the `only`
386389
option set.

lib/internal/errors.js

+17-3
Original file line numberDiff line numberDiff line change
@@ -207,6 +207,16 @@ function isErrorStackTraceLimitWritable() {
207207
desc.set !== undefined;
208208
}
209209

210+
function inspectWithNoCustomRetry(obj, options) {
211+
const utilInspect = lazyInternalUtilInspect();
212+
213+
try {
214+
return utilInspect.inspect(obj, options);
215+
} catch {
216+
return utilInspect.inspect(obj, { ...options, customInspect: false });
217+
}
218+
}
219+
210220
// A specialized Error that includes an additional info property with
211221
// additional information about the error condition.
212222
// It has the properties present in a UVException but with a custom error
@@ -862,6 +872,7 @@ module.exports = {
862872
getMessage,
863873
hideInternalStackFrames,
864874
hideStackFrames,
875+
inspectWithNoCustomRetry,
865876
isErrorStackTraceLimitWritable,
866877
isStackOverflowError,
867878
kEnhanceStackBeforeInspector,
@@ -1549,11 +1560,14 @@ E('ERR_TEST_FAILURE', function(error, failureType) {
15491560
assert(typeof failureType === 'string',
15501561
"The 'failureType' argument must be of type string.");
15511562

1552-
const msg = error?.message ?? lazyInternalUtilInspect().inspect(error);
1563+
let msg = error?.message ?? error;
15531564

1554-
this.failureType = error?.failureType ?? failureType;
1555-
this.cause = error;
1565+
if (typeof msg !== 'string') {
1566+
msg = inspectWithNoCustomRetry(msg);
1567+
}
15561568

1569+
this.failureType = failureType;
1570+
this.cause = error;
15571571
return msg;
15581572
}, Error);
15591573
E('ERR_TLS_CERT_ALTNAME_FORMAT', 'Invalid subject alternative name string',

lib/internal/main/test_runner.js

+160
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
'use strict';
2+
const {
3+
ArrayFrom,
4+
ArrayPrototypeFilter,
5+
ArrayPrototypeIncludes,
6+
ArrayPrototypePush,
7+
ArrayPrototypeSlice,
8+
ArrayPrototypeSort,
9+
Promise,
10+
SafeSet,
11+
StringPrototypeEndsWith,
12+
} = primordials;
13+
const {
14+
prepareMainThreadExecution,
15+
} = require('internal/bootstrap/pre_execution');
16+
const { spawn } = require('child_process');
17+
const { readdirSync, statSync } = require('fs');
18+
const console = require('internal/console/global');
19+
const {
20+
codes: {
21+
ERR_TEST_FAILURE,
22+
},
23+
} = require('internal/errors');
24+
const test = require('internal/test_runner/harness');
25+
const { kSubtestsFailed } = require('internal/test_runner/test');
26+
const {
27+
isSupportedFileType,
28+
doesPathMatchFilter,
29+
} = require('internal/test_runner/utils');
30+
const { join, resolve } = require('path');
31+
const kFilterArgs = ['--test'];
32+
33+
prepareMainThreadExecution(false);
34+
markBootstrapComplete();
35+
36+
// TODO(cjihrig): Replace this with recursive readdir once it lands.
37+
function processPath(path, testFiles, options) {
38+
const stats = statSync(path);
39+
40+
if (stats.isFile()) {
41+
if (options.userSupplied ||
42+
(options.underTestDir && isSupportedFileType(path)) ||
43+
doesPathMatchFilter(path)) {
44+
testFiles.add(path);
45+
}
46+
} else if (stats.isDirectory()) {
47+
if (!options.userSupplied &&
48+
StringPrototypeEndsWith(path, 'node_modules')) {
49+
return;
50+
}
51+
52+
const entries = readdirSync(path);
53+
54+
options.userSupplied = false;
55+
56+
for (let i = 0; i < entries.length; i++) {
57+
processPath(join(path, entries[i]), testFiles, options);
58+
}
59+
}
60+
}
61+
62+
function createTestFileList() {
63+
const cwd = process.cwd();
64+
const hasUserSuppliedPaths = process.argv.length > 1;
65+
const testPaths = hasUserSuppliedPaths ?
66+
ArrayPrototypeSlice(process.argv, 1) : [cwd];
67+
const testFiles = new SafeSet();
68+
69+
try {
70+
for (let i = 0; i < testPaths.length; i++) {
71+
const absolutePath = resolve(testPaths[i]);
72+
73+
processPath(absolutePath, testFiles, { userSupplied: true });
74+
}
75+
} catch (err) {
76+
if (err?.code === 'ENOENT') {
77+
console.error(`Could not find '${err.path}'`);
78+
process.exit(1);
79+
}
80+
81+
throw err;
82+
}
83+
84+
if (!hasUserSuppliedPaths) {
85+
try {
86+
// A 'test' directory in the current working directory gets special
87+
// treatment. Recursively add all .js, .cjs, and .mjs files in the test
88+
// directory if it exists.
89+
const testDir = join(cwd, 'test');
90+
91+
processPath(testDir, testFiles, { underTestDir: true });
92+
} catch (err) {
93+
if (err?.code !== 'ENOENT') {
94+
throw err;
95+
}
96+
}
97+
}
98+
99+
return ArrayPrototypeSort(ArrayFrom(testFiles));
100+
}
101+
102+
function runTestFile(path) {
103+
return test(path, () => {
104+
return new Promise((resolve, reject) => {
105+
const args = ArrayPrototypeFilter(process.execArgv, (arg) => {
106+
return !ArrayPrototypeIncludes(kFilterArgs, arg);
107+
});
108+
ArrayPrototypePush(args, path);
109+
110+
const child = spawn(process.execPath, args);
111+
// TODO(cjihrig): Implement a TAP parser to read the child's stdout
112+
// instead of just displaying it all if the child fails.
113+
let stdout = '';
114+
let stderr = '';
115+
let err;
116+
117+
child.on('error', (error) => {
118+
err = error;
119+
});
120+
121+
child.stdout.setEncoding('utf8');
122+
child.stderr.setEncoding('utf8');
123+
124+
child.stdout.on('data', (chunk) => {
125+
stdout += chunk;
126+
});
127+
128+
child.stderr.on('data', (chunk) => {
129+
stderr += chunk;
130+
});
131+
132+
child.once('exit', (code, signal) => {
133+
if (code !== 0 || signal !== null) {
134+
if (!err) {
135+
err = new ERR_TEST_FAILURE('test failed', kSubtestsFailed);
136+
err.exitCode = code;
137+
err.signal = signal;
138+
err.stdout = stdout;
139+
err.stderr = stderr;
140+
// The stack will not be useful since the failures came from tests
141+
// in a child process.
142+
err.stack = undefined;
143+
}
144+
145+
return reject(err);
146+
}
147+
148+
resolve();
149+
});
150+
});
151+
});
152+
}
153+
154+
(async function main() {
155+
const testFiles = createTestFileList();
156+
157+
for (let i = 0; i < testFiles.length; i++) {
158+
runTestFile(testFiles[i]);
159+
}
160+
})();

0 commit comments

Comments
 (0)