test_runner: add snapshot testing

This commit adds a t.assert.snapshot() method that implements
snapshot testing. Serialization uses JSON.stringify() by default,
but users can configure the serialization to meet their needs.

PR-URL: nodejs#53169
Fixes: nodejs#48260
Reviewed-By: Moshe Atlow <[email protected]>
Reviewed-By: Benjamin Gruenbaum <[email protected]>
Reviewed-By: Geoffrey Booth <[email protected]>

Author: cjihrig

doc/api/cli.md

@@ -991,6 +991,16 @@ added: REPLACEME
 
 Enable module mocking in the test runner.
 
+### `--experimental-test-snapshots`
+
+<!-- YAML
+added: REPLACEME
+-->
+
+> Stability: 1.0 - Early development
+
+Enable [snapshot testing][] in the test runner.
+
 ### `--experimental-vm-modules`
 
 <!-- YAML
@@ -2129,6 +2139,18 @@ added:
 A number of milliseconds the test execution will fail after. If unspecified,
 subtests inherit this value from their parent. The default value is `Infinity`.
 
+### `--test-update-snapshots`
+
+<!-- YAML
+added: REPLACEME
+-->
+
+> Stability: 1.0 - Early development
+
+Regenerates the snapshot file used by the test runner for [snapshot testing][].
+Node.js must be started with the `--experimental-test-snapshots` flag in order
+to use this functionality.
+
 ### `--throw-deprecation`
 
 <!-- YAML
@@ -3269,6 +3291,7 @@ node --stack-trace-limit=12 -p -e "Error.stackTraceLimit" # prints 12
 [security warning]: #warning-binding-inspector-to-a-public-ipport-combination-is-insecure
 [semi-space]: https://www.memorymanagement.org/glossary/s.html#semi.space
 [single executable application]: single-executable-applications.md
+[snapshot testing]: test.md#snapshot-testing
 [test reporters]: test.md#test-reporters
 [timezone IDs]: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
 [tracking issue for user-land snapshots]: https://github.com/nodejs/node/issues/44014

doc/api/test.md

@@ -923,6 +923,61 @@ test('runs timers as setTime passes ticks', (context) => {
 });
 ```
 
+## Snapshot testing
+
+> Stability: 1.0 - Early development
+
+Snapshot tests allow arbitrary values to be serialized into string values and
+compared against a set of known good values. The known good values are known as
+snapshots, and are stored in a snapshot file. Snapshot files are managed by the
+test runner, but are designed to be human readable to aid in debugging. Best
+practice is for snapshot files to be checked into source control along with your
+test files. In order to enable snapshot testing, Node.js must be started with
+the [`--experimental-test-snapshots`][] command-line flag.
+
+Snapshot files are generated by starting Node.js with the
+[`--test-update-snapshots`][] command-line flag. A separate snapshot file is
+generated for each test file. By default, the snapshot file has the same name
+as `process.argv[1]` with a `.snapshot` file extension. This behavior can be
+configured using the `snapshot.setResolveSnapshotPath()` function. Each
+snapshot assertion corresponds to an export in the snapshot file.
+
+An example snapshot test is shown below. The first time this test is executed,
+it will fail because the corresponding snapshot file does not exist.
+
+```js
+// test.js
+suite('suite of snapshot tests', () => {
+  test('snapshot test', (t) => {
+    t.assert.snapshot({ value1: 1, value2: 2 });
+    t.assert.snapshot(5);
+  });
+});
+```
+
+Generate the snapshot file by running the test file with
+`--test-update-snapshots`. The test should pass, and a file named
+`test.js.snapshot` is created in the same directory as the test file. The
+contents of the snapshot file are shown below. Each snapshot is identified by
+the full name of test and a counter to differentiate between snapshots in the
+same test.
+
+```js
+exports[`suite of snapshot tests > snapshot test 1`] = `
+{
+  "value1": 1,
+  "value2": 2
+}
+`;
+
+exports[`suite of snapshot tests > snapshot test 2`] = `
+5
+`;
+```
+
+Once the snapshot file is created, run the tests again without the
+`--test-update-snapshots` flag. The tests should pass now.
+
 ## Test reporters
 
 <!-- YAML
@@ -1625,6 +1680,54 @@ describe('tests', async () => {
 });
 ```
 
+## `snapshot`
+
+<!-- YAML
+added: REPLACEME
+-->
+
+> Stability: 1.0 - Early development
+
+An object whose methods are used to configure default snapshot settings in the
+current process. It is possible to apply the same configuration to all files by
+placing common configuration code in a module preloaded with `--require` or
+`--import`.
+
+### `snapshot.setDefaultSnapshotSerializers(serializers)`
+
+<!-- YAML
+added: REPLACEME
+-->
+
+> Stability: 1.0 - Early development
+
+* `serializers` {Array} An array of synchronous functions used as the default
+  serializers for snapshot tests.
+
+This function is used to customize the default serialization mechanism used by
+the test runner. By default, the test runner performs serialization by calling
+`JSON.stringify(value, null, 2)` on the provided value. `JSON.stringify()` does
+have limitations regarding circular structures and supported data types. If a
+more robust serialization mechanism is required, this function should be used.
+
+### `snapshot.setResolveSnapshotPath(fn)`
+
+<!-- YAML
+added: REPLACEME
+-->
+
+> Stability: 1.0 - Early development
+
+* `fn` {Function} A function used to compute the location of the snapshot file.
+  The function receives the path of the test file as its only argument. If the
+  `process.argv[1]` is not associated with a file (for example in the REPL),
+  the input is undefined. `fn()` must return a string specifying the location of
+  the snapshot file.
+
+This function is used to customize the location of the snapshot file used for
+snapshot testing. By default, the snapshot filename is the same as the entry
+point filename with a `.snapshot` file extension.
+
 ## Class: `MockFunctionContext`
 
 <!-- YAML
@@ -3045,6 +3148,41 @@ test('test', (t) => {
 });
 ```
 
+#### `context.assert.snapshot(value[, options])`
+
+<!-- YAML
+added: REPLACEME
+-->
+
+> Stability: 1.0 - Early development
+
+* `value` {any} A value to serialize to a string. If Node.js was started with
+  the [`--test-update-snapshots`][] flag, the serialized value is written to
+  the snapshot file. Otherwise, the serialized value is compared to the
+  corresponding value in the existing snapshot file.
+* `options` {Object} Optional configuration options. The following properties
+  are supported:
+  * `serializers` {Array} An array of synchronous functions used to serialize
+    `value` into a string. `value` is passed as the only argument to the first
+    serializer function. The return value of each serializer is passed as input
+    to the next serializer. Once all serializers have run, the resulting value
+    is coerced to a string. **Default:** If no serializers are provided, the
+    test runner's default serializers are used.
+
+This function implements assertions for snapshot testing.
+
+```js
+test('snapshot test with default serialization', (t) => {
+  t.assert.snapshot({ value1: 1, value2: 2 });
+});
+
+test('snapshot test with custom serialization', (t) => {
+  t.assert.snapshot({ value3: 3, value4: 4 }, {
+    serializers: [(value) => JSON.stringify(value)]
+  });
+});
+```
+
 ### `context.diagnostic(message)`
 
 <!-- YAML
@@ -3323,13 +3461,15 @@ Can be used to abort test subtasks when the test has been aborted.
 [TAP]: https://testanything.org/
 [TTY]: tty.md
 [`--experimental-test-coverage`]: cli.md#--experimental-test-coverage
+[`--experimental-test-snapshots`]: cli.md#--experimental-test-snapshots
 [`--import`]: cli.md#--importmodule
 [`--test-concurrency`]: cli.md#--test-concurrency
 [`--test-name-pattern`]: cli.md#--test-name-pattern
 [`--test-only`]: cli.md#--test-only
 [`--test-reporter-destination`]: cli.md#--test-reporter-destination
 [`--test-reporter`]: cli.md#--test-reporter
 [`--test-skip-pattern`]: cli.md#--test-skip-pattern
+[`--test-update-snapshots`]: cli.md#--test-update-snapshots
 [`--test`]: cli.md#--test
 [`MockFunctionContext`]: #class-mockfunctioncontext
 [`MockTimers`]: #class-mocktimers

doc/node.1

@@ -188,6 +188,9 @@ Enable code coverage in the test runner.
 .It Fl -experimental-test-module-mocks
 Enable module mocking in the test runner.
 .
+.It Fl -experimental-test-snapshots
+Enable snapshot testing in the test runner.
+.
 .It Fl -experimental-eventsource
 Enable experimental support for the EventSource Web API.
 .
@@ -451,6 +454,9 @@ whose name matches the provided pattern.
 .It Fl -test-timeout
 A number of milliseconds the test execution will fail after.
 .
+.It Fl -test-update-snapshots
+Regenerates the snapshot file used by the test runner for snapshot testing.
+.
 .It Fl -throw-deprecation
 Throw errors for deprecations.
 .

lib/internal/test_runner/harness.js

@@ -210,6 +210,7 @@ function setup(root) {
     counters: null,
     shouldColorizeTestFiles: false,
     teardown: exitHandler,
+    snapshotManager: null,
   };
   root.harness.resetCounters();
   root.startTime = hrtime();