module: unflag detect-module

[GeoffreyBooth]

Notable change

Module syntax detection (the --experimental-detect-module flag) is now enabled by default. Use --no-experimental-detect-module to disable it if needed.

Syntax detection attempts to run ambiguous files as CommonJS, and if the module fails to parse as CommonJS due to ES module syntax, Node.js tries again and runs the file as an ES module. Ambiguous files are those with a .js or no extension, where the nearest parent package.json has no "type" field (either "type": "module" or "type": "commonjs"). Syntax detection should have no performance impact on CommonJS modules, but it incurs a slight performance penalty for ES modules; add "type": "module" to the nearest parent package.json file to eliminate the performance cost. A use case unlocked by this feature is the ability to use ES module syntax in extensionless scripts with no nearby package.json.

Description

This is a PR to unflag --experimental-detect-module. It does two things:

• It makes the --experimental-detect-module flag enabled by default, letting people disable it via --no-experimental-detect-module.
• Per module: resolve format for all situations with auto module detection on #53044 (comment) this PR changes the format hint returned by Node’s internal resolve for ambiguous (no package.json type field, no .mjs or .cjs extension) files from commonjs to null. This is more correct, as the format hasn’t yet been determined at the time of resolution, because with detection enabled we need to load the source and parse it to detect the format.

I also updated the tests because of the unflagging:

Not breaking changes

• test/es-module/test-esm-cjs-exports.js should error on invalid CJS exports: Previously this was testing for the error strings Warning: To load an ES module and Unexpected token \'export\'; now it tests for the error string SyntaxError: The requested module './invalid-cjs.js' does not provide an export named 'default'. Arguably the new version is more appropriate as it’s testing the actual thing the test describes rather than testing an error that was thrown before getting to the invalid CommonJS exports.

• test/es-module/test-esm-import-flag.mjs should import when running –check fails: This test checks that an ESM file passed to --import is evaluated with a CommonJS entry passed to --check fails the syntax check. The file passed to --import is run in both cases, though the previous SyntaxError is now a success; but the point of the test is about the --import file being evaluated at all, not whether its evaluation was successful.

• test/es-module/test-require-module-implicit.js: Some tests were attempting to require an ambiguous, extensionless ESM file and expecting an exception with Unexpected token 'export'. These tests were removed. This file is run via --experimental-require-module so this change shouldn’t be considered breaking.

Possibly breaking changes

• test/es-module/test-esm-detect-ambiguous.mjs should not hint wrong format in resolve hook: Previously this expected a format hint of commonjs from a resolve hook, and now it expects null, an intentional change. The hooks are still experimental and the format hint from resolve is documented as optional, so hooks should be written to expect it not to be present; but the unflagging would cause some code that currently returns a hint to no longer do so.

• test/es-module/test-esm-loader-hooks.mjs should use ESM loader to respond to require.resolve calls when opting in: The load hook in this test previously assumed that most files being imported got a format of commonjs, whereas now many of them are undefined, an intentional change. Same impact as previous.

• test/es-module/test-esm-resolve-type.mjs: Some tests for a format hint of commonjs now check for a format hint of null. Same impact as previous.

• test/es-module/test-esm-extensionless-esm-and-wasm.mjs: extensionless ES modules within no package scope: tests for ambiguous extensionless ESM running as the entry point and ambugious extensionless ESM running via import() used to test for exceptions being thrown; now they test for success. Anyone expecting exceptions on running or importing ambiguous or extensionless ESM (for example, to do detection themselves) will be broken by this change.

[nodejs-github-bot]

Review requested:

• @nodejs/loaders

[joyeecheung]

I wonder for hook users relying on the inaccurate context.format whether they should just do const format = context.format || 'commonjs' because that's normally how it goes. Or we could just maintain an "tentative format" internally in addition to the existing format which is more like "confirmed format" and maintain context.format as the "tentative format", while adding context.confirmedFormat or something that carries around format information that are confirmed by the previous resolution & won't incur any further detection in the default load step (and it would be undefined for format that are pending detection and can only be confirmed after the source code is final).

[GeoffreyBooth]

Update: I figured out the require.resolve test. The load hook had an if (result.format === 'commonjs') condition that needed updating to allow for an undefined format. Now all that remains is #53634.

I’ve updated the top post with a list of tests that were updated, grouped by whether I think they should be considered nonbreaking or might be considered breaking.

[GeoffreyBooth]

With #53642 landed, now all the tests pass. This is ready for review.

Assuming we think this is ready to land, the other question we need to answer is how to release it. I updated the top post with my analysis of each changed test and whether the change might be considered breaking or not; basically I think three are definitely not breaking and three are arguably breaking but possibly not. I’m curious to hear what others think. If we think that some of these arguably breaking ones are breaking, then detection can only be unflagged in >23; otherwise I think this should be acceptable to backport.

We can always land it in 23.0.0 first to see the reaction and only backport it afterward, at the cost of delaying the rollout to the <23 lines until November at the earliest.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/60079/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/60154/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/60161/

[joyeecheung]

Backport in #56927