module: support ESM detection in the CJS loader

[joyeecheung]

module: support ESM detection in the CJS loader

This patch:

1. Adds ESM syntax detection to compileFunctionForCJSLoader()
   for --experimental-detect-module and allow it to emit the
   warning for how to load ESM when it's used to parse ESM as
   CJS but detection is not enabled.
2. Moves the ESM detection of --experimental-detect-module for
   the entrypoint from executeUserEntryPoint() into
   Module.prototype._compile() and handle it directly in the
   CJS loader so that the errors thrown during compilation and
   execution during the loading of the entrypoint does not
   need to be bubbled all the way up. If the entrypoint doesn't
   parse as CJS, and detection is enabled, the CJS loader will
   re-load the entrypoint as ESM on the spot asynchronously using
   runEntryPointWithESMLoader() and cascadedLoader.import(). This
   is fine for the entrypoint because unlike require(ESM) we don't
   the namespace of the entrypoint synchronously, and can just
   ignore the returned value. In this case process.mainModule is
   reset to undefined as they are not available for ESM entrypoints.
3. Supports --experimental-detect-module for require(esm)

[nodejs-github-bot]

Review requested:

• @nodejs/loaders
• @nodejs/startup
• @nodejs/vm

[GeoffreyBooth]

My note from #51977 (comment) still applies; I’m not sure why we need a separate PR for this. If we support detection at all, it should be via one flag, not two; having separate flags for detection for import and detection for require is a footgun for users because they should never be enabling one without the other.

[joyeecheung]

Pasting my reply from #51977 (comment) :

I don't want to couple require(esm) with --experimental-detect-module because I think the implementation of the latter has a lot of issues that we talked about before (reading and parsing the file twice etc.) and it seems unfortunate to me that once you enable require(esm) detection you also get yourself into those troubles. But to fix --experimental-detect-module there are a lot to be done and would unnecessarily complicates require(esm), so might as well just keep detection out of the scope of this PR. I don't think it needs to cover all these to land as a first iteration experiment. Without detection this is already can load all the ~20 high impact esm-only npm packages that I've tested just fine. Detection only serves a less painful point that can be already be addressed with explicit module type while there's no workaround in core for require(esm) at all ATM. IMO the latter is the major pain point of ESM adoption while the former is minor and less of a priority, so I just want to focus on the higher priority.

I don't think require(esm) must support every feature there is (experimental ones, even) to land as an experiment itself. That would just procrastinate the entire thing. My criteria for whether the first iteration is feature complete enough is very simple - can require(esm) load say the top 20 high-impact ESM-only npm packages? If yes, then it's good enough as a MVP. We can sort out other stuff while it's experimental. If we want to enforce that experimental features must work with other features to land, then either SEA shouldn't land because it doesn't support import, or --experimental-detect-module shouldn't land because it doesn't support SEA, for example.

[GeoffreyBooth]

I don’t think require(esm) must support every feature there is (experimental ones, even) to land as an experiment itself.

That’s fine; as I wrote over there, I don’t think the initial require(esm) needs to support detection in order to land. But this PR is just adding detection to require(esm), so I think it’s fair to say that this PR needs to support detection in a way that we agree on.

I also think detection can be part of the first PR, as it’s simple and all I requested was that you change the new flag you’re adding to instead use the existing flag. But if you’d prefer to do two PRs that’s fine too.

[joyeecheung]

I think it’s fair to say that this PR needs to support detection in a way that we agree on.

I agree but I think if this would need a lot more than just what this PR already has to fix --experimental-detect-module. That's why I opened a separate PR to avoid derailing #51977 I personally don't think ContainsModuleSyntax should be implemented the way it is and we've talked about it (i.e. we should rely on V8 to cache successful compilations, reparse it as ESM on syntax failure potentially using V8 API instead of error message matching, etc.). But I would not expect to just fix that with a few lines, and it feels really out of scope of require(esm). For now I just want to prioritize require(esm) and set aside other things in the ESM loader that I think look off.

[GeoffreyBooth]

I personally don’t think ContainsModuleSyntax should be implemented the way it is

Sure, but isn’t that a concern for unflagging it? Like those are just performance optimizations; you could add CommonJS detection to it and keep all detection behind the one flag, and then unflag everything once detection is refactoring to your liking.

[joyeecheung]

Sure, but isn’t that a concern for unflagging it? Like those are just performance optimizations; you could add CommonJS detection to it and keep all detection behind the one flag, and then unflag everything once detection is refactoring to your liking.

I think the detection in CJS should be done in where CJS compiles code - that's where it can be done most efficiently. If done that way the refactoring isn't really a separate thing and can cause constant conflicts.

[joyeecheung]

This needs #52413 to detect the ESM syntax properly in C++

[joyeecheung]

I noticed that this is actually unnecessary, when X is already a file, it's covered by 1 (and even before we supported require(esm), 1 already implied that X.mjs can be loaded as a ECMAScript module by require(), kind of funny..

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/58608/

[nodejs-github-bot]

Landed in 4d59a9d

[joyeecheung]

Backport in #56927