esm: add chaining to loaders

PR-URL: https://github.com/nodejs/node/pull/42623
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

2 files changed, 156 insertions, 89 deletions

diff --git a/doc/api/errors.md b/doc/api/errors.md
index 5bc949d5b12..801f22727f3 100644
--- a/doc/api/errors.md
+++ b/doc/api/errors.md
@@ -2092,6 +2092,17 @@ An attempt was made to open an IPC communication channel with a synchronously
 forked Node.js process. See the documentation for the [`child_process`][] module
 for more information.
 
+<a id="ERR_LOADER_CHAIN_INCOMPLETE"></a>
+
+### `ERR_LOADER_CHAIN_INCOMPLETE`
+
+<!-- YAML
+added: REPLACEME
+-->
+
+An ESM loader hook returned without calling `next()` and without explicitly
+signaling a short circuit.
+
 <a id="ERR_MANIFEST_ASSERT_INTEGRITY"></a>
 
 ### `ERR_MANIFEST_ASSERT_INTEGRITY`
diff --git a/doc/api/esm.md b/doc/api/esm.md
index 3d09cc03bba..fd128397f49 100644
--- a/doc/api/esm.md
+++ b/doc/api/esm.md
@@ -8,6 +8,10 @@
 added: v8.5.0
 changes:
   - version:
+    - REPLACEME
+    pr-url: https://github.com/nodejs/node/pull/42623
+    description: Add support for chaining loaders.
+  - version:
     - v17.1.0
     - v16.14.0
     pr-url: https://github.com/nodejs/node/pull/40250
@@ -675,6 +679,10 @@ of Node.js applications.
 <!-- YAML
 added: v8.8.0
 changes:
+  - version:
+    - REPLACEME
+    pr-url: https://github.com/nodejs/node/pull/42623
+    description: Add support for chaining loaders.
   - version: v16.12.0
     pr-url: https://github.com/nodejs/node/pull/37468
     description: Removed `getFormat`, `getSource`, `transformSource`, and
@@ -693,12 +701,40 @@ provided via a `--experimental-loader ./loader-name.mjs` argument to Node.js.
 When hooks are used they apply to the entry point and all `import` calls. They
 won't apply to `require` calls; those still follow [CommonJS][] rules.
 
+Loaders follow the pattern of `--require`:
+
+```console
+node \
+  --experimental-loader unpkg \
+  --experimental-loader http-to-https \
+  --experimental-loader cache-buster
+```
+
+These are called in the following sequence: `cache-buster` calls
+`http-to-https` which calls `unpkg`.
+
 ### Hooks
 
-#### `resolve(specifier, context, defaultResolve)`
+Hooks are part of a chain, even if that chain consists of only one custom
+(user-provided) hook and the default hook, which is always present. Hook
+functions nest: each one must always return a plain object, and chaining happens
+as a result of each function calling `next<hookName>()`, which is a reference
+to the subsequent loader’s hook.
+
+A hook that returns a value lacking a required property triggers an exception.
+A hook that returns without calling `next<hookName>()` _and_ without returning
+`shortCircuit: true` also triggers an exception. These errors are to help
+prevent unintentional breaks in the chain.
+
+#### `resolve(specifier, context, nextResolve)`
 
 <!-- YAML
 changes:
+  - version: REPLACEME
+    pr-url: https://github.com/nodejs/node/pull/42623
+    description: Add support for chaining resolve hooks. Each hook must either
+      call `nextResolve()` or include a `shortCircuit` property set to `true`
+      in its return.
   - version:
     - v17.1.0
     - v16.14.0
@@ -711,26 +747,35 @@ changes:
 
 * `specifier` {string}
 * `context` {Object}
-  * `conditions` {string\[]}
+  * `conditions` {string\[]} Export conditions of the relevant `package.json`
   * `importAssertions` {Object}
-  * `parentURL` {string|undefined}
-* `defaultResolve` {Function} The Node.js default resolver.
+  * `parentURL` {string|undefined} The module importing this one, or undefined
+    if this is the Node.js entry point
+* `nextResolve` {Function} The subsequent `resolve` hook in the chain, or the
+  Node.js default `resolve` hook after the last user-supplied `resolve` hook
+  * `specifier` {string}
+  * `context` {Object}
 * Returns: {Object}
-  * `format` {string|null|undefined}
+  * `format` {string|null|undefined} A hint to the load hook (it might be
+    ignored)
     `'builtin' | 'commonjs' | 'json' | 'module' | 'wasm'`
-  * `url` {string} The absolute url to the import target (such as `file://…`)
-
-The `resolve` hook returns the resolved file URL for a given module specifier
-and parent URL, and optionally its format (such as `'module'`) as a hint to the
-`load` hook. If a format is specified, the `load` hook is ultimately responsible
-for providing the final `format` value (and it is free to ignore the hint
-provided by `resolve`); if `resolve` provides a `format`, a custom `load`
-hook is required even if only to pass the value to the Node.js default `load`
-hook.
+  * `shortCircuit` {undefined|boolean} A signal that this hook intends to
+    terminate the chain of `resolve` hooks. **Default:** `false`
+  * `url` {string} The absolute URL to which this input resolves
+
+The `resolve` hook chain is responsible for resolving file URL for a given
+module specifier and parent URL, and optionally its format (such as `'module'`)
+as a hint to the `load` hook. If a format is specified, the `load` hook is
+ultimately responsible for providing the final `format` value (and it is free to
+ignore the hint provided by `resolve`); if `resolve` provides a `format`, a
+custom `load` hook is required even if only to pass the value to the Node.js
+default `load` hook.
 
 The module specifier is the string in an `import` statement or
-`import()` expression, and the parent URL is the URL of the module that imported
-this one, or `undefined` if this is the main entry point for the application.
+`import()` expression.
+
+The parent URL is the URL of the module that imported this one, or `undefined`
+if this is the main entry point for the application.
 
 The `conditions` property in `context` is an array of conditions for
 [package exports conditions][Conditional Exports] that apply to this resolution
@@ -744,40 +789,45 @@ Node.js module specifier resolution behavior_ when calling `defaultResolve`, the
 `context.conditions` array originally passed into the `resolve` hook.
 
 ```js
-/**
- * @param {string} specifier
- * @param {{
- *   conditions: string[],
- *   parentURL: string | undefined,
- * }} context
- * @param {Function} defaultResolve
- * @returns {Promise<{ url: string }>}
- */
-export async function resolve(specifier, context, defaultResolve) {
+export async function resolve(specifier, context, nextResolve) {
   const { parentURL = null } = context;
+
   if (Math.random() > 0.5) { // Some condition.
     // For some or all specifiers, do some custom logic for resolving.
     // Always return an object of the form {url: <string>}.
     return {
+      shortCircuit: true,
       url: parentURL ?
         new URL(specifier, parentURL).href :
         new URL(specifier).href,
     };
   }
+
   if (Math.random() < 0.5) { // Another condition.
     // When calling `defaultResolve`, the arguments can be modified. In this
     // case it's adding another value for matching conditional exports.
-    return defaultResolve(specifier, {
+    return nextResolve(specifier, {
       ...context,
       conditions: [...context.conditions, 'another-condition'],
     });
   }
-  // Defer to Node.js for all other specifiers.
-  return defaultResolve(specifier, context, defaultResolve);
+
+  // Defer to the next hook in the chain, which would be the
+  // Node.js default resolve if this is the last user-specified loader.
+  return nextResolve(specifier, context);
 }
 ```
 
-#### `load(url, context, defaultLoad)`
+#### `load(url, context, nextLoad)`
+
+<!-- YAML
+changes:
+  - version: REPLACEME
+    pr-url: https://github.com/nodejs/node/pull/42623
+    description: Add support for chaining load hooks. Each hook must either
+      call `nextLoad()` or include a `shortCircuit` property set to `true` in
+      its return.
+-->
 
 > The loaders API is being redesigned. This hook may disappear or its
 > signature may change. Do not rely on the API described below.
@@ -785,15 +835,21 @@ export async function resolve(specifier, context, defaultResolve) {
 > In a previous version of this API, this was split across 3 separate, now
 > deprecated, hooks (`getFormat`, `getSource`, and `transformSource`).
 
-* `url` {string}
+* `url` {string} The URL returned by the `resolve` chain
 * `context` {Object}
+  * `conditions` {string\[]} Export conditions of the relevant `package.json`
   * `format` {string|null|undefined} The format optionally supplied by the
-    `resolve` hook.
+    `resolve` hook chain
   * `importAssertions` {Object}
-* `defaultLoad` {Function}
+* `nextLoad` {Function} The subsequent `load` hook in the chain, or the
+  Node.js default `load` hook after the last user-supplied `load` hook
+  * `specifier` {string}
+  * `context` {Object}
 * Returns: {Object}
   * `format` {string}
-  * `source` {string|ArrayBuffer|TypedArray}
+  * `shortCircuit` {undefined|boolean} A signal that this hook intends to
+    terminate the chain of `resolve` hooks. **Default:** `false`
+  * `source` {string|ArrayBuffer|TypedArray} The source for Node.js to evaluate
 
 The `load` hook provides a way to define a custom method of determining how
 a URL should be interpreted, retrieved, and parsed. It is also in charge of
@@ -834,20 +890,10 @@ avoid reading files from disk. It could also be used to map an unrecognized
 format to a supported one, for example `yaml` to `module`.
 
 ```js
-/**
- * @param {string} url
- * @param {{
-    format: string,
-  }} context If resolve settled with a `format`, that value is included here.
- * @param {Function} defaultLoad
- * @returns {Promise<{
-    format: string,
-    source: string | ArrayBuffer | SharedArrayBuffer | Uint8Array,
-  }>}
- */
-export async function load(url, context, defaultLoad) {
+export async function load(url, context, nextLoad) {
   const { format } = context;
-  if (Math.random() > 0.5) { // Some condition.
+
+  if (Math.random() > 0.5) { // Some condition
     /*
       For some or all URLs, do some custom logic for retrieving the source.
       Always return an object of the form {
@@ -857,11 +903,13 @@ export async function load(url, context, defaultLoad) {
     */
     return {
       format,
+      shortCircuit: true,
       source: '...',
     };
   }
-  // Defer to Node.js for all other URLs.
-  return defaultLoad(url, context, defaultLoad);
+
+  // Defer to the next hook in the chain.
+  return nextLoad(url, context);
 }
 ```
 
@@ -870,13 +918,22 @@ source to a supported one (see [Examples](#examples) below).
 
 #### `globalPreload()`
 
+<!-- YAML
+changes:
+  - version: REPLACEME
+    pr-url: https://github.com/nodejs/node/pull/42623
+    description: Add support for chaining globalPreload hooks.
+-->
+
 > The loaders API is being redesigned. This hook may disappear or its
 > signature may change. Do not rely on the API described below.
 
 > In a previous version of this API, this hook was named
 > `getGlobalPreloadCode`.
 
-* Returns: {string}
+* `context` {Object} Information to assist the preload code
+  * `port` {MessagePort}
+* Returns: {string} Code to run before application startup
 
 Sometimes it might be necessary to run some code inside of the same global
 scope that the application runs in. This hook allows the return of a string
@@ -890,13 +947,7 @@ If the code needs more advanced `require` features, it has to construct
 its own `require` using  `module.createRequire()`.
 
 ```js
-/**
- * @param {{
-     port: MessagePort,
-   }} utilities Things that preload code might find useful
- * @returns {string} Code to run before application startup
- */
-export function globalPreload(utilities) {
+export function globalPreload(context) {
   return `\
 globalThis.someInjectedProperty = 42;
 console.log('I just set some globals!');
@@ -921,10 +972,6 @@ close normally.
 /**
  * This example has the application context send a message to the loader
  * and sends the message back to the application context
- * @param {{
-     port: MessagePort,
-   }} utilities Things that preload code might find useful
- * @returns {string} Code to run before application startup
  */
 export function globalPreload({ port }) {
   port.onmessage = (evt) => {
@@ -946,9 +993,11 @@ customizations of Node.js’ code loading and evaluation behaviors.
 
 #### HTTPS loader
 
-In current Node.js, specifiers starting with `https://` are unsupported. The
-loader below registers hooks to enable rudimentary support for such specifiers.
-While this may seem like a significant improvement to Node.js core
+In current Node.js, specifiers starting with `https://` are experimental (see
+[HTTPS and HTTP imports][]).
+
+The loader below registers hooks to enable rudimentary support for such
+specifiers. While this may seem like a significant improvement to Node.js core
 functionality, there are substantial downsides to actually using this loader:
 performance is much slower than loading files from disk, there is no caching,
 and there is no security.
@@ -957,7 +1006,7 @@ and there is no security.
 // https-loader.mjs
 import { get } from 'node:https';
 
-export function resolve(specifier, context, defaultResolve) {
+export function resolve(specifier, context, nextResolve) {
   const { parentURL = null } = context;
 
   // Normally Node.js would error on specifiers starting with 'https://', so
@@ -965,19 +1014,21 @@ export function resolve(specifier, context, defaultResolve) {
   // passed along to the later hooks below.
   if (specifier.startsWith('https://')) {
     return {
+      shortCircuit: true,
       url: specifier
     };
   } else if (parentURL && parentURL.startsWith('https://')) {
     return {
-      url: new URL(specifier, parentURL).href
+      shortCircuit: true,
+      url: new URL(specifier, parentURL).href,
     };
   }
 
   // Let Node.js handle all other specifiers.
-  return defaultResolve(specifier, context, defaultResolve);
+  return nextResolve(specifier, context);
 }
 
-export function load(url, context, defaultLoad) {
+export function load(url, context, nextLoad) {
   // For JavaScript to be loaded over the network, we need to fetch and
   // return it.
   if (url.startsWith('https://')) {
@@ -989,6 +1040,7 @@ export function load(url, context, defaultLoad) {
           // This example assumes all network-provided JavaScript is ES module
           // code.
           format: 'module',
+          shortCircuit: true,
           source: data,
         }));
       }).on('error', (err) => reject(err));
@@ -996,7 +1048,7 @@ export function load(url, context, defaultLoad) {
   }
 
   // Let Node.js handle all other URLs.
-  return defaultLoad(url, context, defaultLoad);
+  return nextLoad(url, context);
 }
 ```
 
@@ -1036,27 +1088,29 @@ const baseURL = pathToFileURL(`${cwd()}/`).href;
 // CoffeeScript files end in .coffee, .litcoffee or .coffee.md.
 const extensionsRegex = /\.coffee$|\.litcoffee$|\.coffee\.md$/;
 
-export async function resolve(specifier, context, defaultResolve) {
-  const { parentURL = baseURL } = context;
-
-  // Node.js normally errors on unknown file extensions, so return a URL for
-  // specifiers ending in the CoffeeScript file extensions.
+export async function resolve(specifier, context, nextResolve) {
   if (extensionsRegex.test(specifier)) {
+    const { parentURL = baseURL } = context;
+
+    // Node.js normally errors on unknown file extensions, so return a URL for
+    // specifiers ending in the CoffeeScript file extensions.
     return {
+      shortCircuit: true,
       url: new URL(specifier, parentURL).href
     };
   }
 
   // Let Node.js handle all other specifiers.
-  return defaultResolve(specifier, context, defaultResolve);
+  return nextResolve(specifier, context);
 }
 
-export async function load(url, context, defaultLoad) {
-  // Now that we patched resolve to let CoffeeScript URLs through, we need to
-  // tell Node.js what format such URLs should be interpreted as. Because
-  // CoffeeScript transpiles into JavaScript, it should be one of the two
-  // JavaScript formats: 'commonjs' or 'module'.
+export async function load(url, context, nextLoad) {
   if (extensionsRegex.test(url)) {
+    // Now that we patched resolve to let CoffeeScript URLs through, we need to
+    // tell Node.js what format such URLs should be interpreted as. Because
+    // CoffeeScript transpiles into JavaScript, it should be one of the two
+    // JavaScript formats: 'commonjs' or 'module'.
+
     // CoffeeScript files can be either CommonJS or ES modules, so we want any
     // CoffeeScript file to be treated by Node.js the same as a .js file at the
     // same location. To determine how Node.js would interpret an arbitrary .js
@@ -1069,25 +1123,26 @@ export async function load(url, context, defaultLoad) {
     // loader. Avoiding the need for a separate CommonJS handler is a future
     // enhancement planned for ES module loaders.
     if (format === 'commonjs') {
-      return { format };
+      return {
+        format,
+        shortCircuit: true,
+      };
     }
 
-    const { source: rawSource } = await defaultLoad(url, { format });
+    const { source: rawSource } = await nextLoad(url, { ...context, format });
     // This hook converts CoffeeScript source code into JavaScript source code
     // for all imported CoffeeScript files.
-    const transformedSource = CoffeeScript.compile(rawSource.toString(), {
-      bare: true,
-      filename: url,
-    });
+    const transformedSource = coffeeCompile(rawSource.toString(), url);
 
     return {
       format,
+      shortCircuit: true,
       source: transformedSource,
     };
   }
 
   // Let Node.js handle all other URLs.
-  return defaultLoad(url, context, defaultLoad);
+  return nextLoad(url, context);
 }
 
 async function getPackageType(url) {
@@ -1495,6 +1550,7 @@ success!
 [Determining module system]: packages.md#determining-module-system
 [Dynamic `import()`]: https://wiki.developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import#Dynamic_Imports
 [ES Module Integration Proposal for WebAssembly]: https://github.com/webassembly/esm-integration
+[HTTPS and HTTP imports]: #https-and-http-imports
 [Import Assertions]: #import-assertions
 [Import Assertions proposal]: https://github.com/tc39/proposal-import-assertions
 [JSON modules]: #json-modules
@@ -1525,9 +1581,9 @@ success!
 [`util.TextDecoder`]: util.md#class-utiltextdecoder
 [cjs-module-lexer]: https://github.com/nodejs/cjs-module-lexer/tree/1.2.2
 [custom https loader]: #https-loader
-[load hook]: #loadurl-context-defaultload
+[load hook]: #loadurl-context-nextload
 [percent-encoded]: url.md#percent-encoding-in-urls
-[resolve hook]: #resolvespecifier-context-defaultresolve
+[resolve hook]: #resolvespecifier-context-nextresolve
 [special scheme]: https://url.spec.whatwg.org/#special-scheme
 [status code]: process.md#exit-codes
 [the official standard format]: https://tc39.github.io/ecma262/#sec-modules