test_runner: support 'only' tests

This commit introduces a CLI flag and test runner functionality
to support running a subset of tests that are indicated by an
'only' option passed to the test.

PR-URL: https://github.com/nodejs/node/pull/42514
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

3 files changed, 70 insertions, 0 deletions

diff --git a/doc/api/cli.md b/doc/api/cli.md
index cec177f0410..92acc87e648 100644
--- a/doc/api/cli.md
+++ b/doc/api/cli.md
@@ -1052,6 +1052,15 @@ minimum allocation from the secure heap. The minimum value is `2`.
 The maximum value is the lesser of `--secure-heap` or `2147483647`.
 The value given must be a power of two.
 
+### `--test-only`
+
+<!-- YAML
+added: REPLACEME
+-->
+
+Configures the test runner to only execute top level tests that have the `only`
+option set.
+
 ### `--throw-deprecation`
 
 <!-- YAML
@@ -1641,6 +1650,7 @@ Node.js options that are allowed are:
 * `--require`, `-r`
 * `--secure-heap-min`
 * `--secure-heap`
+* `--test-only`
 * `--throw-deprecation`
 * `--title`
 * `--tls-cipher-list`
diff --git a/doc/api/test.md b/doc/api/test.md
index 86cd1447633..9ce8b731a46 100644
--- a/doc/api/test.md
+++ b/doc/api/test.md
@@ -148,6 +148,42 @@ test('skip() method with message', (t) => {
 });
 ```
 
+### `only` tests
+
+If Node.js is started with the [`--test-only`][] command-line option, it is
+possible to skip all top level tests except for a selected subset by passing
+the `only` option to the tests that should be run. When a test with the `only`
+option set is run, all subtests are also run. The test context's `runOnly()`
+method can be used to implement the same behavior at the subtest level.
+
+```js
+// Assume Node.js is run with the --test-only command-line option.
+// The 'only' option is set, so this test is run.
+test('this test is run', { only: true }, async (t) => {
+  // Within this test, all subtests are run by default.
+  await t.test('running subtest');
+
+  // The test context can be updated to run subtests with the 'only' option.
+  t.runOnly(true);
+  await t.test('this subtest is now skipped');
+  await t.test('this subtest is run', { only: true });
+
+  // Switch the context back to execute all tests.
+  t.runOnly(false);
+  await t.test('this subtest is now run');
+
+  // Explicitly do not run these tests.
+  await t.test('skipped subtest 3', { only: false });
+  await t.test('skipped subtest 4', { skip: true });
+});
+
+// The 'only' option is not set, so this test is skipped.
+test('this test is not run', () => {
+  // This code is not run.
+  throw new Error('fail');
+});
+```
+
 ## Extraneous asynchronous activity
 
 Once a test function finishes executing, the TAP results are output as quickly
@@ -197,6 +233,9 @@ added: REPLACEME
   * `concurrency` {number} The number of tests that can be run at the same time.
     If unspecified, subtests inherit this value from their parent.
     **Default:** `1`.
+  * `only` {boolean} If truthy, and the test context is configured to run
+    `only` tests, then this test will be run. Otherwise, the test is skipped.
+    **Default:** `false`.
   * `skip` {boolean|string} If truthy, the test is skipped. If a string is
     provided, that string is displayed in the test results as the reason for
     skipping the test. **Default:** `false`.
@@ -257,6 +296,19 @@ This function is used to write TAP diagnostics to the output. Any diagnostic
 information is included at the end of the test's results. This function does
 not return a value.
 
+### `context.runOnly(shouldRunOnlyTests)`
+
+<!-- YAML
+added: REPLACEME
+-->
+
+* `shouldRunOnlyTests` {boolean} Whether or not to run `only` tests.
+
+If `shouldRunOnlyTests` is truthy, the test context will only run tests that
+have the `only` option set. Otherwise, all tests are run. If Node.js was not
+started with the [`--test-only`][] command-line option, this function is a
+no-op.
+
 ### `context.skip([message])`
 
 <!-- YAML
@@ -296,6 +348,9 @@ added: REPLACEME
   * `concurrency` {number} The number of tests that can be run at the same time.
     If unspecified, subtests inherit this value from their parent.
     **Default:** `1`.
+  * `only` {boolean} If truthy, and the test context is configured to run
+    `only` tests, then this test will be run. Otherwise, the test is skipped.
+    **Default:** `false`.
   * `skip` {boolean|string} If truthy, the test is skipped. If a string is
     provided, that string is displayed in the test results as the reason for
     skipping the test. **Default:** `false`.
@@ -312,5 +367,6 @@ This function is used to create subtests under the current test. This function
 behaves in the same fashion as the top level [`test()`][] function.
 
 [TAP]: https://testanything.org/
+[`--test-only`]: cli.md#--test-only
 [`TestContext`]: #class-testcontext
 [`test()`]: #testname-options-fn
diff --git a/doc/node.1 b/doc/node.1
index 27b3365140c..bd3550625a7 100644
--- a/doc/node.1
+++ b/doc/node.1
@@ -381,6 +381,10 @@ the secure heap. The default is 0. The value must be a power of two.
 .It Fl -secure-heap-min Ns = Ns Ar n
 Specify the minimum allocation from the OpenSSL secure heap. The default is 2. The value must be a power of two.
 .
+.It Fl -test-only
+Configures the test runner to only execute top level tests that have the `only`
+option set.
+.
 .It Fl -throw-deprecation
 Throw errors for deprecations.
 .