docs: npm-access through npm-dedupe

The big one here is npm-audit, since that changed so significantly in
npm v7.  The others are mostly small copyedits and improvements to the
consistency of ordering sections.

11 files changed, 217 insertions, 131 deletions

diff --git a/docs/content/cli-commands/npm-access.md b/docs/content/cli-commands/npm-access.md
index f4088fe88..ad221c16b 100644
--- a/docs/content/cli-commands/npm-access.md
+++ b/docs/content/cli-commands/npm-access.md
@@ -75,11 +75,12 @@ You must have privileges to set the access of a package:
 * You have been given read-write privileges for a package, either as a member
   of a team or directly as an owner.
 
-If you have two-factor authentication enabled then you'll have to pass in an
-otp with `--otp` when making access changes.
+If you have two-factor authentication enabled then you'll be prompted to
+provide an otp token, or may use the `--otp=...` option to specify it on
+the command line.
 
-If your account is not paid, then attempts to publish scoped packages will fail
-with an HTTP 402 status code (logically enough), unless you use
+If your account is not paid, then attempts to publish scoped packages will
+fail with an HTTP 402 status code (logically enough), unless you use
 `--access=public`.
 
 Management of teams and team memberships is done with the `npm team` command.
diff --git a/docs/content/cli-commands/npm-adduser.md b/docs/content/cli-commands/npm-adduser.md
index ec61cdd39..abacc2905 100644
--- a/docs/content/cli-commands/npm-adduser.md
+++ b/docs/content/cli-commands/npm-adduser.md
@@ -54,8 +54,8 @@ with the specified scope. See [`scope`](/using-npm/scope). You can use both at t
 e.g.
 
 ```bash
-    npm adduser --registry=http://myregistry.example.com --scope=@myco
-```    
+npm adduser --registry=http://myregistry.example.com --scope=@myco
+```
 
 This will set a registry for the given scope and login or create a user for
 that registry at the same time.
@@ -69,7 +69,7 @@ registry should include authorization information. Useful for private
 registries. Can be used with `--registry` and / or `--scope`, e.g.
 
 ```bash
-    npm adduser --registry=http://private-registry.example.com --always-auth
+npm adduser --registry=http://private-registry.example.com --always-auth
 ```
 
 This will ensure that all requests to that registry (including for tarballs)
diff --git a/docs/content/cli-commands/npm-audit.md b/docs/content/cli-commands/npm-audit.md
index 8b944e94c..391537048 100644
--- a/docs/content/cli-commands/npm-audit.md
+++ b/docs/content/cli-commands/npm-audit.md
@@ -17,118 +17,180 @@ npm audit fix [--force|--package-lock-only|--dry-run]
 common options: [--production] [--only=(dev|prod)]
 ```
 
+### Description
+
+The audit command submits a description of the dependencies configured in
+your project to your default registry and asks for a report of known
+vulnerabilities.  If any vulnerabilities are found, then the impact and
+appropriate remediation will be calculated.  If the `fix` argument is
+provided, then remediations will be applied to the package tree.
+
+The command will exit with a 0 exit code if no vulnerabilities were found.
+
+Note that some vulnerabilities cannot be fixed automatically and will
+require manual intervention or review.  Also note that since `npm audit
+fix` runs a full-fledged `npm install` under the hood, all configs that
+apply to the installer will also apply to `npm install` -- so things like
+`npm audit fix --package-lock-only` will work as expected.
+
+By default, the audit command will exit with a non-zero code if any
+vulnerability is found. It may be useful in CI environments to include the
+`--audit-level` parameter to specify the minimum vulnerability level that
+will cause the command to fail. This option does not filter the report
+output, it simply changes the command's failure threshold.
+
+### Audit Endpoints
+
+There are two audit endpoints that npm may use to fetch vulnerability
+information: the `Bulk Advisory` endpoint and the `Quick Audit` endpoint.
+
+#### Bulk Advisory Endpoint
+
+As of version 7, npm uses the much faster `Bulk Advisory` endpoint to
+optimize the speed of calculating audit results.
+
+npm will generate a JSON payload with the name and list of versions of each
+package in the tree, and POST it to the default configured registry at
+the path `/-/npm/v1/security/advisories/bulk`.
+
+Any packages in the tree that do not have a `version` field in their
+package.json file will be ignored.  If any `--omit` options are specified
+(either via the `--omit` config, or one of the shorthands such as
+`--production`, `--only=dev`, and so on), then packages will be omitted
+from the submitted payload as appropriate.
+
+If the registry responds with an error, or with an invalid response, then
+npm will attempt to load advisory data from the `Quick Audit` endpoint.
+
+The expected result will contain a set of advisory objects for each
+dependency that matches the advisory range.  Each advisory object contains
+a `name`, `url`, `id`, `severity`, `vulnerable_versions`, and `title`.
+
+npm then uses these advisory objects to calculate vulnerabilities and
+meta-vulnerabilities of the dependencies within the tree.
+
+#### Quick Audit Endpoint
+
+If the `Bulk Advisory` endpoint returns an error, or invalid data, npm will
+attempt to load advisory data from the `Quick Audit` endpoint, which is
+considerably slower in most cases.
+
+The full package tree as found in `package-lock.json` is submitted, along
+with the following pieces of additional metadata:
+
+* `npm_version`
+* `node_version`
+* `platform`
+* `arch`
+* `node_env`
+
+All packages in the tree are submitted to the Quick Audit endpoint.
+Omitted dependency types are skipped when generating the report.
+
+#### Scrubbing
+
+Out of an abundance of caution, npm versions 5 and 6 would "scrub" any
+packages from the submitted report if their name contained a `/` character,
+so as to avoid leaking the names of potentially private packages or git
+URLs.
+
+However, in practice, this resulted in audits often failing to properly
+detect meta-vulnerabilities, because the tree would appear to be invalid
+due to missing dependencies, and prevented the detection of vulnerabilities
+in package trees that used git dependencies or private modules.
+
+This scrubbing has been removed from npm as of version 7.
+
+#### Calculating Meta-Vulnerabilities and Remediations
+
+npm uses the
+[`@npmcli/metavuln-calculator`](http://npm.im/@npmcli/metavuln-calculator)
+module to turn a set of security advisories into a set of "vulnerability"
+objects.  A "meta-vulnerability" is a dependency that is vulnerable by
+virtue of dependence on vulnerable versions of a vulnerable package.
+
+For example, if the package `foo` is vulnerable in the range `>=1.0.2
+<2.0.0`, and the package `bar` depends on `foo@^1.1.0`, then that version
+of `bar` can only be installed by installing a vulnerable version of `foo`.
+In this case, `bar` is a "metavulnerability".
+
+Once metavulnerabilities for a given package are calculated, they are
+cached in the `~/.npm` folder and only re-evaluated if the advisory range
+changes, or a new version of the package is published (in which case, the
+new version is checked for metavulnerable status as well).
+
+If the chain of metavulnerabilities extends all the way to the root
+project, and it cannot be updated without changing its dependency ranges,
+then `npm audit fix` will require the `--force` option to apply the
+remediation.  If remediations do not require changes to the dependency
+ranges, then all vulnerable packages will be updated to a version that does
+not have an advisory or metavulnerability posted against it.
+
+### Exit Code
+
+The `npm audit` command will exit with a 0 exit code if no vulnerabilities
+were found.  The `npm audit fix` command will exit with 0 exit code if no
+vulnerabilities are found _or_ if the remediation is able to successfully
+fix all vulnerabilities.
+
+If vulnerabilities were found the exit code will depend on the
+`audit-level` configuration setting.
+
 ### Examples
 
 Scan your project for vulnerabilities and automatically install any compatible
 updates to vulnerable dependencies:
+
 ```bash
 $ npm audit fix
 ```
 
 Run `audit fix` without modifying `node_modules`, but still updating the
 pkglock:
+
 ```bash
 $ npm audit fix --package-lock-only
 ```
 
 Skip updating `devDependencies`:
+
 ```bash
 $ npm audit fix --only=prod
 ```
 
-Have `audit fix` install semver-major updates to toplevel dependencies, not just
-semver-compatible ones:
+Have `audit fix` install SemVer-major updates to toplevel dependencies, not
+just SemVer-compatible ones:
+
 ```bash
 $ npm audit fix --force
 ```
 
 Do a dry run to get an idea of what `audit fix` will do, and _also_ output
 install information in JSON format:
+
 ```bash
 $ npm audit fix --dry-run --json
 ```
 
-Scan your project for vulnerabilities and just show the details, without fixing
-anything:
+Scan your project for vulnerabilities and just show the details, without
+fixing anything:
+
 ```bash
 $ npm audit
 ```
 
 Get the detailed audit report in JSON format:
-```bash
-$ npm audit --json
-```
-
-Get the detailed audit report in plain text result, separated by tab characters, allowing for
-future reuse in scripting or command line post processing, like for example, selecting
-some of the columns printed:
-```bash
-$ npm audit --parseable
-```
 
-To parse columns, you can use for example `awk`, and just print some of them:
 ```bash
-$ npm audit --parseable | awk -F $'\t' '{print $1,$4}'
+$ npm audit --json
 ```
 
 Fail an audit only if the results include a vulnerability with a level of moderate or higher:
+
 ```bash
 $ npm audit --audit-level=moderate
 ```
 
-### Description
-
-The audit command submits a description of the dependencies configured in
-your project to your default registry and asks for a report of known
-vulnerabilities. The report returned includes instructions on how to act on
-this information. The command will exit with a 0 exit code if no
-vulnerabilities were found.
-
-You can also have npm automatically fix the vulnerabilities by running `npm
-audit fix`. Note that some vulnerabilities cannot be fixed automatically and
-will require manual intervention or review. Also note that since `npm audit fix`
-runs a full-fledged `npm install` under the hood, all configs that apply to the
-installer will also apply to `npm install` -- so things like `npm audit fix
---package-lock-only` will work as expected.
-
-By default, the audit command will exit with a non-zero code if any vulnerability
-is found. It may be useful in CI environments to include the `--audit-level` parameter
-to specify the minimum vulnerability level that will cause the command to fail. This
-option does not filter the report output, it simply changes the command's failure
-threshold.
-
-### Content Submitted
-
-* npm_version
-* node_version
-* platform
-* node_env
-* A scrubbed version of your package-lock.json or npm-shrinkwrap.json
-
-#### Scrubbing
-
-In order to ensure that potentially sensitive information is not included in
-the audit data bundle, some dependencies may have their names (and sometimes
-versions) replaced with opaque non-reversible identifiers.  It is done for
-the following dependency types:
-
-* Any module referencing a scope that is configured for a non-default
-  registry has its name scrubbed.  (That is, a scope you did a `npm login --scope=@ourscope` for.)
-* All git dependencies have their names and specifiers scrubbed.
-* All remote tarball dependencies have their names and specifiers scrubbed.
-* All local directory and tarball dependencies have their names and specifiers scrubbed.
-
-The non-reversible identifiers are a sha256 of a session-specific UUID and the
-value being replaced, ensuring a consistent value within the payload that is
-different between runs.
-
-### Exit Code
-
-The `npm audit` command will exit with a 0 exit code if no vulnerabilities were found.
-
-If vulnerabilities were found the exit code will depend on the `audit-level`
-configuration setting.
-
 ### See Also
 
 * [npm install](/cli-commands/install)
diff --git a/docs/content/cli-commands/npm-bin.md b/docs/content/cli-commands/npm-bin.md
index eb0912ae4..33b0aa822 100644
--- a/docs/content/cli-commands/npm-bin.md
+++ b/docs/content/cli-commands/npm-bin.md
@@ -9,6 +9,7 @@ description: Display npm bin folder
 ## Display npm bin folder
 
 ### Synopsis
+
 ```bash
 npm bin [-g|--global]
 ```
diff --git a/docs/content/cli-commands/npm-bugs.md b/docs/content/cli-commands/npm-bugs.md
index 1e7272cec..d21e24a27 100644
--- a/docs/content/cli-commands/npm-bugs.md
+++ b/docs/content/cli-commands/npm-bugs.md
@@ -6,9 +6,10 @@ description: Bugs for a package in a web browser maybe
 
 # npm-bugs(1)
 
-## Bugs for a package in a web browser maybe
+## Report bugs for a package in a web browser
 
 ### Synopsis
+
 ```bash
 npm bugs [<pkgname> [<pkgname> ...]]
 
@@ -17,10 +18,10 @@ aliases: issues
 
 ### Description
 
-This command tries to guess at the likely location of a package's
-bug tracker URL, and then tries to open it using the `--browser`
-config param. If no package name is provided, it will search for
-a `package.json` in the current folder and use the `name` property.
+This command tries to guess at the likely location of a package's bug
+tracker URL, and then tries to open it using the `--browser` config param.
+If no package name is provided, it will search for a `package.json` in the
+current folder and use the `name` property.
 
 ### Configuration
 
@@ -43,7 +44,6 @@ Set to `true` to use default system URL opener.
 
 The base URL of the npm package registry.
 
-
 ### See Also
 
 * [npm docs](/cli-commands/docs)
diff --git a/docs/content/cli-commands/npm-bundle.md b/docs/content/cli-commands/npm-bundle.md
index c4fdc5e6b..5b8c015dd 100644
--- a/docs/content/cli-commands/npm-bundle.md
+++ b/docs/content/cli-commands/npm-bundle.md
@@ -11,8 +11,8 @@ description: REMOVED
 ### Description
 
 The `npm bundle` command has been removed in 1.0, for the simple reason
-that it is no longer necessary, as the default behavior is now to
-install packages into the local space.
+that it is no longer necessary, as the default behavior is now to install
+packages into the local space.
 
 Just use `npm install` now to do what `npm bundle` used to do.
 
diff --git a/docs/content/cli-commands/npm-cache.md b/docs/content/cli-commands/npm-cache.md
index 4d19749b8..e3c9d4155 100644
--- a/docs/content/cli-commands/npm-cache.md
+++ b/docs/content/cli-commands/npm-cache.md
@@ -16,7 +16,7 @@ npm cache add <folder>
 npm cache add <tarball url>
 npm cache add <name>@<version>
 
-npm cache clean [<path>]
+npm cache clean
 aliases: npm cache clear, npm cache rm
 
 npm cache verify
@@ -32,41 +32,45 @@ Used to add, list, or clean the npm cache folder.
   add data to the local installation cache explicitly.
 
 * clean:
-  Delete all data out of the cache folder.
+  Delete all data out of the cache folder.  Note that this is typically
+  unnecessary, as npm's cache is self-healing and resistant to data
+  corruption issues.
 
 * verify:
-  Verify the contents of the cache folder, garbage collecting any unneeded data,
-  and verifying the integrity of the cache index and all cached data.
+  Verify the contents of the cache folder, garbage collecting any unneeded
+  data, and verifying the integrity of the cache index and all cached data.
 
 ### Details
 
 npm stores cache data in an opaque directory within the configured `cache`,
-named `_cacache`. This directory is a `cacache`-based content-addressable cache
-that stores all http request data as well as other package-related data. This
-directory is primarily accessed through `pacote`, the library responsible for
-all package fetching as of npm@5.
-
-All data that passes through the cache is fully verified for integrity on both
-insertion and extraction. Cache corruption will either trigger an error, or
-signal to `pacote` that the data must be refetched, which it will do
-automatically. For this reason, it should never be necessary to clear the cache
-for any reason other than reclaiming disk space, thus why `clean` now requires
-`--force` to run.
-
-There is currently no method exposed through npm to inspect or directly manage
-the contents of this cache. In order to access it, `cacache` must be used
-directly.
+named `_cacache`. This directory is a
+[`cacache`](http://npm.im/cacache)-based content-addressable cache that
+stores all http request data as well as other package-related data. This
+directory is primarily accessed through `pacote`, the library responsible
+for all package fetching as of npm@5.
+
+All data that passes through the cache is fully verified for integrity on
+both insertion and extraction. Cache corruption will either trigger an
+error, or signal to `pacote` that the data must be refetched, which it will
+do automatically. For this reason, it should never be necessary to clear
+the cache for any reason other than reclaiming disk space, thus why `clean`
+now requires `--force` to run.
+
+There is currently no method exposed through npm to inspect or directly
+manage the contents of this cache. In order to access it, `cacache` must be
+used directly.
 
 npm will not remove data by itself: the cache will grow as new packages are
 installed.
 
 ### A note about the cache's design
 
-The npm cache is strictly a cache: it should not be relied upon as a persistent
-and reliable data store for package data. npm makes no guarantee that a
-previously-cached piece of data will be available later, and will automatically
-delete corrupted contents. The primary guarantee that the cache makes is that,
-if it does return data, that data will be exactly the data that was inserted.
+The npm cache is strictly a cache: it should not be relied upon as a
+persistent and reliable data store for package data. npm makes no guarantee
+that a previously-cached piece of data will be available later, and will
+automatically delete corrupted contents. The primary guarantee that the
+cache makes is that, if it does return data, that data will be exactly the
+data that was inserted.
 
 To run an offline verification of existing cache contents, use `npm cache
 verify`.
@@ -89,3 +93,5 @@ The root cache folder.
 * [npm pack](/cli-commands/pack)
 * https://npm.im/cacache
 * https://npm.im/pacote
+* https://npm.im/@npmcli/arborist
+* https://npm.im/make-fetch-happen
diff --git a/docs/content/cli-commands/npm-ci.md b/docs/content/cli-commands/npm-ci.md
index b5b6447a4..bb3bf9314 100644
--- a/docs/content/cli-commands/npm-ci.md
+++ b/docs/content/cli-commands/npm-ci.md
@@ -9,10 +9,36 @@ description: Install a project with a clean slate
 ## Install a project with a clean slate
 
 ### Synopsis
+
 ```bash
 npm ci
 ```
 
+### Description
+
+This command is similar to [`npm install`](/cli-commands/install), except
+it's meant to be used in automated environments such as test platforms,
+continuous integration, and deployment -- or any situation where you want
+to make sure you're doing a clean install of your dependencies.
+
+`npm ci` will be significantly faster when:
+
+- There is a `package-lock.json` or `npm-shrinkwrap.json` file.
+- The `node_modules` folder is missing or empty.
+
+In short, the main differences between using `npm install` and `npm ci` are:
+
+* The project **must** have an existing `package-lock.json` or
+  `npm-shrinkwrap.json`.
+* If dependencies in the package lock do not match those in `package.json`,
+  `npm ci` will exit with an error, instead of updating the package lock.
+* `npm ci` can only install entire projects at a time: individual
+  dependencies cannot be added with this command.
+* If a `node_modules` is already present, it will be automatically removed
+  before `npm ci` begins its install.
+* It will never write to `package.json` or any of the package-locks:
+  installs are essentially frozen.
+
 ### Example
 
 Make sure you have a package-lock and an up-to-date install:
@@ -43,24 +69,6 @@ cache:
   - "$HOME/.npm"
 ```
 
-### Description
-
-This command is similar to [`npm install`](/cli-commands/install), except it's meant to be used in
-automated environments such as test platforms, continuous integration, and
-deployment -- or any situation where you want to make sure you're doing a clean
-install of your dependencies. It can be significantly faster than a regular npm
-install by skipping certain user-oriented features. It is also more strict than
-a regular install, which can help catch errors or inconsistencies caused by the
-incrementally-installed local environments of most npm users.
-
-In short, the main differences between using `npm install` and `npm ci` are:
-
-* The project **must** have an existing `package-lock.json` or `npm-shrinkwrap.json`.
-* If dependencies in the package lock do not match those in `package.json`, `npm ci` will exit with an error, instead of updating the package lock.
-* `npm ci` can only install entire projects at a time: individual dependencies cannot be added with this command.
-* If a `node_modules` is already present, it will be automatically removed before `npm ci` begins its install.
-* It will never write to `package.json` or any of the package-locks: installs are essentially frozen.
-
 ### See Also
 
 * [npm install](/cli-commands/install)
diff --git a/docs/content/cli-commands/npm-completion.md b/docs/content/cli-commands/npm-completion.md
index 59bfca503..c79cf176a 100644
--- a/docs/content/cli-commands/npm-completion.md
+++ b/docs/content/cli-commands/npm-completion.md
@@ -9,6 +9,7 @@ description: Tab Completion for npm
 ## Tab Completion for npm
 
 ### Synopsis
+
 ```bash
 source <(npm completion)
 ```
diff --git a/docs/content/cli-commands/npm-config.md b/docs/content/cli-commands/npm-config.md
index 68d403746..0c20e4935 100644
--- a/docs/content/cli-commands/npm-config.md
+++ b/docs/content/cli-commands/npm-config.md
@@ -9,6 +9,7 @@ description: Manage the npm configuration files
 ## Manage the npm configuration files
 
 ### Synopsis
+
 ```bash
 npm config set <key> <value> [-g|--global]
 npm config get <key>
@@ -26,10 +27,11 @@ aliases: c
 npm gets its config settings from the command line, environment
 variables, `npmrc` files, and in some cases, the `package.json` file.
 
-See [npmrc](/configuring-npm/npmrc) for more information about the npmrc files.
+See [npmrc](/configuring-npm/npmrc) for more information about the npmrc
+files.
 
-See [config](/using-npm/config) for a more thorough discussion of the mechanisms
-involved.
+See [config(7)](/using-npm/config) for a more thorough explanation of the
+mechanisms involved, and a full list of config options available.
 
 The `npm config` command can be used to update and edit the contents
 of the user and global npmrc files.
@@ -39,14 +41,17 @@ of the user and global npmrc files.
 Config supports the following sub-commands:
 
 #### set
+
 ```bash
 npm config set key value
 ```
+
 Sets the config key to the value.
 
 If value is omitted, then it sets it to "true".
 
 #### get
+
 ```bash
 npm config get key
 ```
@@ -54,6 +59,7 @@ npm config get key
 Echo the config value to stdout.
 
 #### list
+
 ```bash
 npm config list
 ```
@@ -62,6 +68,7 @@ Show all the config settings. Use `-l` to also show defaults. Use `--json`
 to show the settings in json format.
 
 #### delete
+
 ```bash
 npm config delete key
 ```
@@ -69,6 +76,7 @@ npm config delete key
 Deletes the key from all configuration files.
 
 #### edit
+
 ```bash
 npm config edit
 ```
diff --git a/docs/content/cli-commands/npm-dedupe.md b/docs/content/cli-commands/npm-dedupe.md
index e0493f5e2..84ccb772f 100644
--- a/docs/content/cli-commands/npm-dedupe.md
+++ b/docs/content/cli-commands/npm-dedupe.md
@@ -9,6 +9,7 @@ description: Reduce duplication
 ## Reduce duplication
 
 ### Synopsis
+
 ```bash
 npm dedupe
 npm ddp
@@ -55,8 +56,6 @@ be deleted.
 
 Arguments are ignored. Dedupe always acts on the entire tree.
 
-Modules
-
 Note that this operation transforms the dependency tree, but will never
 result in new modules being installed.