docs: update package-lock/shrinkwrap documentation

4 files changed, 313 insertions, 176 deletions

diff --git a/doc/cli/npm-shrinkwrap.md b/doc/cli/npm-shrinkwrap.md
index ae04bd02b..4c223a86c 100644
--- a/doc/cli/npm-shrinkwrap.md
+++ b/doc/cli/npm-shrinkwrap.md
@@ -1,4 +1,4 @@
-npm-shrinkwrap(1) -- Lock down dependency versions
+npm-shrinkwrap(1) -- Lock down dependency versions for publication
 =====================================================
 
 ## SYNOPSIS
@@ -7,181 +7,11 @@ npm-shrinkwrap(1) -- Lock down dependency versions
 
 ## DESCRIPTION
 
-This command locks down the versions of a package's dependencies so
-that you can control exactly which versions of each dependency will be
-used when your package is installed. The `package.json` file is still
-required if you want to use `npm install`.
-
-By default, `npm install` recursively installs the target's
-dependencies (as specified in `package.json`), choosing the latest
-available version that satisfies the dependency's semver pattern. In
-some situations, particularly when shipping software where each change
-is tightly managed, it's desirable to fully specify each version of
-each dependency recursively so that subsequent builds and deploys do
-not inadvertently pick up newer versions of a dependency that satisfy
-the semver pattern. Specifying specific semver patterns in each
-dependency's `package.json` would facilitate this, but that's not always
-possible or desirable, as when another author owns the npm package.
-It's also possible to check dependencies directly into source control,
-but that may be undesirable for other reasons.
-
-As an example, consider package A:
-
-    {
-      "name": "A",
-      "version": "0.1.0",
-      "dependencies": {
-        "B": "<0.1.0"
-      }
-    }
-
-package B:
-
-    {
-      "name": "B",
-      "version": "0.0.1",
-      "dependencies": {
-        "C": "<0.1.0"
-      }
-    }
-
-and package C:
-
-    {
-      "name": "C",
-      "version": "0.0.1"
-    }
-
-If these are the only versions of A, B, and C available in the
-registry, then a normal `npm install A` will install:
-
-    A@0.1.0
-    `-- B@0.0.1
-        `-- C@0.0.1
-
-However, if B@0.0.2 is published, then a fresh `npm install A` will
-install:
-
-    A@0.1.0
-    `-- B@0.0.2
-        `-- C@0.0.1
-
-assuming the new version did not modify B's dependencies. Of course,
-the new version of B could include a new version of C and any number
-of new dependencies. If such changes are undesirable, the author of A
-could specify a dependency on B@0.0.1. However, if A's author and B's
-author are not the same person, there's no way for A's author to say
-that he or she does not want to pull in newly published versions of C
-when B hasn't changed at all.
-
-In this case, A's author can run
-
-    npm shrinkwrap
-
-This generates `npm-shrinkwrap.json`, which will look something like this:
-
-    {
-      "name": "A",
-      "version": "0.1.0",
-      "dependencies": {
-        "B": {
-          "version": "0.0.1",
-          "from": "B@^0.0.1",
-          "resolved": "https://registry.npmjs.org/B/-/B-0.0.1.tgz",
-          "dependencies": {
-            "C": {
-              "version": "0.0.1",
-              "from": "org/C#v0.0.1",
-              "resolved": "git://github.com/org/C.git#5c380ae319fc4efe9e7f2d9c78b0faa588fd99b4"
-            }
-          }
-        }
-      }
-    }
-
-The shrinkwrap command has locked down the dependencies based on what's
-currently installed in `node_modules`.  The installation behavior is changed to:
-
-1. The module tree described by the shrinkwrap is reproduced. This means
-reproducing the structure described in the file, using the specific files
-referenced in "resolved" if available, falling back to normal package
-resolution using "version" if one isn't.
-
-2. The tree is walked and any missing dependencies are installed in the usual fashion.
-
-If `preshrinkwrap`, `shrinkwrap` or `postshrinkwrap` are in the `scripts` property of the
-`package.json`, they will be executed by running `npm shrinkwrap`.
-`preshrinkwrap` and `shrinkwrap` are executed before the shrinkwrap, `postshrinkwrap` is
-executed afterwards. For example to run some postprocessing on the generated file:
-
-    "scripts": { "postshrinkwrap": "node fix-shrinkwrap.js" }
-
-
-### Using shrinkwrapped packages
-
-Using a shrinkwrapped package is no different than using any other
-package: you can `npm install` it by hand, or add a dependency to your
-`package.json` file and `npm install` it.
-
-### Building shrinkwrapped packages
-
-To shrinkwrap an existing package:
-
-1. Run `npm install` in the package root to install the current
-   versions of all dependencies.
-2. Validate that the package works as expected with these versions.
-3. Run `npm shrinkwrap`, add `npm-shrinkwrap.json` to git, and publish
-   your package.
-
-To add or update a dependency in a shrinkwrapped package:
-
-1. Run `npm install` in the package root to install the current
-   versions of all dependencies.
-2. Add or update dependencies. `npm install --save` or `npm install --save-dev`
-   each new or updated package individually to update the `package.json` and
-   the shrinkwrap. Note that they must be explicitly named in order to be
-   installed: running `npm install` with no arguments will merely reproduce
-   the existing shrinkwrap.
-3. Validate that the package works as expected with the new
-   dependencies.
-4. Commit the new `npm-shrinkwrap.json`, and publish your package.
-
-You can use npm-outdated(1) to view dependencies with newer versions
-available.
-
-### Other Notes
-
-A shrinkwrap file must be consistent with the package's `package.json`
-file. `npm shrinkwrap` will fail if required dependencies are not
-already installed, since that would result in a shrinkwrap that
-wouldn't actually work. Similarly, the command will fail if there are
-extraneous packages (not referenced by `package.json`), since that would
-indicate that `package.json` is not correct.
-
-Starting with npm v4.0.1, `devDependencies` are included when you run
-`npm shrinkwrap` and follow the usual rules as to when they're installed.
-As of npm v3.10.8, if you run `npm install --only=production` or
-`npm install --production` with a shrinkwrap including your development
-dependencies they won't be installed. Similarly, if the environment
-variable `NODE_ENV` is `production` then they won't be installed. If you
-need compatibility with versions of npm prior to v3.10.8 or otherwise
-don't want them in your shrinkwrap you can exclude development
-dependencies with:
-`npm shrinkwrap --only=prod` or `npm shrinkwrap --production`.
-
-If shrinkwrapped package A depends on shrinkwrapped package B, B's
-shrinkwrap will not be used as part of the installation of A. However,
-because A's shrinkwrap is constructed from a valid installation of B
-and recursively specifies all dependencies, the contents of B's
-shrinkwrap will implicitly be included in A's shrinkwrap.
-
-### Caveats
-
-If you wish to lock down the specific bytes included in a package, for
-example to have 100% confidence in being able to reproduce a
-deployment or build, then you ought to check your dependencies into
-source control, or pursue some other mechanism that can verify
-contents rather than versions.
+This command repurposes `package-lock.json` into a publishable
+`npm-shrinkwrap.json` or simply creates a new one. The file created and updated
+by this command will then take precedence over any other existing or future
+`package-lock.json` files. For a detailed explanation of the design and purpose
+of package locks in npm, see npm-package-locks(5).
 
 ## SEE ALSO
 
@@ -189,4 +19,7 @@ contents rather than versions.
 * npm-run-script(1)
 * npm-scripts(7)
 * package.json(5)
+* npm-package-locks(5)
+* package-lock.json(5)
+* npm-shrinkwrap.json(5)
 * npm-ls(1)
diff --git a/doc/files/npm-package-locks.md b/doc/files/npm-package-locks.md
new file mode 100644
index 000000000..73786dc91
--- /dev/null
+++ b/doc/files/npm-package-locks.md
@@ -0,0 +1,145 @@
+npm-package-locks(5) -- An explanation of npm lockfiles
+=====================================================
+
+## DESCRIPTION
+
+Conceptually, the "input" to npm-install(1) is a package.json(5), while its
+"output" is a fully-formed `node_modules` tree: a representation of the
+dependencies you declared. In an ideal world, npm would work like a pure
+function: the same `package.json` should produce the exact same `node_modules`
+tree, any time. In some cases, this is indeed true. But in many others, npm is
+unable to do this. There are multiple reasons for this:
+
+* different versions of npm (or other package managers) may have been used to install a package, each using slightly different installation algorithms.
+
+* a new version of a direct semver-range package may have been published since the last time your packages were installed, and thus a newer version will be used.
+
+* A dependency of one of your dependencies may have published a new version, which will update even if you used pinned dependency specifiers (`1.2.3` instead of `^1.2.3`)
+
+* The registry you installed from is no longer available, or allows mutation of versions (unlike the primary npm registry), and a different version of a package exists under the same version number now.
+
+As an example, consider package A:
+
+    {
+      "name": "A",
+      "version": "0.1.0",
+      "dependencies": {
+        "B": "<0.1.0"
+      }
+    }
+
+package B:
+
+    {
+      "name": "B",
+      "version": "0.0.1",
+      "dependencies": {
+        "C": "<0.1.0"
+      }
+    }
+
+and package C:
+
+    {
+      "name": "C",
+      "version": "0.0.1"
+    }
+
+If these are the only versions of A, B, and C available in the
+registry, then a normal `npm install A` will install:
+
+    A@0.1.0
+    `-- B@0.0.1
+        `-- C@0.0.1
+
+However, if B@0.0.2 is published, then a fresh `npm install A` will
+install:
+
+    A@0.1.0
+    `-- B@0.0.2
+        `-- C@0.0.1
+
+assuming the new version did not modify B's dependencies. Of course,
+the new version of B could include a new version of C and any number
+of new dependencies. If such changes are undesirable, the author of A
+could specify a dependency on B@0.0.1. However, if A's author and B's
+author are not the same person, there's no way for A's author to say
+that he or she does not want to pull in newly published versions of C
+when B hasn't changed at all.
+
+To prevent this potential issue, npm uses package-lock.json(5) or, if present,
+npm-shrinkwrap.json(5). These files are called package locks, or lockfiles.
+
+Whenever you run `npm install`, npm generates or updates your package lock,
+which will look something like this:
+
+    {
+      "name": "A",
+      "version": "0.1.0",
+      ...metadata fields...
+      "dependencies": {
+        "B": {
+          "version": "0.0.1",
+          "resolved": "https://registry.npmjs.org/B/-/B-0.0.1.tgz",
+          "integrity": "sha512-DeAdb33F+"
+          "dependencies": {
+            "C": {
+              "version": "git://github.com/org/C.git#5c380ae319fc4efe9e7f2d9c78b0faa588fd99b4"
+            }
+          }
+        }
+      }
+    }
+
+This file describes an *exact*, and more importantly *reproducible*
+`node_modules` tree. Once it's present, and future installation will base its
+work off this file, instead of recalculating dependency versions off
+package.json(5).
+
+The presence of a package lock changes the installation behavior such that:
+
+1. The module tree described by the package lock is reproduced. This means
+reproducing the structure described in the file, using the specific files
+referenced in "resolved" if available, falling back to normal package resolution
+using "version" if one isn't.
+
+2. The tree is walked and any missing dependencies are installed in the usual
+fashion.
+
+If `preshrinkwrap`, `shrinkwrap` or `postshrinkwrap` are in the `scripts`
+property of the `package.json`, they will be executed in order. `preshrinkwrap`
+and `shrinkwrap` are executed before the shrinkwrap, `postshrinkwrap` is
+executed afterwards. These scripts run for both `package-lock.json` and
+`npm-shrinkwrap.json`. For example to run some postprocessing on the generated
+file:
+
+    "scripts": {
+      "postshrinkwrap": "json -I -e \"this.myMetadata = $MY_APP_METADATA\""
+    }
+
+
+### Using locked packages
+
+Using a locked package is no different than using any package without a package
+lock: any commands that update `node_modules` and/or `package.json`'s
+dependencies will automatically sync the existing lockfile. This includes `npm
+install`, `npm rm`, `npm update`, etc. To prevent this update from happening,
+you can use the `--no-save` option to prevent saving altogether, or
+`--no-shrinkwrap` to allow `package.json` to be updated while leaving
+`package-lock.json` or `npm-shrinkwrap.json` intact.
+
+It is highly recommended you commit the generated package lock to source
+control: this will allow anyone else on your team, your deployments, your
+CI/continuous integration, and anyone else who runs `npm install` in your
+package source to get the exact same dependency tree that you were developing
+on. Additionally, the diffs from these changes are human-readable and will
+inform you of any changes npm has made to your `node_modules`, so you can notice
+if any transitive dependencies were updated, hoisted, etc.
+
+## SEE ALSO
+
+* https://medium.com/@sdboyer/so-you-want-to-write-a-package-manager-4ae9c17d9527
+* package.json(5)
+* package-lock.json(5)
+* npm-shrinkwrap.json(5)
+* npm-shrinkwrap(1)
diff --git a/doc/files/npm-shrinkwrap.json.md b/doc/files/npm-shrinkwrap.json.md
new file mode 100644
index 000000000..8256398e8
--- /dev/null
+++ b/doc/files/npm-shrinkwrap.json.md
@@ -0,0 +1,27 @@
+npm-shrinkwrap.json(5) -- A publishable lockfile
+=====================================================
+
+## DESCRIPTION
+
+`npm-shrinkwrap.json` is a file created by npm-shrinkwrap(1). It is identical to
+`package-lock.json`, with one major caveat: Unlike `package-lock.json`,
+`npm-shrinwkrap.json` may be included when publishing a package.
+
+The recommended use-case for `npm-shrinkwrap.json` is applications deployed
+through the publishing process on the registry: for example, daemons and
+command-line tools intended as global installs or `devDependencies`. It's
+strongly discouraged for library authors to publish this file, since that would
+prevent end users from having control over transitive dependency updates.
+
+Additionally, if both `package-lock.json` and `npm-shrinwkrap.json` are present
+in a package root, `package-lock.json` will be ignored in favor of this file.
+
+For full details and description of the `npm-shrinkwrap.json` file format, refer
+to the manual page for package-lock.json(5).
+
+## SEE ALSO
+
+* npm-shrinkwrap(1)
+* package-lock.json(5)
+* package.json(5)
+* npm-install(1)
diff --git a/doc/files/package-lock.json.md b/doc/files/package-lock.json.md
new file mode 100644
index 000000000..dad219b4a
--- /dev/null
+++ b/doc/files/package-lock.json.md
@@ -0,0 +1,132 @@
+package-lock.json(5) -- A manifestation of the manifest
+=====================================================
+
+## DESCRIPTION
+
+`package-lock.json` is automatically generated for any operations where npm
+modifies either the `node_modules` tree, or `package.json`. It describes the
+exact tree that was generated, such that subsequent installs are able to
+generate identical trees, regardless of intermediate dependency updates.
+
+This file is intended to be committed into source repositories, and serves
+various purposes:
+
+* Describe a single representation of a dependency tree such that teammates, deployments, and continuous integration are guaranteed to install exactly the same dependencies.
+
+* Provide a facility for users to "time-travel" to previous states of `node_modules` without having to commit the directory itself.
+
+* To facilitate greater visibility of tree changes through readable source control diffs.
+
+* And optimize the installation process by allowing npm to skip repeated metadata resolutions for previously-installed packages.
+
+One key detail about `package-lock.json` is that it cannot be published, and it
+will be ignored if found in any place other than the toplevel package. It shares
+a format with npm-shrinkwrap.json(5), which is essentially the same file, but
+allows publication. This is not recommended unless deploying a CLI tool or
+otherwise using the publication process for producing production packages.
+
+If both `package-lock.json` and `npm-shrinkwrap.json` are present in the root of
+a package, `package-lock.json` will be completely ignored.
+
+
+## FILE FORMAT
+
+### name
+
+The name of the package this is a package-lock for. This must match what's in
+`package.json`.
+
+### version
+
+The version of the package this is a package-lock for. This must match what's in
+`package.json`.
+
+### lockfileVersion
+
+An integer version, starting at `1` with the version number of this document
+whose semantics were used when generating this `package-lock.json`.
+
+### packageIntegrity
+
+This is a [subresource
+integrity](https://w3c.github.io/webappsec/specs/subresourceintegrity/) value
+created from the `pacakge.json`. No preprocessing of the `package.json` should
+be done. Subresource integrity strings can be produced by modules like
+[`ssri`](https://www.npmjs.com/package/ssri).
+
+### preserveSymlinks
+
+Indicates that the install was done with the environment variable
+`NODE_PRESERVE_SYMLINKS` enabled. The installer should insist that the value of
+this property match that environment variable.
+
+### dependencies
+
+A mapping of package name to dependency object.  Dependency objects have the
+following properties:
+
+#### version
+
+This is a specifier that uniquely identifies this package and should be
+usable in fetching a new copy of it.
+
+* bundled dependencies: Regardless of source, this is a version number that is purely for informational purposes.
+* registry sources: This is a version number. (eg, `1.2.3`)
+* git sources: This is a git specifier with resolved committish. (eg, `git+https://example.com/foo/bar#115311855adb0789a0466714ed48a1499ffea97e`)
+* http tarball sources: This is the URL of the tarball. (eg, `https://example.com/example-1.3.0.tgz`)
+* local tarball sources: This is the file URL of the tarball. (eg `file:///opt/storage/example-1.3.0.tgz`)
+* local link sources: This is the file URL of the link. (eg `file:libs/our-module`)
+
+#### integrity
+
+This is a [Standard Subresource
+Integrity](https://w3c.github.io/webappsec/specs/subresourceintegrity/) for this
+resource.
+
+* For bundled dependencies this is not included, regardless of source.
+* For registry sources, this is the `integrity` that the registry provided, or if one wasn't provided the SHA1 in `shasum`.
+* For git sources this is the specific commit hash we cloned from.
+* For remote tarball sources this is an integrity based on a SHA512 of
+  the file.
+* For local tarball sources: This is an integrity field based on the SHA512 of the file.
+
+#### resolved
+
+* For bundled dependencies this is not included, regardless of source.
+* For registry sources this is path of the tarball relative to the registry
+  URL.  If the tarball URL isn't on the same server as the registry URL then
+  this is a complete URL.
+
+#### bundled
+
+If true, this is the bundled dependency and will be installed by the parent
+module.  When installing, this module will be extracted from the parent
+module during the extract phase, not installed as a separate dependency.
+
+#### dev
+
+If true then this dependency is either a development dependency ONLY of the
+top level module or a transitive dependency of one.  This is false for
+dependencies that are both a development dependency of the top level and a
+transitive dependency of a non-development dependency of the top level.
+
+#### optional
+
+If true then this dependency is either an optional dependency ONLY of the
+top level module or a transitive dependency of one.  This is false for
+dependencies that are both an optional dependency of the top level and a
+transitive dependency of a non-optional dependency of the top level.
+
+All optional dependencies should be included even if they're uninstallable
+on the current platform.
+
+#### dependencies
+
+The dependencies of this dependency, exactly as at the top level.
+
+## SEE ALSO
+
+* npm-shrinkwrap(1)
+* package-lock.json(5)
+* package.json(5)
+* npm-install(1)