Add latest changes from gitlab-org/gitlab@15-6-stable-eev15.6.0-rc42

7 files changed, 115 insertions, 146 deletions

diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md
index e4aef2db9a8..e7a6fc35ced 100644
--- a/doc/administration/gitaly/configure_gitaly.md
+++ b/doc/administration/gitaly/configure_gitaly.md
@@ -57,21 +57,6 @@ The process for setting up Gitaly on its own server is:
 1. [Configure Gitaly clients](#configure-gitaly-clients).
 1. [Disable Gitaly where not required](#disable-gitaly-where-not-required-optional) (optional).
 
-When running Gitaly on its own server, note the following regarding GitLab versions:
-
-- From GitLab 11.4, Gitaly was able to serve all Git requests without requiring a shared NFS mount
-  for Git repository data, except for the
-  [Elasticsearch indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer).
-- From GitLab 11.8, the Elasticsearch indexer also uses Gitaly for data access. NFS can still be
-  leveraged for redundancy on block-level Git data, but should be mounted only on the Gitaly
-  servers.
-- From GitLab 11.8 to 12.2, it is possible to use Elasticsearch in a Gitaly setup that doesn't use
-  NFS. To use Elasticsearch in these versions, the
-  [repository indexer](../../integration/advanced_search/elasticsearch.md#elasticsearch-repository-indexer)
-  must be enabled in your GitLab configuration.
-- [In GitLab 12.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/6481), the new indexer is
-  the default and no configuration is required.
-
 ### Network architecture
 
 The following list depicts the network architecture of Gitaly:
@@ -1100,8 +1085,8 @@ benefit from it. It is orthogonal to:
 
 - The transport (HTTP or SSH).
 - Git protocol version (v0 or v2).
-- The type of fetch (full clones, incremental fetches, shallow clones,
-  partial clones, and so on).
+- The type of fetch, such as full clones, incremental fetches, shallow clones,
+  or partial clones.
 
 The strength of this cache is its ability to deduplicate concurrent
 identical fetches. It:
@@ -1309,18 +1294,38 @@ following keys (in this example, to disable the `hasDotgit` consistency check):
 - In [GitLab 15.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6800) and later:
 
   ```ruby
+  ignored_blobs = "/etc/gitlab/instance_wide_ignored_git_blobs.txt"
+
   gitaly['gitconfig'] = [
+
+   # Populate a file with one unabbreviated SHA-1 per line.
+   # See https://git-scm.com/docs/git-config#Documentation/git-config.txt-fsckskipList
+   { key: "fsck.skipList", value: ignored_blobs },
+   { key: "fetch.fsck.skipList", value: ignored_blobs },
+   { key: "receive.fsck.skipList", value: ignored_blobs },
+
    { key: "fsck.hasDotgit", value: "ignore" },
    { key: "fetch.fsck.hasDotgit", value: "ignore" },
-   { key: "receive.fsck.hasDotgit", value: "ignore "},
+   { key: "receive.fsck.hasDotgit", value: "ignore" },
+   { key: "fsck.missingSpaceBeforeEmail", value: "ignore" },
   ]
   ```
 
 - In GitLab 15.2 and earlier (legacy method):
 
   ```ruby
-  ignored_git_errors = ["hasDotgit = ignore"]
+  ignored_git_errors = [
+    "hasDotgit = ignore",
+    "missingSpaceBeforeEmail = ignore",
+  ]
   omnibus_gitconfig['system'] = {
+
+   # Populate a file with one unabbreviated SHA-1 per line.
+   # See https://git-scm.com/docs/git-config#Documentation/git-config.txt-fsckskipList
+    "fsck.skipList" => ignored_blobs
+    "fetch.fsck.skipList" => ignored_blobs,
+    "receive.fsck.skipList" => ignored_blobs,
+
     "fsck" => ignored_git_errors,
     "fetch.fsck" => ignored_git_errors,
     "receive.fsck" => ignored_git_errors,
@@ -1342,6 +1347,30 @@ value = "ignore"
 [[git.config]]
 key = "receive.fsck.hasDotgit"
 value = "ignore"
+
+[[git.config]]
+key = "fsck.missingSpaceBeforeEmail"
+value = "ignore"
+
+[[git.config]]
+key = "fetch.fsck.missingSpaceBeforeEmail"
+value = "ignore"
+
+[[git.config]]
+key = "receive.fsck.missingSpaceBeforeEmail"
+value = "ignore"
+
+[[git.config]]
+key = "fsck.skipList"
+value = "/etc/gitlab/instance_wide_ignored_git_blobs.txt"
+
+[[git.config]]
+key = "fetch.fsck.skipList"
+value = "/etc/gitlab/instance_wide_ignored_git_blobs.txt"
+
+[[git.config]]
+key = "receive.fsck.skipList"
+value = "/etc/gitlab/instance_wide_ignored_git_blobs.txt"
 ```
 
 ## Configure commit signing for GitLab UI commits
diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md
index 96447239116..1b7e1f0f7c6 100644
--- a/doc/administration/gitaly/index.md
+++ b/doc/administration/gitaly/index.md
@@ -45,7 +45,7 @@ repository storage is either:
 ## Before deploying Gitaly Cluster
 
 Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management.
-Before deploying Gitaly Cluster, please review:
+Before deploying Gitaly Cluster, review:
 
 - Existing [known issues](#known-issues).
 - [Snapshot limitations](#snapshot-backup-and-recovery-limitations).
@@ -66,13 +66,14 @@ Contact your Technical Account Manager or customer support if you have any quest
 ### Known issues
 
 The following table outlines current known issues impacting the use of Gitaly Cluster. For
-the current status of these issues, please refer to the referenced issues and epics.
+the current status of these issues, refer to the referenced issues and epics.
 
 | Issue                                                                                 | Summary                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | How to avoid |
 |:--------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------|
 | Gitaly Cluster + Geo - Issues retrying failed syncs                             | If Gitaly Cluster is used on a Geo secondary site, repositories that have failed to sync could continue to fail when Geo tries to resync them. Recovering from this state requires assistance from support to run manual steps. | No known solution prior to GitLab 15.0. In GitLab 15.0 to 15.2, enable the [`gitaly_praefect_generated_replica_paths` feature flag](#praefect-generated-replica-paths-gitlab-150-and-later). In GitLab 15.3, the feature flag is enabled by default. |
 | Praefect unable to insert data into the database due to migrations not being applied after an upgrade | If the database is not kept up to date with completed migrations, then the Praefect node is unable to perform normal operation. | Make sure the Praefect database is up and running with all migrations completed (For example: `/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migrate-status` should show a list of all applied migrations). Consider [requesting upgrade assistance](https://about.gitlab.com/support/scheduling-upgrade-assistance/) so your upgrade plan can be reviewed by support. |
-| Restoring a Gitaly Cluster node from a snapshot in a running cluster | Because the Gitaly Cluster runs with consistent state, introducing a single node that is behind will result in the cluster not being able to reconcile the nodes data and other nodes data | Don't restore a single Gitaly Cluster node from a backup snapshot. If you must restore from backup, it's best to [shut down GitLab](../read_only_gitlab.md#shut-down-the-gitlab-ui), snapshot all Gitaly Cluster nodes at the same time and take a database dump of the Praefect database. |
+| Restoring a Gitaly Cluster node from a snapshot in a running cluster | Because the Gitaly Cluster runs with consistent state, introducing a single node that is behind results in the cluster not being able to reconcile the nodes data and other nodes data | Don't restore a single Gitaly Cluster node from a backup snapshot. If you must restore from backup:<br/><br/>1. [Shut down GitLab](../read_only_gitlab.md#shut-down-the-gitlab-ui).<br/>2. Snapshot all Gitaly Cluster nodes at the same time.<br/>3. Take a database dump of the Praefect database. |
+| Rebuilding or replacing an existing Gitaly Cluster node | There is no way to replace existing nodes in place because the Praefect database is relied on to determine the current state of each Gitaly node. Changing how Gitaly Cluster stores repositories is proposed in issue [4218](https://gitlab.com/gitlab-org/gitaly/-/issues/4218). Turning on [repository verification](praefect.md#repository-verification) is proposed in issue [4429](https://gitlab.com/gitlab-org/gitaly/-/issues/4429) so Praefect can detect that data is missing and needs to replicated to a new Gitaly node. | No known solution at this time. |
 
 ### Snapshot backup and recovery limitations
 
@@ -83,11 +84,11 @@ during a restore, we recommend using the [official backup and restore Rake tasks
 The [incremental backup method](../../raketasks/backup_gitlab.md#incremental-repository-backups)
 can be used to speed up Gitaly Cluster backups.
 
-If you are unable to use either method, please contact customer support for restoration help.
+If you are unable to use either method, contact customer support for restoration help.
 
 ### What to do if you are on Gitaly Cluster experiencing an issue or limitation
 
-Please contact customer support for immediate help in restoration or recovery.
+Contact customer support for immediate help in restoration or recovery.
 
 ## Directly accessing repositories
 
@@ -219,8 +220,7 @@ The availability objectives for Gitaly clusters assuming a single node failure a
   second. Failover requires ten consecutive failed health checks on each
   Praefect node.
 
-  Faster outage detection, to improve this speed to less than 1 second,
-  is tracked [in this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/2608).
+Improvements to RPO and RTO are proposed in epic [8903](https://gitlab.com/groups/gitlab-org/-/epics/8903).
 
 WARNING:
 If complete cluster failure occurs, disaster recovery plans should be executed. These can affect the
@@ -320,9 +320,7 @@ follow the [hashed storage](../repository_storage_types.md#hashed-storage) schem
 > - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4218) in GitLab 15.0 [with a flag](../feature_flags.md) named `gitaly_praefect_generated_replica_paths`. Disabled by default.
 > - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitaly/-/issues/4218) in GitLab 15.2.
 > - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4809) in GitLab 15.3.
-
-FLAG:
-On self-managed GitLab, by default this feature is available. To hide the feature, ask an administrator to [disable the feature flag](../feature_flags.md) named `gitaly_praefect_generated_replica_paths`. On GitLab.com, this feature is available but can be configured by GitLab.com administrators only.
+> - [Generally available](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4941) in GitLab 15.6. Feature flag `gitaly_praefect_generated_replica_paths` removed.
 
 When Gitaly Cluster creates a repository, it assigns the repository a unique and permanent ID called the _repository ID_. The repository ID is
 internal to Gitaly Cluster and doesn't relate to any IDs elsewhere in GitLab. If a repository is removed from Gitaly Cluster and later moved
@@ -410,7 +408,7 @@ relative path of the repository in the metadata store.
 ### Moving beyond NFS
 
 Engineering support for NFS for Git repositories is deprecated. Technical support is planned to be unavailable starting
-November 22, 2022. Please see our [statement of support](https://about.gitlab.com/support/statement-of-support/#gitaly-and-nfs)
+November 22, 2022. See our [statement of support](https://about.gitlab.com/support/statement-of-support/#gitaly-and-nfs)
 for more details.
 
 [Network File System (NFS)](https://en.wikipedia.org/wiki/Network_File_System)
@@ -470,12 +468,6 @@ including [horizontally distributing reads](https://gitlab.com/groups/gitlab-org
 
 #### Distributed reads
 
-> - Introduced in GitLab 13.1 in [beta](../../policy/alpha-beta-support.md#beta-features) with feature flag `gitaly_distributed_reads` set to disabled.
-> - [Made generally available and enabled by default](https://gitlab.com/gitlab-org/gitaly/-/issues/2951) in GitLab 13.3.
-> - [Disabled by default](https://gitlab.com/gitlab-org/gitaly/-/issues/3178) in GitLab 13.5.
-> - [Enabled by default](https://gitlab.com/gitlab-org/gitaly/-/issues/3334) in GitLab 13.8.
-> - [Feature flag removed](https://gitlab.com/gitlab-org/gitaly/-/issues/3383) in GitLab 13.11.
-
 Gitaly Cluster supports distribution of read operations across Gitaly nodes that are configured for
 the [virtual storage](#virtual-storage).
 
@@ -496,31 +488,21 @@ You can [monitor distribution of reads](monitoring.md#monitor-gitaly-cluster) us
 
 #### Strong consistency
 
-> - Introduced in GitLab 13.1 in [alpha](../../policy/alpha-beta-support.md#alpha-features), disabled by default.
-> - Entered [beta](../../policy/alpha-beta-support.md#beta-features) in GitLab 13.2, disabled by default.
-> - In GitLab 13.3, disabled unless primary-wins voting strategy is disabled.
-> - From GitLab 13.4, enabled by default.
-> - From GitLab 13.5, you must use Git v2.28.0 or higher on Gitaly nodes to enable strong consistency.
-> - From GitLab 13.6, primary-wins voting strategy and the `gitaly_reference_transactions_primary_wins` feature flag was removed.
-> - From GitLab 14.0, [Gitaly Cluster only supports strong consistency](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3575), and the `gitaly_reference_transactions` feature flag was removed.
+> - In GitLab 13.6 to 13.12, strong consistency must be manually configured. Refer to [the 13.12 documentation](https://docs.gitlab.com/13.12/ee/administration/gitaly/praefect.html#strong-consistency).
+> - In GitLab 14.0, strong consistency is the primary replication method.
 
 Gitaly Cluster provides strong consistency by writing changes synchronously to all healthy, up-to-date replicas. If a
 replica is outdated or unhealthy at the time of the transaction, the write is asynchronously replicated to it.
 
+Strong consistency is the primary replication method. A subset of operations still use replication jobs
+(eventual consistency) instead of strong consistency. Refer to the
+[strong consistency epic](https://gitlab.com/groups/gitlab-org/-/epics/1189) for more information.
+
 If strong consistency is unavailable, Gitaly Cluster guarantees eventual consistency. In this case. Gitaly Cluster
 replicates all writes to secondary Gitaly nodes after the write to the primary Gitaly node has occurred.
 
-Strong consistency:
-
-- Is the primary replication method in GitLab 14.0 and later. A subset of operations still use replication jobs
-  (eventual consistency) instead of strong consistency. Refer to the
-  [strong consistency epic](https://gitlab.com/groups/gitlab-org/-/epics/1189) for more information.
-- Must be configured in GitLab versions 13.1 to 13.12. For configuration information, refer to either:
-  - Documentation on your GitLab instance at `/help`.
-  - The [13.12 documentation](https://docs.gitlab.com/13.12/ee/administration/gitaly/praefect.html#strong-consistency).
-- Is unavailable in GitLab 13.0 and earlier.
-
-For more information on monitoring strong consistency, see the Gitaly Cluster [Prometheus metrics documentation](monitoring.md#monitor-gitaly-cluster).
+For more information on monitoring strong consistency, see the Gitaly Cluster
+[Prometheus metrics documentation](monitoring.md#monitor-gitaly-cluster).
 
 #### Replication factor
 
@@ -598,7 +580,7 @@ To migrate to Gitaly Cluster:
 1. Create the required storage. Refer to
    [repository storage recommendations](praefect.md#repository-storage-recommendations).
 1. Create and configure [Gitaly Cluster](praefect.md).
-1. Configure the existing Gitaly instance [to use TPC](praefect.md#use-tcp-for-existing-gitlab-instances), if not already configured that way.
+1. Configure the existing Gitaly instance [to use TCP](praefect.md#use-tcp-for-existing-gitlab-instances), if not already configured that way.
 1. [Move the repositories](../operations/moving_repositories.md#move-repositories). To migrate to
    Gitaly Cluster, existing repositories stored outside Gitaly Cluster must be moved. There is no
    automatic migration but the moves can be scheduled with the GitLab API.
@@ -718,4 +700,4 @@ The second facet presents the only real solution. For this, we developed
 ## NFS deprecation notice
 
 Engineering support for NFS for Git repositories is deprecated. Technical support is planned to be
-unavailable beginning November 22, 2022. For further information, please see our [NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation) documentation.
+unavailable beginning November 22, 2022. For further information, see our [NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation) documentation.
diff --git a/doc/administration/gitaly/monitoring.md b/doc/administration/gitaly/monitoring.md
index 9b7acd536b3..8d4f30c7c20 100644
--- a/doc/administration/gitaly/monitoring.md
+++ b/doc/administration/gitaly/monitoring.md
@@ -152,9 +152,6 @@ The following metrics are available from the `/metrics` endpoint:
   for replication to complete after the replication job starts. Available in GitLab 12.10 and later.
 - `gitaly_praefect_replication_delay_bucket`, a histogram measuring how much time passes between
   when the replication job is created and when it starts. Available in GitLab 12.10 and later.
-- `gitaly_praefect_node_latency_bucket`, a histogram measuring the latency in Gitaly returning
-  health check information to Praefect. This indicates Praefect connection saturation. Available in
-  GitLab 12.10 and later.
 - `gitaly_praefect_connections_total`, the total number of connections to Praefect. [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4220) in GitLab 14.7.
 
 To monitor [strong consistency](index.md#strong-consistency), you can use the following Prometheus metrics:
diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md
index e3b198d1012..8cf32a6aaa3 100644
--- a/doc/administration/gitaly/praefect.md
+++ b/doc/administration/gitaly/praefect.md
@@ -28,6 +28,9 @@ The minimum recommended configuration for a Gitaly Cluster requires:
 - 3 Praefect nodes
 - 3 Gitaly nodes (1 primary, 2 secondary)
 
+You should configure an odd number of Gitaly nodes so that transactions have a tie-breaker in case one of the
+Gitaly nodes fails in a mutating RPC call.
+
 See the [design document](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/design_ha.md)
 for implementation details.
 
@@ -155,6 +158,14 @@ We note in the instructions below where these secrets are required.
 NOTE:
 Omnibus GitLab installations can use `gitlab-secrets.json` for `GITLAB_SHELL_SECRET_TOKEN`.
 
+### Customize time server setting
+
+By default, Gitaly and Praefect nodes use the time server at `pool.ntp.org` for time synchronization checks. You can customize this setting by adding the
+following to `gitlab.rb` on each node:
+
+- `gitaly['env'] = { "NTP_HOST" => "ntp.example.com" }`, for Gitaly nodes.
+- `praefect['env'] = { "NTP_HOST" => "ntp.example.com" }`, for Praefect nodes.
+
 ### PostgreSQL
 
 NOTE:
@@ -285,7 +296,7 @@ praefect['database_direct_dbname'] = 'praefect_production'
 #praefect['database_direct_sslrootcert'] = '...'
 ```
 
-Once configured, this connection is automatically used for the
+When configured, this connection is automatically used for the
 [SQL LISTEN](https://www.postgresql.org/docs/11/sql-listen.html) feature and
 allows Praefect to receive notifications from PostgreSQL for cache invalidation.
 
@@ -584,11 +595,7 @@ Updates to example must be made at:
    }
    ```
 
-   NOTE:
-   In [GitLab 13.8 and earlier](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4988),
-   Gitaly nodes were configured directly under the virtual storage, and not under the `nodes` key.
-
-1. [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2013) in GitLab 13.1 and later, enable [distribution of reads](index.md#distributed-reads).
+1. Enable [distribution of reads](index.md#distributed-reads).
 
 1. Save the changes to `/etc/gitlab/gitlab.rb` and
    [reconfigure Praefect](../restart_gitlab.md#omnibus-gitlab-reconfigure):
@@ -641,8 +648,6 @@ Updates to example must be made at:
 
 #### Enable TLS support
 
-> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/1698) in GitLab 13.2.
-
 Praefect supports TLS encryption. To communicate with a Praefect instance that listens
 for secure connections, you must:
 
@@ -1369,7 +1374,8 @@ We recommend using [repository-specific primary nodes](#repository-specific-prim
 
 ### Repository-specific primary nodes
 
-> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/3492) in GitLab 13.12.
+> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/3492) in GitLab 13.12, with primary elections run when Praefect starts or the cluster's consensus of a Gitaly node's health changes.
+> - [Changed](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3543) in GitLab 14.1, primary elections are run lazily.
 
 Gitaly Cluster supports electing repository-specific primary Gitaly nodes. Repository-specific
 Gitaly primary nodes are enabled in `/etc/gitlab/gitlab.rb` by setting
@@ -1386,14 +1392,8 @@ The `per_repository` election strategy solves this problem by electing a primary
 repository. Combined with [configurable replication factors](#configure-replication-factor), you can
 horizontally scale storage capacity and distribute write load across Gitaly nodes.
 
-Primary elections are run:
-
-- In GitLab 14.1 and later, lazily. This means that Praefect doesn't immediately elect
-  a new primary node if the current one is unhealthy. A new primary is elected if it is
-  necessary to serve a request while the current primary is unavailable.
-- In GitLab 13.12 to GitLab 14.0 when:
-  - Praefect starts up.
-  - The cluster's consensus of a Gitaly node's health changes.
+Primary elections are run lazily. Praefect doesn't immediately elect a new primary node if the current
+one is unhealthy. A new primary is elected if a request must be served while the current primary is unavailable.
 
 A valid primary node candidate is a Gitaly node that:
 
@@ -1422,9 +1422,9 @@ To migrate existing clusters:
 
 1. Praefect nodes didn't historically keep database records of every repository stored on the cluster. When
    the `per_repository` election strategy is configured, Praefect expects to have database records of
-   each repository. A [background database migration](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/2749) is
-   included in GitLab 13.6 and later to create any missing database records for repositories. Before migrating,
-   check Praefect's logs to verify that the database migration ran.
+   each repository. A [background database migration](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/2749)
+   creates any missing database records for repositories. Before migrating, check Praefect's logs to verify
+   that the database migration ran.
 
    Check Praefect's logs for `repository importer finished` message. The `virtual_storages` field contains
    the names of virtual storages and whether they've had any missing database records created.
diff --git a/doc/administration/gitaly/recovery.md b/doc/administration/gitaly/recovery.md
index b51454aa44e..56894f3e963 100644
--- a/doc/administration/gitaly/recovery.md
+++ b/doc/administration/gitaly/recovery.md
@@ -11,40 +11,15 @@ recovery and has Praefect tracking database tools.
 
 ## Primary node failure
 
-Gitaly Cluster recovers from a failing primary Gitaly node by promoting a healthy secondary as the
-new primary.
+> - Introduced in GitLab 13.0, Gitaly Cluster, elects the secondary with the least unreplicated writes from the primary to be the new primary. There can still be some unreplicated writes, so [data loss can occur](#check-for-data-loss).
+> - Primary node failure recovery support added in GitLab 14.1.
 
-In GitLab 14.1 and later, Gitaly Cluster:
+Gitaly Cluster recovers from a failing primary Gitaly node by promoting a healthy secondary as the new primary. Gitaly
+Cluster:
 
 - Elects a healthy secondary with a fully up to date copy of the repository as the new primary.
 - Repository becomes unavailable if there are no fully up to date copies of it on healthy secondaries.
 
-To minimize data loss in GitLab 13.0 to 14.0, Gitaly Cluster:
-
-- Switches repositories that are outdated on the new primary to [read-only mode](#read-only-mode).
-- Elects the secondary with the least unreplicated writes from the primary to be the new
-  primary. Because there can still be some unreplicated writes,
-  [data loss can occur](#check-for-data-loss).
-
-### Read-only mode
-
-> - Introduced in GitLab 13.0 as [generally available](../../policy/alpha-beta-support.md#generally-available-ga).
-> - Between GitLab 13.0 and GitLab 13.2, read-only mode applied to the whole virtual storage and occurred whenever failover occurred.
-> - [In GitLab 13.3 and later](https://gitlab.com/gitlab-org/gitaly/-/issues/2862), read-only mode applies on a per-repository basis and only occurs if a new primary is out of date. If the failed primary contained unreplicated writes, [data loss can occur](#check-for-data-loss).
-> - Removed in GitLab 14.1. Instead, repositories [become unavailable](#unavailable-repositories).
-
-When Gitaly Cluster switches to a new primary in GitLab 13.0 to 14.0, repositories enter read-only mode if they are
-out-of-date. This can happen after failing over to an outdated secondary. Read-only mode eases data recovery efforts by
-preventing writes that may conflict with the unreplicated writes on other nodes.
-
-To enable writes again in GitLab 13.0 to 14.0, an administrator can:
-
-1. [Check](#check-for-data-loss) for data loss.
-1. Attempt to [recover](#data-recovery) missing data.
-1. Either [enable writes](#enable-writes-or-accept-data-loss) in the virtual storage or
-   [accept data loss](#enable-writes-or-accept-data-loss) if necessary, depending on the version of
-   GitLab.
-
 ### Unavailable repositories
 
 > - From GitLab 13.0 through 14.0, repositories became read-only if they were outdated on the primary but fully up to date on a healthy secondary. `dataloss` sub-command displays read-only repositories by default through these versions.
@@ -144,9 +119,7 @@ Virtual storage: default
 
 #### Unavailable replicas of available repositories
 
-NOTE:
-In GitLab 14.0 and earlier, the flag is `-partially-replicated` and the output shows any repositories with assigned nodes with outdated
-copies.
+> Introduced in GitLab 14.0, flag renamed from `-partially-replicated` and behavior changed.
 
 To also list information of repositories which are available but are unavailable from some of the assigned nodes,
 use the `-partially-unavailable` flag.
@@ -209,26 +182,17 @@ WARNING:
 `accept-dataloss` causes permanent data loss by overwriting other versions of the repository.
 Data [recovery efforts](#data-recovery) must be performed before using it.
 
-Praefect provides the following subcommands to re-enable writes or accept data loss:
-
-- In GitLab 13.2 and earlier, `enable-writes` to re-enable virtual storage for writes after
-  data recovery attempts:
+Praefect provides the following subcommands to re-enable writes or accept data loss. If it is not possible to bring one
+of the up-to-date nodes back online, you might have to accept data loss:
 
-  ```shell
-  sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml enable-writes -virtual-storage <virtual-storage>
-  ```
-
-- In GitLab 13.3 and later, if it is not possible to bring one of the up to date nodes back
-  online, you may have to accept data loss:
-
-  ```shell
-  sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml accept-dataloss -virtual-storage <virtual-storage> -repository <relative-path> -authoritative-storage <storage-name>
-  ```
+```shell
+sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml accept-dataloss -virtual-storage <virtual-storage> -repository <relative-path> -authoritative-storage <storage-name>
+```
 
-  When accepting data loss, Praefect:
+When accepting data loss, Praefect:
 
-  1. Marks the chosen copy of the repository as the latest version.
-  1. Replicates the copy to the other assigned Gitaly nodes.
+1. Marks the chosen copy of the repository as the latest version.
+1. Replicates the copy to the other assigned Gitaly nodes.
 
   This process overwrites any other copy of the repository so care must be taken.
 
diff --git a/doc/administration/gitaly/reference.md b/doc/administration/gitaly/reference.md
index 3bf1e3136c0..8f7dc688e56 100644
--- a/doc/administration/gitaly/reference.md
+++ b/doc/administration/gitaly/reference.md
@@ -192,8 +192,7 @@ For historical reasons
 [GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell) contains
 the Git hooks that allow GitLab to validate and react to Git pushes.
 Because Gitaly "owns" Git pushes, GitLab Shell must therefore be
-installed alongside Gitaly. We plan to
-[simplify this](https://gitlab.com/gitlab-org/gitaly/-/issues/1226).
+installed alongside Gitaly.
 
 | Name | Type | Required | Description |
 | ---- | ---- | -------- | ----------- |
diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md
index 1c5f0d43864..7edce840396 100644
--- a/doc/administration/gitaly/troubleshooting.md
+++ b/doc/administration/gitaly/troubleshooting.md
@@ -41,17 +41,6 @@ The `gitaly-debug` command provides "production debugging" tools for Gitaly and
 performance. It is intended to help production engineers and support
 engineers investigate Gitaly performance problems.
 
-If you're using GitLab 11.6 or newer, this tool should be installed on
-your GitLab or Gitaly server already at `/opt/gitlab/embedded/bin/gitaly-debug`.
-If you're investigating an older GitLab version you can compile this
-tool offline and copy the executable to your server:
-
-```shell
-git clone https://gitlab.com/gitlab-org/gitaly.git
-cd cmd/gitaly-debug
-GOOS=linux GOARCH=amd64 go build -o gitaly-debug
-```
-
 To see the help page of `gitaly-debug` for a list of supported sub-commands, run:
 
 ```shell
@@ -144,9 +133,8 @@ If you have Prometheus set up to scrape your Gitaly process, you can see
 request rates and error codes for individual RPCs in `gitaly-ruby` by
 querying `grpc_client_handled_total`.
 
-- In theory, this metric does not differentiate between `gitaly-ruby` and other RPCs.
-- In practice from GitLab 11.9, all gRPC calls made by Gitaly itself are internal calls from the
-  main Gitaly process to one of its `gitaly-ruby` sidecars.
+All gRPC calls made by `gitaly-ruby` itself are internal calls from the main Gitaly process to one of its `gitaly-ruby`
+sidecars.
 
 Assuming your `grpc_client_handled_total` counter only observes Gitaly,
 the following query shows you RPCs are (most likely) internally
@@ -335,7 +323,7 @@ You might see the following in Gitaly and Praefect logs:
 }
 ```
 
-This is a GRPC call
+This is a gRPC call
 [error response code](https://grpc.github.io/grpc/core/md_doc_statuscodes.html).
 
 If this error occurs, even though
@@ -359,7 +347,7 @@ necessary because [this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/252
 If this error occurs even though file permissions are correct, it's likely that the Gitaly node is
 experiencing [clock drift](https://en.wikipedia.org/wiki/Clock_drift).
 
-Please ensure that the GitLab and Gitaly nodes are synchronized and use an NTP time
+Ensure that the GitLab and Gitaly nodes are synchronized and use an NTP time
 server to keep them synchronized if possible.
 
 ### Health check warnings
@@ -604,6 +592,16 @@ For each replica, the following metadata is available:
 | `Valid Primary`  | Indicates whether the replica is fit to serve as the primary node. If the repository's primary is not a valid primary, a failover occurs on the next write to the repository if there is another replica that is a valid primary. A replica is a valid primary if:<br><br>- It is stored on a healthy Gitaly node.<br>- It is fully up to date.<br>- It is not targeted by a pending deletion job from decreasing replication factor.<br>- It is assigned. |
 | `Verified At` | Indicates last successful verification of the replica by the [verification worker](praefect.md#repository-verification). If the replica has not yet been verified, `unverified` is displayed in place of the last successful verification time. Introduced in GitLab 15.0. |
 
+#### Command fails with 'repository not found'
+
+If the supplied value for `-virtual-storage` is incorrect, the command returns the following error:
+
+```plaintext
+get metadata: rpc error: code = NotFound desc = repository not found
+```
+
+The documented examples specify `-virtual-storage default`. Check the Praefect server setting `praefect['virtual_storages']` in `/etc/gitlab/gitlab.rb`.
+
 ### Check that repositories are in sync
 
 Is [some cases](index.md#known-issues) the Praefect database can get out of sync with the underlying Gitaly nodes. To check that