diff options
Diffstat (limited to 'doc/administration/gitaly/troubleshooting.md')
| -rw-r--r-- | doc/administration/gitaly/troubleshooting.md | 17 |
1 files changed, 15 insertions, 2 deletions
diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index 1be0bf23ed5..c79ed1d1707 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -2,7 +2,6 @@ stage: Create group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference --- # Troubleshooting Gitaly and Gitaly Cluster **(FREE SELF)** @@ -356,7 +355,7 @@ The following sections provide possible solutions to Gitaly Cluster errors. ### Check cluster health -> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/) in GitLab 14.6. +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5688) in GitLab 14.5. The `check` Praefect sub-command runs a series of checks to determine the health of the Gitaly Cluster. @@ -417,6 +416,16 @@ If this check fails: 1. Check if there is a high load on the Praefect database. If the Praefect database is slow to respond, it can lead health checks failing to persist to the database, leading Praefect to think nodes are unhealthy. +#### Check clock synchronization + +> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4225) in GitLab 14.8. + +Authentication between Praefect and the Gitaly servers requires the server times to be +in sync so the token check succeeds. + +This check helps identify the root cause of `permission denied` +[errors being logged by Praefect](#permission-denied-errors-appearing-in-gitaly-or-praefect-logs-when-accessing-repositories). + ### Praefect errors in logs If you receive an error, check `/var/log/gitlab/gitlab-rails/production.log`. @@ -513,16 +522,19 @@ Replicas: Generation: 1, fully up to date Healthy: true Valid Primary: true + Verified At: 2021-04-01 10:04:20 +0000 UTC - Storage: "gitaly-2" Assigned: true Generation: 0, behind by 1 changes Healthy: true Valid Primary: false + Verified At: unverified - Storage: "gitaly-3" Assigned: true Generation: replica not yet created Healthy: false Valid Primary: false + Verified At: unverified ``` #### Available metadata @@ -548,6 +560,7 @@ For each replica, the following metadata is available: | `Generation` | Latest confirmed generation of the replica. It indicates:<br><br>- The replica is fully up to date if the generation matches the repository's generation.<br>- The replica is outdated if the replica's generation is less than the repository's generation.<br>- `replica not yet created` if the replica does not yet exist at all on the storage. | | `Healthy` | Indicates whether the Gitaly node that is hosting this replica is considered healthy by the consensus of Praefect nodes. | | `Valid Primary` | Indicates whether the replica is fit to serve as the primary node. If the repository's primary is not a valid primary, a failover occurs on the next write to the repository if there is another replica that is a valid primary. A replica is a valid primary if:<br><br>- It is stored on a healthy Gitaly node.<br>- It is fully up to date.<br>- It is not targeted by a pending deletion job from decreasing replication factor.<br>- It is assigned. | +| `Verified At` | Indicates last successful verification of the replica by the [verification worker](praefect.md#repository-verification). If the replica has not yet been verified, `unverified` is displayed in place of the last successful verification time. Introduced in GitLab 15.0. | ### Check that repositories are in sync |