document reconciliation fixes in virtual storage documentation

This commit adds documentation about the reconciliation jobs the
reconciler schedules to fix various inconsistencies that might happen
in a virtual storage.

1 files changed, 10 insertions, 1 deletions

diff --git a/doc/virtual_storage.md b/doc/virtual_storage.md
index e14b24894..bbada3c14 100644
--- a/doc/virtual_storage.md
+++ b/doc/virtual_storage.md
@@ -62,7 +62,7 @@ Praefect identifies inconsistencies in the storage cluster by cross-referencing
 
 Expected state of physical storages can be attained by cross joining the configured physical storages with the expected repositories of the virtual storage in the `repositories` table. It's important to use configured storages as some physical storages might have been added to or removed from the virtual storage.
 
-Some possible inconsistencies are listed below. Each of the scenarios assume a virtual storage called `default` with a primary storage `gitaly-1` and a secondary storage  `gitaly-2`.
+Possible inconsistencies and their reconciliations are listed below. Each of the scenarios assume a virtual storage called `default` with a primary storage `gitaly-1` and a secondary storage  `gitaly-2`.
 
 ### Missing Repository
 
@@ -90,6 +90,8 @@ Praefect expects an up to date copy of a repository to be present on every assig
 |-----------------|------------------------------------------------------------------------------------|----------|------------|
 | default         | @hashed/5f/9c/5f9c4ab08cac7457e9111a30e4664920607ea2c115a1433d7be98e97e64244ca.git | gitaly-1 | 0          |
 
+To fix the inconsistency, reconciler schedules `update`-type jobs to the storages missing the repository from random healthy storages with up to date replicas.
+
 ### Outdated Repository
 
 A repository is considered outdated if its `generation` number in `storage_repositories` does not match the expected generation in the `repositories` table. This might be due to two reasons:
@@ -119,6 +121,8 @@ In the case below, `gitaly-2` has an outdated version of the repository as its g
 | default         | @hashed/5f/9c/5f9c4ab08cac7457e9111a30e4664920607ea2c115a1433d7be98e97e64244ca.git | gitaly-1 | 2          |
 | default         | @hashed/5f/9c/5f9c4ab08cac7457e9111a30e4664920607ea2c115a1433d7be98e97e64244ca.git | gitaly-2 | 0          |
 
+To fix the inconsistency, reconciler schedules `update`-type jobs to the storages missing the repository from random healthy storages with up to date replicas.
+
 ### Unexpected Repository
 
 #### Deleted Repository
@@ -143,6 +147,8 @@ A physical storage might contain a repository that is not expected be present on
 |-----------------|------------------------------------------------------------------------------------|----------|------------|
 | default         | @hashed/5f/9c/5f9c4ab08cac7457e9111a30e4664920607ea2c115a1433d7be98e97e64244ca.git | gitaly-2 | 2          |
 
+Praefect's reconciler doesn't fix the inconsistency at this time. A fix is tracked in https://gitlab.com/gitlab-org/gitaly/-/issues/3480.
+
 ### Unassigned Replica
 
 A physical storage might contain a replica of a repository even if it is not assigned to host it. This can happen if the storage was previously
@@ -197,6 +203,9 @@ from the virtual storage. Below, the repository's replication factor is `1` as `
 | default         | @hashed/5f/9c/5f9c4ab08cac7457e9111a30e4664920607ea2c115a1433d7be98e97e64244ca.git | gitaly-1 | 2          |
 | default         | @hashed/5f/9c/5f9c4ab08cac7457e9111a30e4664920607ea2c115a1433d7be98e97e64244ca.git | gitaly-2 | 3          |
 
+The reconciler considers assigned but removed storages as still assigned. This means it won't schedule `delete_replica` jobs to any assigned storage before the assignments
+of the removed storages are manually removed.
+
 ## Known Problems
 
 1. When a primary is demoted, it might be in process of accepting a write. If there is a concurrent write to the new primary, one of the writes is going to be lost as the primary increments its generation even if it was not on the latest one. This issues and the solution proposed is tracked in [#2969](https://gitlab.com/gitlab-org/gitaly/-/issues/2969).