Add latest changes from gitlab-org/gitlab@15-8-stable-eev15.8.0-rc42

7 files changed, 131 insertions, 66 deletions

diff --git a/.gitlab/issue_templates/Broken Master - Flaky.md b/.gitlab/issue_templates/Broken Master - Flaky.md
index bea12615e41..b87d9b5ee1f 100644
--- a/.gitlab/issue_templates/Broken Master - Flaky.md
+++ b/.gitlab/issue_templates/Broken Master - Flaky.md
@@ -16,7 +16,7 @@ Please read the below documentations for a workflow of triaging and resolving br
 
 <!-- If the pipeline failure is reproducible, provide steps to recreate the issue locally. Please use an ordered list. -->
 
-Please refer to [Flaky tests documentation](https://docs.gitlab.com/ee/development/testing_guide/flaky_tests.html) to 
+Please refer to [Flaky tests documentation](https://docs.gitlab.com/ee/development/testing_guide/flaky_tests.html) to
 learn more about how to reproduce them.
 
 ### Proposed Resolution
@@ -25,4 +25,6 @@ learn more about how to reproduce them.
 
 Please refer to the [Resolution guidance](https://about.gitlab.com/handbook/engineering/workflow/#resolution-of-broken-master) to learn more about resolution of broken master.
 
-/label ~"failure::flaky-test" ~"Engineering Productivity" ~"priority::2" ~"severity::3" ~"type::bug" ~"bug::transient"
+Once the flaky failure has been fixed on the default branch, open merge requests to cherry-pick the fix to the active stable branches.
+
+/label ~"type::maintenance" ~"failure::flaky-test" ~"priority::3" ~"severity::3"
diff --git a/.gitlab/issue_templates/Broken Master - Non-flaky.md b/.gitlab/issue_templates/Broken Master - Non-flaky.md
index 43e73fc5c5a..d2dc616ead8 100644
--- a/.gitlab/issue_templates/Broken Master - Non-flaky.md
+++ b/.gitlab/issue_templates/Broken Master - Non-flaky.md
@@ -21,4 +21,4 @@ Please read the below documentations for a workflow of triaging and resolving br
 
 Please refer to the [Resolution guidance](https://about.gitlab.com/handbook/engineering/workflow/#resolution-of-broken-master) to learn more about resolution of broken master.
 
-/label ~"master:broken" ~"Engineering Productivity" ~"priority::1" ~"severity::1" ~"type::bug" ~"bug::transient"
+/label ~"master:broken" ~"Engineering Productivity" ~"priority::1" ~"severity::1" ~"type::maintenance" ~"maintenance::pipelines"
diff --git a/.gitlab/issue_templates/Doc_cleanup.md b/.gitlab/issue_templates/Doc_cleanup.md
index 3ea692ed1ac..1eb3829e281 100644
--- a/.gitlab/issue_templates/Doc_cleanup.md
+++ b/.gitlab/issue_templates/Doc_cleanup.md
@@ -1,5 +1,3 @@
-/labels ~"documentation" ~"docs-only" ~"documentation" ~"docs::improvement" ~"type::maintenance" ~"maintenance::refactor" ~"Seeking community contributions"  ~"quick win" ~"Technical Writing"
-
 <!--
 * Use this template for documentation issues identified
 * by [Vale](https://docs.gitlab.com/ee/development/documentation/testing.html#vale)
@@ -16,6 +14,8 @@ Do you want to work on this issue?
 
 - **If the issue is unassigned**, in a comment, type `@docs-hackathon I would like to work on this issue` and a writer will assign it to you.
 
+  To be fair to others, do not ask for more than three issues at a time.
+
 - **If the issue is assigned to someone already**, choose another issue. Do not open a merge request for this issue if you are not assigned.
 
 ## To resolve the issue
@@ -35,4 +35,4 @@ Thank you again for contributing to the GitLab documentation! :tada:
 
 ## Documentation issue
 
-
+/labels ~"documentation" ~"docs-only" ~"documentation" ~"docs::improvement" ~"type::maintenance" ~"maintenance::refactor" ~"Seeking community contributions"  ~"quick win" ~"Technical Writing"
diff --git a/.gitlab/issue_templates/Experiment Successful Cleanup.md b/.gitlab/issue_templates/Experiment Successful Cleanup.md
index 1dd57332b8e..14a29452e49 100644
--- a/.gitlab/issue_templates/Experiment Successful Cleanup.md
+++ b/.gitlab/issue_templates/Experiment Successful Cleanup.md
@@ -11,6 +11,8 @@ The changes need to become an official part of the product.
 - [ ] Determine whether the feature should apply to EE - and which tiers - and/or Core
 - [ ] Determine if tracking should be kept as is, removed, or modified.
 - [ ] Ensure any relevant documentation has been updated.
+- [ ] Determine whether there are other concerns that need to be considered before removing the feature flag.
+   - These are typically captured in the `Experiment Successful Cleanup Concerns` section of the rollout issue.
 - [ ] Consider changes to any `feature_category:` introduced by the experiment if ownership is changing (PM for Growth and PM for the new category as DRIs)
 - [ ] Check to see if the experiment introduced new design assets. Add them to the appropriate repos and document them if needed.
 - [ ] Optional: Migrate experiment to a default enabled [feature flag](https://docs.gitlab.com/ee/development/feature_flags) for one milestone and add a changelog. Converting to a feature flag can be skipped at the ICs discretion if risk is deemed low with consideration to both SaaS and (if applicable) self managed
diff --git a/.gitlab/issue_templates/Feature Flag Roll Out.md b/.gitlab/issue_templates/Feature Flag Roll Out.md
index 3972368ddc4..8aa631dce76 100644
--- a/.gitlab/issue_templates/Feature Flag Roll Out.md
+++ b/.gitlab/issue_templates/Feature Flag Roll Out.md
@@ -113,12 +113,14 @@ For visibility, all `/chatops` commands that target production should be execute
 For visibility, all `/chatops` commands that target production should be executed in the `#production` slack channel and cross-posted (with the command results) to the responsible team's slack channel (`#g_TEAM_NAME`).
 
 - [ ] [Incrementally roll out](https://docs.gitlab.com/ee/development/feature_flags/controls.html#process) the feature.
+  - [ ] Between every step wait for at least 15 minutes and monitor the appropriate graphs on https://dashboards.gitlab.net.
   - If the feature flag in code has [an actor](https://docs.gitlab.com/ee/development/feature_flags/#feature-actors), perform **actor-based** rollout.
     - [ ] `/chatops run feature set <feature-flag-name> <rollout-percentage> --actors`
   - If the feature flag in code does **NOT** have [an actor](https://docs.gitlab.com/ee/development/feature_flags/#feature-actors), perform time-based rollout (**random** rollout).
     - [ ] `/chatops run feature set <feature-flag-name> <rollout-percentage> --random`
   - Enable the feature globally on production environment.
     - [ ] `/chatops run feature set <feature-flag-name> true`
+- [ ] Observe appropriate graphs on https://dashboards.gitlab.net and verify that services are not affected.
 - [ ] Leave a comment on [the feature issue][main-issue] announcing that the feature has been globally enabled.
 - [ ] Wait for [at least one day for the verification term](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#including-a-feature-behind-feature-flag-in-the-final-release).
 
diff --git a/.gitlab/issue_templates/Geo Replicate a new Git repository type.md b/.gitlab/issue_templates/Geo Replicate a new Git repository type.md
index 571b0db0a30..97f756f0d02 100644
--- a/.gitlab/issue_templates/Geo Replicate a new Git repository type.md
+++ b/.gitlab/issue_templates/Geo Replicate a new Git repository type.md
@@ -18,7 +18,7 @@ If your Model's pluralized form is non-standard, i.e. it doesn't just end in `s`
 
 -->
 
-## Replicate Cool Widgets
+## Replicate Cool Widgets - Repository
 
 This issue is for implementing Geo replication and verification of Cool Widgets.
 
@@ -39,8 +39,6 @@ You can look into the following example for implementing replication/verificatio
 
 ### Modify database schemas to prepare to add Geo support for Cool Widgets
 
-You might do this section in its own merge request, but it is not required.
-
 #### Add the registry table to track replication and verification state
 
 Geo secondary sites have a [Geo tracking database](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/geo.md#tracking-database) independent of the main database. It is used to track the replication and verification state of all replicables. Every Model has a corresponding "registry" table in the Geo tracking database.
@@ -114,7 +112,7 @@ Geo secondary sites have a [Geo tracking database](https://gitlab.com/gitlab-org
   bin/rake db:migrate:geo
   ```
 
-- [ ] Be sure to commit the relevant changes in `ee/db/geo/structure.sql`
+- [ ] Be sure to commit the relevant changes in `ee/db/geo/structure.sql` and the file under `ee/db/geo/schema_migrations`
 
 ### Add verification state to the Model
 
@@ -146,7 +144,7 @@ The Geo primary site needs to checksum every replicable so secondaries can verif
         t.datetime_with_timezone :verified_at
         t.references :cool_widget, primary_key: true, default: nil, index: false, foreign_key: { on_delete: :cascade }
         t.integer :verification_state, default: 0, limit: 2, null: false
-        t.integer :verification_retry_count, limit: 2
+        t.integer :verification_retry_count, default: 0, limit: 2, null: false
         t.binary :verification_checksum, using: 'verification_checksum::bytea'
         t.text :verification_failure, limit: 255
 
@@ -185,7 +183,21 @@ The Geo primary site needs to checksum every replicable so secondaries can verif
   bin/rake db:migrate
   ```
 
-- [ ] Be sure to commit the relevant changes in `db/structure.sql`
+- [ ] Be sure to commit the relevant changes in `db/structure.sql` and the file under `db/schema_migrations`
+
+- [ ] Add an entry for the state table in `db/docs/cool_widget_states.yml`
+
+  ```yaml
+  ---
+  table_name: cool_widget_states
+  classes:
+    - Geo::CoolWidgetState
+  feature_categories:
+    - geo_replication
+  description: Separate table for cool widget verification states
+  introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/XXXXX
+  milestone: 'XX.Y'
+  ```
 
 That's all of the required database changes.
 
@@ -371,7 +383,6 @@ That's all of the required database changes.
 
   ```shell
   bin/feature-flag --ee geo_cool_widget_replication --type development --group 'group::geo'
-  bin/feature-flag --ee geo_cool_widget_verification --type development --group 'group::geo'
   ```
 
 - [ ] Add this replicator class to the method `replicator_classes` in
@@ -382,7 +393,6 @@ That's all of the required database changes.
     ::Geo::PackageFileReplicator,
     ::Geo::CoolWidgetReplicator
   ]
-  end
   ```
 
 - [ ] Create `ee/spec/replicators/geo/cool_widget_replicator_spec.rb` and perform the necessary setup to define the `model_record` variable for the shared examples:
@@ -478,25 +488,29 @@ That's all of the required database changes.
   end
   ```
 
-- [ ] Add the following to `spec/factories/cool_widgets.rb`:
+- [ ] Add the following to `ee/spec/factories/cool_widgets.rb`:
 
   ```ruby
-  trait :verification_succeeded do
-    with_file
-    verification_checksum { 'abc' }
-    verification_state { CoolWidget.verification_state_value(:verification_succeeded) }
-  end
+  FactoryBot.modify do
+    trait :verification_succeeded do
+        with_file
+        verification_checksum { 'abc' }
+        verification_state { CoolWidget.verification_state_value(:verification_succeeded) }
+    end
 
-  trait :verification_failed do
-    with_file
-    verification_failure { 'Could not calculate the checksum' }
-    verification_state { CoolWidget.verification_state_value(:verification_failed) }
+    trait :verification_failed do
+        with_file
+        verification_failure { 'Could not calculate the checksum' }
+        verification_state { CoolWidget.verification_state_value(:verification_failed) }
+    end
   end
   ```
 
+  If there is not an existing factory for the object in `spec/factories/cool_widgets.rb`, wrap the traits in `FactoryBot.create` instead of `FactoryBot.modify`.
+
 - [ ] Make sure the factory also allows setting a `project` attribute. If the model does not have a direct relation to a project, you can use a `transient` attribute. Check out `spec/factories/merge_request_diffs.rb` for an example.
 
-- [ ] Following [the example of Merge Request Diffs](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63309) add a `Geo::CoolWidgetState` model in `ee/app/models/ee/geo/cool_widget_state.rb`:
+- [ ] Following [the example of Merge Request Diffs](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63309) add a `Geo::CoolWidgetState` model in `ee/app/models/geo/cool_widget_state.rb`:
 
   ``` ruby
   # frozen_string_literal: true
@@ -536,6 +550,8 @@ That's all of the required database changes.
   end
   ```
 
+- [ ] Add `[:cool_widget, :remote_store]` and `[:geo_cool_widget_state, any]` to `skipped` in `spec/models/factories_spec.rb`
+
 #### Step 2. Implement metrics gathering
 
 Metrics are gathered by `Geo::MetricsUpdateWorker`, persisted in `GeoNodeStatus` for display in the UI, and sent to Prometheus:
@@ -556,16 +572,18 @@ Metrics are gathered by `Geo::MetricsUpdateWorker`, persisted in `GeoNodeStatus`
 - [ ] Add the same fields to `GET /geo_nodes/status` example response in
   `ee/spec/fixtures/api/schemas/public_api/v4/geo_node_status.json`.
 - [ ] Add the following fields to the `Sidekiq metrics` table in `doc/administration/monitoring/prometheus/gitlab_metrics.md`:
-  - `geo_cool_widgets`
-  - `geo_cool_widgets_checksum_total`
-  - `geo_cool_widgets_checksummed`
-  - `geo_cool_widgets_checksum_failed`
-  - `geo_cool_widgets_synced`
-  - `geo_cool_widgets_failed`
-  - `geo_cool_widgets_registry`
-  - `geo_cool_widgets_verification_total`
-  - `geo_cool_widgets_verified`
-  - `geo_cool_widgets_verification_failed`
+  ```markdown
+  | `geo_cool_widgets` | Gauge | XX.Y | Number of Cool Widgets on primary | `url` |
+  | `geo_cool_widgets_checksum_total` | Gauge | XX.Y | Number of Cool Widgets checksummed successfully on primary | `url` |
+  | `geo_cool_widgets_checksummed` | Gauge | XX.Y | Number of Cool Widgets failed to calculate the checksum on primary | `url` |
+  | `geo_cool_widgets_checksum_failed` | Gauge | XX.Y | Number of Cool Widgets tried to checksum on primary | `url` |
+  | `geo_cool_widgets_synced` | Gauge | XX.Y | Number of syncable Cool Widgets synced on secondary | `url` |
+  | `geo_cool_widgets_failed` | Gauge | XX.Y | Number of syncable Cool Widgets failed to sync on secondary | `url` |
+  | `geo_cool_widgets_registry` | Gauge | XX.Y | Number of Cool Widgets in the registry | `url` |
+  | `geo_cool_widgets_verification_total` | Gauge | XX.Y | Number of Cool Widgets verified on secondary | `url` |
+  | `geo_cool_widgets_verified` | Gauge | XX.Y | Number of Cool Widgets' verifications failed on secondary | `url` |
+  | `geo_cool_widgets_verification_failed` | Gauge | XX.Y | Number of Cool Widgets' verifications tried on secondary | `url` |
+  ```
 
 Cool Widget replication and verification metrics should now be available in the API, the `Admin > Geo > Nodes` view, and Prometheus.
 
@@ -731,6 +749,14 @@ As illustrated by the above two examples, batch destroy logic cannot be handled
     end
   end
 ```
+
+### Code Review
+
+When requesting review from database reviewers:
+
+- [ ] Include a comment mentioning that the change is based on a documented template.
+- [ ] `replicables_for_current_secondary` and `available_replicables` may differ per Model. If their queries are new, then add [query plans](https://docs.gitlab.com/ee/development/database_review.html#query-plans) to the MR description. An easy place to gather SQL queries is your GDK's `log/test.log` when running tests of these methods.
+
 ### Release Geo support of Cool Widgets
 
 - [ ] In the rollout issue you created when creating the feature flag, modify the Roll Out Steps:
diff --git a/.gitlab/issue_templates/Geo Replicate a new blob type.md b/.gitlab/issue_templates/Geo Replicate a new blob type.md
index 121dbdf035f..9dfc83309cc 100644
--- a/.gitlab/issue_templates/Geo Replicate a new blob type.md
+++ b/.gitlab/issue_templates/Geo Replicate a new blob type.md
@@ -18,7 +18,7 @@ If your Model's pluralized form is non-standard, i.e. it doesn't just end in `s`
 
 -->
 
-## Replicate Cool Widgets
+## Replicate Cool Widgets - Blob
 
 This issue is for implementing Geo replication and verification of Cool Widgets.
 
@@ -41,8 +41,6 @@ You can look into the following examples of MRs for implementing replication/ver
 
 ### Modify database schemas to prepare to add Geo support for Cool Widgets
 
-You might do this section in its own merge request, but it is not required.
-
 #### Add the registry table to track replication and verification state
 
 Geo secondary sites have a [Geo tracking database](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/geo.md#tracking-database) independent of the main database. It is used to track the replication and verification state of all replicables. Every Model has a corresponding "registry" table in the Geo tracking database.
@@ -114,7 +112,7 @@ Geo secondary sites have a [Geo tracking database](https://gitlab.com/gitlab-org
   bin/rake db:migrate:geo
   ```
 
-- [ ] Be sure to commit the relevant changes in `ee/db/geo/structure.sql`
+- [ ] Be sure to commit the relevant changes in `ee/db/geo/structure.sql` and the file created under `ee/db/geo/schema_migrations`
 
 ### Add verification state fields on the Geo primary site
 
@@ -148,7 +146,7 @@ The Geo primary site needs to checksum every replicable so secondaries can verif
         t.datetime_with_timezone :verified_at
         t.references :cool_widget, primary_key: true, default: nil, index: false, foreign_key: { on_delete: :cascade }
         t.integer :verification_state, default: 0, limit: 2, null: false
-        t.integer :verification_retry_count, limit: 2
+        t.integer :verification_retry_count, default: 0, limit: 2, null: false
         t.binary :verification_checksum, using: 'verification_checksum::bytea'
         t.text :verification_failure, limit: 255
 
@@ -189,7 +187,21 @@ The Geo primary site needs to checksum every replicable so secondaries can verif
 
 - [ ] If `cool_widgets` is a high-traffic table, follow [the database documentation to use `with_lock_retries`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/migration_style_guide.md#when-to-use-the-helper-method)
 
-- [ ] Be sure to commit the relevant changes in `db/structure.sql`
+- [ ] Be sure to commit the relevant changes in `db/structure.sql` and the file under `db/schema_migrations`
+
+- [ ] Add an entry for the state table in `db/docs/cool_widget_states.yml`
+
+  ```yaml
+  ---
+  table_name: cool_widget_states
+  classes:
+    - Geo::CoolWidgetState
+  feature_categories:
+    - geo_replication
+  description: Separate table for cool widget verification states
+  introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/XXXXX
+  milestone: 'XX.Y'
+  ```
 
 That's all of the required database changes.
 
@@ -248,7 +260,7 @@ That's all of the required database changes.
 
       # @param primary_key_in [Range, CoolWidget] arg to pass to primary_key_in scope
       # @return [ActiveRecord::Relation<CoolWidget>] everything that should be synced to this node, restricted by primary key
-      def self.replicables_for_current_secondary(primary_key_in)
+      def replicables_for_current_secondary(primary_key_in)
         # This issue template does not help you write this method.
         #
         # This method is called only on Geo secondary sites. It is called when
@@ -329,7 +341,6 @@ That's all of the required database changes.
 
   ```shell
   bin/feature-flag --ee geo_cool_widget_replication --type development --group 'group::geo'
-  bin/feature-flag --ee geo_cool_widget_verification --type development --group 'group::geo'
   ```
 
 - [ ] Add this replicator class to the method `replicator_classes` in
@@ -340,7 +351,6 @@ That's all of the required database changes.
     ::Geo::PackageFileReplicator,
     ::Geo::CoolWidgetReplicator
   ]
-  end
   ```
 
 - [ ] Create `ee/spec/replicators/geo/cool_widget_replicator_spec.rb` and perform the necessary setup to define the `model_record` variable for the shared examples:
@@ -439,22 +449,33 @@ That's all of the required database changes.
 - [ ] Add the following to `spec/factories/cool_widgets.rb`:
 
   ```ruby
-  trait :verification_succeeded do
-    with_file
-    verification_checksum { 'abc' }
-    verification_state { CoolWidget.verification_state_value(:verification_succeeded) }
-  end
+  FactoryBot.modify do
+    trait :verification_succeeded do
+        with_file
+        verification_checksum { 'abc' }
+        verification_state { CoolWidget.verification_state_value(:verification_succeeded) }
+    end
 
-  trait :verification_failed do
-    with_file
-    verification_failure { 'Could not calculate the checksum' }
-    verification_state { CoolWidget.verification_state_value(:verification_failed) }
+    trait :verification_failed do
+        with_file
+        verification_failure { 'Could not calculate the checksum' }
+        verification_state { CoolWidget.verification_state_value(:verification_failed) }
+    end
   end
   ```
 
+  If there is not an existing factory for the object in `spec/factories/cool_widgets.rb`, wrap the traits in `FactoryBot.create` instead of `FactoryBot.modify`
+
+ [ ] Make sure the factory supports the `:remote_store` trait. If not, add something like
+
+  ```ruby
+  trait :remote_store do
+    file_store { CoolWidget::FileUploader::Store::REMOTE }
+  end
+  ```
 - [ ] Make sure the factory also allows setting a `project` attribute. If the model does not have a direct relation to a project, you can use a `transient` attribute. Check out `spec/factories/merge_request_diffs.rb` for an example.
 
-- [ ] Following [the example of Merge Request Diffs](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63309) add a `Geo::CoolWidgetState` model in `ee/app/models/ee/geo/cool_widget_state.rb`:
+- [ ] Following [the example of Merge Request Diffs](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63309) add a `Geo::CoolWidgetState` model in `ee/app/models/geo/cool_widget_state.rb`:
 
   ``` ruby
   # frozen_string_literal: true
@@ -494,6 +515,8 @@ That's all of the required database changes.
   end
   ```
 
+- [ ] Add `[:cool_widget, :remote_store]` and `[:geo_cool_widget_state, any]` to `skipped` in `spec/models/factories_spec.rb`
+
 #### Step 2. Implement metrics gathering
 
 Metrics are gathered by `Geo::MetricsUpdateWorker`, persisted in `GeoNodeStatus` for display in the UI, and sent to Prometheus:
@@ -514,18 +537,21 @@ Metrics are gathered by `Geo::MetricsUpdateWorker`, persisted in `GeoNodeStatus`
 - [ ] Add the same fields to `GET /geo_nodes/status` example response in
   `ee/spec/fixtures/api/schemas/public_api/v4/geo_node_status.json`.
 - [ ] Add the following fields to the `Sidekiq metrics` table in `doc/administration/monitoring/prometheus/gitlab_metrics.md`:
-  - `geo_cool_widgets`
-  - `geo_cool_widgets_checksum_total`
-  - `geo_cool_widgets_checksummed`
-  - `geo_cool_widgets_checksum_failed`
-  - `geo_cool_widgets_synced`
-  - `geo_cool_widgets_failed`
-  - `geo_cool_widgets_registry`
-  - `geo_cool_widgets_verification_total`
-  - `geo_cool_widgets_verified`
-  - `geo_cool_widgets_verification_failed`
-
-Cool Widget replication and verification metrics should now be available in the API, the `Admin > Geo > Nodes` view, and Prometheus.
+
+  ```markdown
+  | `geo_cool_widgets` | Gauge | XX.Y | Number of Cool Widgets on primary | `url` |
+  | `geo_cool_widgets_checksum_total` | Gauge | XX.Y | Number of Cool Widgets checksummed successfully on primary | `url` |
+  | `geo_cool_widgets_checksummed` | Gauge | XX.Y | Number of Cool Widgets failed to calculate the checksum on primary | `url` |
+  | `geo_cool_widgets_checksum_failed` | Gauge | XX.Y | Number of Cool Widgets tried to checksum on primary | `url` |
+  | `geo_cool_widgets_synced` | Gauge | XX.Y | Number of syncable Cool Widgets synced on secondary | `url` |
+  | `geo_cool_widgets_failed` | Gauge | XX.Y | Number of syncable Cool Widgets failed to sync on secondary | `url` |
+  | `geo_cool_widgets_registry` | Gauge | XX.Y | Number of Cool Widgets in the registry | `url` |
+  | `geo_cool_widgets_verification_total` | Gauge | XX.Y | Number of Cool Widgets verified on secondary | `url` |
+  | `geo_cool_widgets_verified` | Gauge | XX.Y | Number of Cool Widgets' verifications failed on secondary | `url` |
+  | `geo_cool_widgets_verification_failed` | Gauge | XX.Y | Number of Cool Widgets' verifications tried on secondary | `url` |
+  ```
+
+ Cool Widget replication and verification metrics should now be available in the API, the `Admin > Geo > Nodes` view, and Prometheus.
 
 #### Step 3. Implement the GraphQL API
 
@@ -690,6 +716,13 @@ As illustrated by the above two examples, batch destroy logic cannot be handled
   end
 ```
 
+### Code Review
+
+When requesting review from database reviewers:
+
+- [ ] Include a comment mentioning that the change is based on a documented template.
+- [ ] `replicables_for_current_secondary` and `available_replicables` may differ per Model. If their queries are new, then add [query plans](https://docs.gitlab.com/ee/development/database_review.html#query-plans) to the MR description. An easy place to gather SQL queries is your GDK's `log/test.log` when running tests of these methods.
+
 ### Release Geo support of Cool Widgets
 
 - [ ] In the rollout issue you created when creating the feature flag, modify the Roll Out Steps: