Add latest changes from gitlab-org/gitlab@master

4 files changed, 104 insertions, 11 deletions

diff --git a/app/controllers/projects/pipelines_controller.rb b/app/controllers/projects/pipelines_controller.rb
index 6a6770c2781..62b464fe955 100644
--- a/app/controllers/projects/pipelines_controller.rb
+++ b/app/controllers/projects/pipelines_controller.rb
@@ -32,7 +32,12 @@ class Projects::PipelinesController < Projects::ApplicationController
 
   POLLING_INTERVAL = 10_000
 
-  feature_category :continuous_integration
+  feature_category :continuous_integration, [
+                     :charts, :show, :config_variables, :stage, :cancel, :retry,
+                     :builds, :dag, :failures, :status, :downloadable_artifacts,
+                     :index, :create, :new, :destroy
+                   ]
+  feature_category :code_testing, [:test_report]
 
   def index
     @pipelines = Ci::PipelinesFinder
diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md
index 14e5710e1ee..d85b4d6cc3a 100644
--- a/doc/administration/gitaly/praefect.md
+++ b/doc/administration/gitaly/praefect.md
@@ -1012,10 +1012,7 @@ virtual storage.
 ### Configure replication factor
 
 WARNING:
-The feature is not production ready yet. After you set a replication factor, you can't unset it
-without manually modifying database state. Variable replication factor requires you to enable
-repository-specific primaries by configuring the `per_repository` primary election strategy. The election
-strategy is not production ready yet.
+Configurable replication factors require [repository-specific primary nodes](#repository-specific-primary-nodes) to be used.
 
 Praefect supports configuring a replication factor on a per-repository basis, by assigning
 specific storage nodes to host a repository.
@@ -1067,7 +1064,100 @@ Praefect regularly checks the health of each backend Gitaly node. This
 information can be used to automatically failover to a new primary node if the
 current primary node is found to be unhealthy.
 
-- **PostgreSQL (recommended):** Enabled by default, and equivalent to:
+### Election strategies
+
+We recommend using [repository-specific primary nodes](#repository-specific-primary-nodes),
+which is [planned to be the only available election strategy](https://gitlab.com/gitlab-org/gitaly/-/issues/3574)
+from GitLab 14.0.
+
+In the future, we are likely to implement support for:
+
+- A [Consul](../consul.md) strategy.
+- A cloud-native strategy.
+
+#### Repository-specific primary nodes
+
+Praefect's earlier election strategies elected a primary for each virtual storage, which was used as the
+primary for each repository in the virtual storage. This model prevented horizontal scaling of a virtual
+storage. The primary Gitaly node needed a replica of each repository and thus became the bottleneck.
+
+The `per_repository` election strategy solves this problem by electing a primary separately for each repository.
+Combined with [configurable replication factors](#configure-replication-factor), you can horizontally
+scale storage capacity and distribute write load across Gitaly nodes.
+
+Primary elections are run when:
+
+- Praefect starts up.
+- The cluster's consensus of a Gitaly node's health changes.
+
+A Gitaly node is considered:
+
+- Healthy if `>=50%` Praefect nodes have successfully health checked the Gitaly node in the
+  previous ten seconds.
+- Unhealthy otherwise.
+
+During an election run, Praefect elects a new primary Gitaly node for each repository that has
+an unhealthy primary Gitaly node. The election is made:
+
+- Randomly from healthy secondary Gitaly nodes that are the most up to date.
+- Only from Gitaly nodes assigned to the host repository.
+
+If there are no healthy secondary nodes for a repository:
+
+- The unhealthy primary node is demoted and the repository is left without a primary node.
+- Operations that require a primary node fail until a primary is successfully elected.
+
+Repository-specific primaries are enabled in `/etc/gitlab/gitlab.rb` by setting
+`praefect['failover_election_strategy'] = 'per_repository'`.
+
+##### Migrate to repository-specific primary nodes
+
+New Gitaly clusters can start using the `per_repository` election strategy immediately.
+
+To migrate existing clusters:
+
+1. Praefect didn't historically keep database records of every repository stored on the cluster. When
+   the `per_repository` election strategy is configured, Praefect expects to have database records of
+   each repository. A [background migration](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/2749) is
+   included in GitLab 13.6 and later to create any missing database records for repositories. Before migrating
+   you should verify the migration has run by checking Praefect's logs:
+
+   Check Praefect's logs for `repository importer finished` message. The `virtual_storages` field contains
+   the names of virtual storages and whether they've had any missing database records created.
+
+   For example, the `default` virtual storage has been successfully migrated:
+
+   ```json
+   {"level":"info","msg":"repository importer finished","pid":19752,"time":"2021-04-28T11:41:36.743Z","virtual_storages":{"default":true}}
+   ```
+
+   If a virtual storage has not been successfully migrated, it would have `false` next to it:
+
+   ```json
+   {"level":"info","msg":"repository importer finished","pid":19752,"time":"2021-04-28T11:41:36.743Z","virtual_storages":{"default":false}}
+   ```
+
+   The migration is ran when Praefect starts up. If the migration is unsuccessful, you can restart
+   a Praefect node to reattempt it. The migration only runs with `sql` election strategy configured.
+
+1. Running two different election strategies side by side can cause a split brain, where different Praefects
+   consider repositories to have different primaries. To avoid this, all Praefects should be shut down prior
+   to changing the election strategy.
+
+   This can be done by running `gitlab-ctl stop praefect` on the Praefect nodes.
+
+1. On the Praefect nodes, configure the election strategy in `/etc/gitlab/gitlab.rb` with
+   `praefect['failover_election_strategy'] = 'per_repository'`.
+
+1. Finally, run `gitlab-ctl reconfigure` to reconfigure and restart the Praefects.
+
+#### Deprecated election strategies
+
+WARNING:
+The below election strategies are deprecated and are scheduled for removal in GitLab 14.0.
+Migrate to [repository-specific primary nodes](#repository-specific-primary-nodes).
+
+- **PostgreSQL:** Enabled by default until GitLab 14.0, and equivalent to:
   `praefect['failover_election_strategy'] = 'sql'`. This configuration
   option allows multiple Praefect nodes to coordinate via the
   PostgreSQL database to elect a primary Gitaly node. This configuration
@@ -1080,8 +1170,6 @@ current primary node is found to be unhealthy.
   be elected. **Do not use with multiple Praefect nodes!** Using with multiple
   Praefect nodes is likely to result in a split brain.
 
-We are likely to implement support for Consul, and a cloud native, strategy in the future.
-
 ## Primary Node Failure
 
 Gitaly Cluster recovers from a failing primary Gitaly node by promoting a healthy secondary as the
diff --git a/qa/qa/specs/features/browser_ui/4_verify/ci_variable/pipeline_with_protected_variable_spec.rb b/qa/qa/specs/features/browser_ui/4_verify/ci_variable/pipeline_with_protected_variable_spec.rb
index 5b976ae4126..4938ae3f969 100644
--- a/qa/qa/specs/features/browser_ui/4_verify/ci_variable/pipeline_with_protected_variable_spec.rb
+++ b/qa/qa/specs/features/browser_ui/4_verify/ci_variable/pipeline_with_protected_variable_spec.rb
@@ -115,7 +115,7 @@ module QA
           commit.add_files(
             [
               {
-                file_path: "#{Faker::Lorem.word}.txt",
+                file_path: Faker::File.unique.file_name,
                 content: Faker::Lorem.sentence
               }
             ]
@@ -129,7 +129,7 @@ module QA
           merge_request.project = project
           merge_request.description = Faker::Lorem.sentence
           merge_request.target_new_branch = false
-          merge_request.file_name = "#{Faker::Lorem.word}.txt"
+          merge_request.file_name = Faker::File.unique.file_name
           merge_request.file_content = Faker::Lorem.sentence
         end
       end
diff --git a/qa/qa/specs/features/browser_ui/5_package/rubygems_registry_spec.rb b/qa/qa/specs/features/browser_ui/5_package/rubygems_registry_spec.rb
index ccc82ce7e23..5d15885cd67 100644
--- a/qa/qa/specs/features/browser_ui/5_package/rubygems_registry_spec.rb
+++ b/qa/qa/specs/features/browser_ui/5_package/rubygems_registry_spec.rb
@@ -1,7 +1,7 @@
 # frozen_string_literal: true
 
 module QA
-  RSpec.describe 'Package', :orchestrated, :packages do
+  RSpec.describe 'Package', :orchestrated, :requires_admin, :packages do
     describe 'RubyGems Repository' do
       include Runtime::Fixtures