Add latest changes from gitlab-org/gitlab@master

10 files changed, 148 insertions, 13 deletions

diff --git a/doc/development/cached_queries.md b/doc/development/cached_queries.md
index fbb857106be..e4625a50d79 100644
--- a/doc/development/cached_queries.md
+++ b/doc/development/cached_queries.md
@@ -87,7 +87,7 @@ and the number of executed cached queries:
 
 ![Performance Bar Database Queries](img/performance_bar_members_page.png)
 
-The page included 55 cached queries. Clicking the number displays a modal window
+The page included 55 cached queries. Selecting the number displays a modal window
 with more details about queries. Cached queries are marked with the `cached` label
 below the query. You can see multiple duplicate cached queries in this modal window:
 
diff --git a/doc/development/database/batched_background_migrations.md b/doc/development/database/batched_background_migrations.md
index a48a9c42e27..ca11e9c8dd3 100644
--- a/doc/development/database/batched_background_migrations.md
+++ b/doc/development/database/batched_background_migrations.md
@@ -219,6 +219,7 @@ In this example, `copy_from` returns `name`, and `copy_to` returns `name_convert
 ```ruby
 class CopyColumnUsingBackgroundMigrationJob < BatchedMigrationJob
   job_arguments :copy_from, :copy_to
+  operation_name :update_all
 
   def perform
     from_column = connection.quote_column_name(copy_from)
@@ -226,7 +227,7 @@ class CopyColumnUsingBackgroundMigrationJob < BatchedMigrationJob
 
     assignment_clause = "#{to_column} = #{from_column}"
 
-    each_sub_batch(operation_name: :update_all) do |relation|
+    each_sub_batch do |relation|
       relation.update_all(assignment_clause)
     end
   end
@@ -267,9 +268,10 @@ In the second (filtered) example, we know exactly 100 will be updated with each
    ```ruby
    class BackfillNamespaceType < BatchedMigrationJob
      scope_to ->(relation) { relation.where(type: nil) }
+     operation_name :update_all
 
      def perform
-       each_sub_batch(operation_name: :update_all) do |sub_batch|
+       each_sub_batch do |sub_batch|
          sub_batch.update_all(type: 'User')
        end
      end
@@ -327,9 +329,10 @@ background migration.
      #   self.table_name = 'routes'
      # end
 
+     operation_name :update_all
+
      def perform
        each_sub_batch(
-         operation_name: :update_all,
          batching_scope: -> (relation) { relation.where("source_type <> 'UnusedType'") }
        ) do |sub_batch|
          sub_batch.update_all('namespace_id = source_id')
@@ -483,8 +486,10 @@ module Gitlab
     class BatchProjectsWithIssues < Gitlab::BackgroundMigration::BatchedMigrationJob
       include Gitlab::Database::DynamicModelHelpers
 
+      operation_name :backfill_issues
+
       def perform
-        distinct_each_batch(operation_name: :backfill_issues) do |batch|
+        distinct_each_batch do |batch|
           project_ids = batch.pluck(batch_column)
           # do something with the distinct project_ids
         end
diff --git a/doc/development/database_review.md b/doc/development/database_review.md
index 58776c5330c..2af5cf3691f 100644
--- a/doc/development/database_review.md
+++ b/doc/development/database_review.md
@@ -176,7 +176,7 @@ Include in the MR description:
 - The query plan for each raw SQL query included in the merge request along with the link to the query plan following each raw SQL snippet.
 - Provide a public link to the plan from either:
   - [postgres.ai](https://postgres.ai/): Follow the link in `#database-lab` and generate a shareable, public link
-    by clicking **Share** in the upper right corner.
+    by selecting **Share** in the upper right corner.
   - [explain.depesz.com](https://explain.depesz.com) or [explain.dalibo.com](https://explain.dalibo.com): Paste both the plan and the query used in the form.
 - When providing query plans, make sure it hits enough data:
   - You can use a GitLab production replica to test your queries on a large scale,
diff --git a/doc/development/distributed_tracing.md b/doc/development/distributed_tracing.md
index 0ca10bc93dd..9e62f019fbf 100644
--- a/doc/development/distributed_tracing.md
+++ b/doc/development/distributed_tracing.md
@@ -91,7 +91,7 @@ Once the performance bar is enabled, select **Trace** in the performance bar to
 the Jaeger UI.
 
 The Jaeger search UI returns a query for the `Correlation-ID` of the current request. Normally,
-this search should return a single trace result. Clicking this result shows the detail of the
+this search should return a single trace result. Selecting this result shows the detail of the
 trace in a hierarchical time-line.
 
 ![Jaeger Search UI](img/distributed_tracing_jaeger_ui.png)
diff --git a/doc/development/multi_version_compatibility.md b/doc/development/multi_version_compatibility.md
index 808554d6027..8e2103b2f3e 100644
--- a/doc/development/multi_version_compatibility.md
+++ b/doc/development/multi_version_compatibility.md
@@ -258,7 +258,7 @@ For more information, see [the relevant issue](https://gitlab.com/gitlab-org/git
 
 We bumped the Markdown cache version and found a bug when a user edited a description or comment which was generated from a different Markdown
 cache version. The cached HTML wasn't generated properly after saving. In most cases, this wouldn't have happened because users would have
-viewed the Markdown before clicking **Edit** and that would mean the Markdown cache is refreshed. But because we run mixed versions, this is
+viewed the Markdown before selecting **Edit** and that would mean the Markdown cache is refreshed. But because we run mixed versions, this is
 more likely to happen. Another user on a different version could view the same page and refresh the cache to the other version behind the scenes.
 
 For more information, see [the relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/208255).
diff --git a/doc/development/packages/cleanup_policies.md b/doc/development/packages/cleanup_policies.md
new file mode 100644
index 00000000000..ae6304e940d
--- /dev/null
+++ b/doc/development/packages/cleanup_policies.md
@@ -0,0 +1,122 @@
+---
+stage: Package
+group: Package
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Cleanup policies
+
+Cleanup policies are recurrent background processes that automatically remove
+objects according to some parameters set by users.
+
+## Container Registry
+
+Cleanup policies for the container registry work on all the container repositories
+hosted in a single project. All tags that match the cleanup parameters are removed.
+
+### Parameters
+
+The [ContainerExpirationPolicy](https://gitlab.com/gitlab-org/gitlab/-/blob/37a76cbfb54a9a3f0dba3c3748eaaac82fb8bf4b/app/models/container_expiration_policy.rb)
+holds all parameters for the container registry cleanup policies.
+
+The parameters are split into two groups:
+
+- The parameters that define tags to keep:
+  - `keep_n`. Keep the `n` most recent tags.
+  - `name_regex_keep`. Keep tags matching this regular expression.
+- The parameters that define tags to destroy:
+  - `older_than`. Destroy tags older than this timestamp.
+  - `name_regex`. Destroy tags matching this regular expression.
+  
+The remaining parameters impact when the policy is executed:
+
+- `enabled`. Defines if the policy is enabled or not.
+- `cadence`. Defines the execution cadence of the policy.
+- `next_run_at`. Defines when the next execution should happen.
+
+### Execution
+
+Due to the large number of policies we need to process on GitLab.com, the execution
+follows this design.
+
+- Policy executions are limited in time.
+- Policy executions are either complete or partial.
+- The background jobs will consider the next job to be executed based on two
+priorities:
+  - Policy with a `next_run_at` in the past.
+  - Partially executed policies.
+
+To track the cleanup policy status on a container repository,
+we have an `expiration_policy_cleanup_status` on the `ContainerRepository`
+model.
+
+Background jobs for this execution are organized on:
+
+- A cron background job that runs every hour.
+- A set of background jobs that will loop on container repositories that need
+a policy execution.
+
+#### The cron background job
+
+The [cron background job](https://gitlab.com/gitlab-org/gitlab/-/blob/36454d77a8de76a25896efd7c051d6796985f579/app/workers/container_expiration_policy_worker.rb)
+is quite simple.
+Its main tasks are:
+
+1. Check if there are any container repositories in need of a cleanup. If any,
+enqueue as many limited capacity jobs as necessary, up to a limit.
+1. Compute metrics for cleanup policies and log them.
+
+#### The limited capacity job
+
+This [job](https://gitlab.com/gitlab-org/gitlab/-/blob/36454d77a8de76a25896efd7c051d6796985f579/app/workers/container_expiration_policies/cleanup_container_repository_worker.rb)
+is based on the [limited capacity concern](../sidekiq/limited_capacity_worker.md).
+
+This job will run in parallel up to [a specific capacity](settings.md#container-registry).
+
+The primary responsibility of this job is to select the next container
+repository that requires cleaning and call the related service on it.
+
+This is where the two priorities are evaluated in order. If a container repository
+is found, the cleanup service is called on it.
+
+To ensure that only one cleaning is executed on a given container repository
+at any time, we use a database lock along with the
+`expiration_policy_cleanup_status` column.
+
+This job will re-enqueue itself until no more container repositories require cleanup.
+
+#### Services
+
+Here is the services call that will happen from the limited capacity job:
+
+```mermaid
+flowchart TD
+    job[Limited capacity job] --> cleanup([ContainerExpirationPolicies::CleanupService])
+    cleanup --> cleanup_tags([Projects::ContainerRepository::CleanupTagsService])
+    cleanup_tags --> delete_tags([Projects::ContainerRepository::DeleteTagsService])
+```
+
+- [`ContainerExpirationPolicies::CleanupService`](https://gitlab.com/gitlab-org/gitlab/-/blob/6546ffc6fe4e9b447a1b7f050edddb8926fe4a3d/app/services/container_expiration_policies/cleanup_service.rb).
+This service mainly deals with container repository `expiration_policy_cleanup_status`
+updates and will call the cleanup tags service.
+- [`Projects::ContainerRepository::CleanupTagsService`](https://gitlab.com/gitlab-org/gitlab/-/blob/f23d70b7d638c38d71af102cfd32a3f6751596f9/app/services/projects/container_repository/cleanup_tags_service.rb).
+This service receives the policy parameters and builds the list of tags to
+destroy on the container registry.
+- [`Projects::ContainerRepository::DeleteTagsService`](https://gitlab.com/gitlab-org/gitlab/-/blob/f23d70b7d638c38d71af102cfd32a3f6751596f9/app/services/projects/container_repository/delete_tags_service.rb).
+This service receives a list of tags and loops on that list. For each tag,
+the service will call the container registry API endpoint to destroy the target tag.
+
+The cleanup tags service uses a very specific [execution order](../../user/packages/container_registry/reduce_container_registry_storage.md#how-the-cleanup-policy-works)
+to build the list of tags to destroy.
+
+Lastly, the cleanup tags service and delete tags service work using facades.
+The actual implementation depends on the type of container registry connected.
+If the GitLab container registry is connected, several improvements are available
+and used during cleanup policies execution, such as [better use of the container registry API](https://gitlab.com/groups/gitlab-org/-/epics/8379).
+
+### Historic reference links
+
+- [First iteration](https://gitlab.com/gitlab-org/gitlab/-/issues/15398)
+- [Throttling policy executions](https://gitlab.com/gitlab-org/gitlab/-/issues/208193)
+- [Adding caching](https://gitlab.com/gitlab-org/gitlab/-/issues/339129)
+- [Further improvements](https://gitlab.com/groups/gitlab-org/-/epics/8379)
diff --git a/doc/development/packages/index.md b/doc/development/packages/index.md
index 41d6e1b84fd..d587397adc7 100644
--- a/doc/development/packages/index.md
+++ b/doc/development/packages/index.md
@@ -10,6 +10,7 @@ Development and Architectural documentation for the package registry
 
 - [Debian repository structure](debian_repository.md)
 - [Dependency proxy structure](dependency_proxy.md)
+- [Cleanup policies](cleanup_policies.md)
 - [Developing a new format](new_format_development.md)
 - [Settings](settings.md)
 - [Structure / Schema](structure.md)
diff --git a/doc/development/secure_coding_guidelines.md b/doc/development/secure_coding_guidelines.md
index 67f7c556055..59013c89687 100644
--- a/doc/development/secure_coding_guidelines.md
+++ b/doc/development/secure_coding_guidelines.md
@@ -1218,7 +1218,7 @@ GitLab-specific example can be found in [this issue](https://gitlab.com/gitlab-o
 
 **Example 3:** you need to fetch a remote file, and perform a `HEAD` request to get and validate the content length and content type. When you subsequently make a `GET` request, though, the file delivered is a different size or different file type. (This is stretching the definition of TOCTOU, but things _have_ changed between time of check and time of use).
 
-**Example 4:** you allow users to upvote a comment if they haven't already. The server is multi-threaded, and you aren't using transactions or an applicable database index. By repeatedly clicking upvote in quick succession a malicious user is able to add multiple upvotes: the requests arrive at the same time, the checks run in parallel and confirm that no upvote exists yet, and so each upvote is written to the database.
+**Example 4:** you allow users to upvote a comment if they haven't already. The server is multi-threaded, and you aren't using transactions or an applicable database index. By repeatedly selecting upvote in quick succession a malicious user is able to add multiple upvotes: the requests arrive at the same time, the checks run in parallel and confirm that no upvote exists yet, and so each upvote is written to the database.
 
 Here's some pseudocode showing an example of a potential TOCTOU bug:
 
diff --git a/doc/development/service_ping/metrics_instrumentation.md b/doc/development/service_ping/metrics_instrumentation.md
index 7a3f291460c..4b00d1fc56d 100644
--- a/doc/development/service_ping/metrics_instrumentation.md
+++ b/doc/development/service_ping/metrics_instrumentation.md
@@ -256,6 +256,13 @@ options:
 
 ## Aggregated metrics
 
+<div class="video-fallback">
+  See the video from: <a href="https://www.youtube.com/embed/22LbYqHwtUQ">Product Intelligence Office Hours Oct 6th</a> for an aggregated metrics walk-through.
+</div>
+<figure class="video-container">
+  <iframe src="https://www.youtube.com/embed/22LbYqHwtUQ" frameborder="0" allowfullscreen="true"> </iframe>
+</figure>
+
 The aggregated metrics feature provides insight into the number of data attributes, for example `pseudonymized_user_ids`, that occurred in a collection of events. For example, you can aggregate the number of users who perform multiple actions such as creating a new issue and opening
 a new merge request.
 
diff --git a/doc/development/testing_guide/end_to_end/style_guide.md b/doc/development/testing_guide/end_to_end/style_guide.md
index f380b24d7a5..419942d6b8f 100644
--- a/doc/development/testing_guide/end_to_end/style_guide.md
+++ b/doc/development/testing_guide/end_to_end/style_guide.md
@@ -12,7 +12,7 @@ This document describes the conventions used at GitLab for writing End-to-end (E
 
 ### When to use `click_`?
 
-When clicking in a single link to navigate, use `click_`.
+When selecting a single link to navigate, use `click_`.
 
 For example:
 
@@ -24,9 +24,9 @@ def click_ci_cd_pipelines
 end
 ```
 
-From a testing perspective, if we want to check that clicking a link, or a button (a single interaction) is working as intended, we would want the test to read as:
+From a testing perspective, if we want to check that selecting a link, or a button (a single interaction) is working as intended, we would want the test to read as:
 
-- Click a certain element
+- Select a certain element
 - Verify the action took place
 
 ### When to use `go_to_`?
@@ -47,7 +47,7 @@ end
 
 `go_to_` fits the definition of interacting with multiple elements very well given it's more of a meta-navigation action that includes multiple interactions.
 
-Notice that in the above example, before clicking the `:operations_environments_link`, another element is hovered over.
+Notice that in the above example, before selecting the `:operations_environments_link`, another element is hovered over.
 
 > We can create these methods as helpers to abstract multi-step navigation.