Add latest changes from gitlab-org/gitlab@master

9 files changed, 475 insertions, 460 deletions

diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md
index 05f9ec264f7..cab3ef581cb 100644
--- a/doc/administration/geo/replication/datatypes.md
+++ b/doc/administration/geo/replication/datatypes.md
@@ -189,7 +189,7 @@ successfully, you must replicate their data using some other means.
 |Feature                                                                                                        | Replicated (added in GitLab version)                                    | Verified (added in GitLab version)                                         | GitLab-managed object storage replication (added in GitLab version)                | GitLab-managed object storage verification (added in GitLab version)           | Notes |
 |:--------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------|:---------------------------------------------------------------------------|:--------------------------------------------------------------------|:----------------------------------------------------------------|:------|
 |[Application data in PostgreSQL](../../postgresql/index.md)                                                    | **Yes** (10.2)                                                          | **Yes** (10.2)                                                             | N/A                                                                 | N/A                                                             |       |
-|[Project repository](../../../user/project/repository/index.md)                                                        | **Yes** (10.2)                                                          | **Yes** (10.7)                                                             | N/A                                                                 | N/A                                                             | Migrated to [self-service framework](../../../development/geo/framework.md) in 16.2. See GitLab issue [#367925](https://gitlab.com/gitlab-org/gitlab/-/issues/367925) for more details.<br /><br />Behind feature flag `geo_project_repository_replication`, enabled by default in (16.3).     |
+|[Project repository](../../../user/project/repository/index.md)                                                        | **Yes** (10.2)                                                          | **Yes** (10.7)                                                             | N/A                                                                 | N/A                                                             | Migrated to [self-service framework](../../../development/geo/framework.md) in 16.2. See GitLab issue [#367925](https://gitlab.com/gitlab-org/gitlab/-/issues/367925) for more details.<br /><br />Behind feature flag `geo_project_repository_replication`, enabled by default in (16.3).<br /><br /> All projects, including [archived projects](../../../user/project/settings/index.md#archive-a-project), are replicated. |
 |[Project wiki repository](../../../user/project/wiki/index.md)                                                         | **Yes** (10.2)<sup>2</sup>                                                          | **Yes** (10.7)<sup>2</sup>                                                             | N/A                                                                 | N/A                                                             | Migrated to [self-service framework](../../../development/geo/framework.md) in 15.11. See GitLab issue [#367925](https://gitlab.com/gitlab-org/gitlab/-/issues/367925) for more details.<br /><br />Behind feature flag `geo_project_wiki_repository_replication`, enabled by default in (15.11). |
 |[Group wiki repository](../../../user/project/wiki/group.md)                                                   | [**Yes** (13.10)](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | [**Yes** (16.3)](https://gitlab.com/gitlab-org/gitlab/-/issues/323897)                                                                         | N/A                                                                 | N/A                                                             | Behind feature flag `geo_group_wiki_repository_replication`, enabled by default. |
 |[Uploads](../../uploads.md)                                                                                    | **Yes** (10.2)                                                          | **Yes** (14.6)                                                             | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Replication is behind the feature flag `geo_upload_replication`, enabled by default. Verification was behind the feature flag `geo_upload_verification`, removed in 14.8. |
diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md
index d6f1b39d852..b3778e89b19 100644
--- a/doc/administration/job_artifacts.md
+++ b/doc/administration/job_artifacts.md
@@ -399,456 +399,3 @@ an archive is a very large file.
 When selecting a specific file, [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) extracts it
 from the archive and the download begins. This implementation saves space,
 memory and disk I/O.
-
-## Troubleshooting
-
-### Job artifacts using too much disk space
-
-Job artifacts can fill up your disk space quicker than expected. Some possible
-reasons are:
-
-- Users have configured job artifacts expiration to be longer than necessary.
-- The number of jobs run, and hence artifacts generated, is higher than expected.
-- Job logs are larger than expected, and have accumulated over time.
-- The file system might run out of inodes because
-  [empty directories are left behind by artifact housekeeping](https://gitlab.com/gitlab-org/gitlab/-/issues/17465).
-  [The Rake task for _orphaned_ artifact files](../raketasks/cleanup.md#remove-orphan-artifact-files)
-  removes these.
-- Artifact files might be left on disk and not deleted by housekeeping. Run the
-  [Rake task for _orphaned_ artifact files](../raketasks/cleanup.md#remove-orphan-artifact-files)
-  to remove these. This script should always find work to do, as it also removes empty directories (see above).
-- [Artifact housekeeping was changed significantly](#housekeeping-disabled-in-gitlab-146-to-152),
-  and you might need to enable a feature flag to used the updated system.
-- The [keep latest artifacts from most recent success jobs](../ci/jobs/job_artifacts.md#keep-artifacts-from-most-recent-successful-jobs)
-  feature is enabled.
-
-In these and other cases, identify the projects most responsible
-for disk space usage, figure out what types of artifacts are using the most
-space, and in some cases, manually delete job artifacts to reclaim disk space.
-
-#### Artifacts housekeeping
-
-Artifacts housekeeping is the process that identifies which artifacts are expired
-and can be deleted.
-
-##### Housekeeping disabled in GitLab 14.6 to 15.2
-
-Artifact housekeeping was disabled in GitLab 14.6. It was significantly improved
-in GitLab 14.10, and the changes were back ported to patch versions of GitLab 14.6 and later,
-introduced behind [feature flags](feature_flags.md) disabled by default. The flags were
-enabled by default [in GitLab 15.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/92931).
-
-If artifacts housekeeping does not seem to be working in GitLab 14.6 to GitLab 15.2,
-you should check if the feature flags are enabled.
-
-To check if the feature flags are enabled:
-
-1. Start a [Rails console](operations/rails_console.md#starting-a-rails-console-session).
-
-1. Check if the feature flags are enabled.
-
-   - GitLab 14.10 and earlier:
-
-     ```ruby
-     Feature.enabled?(:ci_detect_wrongly_expired_artifacts, default_enabled: :yaml)
-     Feature.enabled?(:ci_update_unlocked_job_artifacts, default_enabled: :yaml)
-     Feature.enabled?(:ci_job_artifacts_backlog_work, default_enabled: :yaml)
-     ```
-
-   - GitLab 15.0 and later:
-
-     ```ruby
-     Feature.enabled?(:ci_detect_wrongly_expired_artifacts)
-     Feature.enabled?(:ci_update_unlocked_job_artifacts)
-     Feature.enabled?(:ci_job_artifacts_backlog_work)
-     ```
-
-1. If any of the feature flags are disabled, enable them:
-
-   ```ruby
-   Feature.enable(:ci_detect_wrongly_expired_artifacts)
-   Feature.enable(:ci_update_unlocked_job_artifacts)
-   Feature.enable(:ci_destroy_unlocked_job_artifacts)
-   ```
-
-These changes include switching artifacts from `unlocked` to `locked` if
-they [should be retained](../ci/jobs/job_artifacts.md#keep-artifacts-from-most-recent-successful-jobs).
-
-##### Artifacts with `unknown` status
-
-Artifacts created before housekeeping was updated have a status of `unknown`. After they expire,
-these artifacts are not processed by the new housekeeping.
-
-You can check the database to confirm if your instance has artifacts with the `unknown` status:
-
-1. Start a database console:
-
-   ::Tabs
-
-   :::TabTitle Linux package (Omnibus)
-
-   ```shell
-   sudo gitlab-psql
-   ```
-
-   :::TabTitle Helm chart (Kubernetes)
-
-   ```shell
-   # Find the toolbox pod
-   kubectl --namespace <namespace> get pods -lapp=toolbox
-   # Connect to the PostgreSQL console
-   kubectl exec -it <toolbox-pod-name> -- /srv/gitlab/bin/rails dbconsole --include-password --database main
-   ```
-
-   :::TabTitle Docker
-
-   ```shell
-   sudo docker exec -it <container_name> /bin/bash
-   gitlab-psql
-   ```
-
-   :::TabTitle Self-compiled (source)
-
-   ```shell
-   sudo -u git -H psql -d gitlabhq_production
-   ```
-
-   ::EndTabs
-
-1. Run the following query:
-
-   ```sql
-   select expire_at, file_type, locked, count(*) from ci_job_artifacts
-   where expire_at is not null and
-   file_type != 3
-   group by expire_at, file_type, locked having count(*) > 1;
-   ```
-
-If records are returned, then there are artifacts which the housekeeping job
-is unable to process. For example:
-
-```plaintext
-           expire_at           | file_type | locked | count
--------------------------------+-----------+--------+--------
- 2021-06-21 22:00:00+00        |         1 |      2 |  73614
- 2021-06-21 22:00:00+00        |         2 |      2 |  73614
- 2021-06-21 22:00:00+00        |         4 |      2 |   3522
- 2021-06-21 22:00:00+00        |         9 |      2 |     32
- 2021-06-21 22:00:00+00        |        12 |      2 |    163
-```
-
-Artifacts with locked status `2` are `unknown`. Check
-[issue #346261](https://gitlab.com/gitlab-org/gitlab/-/issues/346261#note_1028871458)
-for more details.
-
-##### Clean up `unknown` artifacts
-
-The Sidekiq worker that processes all `unknown` artifacts is enabled by default in
-GitLab 15.3 and later. It analyzes the artifacts returned by the above database query and
-determines which should be `locked` or `unlocked`. Artifacts are then deleted
-by that worker if needed.
-
-The worker can be enabled on self-managed instances running GitLab 14.10 and later:
-
-1. Start a [Rails console](operations/rails_console.md#starting-a-rails-console-session).
-
-1. Check if the feature is enabled.
-
-   - GitLab 14.10:
-
-     ```ruby
-     Feature.enabled?(:ci_job_artifacts_backlog_work, default_enabled: :yaml)
-     ```
-
-   - GitLab 15.0 and later:
-
-     ```ruby
-     Feature.enabled?(:ci_job_artifacts_backlog_work)
-     ```
-
-1. Enable the feature, if needed:
-
-   ```ruby
-   Feature.enable(:ci_job_artifacts_backlog_work)
-   ```
-
-The worker processes 10,000 `unknown` artifacts every seven minutes, or roughly two million
-in 24 hours.
-
-There is a related `ci_job_artifacts_backlog_large_loop_limit` feature flag
-which causes the worker to process `unknown` artifacts
-[in batches that are five times larger](https://gitlab.com/gitlab-org/gitlab/-/issues/356319).
-This flag is not recommended for use on self-managed instances.
-
-#### List projects and builds with artifacts with a specific expiration (or no expiration)
-
-Using a [Rails console](operations/rails_console.md), you can find projects that have job artifacts with either:
-
-- No expiration date.
-- An expiration date more than 7 days in the future.
-
-Similar to [deleting artifacts](#delete-job-artifacts-from-jobs-completed-before-a-specific-date), use the following example time frames
-and alter them as needed:
-
-- `7.days.from_now`
-- `10.days.from_now`
-- `2.weeks.from_now`
-- `3.months.from_now`
-
-Each of the following scripts also limits the search to 50 results with `.limit(50)`, but this number can also be changed as needed:
-
-```ruby
-# Find builds & projects with artifacts that never expire
-builds_with_artifacts_that_never_expire = Ci::Build.with_downloadable_artifacts.where(artifacts_expire_at: nil).limit(50)
-builds_with_artifacts_that_never_expire.find_each do |build|
-  puts "Build with id #{build.id} has artifacts that don't expire and belongs to project #{build.project.full_path}"
-end
-
-# Find builds & projects with artifacts that expire after 7 days from today
-builds_with_artifacts_that_expire_in_a_week = Ci::Build.with_downloadable_artifacts.where('artifacts_expire_at > ?', 7.days.from_now).limit(50)
-builds_with_artifacts_that_expire_in_a_week.find_each do |build|
-  puts "Build with id #{build.id} has artifacts that expire at #{build.artifacts_expire_at} and belongs to project #{build.project.full_path}"
-end
-```
-
-#### List projects by total size of job artifacts stored
-
-List the top 20 projects, sorted by the total size of job artifacts stored, by
-running the following code in the Rails console (`sudo gitlab-rails console`):
-
-```ruby
-include ActionView::Helpers::NumberHelper
-ProjectStatistics.order(build_artifacts_size: :desc).limit(20).each do |s|
-  puts "#{number_to_human_size(s.build_artifacts_size)} \t #{s.project.full_path}"
-end
-```
-
-You can change the number of projects listed by modifying `.limit(20)` to the
-number you want.
-
-#### List largest artifacts in a single project
-
-List the 50 largest job artifacts in a single project by running the following
-code in the Rails console (`sudo gitlab-rails console`):
-
-```ruby
-include ActionView::Helpers::NumberHelper
-project = Project.find_by_full_path('path/to/project')
-Ci::JobArtifact.where(project: project).order(size: :desc).limit(50).map { |a| puts "ID: #{a.id} - #{a.file_type}: #{number_to_human_size(a.size)}" }
-```
-
-You can change the number of job artifacts listed by modifying `.limit(50)` to
-the number you want.
-
-#### List artifacts in a single project
-
-List the artifacts for a single project, sorted by artifact size. The output includes the:
-
-- ID of the job that created the artifact
-- artifact size
-- artifact file type
-- artifact creation date
-- on-disk location of the artifact
-
-```ruby
-p = Project.find_by_id(<project_id>)
-arts = Ci::JobArtifact.where(project: p)
-
-list = arts.order(size: :desc).limit(50).each do |art|
-    puts "Job ID: #{art.job_id} - Size: #{art.size}b - Type: #{art.file_type} - Created: #{art.created_at} - File loc: #{art.file}"
-end
-```
-
-To change the number of job artifacts listed, change the number in `limit(50)`.
-
-#### Delete job artifacts from jobs completed before a specific date
-
-WARNING:
-These commands remove data permanently from both the database and from disk. Before running them, we highly recommend seeking guidance from a Support Engineer, or running them in a test environment with a backup of the instance ready to be restored, just in case.
-
-If you need to manually remove job artifacts associated with multiple jobs while
-**retaining their job logs**, this can be done from the Rails console (`sudo gitlab-rails console`):
-
-1. Select jobs to be deleted:
-
-   To select all jobs with artifacts for a single project:
-
-   ```ruby
-   project = Project.find_by_full_path('path/to/project')
-   builds_with_artifacts =  project.builds.with_downloadable_artifacts
-   ```
-
-   To select all jobs with artifacts across the entire GitLab instance:
-
-   ```ruby
-   builds_with_artifacts = Ci::Build.with_downloadable_artifacts
-   ```
-
-1. Delete job artifacts older than a specific date:
-
-   NOTE:
-   This step also erases artifacts that users have chosen to
-   ["keep"](../ci/jobs/job_artifacts.md#download-job-artifacts).
-
-   ```ruby
-   builds_to_clear = builds_with_artifacts.where("finished_at < ?", 1.week.ago)
-   builds_to_clear.find_each do |build|
-     Ci::JobArtifacts::DeleteService.new(build).execute
-     build.update!(artifacts_expire_at: Time.now)
-   end
-   ```
-
-   In [GitLab 15.3 and earlier](https://gitlab.com/gitlab-org/gitlab/-/issues/372537), use the following instead:
-
-   ```ruby
-   builds_to_clear = builds_with_artifacts.where("finished_at < ?", 1.week.ago)
-   builds_to_clear.find_each do |build|
-     build.artifacts_expire_at = Time.now
-     build.erase_erasable_artifacts!
-   end
-   ```
-
-   `1.week.ago` is a Rails `ActiveSupport::Duration` method which calculates a new
-   date or time in the past. Other valid examples are:
-
-   - `7.days.ago`
-   - `3.months.ago`
-   - `1.year.ago`
-
-   `erase_erasable_artifacts!` is a synchronous method, and upon execution the artifacts are immediately removed;
-   they are not scheduled by a background queue.
-
-#### Delete job artifacts and logs from jobs completed before a specific date
-
-WARNING:
-These commands remove data permanently from both the database and from disk. Before running them, we highly recommend seeking guidance from a Support Engineer, or running them in a test environment with a backup of the instance ready to be restored, just in case.
-
-If you need to manually remove **all** job artifacts associated with multiple jobs,
-**including job logs**, this can be done from the Rails console (`sudo gitlab-rails console`):
-
-1. Select the jobs to be deleted:
-
-   To select jobs with artifacts for a single project:
-
-   ```ruby
-   project = Project.find_by_full_path('path/to/project')
-   builds_with_artifacts =  project.builds.with_downloadable_artifacts
-   ```
-
-   To select jobs with artifacts across the entire GitLab instance:
-
-   ```ruby
-   builds_with_artifacts = Ci::Build.with_downloadable_artifacts
-   ```
-
-   Occasionally, when choosing jobs with artifacts, there could be a risk of the process being terminated due to selecting a large number of rows. This can result in high memory usage and eventually lead to the process being killed due to an Out-of-Memory (OOM) error. To resolve this, you can run in small batches. The example below limits each batch to 1000.
-
-   To select jobs with artifacts for a single project:
-
-   ```ruby
-   project = Project.find_by_full_path('path/to/project')
-   builds_with_artifacts =  project.builds.with_downloadable_artifacts.find_each(batch_size: 1000)
-   ```
-
-   To select jobs with artifacts across the entire GitLab instance:
-
-   ```ruby
-   builds_with_artifacts = Ci::Build.with_downloadable_artifacts.find_each(batch_size: 1000)
-   ```
-
-1. Select the user which is mentioned in the web UI as erasing the job:
-
-   ```ruby
-   admin_user = User.find_by(username: 'username')
-   ```
-
-1. Erase the job artifacts and logs older than a specific date:
-
-   ```ruby
-   builds_to_clear = builds_with_artifacts.where("finished_at < ?", 1.week.ago)
-   builds_to_clear.find_each do |build|
-     print "Ci::Build ID #{build.id}... "
-
-     if build.erasable?
-       Ci::BuildEraseService.new(build, admin_user).execute
-       puts "Erased"
-     else
-       puts "Skipped (Nothing to erase or not erasable)"
-     end
-   end
-   ```
-
-   In [GitLab 15.3 and earlier](https://gitlab.com/gitlab-org/gitlab/-/issues/369132), replace
-   `Ci::BuildEraseService.new(build, admin_user).execute` with `build.erase(erased_by: admin_user)`.
-
-   `1.week.ago` is a Rails `ActiveSupport::Duration` method which calculates a new
-   date or time in the past. Other valid examples are:
-
-   - `7.days.ago`
-   - `3.months.ago`
-   - `1.year.ago`
-
-### Job artifact upload fails with error 500
-
-If you are using object storage for artifacts and a job artifact fails to upload,
-review:
-
-- The job log for an error message similar to:
-
-  ```plaintext
-  WARNING: Uploading artifacts as "archive" to coordinator... failed id=12345 responseStatus=500 Internal Server Error status=500 token=abcd1234
-  ```
-
-- The [workhorse log](logs/index.md#workhorse-logs) for an error message similar to:
-
-  ```json
-  {"error":"MissingRegion: could not find region configuration","level":"error","msg":"error uploading S3 session","time":"2021-03-16T22:10:55-04:00"}
-  ```
-
-In both cases, you might need to add `region` to the job artifact [object storage configuration](object_storage.md).
-
-### Job artifact upload fails with `500 Internal Server Error (Missing file)`
-
-Bucket names that include folder paths are not supported with [consolidated object storage](object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form).
-For example, `bucket/path`. If a bucket name has a path in it, you might receive an error similar to:
-
-```plaintext
-WARNING: Uploading artifacts as "archive" to coordinator... POST https://gitlab.example.com/api/v4/jobs/job_id/artifacts?artifact_format=zip&artifact_type=archive&expire_in=1+day: 500 Internal Server Error (Missing file)
-FATAL: invalid argument
-```
-
-If a job artifact fails to upload with the above error when using consolidated object storage, make sure you are [using separate buckets](object_storage.md#use-separate-buckets) for each data type.
-
-### Job artifacts fail to upload with `FATAL: invalid argument` when using Windows mount
-
-If you are using a Windows mount with CIFS for job artifacts, you may see an
-`invalid argument` error when the runner attempts to upload artifacts:
-
-```plaintext
-WARNING: Uploading artifacts as "dotenv" to coordinator... POST https://<your-gitlab-instance>/api/v4/jobs/<JOB_ID>/artifacts: 500 Internal Server Error  id=1296 responseStatus=500 Internal Server Error status=500 token=*****
-FATAL: invalid argument
-```
-
-To work around this issue, you can try:
-
-- Switching to an ext4 mount instead of CIFS.
-- Upgrading to at least Linux kernel 5.15 which contains a number of important bug fixes
-  relating to CIFS file leases.
-- For older kernels, using the `nolease` mount option to disable file leasing.
-
-For more information, [see the investigation details](https://gitlab.com/gitlab-org/gitlab/-/issues/389995).
-
-### Usage quota shows incorrect artifact storage usage
-
-> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/238536) in GitLab 14.10.
-
-Sometimes the [artifacts storage usage](../user/usage_quotas.md) displays an incorrect
-value for the total storage space used by artifacts. To recalculate the artifact
-usage statistics for all projects in the instance, you can run this background script:
-
-```shell
-bin/rake 'gitlab:refresh_project_statistics_build_artifacts_size[file.csv]'
-```
-
-The artifact usage value can fluctuate to `0` while the script is running. After
-recalculation, usage should display as expected again.
diff --git a/doc/administration/job_artifacts_troubleshooting.md b/doc/administration/job_artifacts_troubleshooting.md
new file mode 100644
index 00000000000..ba60cbd7aba
--- /dev/null
+++ b/doc/administration/job_artifacts_troubleshooting.md
@@ -0,0 +1,460 @@
+---
+stage: Verify
+group: Pipeline Security
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Job artifact troubleshooting for administrators
+
+When administering job artifacts, you might encounter the following issues.
+
+## Job artifacts using too much disk space
+
+Job artifacts can fill up your disk space quicker than expected. Some possible
+reasons are:
+
+- Users have configured job artifacts expiration to be longer than necessary.
+- The number of jobs run, and hence artifacts generated, is higher than expected.
+- Job logs are larger than expected, and have accumulated over time.
+- The file system might run out of inodes because
+  [empty directories are left behind by artifact housekeeping](https://gitlab.com/gitlab-org/gitlab/-/issues/17465).
+  [The Rake task for _orphaned_ artifact files](../raketasks/cleanup.md#remove-orphan-artifact-files)
+  removes these.
+- Artifact files might be left on disk and not deleted by housekeeping. Run the
+  [Rake task for _orphaned_ artifact files](../raketasks/cleanup.md#remove-orphan-artifact-files)
+  to remove these. This script should always find work to do, as it also removes empty directories (see above).
+- [Artifact housekeeping was changed significantly](#housekeeping-disabled-in-gitlab-146-to-152),
+  and you might need to enable a feature flag to used the updated system.
+- The [keep latest artifacts from most recent success jobs](../ci/jobs/job_artifacts.md#keep-artifacts-from-most-recent-successful-jobs)
+  feature is enabled.
+
+In these and other cases, identify the projects most responsible
+for disk space usage, figure out what types of artifacts are using the most
+space, and in some cases, manually delete job artifacts to reclaim disk space.
+
+### Artifacts housekeeping
+
+Artifacts housekeeping is the process that identifies which artifacts are expired
+and can be deleted.
+
+#### Housekeeping disabled in GitLab 14.6 to 15.2
+
+Artifact housekeeping was disabled in GitLab 14.6. It was significantly improved
+in GitLab 14.10, and the changes were back ported to patch versions of GitLab 14.6 and later,
+introduced behind [feature flags](feature_flags.md) disabled by default. The flags were
+enabled by default [in GitLab 15.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/92931).
+
+If artifacts housekeeping does not seem to be working in GitLab 14.6 to GitLab 15.2,
+you should check if the feature flags are enabled.
+
+To check if the feature flags are enabled:
+
+1. Start a [Rails console](operations/rails_console.md#starting-a-rails-console-session).
+
+1. Check if the feature flags are enabled.
+
+   - GitLab 14.10 and earlier:
+
+     ```ruby
+     Feature.enabled?(:ci_detect_wrongly_expired_artifacts, default_enabled: :yaml)
+     Feature.enabled?(:ci_update_unlocked_job_artifacts, default_enabled: :yaml)
+     Feature.enabled?(:ci_job_artifacts_backlog_work, default_enabled: :yaml)
+     ```
+
+   - GitLab 15.0 and later:
+
+     ```ruby
+     Feature.enabled?(:ci_detect_wrongly_expired_artifacts)
+     Feature.enabled?(:ci_update_unlocked_job_artifacts)
+     Feature.enabled?(:ci_job_artifacts_backlog_work)
+     ```
+
+1. If any of the feature flags are disabled, enable them:
+
+   ```ruby
+   Feature.enable(:ci_detect_wrongly_expired_artifacts)
+   Feature.enable(:ci_update_unlocked_job_artifacts)
+   Feature.enable(:ci_destroy_unlocked_job_artifacts)
+   ```
+
+These changes include switching artifacts from `unlocked` to `locked` if
+they [should be retained](../ci/jobs/job_artifacts.md#keep-artifacts-from-most-recent-successful-jobs).
+
+#### Artifacts with `unknown` status
+
+Artifacts created before housekeeping was updated have a status of `unknown`. After they expire,
+these artifacts are not processed by the new housekeeping.
+
+You can check the database to confirm if your instance has artifacts with the `unknown` status:
+
+1. Start a database console:
+
+   ::Tabs
+
+   :::TabTitle Linux package (Omnibus)
+
+   ```shell
+   sudo gitlab-psql
+   ```
+
+   :::TabTitle Helm chart (Kubernetes)
+
+   ```shell
+   # Find the toolbox pod
+   kubectl --namespace <namespace> get pods -lapp=toolbox
+   # Connect to the PostgreSQL console
+   kubectl exec -it <toolbox-pod-name> -- /srv/gitlab/bin/rails dbconsole --include-password --database main
+   ```
+
+   :::TabTitle Docker
+
+   ```shell
+   sudo docker exec -it <container_name> /bin/bash
+   gitlab-psql
+   ```
+
+   :::TabTitle Self-compiled (source)
+
+   ```shell
+   sudo -u git -H psql -d gitlabhq_production
+   ```
+
+   ::EndTabs
+
+1. Run the following query:
+
+   ```sql
+   select expire_at, file_type, locked, count(*) from ci_job_artifacts
+   where expire_at is not null and
+   file_type != 3
+   group by expire_at, file_type, locked having count(*) > 1;
+   ```
+
+If records are returned, then there are artifacts which the housekeeping job
+is unable to process. For example:
+
+```plaintext
+           expire_at           | file_type | locked | count
+-------------------------------+-----------+--------+--------
+ 2021-06-21 22:00:00+00        |         1 |      2 |  73614
+ 2021-06-21 22:00:00+00        |         2 |      2 |  73614
+ 2021-06-21 22:00:00+00        |         4 |      2 |   3522
+ 2021-06-21 22:00:00+00        |         9 |      2 |     32
+ 2021-06-21 22:00:00+00        |        12 |      2 |    163
+```
+
+Artifacts with locked status `2` are `unknown`. Check
+[issue #346261](https://gitlab.com/gitlab-org/gitlab/-/issues/346261#note_1028871458)
+for more details.
+
+#### Clean up `unknown` artifacts
+
+The Sidekiq worker that processes all `unknown` artifacts is enabled by default in
+GitLab 15.3 and later. It analyzes the artifacts returned by the above database query and
+determines which should be `locked` or `unlocked`. Artifacts are then deleted
+by that worker if needed.
+
+The worker can be enabled on self-managed instances running GitLab 14.10 and later:
+
+1. Start a [Rails console](operations/rails_console.md#starting-a-rails-console-session).
+
+1. Check if the feature is enabled.
+
+   - GitLab 14.10:
+
+     ```ruby
+     Feature.enabled?(:ci_job_artifacts_backlog_work, default_enabled: :yaml)
+     ```
+
+   - GitLab 15.0 and later:
+
+     ```ruby
+     Feature.enabled?(:ci_job_artifacts_backlog_work)
+     ```
+
+1. Enable the feature, if needed:
+
+   ```ruby
+   Feature.enable(:ci_job_artifacts_backlog_work)
+   ```
+
+The worker processes 10,000 `unknown` artifacts every seven minutes, or roughly two million
+in 24 hours.
+
+There is a related `ci_job_artifacts_backlog_large_loop_limit` feature flag
+which causes the worker to process `unknown` artifacts
+[in batches that are five times larger](https://gitlab.com/gitlab-org/gitlab/-/issues/356319).
+This flag is not recommended for use on self-managed instances.
+
+### List projects and builds with artifacts with a specific expiration (or no expiration)
+
+Using a [Rails console](operations/rails_console.md), you can find projects that have job artifacts with either:
+
+- No expiration date.
+- An expiration date more than 7 days in the future.
+
+Similar to [deleting artifacts](#delete-job-artifacts-from-jobs-completed-before-a-specific-date), use the following example time frames
+and alter them as needed:
+
+- `7.days.from_now`
+- `10.days.from_now`
+- `2.weeks.from_now`
+- `3.months.from_now`
+
+Each of the following scripts also limits the search to 50 results with `.limit(50)`, but this number can also be changed as needed:
+
+```ruby
+# Find builds & projects with artifacts that never expire
+builds_with_artifacts_that_never_expire = Ci::Build.with_downloadable_artifacts.where(artifacts_expire_at: nil).limit(50)
+builds_with_artifacts_that_never_expire.find_each do |build|
+  puts "Build with id #{build.id} has artifacts that don't expire and belongs to project #{build.project.full_path}"
+end
+
+# Find builds & projects with artifacts that expire after 7 days from today
+builds_with_artifacts_that_expire_in_a_week = Ci::Build.with_downloadable_artifacts.where('artifacts_expire_at > ?', 7.days.from_now).limit(50)
+builds_with_artifacts_that_expire_in_a_week.find_each do |build|
+  puts "Build with id #{build.id} has artifacts that expire at #{build.artifacts_expire_at} and belongs to project #{build.project.full_path}"
+end
+```
+
+### List projects by total size of job artifacts stored
+
+List the top 20 projects, sorted by the total size of job artifacts stored, by
+running the following code in the Rails console (`sudo gitlab-rails console`):
+
+```ruby
+include ActionView::Helpers::NumberHelper
+ProjectStatistics.order(build_artifacts_size: :desc).limit(20).each do |s|
+  puts "#{number_to_human_size(s.build_artifacts_size)} \t #{s.project.full_path}"
+end
+```
+
+You can change the number of projects listed by modifying `.limit(20)` to the
+number you want.
+
+### List largest artifacts in a single project
+
+List the 50 largest job artifacts in a single project by running the following
+code in the Rails console (`sudo gitlab-rails console`):
+
+```ruby
+include ActionView::Helpers::NumberHelper
+project = Project.find_by_full_path('path/to/project')
+Ci::JobArtifact.where(project: project).order(size: :desc).limit(50).map { |a| puts "ID: #{a.id} - #{a.file_type}: #{number_to_human_size(a.size)}" }
+```
+
+You can change the number of job artifacts listed by modifying `.limit(50)` to
+the number you want.
+
+### List artifacts in a single project
+
+List the artifacts for a single project, sorted by artifact size. The output includes the:
+
+- ID of the job that created the artifact
+- artifact size
+- artifact file type
+- artifact creation date
+- on-disk location of the artifact
+
+```ruby
+p = Project.find_by_id(<project_id>)
+arts = Ci::JobArtifact.where(project: p)
+
+list = arts.order(size: :desc).limit(50).each do |art|
+    puts "Job ID: #{art.job_id} - Size: #{art.size}b - Type: #{art.file_type} - Created: #{art.created_at} - File loc: #{art.file}"
+end
+```
+
+To change the number of job artifacts listed, change the number in `limit(50)`.
+
+### Delete job artifacts from jobs completed before a specific date
+
+WARNING:
+These commands remove data permanently from both the database and from disk. Before running them, we highly recommend seeking guidance from a Support Engineer, or running them in a test environment with a backup of the instance ready to be restored, just in case.
+
+If you need to manually remove job artifacts associated with multiple jobs while
+**retaining their job logs**, this can be done from the Rails console (`sudo gitlab-rails console`):
+
+1. Select jobs to be deleted:
+
+   To select all jobs with artifacts for a single project:
+
+   ```ruby
+   project = Project.find_by_full_path('path/to/project')
+   builds_with_artifacts =  project.builds.with_downloadable_artifacts
+   ```
+
+   To select all jobs with artifacts across the entire GitLab instance:
+
+   ```ruby
+   builds_with_artifacts = Ci::Build.with_downloadable_artifacts
+   ```
+
+1. Delete job artifacts older than a specific date:
+
+   NOTE:
+   This step also erases artifacts that users have chosen to
+   ["keep"](../ci/jobs/job_artifacts.md#download-job-artifacts).
+
+   ```ruby
+   builds_to_clear = builds_with_artifacts.where("finished_at < ?", 1.week.ago)
+   builds_to_clear.find_each do |build|
+     Ci::JobArtifacts::DeleteService.new(build).execute
+     build.update!(artifacts_expire_at: Time.now)
+   end
+   ```
+
+   In [GitLab 15.3 and earlier](https://gitlab.com/gitlab-org/gitlab/-/issues/372537), use the following instead:
+
+   ```ruby
+   builds_to_clear = builds_with_artifacts.where("finished_at < ?", 1.week.ago)
+   builds_to_clear.find_each do |build|
+     build.artifacts_expire_at = Time.now
+     build.erase_erasable_artifacts!
+   end
+   ```
+
+   `1.week.ago` is a Rails `ActiveSupport::Duration` method which calculates a new
+   date or time in the past. Other valid examples are:
+
+   - `7.days.ago`
+   - `3.months.ago`
+   - `1.year.ago`
+
+   `erase_erasable_artifacts!` is a synchronous method, and upon execution the artifacts are immediately removed;
+   they are not scheduled by a background queue.
+
+### Delete job artifacts and logs from jobs completed before a specific date
+
+WARNING:
+These commands remove data permanently from both the database and from disk. Before running them, we highly recommend seeking guidance from a Support Engineer, or running them in a test environment with a backup of the instance ready to be restored, just in case.
+
+If you need to manually remove **all** job artifacts associated with multiple jobs,
+**including job logs**, this can be done from the Rails console (`sudo gitlab-rails console`):
+
+1. Select the jobs to be deleted:
+
+   To select jobs with artifacts for a single project:
+
+   ```ruby
+   project = Project.find_by_full_path('path/to/project')
+   builds_with_artifacts =  project.builds.with_downloadable_artifacts
+   ```
+
+   To select jobs with artifacts across the entire GitLab instance:
+
+   ```ruby
+   builds_with_artifacts = Ci::Build.with_downloadable_artifacts
+   ```
+
+   Occasionally, when choosing jobs with artifacts, there could be a risk of the process being terminated due to selecting a large number of rows. This can result in high memory usage and eventually lead to the process being killed due to an Out-of-Memory (OOM) error. To resolve this, you can run in small batches. The example below limits each batch to 1000.
+
+   To select jobs with artifacts for a single project:
+
+   ```ruby
+   project = Project.find_by_full_path('path/to/project')
+   builds_with_artifacts =  project.builds.with_downloadable_artifacts.find_each(batch_size: 1000)
+   ```
+
+   To select jobs with artifacts across the entire GitLab instance:
+
+   ```ruby
+   builds_with_artifacts = Ci::Build.with_downloadable_artifacts.find_each(batch_size: 1000)
+   ```
+
+1. Select the user which is mentioned in the web UI as erasing the job:
+
+   ```ruby
+   admin_user = User.find_by(username: 'username')
+   ```
+
+1. Erase the job artifacts and logs older than a specific date:
+
+   ```ruby
+   builds_to_clear = builds_with_artifacts.where("finished_at < ?", 1.week.ago)
+   builds_to_clear.find_each do |build|
+     print "Ci::Build ID #{build.id}... "
+
+     if build.erasable?
+       Ci::BuildEraseService.new(build, admin_user).execute
+       puts "Erased"
+     else
+       puts "Skipped (Nothing to erase or not erasable)"
+     end
+   end
+   ```
+
+   In [GitLab 15.3 and earlier](https://gitlab.com/gitlab-org/gitlab/-/issues/369132), replace
+   `Ci::BuildEraseService.new(build, admin_user).execute` with `build.erase(erased_by: admin_user)`.
+
+   `1.week.ago` is a Rails `ActiveSupport::Duration` method which calculates a new
+   date or time in the past. Other valid examples are:
+
+   - `7.days.ago`
+   - `3.months.ago`
+   - `1.year.ago`
+
+## Job artifact upload fails with error 500
+
+If you are using object storage for artifacts and a job artifact fails to upload,
+review:
+
+- The job log for an error message similar to:
+
+  ```plaintext
+  WARNING: Uploading artifacts as "archive" to coordinator... failed id=12345 responseStatus=500 Internal Server Error status=500 token=abcd1234
+  ```
+
+- The [workhorse log](logs/index.md#workhorse-logs) for an error message similar to:
+
+  ```json
+  {"error":"MissingRegion: could not find region configuration","level":"error","msg":"error uploading S3 session","time":"2021-03-16T22:10:55-04:00"}
+  ```
+
+In both cases, you might need to add `region` to the job artifact [object storage configuration](object_storage.md).
+
+## Job artifact upload fails with `500 Internal Server Error (Missing file)`
+
+Bucket names that include folder paths are not supported with [consolidated object storage](object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form).
+For example, `bucket/path`. If a bucket name has a path in it, you might receive an error similar to:
+
+```plaintext
+WARNING: Uploading artifacts as "archive" to coordinator... POST https://gitlab.example.com/api/v4/jobs/job_id/artifacts?artifact_format=zip&artifact_type=archive&expire_in=1+day: 500 Internal Server Error (Missing file)
+FATAL: invalid argument
+```
+
+If a job artifact fails to upload with the above error when using consolidated object storage, make sure you are [using separate buckets](object_storage.md#use-separate-buckets) for each data type.
+
+## Job artifacts fail to upload with `FATAL: invalid argument` when using Windows mount
+
+If you are using a Windows mount with CIFS for job artifacts, you may see an
+`invalid argument` error when the runner attempts to upload artifacts:
+
+```plaintext
+WARNING: Uploading artifacts as "dotenv" to coordinator... POST https://<your-gitlab-instance>/api/v4/jobs/<JOB_ID>/artifacts: 500 Internal Server Error  id=1296 responseStatus=500 Internal Server Error status=500 token=*****
+FATAL: invalid argument
+```
+
+To work around this issue, you can try:
+
+- Switching to an ext4 mount instead of CIFS.
+- Upgrading to at least Linux kernel 5.15 which contains a number of important bug fixes
+  relating to CIFS file leases.
+- For older kernels, using the `nolease` mount option to disable file leasing.
+
+For more information, [see the investigation details](https://gitlab.com/gitlab-org/gitlab/-/issues/389995).
+
+## Usage quota shows incorrect artifact storage usage
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/238536) in GitLab 14.10.
+
+Sometimes the [artifacts storage usage](../user/usage_quotas.md) displays an incorrect
+value for the total storage space used by artifacts. To recalculate the artifact
+usage statistics for all projects in the instance, you can run this background script:
+
+```shell
+bin/rake 'gitlab:refresh_project_statistics_build_artifacts_size[file.csv]'
+```
+
+The artifact usage value can fluctuate to `0` while the script is running. After
+recalculation, usage should display as expected again.
diff --git a/doc/administration/settings/continuous_integration.md b/doc/administration/settings/continuous_integration.md
index 6abef613be0..edf61701a33 100644
--- a/doc/administration/settings/continuous_integration.md
+++ b/doc/administration/settings/continuous_integration.md
@@ -139,7 +139,7 @@ NOTE:
 Any changes to this setting applies to new artifacts only. The expiration time is not
 be updated for artifacts created before this setting was changed.
 The administrator may need to manually search for and expire previously-created
-artifacts, as described in the [troubleshooting documentation](../../administration/job_artifacts.md#delete-job-artifacts-from-jobs-completed-before-a-specific-date).
+artifacts, as described in the [troubleshooting documentation](../../administration/job_artifacts_troubleshooting.md#delete-job-artifacts-from-jobs-completed-before-a-specific-date).
 
 ## Keep the latest artifacts for all jobs in the latest successful pipelines
 
diff --git a/doc/api/graphql/reference/index.md b/doc/api/graphql/reference/index.md
index 09c9d138c3f..351c8e69374 100644
--- a/doc/api/graphql/reference/index.md
+++ b/doc/api/graphql/reference/index.md
@@ -6912,6 +6912,7 @@ Input type: `UserAddOnAssignmentRemoveInput`
 
 | Name | Type | Description |
 | ---- | ---- | ----------- |
+| <a id="mutationuseraddonassignmentremoveaddonpurchase"></a>`addOnPurchase` | [`AddOnPurchase`](#addonpurchase) | AddOnPurchase state after mutation. |
 | <a id="mutationuseraddonassignmentremoveclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. |
 | <a id="mutationuseraddonassignmentremoveerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. |
 
diff --git a/doc/api/jobs.md b/doc/api/jobs.md
index a48168c8362..1aa4d34a4c5 100644
--- a/doc/api/jobs.md
+++ b/doc/api/jobs.md
@@ -842,7 +842,7 @@ Example of response
 
 NOTE:
 You can't delete archived jobs with the API, but you can
-[delete job artifacts and logs from jobs completed before a specific date](../administration/job_artifacts.md#delete-job-artifacts-and-logs-from-jobs-completed-before-a-specific-date)
+[delete job artifacts and logs from jobs completed before a specific date](../administration/job_artifacts_troubleshooting.md#delete-job-artifacts-and-logs-from-jobs-completed-before-a-specific-date)
 
 ## Run a job
 
diff --git a/doc/ci/jobs/job_artifacts_troubleshooting.md b/doc/ci/jobs/job_artifacts_troubleshooting.md
index f5951350085..ebed347e966 100644
--- a/doc/ci/jobs/job_artifacts_troubleshooting.md
+++ b/doc/ci/jobs/job_artifacts_troubleshooting.md
@@ -23,7 +23,7 @@ the keyword reference for information on how to fetch artifacts with these keywo
 ## Job artifacts use too much disk space
 
 If job artifacts are using too much disk space, see the
-[job artifacts administration documentation](../../administration/job_artifacts.md#job-artifacts-using-too-much-disk-space).
+[job artifacts administration documentation](../../administration/job_artifacts_troubleshooting.md#job-artifacts-using-too-much-disk-space).
 
 ## Error message `No files to upload`
 
diff --git a/doc/development/database/efficient_in_operator_queries.md b/doc/development/database/efficient_in_operator_queries.md
index a0c71f49e2d..87976df5711 100644
--- a/doc/development/database/efficient_in_operator_queries.md
+++ b/doc/development/database/efficient_in_operator_queries.md
@@ -228,9 +228,6 @@ Gitlab::Pagination::Keyset::InOperatorOptimization::QueryBuilder.new(
   If it's not given, the `IN` operator optimization only makes the `ORDER BY` columns available to
   the end-user and not the full database row.
 
-  If it's not given, the `IN` operator optimization only makes the `ORDER BY` columns available to
-  the end-user and not the full database row.
-
 The following database index on the `issues` table must be present
 to make the query execute efficiently:
 
diff --git a/doc/update/index.md b/doc/update/index.md
index a030770ac64..723b5199e98 100644
--- a/doc/update/index.md
+++ b/doc/update/index.md
@@ -280,6 +280,11 @@ and [Helm Chart deployments](https://docs.gitlab.com/charts/). They come with ap
   See the [configuration workarounds](https://gitlab.com/gitlab-org/gitlab/-/issues/419485#workarounds)
   for more details.
 - Git 2.41.0 and later is required by Gitaly. For installations from source, you should use the [Git version provided by Gitaly](../install/installation.md#git).
+- New job artifacts are not replicated if job artifacts are configured to be stored in object storage and `direct_upload` is enabled. This bug is fixed in GitLab versions 16.1.4,
+  16.2.3, 16.3.0, and later.
+  - Impacted versions: GitLab versions 16.1.0 - 16.1.3 and 16.2.0 - 16.2.2.
+  - If you deployed an affected version, after upgrading to a fixed GitLab version, follow [these instructions](https://gitlab.com/gitlab-org/gitlab/-/issues/419742#to-fix-data)
+    to resync the affected job artifacts.
 
 ### 16.1.0
 
@@ -302,6 +307,11 @@ and [Helm Chart deployments](https://docs.gitlab.com/charts/). They come with ap
   - Impacted versions: GitLab versions 16.1.x.
   - Versions containing fix: GitLab 16.2.0 and later.
 - For self-compiled installations: You must remove any settings related to Puma worker killer from the `puma.rb` configuration file, since those have been [removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/118645). For more information, see the [`puma.rb.example`](https://gitlab.com/gitlab-org/gitlab/-/blob/16-0-stable-ee/config/puma.rb.example) file.
+- New job artifacts are not replicated if job artifacts are configured to be stored in object storage and `direct_upload` is enabled. This bug is fixed in GitLab versions 16.1.4,
+  16.2.3, 16.3.0, and later.
+  - Impacted versions: GitLab versions 16.1.0 - 16.1.3 and 16.2.0 - 16.2.2.
+  - If you deployed an affected version, after upgrading to a fixed GitLab version, follow [these instructions](https://gitlab.com/gitlab-org/gitlab/-/issues/419742#to-fix-data)
+    to resync the affected job artifacts.
 
 ### 16.0.0