Add latest changes from gitlab-org/gitlab@master

14 files changed, 560 insertions, 597 deletions

diff --git a/doc/administration/cicd.md b/doc/administration/cicd.md
index 47cbd7d1aec..9c69a7cbb65 100644
--- a/doc/administration/cicd.md
+++ b/doc/administration/cicd.md
@@ -93,14 +93,96 @@ To change the frequency of the pipeline schedule worker:
 For example, to set the maximum frequency of pipelines to twice a day, set `pipeline_schedule_worker_cron`
 to a cron value of `0 */12 * * *` (`00:00` and `12:00` every day).
 
-<!-- ## Troubleshooting
+## Disaster recovery
 
-Include any troubleshooting steps that you can foresee. If you know beforehand what issues
-one might have when setting this up, or when something is changed, or on upgrading, it's
-important to describe those, too. Think of things that may go wrong and include them here.
-This is important to minimize requests for support, and to avoid doc comments with
-questions that you know someone might ask.
+You can disable some important but computationally expensive parts of the application
+to relieve stress on the database during ongoing downtime.
 
-Each scenario can be a third-level heading, for example `### Getting error message X`.
-If you have none to add when creating a doc, leave this section in place
-but commented out to help encourage others to add to it in the future. -->
+### Disable fair scheduling on shared runners
+
+When clearing a large backlog of jobs, you can temporarily enable the `ci_queueing_disaster_recovery_disable_fair_scheduling`
+[feature flag](../administration/feature_flags.md). This flag disables fair scheduling
+on shared runners, which reduces system resource usage on the `jobs/request` endpoint.
+
+When enabled, jobs are processed in the order they were put in the system, instead of
+balanced across many projects.
+
+### Disable compute quota enforcement
+
+To disable the enforcement of [compute quotas](../ci/pipelines/cicd_minutes.md) on shared runners, you can temporarily
+enable the `ci_queueing_disaster_recovery_disable_quota` [feature flag](../administration/feature_flags.md).
+This flag reduces system resource usage on the `jobs/request` endpoint.
+
+When enabled, jobs created in the last hour can run in projects which are out of quota.
+Earlier jobs are already canceled by a periodic background worker (`StuckCiJobsWorker`).
+
+## CI/CD troubleshooting Rails console commands
+
+The following commands are run in the [Rails console](../administration/operations/rails_console.md#starting-a-rails-console-session).
+
+WARNING:
+Any command that changes data directly could be damaging if not run correctly, or under the right conditions.
+We highly recommend running them in a test environment with a backup of the instance ready to be restored, just in case.
+
+### Cancel stuck pending pipelines
+
+```ruby
+project = Project.find_by_full_path('<project_path>')
+Ci::Pipeline.where(project_id: project.id).where(status: 'pending').count
+Ci::Pipeline.where(project_id: project.id).where(status: 'pending').each {|p| p.cancel if p.stuck?}
+Ci::Pipeline.where(project_id: project.id).where(status: 'pending').count
+```
+
+### Try merge request integration
+
+```ruby
+project = Project.find_by_full_path('<project_path>')
+mr = project.merge_requests.find_by(iid: <merge_request_iid>)
+mr.project.try(:ci_integration)
+```
+
+### Validate the `.gitlab-ci.yml` file
+
+```ruby
+project = Project.find_by_full_path('<project_path>')
+content = p.ci_config_for(project.repository.root_ref_sha)
+Gitlab::Ci::Lint.new(project: project,  current_user: User.first).validate(content)
+```
+
+### Disable AutoDevOps on Existing Projects
+
+```ruby
+Project.all.each do |p|
+  p.auto_devops_attributes={"enabled"=>"0"}
+  p.save
+end
+```
+
+### Obtain runners registration token
+
+```ruby
+Gitlab::CurrentSettings.current_application_settings.runners_registration_token
+```
+
+### Seed runners registration token
+
+```ruby
+appSetting = Gitlab::CurrentSettings.current_application_settings
+appSetting.set_runners_registration_token('<new-runners-registration-token>')
+appSetting.save!
+```
+
+### Run pipeline schedules manually
+
+You can run pipeline schedules manually through the Rails console to reveal any errors that are usually not visible.
+
+```ruby
+# schedule_id can be obtained from Edit Pipeline Schedule page
+schedule = Ci::PipelineSchedule.find_by(id: <schedule_id>)
+
+# Select the user that you want to run the schedule for
+user = User.find_by_username('<username>')
+
+# Run the schedule
+ps = Ci::CreatePipelineService.new(schedule.project, user, ref: schedule.ref).execute!(:schedule, ignore_skip_ci: true, save_on_errors: false, schedule: schedule)
+```
diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md
index 44aa3d648ad..e2fc425b536 100644
--- a/doc/administration/reference_architectures/index.md
+++ b/doc/administration/reference_architectures/index.md
@@ -191,13 +191,22 @@ Before implementing a reference architecture, refer to the following requirement
 These reference architectures were built and tested on Google Cloud Platform (GCP) using the
 [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms)
 CPU platform as a lowest common denominator baseline ([Sysbench benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks)).
+Newer, similarly-sized CPUs are supported and may have improved performance as a result.
 
-Newer, similarly-sized CPUs are supported and may have improved performance as a result. For Linux package environments,
-ARM-based equivalents are also supported.
+ARM CPUs are supported for Linux package environments as well as for any [Cloud Provider services](#cloud-provider-services) where applicable.
 
 NOTE:
 Any "burstable" instance types are not recommended due to inconsistent performance.
 
+### Supported disk types
+
+As a general guidance, most standard disk types are expected to work for GitLab, but be aware of the following specific call outs:
+
+- [Gitaly](../gitaly/index.md#disk-requirements) requires at least 8,000 input/output operations per second (IOPS) for read operations, and 2,000 IOPS for write operations.
+- We don't recommend the use of any disk types that are "burstable" due to inconsistent performance.
+
+Outside the above standard, disk types are expected to work for GitLab and the choice of each depends on your specific requirements around areas, such as durability or costs.
+
 ### Supported infrastructure
 
 As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP, Azure) and
@@ -356,6 +365,12 @@ If you choose to use a third party external service:
 
 Redis is primarily single threaded. For the 10,000 user and above Reference Architectures, separate out the instances as specified into Cache and Persistent data to achieve optimum performance at this scale.
 
+### Recommendation notes for Object Storage
+
+GitLab has been tested against [various Object Storage providers](../object_storage.md#supported-object-storage-providers) that are expected to work.
+
+As a general guidance, it's recommended to use a reputable solution that has full S3 compatibility.
+
 #### Unsupported database services
 
 Several database cloud provider services are known not to support the above or have been found to have other issues and aren't recommended:
@@ -663,8 +678,8 @@ Most setups would only need vertical scaling, but there are some specific areas
 Conversely, if you have robust metrics in place that show the environment is over-provisioned, you can apply the same process for
 scaling downwards. You should take an iterative approach when scaling downwards, however, to ensure there are no issues.
 
-### How to monitor your environment
+### Monitoring
+
+There are numerous options available to monitor your infrastructure, as well as [GitLab itself](../monitoring/index.md), and you should refer to your chosen monitoring solution's documentation for more information.
 
-To monitor your GitLab environment, you can use the tools
-[bundled with GitLab](../monitoring/index.md), but it's also possible to use third-party
-options if desired.
+Of note, the GitLab application is bundled with [Prometheus as well as various Prometheus compatible exporters](../monitoring/prometheus/index.md) that could be hooked into your solution.
diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md
index 9432836c22b..01c75c32366 100644
--- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md
+++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md
@@ -46,11 +46,11 @@ This content has been moved to [Troubleshooting Repository mirroring](../../user
 
 ## CI
 
-This content has been moved to [Troubleshooting CI/CD](../../ci/troubleshooting.md).
+This content has been moved to [Troubleshooting CI/CD](../cicd.md#cicd-troubleshooting-rails-console-commands).
 
 ## License
 
-This content has been moved to [Activate GitLab EE with a license file or key](../../administration/license_file.md).
+This content has been moved to [Activate GitLab EE with a license file or key](../license_file.md).
 
 ## Registry
 
diff --git a/doc/ci/debugging.md b/doc/ci/debugging.md
new file mode 100644
index 00000000000..bc4482a43e2
--- /dev/null
+++ b/doc/ci/debugging.md
@@ -0,0 +1,280 @@
+---
+stage: Verify
+group: Pipeline Authoring
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+type: reference
+---
+
+# Debugging CI/CD pipelines **(FREE ALL)**
+
+GitLab provides several tools to help make it easier to debug your CI/CD configuration.
+
+If you are unable to resolve pipeline issues, you can get help from:
+
+- The [GitLab community forum](https://forum.gitlab.com/)
+- GitLab [Support](https://about.gitlab.com/support/)
+
+## Verify syntax
+
+An early source of problems can be incorrect syntax. The pipeline shows a `yaml invalid`
+badge and does not start running if any syntax or formatting problems are found.
+
+### Edit `.gitlab-ci.yml` with the pipeline editor
+
+The [pipeline editor](pipeline_editor/index.md) is the recommended editing
+experience (rather than the single file editor or the Web IDE). It includes:
+
+- Code completion suggestions that ensure you are only using accepted keywords.
+- Automatic syntax highlighting and validation.
+- The [CI/CD configuration visualization](pipeline_editor/index.md#visualize-ci-configuration),
+  a graphical representation of your `.gitlab-ci.yml` file.
+
+### Edit `.gitlab-ci.yml` locally
+
+If you prefer to edit your pipeline configuration locally, you can use the
+GitLab CI/CD schema in your editor to verify basic syntax issues. Any
+[editor with Schemastore support](https://www.schemastore.org/json/#editors) uses
+the GitLab CI/CD schema by default.
+
+If you need to link to the schema directly, use this URL:
+
+```plaintext
+https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/editor/schema/ci.json
+```
+
+To see the full list of custom tags covered by the CI/CD schema, check the
+latest version of the schema.
+
+### Verify syntax with CI Lint tool
+
+You can use the [CI Lint tool](lint.md) to verify that the syntax of a CI/CD configuration
+snippet is correct. Paste in full `.gitlab-ci.yml` files or individual job configurations,
+to verify the basic syntax.
+
+When a `.gitlab-ci.yml` file is present in a project, you can also use the CI Lint
+tool to [simulate the creation of a full pipeline](lint.md#simulate-a-pipeline).
+It does deeper verification of the configuration syntax.
+
+## Verify variables
+
+A key part of troubleshooting CI/CD is to verify which variables are present in a
+pipeline, and what their values are. A lot of pipeline configuration is dependent
+on variables, and verifying them is one of the fastest ways to find the source of
+a problem.
+
+[Export the full list of variables](variables/index.md#list-all-variables)
+available in each problematic job. Check if the variables you expect are present,
+and check if their values are what you expect.
+
+## Job configuration issues
+
+A lot of common pipeline issues can be fixed by analyzing the behavior of the `rules`
+or `only/except` configuration used to [control when jobs are added to a pipeline](jobs/job_control.md).
+You shouldn't use these two configurations in the same pipeline, as they behave differently.
+It's hard to predict how a pipeline runs with this mixed behavior. `rules` is the preferred
+choice for controlling jobs, as `only` and `except` are no longer being actively developed.
+
+If your `rules` or `only/except` configuration makes use of [predefined variables](variables/predefined_variables.md)
+like `CI_PIPELINE_SOURCE`, `CI_MERGE_REQUEST_ID`, you should [verify them](#verify-variables)
+as the first troubleshooting step.
+
+### Jobs or pipelines don't run when expected
+
+The `rules` or `only/except` keywords are what determine whether or not a job is
+added to a pipeline. If a pipeline runs, but a job is not added to the pipeline,
+it's usually due to `rules` or `only/except` configuration issues.
+
+If a pipeline does not seem to run at all, with no error message, it may also be
+due to `rules` or `only/except` configuration, or the `workflow: rules` keyword.
+
+If you are converting from `only/except` to the `rules` keyword, you should check
+the [`rules` configuration details](yaml/index.md#rules) carefully. The behavior
+of `only/except` and `rules` is different and can cause unexpected behavior when migrating
+between the two.
+
+The [common `if` clauses for `rules`](jobs/job_control.md#common-if-clauses-for-rules)
+can be very helpful for examples of how to write rules that behave the way you expect.
+
+### A job with the `changes` keyword runs unexpectedly
+
+A common reason a job is added to a pipeline unexpectedly is because the `changes`
+keyword always evaluates to true in certain cases. For example, `changes` is always
+true in certain pipeline types, including scheduled pipelines and pipelines for tags.
+
+The `changes` keyword is used in combination with [`only/except`](yaml/index.md#onlychanges--exceptchanges)
+or [`rules`](yaml/index.md#ruleschanges). It's recommended to only use `changes` with
+`if` sections in `rules` or `only/except` configuration that ensures the job is only added to
+branch pipelines or merge request pipelines.
+
+### Two pipelines run at the same time
+
+Two pipelines can run when pushing a commit to a branch that has an open merge request
+associated with it. Usually one pipeline is a merge request pipeline, and the other
+is a branch pipeline.
+
+This situation is usually caused by the `rules` configuration, and there are several ways to
+[prevent duplicate pipelines](jobs/job_control.md#avoid-duplicate-pipelines).
+
+### No pipeline or the wrong type of pipeline runs
+
+Before a pipeline can run, GitLab evaluates all the jobs in the configuration and tries
+to add them to all available pipeline types. A pipeline does not run if no jobs are added
+to it at the end of the evaluation.
+
+If a pipeline did not run, it's likely that all the jobs had `rules` or `only/except` that
+blocked them from being added to the pipeline.
+
+If the wrong pipeline type ran, then the `rules` or `only/except` configuration should
+be checked to make sure the jobs are added to the correct pipeline type. For
+example, if a merge request pipeline did not run, the jobs may have been added to
+a branch pipeline instead.
+
+It's also possible that your [`workflow: rules`](yaml/index.md#workflow) configuration
+blocked the pipeline, or allowed the wrong pipeline type.
+
+### Pipeline with many jobs fails to start
+
+A Pipeline that has more jobs than the instance's defined [CI/CD limits](../administration/settings/continuous_integration.md#set-cicd-limits)
+fails to start.
+
+To reduce the number of jobs in a single pipeline, you can split your `.gitlab-ci.yml`
+configuration into more independent [parent-child pipelines](../ci/pipelines/pipeline_architectures.md#parent-child-pipelines).
+
+## Pipeline warnings
+
+Pipeline configuration warnings are shown when you:
+
+- [Validate configuration with the CI Lint tool](yaml/index.md).
+- [Manually run a pipeline](pipelines/index.md#run-a-pipeline-manually).
+
+### `Job may allow multiple pipelines to run for a single action` warning
+
+When you use [`rules`](yaml/index.md#rules) with a `when` clause without an `if`
+clause, multiple pipelines may run. Usually this occurs when you push a commit to
+a branch that has an open merge request associated with it.
+
+To [prevent duplicate pipelines](jobs/job_control.md#avoid-duplicate-pipelines), use
+[`workflow: rules`](yaml/index.md#workflow) or rewrite your rules to control
+which pipelines can run.
+
+## Troubleshooting
+
+For help with a specific area, see:
+
+- [Caching](caching/index.md#troubleshooting).
+- [CI/CD job tokens](jobs/ci_job_token.md).
+- [Container Registry](../user/packages/container_registry/troubleshoot_container_registry.md).
+- [Docker](docker/using_docker_build.md#troubleshooting).
+- [Downstream pipelines](pipelines/downstream_pipelines.md#troubleshooting).
+- [Environments](environments/deployment_safety.md#ensure-only-one-deployment-job-runs-at-a-time).
+- [GitLab Runner](https://docs.gitlab.com/runner/faq/).
+- [ID tokens](secrets/id_token_authentication.md#troubleshooting).
+- [Jobs](jobs/index.md#troubleshooting).
+- [Job control](jobs/job_control.md).
+- [Job artifacts](jobs/job_artifacts_troubleshooting.md).
+- [Merge request pipelines](pipelines/merge_request_pipelines.md#troubleshooting),
+  [merged results pipelines](pipelines/merged_results_pipelines.md#troubleshooting),
+  and [Merge trains](pipelines/merge_trains.md#troubleshooting).
+- [Pipeline editor](pipeline_editor/index.md#troubleshooting).
+- [Variables](variables/index.md#troubleshooting).
+- [YAML `includes` keyword](yaml/includes.md#troubleshooting).
+- [YAML `script` keyword](yaml/script.md#troubleshooting).
+
+Otherwise, review the following troubleshooting sections for known status messages
+and error messages.
+
+### `A CI/CD pipeline must run and be successful before merge` message
+
+This message is shown if the [**Pipelines must succeed**](../user/project/merge_requests/merge_when_pipeline_succeeds.md#require-a-successful-pipeline-for-merge)
+setting is enabled in the project and a pipeline has not yet run successfully.
+This also applies if the pipeline has not been created yet, or if you are waiting
+for an external CI service.
+
+If you don't use pipelines for your project, then you should disable **Pipelines must succeed**
+so you can accept merge requests.
+
+### `Checking ability to merge automatically` message
+
+If your merge request is stuck with a `Checking ability to merge automatically`
+message that does not disappear after a few minutes, you can try one of these workarounds:
+
+- Refresh the merge request page.
+- Close & Re-open the merge request.
+- Rebase the merge request with the `/rebase` [quick action](../user/project/quick_actions.md).
+- If you have already confirmed the merge request is ready to be merged, you can merge
+  it with the `/merge` quick action.
+
+This issue is [resolved](https://gitlab.com/gitlab-org/gitlab/-/issues/229352) in GitLab 15.5.
+
+### `Checking pipeline status` message
+
+This message displays when the merge request does not yet have a pipeline associated with the
+latest commit. This might be because:
+
+- GitLab hasn't finished creating the pipeline yet.
+- You are using an external CI service and GitLab hasn't heard back from the service yet.
+- You are not using CI/CD pipelines in your project.
+- You are using CI/CD pipelines in your project, but your configuration prevented a pipeline from running on the source branch for your merge request.
+- The latest pipeline was deleted (this is a [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/214323)).
+- The source branch of the merge request is on a private fork.
+
+After the pipeline is created, the message updates with the pipeline status.
+
+### `Project <group/project> not found or access denied` message
+
+This message is shown if configuration is added with [`include`](yaml/index.md#include) and either:
+
+- The configuration refers to a project that can't be found.
+- The user that is running the pipeline is unable to access any included projects.
+
+To resolve this, check that:
+
+- The path of the project is in the format `my-group/my-project` and does not include
+  any folders in the repository.
+- The user running the pipeline is a [member of the projects](../user/project/members/index.md#add-users-to-a-project)
+  that contain the included files. Users must also have the [permission](../user/permissions.md#job-permissions)
+  to run CI/CD jobs in the same projects.
+
+### `The parsed YAML is too big` message
+
+This message displays when the YAML configuration is too large or nested too deeply.
+YAML files with a large number of includes, and thousands of lines overall, are
+more likely to hit this memory limit. For example, a YAML file that is 200 kb is
+likely to hit the default memory limit.
+
+To reduce the configuration size, you can:
+
+- Check the length of the expanded CI/CD configuration in the pipeline editor's
+  [Full configuration](pipeline_editor/index.md#view-full-configuration) tab. Look for
+  duplicated configuration that can be removed or simplified.
+- Move long or repeated `script` sections into standalone scripts in the project.
+- Use [parent and child pipelines](pipelines/downstream_pipelines.md#parent-child-pipelines) to move some
+  work to jobs in an independent child pipeline.
+
+On a self-managed instance, you can [increase the size limits](../administration/instance_limits.md#maximum-size-and-depth-of-cicd-configuration-yaml-files).
+
+### `500` error when editing the `.gitlab-ci.yml` file
+
+A [loop of included configuration files](pipeline_editor/index.md#configuration-validation-currently-not-available-message)
+can cause a `500` error when editing the `.gitlab-ci.yml` file with the [web editor](../user/project/repository/web_editor.md).
+
+Ensure that included configuration files do not create a loop of references to each other.
+
+### `Failed to pull image` message
+
+> **Allow access to this project with a CI_JOB_TOKEN** setting [renamed to **Limit access _to_ this project**](https://gitlab.com/gitlab-org/gitlab/-/issues/411406) in GitLab 16.3.
+
+When a runner tries to pull an image from a private project, the job could fail with the following error:
+
+```shell
+WARNING: Failed to pull image with policy "always": Error response from daemon: pull access denied for registry.example.com/path/to/project, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
+```
+
+This error can happen if the following are both true:
+
+- The **Limit access _to_ this project** option is enabled in the private project
+  hosting the image.
+- The job attempting to fetch the image is running for a project that is not listed in
+  the private project's allowlist.
+
+The recommended solution is to [add your project to the private project's job token scope allowlist](jobs/ci_job_token.md#add-a-project-to-the-job-token-scope-allowlist).
diff --git a/doc/ci/jobs/index.md b/doc/ci/jobs/index.md
index 90a64ea7569..761e9e6dd66 100644
--- a/doc/ci/jobs/index.md
+++ b/doc/ci/jobs/index.md
@@ -397,3 +397,67 @@ The behavior of deployment jobs can be controlled with
 [deployment safety](../environments/deployment_safety.md) settings like
 [preventing outdated deployment jobs](../environments/deployment_safety.md#prevent-outdated-deployment-jobs)
 and [ensuring only one deployment job runs at a time](../environments/deployment_safety.md#ensure-only-one-deployment-job-runs-at-a-time).
+
+## Troubleshooting
+
+### Job log slow to update
+
+When you visit the job log page for a running job, there could be a delay of up to
+60 seconds before a log update. The default refresh time is 60 seconds, but after
+the log is viewed in the UI one time, log updates should occur every 3 seconds.
+
+### `get_sources` job section fails because of an HTTP/2 problem
+
+Sometimes, jobs fail with the following cURL error:
+
+```plaintext
+++ git -c 'http.userAgent=gitlab-runner <version>' fetch origin +refs/pipelines/<id>:refs/pipelines/<id> ...
+error: RPC failed; curl 16 HTTP/2 send again with decreased length
+fatal: ...
+```
+
+You can work around this problem by configuring Git and `libcurl` to
+[use HTTP/1.1](https://git-scm.com/docs/git-config#Documentation/git-config.txt-httpversion).
+The configuration can be added to:
+
+- A job's [`pre_get_sources_script`](../yaml/index.md#hookspre_get_sources_script):
+
+  ```yaml
+  job_name:
+    hooks:
+      pre_get_sources_script:
+        - git config --global http.version "HTTP/1.1"
+  ```
+
+- The [runner's `config.toml`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html)
+  with [Git configuration environment variables](https://git-scm.com/docs/git-config#ENVIRONMENT):
+
+  ```toml
+  [[runners]]
+  ...
+  environment = [
+    "GIT_CONFIG_COUNT=1",
+    "GIT_CONFIG_KEY_0=http.version",
+    "GIT_CONFIG_VALUE_0=HTTP/1.1"
+  ]
+  ```
+
+### Job using `resource_group` gets stuck **(FREE SELF)**
+
+If a job using [`resource_group`](../yaml/index.md#resource_group) gets stuck, a
+GitLab administrator can try run the following commands from the [rails console](../../administration/operations/rails_console.md#starting-a-rails-console-session):
+
+```ruby
+# find resource group by name
+resource_group = Project.find_by_full_path('...').resource_groups.find_by(key: 'the-group-name')
+busy_resources = resource_group.resources.where('build_id IS NOT NULL')
+
+# identify which builds are occupying the resource
+# (I think it should be 1 as of today)
+busy_resources.pluck(:build_id)
+
+# it's good to check why this build is holding the resource.
+# Is it stuck? Has it been forcefully dropped by the system?
+# free up busy resources
+busy_resources.update_all(build_id: nil)
+```
diff --git a/doc/ci/jobs/job_control.md b/doc/ci/jobs/job_control.md
index 418991ab5f7..0c8e4fc593f 100644
--- a/doc/ci/jobs/job_control.md
+++ b/doc/ci/jobs/job_control.md
@@ -174,8 +174,7 @@ multiple pipelines. You don't have to explicitly configure rules for multiple ty
 of pipeline to trigger them accidentally.
 
 Some configurations that have the potential to cause duplicate pipelines cause a
-[pipeline warning](../troubleshooting.md#pipeline-warnings) to be displayed.
-[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/219431) in GitLab 13.3.
+[pipeline warning](../debugging.md#pipeline-warnings) to be displayed.
 
 For example:
 
@@ -209,7 +208,7 @@ To avoid duplicate pipelines, you can:
 
 You can also avoid duplicate pipelines by changing the job rules to avoid either push (branch)
 pipelines or merge request pipelines. However, if you use a `- when: always` rule without
-`workflow: rules`, GitLab still displays a [pipeline warning](../troubleshooting.md#pipeline-warnings).
+`workflow: rules`, GitLab still displays a [pipeline warning](../debugging.md#pipeline-warnings).
 
 For example, the following does not trigger double pipelines, but is not recommended
 without `workflow: rules`:
@@ -1194,3 +1193,20 @@ To run protected manual jobs:
 - Add the administrator as a direct member of the private project (any role)
 - [Impersonate a user](../../administration/admin_area.md#user-impersonation) who is a
   direct member of the project.
+
+### A CI/CD job does not use newer configuration when run again
+
+The configuration for a pipeline is only fetched when the pipeline is created.
+When you rerun a job, uses the same configuration each time. If you update configuration files,
+including separate files added with [`include`](../yaml/index.md#include), you must
+start a new pipeline to use the new configuration.
+
+### `Job may allow multiple pipelines to run for a single action` warning
+
+When you use [`rules`](../yaml/index.md#rules) with a `when` clause without an `if`
+clause, multiple pipelines may run. Usually this occurs when you push a commit to
+a branch that has an open merge request associated with it.
+
+To [prevent duplicate pipelines](#avoid-duplicate-pipelines), use
+[`workflow: rules`](../yaml/index.md#workflow) or rewrite your rules to control
+which pipelines can run.
diff --git a/doc/ci/pipelines/merge_request_pipelines.md b/doc/ci/pipelines/merge_request_pipelines.md
index 37febfd90ee..fb1c19d8770 100644
--- a/doc/ci/pipelines/merge_request_pipelines.md
+++ b/doc/ci/pipelines/merge_request_pipelines.md
@@ -262,3 +262,25 @@ Some possible reasons for this error message:
 If **Run pipeline** is available, but the project does not have merge request pipelines
 enabled, do not use this option. You can push a commit or rebase the branch to trigger
 new branch pipelines.
+
+### `Merge blocked: pipeline must succeed. Push a new commit that fixes the failure` message
+
+This message is shown if the merge request pipeline, [merged results pipeline](merged_results_pipelines.md),
+or [merge train pipeline](merge_trains.md) has failed or been canceled.
+This does not happen when a branch pipeline fails.
+
+If a merge request pipeline or merged result pipeline was canceled or failed, you can:
+
+- Re-run the entire pipeline by selecting **Run pipeline** in the pipeline tab in the merge request.
+- [Retry only the jobs that failed](index.md#view-pipelines). If you re-run the entire pipeline, this is not necessary.
+- Push a new commit to fix the failure.
+
+If the merge train pipeline has failed, you can:
+
+- Check the failure and determine if you can use the [`/merge` quick action](../../user/project/quick_actions.md) to immediately add the merge request to the train again.
+- Re-run the entire pipeline by selecting **Run pipeline** in the pipeline tab in the merge request, then add the merge request to the train again.
+- Push a commit to fix the failure, then add the merge request to the train again.
+
+If the merge train pipeline was canceled before the merge request was merged, without a failure, you can:
+
+- Add it to the train again.
diff --git a/doc/ci/pipelines/merged_results_pipelines.md b/doc/ci/pipelines/merged_results_pipelines.md
index e4f739e8242..afe7a450370 100644
--- a/doc/ci/pipelines/merged_results_pipelines.md
+++ b/doc/ci/pipelines/merged_results_pipelines.md
@@ -61,19 +61,6 @@ Upgrade to 13.8 or later, or make sure the `:merge_ref_auto_sync`
 [feature flag is enabled](../../administration/feature_flags.md#check-if-a-feature-flag-is-enabled)
 on your GitLab instance.
 
-### Pipelines fail intermittently with a `fatal: reference is not a tree:` error
-
-Merged results pipelines run on a merge ref for a merge request
-(`refs/merge-requests/<iid>/merge`), so the Git reference could be overwritten at an
-unexpected time.
-
-For example, when a source or target branch is advanced, the pipeline fails with
-the `fatal: reference is not a tree:` error, which indicates that the checkout-SHA
-is not found in the merge ref.
-
-This behavior was improved in GitLab 12.4 by introducing [persistent pipeline refs](../troubleshooting.md#fatal-reference-is-not-a-tree-error).
-Upgrade to GitLab 12.4 or later to resolve the problem.
-
 ### Successful merged results pipeline overrides a failed branch pipeline
 
 A failed branch pipeline is sometimes ignored when the
diff --git a/doc/ci/troubleshooting.md b/doc/ci/troubleshooting.md
index b09b85f33a0..77ee6b11d92 100644
--- a/doc/ci/troubleshooting.md
+++ b/doc/ci/troubleshooting.md
@@ -1,555 +1,11 @@
 ---
-stage: Verify
-group: Pipeline Authoring
-info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
-type: reference
+redirect_to: 'debugging.md'
+remove_date: '2024-02-01'
 ---
 
-# Troubleshooting CI/CD **(FREE ALL)**
+This document was moved to [another location](debugging.md).
 
-GitLab provides several tools to help make troubleshooting your pipelines easier.
-
-This guide also lists common issues and possible solutions.
-
-## Verify syntax
-
-An early source of problems can be incorrect syntax. The pipeline shows a `yaml invalid`
-badge and does not start running if any syntax or formatting problems are found.
-
-### Edit `.gitlab-ci.yml` with the pipeline editor
-
-The [pipeline editor](pipeline_editor/index.md) is the recommended editing
-experience (rather than the single file editor or the Web IDE). It includes:
-
-- Code completion suggestions that ensure you are only using accepted keywords.
-- Automatic syntax highlighting and validation.
-- The [CI/CD configuration visualization](pipeline_editor/index.md#visualize-ci-configuration),
-  a graphical representation of your `.gitlab-ci.yml` file.
-
-### Edit `.gitlab-ci.yml` locally
-
-If you prefer to edit your pipeline configuration locally, you can use the
-GitLab CI/CD schema in your editor to verify basic syntax issues. Any
-[editor with Schemastore support](https://www.schemastore.org/json/#editors) uses
-the GitLab CI/CD schema by default.
-
-If you need to link to the schema directly, it
-is at:
-
-```plaintext
-https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/editor/schema/ci.json
-```
-
-To see the full list of custom tags covered by the CI/CD schema, check the
-latest version of the schema.
-
-### Verify syntax with CI Lint tool
-
-The [CI Lint tool](lint.md) is a simple way to ensure the syntax of a CI/CD configuration
-file is correct. Paste in full `.gitlab-ci.yml` files or individual jobs configuration,
-to verify the basic syntax.
-
-When a `.gitlab-ci.yml` file is present in a project, you can also use the CI Lint
-tool to [simulate the creation of a full pipeline](lint.md#simulate-a-pipeline).
-It does deeper verification of the configuration syntax.
-
-## Verify variables
-
-A key part of troubleshooting CI/CD is to verify which variables are present in a
-pipeline, and what their values are. A lot of pipeline configuration is dependent
-on variables, and verifying them is one of the fastest ways to find the source of
-a problem.
-
-[Export the full list of variables](variables/index.md#list-all-variables)
-available in each problematic job. Check if the variables you expect are present,
-and check if their values are what you expect.
-
-## GitLab CI/CD documentation
-
-The [complete `.gitlab-ci.yml` reference](yaml/index.md) contains a full list of
-every keyword you can use to configure your pipelines.
-
-You can also look at a large number of pipeline configuration [examples](examples/index.md)
-and [templates](examples/index.md#cicd-templates).
-
-### Documentation for pipeline types
-
-Branch pipelines are the most basic type.
-Other pipeline types have their own detailed usage guides that you should read
-if you are using that type:
-
-- [Multi-project pipelines](pipelines/downstream_pipelines.md#multi-project-pipelines): Have your pipeline trigger
-  a pipeline in a different project.
-- [Parent/child pipelines](pipelines/downstream_pipelines.md#parent-child-pipelines): Have your main pipeline trigger
-  and run separate pipelines in the same project. You can also
-  [dynamically generate the child pipeline's configuration](pipelines/downstream_pipelines.md#dynamic-child-pipelines)
-  at runtime.
-- [Merge request pipelines](pipelines/merge_request_pipelines.md): Run a pipeline
-  in the context of a merge request.
-  - [Merged results pipelines](pipelines/merged_results_pipelines.md):
-    Merge request pipelines that run on the combined source and target branch
-  - [Merge trains](pipelines/merge_trains.md):
-    Multiple merged results pipelines that queue and run automatically before
-    changes are merged.
-
-### Troubleshooting Guides for CI/CD features
-
-Troubleshooting guides are available for some CI/CD features and related topics:
-
-- [Container Registry](../user/packages/container_registry/troubleshoot_container_registry.md)
-- [GitLab Runner](https://docs.gitlab.com/runner/faq/)
-- [Merge Trains](pipelines/merge_trains.md#troubleshooting)
-- [Docker Build](docker/using_docker_build.md#troubleshooting)
-- [Environments](environments/deployment_safety.md#ensure-only-one-deployment-job-runs-at-a-time)
-
-## Common CI/CD issues
-
-A lot of common pipeline issues can be fixed by analyzing the behavior of the `rules`
-or `only/except` configuration. You shouldn't use these two configurations in the same
-pipeline, as they behave differently. It's hard to predict how a pipeline runs with
-this mixed behavior.
-
-If your `rules` or `only/except` configuration makes use of [predefined variables](variables/predefined_variables.md)
-like `CI_PIPELINE_SOURCE`, `CI_MERGE_REQUEST_ID`, you should [verify them](#verify-variables)
-as the first troubleshooting step.
-
-### Jobs or pipelines don't run when expected
-
-The `rules` or `only/except` keywords are what determine whether or not a job is
-added to a pipeline. If a pipeline runs, but a job is not added to the pipeline,
-it's usually due to `rules` or `only/except` configuration issues.
-
-If a pipeline does not seem to run at all, with no error message, it may also be
-due to `rules` or `only/except` configuration, or the `workflow: rules` keyword.
-
-If you are converting from `only/except` to the `rules` keyword, you should check
-the [`rules` configuration details](yaml/index.md#rules) carefully. The behavior
-of `only/except` and `rules` is different and can cause unexpected behavior when migrating
-between the two.
-
-The [common `if` clauses for `rules`](jobs/job_control.md#common-if-clauses-for-rules)
-can be very helpful for examples of how to write rules that behave the way you expect.
-
-#### Two pipelines run at the same time
-
-Two pipelines can run when pushing a commit to a branch that has an open merge request
-associated with it. Usually one pipeline is a merge request pipeline, and the other
-is a branch pipeline.
-
-This situation is usually caused by the `rules` configuration, and there are several ways to
-[prevent duplicate pipelines](jobs/job_control.md#avoid-duplicate-pipelines).
-
-#### A job is not in the pipeline
-
-GitLab determines if a job is added to a pipeline based on the [`only/except`](yaml/index.md#only--except)
-or [`rules`](yaml/index.md#rules) defined for the job. If it didn't run, it's probably
-not evaluating as you expect.
-
-#### No pipeline or the wrong type of pipeline runs
-
-Before a pipeline can run, GitLab evaluates all the jobs in the configuration and tries
-to add them to all available pipeline types. A pipeline does not run if no jobs are added
-to it at the end of the evaluation.
-
-If a pipeline did not run, it's likely that all the jobs had `rules` or `only/except` that
-blocked them from being added to the pipeline.
-
-If the wrong pipeline type ran, then the `rules` or `only/except` configuration should
-be checked to make sure the jobs are added to the correct pipeline type. For
-example, if a merge request pipeline did not run, the jobs may have been added to
-a branch pipeline instead.
-
-It's also possible that your [`workflow: rules`](yaml/index.md#workflow) configuration
-blocked the pipeline, or allowed the wrong pipeline type.
-
-### Pipeline with many jobs fails to start
-
-A Pipeline that has more jobs than the instance's defined [CI/CD limits](../administration/settings/continuous_integration.md#set-cicd-limits)
-fails to start.
-
-To reduce the number of jobs in your pipeline, you can split your `.gitlab-ci.yml`
-configuration using [parent-child pipelines](../ci/pipelines/pipeline_architectures.md#parent-child-pipelines).
-
-### A job runs unexpectedly
-
-A common reason a job is added to a pipeline unexpectedly is because the `changes`
-keyword always evaluates to true in certain cases. For example, `changes` is always
-true in certain pipeline types, including scheduled pipelines and pipelines for tags.
-
-The `changes` keyword is used in combination with [`only/except`](yaml/index.md#onlychanges--exceptchanges)
-or [`rules`](yaml/index.md#ruleschanges)). It's recommended to use `changes` with
-`rules` or `only/except` configuration that ensures the job is only added to branch
-pipelines or merge request pipelines.
-
-### "fatal: reference is not a tree" error
-
-> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17043) in GitLab 12.4.
-
-Previously, you'd have encountered unexpected pipeline failures when you force-pushed
-a branch to its remote repository. To illustrate the problem, suppose you've had the current workflow:
-
-1. A user creates a feature branch named `example` and pushes it to a remote repository.
-1. A new pipeline starts running on the `example` branch.
-1. A user rebases the `example` branch on the latest default branch and force-pushes it to its remote repository.
-1. A new pipeline starts running on the `example` branch again, however,
-   the previous pipeline (2) fails because of `fatal: reference is not a tree:` error.
-
-This occurs because the previous pipeline cannot find a checkout-SHA (which is associated with the pipeline record)
-from the `example` branch that the commit history has already been overwritten by the force-push.
-Similarly, [Merged results pipelines](pipelines/merged_results_pipelines.md)
-might have failed intermittently due to [the same reason](pipelines/merged_results_pipelines.md#pipelines-fail-intermittently-with-a-fatal-reference-is-not-a-tree-error).
-
-As of GitLab 12.4, we've improved this behavior by persisting pipeline refs exclusively.
-To illustrate its life cycle:
-
-1. A pipeline is created on a feature branch named `example`.
-1. A persistent pipeline ref is created at `refs/pipelines/<pipeline-id>`,
-   which retains the checkout-SHA of the associated pipeline record.
-   This persistent ref stays intact during the pipeline execution,
-   even if the commit history of the `example` branch has been overwritten by force-push.
-1. The runner fetches the persistent pipeline ref and gets source code from the checkout-SHA.
-1. When the pipeline finishes, its persistent ref is cleaned up in a background process.
-
-### `get_sources` job section fails because of an HTTP/2 problem
-
-Sometimes, jobs fail with the following cURL error:
-
-```plaintext
-++ git -c 'http.userAgent=gitlab-runner <version>' fetch origin +refs/pipelines/<id>:refs/pipelines/<id> ...
-error: RPC failed; curl 16 HTTP/2 send again with decreased length
-fatal: ...
-```
-
-You can work around this problem by configuring Git and `libcurl` to
-[use HTTP/1.1](https://git-scm.com/docs/git-config#Documentation/git-config.txt-httpversion).
-The configuration can be added to:
-
-- A job's [`pre_get_sources_script`](yaml/index.md#hookspre_get_sources_script):
-
-  ```yaml
-  job_name:
-    hooks:
-      pre_get_sources_script:
-        - git config --global http.version "HTTP/1.1"
-  ```
-
-- The [runner's `config.toml`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html)
-  with [Git configuration environment variables](https://git-scm.com/docs/git-config#ENVIRONMENT):
-
-  ```toml
-  [[runners]]
-  ...
-  environment = [
-    "GIT_CONFIG_COUNT=1",
-    "GIT_CONFIG_KEY_0=http.version",
-    "GIT_CONFIG_VALUE_0=HTTP/1.1"
-  ]
-  ```
-
-### Merge request pipeline messages
-
-The merge request pipeline widget shows information about the pipeline status in
-a merge request. It's displayed above the [ability to merge status widget](#merge-request-status-messages).
-
-#### "Checking ability to merge automatically" message
-
-There is a [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/229352)
-where a merge request can be stuck with the `Checking ability to merge automatically`
-message.
-
-If your merge request has this message and it does not disappear after a few minutes,
-you can try one of these workarounds:
-
-- Refresh the merge request page.
-- Close & Re-open the merge request.
-- Rebase the merge request with the `/rebase` [quick action](../user/project/quick_actions.md).
-- If you have already confirmed the merge request is ready to be merged, you can merge
-  it with the `/merge` quick action.
-
-#### "Checking pipeline status" message
-
-This message is shown when the merge request has no pipeline associated with the
-latest commit yet. This might be because:
-
-- GitLab hasn't finished creating the pipeline yet.
-- You are using an external CI service and GitLab hasn't heard back from the service yet.
-- You are not using CI/CD pipelines in your project.
-- You are using CI/CD pipelines in your project, but your configuration prevented a pipeline from running on the source branch for your merge request.
-- The latest pipeline was deleted (this is a [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/214323)).
-- The source branch of the merge request is on a private fork.
-
-After the pipeline is created, the message updates with the pipeline status.
-
-### Merge request status messages
-
-The merge request status widget shows:
-
-- If the merge request is ready to merge. If the merge request can't be merged, the reason is displayed.
-- **Merge**, if the pipeline is complete, or **Set to auto-merge** if the pipeline is still running.
-
-#### "A CI/CD pipeline must run and be successful before merge" message
-
-This message is shown if the [Pipelines must succeed](../user/project/merge_requests/merge_when_pipeline_succeeds.md#require-a-successful-pipeline-for-merge)
-setting is enabled in the project and a pipeline has not yet run successfully.
-This also applies if the pipeline has not been created yet, or if you are waiting
-for an external CI service. If you don't use pipelines for your project, then you
-should disable **Pipelines must succeed** so you can accept merge requests.
-
-#### "Merge blocked: pipeline must succeed. Push a new commit that fixes the failure" message
-
-This message is shown if the [merge request pipeline](pipelines/merge_request_pipelines.md),
-[merged results pipeline](pipelines/merged_results_pipelines.md),
-or [merge train pipeline](pipelines/merge_trains.md)
-has failed or been canceled.
-This does not happen when a basic branch pipeline fails.
-
-If a merge request pipeline or merged result pipeline was canceled or failed, you can:
-
-- Re-run the entire pipeline by selecting **Run pipeline** in the pipeline tab in the merge request.
-- [Retry only the jobs that failed](pipelines/index.md#view-pipelines). If you re-run the entire pipeline, this is not necessary.
-- Push a new commit to fix the failure.
-
-If the merge train pipeline has failed, you can:
-
-- Check the failure and determine if you can use the [`/merge` quick action](../user/project/quick_actions.md) to immediately add the merge request to the train again.
-- Re-run the entire pipeline by selecting **Run pipeline** in the pipeline tab in the merge request, then add the merge request to the train again.
-- Push a commit to fix the failure, then add the merge request to the train again.
-
-If the merge train pipeline was canceled before the merge request was merged, without a failure, you can:
-
-- Add it to the train again.
-
-### Merge request rules widget shows a scan result policy is invalid or duplicated **(ULTIMATE SELF)**
-
-On GitLab self-managed from 15.0 to 16.4, the most likely cause is that the project was exported from a
-group and imported into another, and had scan result policy rules. These rules are stored in a
-separate project to the one that was exported. As a result, the project contains policy rules that
-reference entities that don't exist in the imported project's group. The result is policy rules that
-are invalid, duplicated, or both.
-
-To remove all invalid scan result policy rules from a GitLab instance, an administrator can run
-the following script in the [Rails console](../administration/operations/rails_console.md).
-
-```ruby
-Project.joins(:approval_rules).where(approval_rules: { report_type: %i[scan_finding license_scanning] }).where.not(approval_rules: { security_orchestration_policy_configuration_id: nil }).find_in_batches.flat_map do |batch|
-  batch.map do |project|
-    # Get projects and their configuration_ids for applicable project rules
-    [project, project.approval_rules.where(report_type: %i[scan_finding license_scanning]).pluck(:security_orchestration_policy_configuration_id).uniq]
-  end.uniq.map do |project, configuration_ids| # We take only unique combinations of project + configuration_ids
-    # If we find more configurations than what is available for the project, we take records with the extra configurations
-    [project, configuration_ids - project.all_security_orchestration_policy_configurations.pluck(:id)]
-  end.select { |_project, configuration_ids| configuration_ids.any? }
-end.each do |project, configuration_ids|
-  # For each found pair project + ghost configuration, we remove these rules for a given project
-  Security::OrchestrationPolicyConfiguration.where(id: configuration_ids).each do |configuration|
-    configuration.delete_scan_finding_rules_for_project(project.id)
-  end
-  # Ensure we sync any potential rules from new group's policy
-  Security::ScanResultPolicies::SyncProjectWorker.perform_async(project.id)
-end
-```
-
-### Project `group/project` not found or access denied
-
-This message is shown if configuration is added with [`include`](yaml/index.md#include) and one of the following:
-
-- The configuration refers to a project that can't be found.
-- The user that is running the pipeline is unable to access any included projects.
-
-To resolve this, check that:
-
-- The path of the project is in the format `my-group/my-project` and does not include
-  any folders in the repository.
-- The user running the pipeline is a [member of the projects](../user/project/members/index.md#add-users-to-a-project)
-  that contain the included files. Users must also have the [permission](../user/permissions.md#job-permissions)
-  to run CI/CD jobs in the same projects.
-
-### "The parsed YAML is too big" message
-
-This message displays when the YAML configuration is too large or nested too deeply.
-YAML files with a large number of includes, and thousands of lines overall, are
-more likely to hit this memory limit. For example, a YAML file that is 200kb is
-likely to hit the default memory limit.
-
-To reduce the configuration size, you can:
-
-- Check the length of the expanded CI/CD configuration in the pipeline editor's
-  [Full configuration](pipeline_editor/index.md#view-full-configuration) tab. Look for
-  duplicated configuration that can be removed or simplified.
-- Move long or repeated `script` sections into standalone scripts in the project.
-- Use [parent and child pipelines](pipelines/downstream_pipelines.md#parent-child-pipelines) to move some
-  work to jobs in an independent child pipeline.
-
-On a self-managed instance, you can [increase the size limits](../administration/instance_limits.md#maximum-size-and-depth-of-cicd-configuration-yaml-files).
-
-### Error 500 when editing the `.gitlab-ci.yml` file
-
-A [loop of included configuration files](pipeline_editor/index.md#configuration-validation-currently-not-available-message)
-can cause a `500` error when editing the `.gitlab-ci.yml` file with the [web editor](../user/project/repository/web_editor.md).
-
-### A CI/CD job does not use newer configuration when run again
-
-The configuration for a pipeline is only fetched when the pipeline is created.
-When you rerun a job, uses the same configuration each time. If you update configuration files,
-including separate files added with [`include`](yaml/index.md#include), you must
-start a new pipeline to use the new configuration.
-
-### Unable to pull image from another project
-
-> **Allow access to this project with a CI_JOB_TOKEN** setting [renamed to **Limit access _to_ this project**](https://gitlab.com/gitlab-org/gitlab/-/issues/411406) in GitLab 16.3.
-
-When a runner tries to pull an image from a private project, the job could fail with the following error:
-
-```shell
-WARNING: Failed to pull image with policy "always": Error response from daemon: pull access denied for registry.example.com/path/to/project, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
-```
-
-This error can happen if the following are both true:
-
-- The **Limit access _to_ this project** option is enabled in the private project
-  hosting the image.
-- The job attempting to fetch the image is running for a project that is not listed in
-  the private project's allowlist.
-
-The recommended solution is to [add your project to the private project's job token scope allowlist](jobs/ci_job_token.md#add-a-project-to-the-job-token-scope-allowlist).
-
-## Pipeline warnings
-
-Pipeline configuration warnings are shown when you:
-
-- [Validate configuration with the CI Lint tool](yaml/index.md).
-- [Manually run a pipeline](pipelines/index.md#run-a-pipeline-manually).
-
-### "Job may allow multiple pipelines to run for a single action" warning
-
-When you use [`rules`](yaml/index.md#rules) with a `when` clause without an `if`
-clause, multiple pipelines may run. Usually this occurs when you push a commit to
-a branch that has an open merge request associated with it.
-
-To [prevent duplicate pipelines](jobs/job_control.md#avoid-duplicate-pipelines), use
-[`workflow: rules`](yaml/index.md#workflow) or rewrite your rules to control
-which pipelines can run.
-
-### Console workaround if job using `resource_group` gets stuck **(FREE SELF)**
-
-```ruby
-# find resource group by name
-resource_group = Project.find_by_full_path('...').resource_groups.find_by(key: 'the-group-name')
-busy_resources = resource_group.resources.where('build_id IS NOT NULL')
-
-# identify which builds are occupying the resource
-# (I think it should be 1 as of today)
-busy_resources.pluck(:build_id)
-
-# it's good to check why this build is holding the resource.
-# Is it stuck? Has it been forcefully dropped by the system?
-# free up busy resources
-busy_resources.update_all(build_id: nil)
-```
-
-### Job log slow to update
-
-When you visit the job log page for a running job, there could be a delay of up to
-60 seconds before the log updates. The default refresh time is 60 seconds, but after
-the log is viewed in the UI, the following log updates should occur every 3 seconds.
-
-## Disaster recovery
-
-You can disable some important but computationally expensive parts of the application
-to relieve stress on the database during ongoing downtime.
-
-### Disable fair scheduling on shared runners
-
-When clearing a large backlog of jobs, you can temporarily enable the `ci_queueing_disaster_recovery_disable_fair_scheduling`
-[feature flag](../administration/feature_flags.md). This flag disables fair scheduling
-on shared runners, which reduces system resource usage on the `jobs/request` endpoint.
-
-When enabled, jobs are processed in the order they were put in the system, instead of
-balanced across many projects.
-
-### Disable compute quota enforcement
-
-To disable the enforcement of [compute quotas](pipelines/cicd_minutes.md) on shared runners, you can temporarily
-enable the `ci_queueing_disaster_recovery_disable_quota` [feature flag](../administration/feature_flags.md).
-This flag reduces system resource usage on the `jobs/request` endpoint.
-
-When enabled, jobs created in the last hour can run in projects which are out of quota.
-Earlier jobs are already canceled by a periodic background worker (`StuckCiJobsWorker`).
-
-## CI/CD troubleshooting Rails console commands
-
-The following commands are run in the [Rails console](../administration/operations/rails_console.md#starting-a-rails-console-session).
-
-WARNING:
-Any command that changes data directly could be damaging if not run correctly, or under the right conditions.
-We highly recommend running them in a test environment with a backup of the instance ready to be restored, just in case.
-
-### Cancel stuck pending pipelines
-
-```ruby
-project = Project.find_by_full_path('<project_path>')
-Ci::Pipeline.where(project_id: project.id).where(status: 'pending').count
-Ci::Pipeline.where(project_id: project.id).where(status: 'pending').each {|p| p.cancel if p.stuck?}
-Ci::Pipeline.where(project_id: project.id).where(status: 'pending').count
-```
-
-### Try merge request integration
-
-```ruby
-project = Project.find_by_full_path('<project_path>')
-mr = project.merge_requests.find_by(iid: <merge_request_iid>)
-mr.project.try(:ci_integration)
-```
-
-### Validate the `.gitlab-ci.yml` file
-
-```ruby
-project = Project.find_by_full_path('<project_path>')
-content = p.ci_config_for(project.repository.root_ref_sha)
-Gitlab::Ci::Lint.new(project: project,  current_user: User.first).validate(content)
-```
-
-### Disable AutoDevOps on Existing Projects
-
-```ruby
-Project.all.each do |p|
-  p.auto_devops_attributes={"enabled"=>"0"}
-  p.save
-end
-```
-
-### Obtain runners registration token
-
-```ruby
-Gitlab::CurrentSettings.current_application_settings.runners_registration_token
-```
-
-### Seed runners registration token
-
-```ruby
-appSetting = Gitlab::CurrentSettings.current_application_settings
-appSetting.set_runners_registration_token('<new-runners-registration-token>')
-appSetting.save!
-```
-
-### Run pipeline schedules manually
-
-You can run pipeline schedules manually through the Rails console to reveal any errors that are usually not visible.
-
-```ruby
-# schedule_id can be obtained from Edit Pipeline Schedule page
-schedule = Ci::PipelineSchedule.find_by(id: <schedule_id>)
-
-# Select the user that you want to run the schedule for
-user = User.find_by_username('<username>')
-
-# Run the schedule
-ps = Ci::CreatePipelineService.new(schedule.project, user, ref: schedule.ref).execute!(:schedule, ignore_skip_ci: true, save_on_errors: false, schedule: schedule)
-```
-
-## How to get help
-
-If you are unable to resolve pipeline issues, you can get help from:
-
-- The [GitLab community forum](https://forum.gitlab.com/)
-- GitLab [Support](https://about.gitlab.com/support/)
+<!-- This redirect file can be deleted after <2024-02-01>. -->
+<!-- Redirects that point to other docs in the same project expire in three months. -->
+<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. -->
+<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
diff --git a/doc/ci/yaml/index.md b/doc/ci/yaml/index.md
index ef573f6f3df..ed1bc190a4d 100644
--- a/doc/ci/yaml/index.md
+++ b/doc/ci/yaml/index.md
@@ -2026,7 +2026,7 @@ Use `hooks:pre_get_sources_script` to specify a list of commands to execute on t
 before cloning the Git repository and any submodules.
 You can use it for example to:
 
-- Adjust the [Git configuration](../troubleshooting.md#get_sources-job-section-fails-because-of-an-http2-problem).
+- Adjust the [Git configuration](../jobs/index.md#get_sources-job-section-fails-because-of-an-http2-problem).
 - Export [tracing variables](../../topics/git/useful_git_commands.md).
 
 **Possible inputs**: An array including:
diff --git a/doc/development/index.md b/doc/development/index.md
index 71ab54c8a73..abc19645ecb 100644
--- a/doc/development/index.md
+++ b/doc/development/index.md
@@ -10,7 +10,7 @@ description: "Development Guidelines: learn how to contribute to GitLab."
 
 Learn how to contribute to the development of the GitLab product.
 
-This content is intended for GitLab team members as well as members of the wider community.
+This content is intended for both GitLab team members and members of the wider community.
 
 - [Contribute to GitLab development](contributing/index.md)
 - [Contribute to GitLab Runner development](https://docs.gitlab.com/runner/development/)
diff --git a/doc/user/ai_features.md b/doc/user/ai_features.md
index de081a7f620..685f1dbd99b 100644
--- a/doc/user/ai_features.md
+++ b/doc/user/ai_features.md
@@ -12,18 +12,18 @@ GitLab is creating AI-assisted features across our DevSecOps platform. These fea
 | Feature | Purpose | Large Language Model | Current availability | Maturity |
 |-|-|-|-|-|
 | [Suggested Reviewers](project/merge_requests/reviews/index.md#gitlab-duo-suggested-reviewers) | Assists in creating faster and higher-quality reviews by automatically suggesting reviewers for your merge request. | GitLab creates a machine learning model for each project, which is used to generate reviewers <br><br> [View the issue](https://gitlab.com/gitlab-org/modelops/applied-ml/applied-ml-updates/-/issues/10) | SaaS only <br><br> Ultimate tier | [Generally Available (GA)](../policy/experiment-beta-support.md#generally-available-ga) |
-| [Code Suggestions](project/repository/code_suggestions/index.md) | Helps you write code more efficiently by viewing code suggestions as you type. | For Code Completion: Vertext AI Codey [`code-gecko`](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/code-completion) and Anthropic [`Claude-instant-1.2`](https://docs.anthropic.com/claude/reference/selecting-a-model) <br><br> For Code Generation: Vertext AI Codey [`code-bison`](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/code-generation) and Anthropic [`Claude-2`](https://docs.anthropic.com/claude/reference/selecting-a-model)| [SaaS: All tiers](project/repository/code_suggestions/saas.md) <br><br> [Self-managed: Premium and Ultimate with Cloud Licensing](project/repository/code_suggestions/self_managed.md) | [Beta](../policy/experiment-beta-support.md#beta) |
-| [Vulnerability summary](application_security/vulnerabilities/index.md#explaining-a-vulnerability) | Helps you remediate vulnerabilities more efficiently, boost your skills, and write more secure code. | Vertext AI Codey [`text-bison`](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/text) <br><br> Anthropic [`Claude-2`](https://docs.anthropic.com/claude/reference/selecting-a-model) if degraded performance | SaaS only <br><br> Ultimate tier | [Beta](../policy/experiment-beta-support.md#beta) |
-| [Code explanation](#explain-code-in-the-web-ui-with-code-explanation) | Helps you understand code by explaining it in English language. | Vertext AI Codey [`codechat-bison`](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/code-chat) | SaaS only <br><br> Ultimate tier | [Experiment](../policy/experiment-beta-support.md#experiment) |
-| [GitLab Duo Chat](gitlab_duo_chat.md) | Process and generate text and code in a conversational manner. Helps you quickly identify useful information in large volumes of text in issues, epics, code, and GitLab documentation. | Anthropic [`Claude-2`](https://docs.anthropic.com/claude/reference/selecting-a-model) <br><br> Vertext AI Codey [`textembedding-gecko`](https://cloud.google.com/vertex-ai/docs/generative-ai/embeddings/get-text-embeddings) | SaaS only <br><br> Ultimate tier | [Experiment](../policy/experiment-beta-support.md#experiment) |
+| [Code Suggestions](project/repository/code_suggestions/index.md) | Helps you write code more efficiently by viewing code suggestions as you type. | For Code Completion: Vertex AI Codey [`code-gecko`](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/code-completion) and Anthropic [`Claude-instant-1.2`](https://docs.anthropic.com/claude/reference/selecting-a-model) <br><br> For Code Generation: Vertex AI Codey [`code-bison`](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/code-generation) and Anthropic [`Claude-2`](https://docs.anthropic.com/claude/reference/selecting-a-model)| [SaaS: All tiers](project/repository/code_suggestions/saas.md) <br><br> [Self-managed: Premium and Ultimate with Cloud Licensing](project/repository/code_suggestions/self_managed.md) | [Beta](../policy/experiment-beta-support.md#beta) |
+| [Vulnerability summary](application_security/vulnerabilities/index.md#explaining-a-vulnerability) | Helps you remediate vulnerabilities more efficiently, boost your skills, and write more secure code. | Vertex AI Codey [`text-bison`](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/text) <br><br> Anthropic [`Claude-2`](https://docs.anthropic.com/claude/reference/selecting-a-model) if degraded performance | SaaS only <br><br> Ultimate tier | [Beta](../policy/experiment-beta-support.md#beta) |
+| [Code explanation](#explain-code-in-the-web-ui-with-code-explanation) | Helps you understand code by explaining it in English language. | Vertex AI Codey [`codechat-bison`](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/code-chat) | SaaS only <br><br> Ultimate tier | [Experiment](../policy/experiment-beta-support.md#experiment) |
+| [GitLab Duo Chat](gitlab_duo_chat.md) | Process and generate text and code in a conversational manner. Helps you quickly identify useful information in large volumes of text in issues, epics, code, and GitLab documentation. | Anthropic [`Claude-2`](https://docs.anthropic.com/claude/reference/selecting-a-model) <br><br> Vertex AI Codey [`textembedding-gecko`](https://cloud.google.com/vertex-ai/docs/generative-ai/embeddings/get-text-embeddings) | SaaS only <br><br> Ultimate tier | [Experiment](../policy/experiment-beta-support.md#experiment) |
 | [Value stream forecasting](#forecast-deployment-frequency-with-value-stream-forecasting) | Assists you with predicting productivity metrics and identifying anomalies across your software development lifecycle. | Statistical forecasting | SaaS only <br> Self-managed <br><br> Ultimate tier | [Experiment](../policy/experiment-beta-support.md#experiment) |
-| [Discussion summary](#summarize-issue-discussions-with-discussion-summary) | Assists with quickly getting everyone up to speed on lengthy conversations to help ensure you are all on the same page. | Vertext AI Codey [`text-bison`](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/text) | SaaS only <br><br> Ultimate tier | [Experiment](../policy/experiment-beta-support.md#experiment) |
-| [Merge request summary](project/merge_requests/ai_in_merge_requests.md#summarize-merge-request-changes) | Efficiently communicate the impact of your merge request changes. | Vertext AI Codey [`text-bison`](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/text) | SaaS only <br><br> Ultimate tier | [Experiment](../policy/experiment-beta-support.md#experiment) |
-| [Code review summary](project/merge_requests/ai_in_merge_requests.md#summarize-my-merge-request-review) | Helps ease merge request handoff between authors and reviewers and help reviewers efficiently understand suggestions. | Vertext AI Codey [`text-bison`](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/text) | SaaS only <br><br> Ultimate tier | [Experiment](../policy/experiment-beta-support.md#experiment) |
-| [Merge request template population](project/merge_requests/ai_in_merge_requests.md#fill-in-merge-request-templates) | Generate a description for the merge request based on the contents of the template. | Vertext AI Codey [`text-bison`](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/text) | SaaS only <br><br> Ultimate tier | [Experiment](../policy/experiment-beta-support.md#experiment) |
-| [Test generation](project/merge_requests/ai_in_merge_requests.md#generate-suggested-tests-in-merge-requests) | Automates repetitive tasks and helps catch bugs early. | Vertext AI Codey [`text-bison`](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/text) | SaaS only <br><br> Ultimate tier | [Experiment](../policy/experiment-beta-support.md#experiment) |
-| [Git suggestions](https://gitlab.com/gitlab-org/gitlab/-/issues/409636) | Helps you discover or recall Git commands when and where you need them. | Vertext AI Codey [`codechat-bison`](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/code-chat) | SaaS only <br><br> Ultimate tier | [Experiment](../policy/experiment-beta-support.md#experiment) |
-| [Root cause analysis](#root-cause-analysis) | Assists you in determining the root cause for a pipeline failure and failed CI/CD build. | Vertext AI Codey [`text-bison`](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/text) | SaaS only <br><br> Ultimate tier | [Experiment](../policy/experiment-beta-support.md#experiment) |
+| [Discussion summary](#summarize-issue-discussions-with-discussion-summary) | Assists with quickly getting everyone up to speed on lengthy conversations to help ensure you are all on the same page. | Vertex AI Codey [`text-bison`](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/text) | SaaS only <br><br> Ultimate tier | [Experiment](../policy/experiment-beta-support.md#experiment) |
+| [Merge request summary](project/merge_requests/ai_in_merge_requests.md#summarize-merge-request-changes) | Efficiently communicate the impact of your merge request changes. | Vertex AI Codey [`text-bison`](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/text) | SaaS only <br><br> Ultimate tier | [Experiment](../policy/experiment-beta-support.md#experiment) |
+| [Code review summary](project/merge_requests/ai_in_merge_requests.md#summarize-my-merge-request-review) | Helps ease merge request handoff between authors and reviewers and help reviewers efficiently understand suggestions. | Vertex AI Codey [`text-bison`](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/text) | SaaS only <br><br> Ultimate tier | [Experiment](../policy/experiment-beta-support.md#experiment) |
+| [Merge request template population](project/merge_requests/ai_in_merge_requests.md#fill-in-merge-request-templates) | Generate a description for the merge request based on the contents of the template. | Vertex AI Codey [`text-bison`](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/text) | SaaS only <br><br> Ultimate tier | [Experiment](../policy/experiment-beta-support.md#experiment) |
+| [Test generation](project/merge_requests/ai_in_merge_requests.md#generate-suggested-tests-in-merge-requests) | Automates repetitive tasks and helps catch bugs early. | Vertex AI Codey [`text-bison`](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/text) | SaaS only <br><br> Ultimate tier | [Experiment](../policy/experiment-beta-support.md#experiment) |
+| [Git suggestions](https://gitlab.com/gitlab-org/gitlab/-/issues/409636) | Helps you discover or recall Git commands when and where you need them. | Vertex AI Codey [`codechat-bison`](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/code-chat) | SaaS only <br><br> Ultimate tier | [Experiment](../policy/experiment-beta-support.md#experiment) |
+| [Root cause analysis](#root-cause-analysis) | Assists you in determining the root cause for a pipeline failure and failed CI/CD build. | Vertex AI Codey [`text-bison`](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/text) | SaaS only <br><br> Ultimate tier | [Experiment](../policy/experiment-beta-support.md#experiment) |
 | [Issue description generation](#summarize-an-issue-with-issue-description-generation) | Generate issue descriptions. | OpenAI's [`GPT-3`](https://platform.openai.com/docs/models/gpt-3) | SaaS only <br><br> Ultimate tier | [Experiment](../policy/experiment-beta-support.md#experiment) |
 
 ## Enable AI/ML features
diff --git a/doc/user/application_security/policies/scan-result-policies.md b/doc/user/application_security/policies/scan-result-policies.md
index 7582dd8bca3..d70463f4056 100644
--- a/doc/user/application_security/policies/scan-result-policies.md
+++ b/doc/user/application_security/policies/scan-result-policies.md
@@ -322,3 +322,35 @@ We have identified in [epic 11020](https://gitlab.com/groups/gitlab-org/-/epics/
 - When using `newly_detected`, some findings may require approval when they are not introduced by the merge request (such as a new CVE on a related dependency). We currently use `main tip` of the target branch for comparison. In the future, we plan to use `merge base` for `newly_detected` policies (see [issue 428518](https://gitlab.com/gitlab-org/gitlab/-/issues/428518)).
 - Findings or errors that cause approval to be required on a scan result policy may not be evident in the Security MR Widget. By using `merge base` in [issue 428518](https://gitlab.com/gitlab-org/gitlab/-/issues/428518) some cases will be addressed. We will additionally be [displaying more granular details](https://gitlab.com/groups/gitlab-org/-/epics/11185) about what caused security policy violations.
 - Security policy violations are distinct compared to findings displayed in the MR widgets. Some violations may not be present in the MR widget. We are working to harmonize our features in [epic 11020](https://gitlab.com/groups/gitlab-org/-/epics/11020) and to display policy violations explicitly in merge requests in [epic 11185](https://gitlab.com/groups/gitlab-org/-/epics/11185).
+
+## Troubleshooting
+
+### Merge request rules widget shows a scan result policy is invalid or duplicated **(ULTIMATE SELF)**
+
+On GitLab self-managed from 15.0 to 16.4, the most likely cause is that the project was exported from a
+group and imported into another, and had scan result policy rules. These rules are stored in a
+separate project to the one that was exported. As a result, the project contains policy rules that
+reference entities that don't exist in the imported project's group. The result is policy rules that
+are invalid, duplicated, or both.
+
+To remove all invalid scan result policy rules from a GitLab instance, an administrator can run
+the following script in the [Rails console](../../../administration/operations/rails_console.md).
+
+```ruby
+Project.joins(:approval_rules).where(approval_rules: { report_type: %i[scan_finding license_scanning] }).where.not(approval_rules: { security_orchestration_policy_configuration_id: nil }).find_in_batches.flat_map do |batch|
+  batch.map do |project|
+    # Get projects and their configuration_ids for applicable project rules
+    [project, project.approval_rules.where(report_type: %i[scan_finding license_scanning]).pluck(:security_orchestration_policy_configuration_id).uniq]
+  end.uniq.map do |project, configuration_ids| # We take only unique combinations of project + configuration_ids
+    # If we find more configurations than what is available for the project, we take records with the extra configurations
+    [project, configuration_ids - project.all_security_orchestration_policy_configurations.pluck(:id)]
+  end.select { |_project, configuration_ids| configuration_ids.any? }
+end.each do |project, configuration_ids|
+  # For each found pair project + ghost configuration, we remove these rules for a given project
+  Security::OrchestrationPolicyConfiguration.where(id: configuration_ids).each do |configuration|
+    configuration.delete_scan_finding_rules_for_project(project.id)
+  end
+  # Ensure we sync any potential rules from new group's policy
+  Security::ScanResultPolicies::SyncProjectWorker.perform_async(project.id)
+end
+```
diff --git a/doc/user/product_analytics/instrumentation/browser_sdk.md b/doc/user/product_analytics/instrumentation/browser_sdk.md
index 912e157f67c..f2beafab8e0 100644
--- a/doc/user/product_analytics/instrumentation/browser_sdk.md
+++ b/doc/user/product_analytics/instrumentation/browser_sdk.md
@@ -161,6 +161,15 @@ glClient.page(eventAttributes);
 | :---------------- | :-------------------------- | :---------------------------------------------------------------- |
 | `eventAttributes` | `Object`/`Null`/`undefined` | The event attributes that need to be added to the pageview event. |
 
+The `eventAttributes` object supports the following optional properties:
+
+| Property         | Type                        | Description                                                                   |
+| :--------------- | :-------------------------- | :---------------------------------------------------------------------------- |
+| `title`          | `String`                    | Override the default page title. |
+| `contextCallback` | `Function` | A callback that fires on the page view. |
+| `context` | `Object` | Add context (additional information) on the page view. |
+| `timestamp` | `timestamp` | Set the true timestamp or overwrite the device-sent timestamp on an event. |
+
 ### `track`
 
 Used to trigger a custom event.