Add latest changes from gitlab-org/gitlab@master

3 files changed, 114 insertions, 17 deletions

diff --git a/doc/administration/cicd.md b/doc/administration/cicd.md
index 47cbd7d1aec..9c69a7cbb65 100644
--- a/doc/administration/cicd.md
+++ b/doc/administration/cicd.md
@@ -93,14 +93,96 @@ To change the frequency of the pipeline schedule worker:
 For example, to set the maximum frequency of pipelines to twice a day, set `pipeline_schedule_worker_cron`
 to a cron value of `0 */12 * * *` (`00:00` and `12:00` every day).
 
-<!-- ## Troubleshooting
+## Disaster recovery
 
-Include any troubleshooting steps that you can foresee. If you know beforehand what issues
-one might have when setting this up, or when something is changed, or on upgrading, it's
-important to describe those, too. Think of things that may go wrong and include them here.
-This is important to minimize requests for support, and to avoid doc comments with
-questions that you know someone might ask.
+You can disable some important but computationally expensive parts of the application
+to relieve stress on the database during ongoing downtime.
 
-Each scenario can be a third-level heading, for example `### Getting error message X`.
-If you have none to add when creating a doc, leave this section in place
-but commented out to help encourage others to add to it in the future. -->
+### Disable fair scheduling on shared runners
+
+When clearing a large backlog of jobs, you can temporarily enable the `ci_queueing_disaster_recovery_disable_fair_scheduling`
+[feature flag](../administration/feature_flags.md). This flag disables fair scheduling
+on shared runners, which reduces system resource usage on the `jobs/request` endpoint.
+
+When enabled, jobs are processed in the order they were put in the system, instead of
+balanced across many projects.
+
+### Disable compute quota enforcement
+
+To disable the enforcement of [compute quotas](../ci/pipelines/cicd_minutes.md) on shared runners, you can temporarily
+enable the `ci_queueing_disaster_recovery_disable_quota` [feature flag](../administration/feature_flags.md).
+This flag reduces system resource usage on the `jobs/request` endpoint.
+
+When enabled, jobs created in the last hour can run in projects which are out of quota.
+Earlier jobs are already canceled by a periodic background worker (`StuckCiJobsWorker`).
+
+## CI/CD troubleshooting Rails console commands
+
+The following commands are run in the [Rails console](../administration/operations/rails_console.md#starting-a-rails-console-session).
+
+WARNING:
+Any command that changes data directly could be damaging if not run correctly, or under the right conditions.
+We highly recommend running them in a test environment with a backup of the instance ready to be restored, just in case.
+
+### Cancel stuck pending pipelines
+
+```ruby
+project = Project.find_by_full_path('<project_path>')
+Ci::Pipeline.where(project_id: project.id).where(status: 'pending').count
+Ci::Pipeline.where(project_id: project.id).where(status: 'pending').each {|p| p.cancel if p.stuck?}
+Ci::Pipeline.where(project_id: project.id).where(status: 'pending').count
+```
+
+### Try merge request integration
+
+```ruby
+project = Project.find_by_full_path('<project_path>')
+mr = project.merge_requests.find_by(iid: <merge_request_iid>)
+mr.project.try(:ci_integration)
+```
+
+### Validate the `.gitlab-ci.yml` file
+
+```ruby
+project = Project.find_by_full_path('<project_path>')
+content = p.ci_config_for(project.repository.root_ref_sha)
+Gitlab::Ci::Lint.new(project: project,  current_user: User.first).validate(content)
+```
+
+### Disable AutoDevOps on Existing Projects
+
+```ruby
+Project.all.each do |p|
+  p.auto_devops_attributes={"enabled"=>"0"}
+  p.save
+end
+```
+
+### Obtain runners registration token
+
+```ruby
+Gitlab::CurrentSettings.current_application_settings.runners_registration_token
+```
+
+### Seed runners registration token
+
+```ruby
+appSetting = Gitlab::CurrentSettings.current_application_settings
+appSetting.set_runners_registration_token('<new-runners-registration-token>')
+appSetting.save!
+```
+
+### Run pipeline schedules manually
+
+You can run pipeline schedules manually through the Rails console to reveal any errors that are usually not visible.
+
+```ruby
+# schedule_id can be obtained from Edit Pipeline Schedule page
+schedule = Ci::PipelineSchedule.find_by(id: <schedule_id>)
+
+# Select the user that you want to run the schedule for
+user = User.find_by_username('<username>')
+
+# Run the schedule
+ps = Ci::CreatePipelineService.new(schedule.project, user, ref: schedule.ref).execute!(:schedule, ignore_skip_ci: true, save_on_errors: false, schedule: schedule)
+```
diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md
index 44aa3d648ad..e2fc425b536 100644
--- a/doc/administration/reference_architectures/index.md
+++ b/doc/administration/reference_architectures/index.md
@@ -191,13 +191,22 @@ Before implementing a reference architecture, refer to the following requirement
 These reference architectures were built and tested on Google Cloud Platform (GCP) using the
 [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms)
 CPU platform as a lowest common denominator baseline ([Sysbench benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks)).
+Newer, similarly-sized CPUs are supported and may have improved performance as a result.
 
-Newer, similarly-sized CPUs are supported and may have improved performance as a result. For Linux package environments,
-ARM-based equivalents are also supported.
+ARM CPUs are supported for Linux package environments as well as for any [Cloud Provider services](#cloud-provider-services) where applicable.
 
 NOTE:
 Any "burstable" instance types are not recommended due to inconsistent performance.
 
+### Supported disk types
+
+As a general guidance, most standard disk types are expected to work for GitLab, but be aware of the following specific call outs:
+
+- [Gitaly](../gitaly/index.md#disk-requirements) requires at least 8,000 input/output operations per second (IOPS) for read operations, and 2,000 IOPS for write operations.
+- We don't recommend the use of any disk types that are "burstable" due to inconsistent performance.
+
+Outside the above standard, disk types are expected to work for GitLab and the choice of each depends on your specific requirements around areas, such as durability or costs.
+
 ### Supported infrastructure
 
 As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP, Azure) and
@@ -356,6 +365,12 @@ If you choose to use a third party external service:
 
 Redis is primarily single threaded. For the 10,000 user and above Reference Architectures, separate out the instances as specified into Cache and Persistent data to achieve optimum performance at this scale.
 
+### Recommendation notes for Object Storage
+
+GitLab has been tested against [various Object Storage providers](../object_storage.md#supported-object-storage-providers) that are expected to work.
+
+As a general guidance, it's recommended to use a reputable solution that has full S3 compatibility.
+
 #### Unsupported database services
 
 Several database cloud provider services are known not to support the above or have been found to have other issues and aren't recommended:
@@ -663,8 +678,8 @@ Most setups would only need vertical scaling, but there are some specific areas
 Conversely, if you have robust metrics in place that show the environment is over-provisioned, you can apply the same process for
 scaling downwards. You should take an iterative approach when scaling downwards, however, to ensure there are no issues.
 
-### How to monitor your environment
+### Monitoring
+
+There are numerous options available to monitor your infrastructure, as well as [GitLab itself](../monitoring/index.md), and you should refer to your chosen monitoring solution's documentation for more information.
 
-To monitor your GitLab environment, you can use the tools
-[bundled with GitLab](../monitoring/index.md), but it's also possible to use third-party
-options if desired.
+Of note, the GitLab application is bundled with [Prometheus as well as various Prometheus compatible exporters](../monitoring/prometheus/index.md) that could be hooked into your solution.
diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md
index 9432836c22b..01c75c32366 100644
--- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md
+++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md
@@ -46,11 +46,11 @@ This content has been moved to [Troubleshooting Repository mirroring](../../user
 
 ## CI
 
-This content has been moved to [Troubleshooting CI/CD](../../ci/troubleshooting.md).
+This content has been moved to [Troubleshooting CI/CD](../cicd.md#cicd-troubleshooting-rails-console-commands).
 
 ## License
 
-This content has been moved to [Activate GitLab EE with a license file or key](../../administration/license_file.md).
+This content has been moved to [Activate GitLab EE with a license file or key](../license_file.md).
 
 ## Registry