diff options
Diffstat (limited to 'doc/administration/cicd.md')
-rw-r--r-- | doc/administration/cicd.md | 100 |
1 files changed, 91 insertions, 9 deletions
diff --git a/doc/administration/cicd.md b/doc/administration/cicd.md index 47cbd7d1aec..9c69a7cbb65 100644 --- a/doc/administration/cicd.md +++ b/doc/administration/cicd.md @@ -93,14 +93,96 @@ To change the frequency of the pipeline schedule worker: For example, to set the maximum frequency of pipelines to twice a day, set `pipeline_schedule_worker_cron` to a cron value of `0 */12 * * *` (`00:00` and `12:00` every day). -<!-- ## Troubleshooting +## Disaster recovery -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. +You can disable some important but computationally expensive parts of the application +to relieve stress on the database during ongoing downtime. -Each scenario can be a third-level heading, for example `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +### Disable fair scheduling on shared runners + +When clearing a large backlog of jobs, you can temporarily enable the `ci_queueing_disaster_recovery_disable_fair_scheduling` +[feature flag](../administration/feature_flags.md). This flag disables fair scheduling +on shared runners, which reduces system resource usage on the `jobs/request` endpoint. + +When enabled, jobs are processed in the order they were put in the system, instead of +balanced across many projects. + +### Disable compute quota enforcement + +To disable the enforcement of [compute quotas](../ci/pipelines/cicd_minutes.md) on shared runners, you can temporarily +enable the `ci_queueing_disaster_recovery_disable_quota` [feature flag](../administration/feature_flags.md). +This flag reduces system resource usage on the `jobs/request` endpoint. + +When enabled, jobs created in the last hour can run in projects which are out of quota. +Earlier jobs are already canceled by a periodic background worker (`StuckCiJobsWorker`). + +## CI/CD troubleshooting Rails console commands + +The following commands are run in the [Rails console](../administration/operations/rails_console.md#starting-a-rails-console-session). + +WARNING: +Any command that changes data directly could be damaging if not run correctly, or under the right conditions. +We highly recommend running them in a test environment with a backup of the instance ready to be restored, just in case. + +### Cancel stuck pending pipelines + +```ruby +project = Project.find_by_full_path('<project_path>') +Ci::Pipeline.where(project_id: project.id).where(status: 'pending').count +Ci::Pipeline.where(project_id: project.id).where(status: 'pending').each {|p| p.cancel if p.stuck?} +Ci::Pipeline.where(project_id: project.id).where(status: 'pending').count +``` + +### Try merge request integration + +```ruby +project = Project.find_by_full_path('<project_path>') +mr = project.merge_requests.find_by(iid: <merge_request_iid>) +mr.project.try(:ci_integration) +``` + +### Validate the `.gitlab-ci.yml` file + +```ruby +project = Project.find_by_full_path('<project_path>') +content = p.ci_config_for(project.repository.root_ref_sha) +Gitlab::Ci::Lint.new(project: project, current_user: User.first).validate(content) +``` + +### Disable AutoDevOps on Existing Projects + +```ruby +Project.all.each do |p| + p.auto_devops_attributes={"enabled"=>"0"} + p.save +end +``` + +### Obtain runners registration token + +```ruby +Gitlab::CurrentSettings.current_application_settings.runners_registration_token +``` + +### Seed runners registration token + +```ruby +appSetting = Gitlab::CurrentSettings.current_application_settings +appSetting.set_runners_registration_token('<new-runners-registration-token>') +appSetting.save! +``` + +### Run pipeline schedules manually + +You can run pipeline schedules manually through the Rails console to reveal any errors that are usually not visible. + +```ruby +# schedule_id can be obtained from Edit Pipeline Schedule page +schedule = Ci::PipelineSchedule.find_by(id: <schedule_id>) + +# Select the user that you want to run the schedule for +user = User.find_by_username('<username>') + +# Run the schedule +ps = Ci::CreatePipelineService.new(schedule.project, user, ref: schedule.ref).execute!(:schedule, ignore_skip_ci: true, save_on_errors: false, schedule: schedule) +``` |