diff options
Diffstat (limited to 'doc/administration/troubleshooting')
16 files changed, 122 insertions, 96 deletions
diff --git a/doc/administration/troubleshooting/debug.md b/doc/administration/troubleshooting/debug.md index 5a8ee1c5c94..6861cdcde4e 100644 --- a/doc/administration/troubleshooting/debug.md +++ b/doc/administration/troubleshooting/debug.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Debugging Tips +# Debugging tips **(FREE SELF)** Sometimes things don't work the way they should. Here are some tips on debugging issues out in production. @@ -26,7 +26,7 @@ You can enable output of Active Record debug logging in the Rails console session by running: ```ruby -ActiveRecord::Base.logger = Logger.new(STDOUT) +ActiveRecord::Base.logger = Logger.new($stdout) ``` This will show information about database queries triggered by any Ruby code @@ -155,7 +155,7 @@ and more. However, this is not enabled by default. To enable it, define the gitlab_rails['env'] = {"ENABLE_RBTRACE" => "1"} ``` -Then reconfigure the system and restart Unicorn and Sidekiq. To run this +Then reconfigure the system and restart Puma and Sidekiq. To run this in Omnibus, run as root: ```ruby @@ -178,7 +178,7 @@ following tips are only recommended if you do NOT mind users being affected by downtime. Otherwise skip to the next section. 1. Load the problematic URL -1. Run `sudo gdb -p <PID>` to attach to the Unicorn process. +1. Run `sudo gdb -p <PID>` to attach to the Puma process. 1. In the GDB window, type: ```plaintext @@ -186,7 +186,7 @@ downtime. Otherwise skip to the next section. ``` 1. This forces the process to generate a Ruby backtrace. Check - `/var/log/gitlab/unicorn/unicorn_stderr.log` for the backtrace. For example, you may see: + `/var/log/gitlab/puma/puma_stderr.log` for the backtrace. For example, you may see: ```plaintext from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:33:in `block in start' @@ -213,16 +213,19 @@ downtime. Otherwise skip to the next section. exit ``` -Note that if the Unicorn process terminates before you are able to run these +Note that if the Puma process terminates before you are able to run these commands, GDB will report an error. To buy more time, you can always raise the -Unicorn timeout. For omnibus users, you can edit `/etc/gitlab/gitlab.rb` and -increase it from 60 seconds to 300: +Puma worker timeout. For omnibus users, you can edit `/etc/gitlab/gitlab.rb` and +increase it from 60 seconds to 600: ```ruby -unicorn['worker_timeout'] = 300 +gitlab_rails['env'] = { + 'GITLAB_RAILS_RACK_TIMEOUT' => 600 +} ``` -For source installations, edit `config/unicorn.rb`. +For source installations, set the environment variable. +Refer to [Puma Worker timeout](https://docs.gitlab.com/omnibus/settings/puma.html#worker-timeout). [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. @@ -267,7 +270,7 @@ is a Unicorn worker that is spinning via `top`. Try to use the `gdb` techniques above. In addition, using `strace` may help isolate issues: ```shell -strace -ttTfyyy -s 1024 -p <PID of unicorn worker> -o /tmp/unicorn.txt +strace -ttTfyyy -s 1024 -p <PID of puma worker> -o /tmp/puma.txt ``` If you cannot isolate which Unicorn worker is the issue, try to run `strace` @@ -275,10 +278,10 @@ on all the Unicorn workers to see where the [`/internal/allowed`](../../development/internal_api.md) endpoint gets stuck: ```shell -ps auwx | grep unicorn | awk '{ print " -p " $2}' | xargs strace -ttTfyyy -s 1024 -o /tmp/unicorn.txt +ps auwx | grep puma | awk '{ print " -p " $2}' | xargs strace -ttTfyyy -s 1024 -o /tmp/puma.txt ``` -The output in `/tmp/unicorn.txt` may help diagnose the root cause. +The output in `/tmp/puma.txt` may help diagnose the root cause. ## More information diff --git a/doc/administration/troubleshooting/defcon.md b/doc/administration/troubleshooting/defcon.md index 09e11553d97..7cae6ea1c8f 100644 --- a/doc/administration/troubleshooting/defcon.md +++ b/doc/administration/troubleshooting/defcon.md @@ -5,21 +5,21 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Disaster Recovery +# Disaster recovery **(FREE SELF)** -This document describes a feature that allows to easily disable some important but computationally -expensive parts of the application, in order to relieve stress on the database in an ongoing downtime. +This document describes a feature that allows you to disable some important but computationally +expensive parts of the application to relieve stress on the database during an ongoing downtime. ## `ci_queueing_disaster_recovery` -This feature flag, if enabled temporarily disables fair scheduling on shared runners. -This can help reduce system resource usage on the `jobs/request` endpoint -by significantly reducing computations being performed. +This feature flag, if temporarily enabled, disables fair scheduling on shared runners. +This can help to reduce system resource usage on the `jobs/request` endpoint +by significantly reducing the computations being performed. Side effects: -- In case of a large backlog of jobs, the jobs will be processed in the order -they were put in the system instead of balancing the jobs across many projects +- In case of a large backlog of jobs, the jobs are processed in the order + they were put in the system, instead of balancing the jobs across many projects. - Projects which are out of quota will be run. This affects -only jobs that were created during the last hour, as prior jobs are canceled -by a periodic background worker (`StuckCiJobsWorker`). + only jobs created during the last hour, as prior jobs are canceled + by a periodic background worker (`StuckCiJobsWorker`). diff --git a/doc/administration/troubleshooting/diagnostics_tools.md b/doc/administration/troubleshooting/diagnostics_tools.md index 27a7493b318..fe85b5d5803 100644 --- a/doc/administration/troubleshooting/diagnostics_tools.md +++ b/doc/administration/troubleshooting/diagnostics_tools.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Diagnostics tools +# Diagnostics tools **(FREE SELF)** These are some of the diagnostics tools the GitLab Support team uses during troubleshooting. They are listed here for transparency, and they may be useful for users with experience diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 588be73e786..92070a86a0d 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -100,7 +100,7 @@ Rails.cache.instance_variable_get(:@data).keys ```ruby # Before 11.6.0 -logger = Logger.new(STDOUT) +logger = Logger.new($stdout) admin_token = User.find_by_username('ADMIN_USERNAME').personal_access_tokens.first.token app.get("URL/?private_token=#{admin_token}") @@ -113,7 +113,7 @@ Gitlab::Profiler.with_user(admin) { app.get(url) } ## Using the GitLab profiler inside console (used as of 10.5) ```ruby -logger = Logger.new(STDOUT) +logger = Logger.new($stdout) admin = User.find_by_username('ADMIN_USERNAME') Gitlab::Profiler.profile('URL', logger: logger, user: admin) ``` @@ -279,6 +279,21 @@ p.each do |project| end ``` +## Bulk update to change all the Jira integrations to Jira instance-level values + +To change all Jira project to use the instance-level integration settings: + +1. In a Rails console: + + ```ruby + jira_service_instance_id = JiraService.find_by(instance: true).id + JiraService.where(active: true, instance: false, template: false, inherit_from_id: nil).find_each do |service| + service.update_attribute(:inherit_from_id, jira_service_instance_id) + end + ``` + +1. Modify and save again the instance-level integration from the UI to propagate the changes to all the group-level and project-level integrations. + ### Bulk update to disable the Slack Notification service To disable notifications for all projects that have Slack service enabled, do: @@ -302,7 +317,18 @@ the displayed size may still show old sizes or commit numbers. To force an updat p = Project.find_by_full_path('<namespace>/<project>') pp p.statistics p.statistics.refresh! -pp p.statistics # compare with earlier values +pp p.statistics +# compare with earlier values + +# check the total artifact storage space separately +builds_with_artifacts = p.builds.with_downloadable_artifacts.all + +artifact_storage = 0 +builds_with_artifacts.find_each do |build| + artifact_storage += build.artifacts_size +end + +puts "#{artifact_storage} bytes" ``` ### Identify deploy keys associated with blocked and non-member users @@ -341,6 +367,16 @@ DeployKeysProject.with_write_access.find_each do |deploy_key_mapping| end ``` +### Find projects using an SQL query + +Find and store an array of projects based on an SQL query: + +```ruby +# Finds projects that end with '%ject' +projects = Project.find_by_sql("SELECT * FROM projects WHERE name LIKE '%ject'") +=> [#<Project id:12 root/my-first-project>>, #<Project id:13 root/my-second-project>>] +``` + ## Wikis ### Recreate @@ -524,7 +560,8 @@ User.billable.count Using cURL and jq (up to a max 100, see the [pagination docs](../../api/README.md#pagination)): ```shell -curl --silent --header "Private-Token: ********************" "https://gitlab.example.com/api/v4/users?per_page=100&active" | jq --compact-output '.[] | [.id,.name,.username]' +curl --silent --header "Private-Token: ********************" \ + "https://gitlab.example.com/api/v4/users?per_page=100&active" | jq --compact-output '.[] | [.id,.name,.username]' ``` ### Block or Delete Users that have no projects or groups @@ -694,6 +731,16 @@ emails.each do |e| end ``` +### Find groups using an SQL query + +Find and store an array of groups based on an SQL query: + +```ruby +# Finds groups and subgroups that end with '%oup' +Group.find_by_sql("SELECT * FROM namespaces WHERE name LIKE '%oup'") +=> [#<Group id:3 @test-group>, #<Group id:4 @template-group/template-subgroup>] +``` + ## Routes ### Remove redirecting routes @@ -839,7 +886,7 @@ License.current.trial? ### Check if a project feature is available on the instance -Features listed in <https://gitlab.com/gitlab-org/gitlab/blob/master/ee/app/models/license.rb>. +Features listed in <https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/license.rb>. ```ruby License.current.feature_available?(:jira_dev_panel_integration) @@ -847,7 +894,7 @@ License.current.feature_available?(:jira_dev_panel_integration) ### Check if a project feature is available in a project -Features listed in [`license.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/app/models/license.rb). +Features listed in [`license.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/license.rb). ```ruby p = Project.find_by_full_path('<group>/<project>') @@ -863,55 +910,6 @@ license.save License.current # check to make sure it applied ``` -## Unicorn - -From [Zendesk ticket #91083](https://gitlab.zendesk.com/agent/tickets/91083) (internal) - -### Poll Unicorn requests by seconds - -```ruby -require 'rubygems' -require 'unicorn' - -# Usage for this program -def usage - puts "ruby unicorn_status.rb <path to unix socket> <poll interval in seconds>" - puts "Polls the given Unix socket every interval in seconds. Will not allow you to drop below 3 second poll intervals." - puts "Example: /opt/gitlab/embedded/bin/ruby poll_unicorn.rb /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket 10" -end - -# Look for required args. Throw usage and exit if they don't exist. -if ARGV.count < 2 - usage - exit 1 -end - -# Get the socket and threshold values. -socket = ARGV[0] -threshold = (ARGV[1]).to_i - -# Check threshold - is it less than 3? If so, set to 3 seconds. Safety first! -if threshold.to_i < 3 - threshold = 3 -end - -# Check - does that socket exist? -unless File.exist?(socket) - puts "Socket file not found: #{socket}" - exit 1 -end - -# Poll the given socket every THRESHOLD seconds as specified above. -puts "Running infinite loop. Use CTRL+C to exit." -puts "------------------------------------------" -loop do - Raindrops::Linux.unix_listener_stats([socket]).each do |addr, stats| - puts DateTime.now.to_s + " Active: " + stats.active.to_s + " Queued: " + stats.queued.to_s - end - sleep threshold -end -``` - ## Registry ### Registry Disk Space Usage by Project @@ -1189,6 +1187,28 @@ Prints the metrics saved in `conversational_development_index_metrics`. rake gitlab:usage_data:generate_and_send ``` +## Kubernetes integration + +Find cluster: + +```ruby +cluster = Clusters::Cluster.find(1) +cluster = Clusters::Cluster.find_by(name: 'cluster_name') +``` + +Delete cluster without associated resources: + +```ruby +# Find an admin user +user = User.find_by(username: 'admin_user') + +# Find the cluster with the ID +cluster = Clusters::Cluster.find(1) + +# Delete the cluster +Clusters::DestroyService.new(user).execute(cluster) +``` + ## Elasticsearch ### Configuration attributes @@ -1206,11 +1226,11 @@ Among other attributes, in the output you will notice that all the settings avai You can then set anyone of Elasticsearch integration settings by issuing a command similar to: ```ruby -ApplicationSetting.last.update_attributes(elasticsearch_url: '<your ES URL and port>') +ApplicationSetting.last.update(elasticsearch_url: '<your ES URL and port>') #or -ApplicationSetting.last.update_attributes(elasticsearch_indexing: false) +ApplicationSetting.last.update(elasticsearch_indexing: false) ``` #### Getting attributes diff --git a/doc/administration/troubleshooting/group_saml_scim.md b/doc/administration/troubleshooting/group_saml_scim.md index 63e69589b54..9e9ef492ebd 100644 --- a/doc/administration/troubleshooting/group_saml_scim.md +++ b/doc/administration/troubleshooting/group_saml_scim.md @@ -42,6 +42,10 @@ SCIM mapping:  +Group Sync: + + + ## Okta Basic SAML app configuration: diff --git a/doc/administration/troubleshooting/img/azure_configure_group_claim.png b/doc/administration/troubleshooting/img/azure_configure_group_claim.png Binary files differnew file mode 100644 index 00000000000..31df5fff625 --- /dev/null +++ b/doc/administration/troubleshooting/img/azure_configure_group_claim.png diff --git a/doc/administration/troubleshooting/index.md b/doc/administration/troubleshooting/index.md index 1c205cc987a..75c5ce460e9 100644 --- a/doc/administration/troubleshooting/index.md +++ b/doc/administration/troubleshooting/index.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Troubleshooting a GitLab installation +# Troubleshooting a GitLab installation **(FREE SELF)** This page documents a collection of resources to help you troubleshoot a GitLab installation. diff --git a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md index 9766b2210ca..b43825092b7 100644 --- a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md +++ b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Kubernetes, GitLab and You +# Kubernetes, GitLab, and you **(FREE SELF)** This is a list of useful information regarding Kubernetes that the GitLab Support Team sometimes uses while troubleshooting. GitLab is making this public, so that anyone @@ -147,7 +147,7 @@ and they will assist you with any issues you are having. You can also use `gitlab-rake`, instead of `/usr/local/bin/gitlab-rake`. -- Troubleshooting **Operations > Kubernetes** integration: +- Troubleshooting **Infrastructure > Kubernetes** integration: - Check the output of `kubectl get events -w --all-namespaces`. - Check the logs of pods within `gitlab-managed-apps` namespace. diff --git a/doc/administration/troubleshooting/linux_cheat_sheet.md b/doc/administration/troubleshooting/linux_cheat_sheet.md index c4e991ccc1b..9eadbad171e 100644 --- a/doc/administration/troubleshooting/linux_cheat_sheet.md +++ b/doc/administration/troubleshooting/linux_cheat_sheet.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Linux Cheat Sheet +# Linux cheat sheet **(FREE SELF)** This is the GitLab Support Team's collection of information regarding Linux, that they sometimes use while troubleshooting. It is listed here for transparency, @@ -177,8 +177,8 @@ strace -tt -T -f -y -yy -s 1024 -p <pid> # -o output file -# run strace on all unicorn processes -ps auwx | grep unicorn | awk '{ print " -p " $2}' | xargs strace -tt -T -f -y -yy -s 1024 -o /tmp/unicorn.txt +# run strace on all puma processes +ps auwx | grep puma | awk '{ print " -p " $2}' | xargs strace -tt -T -f -y -yy -s 1024 -o /tmp/puma.txt ``` Be aware that strace can have major impacts to system performance when it is running. diff --git a/doc/administration/troubleshooting/log_parsing.md b/doc/administration/troubleshooting/log_parsing.md index a0f71960e14..c16302dd251 100644 --- a/doc/administration/troubleshooting/log_parsing.md +++ b/doc/administration/troubleshooting/log_parsing.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Parsing GitLab logs with `jq` +# Parsing GitLab logs with `jq` **(FREE SELF)** We recommend using log aggregation and search tools like Kibana and Splunk whenever possible, but if they are not available you can still quickly parse diff --git a/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md b/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md index 481c95b925a..e55118d7309 100644 --- a/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md +++ b/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Navigating GitLab via Rails console +# Navigating GitLab via Rails console **(FREE SELF)** At the heart of GitLab is a web application [built using the Ruby on Rails framework](https://about.gitlab.com/blog/2018/10/29/why-we-use-rails-to-build-gitlab/). @@ -46,7 +46,7 @@ Let's enable debug logging for Active Record so we can see the underlying database queries made: ```ruby -ActiveRecord::Base.logger = Logger.new(STDOUT) +ActiveRecord::Base.logger = Logger.new($stdout) ``` Now, let's try retrieving a user from the database: diff --git a/doc/administration/troubleshooting/postgresql.md b/doc/administration/troubleshooting/postgresql.md index 9565b7594d6..341c6bfbc65 100644 --- a/doc/administration/troubleshooting/postgresql.md +++ b/doc/administration/troubleshooting/postgresql.md @@ -53,8 +53,7 @@ This section is for links to information elsewhere in the GitLab documentation. - [PostgreSQL scaling](../postgresql/replication_and_failover.md) - Including [troubleshooting](../postgresql/replication_and_failover.md#troubleshooting) - `gitlab-ctl repmgr-check-master` (or `gitlab-ctl patroni check-leader` if - you're using Patroni) and PgBouncer errors. + `gitlab-ctl patroni check-leader` and PgBouncer errors. - [Developer database documentation](../../development/README.md#database-guides), some of which is absolutely not for production use. Including: diff --git a/doc/administration/troubleshooting/sidekiq.md b/doc/administration/troubleshooting/sidekiq.md index 297a8355036..7a8ac8c3dbe 100644 --- a/doc/administration/troubleshooting/sidekiq.md +++ b/doc/administration/troubleshooting/sidekiq.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Troubleshooting Sidekiq +# Troubleshooting Sidekiq **(FREE SELF)** Sidekiq is the background job processor GitLab uses to asynchronously run tasks. When things go wrong it can be difficult to troubleshoot. These diff --git a/doc/administration/troubleshooting/ssl.md b/doc/administration/troubleshooting/ssl.md index 7c0b745f8c2..5ce4c7f0fb2 100644 --- a/doc/administration/troubleshooting/ssl.md +++ b/doc/administration/troubleshooting/ssl.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Troubleshooting SSL +# Troubleshooting SSL **(FREE SELF)** This page contains a list of common SSL-related errors and scenarios that you may encounter while working with GitLab. It should serve as an addition to the diff --git a/doc/administration/troubleshooting/test_environments.md b/doc/administration/troubleshooting/test_environments.md index 16ef4f83978..5fe493a536c 100644 --- a/doc/administration/troubleshooting/test_environments.md +++ b/doc/administration/troubleshooting/test_environments.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Apps for a Testing Environment +# Apps for a testing environment **(FREE SELF)** This is the GitLab Support Team's collection of information regarding testing environments, for use while troubleshooting. It is listed here for transparency, and it may be useful diff --git a/doc/administration/troubleshooting/tracing_correlation_id.md b/doc/administration/troubleshooting/tracing_correlation_id.md index ad2b8586b8b..7b9ce5c6d7b 100644 --- a/doc/administration/troubleshooting/tracing_correlation_id.md +++ b/doc/administration/troubleshooting/tracing_correlation_id.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Finding relevant log entries with a correlation ID +# Finding relevant log entries with a correlation ID **(FREE SELF)** In GitLab 11.6 and later, a unique request tracking ID, known as the "correlation ID" has been logged by the GitLab instance for most requests. Each individual request to GitLab gets |
