diff options
Diffstat (limited to 'doc/administration/troubleshooting')
8 files changed, 78 insertions, 66 deletions
diff --git a/doc/administration/troubleshooting/debug.md b/doc/administration/troubleshooting/debug.md index 1e1b2ad8378..55fd6462bc3 100644 --- a/doc/administration/troubleshooting/debug.md +++ b/doc/administration/troubleshooting/debug.md @@ -244,7 +244,7 @@ separate Rails process to debug the issue: For example: ```ruby - app.get 'https://gitlab.com/gitlab-org/gitlab-foss/issues/1?private_token=123456' + app.get 'https://gitlab.com/gitlab-org/gitlab-foss/-/issues/1?private_token=123456' ``` 1. In a new window, run `top`. It should show this Ruby process using 100% CPU. Write down the PID. diff --git a/doc/administration/troubleshooting/elasticsearch.md b/doc/administration/troubleshooting/elasticsearch.md index a39fe4ba8c3..12b82e4bc48 100644 --- a/doc/administration/troubleshooting/elasticsearch.md +++ b/doc/administration/troubleshooting/elasticsearch.md @@ -189,7 +189,7 @@ Moving past that, it is best to attempt the same search using the [Elasticsearch If the results: -- Sync up, then there is not a technical "issue" per se. Instead, it might be a problem +- Sync up, then there is not a technical "issue." Instead, it might be a problem with the Elasticsearch filters we are using. This can be complicated, so it is best to escalate to GitLab support to check these and guide you on the potential on whether or not a feature request is needed. @@ -330,10 +330,10 @@ feel free to update that page with issues you encounter and solutions. Setting up Elasticsearch isn't too bad, but it can be a bit finicky and time consuming. -The easiest method is to spin up a docker container with the required version and +The easiest method is to spin up a Docker container with the required version and bind ports 9200/9300 so it can be used. -The following is an example of running a docker container of Elasticsearch v7.2.0: +The following is an example of running a Docker container of Elasticsearch v7.2.0: ```shell docker pull docker.elastic.co/elasticsearch/elasticsearch:7.2.0 @@ -342,7 +342,7 @@ docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elas From here, you can: -- Grab the IP of the docker container (use `docker inspect <container_id>`) +- Grab the IP of the Docker container (use `docker inspect <container_id>`) - Use `<IP.add.re.ss:9200>` to communicate with it. This is a quick method to test out Elasticsearch, but by no means is this a diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 2cbc994fb4c..33af356b37d 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -263,7 +263,7 @@ p.import_state.mark_as_failed("Failed manually through console.") In a specific situation, an imported repository needed to be renamed. The Support Team was informed of a backup restore that failed on a single repository, which created -the project with an empty repository. The project was successfully restored to a dev +the project with an empty repository. The project was successfully restored to a development instance, then exported, and imported into a new project under a different name. The Support Team was able to transfer the incorrectly named imported project into the @@ -302,7 +302,7 @@ you will see two pushes with the same "from" SHA: ```ruby p = Project.find_with_namespace('u/p') -p.events.code_push.last(100).each do |e| +p.events.pushed_action.last(100).each do |e| printf "%-20.20s %8s...%8s (%s)\n", e.data[:ref], e.data[:before], e.data[:after], e.author.try(:username) end ``` @@ -311,7 +311,7 @@ GitLab 9.5 and above: ```ruby p = Project.find_by_full_path('u/p') -p.events.code_push.last(100).each do |e| +p.events.pushed_action.last(100).each do |e| printf "%-20.20s %8s...%8s (%s)\n", e.push_event_payload[:ref], e.push_event_payload[:commit_from], e.push_event_payload[:commit_to], e.author.try(:username) end ``` @@ -380,39 +380,6 @@ user = User.find_by_username '' user.skip_reconfirmation! ``` -### Get an admin token - -```ruby -# Get the first admin's first access token (no longer works on 11.9+. see: https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/22743) -User.where(admin:true).first.personal_access_tokens.first.token - -# Get the first admin's private token (no longer works on 10.2+) -User.where(admin:true).private_token -``` - -### Create personal access token - -```ruby -personal_access_token = User.find(123).personal_access_tokens.create( - name: 'apitoken', - impersonation: false, - scopes: [:api] -) - -puts personal_access_token.token -``` - -You might also want to manually set the token string: - -```ruby -User.find(123).personal_access_tokens.create( - name: 'apitoken', - token_digest: Gitlab::CryptoHelper.sha256('some-token-string-here'), - impersonation: false, - scopes: [:api] -) -``` - ### Active users & Historical users ```ruby @@ -518,7 +485,7 @@ group.project_creation_level=0 ### Remove redirecting routes -See <https://gitlab.com/gitlab-org/gitlab-foss/issues/41758#note_54828133>. +See <https://gitlab.com/gitlab-org/gitlab-foss/-/issues/41758#note_54828133>. ```ruby path = 'foo' @@ -576,7 +543,7 @@ This section has been moved to the [job artifacts troubleshooting documentation] ### Find reason failure (for when build trace is empty) (Introduced in 10.3.0) -See <https://gitlab.com/gitlab-org/gitlab-foss/issues/41111>. +See <https://gitlab.com/gitlab-org/gitlab-foss/-/issues/41111>. ```ruby build = Ci::Build.find(78420) @@ -620,10 +587,26 @@ Gitlab::CurrentSettings.current_application_settings.runners_registration_token ## License -### See license plan name (since v9.3.0-ee) +### See current license information ```ruby +# License information (name, company, email address) +License.current.licensee + +# Plan: License.current.plan + +# Uploaded: +License.current.created_at + +# Started: +License.current.starts_at + +# Expires at: +License.current.expires_at + +# Is this a trial license? +License.current.trial? ``` ### Check if a project feature is available on the instance @@ -636,7 +619,7 @@ License.current.feature_available?(:jira_dev_panel_integration) ### Check if a project feature is available in a project -Features listed in <https://gitlab.com/gitlab-org/gitlab/blob/master/ee/app/models/license.rb>. +Features listed in [`license.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/app/models/license.rb). ```ruby p = Project.find_by_full_path('<group>/<project>') diff --git a/doc/administration/troubleshooting/index.md b/doc/administration/troubleshooting/index.md new file mode 100644 index 00000000000..50f192b1983 --- /dev/null +++ b/doc/administration/troubleshooting/index.md @@ -0,0 +1,20 @@ +# Troubleshooting a GitLab installation + +Below are some resources to help you troubleshoot a GitLab installation +in case something goes wrong: + +- [Debugging tips](debug.md) +- [Diagnostics tools](diagnostics_tools.md) +- [Elasticsearch](elasticsearch.md) +- [GitLab Rails console cheat sheet](gitlab_rails_cheat_sheet.md) +- [Group SAML and SCIM troubleshooting](group_saml_scim.md) **(SILVER ONLY)** +- [Kubernetes cheat sheet](kubernetes_cheat_sheet.md) +- [Linux cheat sheet](linux_cheat_sheet.md) +- [Parsing GitLab logs with `jq`](log_parsing.md) +- [Navigating GitLab via Rails console](navigating_gitlab_via_rails_console.md) +- [PostgreSQL](postgresql.md) +- [Sidekiq](sidekiq.md) +- [SSL](ssl.md) + +If you need a testing environment to troubleshoot, see the +[apps for a testing environment](test_environments.md). diff --git a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md index cab073b9924..01532032b49 100644 --- a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md +++ b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md @@ -85,7 +85,7 @@ and they will assist you with any issues you are having. ## GitLab-specific Kubernetes information - Minimal config that can be used to test a Kubernetes Helm chart can be found - [here](https://gitlab.com/gitlab-org/charts/gitlab/issues/620). + [here](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/620). - Tailing logs of a separate pod. An example for a Webservice pod: @@ -186,7 +186,7 @@ and they will assist you with any issues you are having. helm upgrade <release name> <chart path> -f gitlab.yaml ``` - After <https://gitlab.com/gitlab-org/charts/gitlab/issues/780> is fixed, it should + After <https://gitlab.com/gitlab-org/charts/gitlab/-/issues/780> is fixed, it should be possible to use [Updating GitLab using the Helm Chart](https://docs.gitlab.com/charts/index.html#updating-gitlab-using-the-helm-chart) for upgrades. diff --git a/doc/administration/troubleshooting/postgresql.md b/doc/administration/troubleshooting/postgresql.md index 65a6bffca44..e5a4dffb3cc 100644 --- a/doc/administration/troubleshooting/postgresql.md +++ b/doc/administration/troubleshooting/postgresql.md @@ -31,7 +31,7 @@ This section is for links to information elsewhere in the GitLab documentation. - Destructively reseeding the GitLab database. - Guidance around updating packaged PostgreSQL, including how to stop it happening automatically. -- [More about external PostgreSQL](../external_database.md) +- [More about external PostgreSQL](../postgresql/external.md) - [Running Geo with external PostgreSQL](../geo/replication/external_database.md) @@ -45,8 +45,8 @@ This section is for links to information elsewhere in the GitLab documentation. - Managing Omnibus PostgreSQL versions [from the development docs](https://docs.gitlab.com/omnibus/development/managing-postgresql-versions.html) -- [PostgreSQL scaling](../high_availability/database.md) - - including [troubleshooting](../high_availability/database.md#troubleshooting) `gitlab-ctl repmgr-check-master` and PgBouncer errors +- [PostgreSQL scaling](../postgresql/replication_and_failover.md) + - including [troubleshooting](../postgresql/replication_and_failover.md#troubleshooting) `gitlab-ctl repmgr-check-master` and PgBouncer errors - [Developer database documentation](../../development/README.md#database-guides) - some of which is absolutely not for production use. Including: - understanding EXPLAIN plans @@ -55,8 +55,8 @@ This section is for links to information elsewhere in the GitLab documentation. - [GitLab database requirements](../../install/requirements.md#database) including - Support for MySQL was removed in GitLab 12.1; [migrate to PostgreSQL](../../update/mysql_to_postgresql.md) - - required extension pg_trgm - - required extension postgres_fdw for Geo + - required extension `pg_trgm` + - required extension `postgres_fdw` for Geo - Errors like this in the `production/sidekiq` log; see: [Set default_transaction_isolation into read committed](https://docs.gitlab.com/omnibus/settings/database.html#set-default_transaction_isolation-into-read-committed): @@ -95,15 +95,15 @@ This section is for links to information elsewhere in the GitLab documentation. References: -- [Issue #1 Deadlocks with GitLab 12.1, PostgreSQL 10.7](https://gitlab.com/gitlab-org/gitlab/issues/30528) +- [Issue #1 Deadlocks with GitLab 12.1, PostgreSQL 10.7](https://gitlab.com/gitlab-org/gitlab/-/issues/30528) - [Customer ticket (internal) GitLab 12.1.6](https://gitlab.zendesk.com/agent/tickets/134307) and [Google doc (internal)](https://docs.google.com/document/d/19xw2d_D1ChLiU-MO1QzWab-4-QXgsIUcN5e_04WTKy4) -- [Issue #2 deadlocks can occur if an instance is flooded with pushes](https://gitlab.com/gitlab-org/gitlab/issues/33650). Provided for context about how GitLab code can have this sort of unanticipated effect in unusual situations. +- [Issue #2 deadlocks can occur if an instance is flooded with pushes](https://gitlab.com/gitlab-org/gitlab/-/issues/33650). Provided for context about how GitLab code can have this sort of unanticipated effect in unusual situations. ```plaintext ERROR: deadlock detected ``` -Three applicable timeouts are identified in the issue [#1](https://gitlab.com/gitlab-org/gitlab/issues/30528); our recommended settings are as follows: +Three applicable timeouts are identified in the issue [#1](https://gitlab.com/gitlab-org/gitlab/-/issues/30528); our recommended settings are as follows: ```ini deadlock_timeout = 5s @@ -111,20 +111,20 @@ statement_timeout = 15s idle_in_transaction_session_timeout = 60s ``` -Quoting from issue [#1](https://gitlab.com/gitlab-org/gitlab/issues/30528): +Quoting from issue [#1](https://gitlab.com/gitlab-org/gitlab/-/issues/30528): > "If a deadlock is hit, and we resolve it through aborting the transaction after a short period, then the retry mechanisms we already have will make the deadlocked piece of work try again, and it's unlikely we'll deadlock multiple times in a row." TIP: **Tip:** In support, our general approach to reconfiguring timeouts (applies also to the HTTP stack as well) is that it's acceptable to do it temporarily as a workaround. If it makes GitLab usable for the customer, then it buys time to understand the problem more completely, implement a hot fix, or make some other change that addresses the root cause. Generally, the timeouts should be put back to reasonable defaults once the root cause is resolved. -In this case, the guidance we had from development was to drop deadlock_timeout and/or statement_timeout but to leave the third setting at 60s. Setting idle_in_transaction protects the database from sessions potentially hanging for days. There's more discussion in [the issue relating to introducing this timeout on GitLab.com](https://gitlab.com/gitlab-com/gl-infra/production/issues/1053). +In this case, the guidance we had from development was to drop deadlock_timeout and/or statement_timeout but to leave the third setting at 60s. Setting idle_in_transaction protects the database from sessions potentially hanging for days. There's more discussion in [the issue relating to introducing this timeout on GitLab.com](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1053). PostgresSQL defaults: - `statement_timeout = 0` (never) - `idle_in_transaction_session_timeout = 0` (never) -Comments in issue [#1](https://gitlab.com/gitlab-org/gitlab/issues/30528) indicate that these should both be set to at least a number of minutes for all Omnibus installations (so they don't hang indefinitely). However, 15s for statement_timeout is very short, and will only be effective if the underlying infrastructure is very performant. +Comments in issue [#1](https://gitlab.com/gitlab-org/gitlab/-/issues/30528) indicate that these should both be set to at least a number of minutes for all Omnibus installations (so they don't hang indefinitely). However, 15s for statement_timeout is very short, and will only be effective if the underlying infrastructure is very performant. See current settings with: diff --git a/doc/administration/troubleshooting/sidekiq.md b/doc/administration/troubleshooting/sidekiq.md index ca21c038267..5109a3baff2 100644 --- a/doc/administration/troubleshooting/sidekiq.md +++ b/doc/administration/troubleshooting/sidekiq.md @@ -29,9 +29,18 @@ Example: gitlab_rails['env'] = {"SIDEKIQ_LOG_ARGUMENTS" => "1"} ``` -Please note: It is not recommend to enable this setting in production because some -Sidekiq jobs (such as sending a password reset email) take secret arguments (for -example the password reset token). +This does not log all job arguments. To avoid logging sensitive +information (for instance, password reset tokens), it logs numeric +arguments for all workers, with overrides for some specific workers +where their arguments are not sensitive. + +Example log output: + +```json +{"severity":"INFO","time":"2020-06-08T14:37:37.892Z","class":"AdminEmailsWorker","args":["[FILTERED]","[FILTERED]","[FILTERED]"],"retry":3,"queue":"admin_emails","backtrace":true,"jid":"9e35e2674ac7b12d123e13cc","created_at":"2020-06-08T14:37:37.373Z","meta.user":"root","meta.caller_id":"Admin::EmailsController#create","correlation_id":"37D3lArJmT1","uber-trace-id":"2d942cc98cc1b561:6dc94409cfdd4d77:9fbe19bdee865293:1","enqueued_at":"2020-06-08T14:37:37.410Z","pid":65011,"message":"AdminEmailsWorker JID-9e35e2674ac7b12d123e13cc: done: 0.48085 sec","job_status":"done","scheduling_latency_s":0.001012,"redis_calls":9,"redis_duration_s":0.004608,"redis_read_bytes":696,"redis_write_bytes":6141,"duration_s":0.48085,"cpu_s":0.308849,"completed_at":"2020-06-08T14:37:37.892Z","db_duration_s":0.010742} +{"severity":"INFO","time":"2020-06-08T14:37:37.894Z","class":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper","wrapped":"ActionMailer::DeliveryJob","queue":"mailers","args":["[FILTERED]"],"retry":3,"backtrace":true,"jid":"e47a4f6793d475378432e3c8","created_at":"2020-06-08T14:37:37.884Z","meta.user":"root","meta.caller_id":"AdminEmailsWorker","correlation_id":"37D3lArJmT1","uber-trace-id":"2d942cc98cc1b561:29344de0f966446d:5c3b0e0e1bef987b:1","enqueued_at":"2020-06-08T14:37:37.885Z","pid":65011,"message":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper JID-e47a4f6793d475378432e3c8: start","job_status":"start","scheduling_latency_s":0.009473} +{"severity":"INFO","time":"2020-06-08T14:39:50.648Z","class":"NewIssueWorker","args":["455","1"],"retry":3,"queue":"new_issue","backtrace":true,"jid":"a24af71f96fd129ec47f5d1e","created_at":"2020-06-08T14:39:50.643Z","meta.user":"root","meta.project":"h5bp/html5-boilerplate","meta.root_namespace":"h5bp","meta.caller_id":"Projects::IssuesController#create","correlation_id":"f9UCZHqhuP7","uber-trace-id":"28f65730f99f55a3:a5d2b62dec38dffc:48ddd092707fa1b7:1","enqueued_at":"2020-06-08T14:39:50.646Z","pid":65011,"message":"NewIssueWorker JID-a24af71f96fd129ec47f5d1e: start","job_status":"start","scheduling_latency_s":0.001144} +``` When using [Sidekiq JSON logging](../logs.md#sidekiqlog), arguments logs are limited to a maximum size of 10 kilobytes of text; @@ -86,13 +95,13 @@ sudo apt-get install linux-tools-common linux-tools-generic linux-tools-`uname - sudo yum install perf ``` -Run perf against the Sidekiq PID: +Run `perf` against the Sidekiq PID: ```shell sudo perf record -p <sidekiq_pid> ``` -Let this run for 30-60 seconds and then press Ctrl-C. Then view the perf report: +Let this run for 30-60 seconds and then press Ctrl-C. Then view the `perf` report: ```shell $ sudo perf report @@ -105,13 +114,13 @@ Samples: 348K of event 'cycles', Event count (approx.): 280908431073 0.10% ruby libc-2.12.so [.] _int_free ``` -Above you see sample output from a perf report. It shows that 97% of the CPU is +Above you see sample output from a `perf` report. It shows that 97% of the CPU is being spent inside Nokogiri and `xmlXPathNodeSetMergeAndClear`. For something this obvious you should then go investigate what job in GitLab would use Nokogiri and XPath. Combine with `TTIN` or `gdb` output to show the corresponding Ruby code where this is happening. -## The GNU Project Debugger (gdb) +## The GNU Project Debugger (`gdb`) `gdb` can be another effective tool for debugging Sidekiq. It gives you a little more interactive way to look at each thread and see what's causing problems. diff --git a/doc/administration/troubleshooting/test_environments.md b/doc/administration/troubleshooting/test_environments.md index e9db5f64446..80ccd15aa22 100644 --- a/doc/administration/troubleshooting/test_environments.md +++ b/doc/administration/troubleshooting/test_environments.md @@ -16,7 +16,7 @@ are only available internally at GitLab. ## Docker -The following were tested on docker containers running in the cloud. Support Engineers, +The following were tested on Docker containers running in the cloud. Support Engineers, please see [these docs](https://gitlab.com/gitlab-com/dev-resources/tree/master/dev-resources#running-docker-containers) on how to run Docker containers on `dev-resources`. Other setups haven't been tested, but contributions are welcome. |