diff options
Diffstat (limited to 'doc/administration/troubleshooting')
16 files changed, 321 insertions, 148 deletions
diff --git a/doc/administration/troubleshooting/debug.md b/doc/administration/troubleshooting/debug.md index d67c9963092..8c8fa25aa5e 100644 --- a/doc/administration/troubleshooting/debug.md +++ b/doc/administration/troubleshooting/debug.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Debugging Tips diff --git a/doc/administration/troubleshooting/diagnostics_tools.md b/doc/administration/troubleshooting/diagnostics_tools.md index 132f8524ca1..27a7493b318 100644 --- a/doc/administration/troubleshooting/diagnostics_tools.md +++ b/doc/administration/troubleshooting/diagnostics_tools.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- diff --git a/doc/administration/troubleshooting/elasticsearch.md b/doc/administration/troubleshooting/elasticsearch.md index 12aa91e6f14..755273eb06e 100644 --- a/doc/administration/troubleshooting/elasticsearch.md +++ b/doc/administration/troubleshooting/elasticsearch.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Global Search -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Troubleshooting Elasticsearch diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index cae4aa96f75..2482a4fe7ad 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -14,13 +14,13 @@ having an issue with GitLab, it is highly recommended that you check your [support options](https://about.gitlab.com/support/) first, before attempting to use this information. -CAUTION: **Caution:** +WARNING: Please note that some of these scripts could be damaging if not run correctly, or under the right conditions. We highly recommend running them under the guidance of a Support Engineer, or running them in a test environment with a backup of the instance ready to be restored, just in case. -CAUTION: **Caution:** +WARNING: Please also note that as GitLab changes, changes to the code are inevitable, and so some scripts may not work as they once used to. These are not kept up-to-date as these scripts/commands were added as they were found/needed. As @@ -308,7 +308,7 @@ pp p.statistics # compare with earlier values ### Recreate -CAUTION: **Caution:** +WARNING: This is a destructive operation, the Wiki will be empty. A Projects Wiki can be recreated by this command: @@ -374,6 +374,26 @@ Clear the cache: sudo gitlab-rake cache:clear ``` +### Export a repository + +It's typically recommended to export a project through [the web interface](../../user/project/settings/import_export.md#exporting-a-project-and-its-data) or through [the API](../../api/project_import_export.md). In situations where this is not working as expected, it may be preferable to export a project directly via the Rails console: + +```ruby +user = User.find_by_username('USERNAME') +project = Project.find_by_full_path('PROJECT_PATH') +Projects::ImportExport::ExportService.new(project, user).execute +``` + +If the project you wish to export is available at `https://gitlab.example.com/baltig/pipeline-templates`, the value to use for `PROJECT_PATH` would be `baltig/pipeline-templates`. + +If this all runs successfully, you will see output like the following before being returned to the Rails console prompt: + +```ruby +=> nil +``` + +The exported project will be located within a `.tar.gz` file in `/var/opt/gitlab/gitlab-rails/uploads/-/system/import_export_upload/export_file/`. + ## Repository ### Search sequence of pushes to a repository @@ -464,8 +484,9 @@ User.billable.count ::HistoricalData.max_historical_user_count ``` +Using cURL and jq (up to a max 100, see the [pagination docs](../../api/README.md#pagination)): + ```shell -# Using curl and jq (up to a max 100, see pagination docs https://docs.gitlab.com/ee/api/#pagination curl --silent --header "Private-Token: ********************" "https://gitlab.example.com/api/v4/users?per_page=100&active" | jq --compact-output '.[] | [.id,.name,.username]' ``` @@ -491,10 +512,22 @@ users.each do |user| end ``` +### Deactivate Users that have no recent activity + +```ruby +days_inactive = 90 +inactive_users = User.active.where("last_activity_on <= ?", days_inactive.days.ago) + +inactive_users.each do |user| + puts "user '#{user.username}': #{user.last_activity_on}" + user.deactivate! +end +``` + ### Block Users that have no recent activity ```ruby -days_inactive = 60 +days_inactive = 90 inactive_users = User.active.where("last_activity_on <= ?", days_inactive.days.ago) inactive_users.each do |user| @@ -565,6 +598,17 @@ group = Group.find_by_path_or_name('group-name') group.project_creation_level=0 ``` +### Modify group - disable 2FA requirement + +WARNING: +When disabling the 2FA Requirement on a subgroup, the whole parent group (including all subgroups) is affected by this change. + +```ruby +group = Group.find_by_path_or_name('group-name') +group.require_two_factor_authentication=false +group.save +``` + ## SCIM ### Fixing bad SCIM identities @@ -1015,3 +1059,55 @@ This will also refresh the cached usage ping displayed in the admin area ```ruby Gitlab::UsageData.to_json(force_refresh: true) ``` + +#### Generate and print + +Generates usage ping data in JSON format. + +```shell +rake gitlab:usage_data:generate +``` + +#### Generate and send usage ping + +Prints the metrics saved in `conversational_development_index_metrics`. + +```shell +rake gitlab:usage_data:generate_and_send +``` + +## Elasticsearch + +### Configuration attributes + +Open the rails console (`gitlab rails c`) and run the following command to see all the available attributes: + +```ruby +ApplicationSetting.last.attributes +``` + +Among other attributes, in the output you will notice that all the settings available in the [Elasticsearch Integration page](../../integration/elasticsearch.md), like: `elasticsearch_indexing`, `elasticsearch_url`, `elasticsearch_replicas`, `elasticsearch_pause_indexing`, etc. + +#### Setting attributes + +You can then set anyone of Elasticsearch integration settings by issuing a command similar to: + +```ruby +ApplicationSetting.last.update_attributes(elasticsearch_url: '<your ES URL and port>') + +#or + +ApplicationSetting.last.update_attributes(elasticsearch_indexing: false) +``` + +#### Getting attributes + +You can then check if the settings have been set in the [Elasticsearch Integration page](../../integration/elasticsearch.md) or in the rails console by issuing: + +```ruby +Gitlab::CurrentSettings.elasticsearch_url + +#or + +Gitlab::CurrentSettings.elasticsearch_indexing +``` diff --git a/doc/administration/troubleshooting/group_saml_scim.md b/doc/administration/troubleshooting/group_saml_scim.md index efc8eaab198..e9dbbfbde12 100644 --- a/doc/administration/troubleshooting/group_saml_scim.md +++ b/doc/administration/troubleshooting/group_saml_scim.md @@ -1,7 +1,7 @@ --- stage: Manage group: Access -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -9,7 +9,7 @@ type: reference These are notes and screenshots regarding Group SAML and SCIM that the GitLab Support Team sometimes uses while troubleshooting, but which do not fit into the official documentation. GitLab is making this public, so that anyone can make use of the Support team’s collected knowledge. -Please refer to GitLab's [Group SAML](../../user/group/saml_sso/index.md) docs for information on the feature and how to set it up. +Please refer to the GitLab [Group SAML](../../user/group/saml_sso/index.md) docs for information on the feature and how to set it up. When troubleshooting a SAML configuration, GitLab team members will frequently start with the [SAML troubleshooting section](../../user/group/saml_sso/index.md#troubleshooting). @@ -22,7 +22,7 @@ This section includes relevant screenshots of the following example configuratio - [Azure Active Directory](#azure-active-directory) - [OneLogin](#onelogin) -CAUTION: **Caution:** +WARNING: These screenshots are updated only as needed by GitLab Support. They are **not** official documentation. If you are currently having an issue with GitLab, you may want to check your [support options](https://about.gitlab.com/support/). diff --git a/doc/administration/troubleshooting/img/network_monitor_xid.png b/doc/administration/troubleshooting/img/network_monitor_xid.png Binary files differindex 7fc2cf47ea0..fa64fd1509c 100644 --- a/doc/administration/troubleshooting/img/network_monitor_xid.png +++ b/doc/administration/troubleshooting/img/network_monitor_xid.png diff --git a/doc/administration/troubleshooting/index.md b/doc/administration/troubleshooting/index.md index 8a4a0a4caac..67115ce31c0 100644 --- a/doc/administration/troubleshooting/index.md +++ b/doc/administration/troubleshooting/index.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Troubleshooting a GitLab installation diff --git a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md index 21fd183dfd0..07a7baf338b 100644 --- a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md +++ b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -11,7 +11,7 @@ This is a list of useful information regarding Kubernetes that the GitLab Suppor Team sometimes uses while troubleshooting. GitLab is making this public, so that anyone can make use of the Support team's collected knowledge -CAUTION: **Caution:** +WARNING: These commands **can alter or break** your Kubernetes components so use these at your own risk. If you are on a [paid tier](https://about.gitlab.com/pricing/) and are not sure how @@ -126,7 +126,7 @@ and they will assist you with any issues you are having. kubectl get pods | grep task-runner # enter it - kubectl exec -it <task-runner-pod-name> bash + kubectl exec -it <task-runner-pod-name> -- bash # open rails console # rails console can be also called from other GitLab pods @@ -139,10 +139,10 @@ and they will assist you with any issues you are having. /usr/local/bin/gitlab-rake gitlab:check # open console without entering pod - kubectl exec -it <task-runner-pod-name> /srv/gitlab/bin/rails console + kubectl exec -it <task-runner-pod-name> -- /srv/gitlab/bin/rails console # check the status of DB migrations - kubectl exec -it <task-runner-pod-name> /usr/local/bin/gitlab-rake db:migrate:status + kubectl exec -it <task-runner-pod-name> -- /usr/local/bin/gitlab-rake db:migrate:status ``` You can also use `gitlab-rake`, instead of `/usr/local/bin/gitlab-rake`. @@ -207,7 +207,7 @@ all Kubernetes resources and dependent charts: helm get manifest <release name> ``` -## Installation of minimal GitLab config via Minikube on macOS +## Installation of minimal GitLab configuration via Minikube on macOS This section is based on [Developing for Kubernetes with Minikube](https://docs.gitlab.com/charts/development/minikube/index.html) and [Helm](https://docs.gitlab.com/charts/installation/tools.html#helm). Refer diff --git a/doc/administration/troubleshooting/linux_cheat_sheet.md b/doc/administration/troubleshooting/linux_cheat_sheet.md index b1042a9402b..c61a78624c3 100644 --- a/doc/administration/troubleshooting/linux_cheat_sheet.md +++ b/doc/administration/troubleshooting/linux_cheat_sheet.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -13,7 +13,7 @@ and it may be useful for users with experience with Linux. If you are currently having an issue with GitLab, you may want to check your [support options](https://about.gitlab.com/support/) first, before attempting to use this information. -CAUTION: **Caution:** +WARNING: If you are administering GitLab you are expected to know these commands for your distribution of choice. If you are a GitLab Support Engineer, consider this a cross-reference to translate `yum` -> `apt-get` and the like. @@ -23,7 +23,7 @@ on. Contributions are welcome to help add them. ## System Commands -### Distro Information +### Distribution Information ```shell # Debian/Ubuntu @@ -200,7 +200,7 @@ or you can build it from source if you have the Rust compiler. #### How to use the tool -First run the tool with no arguments other than the strace output file name to get +First run the tool with no arguments other than the strace output filename to get a summary of the top processes sorted by time spent actively performing tasks. You can also sort based on total time, # of syscalls made, PID #, and # of child processes using the `-S` or `--sort` flag. The number of results defaults to 25 processes, but @@ -303,7 +303,7 @@ nslookup example.com 1.1.1.1 whois <ip_address> | grep -i "orgname\|netname" # Curl headers with redirect -curl --head --location https://example.com +curl --head --location "https://example.com" ``` ## Package Management diff --git a/doc/administration/troubleshooting/log_parsing.md b/doc/administration/troubleshooting/log_parsing.md index 4e9e8cd591f..144aa0f6d3b 100644 --- a/doc/administration/troubleshooting/log_parsing.md +++ b/doc/administration/troubleshooting/log_parsing.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Parsing GitLab logs with `jq` @@ -143,16 +143,39 @@ jq 'select(."grpc.code" != null and ."grpc.code" != "OK")' current jq 'select(."grpc.time_ms" > 30000)' current ``` -#### Print top three projects by request volume and their three longest durations - -```shell -jq -s -r 'map(select(."grpc.request.glProjectPath" != null and ."grpc.request.glProjectPath" != "" and ."grpc.time_ms" != null)) | group_by(."grpc.request.glProjectPath") | sort_by(-length) | limit(3; .[]) | sort_by(-."grpc.time_ms") | "CT: \(length)\tPROJECT: \(.[0]."grpc.request.glProjectPath")\tDURS: \(.[0]."grpc.time_ms"), \(.[1]."grpc.time_ms"), \(.[2]."grpc.time_ms")"' current +#### Print top ten projects by request volume and their three longest durations + +```shell +jq --raw-output --slurp ' + map( + select( + ."grpc.request.glProjectPath" != null + and ."grpc.request.glProjectPath" != "" + and ."grpc.time_ms" != null + ) + ) + | group_by(."grpc.request.glProjectPath") + | sort_by(-length) + | limit(10; .[]) + | sort_by(-."grpc.time_ms") + | [ + length, + .[0]."grpc.time_ms", + .[1]."grpc.time_ms", + .[2]."grpc.time_ms", + .[0]."grpc.request.glProjectPath" + ] + | @sh' /var/log/gitlab/gitaly/current \ +| awk 'BEGIN { printf "%7s %10s %10s %10s\t%s\n", "CT", "MAX DURS", "", "", "PROJECT" } + { printf "%7u %7u ms, %7u ms, %7u ms\t%s\n", $1, $2, $3, $4, $5 }' ``` **Example output** ```plaintext -CT: 635 PROJECT: groupA/project1 DURS: 4292.269, 4228.853, 2885.548 -CT: 462 PROJECT: groupB/project5 DURS: 4368.981, 3623.553, 361.399 -CT: 455 PROJECT: groupC/project7 DURS: 387.295, 381.874, 373.988 + CT MAX DURS PROJECT + 206 4898 ms, 1101 ms, 1032 ms 'groupD/project4' + 109 1420 ms, 962 ms, 875 ms 'groupEF/project56' + 663 106 ms, 96 ms, 94 ms 'groupABC/project123' + ... ``` diff --git a/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md b/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md index 475f3d56836..68c12117222 100644 --- a/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md +++ b/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Navigating GitLab via Rails console @@ -12,7 +12,7 @@ Thanks to this, we also get access to the amazing tools built right into Rails. In this guide, we'll introduce the [Rails console](../operations/rails_console.md#starting-a-rails-console-session) and the basics of interacting with your GitLab instance from the command line. -CAUTION: **Caution:** +WARNING: The Rails console interacts directly with your GitLab instance. In many cases, there are no handrails to prevent you from permanently modifying, corrupting or destroying production data. If you would like to explore the Rails console diff --git a/doc/administration/troubleshooting/postgresql.md b/doc/administration/troubleshooting/postgresql.md index d22e76a505a..7052b68370c 100644 --- a/doc/administration/troubleshooting/postgresql.md +++ b/doc/administration/troubleshooting/postgresql.md @@ -1,22 +1,23 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- # PostgreSQL -This page is useful information about PostgreSQL that the GitLab Support -Team sometimes uses while troubleshooting. GitLab is making this public, so that anyone -can make use of the Support team's collected knowledge. +This page contains information about PostgreSQL the GitLab Support team uses +when troubleshooting. GitLab makes this information public, so that anyone can +make use of the Support team's collected knowledge. -CAUTION: **Caution:** -Some procedures documented here may break your GitLab instance. Use at your own risk. +WARNING: +Some procedures documented here may break your GitLab instance. Use at your +own risk. -If you are on a [paid tier](https://about.gitlab.com/pricing/) and are not sure how -to use these commands, it is best to [contact Support](https://about.gitlab.com/support/) -and they will assist you with any issues you are having. +If you're on a [paid tier](https://about.gitlab.com/pricing/) and aren't sure +how to use these commands, [contact Support](https://about.gitlab.com/support/) +for assistance with any issues you're having. ## Other GitLab PostgreSQL documentation @@ -24,51 +25,57 @@ This section is for links to information elsewhere in the GitLab documentation. ### Procedures -- [Connect to the PostgreSQL console.](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database) +- [Connect to the PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database). -- [Omnibus database procedures](https://docs.gitlab.com/omnibus/settings/database.html) including +- [Omnibus database procedures](https://docs.gitlab.com/omnibus/settings/database.html) including: - SSL: enabling, disabling, and verifying. - Enabling Write Ahead Log (WAL) archiving. - Using an external (non-Omnibus) PostgreSQL installation; and backing it up. - Listening on TCP/IP as well as or instead of sockets. - Storing data in another location. - Destructively reseeding the GitLab database. - - Guidance around updating packaged PostgreSQL, including how to stop it happening automatically. + - Guidance around updating packaged PostgreSQL, including how to stop it + happening automatically. -- [More about external PostgreSQL](../postgresql/external.md) +- [Information about external PostgreSQL](../postgresql/external.md). -- [Running Geo with external PostgreSQL](../geo/setup/external_database.md) +- [Running Geo with external PostgreSQL](../geo/setup/external_database.md). -- [Upgrades when running PostgreSQL configured for HA.](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-gitlab-ha-cluster) +- [Upgrades when running PostgreSQL configured for HA](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-gitlab-ha-cluster). -- Consuming PostgreSQL from [within CI runners](../../ci/services/postgres.md) +- Consuming PostgreSQL from [within CI runners](../../ci/services/postgres.md). -- [Using Slony to update PostgreSQL](../../update/upgrading_postgresql_using_slony.md) - - Uses replication to handle PostgreSQL upgrades - providing the schemas are the same. - - Reduces downtime to a short window for swinging over to the newer version. +- [Using Slony to update PostgreSQL](../../update/upgrading_postgresql_using_slony.md). + - Uses replication to handle PostgreSQL upgrades if the schemas are the same. + - Reduces downtime to a short window for switching to the newer version. -- Managing Omnibus PostgreSQL versions [from the development docs](https://docs.gitlab.com/omnibus/development/managing-postgresql-versions.html) +- Managing Omnibus PostgreSQL versions [from the development docs](https://docs.gitlab.com/omnibus/development/managing-postgresql-versions.html). - [PostgreSQL scaling](../postgresql/replication_and_failover.md) - - including [troubleshooting](../postgresql/replication_and_failover.md#troubleshooting) `gitlab-ctl repmgr-check-master` (or `gitlab-ctl patroni check-leader` if you are using Patroni) and PgBouncer errors + - Including [troubleshooting](../postgresql/replication_and_failover.md#troubleshooting) + `gitlab-ctl repmgr-check-master` (or `gitlab-ctl patroni check-leader` if + you're using Patroni) and PgBouncer errors. -- [Developer database documentation](../../development/README.md#database-guides) - some of which is absolutely not for production use. Including: - - understanding EXPLAIN plans +- [Developer database documentation](../../development/README.md#database-guides), + some of which is absolutely not for production use. Including: + - Understanding EXPLAIN plans. ### Troubleshooting/Fixes -- [GitLab database requirements](../../install/requirements.md#database) including - - Support for MySQL was removed in GitLab 12.1; [migrate to PostgreSQL](../../update/mysql_to_postgresql.md) - - required extension `pg_trgm` - - required extension `btree_gist` +- [GitLab database requirements](../../install/requirements.md#database), + including + - Support for MySQL was removed in GitLab 12.1; [migrate to PostgreSQL](../../update/mysql_to_postgresql.md). + - Required extension: `pg_trgm` + - Required extension: `btree_gist` -- Errors like this in the `production/sidekiq` log; see: [Set default_transaction_isolation into read committed](https://docs.gitlab.com/omnibus/settings/database.html#set-default_transaction_isolation-into-read-committed): +- Errors like this in the `production/sidekiq` log; see: + [Set default_transaction_isolation into read committed](https://docs.gitlab.com/omnibus/settings/database.html#set-default_transaction_isolation-into-read-committed): ```plaintext ActiveRecord::StatementInvalid PG::TRSerializationFailure: ERROR: could not serialize access due to concurrent update ``` -- PostgreSQL HA - [replication slot errors](https://docs.gitlab.com/omnibus/settings/database.html#troubleshooting-upgrades-in-an-ha-cluster): +- PostgreSQL HA [replication slot errors](https://docs.gitlab.com/omnibus/settings/database.html#troubleshooting-upgrades-in-an-ha-cluster): ```plaintext pg_basebackup: could not create temporary replication slot "pg_basebackup_12345": ERROR: all replication slots are in use @@ -87,11 +94,11 @@ This section is for links to information elsewhere in the GitLab documentation. PANIC: could not write to file ‘pg_xlog/xlogtemp.123’: No space left on device ``` -- [Checking Geo configuration](../geo/replication/troubleshooting.md) including - - reconfiguring hosts/ports - - checking and fixing user/password mappings +- [Checking Geo configuration](../geo/replication/troubleshooting.md), including: + - Reconfiguring hosts/ports. + - Checking and fixing user/password mappings. -- [Common Geo errors](../geo/replication/troubleshooting.md#fixing-common-errors) +- [Common Geo errors](../geo/replication/troubleshooting.md#fixing-common-errors). ## Support topics @@ -99,9 +106,12 @@ This section is for links to information elsewhere in the GitLab documentation. References: -- [Issue #1 Deadlocks with GitLab 12.1, PostgreSQL 10.7](https://gitlab.com/gitlab-org/gitlab/-/issues/30528) -- [Customer ticket (internal) GitLab 12.1.6](https://gitlab.zendesk.com/agent/tickets/134307) and [Google doc (internal)](https://docs.google.com/document/d/19xw2d_D1ChLiU-MO1QzWab-4-QXgsIUcN5e_04WTKy4) -- [Issue #2 deadlocks can occur if an instance is flooded with pushes](https://gitlab.com/gitlab-org/gitlab/-/issues/33650). Provided for context about how GitLab code can have this sort of unanticipated effect in unusual situations. +- [Issue #1 Deadlocks with GitLab 12.1, PostgreSQL 10.7](https://gitlab.com/gitlab-org/gitlab/-/issues/30528). +- [Customer ticket (internal) GitLab 12.1.6](https://gitlab.zendesk.com/agent/tickets/134307) + and [Google doc (internal)](https://docs.google.com/document/d/19xw2d_D1ChLiU-MO1QzWab-4-QXgsIUcN5e_04WTKy4). +- [Issue #2 deadlocks can occur if an instance is flooded with pushes](https://gitlab.com/gitlab-org/gitlab/-/issues/33650). + Provided for context about how GitLab code can have this sort of + unanticipated effect in unusual situations. ```plaintext ERROR: deadlock detected @@ -119,17 +129,29 @@ Quoting from issue [#1](https://gitlab.com/gitlab-org/gitlab/-/issues/30528): > "If a deadlock is hit, and we resolve it through aborting the transaction after a short period, then the retry mechanisms we already have will make the deadlocked piece of work try again, and it's unlikely we'll deadlock multiple times in a row." -TIP: **Tip:** -In support, our general approach to reconfiguring timeouts (applies also to the HTTP stack as well) is that it's acceptable to do it temporarily as a workaround. If it makes GitLab usable for the customer, then it buys time to understand the problem more completely, implement a hot fix, or make some other change that addresses the root cause. Generally, the timeouts should be put back to reasonable defaults once the root cause is resolved. +NOTE: +In Support, our general approach to reconfiguring timeouts (applies also to the +HTTP stack) is that it's acceptable to do it temporarily as a workaround. If it +makes GitLab usable for the customer, then it buys time to understand the +problem more completely, implement a hot fix, or make some other change that +addresses the root cause. Generally, the timeouts should be put back to +reasonable defaults after the root cause is resolved. -In this case, the guidance we had from development was to drop deadlock_timeout and/or statement_timeout but to leave the third setting at 60s. Setting idle_in_transaction protects the database from sessions potentially hanging for days. There's more discussion in [the issue relating to introducing this timeout on GitLab.com](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1053). +In this case, the guidance we had from development was to drop deadlock_timeout +or statement_timeout, but to leave the third setting at 60s. Setting +idle_in_transaction protects the database from sessions potentially hanging for +days. There's more discussion in [the issue relating to introducing this timeout on GitLab.com](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1053). PostgresSQL defaults: - `statement_timeout = 0` (never) - `idle_in_transaction_session_timeout = 0` (never) -Comments in issue [#1](https://gitlab.com/gitlab-org/gitlab/-/issues/30528) indicate that these should both be set to at least a number of minutes for all Omnibus installations (so they don't hang indefinitely). However, 15s for statement_timeout is very short, and will only be effective if the underlying infrastructure is very performant. +Comments in issue [#1](https://gitlab.com/gitlab-org/gitlab/-/issues/30528) +indicate that these should both be set to at least a number of minutes for all +Omnibus GitLab installations (so they don't hang indefinitely). However, 15s +for statement_timeout is very short, and will only be effective if the +underlying infrastructure is very performant. See current settings with: @@ -147,5 +169,5 @@ It may take a little while to respond. {"idle_in_transaction_session_timeout"=>"1min"} ``` -NOTE: **Note:** +NOTE: These are Omnibus GitLab settings. If an external database, such as a customer's PostgreSQL installation or Amazon RDS is being used, these values don't get set, and would have to be set externally. diff --git a/doc/administration/troubleshooting/sidekiq.md b/doc/administration/troubleshooting/sidekiq.md index d415aa0d980..e4082f87c7d 100644 --- a/doc/administration/troubleshooting/sidekiq.md +++ b/doc/administration/troubleshooting/sidekiq.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Troubleshooting Sidekiq diff --git a/doc/administration/troubleshooting/ssl.md b/doc/administration/troubleshooting/ssl.md index 996856521b7..d7bfd537eca 100644 --- a/doc/administration/troubleshooting/ssl.md +++ b/doc/administration/troubleshooting/ssl.md @@ -1,32 +1,36 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- # Troubleshooting SSL -This page contains a list of common SSL-related errors and scenarios that you may face while working with GitLab. -It should serve as an addition to the main SSL docs available here: +This page contains a list of common SSL-related errors and scenarios that you +may encounter while working with GitLab. It should serve as an addition to the +main SSL docs available here: -- [Omnibus SSL Configuration](https://docs.gitlab.com/omnibus/settings/ssl.html) -- [Self-signed certificates or custom Certification Authorities for GitLab Runner](https://docs.gitlab.com/runner/configuration/tls-self-signed.html) -- [Manually configuring HTTPS](https://docs.gitlab.com/omnibus/settings/nginx.html#manually-configuring-https) +- [Omnibus SSL Configuration](https://docs.gitlab.com/omnibus/settings/ssl.html). +- [Self-signed certificates or custom Certification Authorities for GitLab Runner](https://docs.gitlab.com/runner/configuration/tls-self-signed.html). +- [Manually configuring HTTPS](https://docs.gitlab.com/omnibus/settings/nginx.html#manually-configuring-https). ## Using an internal CA certificate with GitLab -After configuring a GitLab instance with an internal CA certificate, you might not be able to access it via various CLI tools. You may see the following symptoms: +After configuring a GitLab instance with an internal CA certificate, you might +not be able to access it by using various CLI tools. You may see experience the +following issues: - `curl` fails: ```shell - curl https://gitlab.domain.tld + curl "https://gitlab.domain.tld" curl: (60) SSL certificate problem: unable to get local issuer certificate More details here: https://curl.haxx.se/docs/sslcerts.html ``` -- Testing via the [rails console](../operations/rails_console.md#starting-a-rails-console-session) also fails: +- Testing by using the [rails console](../operations/rails_console.md#starting-a-rails-console-session) + also fails: ```ruby uri = URI.parse("https://gitlab.domain.tld") @@ -40,33 +44,36 @@ After configuring a GitLab instance with an internal CA certificate, you might n OpenSSL::SSL::SSLError (SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)) ``` -- The error `SSL certificate problem: unable to get local issuer certificate` is shown when setting up a [mirror](../../user/project/repository/repository_mirroring.md#repository-mirroring) from this GitLab instance. +- The error `SSL certificate problem: unable to get local issuer certificate` + is displayed when setting up a [mirror](../../user/project/repository/repository_mirroring.md#repository-mirroring) + from this GitLab instance. - `openssl` works when specifying the path to the certificate: ```shell /opt/gitlab/embedded/bin/openssl s_client -CAfile /root/my-cert.crt -connect gitlab.domain.tld:443 ``` -If you have the problems listed above, add your certificate to `/etc/gitlab/trusted-certs` and run `sudo gitlab-ctl reconfigure`. +If you have the previously described issues, add your certificate to +`/etc/gitlab/trusted-certs`, and then run `sudo gitlab-ctl reconfigure`. ## X.509 key values mismatch error -After configuring your instance with a certificate bundle, NGINX may throw the -following error: +After configuring your instance with a certificate bundle, NGINX may display +the following error message: `SSL: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch` -This error means that the server certificate and key you have provided do not -match. You can confirm this by running the following command and comparing the -output: +This error message means that the server certificate and key you have provided +don't match. You can confirm this by running the following command and then +comparing the output: ```shell openssl rsa -noout -modulus -in path/to/your/.key | openssl md5 openssl x509 -noout -modulus -in path/to/your/.crt | openssl md5 ``` -The following is an example of an md5 output between a matching key and certificate. Note the -matching md5 hashes: +The following is an example of an md5 output between a matching key and +certificate. Note the matching md5 hashes: ```shell $ openssl rsa -noout -modulus -in private.key | openssl md5 @@ -75,7 +82,8 @@ $ openssl x509 -noout -modulus -in public.crt | openssl md5 4f49b61b25225abeb7542b29ae20e98c ``` -This is an opposing output with a non-matching key and certificate which shows different md5 hashes: +This is an opposing output with a non-matching key and certificate which shows +different md5 hashes: ```shell $ openssl rsa -noout -modulus -in private.key | openssl md5 @@ -84,14 +92,16 @@ $ openssl x509 -noout -modulus -in public.crt | openssl md5 4f49b61b25225abeb7542b29ae20e98c ``` -If the two outputs differ like the above example, there is a mismatch between the certificate -and key. You should contact the provider of the SSL certificate for further support. +If the two outputs differ like the previous example, there's a mismatch between +the certificate and key. Contact the provider of the SSL certificate for +further support. ## Using GitLab Runner with a GitLab instance configured with internal CA certificate or self-signed certificate Besides getting the errors mentioned in [Using an internal CA certificate with GitLab](ssl.md#using-an-internal-ca-certificate-with-gitlab), -your CI pipelines may get stuck in `Pending` status. In the runner logs you may see the below error: +your CI pipelines may get stuck in `Pending` status. In the runner logs you may +see the following error message: ```shell Dec 6 02:43:17 runner-host01 gitlab-runner[15131]: #033[0;33mWARNING: Checking for jobs... failed @@ -100,23 +110,27 @@ https://gitlab.domain.tld/api/v4/jobs/request: Post https://gitlab.domain.tld/ap x509: certificate signed by unknown authority ``` -If you face similar problem, add your certificate to `/etc/gitlab-runner/certs` and restart the runner via `gitlab-runner restart`. +If you encounter a similar problem, add your certificate to `/etc/gitlab-runner/certs`, +and the restart the runner by running `gitlab-runner restart`. ## Mirroring a remote GitLab repository that uses a self-signed SSL certificate -**Scenario:** When configuring a local GitLab instance to [mirror a repository](../../user/project/repository/repository_mirroring.md) from a remote GitLab instance that uses a self-signed certificate, you may see the `SSL certificate problem: self signed certificate` error in the UI. +When configuring a local GitLab instance to [mirror a repository](../../user/project/repository/repository_mirroring.md) +from a remote GitLab instance that uses a self-signed certificate, you may see +the `SSL certificate problem: self signed certificate` error message in the +user interface. The cause of the issue can be confirmed by checking if: - `curl` fails: ```shell - $ curl https://gitlab.domain.tld + $ curl "https://gitlab.domain.tld" curl: (60) SSL certificate problem: self signed certificate More details here: https://curl.haxx.se/docs/sslcerts.html ``` -- Testing via the Rails console also fails: +- Testing by using the Rails console also fails: ```ruby uri = URI.parse("https://gitlab.domain.tld") @@ -132,10 +146,15 @@ The cause of the issue can be confirmed by checking if: To fix this problem: -- Add the self-signed certificate from the remote GitLab instance to the `/etc/gitlab/trusted-certs` directory on the local GitLab instance and run `sudo gitlab-ctl reconfigure` as per the instructions for [installing custom public certificates](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). -- If your local GitLab instance was installed using the Helm Charts, you can [add your self-signed certificate to your GitLab instance](https://docs.gitlab.com/runner/install/kubernetes.html#providing-a-custom-certificate-for-accessing-gitlab). +- Add the self-signed certificate from the remote GitLab instance to the + `/etc/gitlab/trusted-certs` directory on the local GitLab instance, and then + run `sudo gitlab-ctl reconfigure` as per the instructions for + [installing custom public certificates](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). +- If your local GitLab instance was installed using the Helm Charts, you can + [add your self-signed certificate to your GitLab instance](https://docs.gitlab.com/runner/install/kubernetes.html#providing-a-custom-certificate-for-accessing-gitlab). -You may also get another error when trying to mirror a repository from a remote GitLab instance that uses a self-signed certificate: +You may also get another error message when trying to mirror a repository from +a remote GitLab instance that uses a self-signed certificate: ```shell 2:Fetching remote upstream failed: fatal: unable to access &#39;https://gitlab.domain.tld/root/test-repo/&#39;: @@ -144,12 +163,16 @@ SSL: unable to obtain common name from peer certificate In this case, the problem can be related to the certificate itself: -- Double check that your self-signed certificate is not missing a common name. If it is then regenerate a valid certificate -- add it to `/etc/gitlab/trusted-certs` and run `sudo gitlab-ctl reconfigure` +1. Validate that your self-signed certificate isn't missing a common name. If it + is, regenerate a valid certificate +1. Add the certificate to `/etc/gitlab/trusted-certs`. +1. Run `sudo gitlab-ctl reconfigure`. ## Unable to perform Git operations due to an internal or self-signed certificate -If your GitLab instance is using a self-signed certificate, or the certificate is signed by an internal certificate authority (CA), you might run into the following errors when attempting to perform Git operations: +If your GitLab instance is using a self-signed certificate, or if the +certificate is signed by an internal certificate authority (CA), you might +experience the following errors when attempting to perform Git operations: ```shell $ git clone https://gitlab.domain.tld/group/project.git @@ -165,15 +188,19 @@ fatal: unable to access 'https://gitlab.domain.tld/group/project.git/': server c To fix this problem: -- If possible, use SSH remotes for all Git operations. This is considered more secure and convenient to use. +- If possible, use SSH remotes for all Git operations. This is considered more + secure and convenient to use. - If you must use HTTPS remotes, you can try the following: - - Copy the self signed certificate or the internal root CA certificate to a local directory (for example, `~/.ssl`) and configure Git to trust your certificate: + - Copy the self-signed certificate or the internal root CA certificate to a + local directory (for example, `~/.ssl`) and configure Git to trust your + certificate: ```shell git config --global http.sslCAInfo ~/.ssl/gitlab.domain.tld.crt ``` - - Disable SSL verification in your Git client. Note that this intended as a temporary measure as it could be considered a **security risk**. + - Disable SSL verification in your Git client. Note that this intended as a + temporary measure, as it could be considered a security risk. ```shell git config --global http.sslVerify false @@ -204,10 +231,10 @@ A misconfiguration may result in: message: SSL_connect returned=1 errno=0 state=error: wrong version number (OpenSSL::SSL::SSLError) ``` -Some of these errors come from the Excon Ruby gem, and could be generated in circumstances -where GitLab is configured to initiate an HTTPS session to a remote server -that is serving just HTTP. +Some of these errors come from the Excon Ruby gem, and could be generated in +circumstances where GitLab is configured to initiate an HTTPS session to a +remote server that is serving only HTTP. -One scenario is that you're using [object storage](../object_storage.md) -which is not served under HTTPS. GitLab is misconfigured and attempts a TLS handshake, +One scenario is that you're using [object storage](../object_storage.md), which +isn't served under HTTPS. GitLab is misconfigured and attempts a TLS handshake, but the object storage will respond with plain HTTP. diff --git a/doc/administration/troubleshooting/test_environments.md b/doc/administration/troubleshooting/test_environments.md index 9855a27ca30..53e51bbfe80 100644 --- a/doc/administration/troubleshooting/test_environments.md +++ b/doc/administration/troubleshooting/test_environments.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -13,7 +13,7 @@ for users with experience with these tools. If you are currently having an issue GitLab, you may want to check your [support options](https://about.gitlab.com/support/) first, before attempting to use this information. -NOTE: **Note:** +NOTE: This page was initially written for Support Engineers, so some of the links are only available internally at GitLab. @@ -103,9 +103,14 @@ docker run -d --name elasticsearch \ docker.elastic.co/elasticsearch/elasticsearch:5.5.1 ``` -Then confirm it works in the browser at `curl http://<IP_ADDRESS>:9200/_cat/health`. +Then confirm it works in the browser at `curl "http://<IP_ADDRESS>:9200/_cat/health"`. Elasticsearch's default username is `elastic` and password is `changeme`. +### Kroki + +See [our Kroki docs](../integration/kroki.md#docker) +on running Kroki in Docker. + ### PlantUML See [our PlantUML docs](../integration/plantuml.md#docker) diff --git a/doc/administration/troubleshooting/tracing_correlation_id.md b/doc/administration/troubleshooting/tracing_correlation_id.md index 8840c2706d6..2981b9e1368 100644 --- a/doc/administration/troubleshooting/tracing_correlation_id.md +++ b/doc/administration/troubleshooting/tracing_correlation_id.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -56,10 +56,10 @@ interested in. ### Getting the correlation ID from curl -If you're using `curl` then you can use the verbose option to show request and response headers, as well as other debug info. +If you're using `curl` then you can use the verbose option to show request and response headers, as well as other debug information. ```shell -➜ ~ curl --verbose https://gitlab.example.com/api/v4/projects +➜ ~ curl --verbose "https://gitlab.example.com/api/v4/projects" # look for a line that looks like this < x-request-id: 4rAMkV3gof4 ``` |
