diff options
Diffstat (limited to 'doc/administration/troubleshooting')
35 files changed, 133 insertions, 1676 deletions
diff --git a/doc/administration/troubleshooting/elasticsearch.md b/doc/administration/troubleshooting/elasticsearch.md index 7ce09252680..7390f4bc816 100644 --- a/doc/administration/troubleshooting/elasticsearch.md +++ b/doc/administration/troubleshooting/elasticsearch.md @@ -1,404 +1,11 @@ --- -stage: Data Stores -group: Global Search -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../../integration/advanced_search/elasticsearch_troubleshooting.md' +remove_date: '2022-11-02' --- -# Troubleshooting Elasticsearch **(PREMIUM SELF)** +This document was moved to [another location](../../integration/advanced_search/elasticsearch_troubleshooting.md). -To install and configure Elasticsearch, -visit the [administrator documentation](../../integration/advanced_search/elasticsearch.md). - -For troubleshooting, visit the -[administrator troubleshooting documentation](../../integration/advanced_search/elasticsearch_troubleshooting.md). - -Troubleshooting Elasticsearch requires: - -- Knowledge of common terms. -- Establishing within which category the problem fits. - -## Common terminology - -- **Lucene**: A full-text search library written in Java. -- **Near real time (NRT)**: Refers to the slight latency from the time to index a - document to the time when it becomes searchable. -- **Cluster**: A collection of one or more nodes that work together to hold all - the data, providing indexing and search capabilities. -- **Node**: A single server that works as part of a cluster. -- **Index**: A collection of documents that have somewhat similar characteristics. -- **Document**: A basic unit of information that can be indexed. -- **Shards**: Fully-functional and independent subdivisions of indices. Each shard is actually - a Lucene index. -- **Replicas**: Failover mechanisms that duplicate indices. - -## Troubleshooting workflows - -The type of problem will determine what steps to take. The possible troubleshooting workflows are for: - -- Search results. -- Indexing. -- Integration. -- Performance. -- Advanced Search Migrations. - -### Search Results workflow - -The following workflow is for Elasticsearch search results issues: - -```mermaid -graph TD; - B --> |No| B1 - B --> |Yes| B4 - B1 --> B2 - B2 --> B3 - B4 --> B5 - B5 --> |Yes| B6 - B5 --> |No| B7 - B7 --> B8 - B{Is GitLab using<br>Elasticsearch for<br>searching?} - B1[From the Admin Area, select<br>Integrations from the left<br>sidebar to ensure the settings<br>are correct.] - B2[Perform a search via<br>the rails console] - B3[If all settings are correct<br>and it still doesn't show Elasticsearch<br>doing the searches, escalate<br>to GitLab support.] - B4[Perform<br>the same search via the<br>Elasticsearch API] - B5{Are the results<br>the same?} - B6[This means it is working as intended.<br>Speak with GitLab support<br>to confirm if the issue lies with<br>the filters.] - B7[Check the index status of the project<br>containing the missing search<br>results.] - B8(Indexing Troubleshooting) -``` - -### Indexing workflow - -The following workflow is for Elasticsearch indexing issues: - -```mermaid -graph TD; - C --> |Yes| C1 - C1 --> |Yes| C2 - C1 --> |No| C3 - C3 --> |Yes| C4 - C3 --> |No| C5 - C --> |No| C6 - C6 --> |No| C10 - C7 --> |GitLab| C8 - C7 --> |Elasticsearch| C9 - C6 --> |Yes| C7 - C10 --> |No| C12 - C10 --> |Yes| C11 - C12 --> |Yes| C13 - C12 --> |No| C14 - C14 --> |Yes| C15 - C14 --> |No| C16 - C{Is the problem with<br>creating an empty<br>index?} - C1{Does the gitlab-production<br>index exist on the<br>Elasticsearch instance?} - C2(Try to manually<br>delete the index on the<br>Elasticsearch instance and<br>retry creating an empty index.) - C3{Can indices be made<br>manually on the Elasticsearch<br>instance?} - C4(Retry the creation of an empty index) - C5(It is best to speak with an<br>Elasticsearch admin concerning the<br>instance's inability to create indices.) - C6{Is the indexer presenting<br>errors during indexing?} - C7{Is the error a GitLab<br>error or an Elasticsearch<br>error?} - C8[Escalate to<br>GitLab support] - C9[You will want<br>to speak with an<br>Elasticsearch admin.] - C10{Does the index status<br>show 100%?} - C11[Escalate to<br>GitLab support] - C12{Does re-indexing the project<br> present any GitLab errors?} - C13[Rectify the GitLab errors and<br>restart troubleshooting, or<br>escalate to GitLab support.] - C14{Does re-indexing the project<br>present errors on the <br>Elasticsearch instance?} - C15[It would be best<br>to speak with an<br>Elasticsearch admin.] - C16[This is likely a bug/issue<br>in GitLab and will require<br>deeper investigation. Escalate<br>to GitLab support.] -``` - -### Integration workflow - -The following workflow is for Elasticsearch integration issues: - -```mermaid -graph TD; - D --> |No| D1 - D --> |Yes| D2 - D2 --> |No| D3 - D2 --> |Yes| D4 - D4 --> |No| D5 - D4 --> |Yes| D6 - D{Is the error concerning<br>the Go indexer?} - D1[It would be best<br>to speak with an<br>Elasticsearch admin.] - D2{Is the ICU development<br>package installed?} - D3>This package is required.<br>Install the package<br>and retry.] - D4{Is the error stemming<br>from the indexer?} - D5[This would indicate an OS level<br> issue. It would be best to<br>contact your sysadmin.] - D6[This is likely a bug/issue<br>in GitLab and will require<br>deeper investigation. Escalate<br>to GitLab support.] -``` - -### Performance workflow - -The following workflow is for Elasticsearch performance issues: - -```mermaid -graph TD; - F --> |Yes| F1 - F --> |No| F2 - F2 --> |No| F3 - F2 --> |Yes| F4 - F4 --> F5 - F5 --> |No| F6 - F5 --> |Yes| F7 - F{Is the Elasticsearch instance<br>running on the same server<br>as the GitLab instance?} - F1(This is not advised and will cause issues.<br>We recommend moving the Elasticsearch<br>instance to a different server.) - F2{Does the Elasticsearch<br>server have at least 8<br>GB of RAM and 2 CPU<br>cores?} - F3(According to Elasticsearch, a non-prod<br>server needs these as a base requirement.<br>Production often requires more. We recommend<br>you increase the server specifications.) - F4(Obtain the <br>cluster health information) - F5(Does it show the<br>status as green?) - F6(We recommend you speak with<br>an Elasticsearch admin<br>about implementing sharding.) - F7(Escalate to<br>GitLab support.) -``` - -### Advanced Search Migrations workflow - -```mermaid -graph TD; - D --> |No| D1 - D --> |Yes| D2 - D2 --> |No| D3 - D2 --> |Yes| D4 - D4 --> |No| D5 - D4 --> |Yes| D6 - D6 --> |No| D8 - D6 --> |Yes| D7 - - D{Is there a halted migration?} - D1[Migrations run in the<br>background and will<br>stop when completed.] - D2{Does the elasticsearch.log<br>file contain errors?} - D3[This is likely a bug/issue<br>in GitLab and will require<br>deeper investigation. Escalate<br>to GitLab support.] - D4{Have the errors<br>been addressed?} - D5[Have an Elasticsearch admin<br>review and address<br>the errors.] - D6{Has the migration<br>been retried?} - D7[This is likely a bug/issue<br>in GitLab and will require<br>deeper investigation. Escalate<br>to GitLab support.] - D8[Retry the migration from<br>the Admin > Settings ><br>Advanced Search UI.] -``` - -## Troubleshooting walkthrough - -Most Elasticsearch troubleshooting can be broken down into 4 categories: - -- [Troubleshooting search results](#troubleshooting-search-results) -- [Troubleshooting indexing](#troubleshooting-indexing) -- [Troubleshooting integration](#troubleshooting-integration) -- [Troubleshooting performance](#troubleshooting-performance) -- [Troubleshooting Advanced Search migrations](#troubleshooting-advanced-search-migrations) - -Generally speaking, if it does not fall into those four categories, it is either: - -- Something GitLab support needs to look into. -- Not a true Elasticsearch issue. - -Exercise caution. Issues that appear to be Elasticsearch problems can be OS-level issues. - -### Troubleshooting search results - -Troubleshooting search result issues is rather straight forward on Elasticsearch. - -The first step is to confirm GitLab is using Elasticsearch for the search function. -To do this: - -1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **Settings > General**, and then confirm the - integration is enabled. -1. Confirm searches use Elasticsearch by accessing the rails console - (`sudo gitlab-rails console`) and running the following commands: - - ```rails - u = User.find_by_email('email_of_user_doing_search') - s = SearchService.new(u, {:search => 'search_term'}) - pp s.search_objects.class - ``` - -The output from the last command is the key here. If it shows: - -- `ActiveRecord::Relation`, **it is not** using Elasticsearch. -- `Kaminari::PaginatableArray`, **it is** using Elasticsearch. - -| Not using Elasticsearch | Using Elasticsearch | -|--------------------------|------------------------------| -| `ActiveRecord::Relation` | `Kaminari::PaginatableArray` | - -If all the settings look correct and it is still not using Elasticsearch for the search function, it is best to escalate to GitLab support. This could be a bug/issue. - -Moving past that, it is best to attempt the same [search via the Rails console](../../integration/advanced_search/elasticsearch_troubleshooting.md#i-indexed-all-the-repositories-but-i-cant-get-any-hits-for-my-search-term-in-the-ui) -or the [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html), -and compare the results from what you see in GitLab. - -If the results: - -- Sync up, then there is not a technical "issue." Instead, it might be a problem - with the Elasticsearch filters we are using. This can be complicated, so it is best to - escalate to GitLab support to check these and guide you on the potential on whether or - not a feature request is needed. -- Do not match up, this indicates a problem with the documents generated from the - project. It is best to re-index that project and proceed with - [Troubleshooting indexing](#troubleshooting-indexing). - -### Troubleshooting indexing - -Troubleshooting indexing issues can be tricky. It can pretty quickly go to either GitLab -support or your Elasticsearch administrator. - -The best place to start is to determine if the issue is with creating an empty index. -If it is, check on the Elasticsearch side to determine if the `gitlab-production` (the -name for the GitLab index) exists. If it exists, manually delete it on the Elasticsearch -side and attempt to recreate it from the -[`recreate_index`](../../integration/advanced_search/elasticsearch.md#gitlab-advanced-search-rake-tasks) -Rake task. - -If you still encounter issues, try creating an index manually on the Elasticsearch -instance. The details of the index aren't important here, as we want to test if indices -can be made. If the indices: - -- Cannot be made, speak with your Elasticsearch administrator. -- Can be made, Escalate this to GitLab support. - -If the issue is not with creating an empty index, the next step is to check for errors -during the indexing of projects. If errors do occur, they stem from either the indexing: - -- On the GitLab side. You need to rectify those. If they are not - something you are familiar with, contact GitLab support for guidance. -- Within the Elasticsearch instance itself. See if the error is [documented and has a fix](../../integration/advanced_search/elasticsearch_troubleshooting.md). If not, speak with your Elasticsearch administrator. - -If the indexing process does not present errors, check the status of the indexed projects. You can do this via the following Rake tasks: - -- [`sudo gitlab-rake gitlab:elastic:index_projects_status`](../../integration/advanced_search/elasticsearch.md#gitlab-advanced-search-rake-tasks) (shows the overall status) -- [`sudo gitlab-rake gitlab:elastic:projects_not_indexed`](../../integration/advanced_search/elasticsearch.md#gitlab-advanced-search-rake-tasks) (shows specific projects that are not indexed) - -If: - -- Everything is showing at 100%, escalate to GitLab support. This could be a potential - bug/issue. -- You do see something not at 100%, attempt to reindex that project. To do this, - run `sudo gitlab-rake gitlab:elastic:index_projects ID_FROM=<project ID> ID_TO=<project ID>`. - -If reindexing the project shows: - -- Errors on the GitLab side, escalate those to GitLab support. -- Elasticsearch errors or doesn't present any errors at all, reach out to your - Elasticsearch administrator to check the instance. - -### Troubleshooting integration - -Troubleshooting integration tends to be pretty straight forward, as there really isn't -much to "integrate" here. - -If the issue is: - -- With the Go indexer, check if the ICU development package is installed. - This is a required package so make sure you install it. - Go indexer was a beta indexer which can be optionally turned on/off, but in 12.3 it reached stable status and is now the default. -- Not concerning the Go indexer, it is almost always an - Elasticsearch-side issue. This means you should reach out to your Elasticsearch administrator - regarding the errors you are seeing. If you are unsure here, it never hurts to reach - out to GitLab support. - -Beyond that, review the error. If it is: - -- Specifically from the indexer, this could be a bug/issue and should be escalated to - GitLab support. -- An OS issue, you should reach out to your systems administrator. -- A `Faraday::TimeoutError (execution expired)` error **and** you're using a proxy, - [set a custom `gitlab_rails['env']` environment variable, called `no_proxy`](https://docs.gitlab.com/omnibus/settings/environment-variables.html) - with the IP address of your Elasticsearch host. - -### Troubleshooting performance - -Troubleshooting performance can be difficult on Elasticsearch. There is a ton of tuning -that *can* be done, but the majority of this falls on shoulders of a skilled -Elasticsearch administrator. - -Generally speaking, ensure: - -- The Elasticsearch server **is not** running on the same node as GitLab. -- The Elasticsearch server have enough RAM and CPU cores. -- That sharding **is** being used. - -Going into some more detail here, if Elasticsearch is running on the same server as GitLab, resource contention is **very** likely to occur. Ideally, Elasticsearch, which requires ample resources, should be running on its own server (maybe coupled with Logstash and Kibana). - -When it comes to Elasticsearch, RAM is the key resource. Elasticsearch themselves recommend: - -- **At least** 8 GB of RAM for a non-production instance. -- **At least** 16 GB of RAM for a production instance. -- Ideally, 64 GB of RAM. - -For CPU, Elasticsearch recommends at least 2 CPU cores, but Elasticsearch states common -setups use up to 8 cores. For more details on server specs, check out -[Elasticsearch's hardware guide](https://www.elastic.co/guide/en/elasticsearch/guide/current/hardware.html). - -Beyond the obvious, sharding comes into play. Sharding is a core part of Elasticsearch. -It allows for horizontal scaling of indices, which is helpful when you are dealing with -a large amount of data. - -With the way GitLab does indexing, there is a **huge** amount of documents being -indexed. By utilizing sharding, you can speed up Elasticsearch's ability to locate -data, since each shard is a Lucene index. - -If you are not using sharding, you are likely to hit issues when you start using -Elasticsearch in a production environment. - -Keep in mind that an index with only one shard has **no scale factor** and will -likely encounter issues when called upon with some frequency. - -If you need to know how many shards, read -[Elasticsearch's documentation on capacity planning](https://www.elastic.co/guide/en/elasticsearch/guide/2.x/capacity-planning.html), -as the answer is not straight forward. - -The easiest way to determine if sharding is in use is to check the output of the -[Elasticsearch Health API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html): - -- Red means the cluster is down. -- Yellow means it is up with no sharding/replication. -- Green means it is healthy (up, sharding, replicating). - -For production use, it should always be green. - -Beyond these steps, you get into some of the more complicated things to check, -such as merges and caching. These can get complicated and it takes some time to -learn them, so it is best to escalate/pair with an Elasticsearch expert if you need to -dig further into these. - -Feel free to reach out to GitLab support, but this is likely to be something a skilled -Elasticsearch administrator has more experience with. - -### Troubleshooting Advanced Search migrations - -Troubleshooting Advanced Search migration failures can be difficult and may -require contacting an Elasticsearch administrator or GitLab Support. - -The best place to start while debugging issues with an Advanced Search -migration is the [`elasticsearch.log` file](../logs.md#elasticsearchlog). -Migrations log information while a migration is in progress and any -errors encountered. Apply fixes for any errors found in the log and retry -the migration. - -If you still encounter issues after retrying the migration, reach out to GitLab support. - -## Common issues - -All common issues [should be documented](../../integration/advanced_search/elasticsearch_troubleshooting.md). If not, -feel free to update that page with issues you encounter and solutions. - -## Replication - -Setting up Elasticsearch isn't too bad, but it can be a bit finicky and time consuming. - -The easiest method is to spin up a Docker container with the required version and -bind ports 9200/9300 so it can be used. - -The following is an example of running a Docker container of Elasticsearch v7.2.0: - -```shell -docker pull docker.elastic.co/elasticsearch/elasticsearch:7.2.0 -docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.2.0 -``` - -From here, you can: - -- Grab the IP of the Docker container (use `docker inspect <container_id>`) -- Use `<IP.add.re.ss:9200>` to communicate with it. - -This is a quick method to test out Elasticsearch, but by no means is this a -production solution. +<!-- This redirect file can be deleted after <2022-11-02>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 0ff1afa86ed..aa4dbec4f95 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -178,26 +178,6 @@ Feature.all Feature.all.map {|f| [f.name, f.state]} ``` -## Command Line - -### Check the GitLab version fast - -```shell -grep -m 1 gitlab /opt/gitlab/version-manifest.txt -``` - -### Debugging SSH - -```shell -GIT_SSH_COMMAND="ssh -vvv" git clone <repository> -``` - -### Debugging over HTTPS - -```shell -GIT_CURL_VERBOSE=1 GIT_TRACE=1 git clone <repository> -``` - ## Projects ### Clear a project's cache @@ -326,7 +306,7 @@ Project.find_each do |p| end ``` -## Bulk update to change all the Jira integrations to Jira instance-level values +### Bulk update to change all the Jira integrations to Jira instance-level values To change all Jira project to use the instance-level integration settings: @@ -341,6 +321,25 @@ To change all Jira project to use the instance-level integration settings: 1. Modify and save again the instance-level integration from the UI to propagate the changes to all the group-level and project-level integrations. +### Check if Jira Cloud is linked to a namespace + +```ruby +JiraConnectSubscription.where(namespace: Namespace.by_path('group/subgroup')) +``` + +### Check if Jira Cloud is linked to a project + +```ruby +Project.find_by_full_path('path/to/project').jira_subscription_exists? +``` + +### Check if Jira Cloud URL is linked to any namespace + +```ruby +installation = JiraConnectInstallation.find_by_base_url("https://customer_name.atlassian.net") +installation.subscriptions +``` + ### Bulk update to disable the Slack Notification service To disable notifications for all projects that have Slack service enabled, do: @@ -424,33 +423,6 @@ projects = Project.find_by_sql("SELECT * FROM projects WHERE name LIKE '%ject'") => [#<Project id:12 root/my-first-project>>, #<Project id:13 root/my-second-project>>] ``` -## Wikis - -### Recreate - -WARNING: -This is a destructive operation, the Wiki becomes empty. - -A Projects Wiki can be recreated by this command: - -```ruby -p = Project.find_by_full_path('<username-or-group>/<project-name>') ### enter your projects path - -GitlabShellWorker.perform_in(0, :remove_repository, p.repository_storage, p.wiki.disk_path) ### deletes the wiki project from the filesystem - -p.create_wiki ### creates the wiki project on the filesystem -``` - -## Issue boards - -### In case of issue boards not loading properly and it's getting time out. Call the Issue Rebalancing service to fix this - -```ruby -p = Project.find_by_full_path('<username-or-group>/<project-name>') - -Issues::RelativePositionRebalancingService.new(p.root_namespace.all_projects).execute -``` - ## Imports and exports ### Import a project @@ -855,52 +827,6 @@ Gitlab::CurrentSettings.update!(password_authentication_enabled_for_web: true) ## SCIM -### Fixing bad SCIM identities - -```ruby -def delete_bad_scim(email, group_path) - output = "" - u = User.find_by_email(email) - uid = u.id - g = Group.find_by_full_path(group_path) - saml_prov_id = SamlProvider.find_by(group_id: g.id).id - saml = Identity.where(user_id: uid, saml_provider_id: saml_prov_id) - scim = ScimIdentity.where(user_id: uid , group_id: g.id) - if saml[0] - saml_eid = saml[0].extern_uid - output += "%s," % [email] - output += "SAML: %s," % [saml_eid] - if scim[0] - scim_eid = scim[0].extern_uid - output += "SCIM: %s" % [scim_eid] - if saml_eid == scim_eid - output += " Identities matched, not deleted \n" - else - scim[0].destroy - output += " Deleted \n" - end - else - output = "ERROR No SCIM identify found for: [%s]\n" % [email] - puts output - return 1 - end - else - output = "ERROR No SAML identify found for: [%s]\n" % [email] - puts output - return 1 - end - puts output - return 0 -end - -# In case of multiple emails -emails = [email1, email2] - -emails.each do |e| - delete_bad_scim(e,'<group-path>') -end -``` - ### Find groups using an SQL query Find and store an array of groups based on an SQL query: @@ -927,13 +853,13 @@ conflicting_permanent_redirects.destroy_all ## Merge requests -### Close a merge request properly (if merged but still marked as open) +### Close a merge request ```ruby u = User.find_by_username('<username>') p = Project.find_by_full_path('<namespace/project>') m = p.merge_requests.find_by(iid: <iid>) -MergeRequests::PostMergeService.new(project: p, current_user: u).execute(m) +MergeRequests::CloseService.new(project: p, current_user: u).execute(m) ``` ### Delete a merge request @@ -954,6 +880,21 @@ m = p.merge_requests.find_by(iid: <iid>) MergeRequests::RebaseService.new(project: m.target_project, current_user: u).execute(m) ``` +### Set a merge request as merged + +Use when a merge request was accepted and the changes merged into the Git repository, +but the merge request still shows as open. + +If the changes are not merged yet, this action causes the merge request to +incorrectly show `merged into <branch-name>`. + +```ruby +u = User.find_by_username('<username>') +p = Project.find_by_full_path('<namespace/project>') +m = p.merge_requests.find_by(iid: <iid>) +MergeRequests::PostMergeService.new(project: p, current_user: u).execute(m) +``` + ## CI ### Cancel stuck pending pipelines @@ -1063,6 +1004,9 @@ License.current.trial? # License ID for lookup on CustomersDot License.current.license_id + +# License data in Base64-encoded ASCII format +License.current.data ``` ### Check if a project feature is available on the instance @@ -1378,16 +1322,6 @@ registry = Geo::SnippetRepositoryRegistry.find(registry_id) registry.replicator.send(:sync_repository) ``` -## Gitaly - -### Find available and used space - -A Gitaly storage resource can be polled through Rails to determine the available and used space. - -```ruby -Gitlab::GitalyClient::ServerService.new("default").storage_disk_statistics -``` - ## Generate Service Ping The [Service Ping Guide](../../development/service_ping/index.md) in our developer documentation @@ -1429,28 +1363,6 @@ Prints the metrics saved in `conversational_development_index_metrics`. rake gitlab:usage_data:generate_and_send ``` -## Kubernetes integration - -Find cluster: - -```ruby -cluster = Clusters::Cluster.find(1) -cluster = Clusters::Cluster.find_by(name: 'cluster_name') -``` - -Delete cluster without associated resources: - -```ruby -# Find users with the administrator access -user = User.find_by(username: 'admin_user') - -# Find the cluster with the ID -cluster = Clusters::Cluster.find(1) - -# Delete the cluster -Clusters::DestroyService.new(user).execute(cluster) -``` - ## Elasticsearch ### Configuration attributes diff --git a/doc/administration/troubleshooting/group_saml_scim.md b/doc/administration/troubleshooting/group_saml_scim.md index 145eb5f65ae..b5187504231 100644 --- a/doc/administration/troubleshooting/group_saml_scim.md +++ b/doc/administration/troubleshooting/group_saml_scim.md @@ -1,207 +1,11 @@ --- -stage: Manage -group: Authentication and Authorization -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference +redirect_to: '../../user/group/saml_sso/example_saml_config.md' +remove_date: '2022-10-29' --- -# Troubleshooting Group SAML and SCIM **(PREMIUM SAAS)** +This document was moved to [another location](../../user/group/saml_sso/example_saml_config.md). -These are notes and screenshots regarding Group SAML and SCIM that the GitLab Support Team sometimes uses while troubleshooting, but which do not fit into the official documentation. GitLab is making this public, so that anyone can make use of the Support team's collected knowledge. - -Please refer to the GitLab [Group SAML](../../user/group/saml_sso/index.md) docs for information on the feature and how to set it up. - -When troubleshooting a SAML configuration, GitLab team members will frequently start with the [SAML troubleshooting section](../../user/group/saml_sso/index.md#troubleshooting). - -They may then set up a test configuration of the desired identity provider. We include example screenshots in this section. - -## SAML and SCIM screenshots - -This section includes relevant screenshots of the following example configurations of [Group SAML](../../user/group/saml_sso/index.md) and [Group SCIM](../../user/group/saml_sso/scim_setup.md): - -- [Azure Active Directory](#azure-active-directory) -- [Google Workspace](#google-workspace) -- [Okta](#okta) -- [OneLogin](#onelogin) - -WARNING: -These screenshots are updated only as needed by GitLab Support. They are **not** official documentation. - -If you are currently having an issue with GitLab, you may want to check your [support options](https://about.gitlab.com/support/). - -## Azure Active Directory - -Basic SAML app configuration: - - - -User claims and attributes: - - - -SCIM mapping: - - - - -Group Sync: - - - -## Google Workspace - -Basic SAML app configuration: - - - -User claims and attributes: - - - -IdP links and certificate: - -NOTE: -Google Workspace displays a SHA256 fingerprint. To retrieve the SHA1 fingerprint required by GitLab for configuring SAML, download the certificate and calculate the SHA1 certificate -fingerprint. - - - -## Okta - -Basic SAML app configuration: - - - -User claims and attributes: - - - -Advanced SAML app settings (defaults): - - - -IdP Links and Certificate: - - - -Sign on settings: - - - -Self-managed instance example: - - - -Setting the username for the newly provisioned users when assigning them the SCIM app: - - - -## OneLogin - -Application details: - - - -Parameters: - - - -Adding a user: - - - -SSO settings: - - - -## SAML response example - -When a user signs in using SAML, GitLab receives a SAML response. The SAML response can be found in `production.log` logs as a base64-encoded message. Locate the response by -searching for `SAMLResponse`. The decoded SAML response is in XML format. For example: - -```xml -<?xml version="1.0" encoding="UTF-8"?> -<saml2p:Response xmlns:saml2p="urn:oasis:names:tc:SAML:2.0:protocol" xmlns:xs="http://www.w3.org/2001/XMLSchema" Destination="https://gitlabexample/-/saml/callback" ID="id4898983630840142426821432" InResponseTo="_c65e4c88-9425-4472-b42c-37f4186ac0ee" IssueInstant="2022-05-30T21:30:35.696Z" Version="2.0"> - <saml2:Issuer xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion" Format="urn:oasis:names:tc:SAML:2.0:nameid-format:entity">http://www.okta.com/exk2y6j57o1Pdr2lI8qh7</saml2:Issuer> - <ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#"> - <ds:SignedInfo> - <ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/> - <ds:SignatureMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"/> - <ds:Reference URI="#id4898983630840142426821432"> - <ds:Transforms> - <ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/> - <ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"> - <ec:InclusiveNamespaces xmlns:ec="http://www.w3.org/2001/10/xml-exc-c14n#" PrefixList="xs"/> - </ds:Transform> - </ds:Transforms> - <ds:DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/> - <ds:DigestValue>neiQvv9d3OgS4GZW8Nptp4JhjpKs3GCefibn+vmRgk4=</ds:DigestValue> - </ds:Reference> - </ds:SignedInfo> - <ds:SignatureValue>dMsQX8ivi...HMuKGhyLRvabGU6CuPrf7==</ds:SignatureValue> - <ds:KeyInfo> - <ds:X509Data> - <ds:X509Certificate>MIIDq...cptGr3vN9TQ==</ds:X509Certificate> - </ds:X509Data> - </ds:KeyInfo> - </ds:Signature> - <saml2p:Status xmlns:saml2p="urn:oasis:names:tc:SAML:2.0:protocol"> - <saml2p:StatusCode Value="urn:oasis:names:tc:SAML:2.0:status:Success"/> - </saml2p:Status> - <saml2:Assertion xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion" xmlns:xs="http://www.w3.org/2001/XMLSchema" ID="id489" IssueInstant="2022-05-30T21:30:35.696Z" Version="2.0"> - <saml2:Issuer xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion" Format="urn:oasis:names:tc:SAML:2.0:nameid-format:entity">http://www.okta.com/exk2y6j57o1Pdr2lI8qh7</saml2:Issuer> - <ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#"> - <ds:SignedInfo> - <ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/> - <ds:SignatureMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"/> - <ds:Reference URI="#id48989836309833801859473359"> - <ds:Transforms> - <ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/> - <ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"> - <ec:InclusiveNamespaces xmlns:ec="http://www.w3.org/2001/10/xml-exc-c14n#" PrefixList="xs"/> - </ds:Transform> - </ds:Transforms> - <ds:DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/> - <ds:DigestValue>MaIsoi8hbT9gsi/mNZsz449mUuAcuEWY0q3bc4asOQs=</ds:DigestValue> - </ds:Reference> - </ds:SignedInfo> - <ds:SignatureValue>dMsQX8ivi...HMuKGhyLRvabGU6CuPrf7==<</ds:SignatureValue> - <ds:KeyInfo> - <ds:X509Data> - <ds:X509Certificate>MIIDq...cptGr3vN9TQ==</ds:X509Certificate> - </ds:X509Data> - </ds:KeyInfo> - </ds:Signature> - <saml2:Subject xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion"> - <saml2:NameID Format="urn:oasis:names:tc:SAML:2.0:nameid-format:persistent">useremail@domain.com</saml2:NameID> - <saml2:SubjectConfirmation Method="urn:oasis:names:tc:SAML:2.0:cm:bearer"> - <saml2:SubjectConfirmationData InResponseTo="_c65e4c88-9425-4472-b42c-37f4186ac0ee" NotOnOrAfter="2022-05-30T21:35:35.696Z" Recipient="https://gitlab.example.com/-/saml/callback"/> - </saml2:SubjectConfirmation> - </saml2:Subject> - <saml2:Conditions xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion" NotBefore="2022-05-30T21:25:35.696Z" NotOnOrAfter="2022-05-30T21:35:35.696Z"> - <saml2:AudienceRestriction> - <saml2:Audience>https://gitlab.example.com/</saml2:Audience> - </saml2:AudienceRestriction> - </saml2:Conditions> - <saml2:AuthnStatement xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion" AuthnInstant="2022-05-30T21:30:35.696Z" SessionIndex="_c65e4c88-9425-4472-b42c-37f4186ac0ee"> - <saml2:AuthnContext> - <saml2:AuthnContextClassRef>urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport</saml2:AuthnContextClassRef> - </saml2:AuthnContext> - </saml2:AuthnStatement> - <saml2:AttributeStatement xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion"> - <saml2:Attribute Name="email" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified"> - <saml2:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">useremail@domain.com</saml2:AttributeValue> - </saml2:Attribute> - <saml2:Attribute Name="firtname" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified"> - <saml2:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">John</saml2:AttributeValue> - </saml2:Attribute> - <saml2:Attribute Name="lastname" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified"> - <saml2:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">Doe</saml2:AttributeValue> - </saml2:Attribute> - <saml2:Attribute Name="Groups" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified"> - <saml2:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">Super-awesome-group</saml2:AttributeValue> - </saml2:Attribute> - </saml2:AttributeStatement> - </saml2:Assertion> -</saml2p:Response> -``` +<!-- This redirect file can be deleted after <2022-10-29>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/troubleshooting/img/AzureAD-basic_SAML.png b/doc/administration/troubleshooting/img/AzureAD-basic_SAML.png Binary files differdeleted file mode 100644 index 7a0d83ab2dd..00000000000 --- a/doc/administration/troubleshooting/img/AzureAD-basic_SAML.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/AzureAD-claims.png b/doc/administration/troubleshooting/img/AzureAD-claims.png Binary files differdeleted file mode 100644 index 576040be337..00000000000 --- a/doc/administration/troubleshooting/img/AzureAD-claims.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/AzureAD-scim_attribute_mapping.png b/doc/administration/troubleshooting/img/AzureAD-scim_attribute_mapping.png Binary files differdeleted file mode 100644 index 5ad82003eed..00000000000 --- a/doc/administration/troubleshooting/img/AzureAD-scim_attribute_mapping.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/AzureAD-scim_provisioning.png b/doc/administration/troubleshooting/img/AzureAD-scim_provisioning.png Binary files differdeleted file mode 100644 index b2c385151a6..00000000000 --- a/doc/administration/troubleshooting/img/AzureAD-scim_provisioning.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/GoogleWorkspace-basic-SAML_v14_10.png b/doc/administration/troubleshooting/img/GoogleWorkspace-basic-SAML_v14_10.png Binary files differdeleted file mode 100644 index e002503ddc0..00000000000 --- a/doc/administration/troubleshooting/img/GoogleWorkspace-basic-SAML_v14_10.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/GoogleWorkspace-claims_v14_10.png b/doc/administration/troubleshooting/img/GoogleWorkspace-claims_v14_10.png Binary files differdeleted file mode 100644 index 2b827e122a8..00000000000 --- a/doc/administration/troubleshooting/img/GoogleWorkspace-claims_v14_10.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/GoogleWorkspace-linkscert_v14_10.png b/doc/administration/troubleshooting/img/GoogleWorkspace-linkscert_v14_10.png Binary files differdeleted file mode 100644 index 8d6c5010670..00000000000 --- a/doc/administration/troubleshooting/img/GoogleWorkspace-linkscert_v14_10.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/Okta-SAMLsetup.png b/doc/administration/troubleshooting/img/Okta-SAMLsetup.png Binary files differdeleted file mode 100644 index 1bd9bf4d7e9..00000000000 --- a/doc/administration/troubleshooting/img/Okta-SAMLsetup.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/Okta-advancedsettings.png b/doc/administration/troubleshooting/img/Okta-advancedsettings.png Binary files differdeleted file mode 100644 index 45e378d1d12..00000000000 --- a/doc/administration/troubleshooting/img/Okta-advancedsettings.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/Okta-attributes.png b/doc/administration/troubleshooting/img/Okta-attributes.png Binary files differdeleted file mode 100644 index a3405e4de9b..00000000000 --- a/doc/administration/troubleshooting/img/Okta-attributes.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/Okta-linkscert.png b/doc/administration/troubleshooting/img/Okta-linkscert.png Binary files differdeleted file mode 100644 index 38cae415f7e..00000000000 --- a/doc/administration/troubleshooting/img/Okta-linkscert.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/OneLogin-SSOsettings.png b/doc/administration/troubleshooting/img/OneLogin-SSOsettings.png Binary files differdeleted file mode 100644 index 58f936d8567..00000000000 --- a/doc/administration/troubleshooting/img/OneLogin-SSOsettings.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/OneLogin-app_details.png b/doc/administration/troubleshooting/img/OneLogin-app_details.png Binary files differdeleted file mode 100644 index 77618960897..00000000000 --- a/doc/administration/troubleshooting/img/OneLogin-app_details.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/OneLogin-parameters.png b/doc/administration/troubleshooting/img/OneLogin-parameters.png Binary files differdeleted file mode 100644 index a2fa734152c..00000000000 --- a/doc/administration/troubleshooting/img/OneLogin-parameters.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/OneLogin-userAdd.png b/doc/administration/troubleshooting/img/OneLogin-userAdd.png Binary files differdeleted file mode 100644 index 54c1ecd2e68..00000000000 --- a/doc/administration/troubleshooting/img/OneLogin-userAdd.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/azure_configure_group_claim.png b/doc/administration/troubleshooting/img/azure_configure_group_claim.png Binary files differdeleted file mode 100644 index 9d8c5348273..00000000000 --- a/doc/administration/troubleshooting/img/azure_configure_group_claim.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/database-query-dialog_v14_3.png b/doc/administration/troubleshooting/img/database-query-dialog_v14_3.png Binary files differdeleted file mode 100644 index 197cfa17da2..00000000000 --- a/doc/administration/troubleshooting/img/database-query-dialog_v14_3.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/network_monitor_xid.png b/doc/administration/troubleshooting/img/network_monitor_xid.png Binary files differdeleted file mode 100644 index fa64fd1509c..00000000000 --- a/doc/administration/troubleshooting/img/network_monitor_xid.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/obtaining-a-session-cookie-for-request_v14_3.png b/doc/administration/troubleshooting/img/obtaining-a-session-cookie-for-request_v14_3.png Binary files differdeleted file mode 100644 index a63ebf9ecf2..00000000000 --- a/doc/administration/troubleshooting/img/obtaining-a-session-cookie-for-request_v14_3.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/okta_admin_panel_v13_9.png b/doc/administration/troubleshooting/img/okta_admin_panel_v13_9.png Binary files differdeleted file mode 100644 index 2ebb1f0112c..00000000000 --- a/doc/administration/troubleshooting/img/okta_admin_panel_v13_9.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/okta_saml_settings.png b/doc/administration/troubleshooting/img/okta_saml_settings.png Binary files differdeleted file mode 100644 index ee275ece369..00000000000 --- a/doc/administration/troubleshooting/img/okta_saml_settings.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/okta_setting_username.png b/doc/administration/troubleshooting/img/okta_setting_username.png Binary files differdeleted file mode 100644 index 5f87d14bb8b..00000000000 --- a/doc/administration/troubleshooting/img/okta_setting_username.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/paste-request-id-into-progress-bar_v14_3.png b/doc/administration/troubleshooting/img/paste-request-id-into-progress-bar_v14_3.png Binary files differdeleted file mode 100644 index a19585d7a89..00000000000 --- a/doc/administration/troubleshooting/img/paste-request-id-into-progress-bar_v14_3.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/select-request-id-from-request-selector-drop-down-menu_v14_3.png b/doc/administration/troubleshooting/img/select-request-id-from-request-selector-drop-down-menu_v14_3.png Binary files differdeleted file mode 100644 index b8314056c9b..00000000000 --- a/doc/administration/troubleshooting/img/select-request-id-from-request-selector-drop-down-menu_v14_3.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/sidekiq_flamegraph.png b/doc/administration/troubleshooting/img/sidekiq_flamegraph.png Binary files differdeleted file mode 100644 index 89d6e8da3ce..00000000000 --- a/doc/administration/troubleshooting/img/sidekiq_flamegraph.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/view-pg-details_v14_3.png b/doc/administration/troubleshooting/img/view-pg-details_v14_3.png Binary files differdeleted file mode 100644 index 1fe12462e4e..00000000000 --- a/doc/administration/troubleshooting/img/view-pg-details_v14_3.png +++ /dev/null diff --git a/doc/administration/troubleshooting/index.md b/doc/administration/troubleshooting/index.md index 7fe731bda66..429dc79e95f 100644 --- a/doc/administration/troubleshooting/index.md +++ b/doc/administration/troubleshooting/index.md @@ -13,15 +13,12 @@ installation. - [SSL](ssl.md) - [Geo](../geo/replication/troubleshooting.md) -- [Elasticsearch](elasticsearch.md) -- [Sidekiq](sidekiq.md) - [GitLab Rails console cheat sheet](gitlab_rails_cheat_sheet.md) -- [Group SAML and SCIM troubleshooting](group_saml_scim.md) **(PREMIUM SAAS)** +- [Example group SAML and SCIM configurations](../../user/group/saml_sso/example_saml_config.md) - [Kubernetes cheat sheet](https://docs.gitlab.com/charts/troubleshooting/kubernetes_cheat_sheet.html) - [Linux cheat sheet](linux_cheat_sheet.md) -- [Parsing GitLab logs with `jq`](log_parsing.md) +- [Parsing GitLab logs with `jq`](../logs/log_parsing.md) - [Diagnostics tools](diagnostics_tools.md) -- [Tracing requests with correlation ID](tracing_correlation_id.md) Some feature documentation pages also have a troubleshooting section at the end that you can check for feature-specific help. diff --git a/doc/administration/troubleshooting/linux_cheat_sheet.md b/doc/administration/troubleshooting/linux_cheat_sheet.md index 66d5fb82936..6ff6e562a7d 100644 --- a/doc/administration/troubleshooting/linux_cheat_sheet.md +++ b/doc/administration/troubleshooting/linux_cheat_sheet.md @@ -14,7 +14,7 @@ having an issue with GitLab, you may want to check your [support options](https: first, before attempting to use this information. WARNING: -It is [beyond the scope of GitLab Support to assist in systems administration](https://about.gitlab.com/support/statement-of-support.html#training). GitLab administrators are expected to know these commands for their distribution +It is [beyond the scope of GitLab Support to assist in systems administration](https://about.gitlab.com/support/statement-of-support/#training). GitLab administrators are expected to know these commands for their distribution of choice. If you are a GitLab Support Engineer, consider this a cross-reference to translate `yum` -> `apt-get` and the like. @@ -204,24 +204,39 @@ or you can build it from source if you have the Rust compiler. #### How to use the tool -First run the tool with no arguments other than the strace output filename to get -a summary of the top processes sorted by time spent actively performing tasks. You -can also sort based on total time, # of system calls made, PID #, and # of child processes -using the `-S` or `--sort` flag. The number of results defaults to 25 processes, but +First run the tool with `summary` flag to get a summary of the top processes sorted by time spent actively performing tasks. +You can also sort based on total time, # of system calls made, PID #, and # of child processes +using the `-s` or `--sort` flag. The number of results defaults to 25 processes, but can be changed using the `-c`/`--count` option. See `--help` for full details. ```shell -$ ./strace-parser strace.txt +$ ./strace-parser sidekiq_trace.txt summary -c15 -s=pid -Top 25 PIDs +Top 15 PIDs by PID # ----------- - pid active (ms) wait (ms) total (ms) % active syscalls - ---------- ---------- --------- --------- --------- --------- - 8795 689.072 45773.832 46462.902 16.89% 23018 - 13408 679.432 55910.891 56590.320 16.65% 28593 - 6423 554.822 13175.485 13730.308 13.60% 13735 -... + pid actv (ms) wait (ms) user (ms) total (ms) % of actv syscalls children + ------- ---------- ---------- ---------- ---------- --------- --------- --------- + 16706 0.000 0.000 0.000 0.000 0.00% 0 0 + 16708 0.000 0.000 0.000 0.000 0.00% 0 0 + 16716 0.000 0.000 0.000 0.000 0.00% 0 0 + 16717 0.000 0.000 0.000 0.000 0.00% 0 0 + 16718 0.000 0.000 0.000 0.000 0.00% 0 0 + 16719 0.000 0.000 0.000 0.000 0.00% 0 0 + 16720 0.389 9796.434 1.090 9797.912 0.02% 16 0 + 16721 0.000 0.000 0.000 0.000 0.00% 0 0 + 16722 0.000 0.000 0.000 0.000 0.00% 0 0 + 16723 0.000 0.000 0.000 0.000 0.00% 0 0 + 16804 0.218 11099.535 1.881 11101.634 0.01% 36 0 + 16813 0.000 0.000 0.000 0.000 0.00% 0 0 + 16814 1.740 11825.640 4.616 11831.996 0.10% 57 0 + 16815 2.364 12039.993 7.669 12050.026 0.14% 80 0 + 16816 0.000 0.000 0.000 0.000 0.00% 0 0 + +PIDs 93 +real 0m12.287s +user 0m1.474s +sys 0m1.686s ``` Based on the summary, you can then view the details of system calls made by one or more @@ -229,36 +244,38 @@ processes using the `-p`/`--pid` for a specific process, or `-s`/`--stats` flags a sorted list. `--stats` takes the same sorting and count options as summary. ```shell -$ ./strace-parse strace.text -p 6423 - -PID 6423 -13735 syscalls, active time: 554.822ms, total time: 13730.308ms - - syscall count total max avg min errors - (ms) (ms) (ms) (ms) - --------------- -------- ---------- ---------- ---------- ---------- -------- - epoll_wait 628 13175.485 21.259 20.980 0.020 - clock_gettime 7326 199.500 0.249 0.027 0.013 - stat 2101 110.768 19.056 0.053 0.017 ENOENT: 2076 - ... +./strace-parser sidekiq_trace.txt p 16815 + +PID 16815 + + 80 syscalls, active time: 2.364ms, user time: 7.669ms, total time: 12050.026ms + start time: 22:46:14.830267 end time: 22:46:26.880293 + + syscall count total (ms) max (ms) avg (ms) min (ms) errors + ----------------- -------- ---------- ---------- ---------- ---------- -------- + futex 5 10100.229 5400.106 2020.046 0.022 ETIMEDOUT: 2 + restart_syscall 1 1939.764 1939.764 1939.764 1939.764 ETIMEDOUT: 1 + getpid 33 1.020 0.046 0.031 0.018 + clock_gettime 14 0.420 0.038 0.030 0.021 + stat 6 0.277 0.072 0.046 0.031 + read 6 0.170 0.036 0.028 0.020 + openat 3 0.126 0.045 0.042 0.038 + close 3 0.099 0.034 0.033 0.031 + lseek 3 0.089 0.035 0.030 0.021 + ioctl 3 0.082 0.033 0.027 0.023 ENOTTY: 3 + fstat 3 0.081 0.034 0.027 0.022 --------------- - Parent PID: 495 - Child PIDs: 8383, 8418, 8419, 8420, 8421 + Slowest file open times for PID 16815: - Slowest file access times for PID 6423: - - open (ms) timestamp error file name - ----------- --------------- --------------- ---------- - 29.818 10:53:11.528954 /srv/gitlab-data/builds/2018_08/6174/954448.log - 12.309 10:53:46.708274 /srv/gitlab-data/builds/2018_08/5342/954186.log - 0.039 10:53:49.222110 /opt/gitlab/embedded/service/gitlab-rails/app/views/events/event/_note.html.haml - 0.035 10:53:49.125115 /opt/gitlab/embedded/service/gitlab-rails/app/views/events/event/_push.html.haml - ... + dur (ms) timestamp error file name + ---------- --------------- --------------- --------- + 0.045 22:46:16.771318 - /opt/gitlab/embedded/service/gitlab-rails/config/database.yml + 0.043 22:46:26.877954 - /opt/gitlab/embedded/service/gitlab-rails/config/database.yml + 0.038 22:46:22.174610 - /opt/gitlab/embedded/service/gitlab-rails/config/database.yml ``` -In the example above, we can see that file opening times on `/srv/gitlab-data` are -extremely slow, about 100X slower than `/opt/gitlab`. +In the example above, we can see which files took longer to open for `PID 16815`. When nothing stands out in the results, a good way to get more context is to run `strace` on your own GitLab instance while performing the action performed by the customer, diff --git a/doc/administration/troubleshooting/log_parsing.md b/doc/administration/troubleshooting/log_parsing.md index 0320b2e52ce..929a49494be 100644 --- a/doc/administration/troubleshooting/log_parsing.md +++ b/doc/administration/troubleshooting/log_parsing.md @@ -1,316 +1,11 @@ --- -stage: Systems -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../logs/log_parsing.md' +remove_date: '2022-10-24' --- -# Parsing GitLab logs with `jq` **(FREE SELF)** +This document was moved to [another location](../logs/log_parsing.md). -We recommend using log aggregation and search tools like Kibana and Splunk whenever possible, -but if they are not available you can still quickly parse -[GitLab logs](../logs.md) in JSON format -(the default in GitLab 12.0 and later) using [`jq`](https://stedolan.github.io/jq/). - -NOTE: -Specifically for summarizing error events and basic usage statistics, -the GitLab Support Team provides the specialised -[`fast-stats` tool](https://gitlab.com/gitlab-com/support/toolbox/fast-stats/#when-to-use-it). - -## What is JQ? - -As noted in its [manual](https://stedolan.github.io/jq/manual/), `jq` is a command-line JSON processor. The following examples -include use cases targeted for parsing GitLab log files. - -## Parsing Logs - -The examples listed below address their respective log files by -their relative Omnibus paths and default filenames. -Find the respective full paths in the [GitLab logs sections](../logs.md#production_jsonlog). - -### General Commands - -#### Pipe colorized `jq` output into `less` - -```shell -jq . <FILE> -C | less -R -``` - -#### Search for a term and pretty-print all matching lines - -```shell -grep <TERM> <FILE> | jq . -``` - -#### Skip invalid lines of JSON - -```shell -jq -cR 'fromjson?' file.json | jq <COMMAND> -``` - -By default `jq` errors out when it encounters a line that is not valid JSON. -This skips over all invalid lines and parses the rest. - -#### Print a JSON log's time range - -```shell -cat log.json | (head -1; tail -1) | jq '.time' -``` - -Use `zcat` if the file has been rotated and compressed: - -```shell -zcat @400000006026b71d1a7af804.s | (head -1; tail -1) | jq '.time' - -zcat some_json.log.25.gz | (head -1; tail -1) | jq '.time' -``` - -#### Get activity for correlation ID across multiple JSON logs in chronological order - -```shell -grep -hR <correlationID> | jq -c -R 'fromjson?' | jq -C -s 'sort_by(.time)' | less -R -``` - -### Parsing `gitlab-rails/production_json.log` and `gitlab-rails/api_json.log` - -#### Find all requests with a 5XX status code - -```shell -jq 'select(.status >= 500)' <FILE> -``` - -#### Top 10 slowest requests - -```shell -jq -s 'sort_by(-.duration_s) | limit(10; .[])' <FILE> -``` - -#### Find and pretty print all requests related to a project - -```shell -grep <PROJECT_NAME> <FILE> | jq . -``` - -#### Find all requests with a total duration > 5 seconds - -```shell -jq 'select(.duration_s > 5000)' <FILE> -``` - -#### Find all project requests with more than 5 rugged calls - -```shell -grep <PROJECT_NAME> <FILE> | jq 'select(.rugged_calls > 5)' -``` - -#### Find all requests with a Gitaly duration > 10 seconds - -```shell -jq 'select(.gitaly_duration_s > 10000)' <FILE> -``` - -#### Find all requests with a queue duration > 10 seconds - -```shell -jq 'select(.queue_duration_s > 10000)' <FILE> -``` - -#### Top 10 requests by # of Gitaly calls - -```shell -jq -s 'map(select(.gitaly_calls != null)) | sort_by(-.gitaly_calls) | limit(10; .[])' <FILE> -``` - -### Parsing `gitlab-rails/production_json.log` - -#### Print the top three controller methods by request volume and their three longest durations - -```shell -jq -s -r 'group_by(.controller+.action) | sort_by(-length) | limit(3; .[]) | sort_by(-.duration_s) | "CT: \(length)\tMETHOD: \(.[0].controller)#\(.[0].action)\tDURS: \(.[0].duration_s), \(.[1].duration_s), \(.[2].duration_s)"' production_json.log -``` - -**Example output** - -```plaintext -CT: 2721 METHOD: SessionsController#new DURS: 844.06, 713.81, 704.66 -CT: 2435 METHOD: MetricsController#index DURS: 299.29, 284.01, 158.57 -CT: 1328 METHOD: Projects::NotesController#index DURS: 403.99, 386.29, 384.39 -``` - -### Parsing `gitlab-rails/api_json.log` - -#### Print top three routes with request count and their three longest durations - -```shell -jq -s -r 'group_by(.route) | sort_by(-length) | limit(3; .[]) | sort_by(-.duration_s) | "CT: \(length)\tROUTE: \(.[0].route)\tDURS: \(.[0].duration_s), \(.[1].duration_s), \(.[2].duration_s)"' api_json.log -``` - -**Example output** - -```plaintext -CT: 2472 ROUTE: /api/:version/internal/allowed DURS: 56402.65, 38411.43, 19500.41 -CT: 297 ROUTE: /api/:version/projects/:id/repository/tags DURS: 731.39, 685.57, 480.86 -CT: 190 ROUTE: /api/:version/projects/:id/repository/commits DURS: 1079.02, 979.68, 958.21 -``` - -### Print top API user agents - -```shell -jq --raw-output '[.route, .ua] | @tsv' api_json.log | sort | uniq -c | sort -n -``` - -**Example output**: - -```plaintext - 89 /api/:version/usage_data/increment_unique_users # plus browser details - 567 /api/:version/jobs/:id/trace gitlab-runner # plus version details -1234 /api/:version/internal/allowed GitLab-Shell -``` - -This sample response seems normal. A custom tool or script might be causing a high load -if the output contains many: - -- Third party libraries like `python-requests` or `curl`. -- [GitLab CLI clients](https://about.gitlab.com/partners/technology-partners/#cli-clients). - -You can also [use `fast-stats top`](#parsing-gitlab-logs-with-jq) to extract performance statistics. - -### Parsing `gitlab-workhorse/current` - -### Print top Workhorse user agents - -```shell -jq --raw-output '[.uri, .user_agent] | @tsv' current | sort | uniq -c | sort -n -``` - -**Example output**: - -```plaintext - 89 /api/graphql # plus browser details - 567 /api/v4/internal/allowed GitLab-Shell -1234 /api/v4/jobs/request gitlab-runner # plus version details -``` - -Similar to the [API `ua` data](#print-top-api-user-agents), -deviations from this common order might indicate scripts that could be optimized. - -The performance impact of runners checking for new jobs can be reduced by increasing -[the `check_interval` setting](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section), -for example. - -### Parsing `gitlab-rails/geo.log` - -#### Find most common Geo sync errors - -If [the `geo:status` Rake task](../geo/replication/troubleshooting.md#sync-status-rake-task) -repeatedly reports that some items never reach 100%, -the following command helps to focus on the most common errors. - -```shell -jq --raw-output 'select(.severity == "ERROR") | [.project_path, .message] | @tsv' geo.log | sort | uniq -c | sort | tail -``` - -### Parsing `gitaly/current` - -Use the following examples to [troubleshoot Gitaly](../gitaly/troubleshooting.md). - -#### Find all Gitaly requests sent from web UI - -```shell -jq 'select(."grpc.meta.client_name" == "gitlab-web")' current -``` - -#### Find all failed Gitaly requests - -```shell -jq 'select(."grpc.code" != null and ."grpc.code" != "OK")' current -``` - -#### Find all requests that took longer than 30 seconds - -```shell -jq 'select(."grpc.time_ms" > 30000)' current -``` - -#### Print top ten projects by request volume and their three longest durations - -```shell -jq --raw-output --slurp ' - map( - select( - ."grpc.request.glProjectPath" != null - and ."grpc.request.glProjectPath" != "" - and ."grpc.time_ms" != null - ) - ) - | group_by(."grpc.request.glProjectPath") - | sort_by(-length) - | limit(10; .[]) - | sort_by(-."grpc.time_ms") - | [ - length, - .[0]."grpc.time_ms", - .[1]."grpc.time_ms", - .[2]."grpc.time_ms", - .[0]."grpc.request.glProjectPath" - ] - | @sh' current \ -| awk 'BEGIN { printf "%7s %10s %10s %10s\t%s\n", "CT", "MAX DURS", "", "", "PROJECT" } - { printf "%7u %7u ms, %7u ms, %7u ms\t%s\n", $1, $2, $3, $4, $5 }' -``` - -**Example output** - -```plaintext - CT MAX DURS PROJECT - 206 4898 ms, 1101 ms, 1032 ms 'groupD/project4' - 109 1420 ms, 962 ms, 875 ms 'groupEF/project56' - 663 106 ms, 96 ms, 94 ms 'groupABC/project123' - ... -``` - -#### Find all projects affected by a fatal Git problem - -```shell -grep "fatal: " current | \ - jq '."grpc.request.glProjectPath"' | \ - sort | uniq -``` - -### Parsing `gitlab-shell/gitlab-shell.log` - -For investigating Git calls via SSH, from [GitLab 12.10](https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/367). - -Find the top 20 calls by project and user: - -```shell -jq --raw-output --slurp ' - map( - select( - .username != null and - .gl_project_path !=null - ) - ) - | group_by(.username+.gl_project_path) - | sort_by(-length) - | limit(20; .[]) - | "count: \(length)\tuser: \(.[0].username)\tproject: \(.[0].gl_project_path)" ' \ - gitlab-shell.log -``` - -Find the top 20 calls by project, user, and command: - -```shell -jq --raw-output --slurp ' - map( - select( - .command != null and - .username != null and - .gl_project_path !=null - ) - ) - | group_by(.username+.gl_project_path+.command) - | sort_by(-length) - | limit(20; .[]) - | "count: \(length)\tcommand: \(.[0].command)\tuser: \(.[0].username)\tproject: \(.[0].gl_project_path)" ' \ - gitlab-shell.log -``` +<!-- This redirect file can be deleted after <2022-10-24>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/troubleshooting/postgresql.md b/doc/administration/troubleshooting/postgresql.md index 61b661d45f8..9e3d3d47a10 100644 --- a/doc/administration/troubleshooting/postgresql.md +++ b/doc/administration/troubleshooting/postgresql.md @@ -216,7 +216,7 @@ because the statement timeout was too short: pg_dump: error: Error message from server: server closed the connection unexpectedly ``` -You may also see errors in the [PostgreSQL logs](../logs.md#postgresql-logs): +You may also see errors in the [PostgreSQL logs](../logs/index.md#postgresql-logs): ```plaintext canceling statement due to statement timeout diff --git a/doc/administration/troubleshooting/sidekiq.md b/doc/administration/troubleshooting/sidekiq.md index 40bfe22ac63..e49e0ed4f1c 100644 --- a/doc/administration/troubleshooting/sidekiq.md +++ b/doc/administration/troubleshooting/sidekiq.md @@ -1,395 +1,11 @@ --- -stage: Systems -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../sidekiq/sidekiq_troubleshooting.md' +remove_date: '2022-11-11' --- -# Troubleshooting Sidekiq **(FREE SELF)** +This document was moved to [another location](../sidekiq/sidekiq_troubleshooting.md). -Sidekiq is the background job processor GitLab uses to asynchronously run -tasks. When things go wrong it can be difficult to troubleshoot. These -situations also tend to be high-pressure because a production system job queue -may be filling up. Users will notice when this happens because new branches -may not show up and merge requests may not be updated. The following are some -troubleshooting steps to help you diagnose the bottleneck. - -GitLab administrators/users should consider working through these -debug steps with GitLab Support so the backtraces can be analyzed by our team. -It may reveal a bug or necessary improvement in GitLab. - -In any of the backtraces, be wary of suspecting cases where every -thread appears to be waiting in the database, Redis, or waiting to acquire -a mutex. This **may** mean there's contention in the database, for example, -but look for one thread that is different than the rest. This other thread -may be using all available CPU, or have a Ruby Global Interpreter Lock, -preventing other threads from continuing. - -## Log arguments to Sidekiq jobs - -[In GitLab 13.6 and later](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/44853) -some arguments passed to Sidekiq jobs are logged by default. -To avoid logging sensitive information (for instance, password reset tokens), -GitLab logs numeric arguments for all workers, with overrides for some specific -workers where their arguments are not sensitive. - -Example log output: - -```json -{"severity":"INFO","time":"2020-06-08T14:37:37.892Z","class":"AdminEmailsWorker","args":["[FILTERED]","[FILTERED]","[FILTERED]"],"retry":3,"queue":"admin_emails","backtrace":true,"jid":"9e35e2674ac7b12d123e13cc","created_at":"2020-06-08T14:37:37.373Z","meta.user":"root","meta.caller_id":"Admin::EmailsController#create","correlation_id":"37D3lArJmT1","uber-trace-id":"2d942cc98cc1b561:6dc94409cfdd4d77:9fbe19bdee865293:1","enqueued_at":"2020-06-08T14:37:37.410Z","pid":65011,"message":"AdminEmailsWorker JID-9e35e2674ac7b12d123e13cc: done: 0.48085 sec","job_status":"done","scheduling_latency_s":0.001012,"redis_calls":9,"redis_duration_s":0.004608,"redis_read_bytes":696,"redis_write_bytes":6141,"duration_s":0.48085,"cpu_s":0.308849,"completed_at":"2020-06-08T14:37:37.892Z","db_duration_s":0.010742} -{"severity":"INFO","time":"2020-06-08T14:37:37.894Z","class":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper","wrapped":"ActionMailer::MailDeliveryJob","queue":"mailers","args":["[FILTERED]"],"retry":3,"backtrace":true,"jid":"e47a4f6793d475378432e3c8","created_at":"2020-06-08T14:37:37.884Z","meta.user":"root","meta.caller_id":"AdminEmailsWorker","correlation_id":"37D3lArJmT1","uber-trace-id":"2d942cc98cc1b561:29344de0f966446d:5c3b0e0e1bef987b:1","enqueued_at":"2020-06-08T14:37:37.885Z","pid":65011,"message":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper JID-e47a4f6793d475378432e3c8: start","job_status":"start","scheduling_latency_s":0.009473} -{"severity":"INFO","time":"2020-06-08T14:39:50.648Z","class":"NewIssueWorker","args":["455","1"],"retry":3,"queue":"new_issue","backtrace":true,"jid":"a24af71f96fd129ec47f5d1e","created_at":"2020-06-08T14:39:50.643Z","meta.user":"root","meta.project":"h5bp/html5-boilerplate","meta.root_namespace":"h5bp","meta.caller_id":"Projects::IssuesController#create","correlation_id":"f9UCZHqhuP7","uber-trace-id":"28f65730f99f55a3:a5d2b62dec38dffc:48ddd092707fa1b7:1","enqueued_at":"2020-06-08T14:39:50.646Z","pid":65011,"message":"NewIssueWorker JID-a24af71f96fd129ec47f5d1e: start","job_status":"start","scheduling_latency_s":0.001144} -``` - -When using [Sidekiq JSON logging](../logs.md#sidekiqlog), -arguments logs are limited to a maximum size of 10 kilobytes of text; -any arguments after this limit are discarded and replaced with a -single argument containing the string `"..."`. - -You can set `SIDEKIQ_LOG_ARGUMENTS` [environment variable](https://docs.gitlab.com/omnibus/settings/environment-variables.html) -to `0` (false) to disable argument logging. - -Example: - -```ruby -gitlab_rails['env'] = {"SIDEKIQ_LOG_ARGUMENTS" => "0"} -``` - -In GitLab 13.5 and earlier, set `SIDEKIQ_LOG_ARGUMENTS` to `1` to start logging arguments passed to Sidekiq. - -## Thread dump - -Send the Sidekiq process ID the `TTIN` signal to output thread -backtraces in the log file. - -```shell -kill -TTIN <sidekiq_pid> -``` - -Check in `/var/log/gitlab/sidekiq/current` or `$GITLAB_HOME/log/sidekiq.log` for -the backtrace output. The backtraces are lengthy and generally start with -several `WARN` level messages. Here's an example of a single thread's backtrace: - -```plaintext -2016-04-13T06:21:20.022Z 31517 TID-orn4urby0 WARN: ActiveRecord::RecordNotFound: Couldn't find Note with 'id'=3375386 -2016-04-13T06:21:20.022Z 31517 TID-orn4urby0 WARN: /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.2/lib/active_record/core.rb:155:in `find' -/opt/gitlab/embedded/service/gitlab-rails/app/workers/new_note_worker.rb:7:in `perform' -/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:150:in `execute_job' -/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:132:in `block (2 levels) in process' -/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:127:in `block in invoke' -/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/memory_killer.rb:17:in `call' -/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke' -/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/arguments_logger.rb:6:in `call' -... -``` - -In some cases Sidekiq may be hung and unable to respond to the `TTIN` signal. -Move on to other troubleshooting methods if this happens. - -## Ruby profiling with `rbspy` - -[rbspy](https://rbspy.github.io) is an easy to use and low-overhead Ruby profiler that can be used to create -flamegraph-style diagrams of CPU usage by Ruby processes. - -No changes to GitLab are required to use it and it has no dependencies. To install it: - -1. Download the binary from the [`rbspy` releases page](https://github.com/rbspy/rbspy/releases). -1. Make the binary executable. - -To profile a Sidekiq worker for one minute, run: - -```shell -sudo ./rbspy record --pid <sidekiq_pid> --duration 60 --file /tmp/sidekiq_profile.svg -``` - - - -In this example of a flamegraph generated by `rbspy`, almost all of the Sidekiq process's time is spent in `rev_parse`, a native C -function in Rugged. In the stack, we can see `rev_parse` is being called by the `ExpirePipelineCacheWorker`. - -## Process profiling with `perf` - -Linux has a process profiling tool called `perf` that is helpful when a certain -process is eating up a lot of CPU. If you see high CPU usage and Sidekiq isn't -responding to the `TTIN` signal, this is a good next step. - -If `perf` is not installed on your system, install it with `apt-get` or `yum`: - -```shell -# Debian -sudo apt-get install linux-tools - -# Ubuntu (may require these additional Kernel packages) -sudo apt-get install linux-tools-common linux-tools-generic linux-tools-`uname -r` - -# Red Hat/CentOS -sudo yum install perf -``` - -Run `perf` against the Sidekiq PID: - -```shell -sudo perf record -p <sidekiq_pid> -``` - -Let this run for 30-60 seconds and then press Ctrl-C. Then view the `perf` report: - -```shell -$ sudo perf report - -# Sample output -Samples: 348K of event 'cycles', Event count (approx.): 280908431073 - 97.69% ruby nokogiri.so [.] xmlXPathNodeSetMergeAndClear - 0.18% ruby libruby.so.2.1.0 [.] objspace_malloc_increase - 0.12% ruby libc-2.12.so [.] _int_malloc - 0.10% ruby libc-2.12.so [.] _int_free -``` - -Above you see sample output from a `perf` report. It shows that 97% of the CPU is -being spent inside Nokogiri and `xmlXPathNodeSetMergeAndClear`. For something -this obvious you should then go investigate what job in GitLab would use -Nokogiri and XPath. Combine with `TTIN` or `gdb` output to show the -corresponding Ruby code where this is happening. - -## The GNU Project Debugger (`gdb`) - -`gdb` can be another effective tool for debugging Sidekiq. It gives you a little -more interactive way to look at each thread and see what's causing problems. - -Attaching to a process with `gdb` suspends the normal operation -of the process (Sidekiq does not process jobs while `gdb` is attached). - -Start by attaching to the Sidekiq PID: - -```shell -gdb -p <sidekiq_pid> -``` - -Then gather information on all the threads: - -```plaintext -info threads - -# Example output -30 Thread 0x7fe5fbd63700 (LWP 26060) 0x0000003f7cadf113 in poll () from /lib64/libc.so.6 -29 Thread 0x7fe5f2b3b700 (LWP 26533) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 -28 Thread 0x7fe5f2a3a700 (LWP 26534) 0x0000003f7ce0ba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 -27 Thread 0x7fe5f2939700 (LWP 26535) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 -26 Thread 0x7fe5f2838700 (LWP 26537) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 -25 Thread 0x7fe5f2737700 (LWP 26538) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 -24 Thread 0x7fe5f2535700 (LWP 26540) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 -23 Thread 0x7fe5f2434700 (LWP 26541) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 -22 Thread 0x7fe5f2232700 (LWP 26543) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 -21 Thread 0x7fe5f2131700 (LWP 26544) 0x00007fe5f7b570f0 in xmlXPathNodeSetMergeAndClear () -from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so -... -``` - -If you see a suspicious thread, like the Nokogiri one above, you may want -to get more information: - -```plaintext -thread 21 -bt - -# Example output -#0 0x00007ff0d6afe111 in xmlXPathNodeSetMergeAndClear () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so -#1 0x00007ff0d6b0b836 in xmlXPathNodeCollectAndTest () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so -#2 0x00007ff0d6b09037 in xmlXPathCompOpEval () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so -#3 0x00007ff0d6b09017 in xmlXPathCompOpEval () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so -#4 0x00007ff0d6b092e0 in xmlXPathCompOpEval () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so -#5 0x00007ff0d6b0bc37 in xmlXPathRunEval () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so -#6 0x00007ff0d6b0be5f in xmlXPathEvalExpression () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so -#7 0x00007ff0d6a97dc3 in evaluate (argc=2, argv=0x1022d058, self=<value optimized out>) at xml_xpath_context.c:221 -#8 0x00007ff0daeab0ea in vm_call_cfunc_with_frame (th=0x1022a4f0, reg_cfp=0x1032b810, ci=<value optimized out>) at vm_insnhelper.c:1510 -``` - -To output a backtrace from all threads at once: - -```plaintext -set pagination off -thread apply all bt -``` - -Once you're done debugging with `gdb`, be sure to detach from the process and -exit: - -```plaintext -detach -exit -``` - -## Sidekiq kill signals - -TTIN was described above as the signal to print backtraces for logging, however -Sidekiq responds to other signals as well. For example, TSTP and TERM can be used -to gracefully shut Sidekiq down, see -[the Sidekiq Signals docs](https://github.com/mperham/sidekiq/wiki/Signals#ttin). - -## Check for blocking queries - -Sometimes the speed at which Sidekiq processes jobs can be so fast that it can -cause database contention. Check for blocking queries when backtraces above -show that many threads are stuck in the database adapter. - -The PostgreSQL wiki has details on the query you can run to see blocking -queries. The query is different based on PostgreSQL version. See -[Lock Monitoring](https://wiki.postgresql.org/wiki/Lock_Monitoring) for -the query details. - -## Managing Sidekiq queues - -It is possible to use [Sidekiq API](https://github.com/mperham/sidekiq/wiki/API) -to perform a number of troubleshooting steps on Sidekiq. - -These are the administrative commands and it should only be used if currently -administration interface is not suitable due to scale of installation. - -All these commands should be run using `gitlab-rails console`. - -### View the queue size - -```ruby -Sidekiq::Queue.new("pipeline_processing:build_queue").size -``` - -### Enumerate all enqueued jobs - -```ruby -queue = Sidekiq::Queue.new("chaos:chaos_sleep") -queue.each do |job| - # job.klass # => 'MyWorker' - # job.args # => [1, 2, 3] - # job.jid # => jid - # job.queue # => chaos:chaos_sleep - # job["retry"] # => 3 - # job.item # => { - # "class"=>"Chaos::SleepWorker", - # "args"=>[1000], - # "retry"=>3, - # "queue"=>"chaos:chaos_sleep", - # "backtrace"=>true, - # "queue_namespace"=>"chaos", - # "jid"=>"39bc482b823cceaf07213523", - # "created_at"=>1566317076.266069, - # "correlation_id"=>"c323b832-a857-4858-b695-672de6f0e1af", - # "enqueued_at"=>1566317076.26761}, - # } - - # job.delete if job.jid == 'abcdef1234567890' -end -``` - -### Enumerate currently running jobs - -```ruby -workers = Sidekiq::Workers.new -workers.each do |process_id, thread_id, work| - # process_id is a unique identifier per Sidekiq process - # thread_id is a unique identifier per thread - # work is a Hash which looks like: - # {"queue"=>"chaos:chaos_sleep", - # "payload"=> - # { "class"=>"Chaos::SleepWorker", - # "args"=>[1000], - # "retry"=>3, - # "queue"=>"chaos:chaos_sleep", - # "backtrace"=>true, - # "queue_namespace"=>"chaos", - # "jid"=>"b2a31e3eac7b1a99ff235869", - # "created_at"=>1566316974.9215662, - # "correlation_id"=>"e484fb26-7576-45f9-bf21-b99389e1c53c", - # "enqueued_at"=>1566316974.9229589}, - # "run_at"=>1566316974}], -end -``` - -### Remove Sidekiq jobs for given parameters (destructive) - -The general method to kill jobs conditionally is the following command, which -removes jobs that are queued but not started. Running jobs can not be killed. - -```ruby -queue = Sidekiq::Queue.new('<queue name>') -queue.each { |job| job.delete if <condition>} -``` - -Have a look at the section below for cancelling running jobs. - -In the method above, `<queue-name>` is the name of the queue that contains the jobs you want to delete and `<condition>` decides which jobs get deleted. - -Commonly, `<condition>` references the job arguments, which depend on the type of job in question. To find the arguments for a specific queue, you can have a look at the `perform` function of the related worker file, commonly found at `/app/workers/<queue-name>_worker.rb`. - -For example, `repository_import` has `project_id` as the job argument, while `update_merge_requests` has `project_id, user_id, oldrev, newrev, ref`. - -Arguments need to be referenced by their sequence ID using `job.args[<id>]` because `job.args` is a list of all arguments provided to the Sidekiq job. - -Here are some examples: - -```ruby -queue = Sidekiq::Queue.new('update_merge_requests') -# In this example, we want to remove any update_merge_requests jobs -# for the Project with ID 125 and ref `ref/heads/my_branch` -queue.each { |job| job.delete if job.args[0] == 125 and job.args[4] == 'ref/heads/my_branch' } -``` - -```ruby -# Cancelling jobs like: `RepositoryImportWorker.new.perform_async(100)` -id_list = [100] - -queue = Sidekiq::Queue.new('repository_import') -queue.each do |job| - job.delete if id_list.include?(job.args[0]) -end -``` - -### Remove specific job ID (destructive) - -```ruby -queue = Sidekiq::Queue.new('repository_import') -queue.each do |job| - job.delete if job.jid == 'my-job-id' -end -``` - -## Canceling running jobs (destructive) - -> Introduced in GitLab 12.3. - -This is highly risky operation and use it as last resort. -Doing that might result in data corruption, as the job -is interrupted mid-execution and it is not guaranteed -that proper rollback of transactions is implemented. - -```ruby -Gitlab::SidekiqDaemon::Monitor.cancel_job('job-id') -``` - -> This requires the Sidekiq to be run with `SIDEKIQ_MONITOR_WORKER=1` -> environment variable. - -To perform of the interrupt we use `Thread.raise` which -has number of drawbacks, as mentioned in [Why Ruby's Timeout is dangerous (and Thread.raise is terrifying)](https://jvns.ca/blog/2015/11/27/why-rubys-timeout-is-dangerous-and-thread-dot-raise-is-terrifying/): - -> This is where the implications get interesting, and terrifying. This means that an exception can get raised: -> -> - during a network request (ok, as long as the surrounding code is prepared to catch Timeout::Error) -> - during the cleanup for the network request -> - during a rescue block -> - while creating an object to save to the database afterwards -> - in any of your code, regardless of whether it could have possibly raised an exception before -> -> Nobody writes code to defend against an exception being raised on literally any line. That's not even possible. So Thread.raise is basically like a sneak attack on your code that could result in almost anything. It would probably be okay if it were pure-functional code that did not modify any state. But this is Ruby, so that's unlikely :) - -## Disable Rugged - -Calls into Rugged, Ruby bindings for `libgit2`, [lock the Sidekiq processes's GVL](https://silverhammermba.github.io/emberb/c/#c-in-ruby-threads), -blocking all jobs on that worker from proceeding. If Rugged calls performed by Sidekiq are slow, this can cause significant delays in -background task processing. - -By default, Rugged is used when Git repository data is stored on local storage or on an NFS mount. -[Using Rugged is recommended when using NFS](../nfs.md#improving-nfs-performance-with-gitlab), but if -you are using local storage, disabling Rugged can improve Sidekiq performance: - -```shell -sudo gitlab-rake gitlab:features:disable_rugged -``` +<!-- This redirect file can be deleted after <2022-11-11>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/troubleshooting/tracing_correlation_id.md b/doc/administration/troubleshooting/tracing_correlation_id.md index 418dd729066..ee59b7c2504 100644 --- a/doc/administration/troubleshooting/tracing_correlation_id.md +++ b/doc/administration/troubleshooting/tracing_correlation_id.md @@ -1,202 +1,11 @@ --- -stage: Systems -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../logs/tracing_correlation_id.md' +remove_date: '2022-11-12' --- -# Finding relevant log entries with a correlation ID **(FREE SELF)** +This document was moved to [another location](../logs/tracing_correlation_id.md). -GitLab instances log a unique request tracking ID (known as the -"correlation ID") for most requests. Each individual request to GitLab gets -its own correlation ID, which then gets logged in each GitLab component's logs for that -request. This makes it easier to trace behavior in a -distributed system. Without this ID it can be difficult or -impossible to match correlating log entries. - -## Identify the correlation ID for a request - -The correlation ID is logged in structured logs under the key `correlation_id` -and in all response headers GitLab sends under the header `x-request-id`. -You can find your correlation ID by searching in either place. - -### Getting the correlation ID in your browser - -You can use your browser's developer tools to monitor and inspect network -activity with the site that you're visiting. See the links below for network monitoring -documentation for some popular browsers. - -- [Network Monitor - Firefox Developer Tools](https://developer.mozilla.org/en-US/docs/Tools/Network_Monitor) -- [Inspect Network Activity In Chrome DevTools](https://developer.chrome.com/docs/devtools/network/) -- [Safari Web Development Tools](https://developer.apple.com/safari/tools/) -- [Microsoft Edge Network panel](https://docs.microsoft.com/en-us/microsoft-edge/devtools-guide-chromium/network/) - -To locate a relevant request and view its correlation ID: - -1. Enable persistent logging in your network monitor. Some actions in GitLab redirect you quickly after you submit a form, so this helps capture all relevant activity. -1. To help isolate the requests you are looking for, you can filter for `document` requests. -1. Select the request of interest to view further detail. -1. Go to the **Headers** section and look for **Response Headers**. There you should find an `x-request-id` header with a -value that was randomly generated by GitLab for the request. - -See the following example: - - - -### Getting the correlation ID from your logs - -Another approach to finding the correct correlation ID is to search or watch -your logs and find the `correlation_id` value for the log entry that you're -watching for. - -For example, let's say that you want learn what's happening or breaking when -you reproduce an action in GitLab. You could tail the GitLab logs, filtering -to requests by your user, and then watch the requests until you see what you're -interested in. - -### Getting the correlation ID from curl - -If you're using `curl` then you can use the verbose option to show request and response headers, as well as other debug information. - -```shell -➜ ~ curl --verbose "https://gitlab.example.com/api/v4/projects" -# look for a line that looks like this -< x-request-id: 4rAMkV3gof4 -``` - -#### Using jq - -This example uses [jq](https://stedolan.github.io/jq/) to filter results and -display values we most likely care about. - -```shell -sudo gitlab-ctl tail gitlab-rails/production_json.log | jq 'select(.username == "bob") | "User: \(.username), \(.method) \(.path), \(.controller)#\(.action), ID: \(.correlation_id)"' -``` - -```plaintext -"User: bob, GET /root/linux, ProjectsController#show, ID: U7k7fh6NpW3" -"User: bob, GET /root/linux/commits/master/signatures, Projects::CommitsController#signatures, ID: XPIHpctzEg1" -"User: bob, GET /root/linux/blob/master/README, Projects::BlobController#show, ID: LOt9hgi1TV4" -``` - -#### Using grep - -This example uses only `grep` and `tr`, which are more likely to be installed than `jq`. - -```shell -sudo gitlab-ctl tail gitlab-rails/production_json.log | grep '"username":"bob"' | tr ',' '\n' | egrep 'method|path|correlation_id' -``` - -```plaintext -{"method":"GET" -"path":"/root/linux" -"username":"bob" -"correlation_id":"U7k7fh6NpW3"} -{"method":"GET" -"path":"/root/linux/commits/master/signatures" -"username":"bob" -"correlation_id":"XPIHpctzEg1"} -{"method":"GET" -"path":"/root/linux/blob/master/README" -"username":"bob" -"correlation_id":"LOt9hgi1TV4"} -``` - -## Searching your logs for the correlation ID - -Once you have the correlation ID you can start searching for relevant log -entries. You can filter the lines by the correlation ID itself. -Combining a `find` and `grep` should be sufficient to find the entries you are looking for. - -```shell -# find <gitlab log directory> -type f -mtime -0 exec grep '<correlation ID>' '{}' '+' -find /var/log/gitlab -type f -mtime 0 -exec grep 'LOt9hgi1TV4' '{}' '+' -``` - -```plaintext -/var/log/gitlab/gitlab-workhorse/current:{"correlation_id":"LOt9hgi1TV4","duration_ms":2478,"host":"gitlab.domain.tld","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://gitlab.domain.tld/root/linux","remote_addr":"68.0.116.160:0","remote_ip":"[filtered]","status":200,"system":"http","time":"2019-09-17T22:17:19Z","uri":"/root/linux/blob/master/README?format=json\u0026viewer=rich","user_agent":"Mozilla/5.0 (Mac) Gecko Firefox/69.0","written_bytes":1743} -/var/log/gitlab/gitaly/current:{"correlation_id":"LOt9hgi1TV4","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"FindCommits","grpc.request.deadline":"2019-09-17T22:17:47Z","grpc.request.fullMethod":"/gitaly.CommitService/FindCommits","grpc.request.glProjectPath":"root/linux","grpc.request.glRepository":"project-1","grpc.request.repoPath":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"@hashed","grpc.service":"gitaly.CommitService","grpc.start_time":"2019-09-17T22:17:17Z","grpc.time_ms":2319.161,"level":"info","msg":"finished streaming call with code OK","peer.address":"@","span.kind":"server","system":"grpc","time":"2019-09-17T22:17:19Z"} -/var/log/gitlab/gitlab-rails/production_json.log:{"method":"GET","path":"/root/linux/blob/master/README","format":"json","controller":"Projects::BlobController","action":"show","status":200,"duration":2448.77,"view":0.49,"db":21.63,"time":"2019-09-17T22:17:19.800Z","params":[{"key":"viewer","value":"rich"},{"key":"namespace_id","value":"root"},{"key":"project_id","value":"linux"},{"key":"id","value":"master/README"}],"remote_ip":"[filtered]","user_id":2,"username":"bob","ua":"Mozilla/5.0 (Mac) Gecko Firefox/69.0","queue_duration":3.38,"gitaly_calls":1,"gitaly_duration":0.77,"rugged_calls":4,"rugged_duration_ms":28.74,"correlation_id":"LOt9hgi1TV4"} -``` - -### Searching in distributed architectures - -If you have done some horizontal scaling in your GitLab infrastructure, then -you must search across _all_ of your GitLab nodes. You can do this with -some sort of log aggregation software like Loki, ELK, Splunk, or others. - -You can use a tool like Ansible or PSSH (parallel SSH) that can execute identical commands across your servers in -parallel, or craft your own solution. - -### Viewing the request in the Performance Bar - -You can use the [performance bar](../monitoring/performance/performance_bar.md) to view interesting data including calls made to SQL and Gitaly. - -To view the data, the correlation ID of the request must match the same session as the user -viewing the performance bar. For API requests, this means that you must perform the request -using the session cookie of the signed-in user. - -For example, if you want to view the database queries executed for the following API endpoint: - -```shell -https://gitlab.com/api/v4/groups/2564205/projects?with_security_reports=true&page=1&per_page=1 -``` - -First, enable the **Developer Tools** panel. See [Getting the correlation ID in your browser](#getting-the-correlation-id-in-your-browser) for details on how to do this. - -After developer tools have been enabled, obtain a session cookie as follows: - -1. Visit <https://gitlab.com> while logged in. -1. Optional. Select **Fetch/XHR** request filter in the **Developer Tools** panel. This step is described for Google Chrome developer tools and is not strictly necessary, it just makes it easier to find the correct request. -1. Select the `results?request_id=<some-request-id>` request on the left hand side. -1. The session cookie is displayed under the `Request Headers` section of the `Headers` panel. Right-click on the cookie value and select `Copy value`. - - - -You have the value of the session cookie copied to your clipboard, for example: - -```shell -experimentation_subject_id=<subject-id>; _gitlab_session=<session-id>; event_filter=all; visitor_id=<visitor-id>; perf_bar_enabled=true; sidebar_collapsed=true; diff_view=inline; sast_entry_point_dismissed=true; auto_devops_settings_dismissed=true; cf_clearance=<cf-clearance>; collapsed_gutter=false; frequently_used_emojis=clap,thumbsup,rofl,tada,eyes,bow -``` - -Use the value of the session cookie to craft an API request by pasting it into a custom header of a `curl` request: - -```shell -$ curl --include "https://gitlab.com/api/v4/groups/2564205/projects?with_security_reports=true&page=1&per_page=1" \ ---header 'cookie: experimentation_subject_id=<subject-id>; _gitlab_session=<session-id>; event_filter=all; visitor_id=<visitor-id>; perf_bar_enabled=true; sidebar_collapsed=true; diff_view=inline; sast_entry_point_dismissed=true; auto_devops_settings_dismissed=true; cf_clearance=<cf-clearance>; collapsed_gutter=false; frequently_used_emojis=clap,thumbsup,rofl,tada,eyes,bow' - - date: Tue, 28 Sep 2021 03:55:33 GMT - content-type: application/json - ... - x-request-id: 01FGN8P881GF2E5J91JYA338Y3 - ... - [ - { - "id":27497069, - "description":"Analyzer for images used on live K8S containers based on Starboard" - }, - "container_registry_image_prefix":"registry.gitlab.com/gitlab-org/security-products/analyzers/cluster-image-scanning", - "..." - ] -``` - -The response contains the data from the API endpoint, and a `correlation_id` value, returned in the `x-request-id` header, as described in the [Identify the correlation ID for a request](#identify-the-correlation-id-for-a-request) section. - -You can then view the database details for this request: - -1. Paste the `x-request-id` value into the `request details` field of the [performance bar](../monitoring/performance/performance_bar.md) and press <kbd>Enter/Return</kbd>. This example uses the `x-request-id` value `01FGN8P881GF2E5J91JYA338Y3`, returned by the above response: - -  - -1. A new request is inserted into the `Request Selector` dropdown on the right-hand side of the Performance Bar. Select the new request to view the metrics of the API request: - -  - - <!-- vale gitlab.Substitutions = NO --> -1. Select the `pg` link in the Progress Bar to view the database queries executed by the API request: - -  - <!-- vale gitlab.Substitutions = YES --> - - The database query dialog is displayed: - -  +<!-- This redirect file can be deleted after 2022-11-12. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
\ No newline at end of file |
