diff options
Diffstat (limited to 'doc/administration/troubleshooting')
10 files changed, 63 insertions, 32 deletions
diff --git a/doc/administration/troubleshooting/debug.md b/doc/administration/troubleshooting/debug.md index 8c8fa25aa5e..a6073e34d58 100644 --- a/doc/administration/troubleshooting/debug.md +++ b/doc/administration/troubleshooting/debug.md @@ -126,7 +126,7 @@ an SMTP server, but you're not seeing mail delivered. Here's how to check the se For more advanced issues, `gdb` is a must-have tool for debugging issues. -### The GNU Project Debugger (gdb) +### The GNU Project Debugger (GDB) To install on Ubuntu/Debian: @@ -140,9 +140,13 @@ On CentOS: sudo yum install gdb ``` +<!-- vale gitlab.Spelling = NO --> + ### rbtrace -GitLab 11.2 ships with [rbtrace](https://github.com/tmm1/rbtrace), which +<!-- vale gitlab.Spelling = YES --> + +GitLab 11.2 ships with [`rbtrace`](https://github.com/tmm1/rbtrace), which allows you to trace Ruby code, view all running threads, take memory dumps, and more. However, this is not enabled by default. To enable it, define the `ENABLE_RBTRACE` variable to the environment. For example, in Omnibus: @@ -175,7 +179,7 @@ downtime. Otherwise skip to the next section. 1. Load the problematic URL 1. Run `sudo gdb -p <PID>` to attach to the Unicorn process. -1. In the gdb window, type: +1. In the GDB window, type: ```plaintext call (void) rb_backtrace() @@ -210,7 +214,7 @@ downtime. Otherwise skip to the next section. ``` Note that if the Unicorn process terminates before you are able to run these -commands, gdb will report an error. To buy more time, you can always raise the +commands, GDB will report an error. To buy more time, you can always raise the Unicorn timeout. For omnibus users, you can edit `/etc/gitlab/gitlab.rb` and increase it from 60 seconds to 300: @@ -231,7 +235,7 @@ separate Rails process to debug the issue: 1. Log in to your GitLab account. 1. Copy the URL that is causing problems (e.g. `https://gitlab.com/ABC`). -1. Create a Personal Access Token for your user (Profile Settings -> Access Tokens). +1. Create a Personal Access Token for your user (User Settings -> Access Tokens). 1. Bring up the [GitLab Rails console.](../operations/rails_console.md#starting-a-rails-console-session) 1. At the Rails console, run: @@ -246,7 +250,7 @@ separate Rails process to debug the issue: ``` 1. In a new window, run `top`. It should show this Ruby process using 100% CPU. Write down the PID. -1. Follow step 2 from the previous section on using gdb. +1. Follow step 2 from the previous section on using GDB. ### GitLab: API is not accessible @@ -279,4 +283,4 @@ The output in `/tmp/unicorn.txt` may help diagnose the root cause. ## More information - [Debugging Stuck Ruby Processes](https://blog.newrelic.com/engineering/debugging-stuck-ruby-processes-what-to-do-before-you-kill-9/) -- [Cheatsheet of using gdb and Ruby processes](gdb-stuck-ruby.txt) +- [Cheat sheet of using GDB and Ruby processes](gdb-stuck-ruby.txt) diff --git a/doc/administration/troubleshooting/elasticsearch.md b/doc/administration/troubleshooting/elasticsearch.md index fb153adfeab..0f60c43ef9e 100644 --- a/doc/administration/troubleshooting/elasticsearch.md +++ b/doc/administration/troubleshooting/elasticsearch.md @@ -232,7 +232,7 @@ If the results: ### Troubleshooting indexing Troubleshooting indexing issues can be tricky. It can pretty quickly go to either GitLab -support or your Elasticsearch admin. +support or your Elasticsearch administrator. The best place to start is to determine if the issue is with creating an empty index. If it is, check on the Elasticsearch side to determine if the `gitlab-production` (the @@ -245,7 +245,7 @@ If you still encounter issues, try creating an index manually on the Elasticsear instance. The details of the index aren't important here, as we want to test if indices can be made. If the indices: -- Cannot be made, speak with your Elasticsearch admin. +- Cannot be made, speak with your Elasticsearch administrator. - Can be made, Escalate this to GitLab support. If the issue is not with creating an empty index, the next step is to check for errors @@ -253,7 +253,7 @@ during the indexing of projects. If errors do occur, they will either stem from - On the GitLab side. You need to rectify those. If they are not something you are familiar with, contact GitLab support for guidance. -- Within the Elasticsearch instance itself. See if the error is [documented and has a fix](../../integration/elasticsearch.md#troubleshooting). If not, speak with your Elasticsearch admin. +- Within the Elasticsearch instance itself. See if the error is [documented and has a fix](../../integration/elasticsearch.md#troubleshooting). If not, speak with your Elasticsearch administrator. If the indexing process does not present errors, you will want to check the status of the indexed projects. You can do this via the following Rake tasks: @@ -271,7 +271,7 @@ If reindexing the project shows: - Errors on the GitLab side, escalate those to GitLab support. - Elasticsearch errors or doesn't present any errors at all, reach out to your - Elasticsearch admin to check the instance. + Elasticsearch administrator to check the instance. ### Troubleshooting integration @@ -284,7 +284,7 @@ If the issue is: This is a required package so make sure you install it. Go indexer was a beta indexer which can be optionally turned on/off, but in 12.3 it reached stable status and is now the default. - Not concerning the Go indexer, it is almost always an - Elasticsearch-side issue. This means you should reach out to your Elasticsearch admin + Elasticsearch-side issue. This means you should reach out to your Elasticsearch administrator regarding the error(s) you are seeing. If you are unsure here, it never hurts to reach out to GitLab support. @@ -354,12 +354,12 @@ learn them, so it is best to escalate/pair with an Elasticsearch expert if you n dig further into these. Feel free to reach out to GitLab support, but this is likely to be something a skilled -Elasticsearch admin has more experience with. +Elasticsearch administrator has more experience with. ### Troubleshooting background migrations Troubleshooting background migration failures can be difficult and may require contacting -an Elasticsearch admin or GitLab Support. +an Elasticsearch administrator or GitLab Support. The best place to start while debugging issues with a background migration is the [`elasticsearch.log` file](../logs.md#elasticsearchlog). Migrations will diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 4a112733bfa..55e707042ba 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -5,13 +5,14 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# GitLab Rails Console Cheat Sheet **(CORE ONLY)** +# GitLab Rails Console Cheat Sheet **(FREE SELF)** This is the GitLab Support Team's collection of information regarding the GitLab Rails console, for use while troubleshooting. It is listed here for transparency, and it may be useful for users with experience with these tools. If you are currently -having an issue with GitLab, it is highly recommended that you check your -[support options](https://about.gitlab.com/support/) first, before attempting to use +having an issue with GitLab, it is highly recommended that you first check +our guide on [navigating our Rails console](navigating_gitlab_via_rails_console.md), +and your [support options](https://about.gitlab.com/support/), before attempting to use this information. WARNING: @@ -562,7 +563,7 @@ service = ::Groups::TransferService.new(group, user) service.execute(parent_group) ``` -### Count unique users in a group and sub-groups +### Count unique users in a group and subgroups ```ruby group = Group.find_by_path_or_name("groupname") diff --git a/doc/administration/troubleshooting/group_saml_scim.md b/doc/administration/troubleshooting/group_saml_scim.md index e9dbbfbde12..f0d44a578ff 100644 --- a/doc/administration/troubleshooting/group_saml_scim.md +++ b/doc/administration/troubleshooting/group_saml_scim.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Troubleshooting Group SAML and SCIM **(SILVER ONLY)** +# Troubleshooting Group SAML and SCIM **(PREMIUM SAAS)** These are notes and screenshots regarding Group SAML and SCIM that the GitLab Support Team sometimes uses while troubleshooting, but which do not fit into the official documentation. GitLab is making this public, so that anyone can make use of the Support team’s collected knowledge. diff --git a/doc/administration/troubleshooting/index.md b/doc/administration/troubleshooting/index.md index 67115ce31c0..63b056df8b7 100644 --- a/doc/administration/troubleshooting/index.md +++ b/doc/administration/troubleshooting/index.md @@ -13,7 +13,7 @@ in case something goes wrong: - [Diagnostics tools](diagnostics_tools.md) - [Elasticsearch](elasticsearch.md) - [GitLab Rails console cheat sheet](gitlab_rails_cheat_sheet.md) -- [Group SAML and SCIM troubleshooting](group_saml_scim.md) **(SILVER ONLY)** +- [Group SAML and SCIM troubleshooting](group_saml_scim.md) **(PREMIUM SAAS)** - [Kubernetes cheat sheet](kubernetes_cheat_sheet.md) - [Linux cheat sheet](linux_cheat_sheet.md) - [Parsing GitLab logs with `jq`](log_parsing.md) diff --git a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md index 07a7baf338b..9766b2210ca 100644 --- a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md +++ b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md @@ -70,7 +70,7 @@ and they will assist you with any issues you are having. kubectl logs <pod-name> --previous ``` - No logs are kept in the containers/pods themselves. Everything is written to stdout. + No logs are kept in the containers/pods themselves. Everything is written to `stdout`. This is the principle of Kubernetes, read [Twelve-factor app](https://12factor.net/) for details. @@ -88,7 +88,7 @@ and they will assist you with any issues you are having. - Minimal configuration that can be used to [test a Kubernetes Helm chart](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/620). -- Tailing logs of a separate pod. An example for a Webservice pod: +- Tailing logs of a separate pod. An example for a `webservice` pod: ```shell kubectl logs gitlab-webservice-54fbf6698b-hpckq -c webservice @@ -154,7 +154,7 @@ and they will assist you with any issues you are having. - On the side of GitLab check Sidekiq log and Kubernetes log. When GitLab is installed via Helm Chart, `kubernetes.log` can be found inside the Sidekiq pod. -- How to get your initial admin password <https://docs.gitlab.com/charts/installation/deployment.html#initial-login>: +- How to get your initial administrator password <https://docs.gitlab.com/charts/installation/deployment.html#initial-login>: ```shell # find the name of the secret containing the password diff --git a/doc/administration/troubleshooting/linux_cheat_sheet.md b/doc/administration/troubleshooting/linux_cheat_sheet.md index c61a78624c3..c4e991ccc1b 100644 --- a/doc/administration/troubleshooting/linux_cheat_sheet.md +++ b/doc/administration/troubleshooting/linux_cheat_sheet.md @@ -55,7 +55,7 @@ chown root:git <file_or_dir> chmod u+x <file> ``` -### Files & Dirs +### Files and directories ```shell # create a new directory and all subdirectories @@ -202,7 +202,7 @@ or you can build it from source if you have the Rust compiler. First run the tool with no arguments other than the strace output filename to get a summary of the top processes sorted by time spent actively performing tasks. You -can also sort based on total time, # of syscalls made, PID #, and # of child processes +can also sort based on total time, # of system calls made, PID #, and # of child processes using the `-S` or `--sort` flag. The number of results defaults to 25 processes, but can be changed using the `-c`/`--count` option. See `--help` for full details. @@ -220,7 +220,7 @@ Top 25 PIDs ... ``` -Based on the summary, you can then view the details of syscalls made by one or more +Based on the summary, you can then view the details of system calls made by one or more processes using the `-p`/`--pid` for a specific process, or `-s`/`--stats` flags for a sorted list. `--stats` takes the same sorting and count options as summary. @@ -266,7 +266,7 @@ Rough numbers for calls to `open` and `openat` (used to access files) on various Slow storage can cause the dreaded `DeadlineExceeded` error in Gitaly. Also [see this entry](../operations/filesystem_benchmarking.md) -in the handbook for quick tests customers can perform to check their filesystem performance. +in the handbook for quick tests customers can perform to check their file system performance. Keep in mind that timing information from `strace` is often somewhat inaccurate, so small differences should not be considered significant. @@ -304,6 +304,24 @@ whois <ip_address> | grep -i "orgname\|netname" # Curl headers with redirect curl --head --location "https://example.com" + +# Test if a host is reachable on the network. `ping6` works on IPv6 networks. +ping example.com + +# Show the route taken to a host. `traceroute6` works on IPv6 networks. +traceroute example.com +mtr example.com + +# List details of network interfaces +ip address + +# Check local DNS settings +cat /etc/hosts +cat /etc/resolv.conf +systemd-resolve --status + +# Capture traffic to/from a host +sudo tcpdump host www.example.com ``` ## Package Management diff --git a/doc/administration/troubleshooting/log_parsing.md b/doc/administration/troubleshooting/log_parsing.md index 144aa0f6d3b..25300d036ed 100644 --- a/doc/administration/troubleshooting/log_parsing.md +++ b/doc/administration/troubleshooting/log_parsing.md @@ -179,3 +179,11 @@ jq --raw-output --slurp ' 663 106 ms, 96 ms, 94 ms 'groupABC/project123' ... ``` + +#### Find all projects affected by a fatal Git problem + +```shell +grep "fatal: " /var/log/gitlab/gitaly/current | \ + jq '."grpc.request.glProjectPath"' | \ + sort | uniq +``` diff --git a/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md b/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md index 68c12117222..481c95b925a 100644 --- a/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md +++ b/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md @@ -107,7 +107,7 @@ Up to now, we've been using `.find` or `.find_by`, which are designed to return only a single object (notice the `LIMIT 1` in the generated SQL query?). `.where` is used when it is desirable to get a collection of objects. -Let's get a collection of non-admin users and see what we can do with it: +Let's get a collection of non-administrator users and see what we can do with it: ```ruby users = User.where.not(admin: true) @@ -364,7 +364,7 @@ StateMachines::InvalidTransition (Cannot transition state via :block from :activ We see that a validation error from what feels like a completely separate attribute comes back to haunt us when we try to update the user in any way. -In practical terms, we sometimes see this happen with GitLab admin settings -- +In practical terms, we sometimes see this happen with GitLab administration settings -- validations are sometimes added or changed in a GitLab update, resulting in previously saved settings now failing validation. Because you can only update a subset of settings at once through the UI, in this case the only way to get @@ -388,7 +388,7 @@ User.find_by_any_email('user@example.com') The `find_by_any_email` method is a custom method added by GitLab developers rather than a Rails-provided default method. -**Get a collection of admin users:** +**Get a collection of administrator users:** ```ruby User.admins @@ -443,7 +443,7 @@ group = Group.find_by_full_path('group/subgroup') # Get group's immediate child projects group.projects -# Get group's child projects, including those in sub-groups +# Get group's child projects, including those in subgroups group.all_projects ``` diff --git a/doc/administration/troubleshooting/sidekiq.md b/doc/administration/troubleshooting/sidekiq.md index e4082f87c7d..147bf1123ad 100644 --- a/doc/administration/troubleshooting/sidekiq.md +++ b/doc/administration/troubleshooting/sidekiq.md @@ -221,7 +221,7 @@ It is possible to use [Sidekiq API](https://github.com/mperham/sidekiq/wiki/API) to perform a number of troubleshooting steps on Sidekiq. These are the administrative commands and it should only be used if currently -admin interface is not suitable due to scale of installation. +administration interface is not suitable due to scale of installation. All these commands should be run using `gitlab-rails console`. |