diff options
author | Harish Ramachandran <harish@gitlab.com> | 2019-08-01 12:27:04 +0300 |
---|---|---|
committer | Achilleas Pipinellis <axil@gitlab.com> | 2019-08-01 12:27:04 +0300 |
commit | a0ccab454667157a6f7717dd8ed7dc1641618fc0 (patch) | |
tree | 235f43fa7b320b108746711c2a087ffeced02064 /doc/administration | |
parent | b559d9909f356f718a7da711dcf574dcddbbcc59 (diff) |
Troubleshooting steps when there are different Gitaly node secrets
Diffstat (limited to 'doc/administration')
-rw-r--r-- | doc/administration/gitaly/index.md | 91 |
1 files changed, 91 insertions, 0 deletions
diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 0f547ef03bf..f6f02221fe3 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -510,3 +510,94 @@ implemented as calls to `gitaly-ruby`: ``` sum(rate(grpc_client_handled_total[5m])) by (grpc_method) > 0 ``` + +### Repository changes fail with a `401 Unauthorized` error + +If you're running Gitaly on its own server and notice that users can +successfully clone and fetch repositories (via both SSH and HTTPS), but can't +push to them or make changes to the repository in the web UI without getting a +`401 Unauthorized` message, then it's possible Gitaly is failing to authenticate +with the other nodes due to having the [wrong secrets file](#3-gitaly-server-configuration). + +Confirm the following are all true: + +- When any user performs a `git push` to any repository on this Gitaly node, it + fails with the following error (note the `401 Unauthorized`): + + ```sh + remote: GitLab: 401 Unauthorized + To <REMOTE_URL> + ! [remote rejected] branch-name -> branch-name (pre-receive hook declined) + error: failed to push some refs to '<REMOTE_URL>' + ``` + +- When any user adds or modifies a file from the repository using the GitLab + UI, it immediatley fails with a red `401 Unauthorized` banner. +- Creating a new project and [initializing it with a README](../../gitlab-basics/create-project.md#blank-projects) + successfully creates the project but doesn't create the README. +- When [tailing the logs](https://docs.gitlab.com/omnibus/settings/logs.md#tail-logs-in-a-console-on-the-server) on an app node and reproducing the error, you get `401` errors + when reaching the `/api/v4/internal/allowed` endpoint: + + ```sh + # api_json.log + { + "time": "2019-07-18T00:30:14.967Z", + "severity": "INFO", + "duration": 0.57, + "db": 0, + "view": 0.57, + "status": 401, + "method": "POST", + "path": "\/api\/v4\/internal\/allowed", + "params": [ + { + "key": "action", + "value": "git-receive-pack" + }, + { + "key": "changes", + "value": "REDACTED" + }, + { + "key": "gl_repository", + "value": "REDACTED" + }, + { + "key": "project", + "value": "\/path\/to\/project.git" + }, + { + "key": "protocol", + "value": "web" + }, + { + "key": "env", + "value": "{\"GIT_ALTERNATE_OBJECT_DIRECTORIES\":[],\"GIT_ALTERNATE_OBJECT_DIRECTORIES_RELATIVE\":[],\"GIT_OBJECT_DIRECTORY\":null,\"GIT_OBJECT_DIRECTORY_RELATIVE\":null}" + }, + { + "key": "user_id", + "value": "2" + }, + { + "key": "secret_token", + "value": "[FILTERED]" + } + ], + "host": "gitlab.example.com", + "ip": "REDACTED", + "ua": "Ruby", + "route": "\/api\/:version\/internal\/allowed", + "queue_duration": 4.24, + "gitaly_calls": 0, + "gitaly_duration": 0, + "correlation_id": "XPUZqTukaP3" + } + + # nginx_access.log + [IP] - - [18/Jul/2019:00:30:14 +0000] "POST /api/v4/internal/allowed HTTP/1.1" 401 30 "" "Ruby" + ``` + +To fix this problem, confirm that your [`gitlab-secrets.json` file](#3-gitaly-server-configuration) +on the Gitaly node matches the one on all other nodes. If it doesn't match, +update the secrets file on the Gitaly node to match the others, then +[reconfigure the node](../restart_gitlab.md#omnibus-gitlab-reconfigure). |