From 905c1110b08f93a19661cf42a276c7ea90d0a0ff Mon Sep 17 00:00:00 2001 From: GitLab Bot Date: Tue, 22 Oct 2019 11:31:16 +0000 Subject: Add latest changes from gitlab-org/gitlab@12-4-stable-ee --- .../operations/cleaning_up_redis_sessions.md | 4 +- .../operations/fast_ssh_key_lookup.md | 4 +- .../operations/moving_repositories.md | 6 +- .../operations/sidekiq_memory_killer.md | 66 ++++++++++++++++------ doc/administration/operations/ssh_certificates.md | 2 +- doc/administration/operations/unicorn.md | 2 +- 6 files changed, 58 insertions(+), 26 deletions(-) (limited to 'doc/administration/operations') diff --git a/doc/administration/operations/cleaning_up_redis_sessions.md b/doc/administration/operations/cleaning_up_redis_sessions.md index c9b5ab9d290..fd469ae23e3 100644 --- a/doc/administration/operations/cleaning_up_redis_sessions.md +++ b/doc/administration/operations/cleaning_up_redis_sessions.md @@ -11,8 +11,8 @@ start building up again after you clean up. In GitLab versions prior to 7.3.0, the session keys in Redis are 16-byte hexadecimal values such as '976aa289e2189b17d7ef525a6702ace9'. Starting with GitLab 7.3.0, the keys are -prefixed with 'session:gitlab:', so they would look like -'session:gitlab:976aa289e2189b17d7ef525a6702ace9'. Below we describe how to +prefixed with `session:gitlab:`, so they would look like +`session:gitlab:976aa289e2189b17d7ef525a6702ace9`. Below we describe how to remove the keys in the old format. **Note:** the instructions below must be modified in accordance with your diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index 16424c25a98..9a38e8ddd23 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -2,7 +2,7 @@ NOTE: **Note:** This document describes a drop-in replacement for the `authorized_keys` file for normal (non-deploy key) users. Consider -using [ssh certificates](ssh_certificates.md), they are even faster, +using [SSH certificates](ssh_certificates.md), they are even faster, but are not a drop-in replacement. > [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/1631) in @@ -78,7 +78,7 @@ CAUTION: **Caution:** Do not disable writes until SSH is confirmed to be working perfectly, because the file will quickly become out-of-date. In the case of lookup failures (which are common), the `authorized_keys` -file will still be scanned. So git SSH performance will still be slow for many +file will still be scanned. So Git SSH performance will still be slow for many users as long as a large file exists. You can disable any more writes to the `authorized_keys` file by unchecking diff --git a/doc/administration/operations/moving_repositories.md b/doc/administration/operations/moving_repositories.md index ec11a92db1b..d54ffacd281 100644 --- a/doc/administration/operations/moving_repositories.md +++ b/doc/administration/operations/moving_repositories.md @@ -31,7 +31,7 @@ If you want to see progress, replace `-xf` with `-xvf`. ### Tar pipe to another server You can also use a tar pipe to copy data to another server. If your -'git' user has SSH access to the newserver as 'git@newserver', you +`git` user has SSH access to the newserver as `git@newserver`, you can pipe the data through SSH. ``` @@ -61,7 +61,7 @@ If you want to see progress, replace `-a` with `-av`. ### Single rsync to another server -If the 'git' user on your source system has SSH access to the target +If the `git` user on your source system has SSH access to the target server you can send the repositories over the network with rsync. ``` @@ -95,7 +95,7 @@ after switching to the new repository storage directory. This will sync repositories with 10 rsync processes at a time. We keep track of progress so that the transfer can be restarted if necessary. -First we create a new directory, owned by 'git', to hold transfer +First we create a new directory, owned by `git`, to hold transfer logs. We assume the directory is empty before we start the transfer procedure, and that we are the only ones writing files in it. diff --git a/doc/administration/operations/sidekiq_memory_killer.md b/doc/administration/operations/sidekiq_memory_killer.md index 8eac42f2fe2..79e9fb778b6 100644 --- a/doc/administration/operations/sidekiq_memory_killer.md +++ b/doc/administration/operations/sidekiq_memory_killer.md @@ -2,7 +2,7 @@ The GitLab Rails application code suffers from memory leaks. For web requests this problem is made manageable using -[unicorn-worker-killer](https://github.com/kzk/unicorn-worker-killer) which +[`unicorn-worker-killer`](https://github.com/kzk/unicorn-worker-killer) which restarts Unicorn worker processes in between requests when needed. The Sidekiq MemoryKiller applies the same approach to the Sidekiq processes used by GitLab to process background jobs. @@ -10,8 +10,8 @@ to process background jobs. Unlike unicorn-worker-killer, which is enabled by default for all GitLab installations since GitLab 6.4, the Sidekiq MemoryKiller is enabled by default _only_ for Omnibus packages. The reason for this is that the MemoryKiller -relies on Runit to restart Sidekiq after a memory-induced shutdown and GitLab -installations from source do not all use Runit or an equivalent. +relies on runit to restart Sidekiq after a memory-induced shutdown and GitLab +installations from source do not all use runit or an equivalent. With the default settings, the MemoryKiller will cause a Sidekiq restart no more often than once every 15 minutes, with the restart causing about one @@ -26,18 +26,50 @@ run as a process group leader (e.g., using `chpst -P`). If using Omnibus or the The MemoryKiller is controlled using environment variables. -- `SIDEKIQ_MEMORY_KILLER_MAX_RSS`: if this variable is set, and its value is - greater than 0, then after each Sidekiq job, the MemoryKiller will check the - RSS of the Sidekiq process that executed the job. If the RSS of the Sidekiq - process (expressed in kilobytes) exceeds SIDEKIQ_MEMORY_KILLER_MAX_RSS, a - delayed shutdown is triggered. The default value for Omnibus packages is set - [in the omnibus-gitlab +- `SIDEKIQ_DAEMON_MEMORY_KILLER`: defaults to 0. When set to 1, the MemoryKiller + works in _daemon_ mode. Otherwise, the MemoryKiller works in _legacy_ mode. + + In _legacy_ mode, the MemoryKiller checks the Sidekiq process RSS after each job. + + In _daemon_ mode, the MemoryKiller checks the Sidekiq process RSS every 3 seconds + (defined by `SIDEKIQ_MEMORY_KILLER_CHECK_INTERVAL`). + +- `SIDEKIQ_MEMORY_KILLER_MAX_RSS`: if this variable is set, and its value is greater + than 0, the MemoryKiller is enabled. Otherwise the MemoryKiller is disabled. + + `SIDEKIQ_MEMORY_KILLER_MAX_RSS` defines the Sidekiq process allowed RSS. + + In _legacy_ mode, if the Sidekiq process exceeds the allowed RSS then an irreversible + delayed graceful restart will be triggered. The restart of Sidekiq will happen + after `SIDEKIQ_MEMORY_KILLER_GRACE_TIME` seconds. + + In _daemon_ mode, if the Sidekiq process exceeds the allowed RSS for longer than + `SIDEKIQ_MEMORY_KILLER_GRACE_TIME` the graceful restart will be triggered. If the + Sidekiq process go below the allowed RSS within `SIDEKIQ_MEMORY_KILLER_GRACE_TIME`, + the restart will be aborted. + + The default value for Omnibus packages is set + [in the Omnibus GitLab repository](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/attributes/default.rb). -- `SIDEKIQ_MEMORY_KILLER_GRACE_TIME`: defaults to 900 seconds (15 minutes). When - a shutdown is triggered, the Sidekiq process will keep working normally for - another 15 minutes. -- `SIDEKIQ_MEMORY_KILLER_SHUTDOWN_WAIT`: defaults to 30 seconds. When the grace - time has expired, the MemoryKiller tells Sidekiq to stop accepting new jobs. - Existing jobs get 30 seconds to finish. After that, the MemoryKiller tells - Sidekiq to shut down, and an external supervision mechanism (e.g. Runit) must - restart Sidekiq. + +- `SIDEKIQ_MEMORY_KILLER_HARD_LIMIT_RSS`: is used by _daemon_ mode. If the Sidekiq + process RSS (expressed in kilobytes) exceeds `SIDEKIQ_MEMORY_KILLER_HARD_LIMIT_RSS`, + an immediate graceful restart of Sidekiq is triggered. + +- `SIDEKIQ_MEMORY_KILLER_CHECK_INTERVAL`: used in _daemon_ mode to define how + often to check process RSS, default to 3 seconds. + +- `SIDEKIQ_MEMORY_KILLER_GRACE_TIME`: defaults to 900 seconds (15 minutes). + The usage of this variable is described as part of `SIDEKIQ_MEMORY_KILLER_MAX_RSS`. + +- `SIDEKIQ_MEMORY_KILLER_SHUTDOWN_WAIT`: defaults to 30 seconds. This defines the + maximum time allowed for all Sidekiq jobs to finish. No new jobs will be accepted + during that time, and the process will exit as soon as all jobs finish. + + If jobs do not finish during that time, the MemoryKiller will interrupt all currently + running jobs by sending `SIGTERM` to the Sidekiq process. + + If the process hard shutdown/restart is not performed by Sidekiq, + the Sidekiq process will be forcefully terminated after + `Sidekiq.options[:timeout] * 2` seconds. An external supervision mechanism + (e.g. runit) must restart Sidekiq afterwards. diff --git a/doc/administration/operations/ssh_certificates.md b/doc/administration/operations/ssh_certificates.md index 3792bcd3bca..2a9a4cff34e 100644 --- a/doc/administration/operations/ssh_certificates.md +++ b/doc/administration/operations/ssh_certificates.md @@ -3,7 +3,7 @@ > [Available in](https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/19911) GitLab > Community Edition 11.2. -GitLab's default SSH authentication requires users to upload their ssh +GitLab's default SSH authentication requires users to upload their SSH public keys before they can use the SSH transport. In centralized (e.g. corporate) environments this can be a hassle diff --git a/doc/administration/operations/unicorn.md b/doc/administration/operations/unicorn.md index 8178cb243f3..969f1211643 100644 --- a/doc/administration/operations/unicorn.md +++ b/doc/administration/operations/unicorn.md @@ -40,7 +40,7 @@ master process has PID 56227 below. The main tunables for Unicorn are the number of worker processes and the request timeout after which the Unicorn master terminates a worker process. -See the [omnibus-gitlab Unicorn settings +See the [Omnibus GitLab Unicorn settings documentation](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/unicorn.md) if you want to adjust these settings. -- cgit v1.2.3