From b08279013423a66f06f5edde4e067f328fe135bd Mon Sep 17 00:00:00 2001 From: GitLab Bot Date: Wed, 18 Sep 2019 18:06:14 +0000 Subject: Add latest changes from gitlab-org/gitlab@master --- .../operations/sidekiq_memory_killer.md | 58 +++++++++++++++++----- 1 file changed, 45 insertions(+), 13 deletions(-) (limited to 'doc/administration/operations') diff --git a/doc/administration/operations/sidekiq_memory_killer.md b/doc/administration/operations/sidekiq_memory_killer.md index 8eac42f2fe2..509bc77a8c3 100644 --- a/doc/administration/operations/sidekiq_memory_killer.md +++ b/doc/administration/operations/sidekiq_memory_killer.md @@ -26,18 +26,50 @@ run as a process group leader (e.g., using `chpst -P`). If using Omnibus or the The MemoryKiller is controlled using environment variables. -- `SIDEKIQ_MEMORY_KILLER_MAX_RSS`: if this variable is set, and its value is - greater than 0, then after each Sidekiq job, the MemoryKiller will check the - RSS of the Sidekiq process that executed the job. If the RSS of the Sidekiq - process (expressed in kilobytes) exceeds SIDEKIQ_MEMORY_KILLER_MAX_RSS, a - delayed shutdown is triggered. The default value for Omnibus packages is set +- `SIDEKIQ_DAEMON_MEMORY_KILLER`: defaults to 0. When set to 1, the MemoryKiller + works in _daemon_ mode. Otherwise, the MemoryKiller works in _legacy_ mode. + + In _legacy_ mode, the MemoryKiller checks the Sidekiq process RSS after each job. + + In _daemon_ mode, the MemoryKiller checks the Sidekiq process RSS every 3 seconds + (defined by `SIDEKIQ_MEMORY_KILLER_CHECK_INTERVAL`). + +- `SIDEKIQ_MEMORY_KILLER_MAX_RSS`: if this variable is set, and its value is greater + than 0, the MemoryKiller is enabled. Otherwise the MemoryKiller is disabled. + + `SIDEKIQ_MEMORY_KILLER_MAX_RSS` defines the Sidekiq process allowed RSS. + + In _legacy_ mode, if the Sidekiq process exceeds the allowed RSS then an irreversible + delayed graceful restart will be triggered. The restart of Sidekiq will happen + after `SIDEKIQ_MEMORY_KILLER_GRACE_TIME` seconds. + + In _daemon_ mode, if the Sidekiq process exceeds the allowed RSS for longer than + `SIDEKIQ_MEMORY_KILLER_GRACE_TIME` the graceful restart will be triggered. If the + Sidekiq process go below the allowed RSS within `SIDEKIQ_MEMORY_KILLER_GRACE_TIME`, + the restart will be aborted. + + The default value for Omnibus packages is set [in the omnibus-gitlab repository](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/attributes/default.rb). -- `SIDEKIQ_MEMORY_KILLER_GRACE_TIME`: defaults to 900 seconds (15 minutes). When - a shutdown is triggered, the Sidekiq process will keep working normally for - another 15 minutes. -- `SIDEKIQ_MEMORY_KILLER_SHUTDOWN_WAIT`: defaults to 30 seconds. When the grace - time has expired, the MemoryKiller tells Sidekiq to stop accepting new jobs. - Existing jobs get 30 seconds to finish. After that, the MemoryKiller tells - Sidekiq to shut down, and an external supervision mechanism (e.g. Runit) must - restart Sidekiq. + +- `SIDEKIQ_MEMORY_KILLER_HARD_LIMIT_RSS`: is used by _daemon_ mode. If the Sidekiq + process RSS (expressed in kilobytes) exceeds `SIDEKIQ_MEMORY_KILLER_HARD_LIMIT_RSS`, + an immediate graceful restart of Sidekiq is triggered. + +- `SIDEKIQ_MEMORY_KILLER_CHECK_INTERVAL`: used in _daemon_ mode to define how + often to check process RSS, default to 3 seconds. + +- `SIDEKIQ_MEMORY_KILLER_GRACE_TIME`: defaults to 900 seconds (15 minutes). + The usage of this variable is described as part of `SIDEKIQ_MEMORY_KILLER_MAX_RSS`. + +- `SIDEKIQ_MEMORY_KILLER_SHUTDOWN_WAIT`: defaults to 30 seconds. This defines the + maximum time allowed for all Sidekiq jobs to finish. No new jobs will be accepted + during that time, and the process will exit as soon as all jobs finish. + + If jobs do not finish during that time, the MemoryKiller will interrupt all currently + running jobs by sending `SIGTERM` to the Sidekiq process. + + If the process hard shutdown/restart is not performed by Sidekiq, + the Sidekiq process will be forcefully terminated after + `Sidekiq.options[:timeout] * 2` seconds. An external supervision mechanism + (e.g. Runit) must restart Sidekiq afterwards. -- cgit v1.2.3