diff options
author | Nick Thomas <nick@gitlab.com> | 2019-03-01 02:25:37 +0300 |
---|---|---|
committer | Stan Hu <stanhu@gmail.com> | 2019-03-04 20:06:41 +0300 |
commit | f0c52df5e540e825be0babd04cc557f3f40cf1c6 (patch) | |
tree | 0d39c1df112c4ac71938c6051d65dc4751c9d526 /doc/administration | |
parent | 6b507626ed3499c1796706b507c30b8f46c706b7 (diff) |
sidekiq: terminate child processes at shutdown
Sidekiq jobs frequently spawn long-lived child processes to do work.
In some circumstances, these can be reparented to init when sidekiq is
terminated, leading to duplication of work and strange concurrency
problems.
This commit changes sidekiq so that, if run as a process group leader,
it will forward `INT` and `TERM` signals to the whole process group. If
the memory killer is active, it will also use the process group when
resorting to `kill -9` to shut down.
These changes mean that a naive `kill <pid-of-sidekiq>` will now do the
right thing, killing any child processes spawned by sidekiq, as long as
the process supervisor placed it in its own process group.
If sidekiq isn't a process group leader, this new code is skipped.
Diffstat (limited to 'doc/administration')
-rw-r--r-- | doc/administration/operations/sidekiq_memory_killer.md | 5 |
1 files changed, 5 insertions, 0 deletions
diff --git a/doc/administration/operations/sidekiq_memory_killer.md b/doc/administration/operations/sidekiq_memory_killer.md index cbffd883774..8eac42f2fe2 100644 --- a/doc/administration/operations/sidekiq_memory_killer.md +++ b/doc/administration/operations/sidekiq_memory_killer.md @@ -17,6 +17,11 @@ With the default settings, the MemoryKiller will cause a Sidekiq restart no more often than once every 15 minutes, with the restart causing about one minute of delay for incoming background jobs. +Some background jobs rely on long-running external processes. To ensure these +are cleanly terminated when Sidekiq is restarted, each Sidekiq process should be +run as a process group leader (e.g., using `chpst -P`). If using Omnibus or the +`bin/background_jobs` script with `runit` installed, this is handled for you. + ## Configuring the MemoryKiller The MemoryKiller is controlled using environment variables. |