diff options
author | Nick Thomas <nick@gitlab.com> | 2019-03-01 02:25:37 +0300 |
---|---|---|
committer | Stan Hu <stanhu@gmail.com> | 2019-03-04 20:06:41 +0300 |
commit | f0c52df5e540e825be0babd04cc557f3f40cf1c6 (patch) | |
tree | 0d39c1df112c4ac71938c6051d65dc4751c9d526 /spec/lib/gitlab/sidekiq_middleware | |
parent | 6b507626ed3499c1796706b507c30b8f46c706b7 (diff) |
sidekiq: terminate child processes at shutdown
Sidekiq jobs frequently spawn long-lived child processes to do work.
In some circumstances, these can be reparented to init when sidekiq is
terminated, leading to duplication of work and strange concurrency
problems.
This commit changes sidekiq so that, if run as a process group leader,
it will forward `INT` and `TERM` signals to the whole process group. If
the memory killer is active, it will also use the process group when
resorting to `kill -9` to shut down.
These changes mean that a naive `kill <pid-of-sidekiq>` will now do the
right thing, killing any child processes spawned by sidekiq, as long as
the process supervisor placed it in its own process group.
If sidekiq isn't a process group leader, this new code is skipped.
Diffstat (limited to 'spec/lib/gitlab/sidekiq_middleware')
-rw-r--r-- | spec/lib/gitlab/sidekiq_middleware/memory_killer_spec.rb | 13 |
1 files changed, 12 insertions, 1 deletions
diff --git a/spec/lib/gitlab/sidekiq_middleware/memory_killer_spec.rb b/spec/lib/gitlab/sidekiq_middleware/memory_killer_spec.rb index 5df56178df2..ff8c0825ee4 100644 --- a/spec/lib/gitlab/sidekiq_middleware/memory_killer_spec.rb +++ b/spec/lib/gitlab/sidekiq_middleware/memory_killer_spec.rb @@ -35,7 +35,7 @@ describe Gitlab::SidekiqMiddleware::MemoryKiller do stub_const("#{described_class}::MAX_RSS", 5.kilobytes) end - it 'sends the STP, TERM and KILL signals at expected times' do + it 'sends the TSTP, TERM and KILL signals at expected times' do expect(subject).to receive(:sleep).with(15 * 60).ordered expect(Process).to receive(:kill).with('SIGTSTP', pid).ordered @@ -47,6 +47,17 @@ describe Gitlab::SidekiqMiddleware::MemoryKiller do run end + + it 'sends TSTP and TERM to the pid, but KILL to the pgroup, when running as process leader' do + allow(Process).to receive(:getpgrp) { pid } + allow(subject).to receive(:sleep) + + expect(Process).to receive(:kill).with('SIGTSTP', pid).ordered + expect(Process).to receive(:kill).with('SIGTERM', pid).ordered + expect(Process).to receive(:kill).with('SIGKILL', "-#{pid}").ordered + + run + end end context 'when MAX_RSS is not exceeded' do |