Prevent amplification of ReactiveCachingWorker jobs upon failures

When `ReactiveCachingWorker` hits an SSL or other exception that occurs
quickly and reliably, automatically rescheduling a new worker could lead
to excessive number of jobs being scheduled. This happens because not
only does the failed job get rescheduled in a minute, but each Sidekiq
retry will also add even more rescheduled jobs.

In busy instances, this can become an issue because large numbers of
`ReactiveCachingWorker` running can cause high rates of `ExclusiveLease`
reads to occur and possibly saturate the Redis server with queries.

We now disable this automatic retry and rely on Sidekiq to perform its 3
retries with a backoff period.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/64176

1 files changed, 3 insertions, 2 deletions

diff --git a/spec/models/concerns/reactive_caching_spec.rb b/spec/models/concerns/reactive_caching_spec.rb
index 7faa196623f..e2ab9ddd4a5 100644
--- a/spec/models/concerns/reactive_caching_spec.rb
+++ b/spec/models/concerns/reactive_caching_spec.rb
@@ -206,8 +206,9 @@ describe ReactiveCaching, :use_clean_rails_memory_store_caching do
           expect(read_reactive_cache(instance)).to eq("preexisting")
         end
 
-        it 'enqueues a repeat worker' do
-          expect_reactive_cache_update_queued(instance)
+        it 'does not enqueue a repeat worker' do
+          expect(ReactiveCachingWorker)
+            .not_to receive(:perform_in)
 
           expect { go! }.to raise_error("foo")
         end