Refactor cache refreshing/expiring

This refactors repository caching so it's possible to selectively
refresh certain caches, instead of just expiring and refreshing
everything.

To allow this the various methods that were cached (e.g. "tag_count" and
"readme") use a similar pattern that makes expiring and refreshing
their data much easier.

In this new setup caches are refreshed as follows:

1. After a commit (but before running ProjectCacheWorker) we expire some
   basic caches such as the commit count and repository size.

2. ProjectCacheWorker will recalculate the commit count, repository
   size, then refresh a specific set of caches based on the list of
   files changed in a push payload.

This requires a bunch of changes to the various methods that may be
cached. For one, data should not be cached if a branch used or the
entire repository does not exist. To prevent all these methods from
handling this manually this is taken care of in
Repository#cache_method_output. Some methods still manually check for
the existence of a repository but this result is also cached.

With selective flushing implemented ProjectCacheWorker no longer uses an
exclusive lease for all of its work. Instead this worker only uses a
lease to limit the number of times the repository size is updated as
this is a fairly expensive operation.

1 files changed, 19 insertions, 35 deletions

diff --git a/app/workers/project_cache_worker.rb b/app/workers/project_cache_worker.rb
index 4dfa745fb50..27d7e652721 100644
--- a/app/workers/project_cache_worker.rb
+++ b/app/workers/project_cache_worker.rb
@@ -1,54 +1,38 @@
 # Worker for updating any project specific caches.
-#
-# This worker runs at most once every 15 minutes per project. This is to ensure
-# that multiple instances of jobs for this worker don't hammer the underlying
-# storage engine as much.
 class ProjectCacheWorker
   include Sidekiq::Worker
   include DedicatedSidekiqQueue
 
   LEASE_TIMEOUT = 15.minutes.to_i
 
-  def self.lease_for(project_id)
-    Gitlab::ExclusiveLease.
-      new("project_cache_worker:#{project_id}", timeout: LEASE_TIMEOUT)
-  end
+  # project_id - The ID of the project for which to flush the cache.
+  # refresh - An Array containing extra types of data to refresh such as
+  #           `:readme` to flush the README and `:changelog` to flush the
+  #           CHANGELOG.
+  def perform(project_id, refresh = [])
+    project = Project.find_by(id: project_id)
 
-  # Overwrite Sidekiq's implementation so we only schedule when actually needed.
-  def self.perform_async(project_id)
-    # If a lease for this project is still being held there's no point in
-    # scheduling a new job.
-    super unless lease_for(project_id).exists?
-  end
+    return unless project && project.repository.exists?
 
-  def perform(project_id)
-    if try_obtain_lease_for(project_id)
-      Rails.logger.
-        info("Obtained ProjectCacheWorker lease for project #{project_id}")
-    else
-      Rails.logger.
-        info("Could not obtain ProjectCacheWorker lease for project #{project_id}")
-
-      return
-    end
+    update_repository_size(project)
+    project.update_commit_count
 
-    update_caches(project_id)
+    project.repository.refresh_method_caches(refresh.map(&:to_sym))
   end
 
-  def update_caches(project_id)
-    project = Project.find(project_id)
+  def update_repository_size(project)
+    return unless try_obtain_lease_for(project.id, :update_repository_size)
 
-    return unless project.repository.exists?
+    Rails.logger.info("Updating repository size for project #{project.id}")
 
     project.update_repository_size
-    project.update_commit_count
-
-    if project.repository.root_ref
-      project.repository.build_cache
-    end
   end
 
-  def try_obtain_lease_for(project_id)
-    self.class.lease_for(project_id).try_obtain
+  private
+
+  def try_obtain_lease_for(project_id, section)
+    Gitlab::ExclusiveLease.
+      new("project_cache_worker:#{project_id}:#{section}", timeout: LEASE_TIMEOUT).
+      try_obtain
   end
 end