Improve performance of the cycle analytics page.

1. These changes bring down page load time for 100 issues from more than
   a minute to about 1.5 seconds.

2. This entire commit is composed of these types of performance
   enhancements:

     - Cache relevant data in `IssueMetrics` wherever possible.
     - Cache relevant data in `MergeRequestMetrics` wherever possible.
     - Preload metrics

3. Given these improvements, we now only need to make 4 SQL calls:

    - Load all issues
    - Load all merge requests
    - Load all metrics for the issues
    - Load all metrics for the merge requests

4. A list of all the data points that are now being pre-calculated:

    a. The first time an issue is mentioned in a commit

      - In `GitPushService`, find all issues mentioned by the given commit
        using `ReferenceExtractor`. Set the `first_mentioned_in_commit_at`
        flag for each of them.

      - There seems to be a (pre-existing) bug here - files (and
        therefore commits) created using the Web CI don't have
        cross-references created, and issues are not closed even when
        the commit title is "Fixes #xx".

    b. The first time a merge request is deployed to production

      When a `Deployment` is created, find all merge requests that
      were merged in before the deployment, and set the
      `first_deployed_to_production_at` flag for each of them.

    c. The start / end time for a merge request pipeline

      Hook into the `Pipeline` state machine. When the `status` moves to
      `running`, find the merge requests whose tip commit matches the
      pipeline, and record the `latest_build_started_at` time for each
      of them. When the `status` moves to `success`, record the
      `latest_build_finished_at` time.

    d. The merge requests that close an issue

      - This was a big cause of the performance problems we were having
        with Cycle Analytics. We need to use `ReferenceExtractor` to make
        this calculation, which is slow when we have to run it on a large
        number of merge requests.

      - When a merge request is created, updated, or refreshed, find the
        issues it closes, and create an instance of
        `MergeRequestsClosingIssues`, which acts as a join model between
        merge requests and issues.

      - If a `MergeRequestsClosingIssues` instance links a merge request
        and an issue, that issue closes that merge request.

5. The `Queries` module was changed into a class, so we can cache the
   results of `issues` and `merge_requests_closing_issues` across
   various cycle analytics stages.

6. The code added in this commit is untested. Tests will be added in the
   next commit.

1 files changed, 19 insertions, 4 deletions

diff --git a/db/fixtures/development/17_cycle_analytics.rb b/db/fixtures/development/17_cycle_analytics.rb
index 6d5a28c2e81..46cf0fbd805 100644
--- a/db/fixtures/development/17_cycle_analytics.rb
+++ b/db/fixtures/development/17_cycle_analytics.rb
@@ -24,30 +24,37 @@ class Gitlab::Seeder::CycleAnalytics
   def seed!
     Sidekiq::Testing.inline! do
       issues = create_issues(@project)
+      print '.'
 
       # Stage 1
       Timecop.travel 5.days.from_now
       add_milestones_and_list_labels(issues)
+      print '.'
 
       # Stage 2
       Timecop.travel 5.days.from_now
       branches = mention_in_commits(issues)
+      print '.'
 
       # Stage 3
       Timecop.travel 5.days.from_now
       merge_requests = create_merge_requests_closing_issues(issues, branches)
+      print '.'
 
       # Stage 4
       Timecop.travel 5.days.from_now
       run_builds(merge_requests)
+      print '.'
 
       # Stage 5
       Timecop.travel 5.days.from_now
       merge_merge_requests(merge_requests)
+      print '.'
 
       # Stage 6 / 7
       Timecop.travel 5.days.from_now
       deploy_to_production(merge_requests)
+      print '.'
     end
 
     print '.'
@@ -101,8 +108,14 @@ class Gitlab::Seeder::CycleAnalytics
       }
 
       commit_sha = Gitlab::Git::Blob.commit(issue.project.repository, options)
-      commit = issue.project.repository.commit(commit_sha)
-      commit.create_cross_references!
+      issue.project.repository.commit(commit_sha)
+
+
+      GitPushService.new(issue.project,
+                         @user,
+                         oldrev: issue.project.repository.commit("master").sha,
+                         newrev: commit_sha,
+                         ref: 'refs/heads/master').execute
 
       branch_name
     end
@@ -162,10 +175,12 @@ end
 
 Gitlab::Seeder.quiet do
   if ENV['SEED_CYCLE_ANALYTICS']
-      seeder = Gitlab::Seeder::CycleAnalytics.new(Project.find(1))
+    Project.all.each do |project|
+      seeder = Gitlab::Seeder::CycleAnalytics.new(project)
       seeder.seed!
+    end
   elsif ENV['CYCLE_ANALYTICS_PERF_TEST']
-    seeder = Gitlab::Seeder::CycleAnalytics.new(Project.first, perf: true)
+    seeder = Gitlab::Seeder::CycleAnalytics.new(Project.order(:id).first, perf: true)
     seeder.seed!
   else
     puts "Not running the cycle analytics seed file. Use the `SEED_CYCLE_ANALYTICS` environment variable to enable it."