Add indexes and change SQL for expired artifacts to deal with artifacts migration efficiently

Artifacts are in the middle of being migrated from ci_builds to
ci_job_artifacts. The expiration date is currently visible in both of
these tables and the test for whether an expired artifact is present
for a job is complex as it requires checking both the of the tables.

Add two new indexes, one on ci_builds.artifacts_expire_at and one on
ci_job_artifacts.expire_at to enable finding expired artifacts
efficiently.

And until the migration is finished, replace the SQL for finding
expired and non-expired artifacts with a hand-crafted UNION ALL based
query instead of using OR. This overcomes a database optimizer
limitation that prevents it from using these indexes.

When the migration is finished the next version should remove this
query and replace it with a much simpler query on just
ci_job_artifacts. See
https://gitlab.com/gitlab-org/gitlab-ce/issues/42561 for followup.

2 files changed, 25 insertions, 0 deletions

diff --git a/db/migrate/20180119160751_optimize_ci_job_artifacts.rb b/db/migrate/20180119160751_optimize_ci_job_artifacts.rb
new file mode 100644
index 00000000000..9b4340ed7b7
--- /dev/null
+++ b/db/migrate/20180119160751_optimize_ci_job_artifacts.rb
@@ -0,0 +1,23 @@
+# See http://doc.gitlab.com/ce/development/migration_style_guide.html
+# for more information on how to write migrations for GitLab.
+
+class OptimizeCiJobArtifacts < ActiveRecord::Migration
+  include Gitlab::Database::MigrationHelpers
+
+  # Set this constant to true if this migration requires downtime.
+  DOWNTIME = false
+
+  disable_ddl_transaction!
+
+  def up
+    # job_id is just here to be a covering index for index only scans
+    # since we'll almost always be joining against ci_builds on job_id
+    add_concurrent_index(:ci_job_artifacts, [:expire_at, :job_id])
+    add_concurrent_index(:ci_builds, [:artifacts_expire_at], where: "artifacts_file <> ''")
+  end
+
+  def down
+    remove_concurrent_index(:ci_job_artifacts, [:expire_at, :job_id])
+    remove_concurrent_index(:ci_builds, [:artifacts_expire_at], where: "artifacts_file <> ''")
+  end
+end
diff --git a/db/schema.rb b/db/schema.rb
index d07a4c31618..57b3f4c9988 100644
--- a/db/schema.rb
+++ b/db/schema.rb
@@ -293,6 +293,7 @@ ActiveRecord::Schema.define(version: 20180206200543) do
     t.integer "failure_reason"
   end
 
+  add_index "ci_builds", ["artifacts_expire_at"], name: "index_ci_builds_on_artifacts_expire_at", where: "(artifacts_file <> ''::text)", using: :btree
   add_index "ci_builds", ["auto_canceled_by_id"], name: "index_ci_builds_on_auto_canceled_by_id", using: :btree
   add_index "ci_builds", ["commit_id", "stage_idx", "created_at"], name: "index_ci_builds_on_commit_id_and_stage_idx_and_created_at", using: :btree
   add_index "ci_builds", ["commit_id", "status", "type"], name: "index_ci_builds_on_commit_id_and_status_and_type", using: :btree
@@ -333,6 +334,7 @@ ActiveRecord::Schema.define(version: 20180206200543) do
     t.string "file"
   end
 
+  add_index "ci_job_artifacts", ["expire_at", "job_id"], name: "index_ci_job_artifacts_on_expire_at_and_job_id", using: :btree
   add_index "ci_job_artifacts", ["job_id", "file_type"], name: "index_ci_job_artifacts_on_job_id_and_file_type", unique: true, using: :btree
   add_index "ci_job_artifacts", ["project_id"], name: "index_ci_job_artifacts_on_project_id", using: :btree