diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2022-02-18 12:45:46 +0300 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2022-02-18 12:45:46 +0300 |
commit | a7b3560714b4d9cc4ab32dffcd1f74a284b93580 (patch) | |
tree | 7452bd5c3545c2fa67a28aa013835fb4fa071baf /doc/development/database | |
parent | ee9173579ae56a3dbfe5afe9f9410c65bb327ca7 (diff) |
Add latest changes from gitlab-org/gitlab@14-8-stable-eev14.8.0-rc42
Diffstat (limited to 'doc/development/database')
-rw-r--r-- | doc/development/database/dbcheck-migrations-job.md | 67 | ||||
-rw-r--r-- | doc/development/database/efficient_in_operator_queries.md | 6 | ||||
-rw-r--r-- | doc/development/database/index.md | 1 | ||||
-rw-r--r-- | doc/development/database/loose_foreign_keys.md | 2 |
4 files changed, 72 insertions, 4 deletions
diff --git a/doc/development/database/dbcheck-migrations-job.md b/doc/development/database/dbcheck-migrations-job.md new file mode 100644 index 00000000000..af72e28a875 --- /dev/null +++ b/doc/development/database/dbcheck-migrations-job.md @@ -0,0 +1,67 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# db:check-migrations job + +This job runs on the test stage of a merge request pipeline. It checks: + +1. A schema dump comparison between the author's working branch and the target branch, + after executing a rollback of your new migrations. This check validates that the + schema properly resets to what it was before executing this new migration. +1. A schema dump comparison between the author's working branch and the `db/structure.sql` + file that the author committed. This check validates that it contains all the expected changes + in the migration. +1. A Git diff between the `db/schema_migrations` that the author committed and the + one that the script generated after running migrations. This check validates that everything + was properly committed. + +## Troubleshooting + +### False positives + +This job is allowed to fail, because it can throw some false positives. + +For example, when we drop a column and then roll back, this column is always +re-added at the end of the list of columns. If the column was previously in +the middle of the list, the rollback can't return the schema back exactly to +its previous state. The job fails, but it's an acceptable situation. + +For a real-life example, refer to +[this failed job](https://gitlab.com/gitlab-org/gitlab/-/jobs/2006544972#L138). +Here, the author dropped the `position` column. + +### Schema dump comparison fails after rollback + +This failure often happens if your working branch is behind the target branch. +A real scenario: + +```mermaid +graph LR + Main((main<br>commit A)) ===> |remove constraint<br>fk_rails_dbebdaa8fe| MainB((main<br>commit B)) + Main((main<br>commit A)) --> |checkout<br>dev| DevA((dev<br>commit A)):::dev + DevA((dev<br>commit A)) --> |add column<br>dependency_proxy_size| DevC((dev<br>commit C)):::dev + DevC -.-> |CI pipeline<br>executes| JOB-FAILED((JOB FAILED!)):::error + + classDef main fill:#f4f0ff,stroke:#7b58cf + classDef dev fill:#e9f3fc,stroke:#1f75cb + classDef error fill:#f15146,stroke:#d4121a +``` + +1. You check out the `dev` working branch from the `main` target branch. At this point, + each branch has their `HEAD` at commit A. +1. Someone works on the `main` branch and drops the `fk_rails_dbebdaa8fe` constraint, + thus creating commit B on `main`. +1. You add column `dependency_proxy_size` to your `dev` branch. +1. The `db:check-migrations` job fails for your `dev` branch's CI/CD pipeline, because + the `structure.sql` file did not rollback to its expected state. + +This happened because branch `dev` contained commits A and C, not B. Its database schema +did not know about the removal of the `fk_rails_dbebdaa8fe` constraint. When comparing the two +schemas, the `dev` branch contained this constraint while the `main` branch didn't. + +This example really happened. Read the [job failure logs](https://gitlab.com/gitlab-org/gitlab/-/jobs/1992050890). + +To fix these kind of issues, rebase the working branch onto the target branch to get the latest changes. diff --git a/doc/development/database/efficient_in_operator_queries.md b/doc/development/database/efficient_in_operator_queries.md index 76518a6f5e2..2503be826ea 100644 --- a/doc/development/database/efficient_in_operator_queries.md +++ b/doc/development/database/efficient_in_operator_queries.md @@ -368,7 +368,7 @@ then the query might perform worse than the non-optimized query. The `milestone_ "index_issues_on_milestone_id" btree (milestone_id) ``` -Adding the `miletone_id = X` filter to the `scope` argument or to the optimized scope causes bad performance. +Adding the `milestone_id = X` filter to the `scope` argument or to the optimized scope causes bad performance. Example (bad): @@ -618,7 +618,7 @@ The following example shows the final `ORDER BY` clause: ORDER BY extract('epoch' FROM epics.closed_at - epics.created_at) DESC, epics.id DESC ``` -Snippet for loading records ordered by the calcualted duration: +Snippet for loading records ordered by the calculated duration: ```ruby arel_table = Epic.arel_table @@ -641,7 +641,7 @@ records = Gitlab::Pagination::Keyset::InOperatorOptimization::QueryBuilder.new( array_mapping_scope: -> (id_expression) { Epic.where(Epic.arel_table[:group_id].eq(id_expression)) } ).execute.limit(20) -puts records.pluck(:duration_in_seconds, :id) # other columnns are not available +puts records.pluck(:duration_in_seconds, :id) # other columns are not available ``` Building the query requires quite a bit of configuration. For the order configuration you diff --git a/doc/development/database/index.md b/doc/development/database/index.md index a7b752e14ef..efc48f72d00 100644 --- a/doc/development/database/index.md +++ b/doc/development/database/index.md @@ -19,6 +19,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w - [Understanding EXPLAIN plans](../understanding_explain_plans.md) - [explain.depesz.com](https://explain.depesz.com/) or [explain.dalibo.com](https://explain.dalibo.com/) for visualizing the output of `EXPLAIN` - [pgFormatter](https://sqlformat.darold.net/) a PostgreSQL SQL syntax beautifier +- [db:check-migrations job](dbcheck-migrations-job.md) ## Migrations diff --git a/doc/development/database/loose_foreign_keys.md b/doc/development/database/loose_foreign_keys.md index 97d150b1a18..d08e90683fe 100644 --- a/doc/development/database/loose_foreign_keys.md +++ b/doc/development/database/loose_foreign_keys.md @@ -61,7 +61,7 @@ following information: - Child table name (`ci_pipelines`) - The data cleanup method (`async_delete` or `async_nullify`) -The YAML file is located at `lib/gitlab/database/gitlab_loose_foreign_keys.yml`. The file groups +The YAML file is located at `config/gitlab_loose_foreign_keys.yml`. The file groups foreign key definitions by the name of the child table. The child table can have multiple loose foreign key definitions, therefore we store them as an array. |