diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2022-05-31 18:09:02 +0300 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2022-05-31 18:09:02 +0300 |
| commit | eb6b17504921dc3b8c22715b9ec7790a34b4abe0 (patch) | |
| tree | b4ca44c1e2028cfa6864cecad568b12b04d4a024 /doc | |
| parent | 8243505178033432b7fc6834eef425c9dcdfd7bc (diff) | |
Add latest changes from gitlab-org/gitlab@master
Diffstat (limited to 'doc')
78 files changed, 530 insertions, 77 deletions
diff --git a/doc/.vale/gitlab/BadPlurals.yml b/doc/.vale/gitlab/BadPlurals.yml index efc55ac3d56..63f002fec94 100644 --- a/doc/.vale/gitlab/BadPlurals.yml +++ b/doc/.vale/gitlab/BadPlurals.yml @@ -6,9 +6,8 @@ # For a list of all options, see https://docs.errata.ai/vale/styles extends: existence message: 'Rewrite "%s" to be plural, without parentheses.' -link: https://docs.gitlab.com/ee/development/documentation/styleguide/word_list.html +link: https://docs.gitlab.com/ee/development/documentation/styleguide/word_list.html#s level: warning -scope: raw ignorecase: true raw: - '\b\w+\(s\)(?<!http\(s\))' diff --git a/doc/administration/application_settings_cache.md b/doc/administration/application_settings_cache.md index b3aac932fa8..33649cf53fc 100644 --- a/doc/administration/application_settings_cache.md +++ b/doc/administration/application_settings_cache.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Memory info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/operations/sidekiq_memory_killer.md b/doc/administration/operations/sidekiq_memory_killer.md index e48ac6e65eb..d9558c3d7a6 100644 --- a/doc/administration/operations/sidekiq_memory_killer.md +++ b/doc/administration/operations/sidekiq_memory_killer.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Memory info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/postgresql/database_load_balancing.md b/doc/administration/postgresql/database_load_balancing.md index 3b37ffd2d9d..0b6e00902e9 100644 --- a/doc/administration/postgresql/database_load_balancing.md +++ b/doc/administration/postgresql/database_load_balancing.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/postgresql/external.md b/doc/administration/postgresql/external.md index f4e4c7f8bef..5d693793a92 100644 --- a/doc/administration/postgresql/external.md +++ b/doc/administration/postgresql/external.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/postgresql/index.md b/doc/administration/postgresql/index.md index 15425a6d9f2..a8e69041f89 100644 --- a/doc/administration/postgresql/index.md +++ b/doc/administration/postgresql/index.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/postgresql/pgbouncer.md b/doc/administration/postgresql/pgbouncer.md index 046b8da691f..5a39565cef7 100644 --- a/doc/administration/postgresql/pgbouncer.md +++ b/doc/administration/postgresql/pgbouncer.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index bcac2ff3a2d..e9b607ad5d4 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/postgresql/standalone.md b/doc/administration/postgresql/standalone.md index b21625acb56..5428e44ccc0 100644 --- a/doc/administration/postgresql/standalone.md +++ b/doc/administration/postgresql/standalone.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/troubleshooting/elasticsearch.md b/doc/administration/troubleshooting/elasticsearch.md index 97e625eb0a3..52de2f696dd 100644 --- a/doc/administration/troubleshooting/elasticsearch.md +++ b/doc/administration/troubleshooting/elasticsearch.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Global Search info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/troubleshooting/postgresql.md b/doc/administration/troubleshooting/postgresql.md index d0ed3c5c12a..7f32d01d617 100644 --- a/doc/administration/troubleshooting/postgresql.md +++ b/doc/administration/troubleshooting/postgresql.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/api/merge_requests.md b/doc/api/merge_requests.md index 124bbd3d5f4..850440040a4 100644 --- a/doc/api/merge_requests.md +++ b/doc/api/merge_requests.md @@ -69,7 +69,7 @@ Parameters: | `in` | string | no | Modify the scope of the `search` attribute. `title`, `description`, or a string joining them with comma. Default is `title,description`. | | `wip` | string | no | Filter merge requests against their `wip` status. `yes` to return *only* draft merge requests, `no` to return *non-draft* merge requests. | | `not` | Hash | no | Return merge requests that do not match the parameters supplied. Accepts: `labels`, `milestone`, `author_id`, `author_username`, `assignee_id`, `assignee_username`, `reviewer_id`, `reviewer_username`, `my_reaction_emoji`. | -| `environment` | string | no | Returns merge requests deployed to the given environment. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | +| `environment` | string | no | Returns merge requests deployed to the given environment. | | `deployed_before` | datetime | no | Return merge requests deployed before the given date/time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | | `deployed_after` | datetime | no | Return merge requests deployed after the given date/time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | @@ -265,13 +265,13 @@ Parameters: | `approved_by_ids` **(PREMIUM)** | integer array | no | Returns merge requests which have been approved by all the users with the given `id`s (Max: 5). `None` returns merge requests with no approvals. `Any` returns merge requests with an approval. | | `reviewer_id` | integer | no | Returns merge requests which have the user as a [reviewer](../user/project/merge_requests/getting_started.md#reviewer) with the given user `id`. `None` returns merge requests with no reviewers. `Any` returns merge requests with any reviewer. Mutually exclusive with `reviewer_username`. | | `reviewer_username` | string | no | Returns merge requests which have the user as a [reviewer](../user/project/merge_requests/getting_started.md#reviewer) with the given `username`. `None` returns merge requests with no reviewers. `Any` returns merge requests with any reviewer. Mutually exclusive with `reviewer_id`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/49341) in GitLab 13.8. | - | `my_reaction_emoji` | string | no | Return merge requests reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. | | `source_branch` | string | no | Return merge requests with the given source branch. | | `target_branch` | string | no | Return merge requests with the given target branch. | | `search` | string | no | Search merge requests against their `title` and `description`. | | `wip` | string | no | Filter merge requests against their `wip` status. `yes` to return *only* draft merge requests, `no` to return *non-draft* merge requests. | | `not` | Hash | no | Return merge requests that do not match the parameters supplied. Accepts: `labels`, `milestone`, `author_id`, `author_username`, `assignee_id`, `assignee_username`, `reviewer_id`, `reviewer_username`, `my_reaction_emoji`. | +| `environment` | string | no | Returns merge requests deployed to the given environment. | ```json [ diff --git a/doc/api/suggestions.md b/doc/api/suggestions.md index 0a3e7d54f88..c174c0f5264 100644 --- a/doc/api/suggestions.md +++ b/doc/api/suggestions.md @@ -36,7 +36,7 @@ Example response: "id": 36, "from_line": 10, "to_line": 10, - "appliable": false, + "applicable": false, "applied": true, "from_content": " \"--talk-name=org.freedesktop.\",\n", "to_content": " \"--talk-name=org.free.\",\n \"--talk-name=org.desktop.\",\n" diff --git a/doc/architecture/blueprints/ci_data_decay/decomposition_partitioning_comparison.png b/doc/architecture/blueprints/ci_data_decay/decomposition_partitioning_comparison.png Binary files differnew file mode 100644 index 00000000000..47a53ad9be5 --- /dev/null +++ b/doc/architecture/blueprints/ci_data_decay/decomposition_partitioning_comparison.png diff --git a/doc/architecture/blueprints/ci_data_decay/pipeline_partitioning.md b/doc/architecture/blueprints/ci_data_decay/pipeline_partitioning.md new file mode 100644 index 00000000000..62a4e0f7b7d --- /dev/null +++ b/doc/architecture/blueprints/ci_data_decay/pipeline_partitioning.md @@ -0,0 +1,435 @@ +--- +stage: none +group: unassigned +comments: false +description: 'Pipeline data partitioning design' +--- + +# Pipeline data partitioning design + +_Disclaimer: The following contains information related to upcoming products, +features, and functionality._ + +_It is important to note that the information presented is for informational +purposes only. Please do not rely on this information for purchasing or +planning purposes._ + +_As with all projects, the items mentioned in this document and linked pages +are subject to change or delay. The development, release and timing of any +products, features, or functionality remain at the sole discretion of GitLab +Inc._ + +## What problem are we trying to solve? + +We want to partition the CI/CD dataset, because some of the database tables are +extremely large, which might be challenging in terms of scaling single node +reads, even after we ship the CI/CD database decomposition. + +We want to reduce the risk of database performance degradation by transforming +a few of the largest database tables into smaller ones using PostgreSQL +declarative partitioning. + +See more details about this effort in [the parent blueprint](index.md). + + + +## How are CI/CD data decomposition, partitioning, and time-decay related? + +CI/CD decomposition is an extraction of a CI/CD database cluster out of the +"main" database cluster, to make it possible to have a different primary +database receiving writes. The main benefit is doubling the capacity for writes +and data storage. The new database cluster will not have to serve reads / +writes for non-CI/CD database tables, so this offers some additional capacity +for reads too. + +CI/CD partitioning is dividing large CI/CD database tables into smaller ones. +This will improve reads capacity on every CI/CD database node, because it is +much less expensive to read data from small tables, than from large +multi-terabytes tables. We can add more CI/CD database replicas to better +handle the increase in the number of SQL queries that are reading data, but we +need partitioning to perform a single read more efficiently. Performance in +other aspects will improve too, because PostgreSQL will be more efficient in +maintaining multiple small tables than in maintaining a very large database +table. + +CI/CD time-decay allows us to benefit from the strong time-decay +characteristics of pipeline data. It can be implemented in many different ways, +but using partitioning to implement time-decay might be especially beneficial. +When implementing a time decay we usually mark data as archived, and migrate it +out of a database to a different place when data is no longer relevant or +needed. Our dataset is extremely large (tens of terabytes), so moving such a +high volume of data is challenging. When time-decay is implemented using +partitioning, we can archive the entire partition (or set of partitions) by +simply updating a single record in one of our database tables. It is one of the +least expensive ways to implement time-decay patterns at a database level. + + + +## Why do we need to partition CI/CD data? + +We need to partition CI/CD data because our database tables storing pipelines, +builds, and artifacts are too large. The `ci_builds` database table size is +currently around 2.5 TB with an index of around 1.4 GB. This is too much and +violates our [principle of 100 GB max size](../database_scaling/size-limits.md). +We also want to [build alerting](https://gitlab.com/gitlab-com/gl-infra/tamland/-/issues/5) +to notify us when this number is exceeded. + +We’ve seen numerous S1 and S2 database-related production environment +incidents, over the last couple of months, for example: + +- S1: 2022-03-17 [Increase in writes in `ci_builds` table](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6625) +- S1: 2021-11-22 [Excessive buffer read in replicas for `ci_job_artifacts`](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/5952) +- S2: 2022-04-12 [Transactions detected that have been running for more than 10m](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6821) +- S2: 2022-04-06 [Database contention plausibly caused by excessive `ci_builds` reads](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6773) +- S2: 2022-03-18 [Unable to remove a foreign key on `ci_builds`](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6642) + +We have approximately 50 `ci_*` prefixed database tables, and some of them +would benefit from partitioning. + +A simple SQL query to get this data: + +```sql +WITH tables AS (SELECT table_name FROM information_schema.tables WHERE table_name LIKE 'ci_%') + SELECT table_name, + pg_size_pretty(pg_total_relation_size(quote_ident(table_name))) AS total_size, + pg_size_pretty(pg_relation_size(quote_ident(table_name))) AS table_size, + pg_size_pretty(pg_indexes_size(quote_ident(table_name))) AS index_size, + pg_total_relation_size(quote_ident(table_name)) AS total_size_bytes + FROM tables ORDER BY total_size_bytes DESC; +``` + +See data from March 2022: + +| Table name | Total size | Index size | +|-------------------------|------------|------------| +| `ci_builds` | 3.5 TB | 1 TB | +| `ci_builds_metadata` | 1.8 TB | 150 GB | +| `ci_job_artifacts` | 600 GB | 300 GB | +| `ci_pipelines` | 400 GB | 300 GB | +| `ci_stages` | 200 GB | 120 GB | +| `ci_pipeline_variables` | 100 GB | 20 GB | +| (...around 40 more) | | | + +Based on the table above, it is clear that there are tables with a lot of +stored data. + +While we have almost 50 CI/CD-related database tables, we are initially +interested in partitioning only 6 of them. We can start by partitioning the +most interesting tables in an iterative way, but we also should have a strategy +for partitioning the remaining ones if needed. This document is an attempt to +capture this strategy, describe as many details as possible, to share this +knowledge among engineering teams. + +## How do we want to partition CI/CD data? + +We want to partition the CI/CD tables in iterations. It might not be feasible +to partition all of the 6 initial tables at once, so an iterative strategy +might be necessary. We also want to have a strategy for partitioning the +remaining database tables when it becomes necessary. + +It is also important to avoid large data migrations. We store almost 6 +terabytes of data in the biggest CI/CD tables, in many different columns and +indexes. Migrating this amount of data might be challenging and could cause +instability in the production environment. Due to this concern, we’ve developed +a way to attach an existing database table as a partition zero without downtime +and excessive database locking, what has been demonstrated in one of the +[first proofs of concept](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/80186). +This makes creation of a partitioned schema possible without a downtime (for +example using a routing table `p_ci_pipelines`), by attaching an existing +`ci_pipelines` table as partition zero without exclusive locking. It will be +possible to use the legacy table as usual, but we can create the next partition +when needed and the `p_ci_pipelines` table will be used for routing queries. To +use the routing table we need to find a good partitioning key. + +Our plan is to use logical partition IDs. We want to start with the +`ci_pipelines` table and create a `partition_id` column with a `DEFAULT` value +of `100` or `1000`. Using a `DEFAULT` value avoids the challenge of backfilling +this value for every row. Adding a `CHECK` constraint prior to attaching the +first partition tells PostgreSQL that we’ve already ensured consistency and +there is no need to check it while holding an exclusive table lock when +attaching this table as a partition to the routing table (partitioned schema +definition). We will increment this value every time we create a new partition +for `p_ci_pipelines`, and the partitioning strategy will be `LIST` +partitioning. + +We will also create a `partition_id` column in the other initial 6 database +tables we want to iteratively partition. After a new pipeline is created, it +will get a `partition_id` assigned, and all the related resources, like builds +and artifacts, will share the same value. We want to add the `partition_id` +column into all 6 problematic tables because we can avoid backfilling this data +when we decide it is time to start partitioning them. + +We want to partition CI/CD data iteratively, so we will start with the +pipelines table, and create at least one, but likely two, partitions. The +pipelines table will be partitioned using the `LIST` partitioning strategy. It +is possible that, after some time, `p_ci_pipelines` will store data in two +partitions with IDs of `100` and `101`. Then we will try partitioning +`ci_builds`. Therefore we might want to use `RANGE` partitioning in +`p_ci_builds` with IDs `100` and `101`, because builds for the two logical +partitions used will still be stored in a single table. + +Physical partitioning and logical partitioning will be separated, and a +strategy will be determined when we implement partitioning for the respective +database tables. Using `RANGE` partitioning works similarly to using `LIST` +partitioning in database tables other than `ci_pipelines`, but because we can +guarantee continuity of `partition_id` values, using `RANGE` partitioning might +be a better strategy. + +## Why do we want to use explicit logical partition ids? + +Partitioning CI/CD data using a logical `partition_id` has several benefits. We +could partition by a primary key, but this would introduce much more complexity +and additional cognitive load required to understand how the data is being +structured and stored in partitions. + +CI/CD data is hierarchical data. Stages belong to pipelines, builds belong to +stages, artifacts belong to builds (with rare exceptions). We are designing a +partitioning strategy that reflects this hierarchy, to reduce the complexity +and therefore cognitive load for contributors. With an explicit `partition_id` +associated with a pipeline, we can cascade the partition ID number when trying +to retrieve all resources associated with a pipeline. We know that for a +pipeline `12345` with a `partition_id` of `102`, we are always able to find +associated resources in logical partitions with number `102` in other routing +tables, and PostgreSQL will know in which partitions these records are being +stored in for every table. + +Another interesting benefit for using a single and incremental latest +`partition_id` number, associated with pipelines, is that in theory we can +cache it in Redis or in memory to avoid excessive reads from the database to +find this number, though we might not need to do this. + +The single and uniform `partition_id` value for pipeline data gives us more +choices later on than primary-keys-based partitioning. + +## Splitting large partitions into smaller ones + +We want to start with the initial `pipeline_id` number `100` (or higher, like +`1000`, depending on our calculations and estimations). We do not want to start +from 1, because existing tables are also large already, and we might want to +split them into smaller partitions. If we start with `100`, we will be able to +create partitions for `partition_id` of `1`, `20`, `45`, and move existing +records there by updating `partition_id` from `100` to a smaller number. + +PostgreSQL will move these records into their respective partitions in a +consistent way, provided that we do it in a transaction for all pipeline +resources at the same time. If we ever decide to split large partitions into +smaller ones (it's not yet clear if we will need to do this), we might be able +to just use background migrations to update partition IDs, and PostgreSQL is +smart enough to move rows between partitions on its own. + +## Storing partitions metadata in the database + +In order to build an efficient mechanism that will be responsible for creating +new partitions, and to implement time decay we want to introduce a partitioning +metadata table, called `ci_partitions`. In that table we would store metadata +about all the logical partitions, with many pipelines per partition. We may +need to store a range of pipeline ids per logical partition. Using it we will +be able to find the `partition_id` number for a given pipeline ID and we will +also find information about which logical partitions are “active” or +“archived”, which will help us to implement a time-decay pattern using database +declarative partitioning. + +`ci_partitions` table will store information about a partition identifier, +pipeline ids range it is valid for and whether the partitions have been +archived or not. Additional columns with timestamps may be helpful too. + +## Implementing a time-decay pattern using partitioning + +We can use `ci_partitions` to implement a time-decay pattern using declarative +partitioning. By telling PostgreSQL which logical partitions are archived we +can stop reading from these partitions using a SQL query like the one below. + +```sql +SELECT * FROM ci_builds WHERE partition_id IN ( + SELECT id FROM ci_partitions WHERE active = true +); +``` + +This query will make it possible to limit the number of partitions we will read +from, and therefore will cut access to "archived" pipeline data, using our data +retention policy for CI/CD data. Ideally we do not want to read from more than +two partitions at once, so we need to align the automatic partitioning +mechanisms with the time-decay policy. We will still need to implement new +access patterns for the archived data, presumably through the API, but the cost +of storing archived data in PostgreSQL will be reduced significantly this way. + +There are some technical details here that are out of the scope of this +description, but by using this strategy we can "archive" data, and make it much +less expensive to reside in our PostgreSQL cluster by simply toggling a boolean +column value. + +## Accessing partitioned data + +It will be possible to access partitioned data whether it has been archived or +not, in most places in GitLab. On a merge request page, we will always show +pipeline details even if the merge request was created years ago. We can do +that because `ci_partitions` will be a lookup table associating a pipeline ID +with its `partition_id`, and we will be able to find the partition that the +pipeline data is stored in. + +We will need to constrain access to searching through pipelines, builds, +artifacts etc. Search can not be done through all partitions, as it would not +be efficient enough, hence we will need to find a better way of searching +through archived pipelines data. It will be necessary to have different access +patterns to access archived data in the UI and API. + +There are a few challenges in enforcing usage of the `partition_id` +partitioning key in PostgreSQL. To make it easier to update our application to +support this, we have designed a new queries analyzer in our +[proof of concept merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/80186). +It helps to find queries that are not using the partitioning key. + +In a [separate proof of concept merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/84071) +and [related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/357090) we +demonstrated that using the uniform `partition_id` makes it possible to extend +Rails associations with an additional scope modifier so we can provide the +partitioning key in the SQL query. + +Using instance dependent associations, we can easily append a partitioning key +to SQL queries that are supposed to retrieve associated pipeline resources, for +example: + +```ruby +has_many :builds, -> (pipeline) { where(partition_id: pipeline.partition_id) } +``` + +The problem with this approach is that it makes preloading much more difficult +as instance dependent associations can not be used with preloads: + +```plaintext +ArgumentError: The association scope 'builds' is instance dependent (the +scope block takes an argument). Preloading instance dependent scopes is not +supported. +``` + +We also need to build a proof of concept for removing data on the PostgreSQL +side (using foreign keys with `ON DELETE CASCADE`) and removing data through +Rails associations, as this might be an important area of uncertainty. + +We need to [better understand](https://gitlab.com/gitlab-org/gitlab/-/issues/360148) +how unique constraints we are currently using will perform when using the +partitioned schema. + +We have also designed a query analyzer that makes it possible to detect direct +usage of zero partitions, legacy tables that have been attached as first +partitions to routing tables, to ensure that all queries are targeting +partitioned schema or partitioned routing tables, like `p_ci_pipelines`. + +## Why not partition using the project or namespace ID? + +We do not want to partition using `project_id` or `namespace_id` because +sharding and podding is a different problem to solve, on a different layer of +the application. It doesn't solve the original problem statement of performance +growing worse over time as we build up infrequently read data. We may want to +introduce pods in the future, and that might become the primary mechanism of +separating data based on the group or project the data is associated with. + +In theory we could use either `project_id` or `namespace_id` as a second +partitioning dimension, but this would add more complexity to a problem that is +already very complex. + +## Partitioning builds queuing tables + +We also want to partition our builds queuing tables. We currently have two: +`ci_pending_builds` and `ci_running_builds`. These tables are different from +other CI/CD data tables, as there are business rules in our product that make +all data stored in them invalid after 24 hours. + +As a result, we will need to use a different strategy to partition those +database tables, by removing partitions entirely after these are older than 24 +hours, and always reading from two partitions through a routing table. The +strategy to partition these tables is well understood, but requires a solid +Ruby-based automation to manage the creation and deletion of these partitions. +To achieve that we will collaborate with the Database team to adapt +[existing database partitioning tools](../../../development/database/table_partitioning.md) +to support CI/CD data partitioning. + +## Iterating to reduce the risk + +This strategy should reduce the risk of implementing CI/CD partitioning to +acceptable levels. We are also focusing on implementing partitioning for +reading only two partitions initially to make it possible to detach zero +partitions in case of problems in our production environment. Every iteration +phase, described below has a revert strategy and before shipping database +changes we want to test them in our benchmarking environment. + +The main way of reducing risk in case of this effort is iteration and making +things reversible. Shipping changes, described in this document, in a safe and +reliable way is our priority. + +## Iterations + +We want to focus on Phase 1 iteration first. The goal and the main objective of +this iteration is to partition the biggest 6 CI/CD database tables into 6 +routing tables (partitioned schema) and 12 partitions. This will leave our +Rails SQL queries mostly unchanged, but it will also make it possible to +perform emergency detachment of "zero partitions" if there is a database +performance degradation. This will cut users off their old data, but the +application will remain up and running, which is a better alternative to +application-wide outage. + +1. **Phase 0**: Build CI/CD data partitioning strategy: Done. ✅ +1. **Phase 1**: Partition the 6 biggest CI/CD database tables. + 1. Create partitioned schemas for all 6 database tables. + 1. Design a way to cascade `partition_id` to all partitioned resources. + 1. Implement initial query analyzers validating that we target routing tables. + 1. Attach zero partitions to the partitioned database tables. + 1. Update the application to target routing tables and partitioned tables. + 1. Measure the performance and efficiency of this solution. + + **Revert strategy**: Switch back to using concrete partitions instead of routing tables. + +1. **Phase 2**: Add a partitioning key to add SQL queries targeting partitioned tables. + 1. Implement query analyzer to check if queries targeting partitioned tables + are using proper partitioning keys. + 1. Modify existing queries to make sure that all of them are using a + partitioning key as a filter. + + **Revert strategy**: Use feature flags, query by query. + +1. **Phase 3**: Build new partitioned data access patterns. + 1. Build a new API or extend an existing one to allow access to data stored in + partitions that are supposed to be excluded based on the time-decay data + retention policy. + + **Revert strategy**: Feature flags. + +1. **Phase 4**: Introduce time-decay mechanisms built on top of partitioning. + 1. Build time-decay policy mechanisms. + 1. Enable the time-decay strategy on GitLab.com. +1. **Phase 5**: Introduce mechanisms for creating partitions automatically. + 1. Make it possible to create partitions in an automatic way. + 1. Deliver the new architecture to self-managed instances. + +## Conclusions + +We want to build a solid strategy for partitioning CI/CD data. We are aware of +the fact that it is difficult to iterate on this design, because a mistake made +in managing the database schema of our multi-terabyte PostgreSQL instance might +not be easily reversible without potential downtime. That is the reason we are +spending a significant amount of time to research and refine our partitioning +strategy. The strategy, described in this document, is subject to iteration as +well. Whenever we find a better way to reduce the risk and improve our plan, we +should update this document as well. + +We’ve managed to find a way to avoid large-scale data migrations, and we are +building an iterative strategy for partitioning CI/CD data. We documented our +strategy here to share knowledge and solicit feedback from other team members. + +## Who + +Authors: + +<!-- vale gitlab.Spelling = NO --> + +| Role | Who | +|--------|----------------| +| Author | Grzegorz Bizon | + +Recommenders: + +| Role | Who | +|------------------------|-----------------| +| Distingiushed Engineer | Kamil Trzciński | + +<!-- vale gitlab.Spelling = YES --> diff --git a/doc/architecture/blueprints/database/scalability/patterns/index.md b/doc/architecture/blueprints/database/scalability/patterns/index.md index dadf3634407..4a9bb003763 100644 --- a/doc/architecture/blueprints/database/scalability/patterns/index.md +++ b/doc/architecture/blueprints/database/scalability/patterns/index.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments comments: false diff --git a/doc/architecture/blueprints/database/scalability/patterns/read_mostly.md b/doc/architecture/blueprints/database/scalability/patterns/read_mostly.md index 02b56841507..75fcf10f048 100644 --- a/doc/architecture/blueprints/database/scalability/patterns/read_mostly.md +++ b/doc/architecture/blueprints/database/scalability/patterns/read_mostly.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments comments: false diff --git a/doc/architecture/blueprints/database/scalability/patterns/time_decay.md b/doc/architecture/blueprints/database/scalability/patterns/time_decay.md index b4614cde9d4..7399fd8048d 100644 --- a/doc/architecture/blueprints/database/scalability/patterns/time_decay.md +++ b/doc/architecture/blueprints/database/scalability/patterns/time_decay.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments comments: false diff --git a/doc/development/adding_database_indexes.md b/doc/development/adding_database_indexes.md index 0f30734c04b..b44a2861735 100644 --- a/doc/development/adding_database_indexes.md +++ b/doc/development/adding_database_indexes.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/cached_queries.md b/doc/development/cached_queries.md index bfa518ff7bf..b0bf7c7b6f5 100644 --- a/doc/development/cached_queries.md +++ b/doc/development/cached_queries.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Memory info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/creating_enums.md b/doc/development/creating_enums.md index 1f04f4c9712..a5a642eed86 100644 --- a/doc/development/creating_enums.md +++ b/doc/development/creating_enums.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/database/add_foreign_key_to_existing_column.md b/doc/development/database/add_foreign_key_to_existing_column.md index bfd455ef9da..9842814816f 100644 --- a/doc/development/database/add_foreign_key_to_existing_column.md +++ b/doc/development/database/add_foreign_key_to_existing_column.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/database/avoiding_downtime_in_migrations.md b/doc/development/database/avoiding_downtime_in_migrations.md index 57d8a1fab78..e7c57ca242b 100644 --- a/doc/development/database/avoiding_downtime_in_migrations.md +++ b/doc/development/database/avoiding_downtime_in_migrations.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/database/background_migrations.md b/doc/development/database/background_migrations.md index 80ba0336bda..7519f7276b6 100644 --- a/doc/development/database/background_migrations.md +++ b/doc/development/database/background_migrations.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/database/batched_background_migrations.md b/doc/development/database/batched_background_migrations.md index b466b63e9c6..c05ffe2cbf5 100644 --- a/doc/development/database/batched_background_migrations.md +++ b/doc/development/database/batched_background_migrations.md @@ -1,6 +1,6 @@ --- type: reference, dev -stage: Enablement +stage: Data Stores group: Database info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" --- diff --git a/doc/development/database/client_side_connection_pool.md b/doc/development/database/client_side_connection_pool.md index 60c8665df87..dc52a551407 100644 --- a/doc/development/database/client_side_connection_pool.md +++ b/doc/development/database/client_side_connection_pool.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/database/constraint_naming_convention.md b/doc/development/database/constraint_naming_convention.md index a22ddc1551c..72f16c20559 100644 --- a/doc/development/database/constraint_naming_convention.md +++ b/doc/development/database/constraint_naming_convention.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/database/database_lab.md b/doc/development/database/database_lab.md index 1c8694b113d..5346df2690d 100644 --- a/doc/development/database/database_lab.md +++ b/doc/development/database/database_lab.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/database/database_migration_pipeline.md b/doc/development/database/database_migration_pipeline.md index ce7e1801abc..496bd09bf1d 100644 --- a/doc/development/database/database_migration_pipeline.md +++ b/doc/development/database/database_migration_pipeline.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/database/database_reviewer_guidelines.md b/doc/development/database/database_reviewer_guidelines.md index ca9ca36b156..c8ab1e7353a 100644 --- a/doc/development/database/database_reviewer_guidelines.md +++ b/doc/development/database/database_reviewer_guidelines.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/database/dbcheck-migrations-job.md b/doc/development/database/dbcheck-migrations-job.md index af72e28a875..49f8b183272 100644 --- a/doc/development/database/dbcheck-migrations-job.md +++ b/doc/development/database/dbcheck-migrations-job.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/database/deleting_migrations.md b/doc/development/database/deleting_migrations.md index be9009f365d..8354cb62d0c 100644 --- a/doc/development/database/deleting_migrations.md +++ b/doc/development/database/deleting_migrations.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/database/efficient_in_operator_queries.md b/doc/development/database/efficient_in_operator_queries.md index 2503be826ea..a8104c3d7e9 100644 --- a/doc/development/database/efficient_in_operator_queries.md +++ b/doc/development/database/efficient_in_operator_queries.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/database/index.md b/doc/development/database/index.md index 0363d13ed4c..b427f54ff3c 100644 --- a/doc/development/database/index.md +++ b/doc/development/database/index.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/database/keyset_pagination.md b/doc/development/database/keyset_pagination.md index 88928feb927..d87d09d82bc 100644 --- a/doc/development/database/keyset_pagination.md +++ b/doc/development/database/keyset_pagination.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/database/layout_and_access_patterns.md b/doc/development/database/layout_and_access_patterns.md index a3e2fefb2a3..99a50b503aa 100644 --- a/doc/development/database/layout_and_access_patterns.md +++ b/doc/development/database/layout_and_access_patterns.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/database/loose_foreign_keys.md b/doc/development/database/loose_foreign_keys.md index ef6d13d3cfc..6889b9123ca 100644 --- a/doc/development/database/loose_foreign_keys.md +++ b/doc/development/database/loose_foreign_keys.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/database/maintenance_operations.md b/doc/development/database/maintenance_operations.md index 9e7a35531ca..85df185c024 100644 --- a/doc/development/database/maintenance_operations.md +++ b/doc/development/database/maintenance_operations.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/database/migrations_for_multiple_databases.md b/doc/development/database/migrations_for_multiple_databases.md index 023b28ce6a6..2500071f4cf 100644 --- a/doc/development/database/migrations_for_multiple_databases.md +++ b/doc/development/database/migrations_for_multiple_databases.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/database/multiple_databases.md b/doc/development/database/multiple_databases.md index c622d4f50ff..324d1a6c6ef 100644 --- a/doc/development/database/multiple_databases.md +++ b/doc/development/database/multiple_databases.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Sharding info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -597,3 +597,21 @@ way to replace cascading deletes so we don't end up with orphaned data or records that point to nowhere, which might lead to bugs. As such we created ["loose foreign keys"](loose_foreign_keys.md) which is an asynchronous process of cleaning up orphaned records. + +## Locking writes on the tables that don't belong to the database schemas + +When the CI database is promoted and the two databases are fully split, +as an extra safeguard against creating a split brain situation, +run the Rake task `gitlab:db:lock_writes`. This command locks writes on: + +- The `gitlab_main` tables on the CI Database. +- The `gitlab_ci` tables on the Main Database. + +This Rake task adds triggers to all the tables, to prevent any +`INSERT`, `UPDATE`, `DELETE`, or `TRUNCATE` statements from running +against the tables that need to be locked. + +If this task was run against a GitLab setup that uses only a single database +for both `gitlab_main` and `gitlab_ci` tables, then no tables will be locked. + +To undo the operation, run the opposite Rake task: `gitlab:db:unlock_writes`. diff --git a/doc/development/database/not_null_constraints.md b/doc/development/database/not_null_constraints.md index af7d569e282..947ac6cd875 100644 --- a/doc/development/database/not_null_constraints.md +++ b/doc/development/database/not_null_constraints.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/database/pagination_guidelines.md b/doc/development/database/pagination_guidelines.md index b5cbae9ad05..d70a34519fb 100644 --- a/doc/development/database/pagination_guidelines.md +++ b/doc/development/database/pagination_guidelines.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/database/pagination_performance_guidelines.md b/doc/development/database/pagination_performance_guidelines.md index 90e4faf2de7..7a0c6131e2c 100644 --- a/doc/development/database/pagination_performance_guidelines.md +++ b/doc/development/database/pagination_performance_guidelines.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/database/post_deployment_migrations.md b/doc/development/database/post_deployment_migrations.md index 799eefdb875..a49c77ca047 100644 --- a/doc/development/database/post_deployment_migrations.md +++ b/doc/development/database/post_deployment_migrations.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/database/rename_database_tables.md b/doc/development/database/rename_database_tables.md index 7a76c028042..44f95b1d277 100644 --- a/doc/development/database/rename_database_tables.md +++ b/doc/development/database/rename_database_tables.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/database/setting_multiple_values.md b/doc/development/database/setting_multiple_values.md index 0f23aae9f79..cba15a73430 100644 --- a/doc/development/database/setting_multiple_values.md +++ b/doc/development/database/setting_multiple_values.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/database/strings_and_the_text_data_type.md b/doc/development/database/strings_and_the_text_data_type.md index 7aa529e1518..d764e54aa76 100644 --- a/doc/development/database/strings_and_the_text_data_type.md +++ b/doc/development/database/strings_and_the_text_data_type.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/database/table_partitioning.md b/doc/development/database/table_partitioning.md index 34cb73978bc..51f2c443047 100644 --- a/doc/development/database/table_partitioning.md +++ b/doc/development/database/table_partitioning.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/database/transaction_guidelines.md b/doc/development/database/transaction_guidelines.md index 2806bd217db..503a713443a 100644 --- a/doc/development/database/transaction_guidelines.md +++ b/doc/development/database/transaction_guidelines.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/database_debugging.md b/doc/development/database_debugging.md index e749fd6453b..c734799cdc4 100644 --- a/doc/development/database_debugging.md +++ b/doc/development/database_debugging.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/database_query_comments.md b/doc/development/database_query_comments.md index e4133633a77..9cfa5540c83 100644 --- a/doc/development/database_query_comments.md +++ b/doc/development/database_query_comments.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/database_review.md b/doc/development/database_review.md index ceb8c9e5906..2f4b7ca7c10 100644 --- a/doc/development/database_review.md +++ b/doc/development/database_review.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/db_dump.md b/doc/development/db_dump.md index 0c63bf06e07..f2076cbc410 100644 --- a/doc/development/db_dump.md +++ b/doc/development/db_dump.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/documentation/versions.md b/doc/development/documentation/versions.md index 81b9108364e..067c37d30aa 100644 --- a/doc/development/documentation/versions.md +++ b/doc/development/documentation/versions.md @@ -132,7 +132,7 @@ To remove a page: ```markdown --- - stage: Enablement + stage: Data Stores group: Global Search info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments remove_date: '2022-08-02' diff --git a/doc/development/elasticsearch.md b/doc/development/elasticsearch.md index 7c67b3495ba..3745a9e50ad 100644 --- a/doc/development/elasticsearch.md +++ b/doc/development/elasticsearch.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Global Search info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/foreign_keys.md b/doc/development/foreign_keys.md index c20c70623ae..7709197a43a 100644 --- a/doc/development/foreign_keys.md +++ b/doc/development/foreign_keys.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/graphql_guide/batchloader.md b/doc/development/graphql_guide/batchloader.md index 0e90f89ff7a..4cdf379fe5e 100644 --- a/doc/development/graphql_guide/batchloader.md +++ b/doc/development/graphql_guide/batchloader.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/hash_indexes.md b/doc/development/hash_indexes.md index 881369e429b..731639b6f06 100644 --- a/doc/development/hash_indexes.md +++ b/doc/development/hash_indexes.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/image_scaling.md b/doc/development/image_scaling.md index e1ffbdb766a..93575429369 100644 --- a/doc/development/image_scaling.md +++ b/doc/development/image_scaling.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Memory info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/insert_into_tables_in_batches.md b/doc/development/insert_into_tables_in_batches.md index cd659a3d19b..c8bb4ce1c6d 100644 --- a/doc/development/insert_into_tables_in_batches.md +++ b/doc/development/insert_into_tables_in_batches.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments description: "Sometimes it is necessary to store large amounts of records at once, which can be inefficient diff --git a/doc/development/iterating_tables_in_batches.md b/doc/development/iterating_tables_in_batches.md index 8813fe560db..384b19cf618 100644 --- a/doc/development/iterating_tables_in_batches.md +++ b/doc/development/iterating_tables_in_batches.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/ordering_table_columns.md b/doc/development/ordering_table_columns.md index 00ce15fcc10..42e5588e010 100644 --- a/doc/development/ordering_table_columns.md +++ b/doc/development/ordering_table_columns.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/query_performance.md b/doc/development/query_performance.md index bc1f753c012..87e41c78e19 100644 --- a/doc/development/query_performance.md +++ b/doc/development/query_performance.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/query_recorder.md b/doc/development/query_recorder.md index 17f2fecc1bc..263bcf66fad 100644 --- a/doc/development/query_recorder.md +++ b/doc/development/query_recorder.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/serializing_data.md b/doc/development/serializing_data.md index 48e756d015b..76e9dc0617e 100644 --- a/doc/development/serializing_data.md +++ b/doc/development/serializing_data.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/service_ping/metrics_instrumentation.md b/doc/development/service_ping/metrics_instrumentation.md index ff567e3b1b3..b2332241c8a 100644 --- a/doc/development/service_ping/metrics_instrumentation.md +++ b/doc/development/service_ping/metrics_instrumentation.md @@ -293,7 +293,7 @@ There is support for: There is no support for: -- `add`, `sum`, `histogram` for database metrics. +- `add`, `histogram` for database metrics. You can [track the progress to support these](https://gitlab.com/groups/gitlab-org/-/epics/6118). diff --git a/doc/development/single_table_inheritance.md b/doc/development/single_table_inheritance.md index 0783721e628..c8d082e8a67 100644 --- a/doc/development/single_table_inheritance.md +++ b/doc/development/single_table_inheritance.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/sql.md b/doc/development/sql.md index 4b6153b7205..17250992588 100644 --- a/doc/development/sql.md +++ b/doc/development/sql.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/swapping_tables.md b/doc/development/swapping_tables.md index cb038a3b85a..d6c5b8f0662 100644 --- a/doc/development/swapping_tables.md +++ b/doc/development/swapping_tables.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/understanding_explain_plans.md b/doc/development/understanding_explain_plans.md index e06ece38135..3fc071bc5ff 100644 --- a/doc/development/understanding_explain_plans.md +++ b/doc/development/understanding_explain_plans.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/verifying_database_capabilities.md b/doc/development/verifying_database_capabilities.md index bda9c68eae5..9433e9d5f78 100644 --- a/doc/development/verifying_database_capabilities.md +++ b/doc/development/verifying_database_capabilities.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/integration/elasticsearch.md b/doc/integration/elasticsearch.md index ded1449682d..64d229b39b8 100644 --- a/doc/integration/elasticsearch.md +++ b/doc/integration/elasticsearch.md @@ -1,6 +1,6 @@ --- type: reference -stage: Enablement +stage: Data Stores group: Global Search info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/update/mysql_to_postgresql.md b/doc/update/mysql_to_postgresql.md index 40dc57c554b..8ac2791c82b 100644 --- a/doc/update/mysql_to_postgresql.md +++ b/doc/update/mysql_to_postgresql.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/update/upgrading_postgresql_using_slony.md b/doc/update/upgrading_postgresql_using_slony.md index f2bbc8d7558..d3102ca4591 100644 --- a/doc/update/upgrading_postgresql_using_slony.md +++ b/doc/update/upgrading_postgresql_using_slony.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/admin_area/monitoring/background_migrations.md b/doc/user/admin_area/monitoring/background_migrations.md index b666c0c5ad2..53d5056bb65 100644 --- a/doc/user/admin_area/monitoring/background_migrations.md +++ b/doc/user/admin_area/monitoring/background_migrations.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/permissions.md b/doc/user/permissions.md index d429312161e..de2bf0a8eb6 100644 --- a/doc/user/permissions.md +++ b/doc/user/permissions.md @@ -270,6 +270,7 @@ More details about the permissions for some project-level features follow. | Cancel and retry jobs | | | | ✓ | ✓ | ✓ | | Create new [environments](../ci/environments/index.md) | | | | ✓ | ✓ | ✓ | | Delete job logs or job artifacts | | | | ✓ (*4*) | ✓ | ✓ | +| Run CI/CD pipeline | | | | ✓ | ✓ | ✓ | | Run CI/CD pipeline for a protected branch | | | | ✓ (*5*) | ✓ (*5*) | ✓ | | Stop [environments](../ci/environments/index.md) | | | | ✓ | ✓ | ✓ | | View a job with [debug logging](../ci/variables/index.md#debug-logging) | | | | ✓ | ✓ | ✓ | diff --git a/doc/user/search/advanced_search.md b/doc/user/search/advanced_search.md index 5435a9d027c..4caef55a0b3 100644 --- a/doc/user/search/advanced_search.md +++ b/doc/user/search/advanced_search.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Global Search info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" type: reference diff --git a/doc/user/search/global_search/advanced_search_syntax.md b/doc/user/search/global_search/advanced_search_syntax.md index 219531070b3..9daa7afd6de 100644 --- a/doc/user/search/global_search/advanced_search_syntax.md +++ b/doc/user/search/global_search/advanced_search_syntax.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Global Search info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- |
