diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2023-06-16 21:09:42 +0300 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2023-06-16 21:09:42 +0300 |
commit | c44eade0d7b09c9d80c42aba6d1974ce9cd3146b (patch) | |
tree | da0710de0a519f225755737cf5ffd93c75d1da8f /doc | |
parent | 0c87da93750c6428328a3e3cd2ebd0882f6294e3 (diff) |
Add latest changes from gitlab-org/gitlab@master
Diffstat (limited to 'doc')
-rw-r--r-- | doc/ci/review_apps/index.md | 2 | ||||
-rw-r--r-- | doc/development/database/batched_background_migrations.md | 96 | ||||
-rw-r--r-- | doc/index.md | 29 |
3 files changed, 62 insertions, 65 deletions
diff --git a/doc/ci/review_apps/index.md b/doc/ci/review_apps/index.md index ae6df691edc..ddb8f0aaa61 100644 --- a/doc/ci/review_apps/index.md +++ b/doc/ci/review_apps/index.md @@ -16,7 +16,7 @@ Review apps: - Provide an automatic live preview of changes made in a feature branch by spinning up a dynamic environment for your merge requests. - Allow designers and product managers to see your changes without needing to check out your branch and run your changes in a sandbox environment. -- Are fully integrated with the [GitLab DevOps LifeCycle](../../index.md#the-entire-devops-lifecycle). +- Are fully integrated with the [GitLab DevOps LifeCycle](https://about.gitlab.com/stages-devops-lifecycle/). - Allow you to deploy your changes wherever you want. ![review apps workflow](img/continuous-delivery-review-apps.svg) diff --git a/doc/development/database/batched_background_migrations.md b/doc/development/database/batched_background_migrations.md index abc96664481..3e54a78757a 100644 --- a/doc/development/database/batched_background_migrations.md +++ b/doc/development/database/batched_background_migrations.md @@ -56,44 +56,66 @@ in the directory `lib/gitlab/background_migration/`. ### Execution mechanism -After being enqueued, batched background migrations are picked in sequential order by a Sidekiq worker. -At this point, batched jobs are created - or retried and iterate over a range of records that are fetched based -on the configuration defined inside the migration class. Each job instance receives a set of -arguments based on this migration configuration. After each job execution, the optimization mechanism takes -place to decide if the migration can be optimized. +Batched background migrations are picked from the queue in the order they are enqueued. Multiple migrations are fetched +and executed in parallel, as long they are in active state and do not target the same database table. +The default number of migrations processed in parallel is 2, for GitLab.com this limit is configured to 4. +Once migration is picked for execution, a job is created for the specific batch. After each job execution, migration's +batch size may be increased or decreased, based on the performance of the last 20 jobs. ```plantuml @startuml hide empty description skinparam ConditionEndStyle hline -start -rectangle Queue { - :Migration Three; - note right: Waiting for Migration Two - :Migration Two; - note right: Waiting for Migration One +left to right direction +rectangle "Batched Background Migration Queue" as migrations { + rectangle "Migration N (active)" as migrationn + rectangle "Migration 1 (completed)" as migration1 + rectangle "Migration 2 (active)" as migration2 + rectangle "Migration 3 (on hold)" as migration3 + rectangle "Migration 4 (active)" as migration4 + migration1 -[hidden]> migration2 + migration2 -[hidden]> migration3 + migration3 -[hidden]> migration4 + migration4 -[hidden]> migrationn +} +rectangle "Execution Workers" as workers { + rectangle "Execution Worker 1 (busy)" as worker1 + rectangle "Execution Worker 2 (available)" as worker2 + worker1 -[hidden]> worker2 } +migration2 --> [Scheduling Worker] +migration4 --> [Scheduling Worker] +[Scheduling Worker] --> worker2 +@enduml +``` + +Soon as a worker is available, the BBM is processed by the runner. + +```plantuml +@startuml +hide empty description +start rectangle Runner { - :Migration One; - note right: Active migration - if (Still have records to migrate?) then (Yes) - if (Have a pending batched job?) then (Yes) - :Retry batched job; - else (No) - :Create a batched job; - endif - :Evaluate DB health; - note right: Table autovacuum, Patroni Apdex, Write-ahead logging - if (Evaluation signs to stop?) then (Yes) - :Put migration on hold; + :Migration; + if (Have reached batching bounds?) then (Yes) + if (Have jobs to retry?) then (Yes) + :Fetch the batched job; else (No) - :Optimize migration; + :Finish active migration; + stop endif else (No) - :Finish active migration; + :Create a batched job; + endif + :Execute batched job; + :Evaluate DB health; + note right: Checks for table autovacuum, Patroni Apdex, Write-ahead logging + if (Evaluation signs to stop?) then (Yes) + :Put migration on hold; + else (No) + :Optimize migration; endif } -stop @enduml ``` @@ -161,13 +183,16 @@ the migration as `failed`) if any of the following is true: Because batched migrations are update heavy and there were few incidents in the past because of the heavy load from migrations while the database was underperforming, a throttling mechanism exists to mitigate them. -These database indicators are checked to throttle a migration. On getting a stop signal, the migration is paused for a set time (10 minutes): +These database indicators are checked to throttle a migration. On getting a +stop signal, the migration is paused for a set time (10 minutes): - WAL queue pending archival crossing a threshold. - Active autovacuum on the tables on which the migration works on. - Patroni apdex SLI dropping below the SLO. -It's an ongoing effort to add more indicators to further enhance the database health check framework, more details can be found in this [epic](https://gitlab.com/groups/gitlab-org/-/epics/7594). +It's an ongoing effort to add more indicators to further enhance the +database health check framework. For more details, see +[epic 7594](https://gitlab.com/groups/gitlab-org/-/epics/7594). ### Isolation @@ -348,21 +373,6 @@ class CopyColumnUsingBackgroundMigrationJob < BatchedMigrationJob end ``` -## Throttling batched migrations - -Because batched migrations are update heavy and there were few incidents in the past because of the heavy load from migrations while the database was underperforming, a throttling mechanism exists to mitigate them. - -These database indicators are checked to throttle a migration. On getting a -stop signal, the migration is paused for a set time (10 minutes): - -- WAL queue pending archival crossing a threshold. -- Active autovacuum on the tables on which the migration works on. -- Patroni apdex SLI dropping below the SLO. - -It's an ongoing effort to add more indicators to further enhance the -database health check framework. For more details, see -[epic 7594](https://gitlab.com/groups/gitlab-org/-/epics/7594). - ## Additional filters By default, when creating background jobs to perform the migration, batched background migrations diff --git a/doc/index.md b/doc/index.md index 654204048af..367f18ec159 100644 --- a/doc/index.md +++ b/doc/index.md @@ -19,16 +19,15 @@ description: 'Learn how to use and administer GitLab, the most scalable Git-base # GitLab Docs -Welcome to the GitLab documentation! +Explore the different areas of the documentation: | | | |:------------------------|:------------------------| | [**Use GitLab**](user/index.md)<br>Get started with GitLab features and functionality. | [**Administer GitLab**](administration/index.md)<br/>Administer a self-managed GitLab instance. | -| [**New to Git and GitLab?**](tutorials/index.md)<br/>Start learning about Git and GitLab. | [**Contribute to GitLab development**](#contributing-to-gitlab)<br/>Create new GitLab functionality and documentation. | +| [**New to Git and GitLab?**](tutorials/index.md)<br/>Start learning about Git and GitLab. | [**Contribute to GitLab development**](#contribute-to-gitlab)<br/>Create new GitLab functionality and documentation. | | [**Coming to GitLab from another platform?**](#coming-to-gitlab-from-another-platform)<br/>Learn how to move to GitLab. | [**Build an integration with GitLab**](#build-an-integration-with-gitlab)<br/>Integrate with Jira and other common applications. | -| [**Choose a subscription**](subscriptions/index.md)<br/>Determine which subscription tier makes sense for you. | [**Install GitLab**](https://about.gitlab.com/install/)<br/>Install GitLab on different platforms. | +| [**Choose a subscription**](subscriptions/index.md)<br/>Determine which subscription tier makes sense for you. | [**Install GitLab**](install/index.md)<br/>Install GitLab on different platforms. | | [**Reference architectures**](administration/reference_architectures/index.md)<br/>View recommended deployments at scale. | [**Upgrade GitLab**](update/index.md)<br/>Upgrade your GitLab self-managed instance to the latest version. | -| [**GitLab releases**](https://about.gitlab.com/releases/)<br/>See what's new in GitLab. | | ## Popular topics @@ -48,15 +47,7 @@ Have a look at some of our most popular topics: | [SSL configuration for Linux package installations](https://docs.gitlab.com/omnibus/settings/ssl/index.html) | SSL settings for self-managed instances installed using Linux packages. | | [GitLab.com settings](user/gitlab_com/index.md) | Settings used for GitLab.com. | -## The entire DevOps lifecycle - -GitLab is the first single application for software development, security, -and operations that enables [Concurrent DevOps](https://about.gitlab.com/topics/concurrent-devops/). -GitLab makes the software lifecycle faster and radically improves the speed of business. - -GitLab provides solutions for [each of the stages of the DevOps lifecycle](https://about.gitlab.com/stages-devops-lifecycle/). - -### User account +## User account For more information about GitLab account management, see: @@ -78,7 +69,7 @@ If you are coming to GitLab from another platform, the following information is ## Build an integration with GitLab -There are many ways to integrate with GitLab, including: +You can build integrations with GitLab: | Topic | Description | |:-------------------------------------------|:------------| @@ -86,15 +77,11 @@ There are many ways to integrate with GitLab, including: | [GitLab GraphQL API](api/graphql/index.md) | Integrate with GitLab using our GraphQL API. | | [Integrations](integration/index.md) | Integrations with third-party products. | -## Contributing to GitLab - -GitLab Community Edition is [open source](https://gitlab.com/gitlab-org/gitlab-foss/) -and GitLab Enterprise Edition is [open-core](https://gitlab.com/gitlab-org/gitlab/). +## Contribute to GitLab Learn how to contribute to GitLab with the following resources: | Topic | Description | |:------------------------------------------------------------|:------------| -| [Development](development/index.md) | How to contribute to GitLab development. | -| [Legal](legal/index.md) | Contributor license agreements. | -| [Writing documentation](development/documentation/index.md) | How to contribute to GitLab Docs. | +| [Contribute to GitLab development](development/index.md). | How to contribute to GitLab development. | +| [Contribute to the documentation](development/documentation/index.md) | How to contribute to GitLab documentation. | |