diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2023-12-05 06:13:51 +0300 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2023-12-05 06:13:51 +0300 |
commit | c20abe491ceeb1b8e1350f18f0c91e0a31c73cb5 (patch) | |
tree | b8f9c5888b0ec18b24eb49f790a253ddc7905f1a /doc | |
parent | 35850d3d1643a9e2c5328f45b8a9026ac3d86e35 (diff) |
Add latest changes from gitlab-org/gitlab@master
Diffstat (limited to 'doc')
-rw-r--r-- | doc/administration/auditor_users.md | 2 | ||||
-rw-r--r-- | doc/administration/backup_restore/index.md | 168 | ||||
-rw-r--r-- | doc/administration/gitaly/configure_gitaly.md | 47 | ||||
-rw-r--r-- | doc/administration/settings/gitaly_timeouts.md | 70 | ||||
-rw-r--r-- | doc/development/mass_insert.md | 2 | ||||
-rw-r--r-- | doc/user/compliance/compliance_center/index.md | 17 | ||||
-rw-r--r-- | doc/user/project/repository/monorepos/index.md | 2 |
7 files changed, 241 insertions, 67 deletions
diff --git a/doc/administration/auditor_users.md b/doc/administration/auditor_users.md index 0355f7a3661..f57d09f79b8 100644 --- a/doc/administration/auditor_users.md +++ b/doc/administration/auditor_users.md @@ -35,7 +35,7 @@ To create a new user account with auditor access (or change an existing user): To create a user account with auditor access: 1. On the left sidebar, at the bottom, select **Admin Area**. -1. On the left sidebar, select **Overview > Users**. +1. Select **Overview > Users**. 1. Create a new user or edit an existing one. Set **Access Level** to **Auditor**. 1. If you created a user, select **Create user**. For an existing user, select **Save changes**. diff --git a/doc/administration/backup_restore/index.md b/doc/administration/backup_restore/index.md index cf7c5c3db9a..d8ee58e42d7 100644 --- a/doc/administration/backup_restore/index.md +++ b/doc/administration/backup_restore/index.md @@ -36,6 +36,174 @@ methods to export or back up your data yourself from GitLab.com. Issues are stored in the database, and can't be stored in Git itself. +## GitLab backup archive creation process + +When working with GitLab backups, you might need to know how GitLab creates backup archives. To create backup archives, GitLab: + +1. If creating an incremental backup, extracts the previous backup archive and read its `backup_information.yml` file. +1. Updates or generates the `backup_information.yml` file. +1. Runs all backup sub-tasks: + - `db` to backup the GitLab PostgreSQL database (not Gitaly Cluster). + - `repositories` to back up Git repositories. + - `uploads` to back up attachments. + - `builds` to back up CI job output logs. + - `artifacts` to back up CI job artifacts. + - `pages` to back up page content. + - `lfs` to back up LFS objects. + - `terraform_state` to back up Terraform states. + - `registry` to back up container registry images. + - `packages` to back up packages. + - `ci_secure_files` to back up project-level secure files. +1. Archives the backup staging area into a tar file. +1. Optional. Uploads the new backup archive to object-storage. +1. Cleans up backup staging directory files that are now archived. + +## Backup staging directory + +The backup staging directory is a place to temporarily: + +- Store backup artifacts on disk before the GitLab backup archive is created. +- Extract backup archives on disk before restoring a backup or creating an incremental backup. + +This directory is the same directory where completed GitLab backup archives are created. When creating an untarred backup, the backup artifacts are left in this directory and no +archive is created. + +Example backup staging directory with untarred backup: + +```plaintext +backups/ +├── 1701728344_2023_12_04_16.7.0-pre_gitlab_backup.tar +├── 1701728447_2023_12_04_16.7.0-pre_gitlab_backup.tar +├── artifacts.tar.gz +├── backup_information.yml +├── builds.tar.gz +├── ci_secure_files.tar.gz +├── db +│ ├── ci_database.sql.gz +│ └── database.sql.gz +├── lfs.tar.gz +├── packages.tar.gz +├── pages.tar.gz +├── repositories +│ ├── manifests/ +│ ├── @hashed/ +│ └── @snippets/ +├── terraform_state.tar.gz +└── uploads.tar.gz +``` + +## `backup_information.yml` file + +The `backup_information.yml` file saves all backup inputs that are not included in the backup itself. It includes information such as: + +- The time the backup was created. +- The version of GitLab that generated the backup. +- Any options that were specified, such as skipped sub-tasks. + +This information is used by some sub-tasks to determine how: + +- To restore. +- To link data in the backup with external services (such as server-side repository backups). + +This file is saved into the backup staging directory. + +## Database backups + +Database backups are created and restored by a GitLab backup sub-task called `db`. The database sub-task uses `pg_dump` to create [a SQL dump](https://www.postgresql.org/docs/14/backup-dump.html). The output of `pg_dump` is piped through `gzip` in order to create a compressed SQL file. This file is saved to the backup staging directory. + +## Repository backups + +Repository backups are created and restored by a GitLab backup sub-task called `repositories`. The repositories sub-task uses a Gitaly command +[`gitaly-backup`](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/gitaly-backup.md) to create Git repository backups: + +- GitLab uses its database to tell `gitaly-backup` which repositories to back up. +- `gitaly-backup` then calls a series of RPCs on Gitaly to collect the repository backup data for each repository. This data is streamed into a directory structure in the GitLab backup staging directory. + +```mermaid +sequenceDiagram + box Backup host + participant Repositories sub-task + participant gitaly-backup + end + + Repositories sub-task->>+gitaly-backup: List of repositories + + loop Each repository + gitaly-backup->>+Gitaly: ListRefs request + Gitaly->>-gitaly-backup: List of Git references + + gitaly-backup->>+Gitaly: CreateBundleFromRefList request + Gitaly->>-gitaly-backup: Git bundle file + + gitaly-backup->>+Gitaly: GetCustomHooks request + Gitaly->>-gitaly-backup: Custom hooks archive + end + + gitaly-backup->>-Repositories sub-task: Success/failure +``` + +Storages configured to Gitaly Cluster are backed up the same as standalone Gitaly. When Gitaly Cluster receives the RPC calls from `gitaly-backup`, it is responsible for +rebuilding its own database. This means that there is no need to backup the Gitaly Cluster database separately. Because backups operate through RPCs, each repository is only backed +up once no matter the replication factor. + +### Server-side repository backups + +You can configure repository backups as server-side repository backups. When specified, `gitaly-backup` makes a single RPC call for each repository to create the backup. This RPC +does not transmit any repository data. Instead, the RPC triggers the Gitaly node that stores that physical repository to upload the backup data directly to object-storage. Because +the data is no longer transmitted through RPCs from Gitaly, server-side backups require much less network transfer and require no disk storage on the machine that is running the +backup Rake task. The backups stored on object-storage are linked to the created backup archive by [the backup name](backup_gitlab.md#backup-timestamp). + +```mermaid +sequenceDiagram + box Backup host + participant Repositories sub-task + participant gitaly-backup + end + + Repositories sub-task->>+gitaly-backup: List of repositories + + loop Each repository + gitaly-backup->>+Gitaly: BackupRepository request + + Gitaly->>+Object-storage: Git references file + Object-storage->>-Gitaly: Success/failure + + Gitaly->>+Object-storage: Git bundle file + Object-storage->>-Gitaly: Success/failure + + Gitaly->>+Object-storage: Custom hooks archive + Object-storage->>-Gitaly: Success/failure + + Gitaly->>+Object-storage: Backup manifest file + Object-storage->>-Gitaly: Success/failure + + Gitaly->>-gitaly-backup: Success/failure + end + + gitaly-backup->>-Repositories sub-task: Success/failure +``` + +## File backups + +The following GitLab backup sub-tasks back up files: + +- `uploads` +- `builds` +- `artifacts` +- `pages` +- `lfs` +- `terraform_state` +- `registry` +- `packages` +- `ci_secure_files` + +These file sub-tasks determine a set of files within a directory specific to the task. This set of files is then passed to `tar` +to create an archive. This archive is piped (not saved to disk) through `gzip` to save a compressed tar file to the backup staging directory. + +Because backups are created from live instances, the files that tar is trying to archive can sometimes be modified while creating the backup. In this case, an alternate "copy" +strategy can be used. When this strategy is used, `rsync` is first used to create a copy of the files to back up. Then, these copies are passed to `tar` as usual. In this case, +the machine running the backup Rake task must have enough storage for the copied files and the compressed archive. + ## Related features - [Geo](../geo/index.md) diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md index 891416a7deb..535e35cc012 100644 --- a/doc/administration/gitaly/configure_gitaly.md +++ b/doc/administration/gitaly/configure_gitaly.md @@ -1542,50 +1542,3 @@ go_cloud_url = "s3://gitaly-backups?region=minio&endpoint=my.minio.local:8080&di ``` ::EndTabs - -## Configure negotiation timeouts - -> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/5574) in GitLab 16.5. - -Gitaly supports configurable negotiation timeouts. - -Negotiation timeouts can be configured for the `git-upload-pack(1)` and `git-upload-archive(1)` -operations, which are invoked by a Gitaly node when you execute the `git fetch` and -`git archive --remote` commands respectively. You might need to increase the negotiation timeout: - -- For particularly large repositories. -- When performing these commands in parallel. - -These timeouts affect only the [negotiation phase](https://git-scm.com/docs/pack-protocol/2.2.3#_packfile_negotiation) of -remote Git operations, not the entire transfer. - -Valid values for timeouts follow the format of [`ParseDuration`](https://pkg.go.dev/time#ParseDuration) in Go. - -How you configure negotiation timeouts depends on the type of installation you have: - -::Tabs - -:::TabTitle Linux package (Omnibus) - -Edit `/etc/gitlab/gitlab.rb`: - -```ruby -gitaly['configuration'] = { - timeout: { - upload_pack_negotiation: '10m', # 10 minutes - upload_archive_negotiation: '20m', # 20 minutes - } -} -``` - -:::TabTitle Self-compiled (source) - -Edit `/home/git/gitaly/config.toml`: - -```toml -[timeout] -upload_pack_negotiation = "10m" -upload_archive_negotiation = "20m" -``` - -::EndTabs diff --git a/doc/administration/settings/gitaly_timeouts.md b/doc/administration/settings/gitaly_timeouts.md index b60cf141cf6..93db0467d70 100644 --- a/doc/administration/settings/gitaly_timeouts.md +++ b/doc/administration/settings/gitaly_timeouts.md @@ -6,23 +6,75 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Gitaly timeouts **(FREE SELF)** -[Gitaly](../gitaly/index.md) timeouts are configurable. The timeouts can be -configured to make sure that long-running Gitaly calls don't needlessly take up resources. +[Gitaly](../gitaly/index.md) provides two types of configurable timeouts: -To access Gitaly timeout settings: +- Call timeouts, configured by using the GitLab UI. +- Negotiation timeouts, configured by using Gitaly configuration files. + +## Configure the call timeouts + +Configure the following call timeouts to make sure that long-running Gitaly calls don't needlessly take up resources. To +configure the call timeouts: 1. On the left sidebar, at the bottom, select **Admin Area**. 1. Select **Settings > Preferences**. 1. Expand the **Gitaly timeouts** section. +1. Set each timeout as required. -## Available timeouts +### Available call timeouts -The following timeouts are available. +Different call timeouts are available for different Gitaly operations. | Timeout | Default | Description | |:--------|:-----------|:------------| -| Default | 55 seconds | Timeout for most Gitaly calls (not enforced for `git` `fetch` and `push` operations, or Sidekiq jobs). For example, checking if a repository exists on disk. Makes sure that Gitaly calls made within a web request cannot exceed the entire request timeout. It should be shorter than the [worker timeout](../operations/puma.md#change-the-worker-timeout) that can be configured for [Puma](../../install/requirements.md#puma-settings). If a Gitaly call timeout exceeds the worker timeout, the remaining time from the worker timeout is used to avoid having to terminate the worker. | -| Fast | 10 seconds | Timeout for fast Gitaly operations used within requests, sometimes multiple times. For example, checking if a repository exists on disk. If fast operations exceed this threshold, there may be a problem with a storage shard. Failing fast can help maintain the stability of the GitLab instance. | -| Medium | 30 seconds | Timeout for Gitaly operations that should be fast (possibly within requests) but preferably not used multiple times within a request. For example, loading blobs. Timeout that should be set between Default and Fast. | +| Default | 55 seconds | Timeout for most Gitaly calls (not enforced for `git` `fetch` and `push` operations, or Sidekiq jobs). For example, checking if a repository exists on disk. Makes sure that Gitaly calls made in a web request cannot exceed the entire request timeout. It should be shorter than the [worker timeout](../operations/puma.md#change-the-worker-timeout) that can be configured for [Puma](../../install/requirements.md#puma-settings). If a Gitaly call timeout exceeds the worker timeout, the remaining time from the worker timeout is used to avoid having to terminate the worker. | +| Fast | 10 seconds | Timeout for fast Gitaly operations used in requests, sometimes multiple times. For example, checking if a repository exists on disk. If fast operations exceed this threshold, there may be a problem with a storage shard. Failing fast can help maintain the stability of the GitLab instance. | +| Medium | 30 seconds | Timeout for Gitaly operations that should be fast (possibly in requests) but preferably not used multiple times in a request. For example, loading blobs. Timeout that should be set between Default and Fast. | + +## Configure the negotiation timeouts + +> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/5574) in GitLab 16.5. + +You might need to increase the negotiation timeout: + +- For particularly large repositories. +- When performing these commands in parallel. + +You can configure negotiation timeouts for: + +- `git-upload-pack(1)`, which is invoked by a Gitaly node when you execute `git fetch`. +- `git-upload-archive(1)`, which is invoked by a Gitaly node when you execute `git archive --remote`. + +To configure these timeouts: + +::Tabs + +:::TabTitle Linux package (Omnibus) + +Edit `/etc/gitlab/gitlab.rb`: + +```ruby +gitaly['configuration'] = { + timeout: { + upload_pack_negotiation: '10m', # 10 minutes + upload_archive_negotiation: '20m', # 20 minutes + } +} +``` + +:::TabTitle Self-compiled (source) + +Edit `/home/git/gitaly/config.toml`: + +```toml +[timeout] +upload_pack_negotiation = "10m" +upload_archive_negotiation = "20m" +``` + +::EndTabs + +For the values, use the format of [`ParseDuration`](https://pkg.go.dev/time#ParseDuration) in Go. -You can also [configure negotiation timeouts](../gitaly/configure_gitaly.md#configure-negotiation-timeouts). +These timeouts affect only the [negotiation phase](https://git-scm.com/docs/pack-protocol/2.2.3#_packfile_negotiation) of +remote Git operations, not the entire transfer. diff --git a/doc/development/mass_insert.md b/doc/development/mass_insert.md index e16f162abfa..412639927a7 100644 --- a/doc/development/mass_insert.md +++ b/doc/development/mass_insert.md @@ -15,5 +15,5 @@ the following snippet in the rails console. ```ruby u = User.find(1) -Project.last(100).each { |p| p.set_timestamps_for_create && p.add_maintainer(u, current_user: u) } # Change 100 to whatever number of projects you need access to +Project.last(100).each { |p| p.send(:set_timestamps_for_create) && p.add_maintainer(u, current_user: u) } # Change 100 to whatever number of projects you need access to ``` diff --git a/doc/user/compliance/compliance_center/index.md b/doc/user/compliance/compliance_center/index.md index 0f2cce67721..df1883f95d9 100644 --- a/doc/user/compliance/compliance_center/index.md +++ b/doc/user/compliance/compliance_center/index.md @@ -188,8 +188,8 @@ To export a report of merge request compliance violations for projects in a grou 1. On the left sidebar, select **Search or go to** and find your group. 1. On the left sidebar, select **Secure > Compliance center**. -1. On the page, select the **Violations** tab. -1. On the Violations tab, select the **Export full report as CSV** action in the top right corner +1. Select the **Export** action in the top right corner +1. Select **Export violations report** A report is compiled and delivered to your email inbox as an attachment. @@ -235,7 +235,8 @@ To generate the Chain of Custody report: 1. On the left sidebar, select **Search or go to** and find your group. 1. On the left sidebar, select **Secure > Compliance center**. -1. Select **List of all merge commits**. +1. Select the **Export** action in the top right corner +1. Select **Export chain of custody report** Depending on your version of GitLab, the Chain of Custody report is either sent through email or available for download. @@ -251,9 +252,9 @@ To generate a commit-specific Chain of Custody report: 1. On the left sidebar, select **Search or go to** and find your group. 1. On the left sidebar, select **Secure > Compliance center**. -1. At the top of the compliance report, to the right of **List of all commits**, select the down arrow - (**{chevron-lg-down}**). -1. Enter the commit SHA, and then select **Export commit custody report**. +1. Select the **Export** action in the top right corner +1. Select **Export custody report of a specific commit** +1. Enter the commit SHA, and then select **Export custody report**. Depending on your version of GitLab, the Chain of Custody report is either sent through email or available for download. @@ -358,8 +359,8 @@ To export a report of compliance frameworks on projects in a group: 1. On the left sidebar, select **Search or go to** and find your group. 1. On the left sidebar, select **Secure > Compliance center**. -1. On the page, select the **Projects** tab. -1. On the Frameworks tab, select the **Export as CSV** action in the top right corner +1. Select the **Export** action in the top right corner +1. Select **Export list of project frameworks** A report is compiled and delivered to your email inbox as an attachment. diff --git a/doc/user/project/repository/monorepos/index.md b/doc/user/project/repository/monorepos/index.md index 144f46cd7d5..b73757a952e 100644 --- a/doc/user/project/repository/monorepos/index.md +++ b/doc/user/project/repository/monorepos/index.md @@ -189,7 +189,7 @@ You might experience a `fatal: the remote end hung up unexpectedly` error when a - The same large repository in parallel. You can attempt to mitigate this issue by increasing the default negotiation timeout values. For more information, see -[Configure negotiation timeouts](../../../../administration/gitaly/configure_gitaly.md#configure-negotiation-timeouts). +[Configure the negotiation timeouts](../../../../administration/settings/gitaly_timeouts.md#configure-the-negotiation-timeouts). ## Optimize your repository |