diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2022-01-20 12:16:11 +0300 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2022-01-20 12:16:11 +0300 |
commit | edaa33dee2ff2f7ea3fac488d41558eb5f86d68c (patch) | |
tree | 11f143effbfeba52329fb7afbd05e6e2a3790241 /doc/raketasks/backup_restore.md | |
parent | d8a5691316400a0f7ec4f83832698f1988eb27c1 (diff) |
Add latest changes from gitlab-org/gitlab@14-7-stable-eev14.7.0-rc42
Diffstat (limited to 'doc/raketasks/backup_restore.md')
-rw-r--r-- | doc/raketasks/backup_restore.md | 245 |
1 files changed, 224 insertions, 21 deletions
diff --git a/doc/raketasks/backup_restore.md b/doc/raketasks/backup_restore.md index 676cc529c98..539908cd7bc 100644 --- a/doc/raketasks/backup_restore.md +++ b/doc/raketasks/backup_restore.md @@ -12,11 +12,11 @@ An application data backup creates an archive file that contains the database, all repositories and all attachments. You can only restore a backup to **exactly the same version and type (CE/EE)** -of GitLab on which it was created. The best way to migrate your repositories -from one server to another is through a backup and restore. +of GitLab on which it was created. The best way to [migrate your projects +from one server to another](#migrate-to-a-new-server) is through a backup and restore. WARNING: -GitLab doesn't back up items that aren't stored in the file system. If you're +GitLab doesn't back up items that aren't stored on the file system. If you're using [object storage](../administration/object_storage.md), be sure to enable backups with your object storage provider, if desired. @@ -58,16 +58,17 @@ including: - CI/CD job output logs - CI/CD job artifacts - LFS objects +- Terraform states ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/331806) in GitLab 14.7) - Container Registry images - GitLab Pages content +- Packages ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332006) in GitLab 14.7) - Snippets - [Group wikis](../user/project/wiki/group.md) Backups do not include: -- [Terraform state files](../administration/terraform_state.md) -- [Package registry files](../administration/packages/index.md) - [Mattermost data](https://docs.mattermost.com/administration/config-settings.html#file-storage) +- Redis (and thus Sidekiq jobs) WARNING: GitLab does not back up any configuration files (`/etc/gitlab`), TLS keys and certificates, or system @@ -115,12 +116,8 @@ the host, based on your installed version of GitLab: If you're using the [GitLab Helm chart](https://gitlab.com/gitlab-org/charts/gitlab) on a Kubernetes cluster, you can run the backup task by using `kubectl` to run the `backup-utility` -script on the GitLab task runner pod. For more details, see -[backing up a GitLab installation](https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/backup-restore/backup.md#backing-up-a-gitlab-installation). - -```shell -kubectl exec -it <gitlab task-runner pod> -- backup-utility -``` +script on the GitLab toolbox pod. For more details, see the +[charts backup documentation](https://docs.gitlab.com/charts/backup-restore/backup.html). Similar to the Kubernetes case, if you have scaled out your GitLab cluster to use multiple application servers, you should pick a designated node (that isn't @@ -276,9 +273,11 @@ You can exclude specific directories from the backup by adding the environment v - `builds` (CI job output logs) - `artifacts` (CI job artifacts) - `lfs` (LFS objects) +- `terraform_state` (Terraform states) - `registry` (Container Registry images) - `pages` (Pages content) - `repositories` (Git repositories data) +- `packages` (Packages) All wikis are backed up as part of the `repositories` group. Non-existent wikis are skipped during a backup. @@ -985,7 +984,7 @@ your installation is using PgBouncer, for either performance reasons or when usi Next, restore `/etc/gitlab/gitlab-secrets.json` if necessary, [as previously mentioned](#restore-prerequisites). -Reconfigure, restart and check GitLab: +Reconfigure, restart and [check](../administration/raketasks/maintenance.md#check-gitlab-configuration) GitLab: ```shell sudo gitlab-ctl reconfigure @@ -993,7 +992,7 @@ sudo gitlab-ctl restart sudo gitlab-rake gitlab:check SANITIZE=true ``` -In GitLab 13.1 and later, check [database values can be decrypted](../administration/raketasks/doctor.md) +In GitLab 13.1 and later, check [database values can be decrypted](../administration/raketasks/check.md#verify-database-values-can-be-decrypted-using-the-current-secrets) especially if `/etc/gitlab/gitlab-secrets.json` was restored, or if a different server is the target for the restore. @@ -1001,6 +1000,14 @@ the target for the restore. sudo gitlab-rake gitlab:doctor:secrets ``` +For added assurance, you can perform [an integrity check on the uploaded files](../administration/raketasks/check.md#uploaded-files-integrity): + +```shell +sudo gitlab-rake gitlab:artifacts:check +sudo gitlab-rake gitlab:lfs:check +sudo gitlab-rake gitlab:uploads:check +``` + ### Restore for Docker image and GitLab Helm chart installations For GitLab installations using the Docker image or the GitLab Helm chart on a @@ -1182,7 +1189,7 @@ has a longer discussion explaining the potential problems. To prevent writes to the Git repository data, there are two possible approaches: -- Use [maintenance mode](../administration/maintenance_mode/index.md) to place GitLab in a read-only state. +- Use [maintenance mode](../administration/maintenance_mode/index.md) **(PREMIUM SELF)** to place GitLab in a read-only state. - Create explicit downtime by stopping all Gitaly services before backing up the repositories: ```shell @@ -1284,6 +1291,198 @@ sudo GITLAB_BACKUP_PGHOST=192.168.1.10 GITLAB_BACKUP_PGPORT=5432 /opt/gitlab/bin See the [PostgreSQL documentation](https://www.postgresql.org/docs/12/libpq-envars.html) for more details on what these parameters do. +## Migrate to a new server + +<!-- some details borrowed from GitLab.com move from Azure to GCP detailed at https://gitlab.com/gitlab-com/migration/-/blob/master/.gitlab/issue_templates/failover.md --> + +You can use GitLab backup and restore to migrate your instance to a new server. This section outlines a typical procedure for a GitLab deployment running on a single server. +If you're running GitLab Geo, an alternative option is [Geo disaster recovery for planned failover](../administration/geo/disaster_recovery/planned_failover.md). + +WARNING: +Avoid uncoordinated data processing by both the new and old servers, where multiple +servers could connect concurrently and process the same data. For example, when using +[incoming email](../administration/incoming_email.md), if both GitLab instances are +processing email at the same time, then both instances will end up missing some data. +This type of problem can occur with other services as well, such as a +[non-packaged database](https://docs.gitlab.com/omnibus/settings/database.html#using-a-non-packaged-postgresql-database-management-server), +a non-packaged Redis instance, or non-packaged Sidekiq. + +Prerequisites: + +- Some time before your migration, consider notifying your users of upcoming + scheduled maintenance with a [broadcast message banner](../user/admin_area/broadcast_messages.md). +- Ensure your backups are complete and current. Create a complete system-level backup, or + take a snapshot of all servers involved in the migration, in case destructive commands + (like `rm`) are run incorrectly. + +### Prepare the new server + +To prepare the new server: + +1. Copy the + [SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079) + from the old server to avoid man-in-the-middle attack warnings. +1. [Install and configure GitLab](https://about.gitlab.com/install) except + [incoming email](../administration/incoming_email.md): + 1. Install GitLab. + 1. Configure by copying `/etc/gitlab` files from the old server to the new server, and update as necessary. + Read the + [Omnibus configuration backup and restore instructions](https://docs.gitlab.com/omnibus/settings/backups.html) for more detail. + 1. If applicable, disable [incoming email](../administration/incoming_email.md). + 1. Block new CI/CD jobs from starting upon initial startup after the backup and restore. + Edit `/etc/gitlab/gitlab.rb` and set the following: + + ```ruby + nginx['custom_gitlab_server_config'] = "location /api/v4/jobs/request {\n deny all;\n return 503;\n}\n" + ``` + + 1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. Stop GitLab to avoid any potential unnecessary and unintentional data processing: + + ```shell + sudo gitlab-ctl stop + ``` + +1. Configure the new server to allow receiving the Redis database and GitLab backup files: + + ```shell + sudo rm -f /var/opt/gitlab/redis/dump.rdb + sudo chown <your-linux-username> /var/opt/gitlab/redis + sudo mkdir /var/opt/gitlab/backups + sudo chown <your-linux-username> /var/opt/gitlab/backups + ``` + +### Prepare and transfer content from the old server + +1. Ensure you have an up-to-date system-level backup or snapshot of the old server. +1. Enable [maintenance mode](../administration/maintenance_mode/index.md) **(PREMIUM SELF)**, + if supported by your GitLab edition. +1. Block new CI/CD jobs from starting: + 1. Edit `/etc/gitlab/gitlab.rb`, and set the following: + + ```ruby + nginx['custom_gitlab_server_config'] = "location /api/v4/jobs/request {\n deny all;\n return 503;\n}\n" + ``` + + 1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. Disable periodic background jobs: + 1. On the top bar, select **Menu > Admin**. + 1. On the left sidebar, select **Monitoring > Background Jobs**. + 1. Under the Sidekiq dashboard, select **Cron** tab and then + **Disable All**. +1. Wait for the currently running CI/CD jobs to finish, or accept that jobs that have not completed may be lost. + To view jobs currently running, on the left sidebar, select **Overviews > Jobs**, + and then select **Running**. +1. Wait for Sidekiq jobs to finish: + 1. On the left sidebar, select **Monitoring > Background Jobs**. + 1. Under the Sidekiq dashboard, select **Queues** and then **Live Poll**. + Wait for **Busy** and **Enqueued** to drop to 0. + These queues contain work that has been submitted by your users; + shutting down before these jobs complete may cause the work to be lost. + Make note of the numbers shown in the Sidekiq dashboard for post-migration verification. +1. Flush the Redis database to disk, and stop GitLab other than the services needed for migration: + + ```shell + sudo /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket save && sudo gitlab-ctl stop && sudo gitlab-ctl start postgresql + ``` + +1. Create a GitLab backup: + + ```shell + sudo gitlab-backup create + ``` + +1. Disable the following GitLab services and prevent unintentional restarts by adding the following to the bottom of `/etc/gitlab/gitlab.rb`: + + ```ruby + alertmanager['enable'] = false + gitlab_exporter['enable'] = false + gitlab_pages['enable'] = false + gitlab_workhorse['enable'] = false + grafana['enable'] = false + logrotate['enable'] = false + gitlab_rails['incoming_email_enabled'] = false + nginx['enable'] = false + node_exporter['enable'] = false + postgres_exporter['enable'] = false + postgresql['enable'] = false + prometheus['enable'] = false + puma['enable'] = false + redis['enable'] = false + redis_exporter['enable'] = false + registry['enable'] = false + sidekiq['enable'] = false + ``` + +1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. Verify everything is stopped, and confirm no services are running: + + ```shell + sudo gitlab-ctl status + ``` + +1. Transfer the Redis database and GitLab backups to the new server: + + ```shell + sudo scp /var/opt/gitlab/redis/dump.rdb <your-linux-username>@new-server:/var/opt/gitlab/redis + sudo scp /var/opt/gitlab/backups/your-backup.tar <your-linux-username>@new-server:/var/opt/gitlab/backups + ``` + +### Restore data on the new server + +1. Restore appropriate file system permissions: + + ```shell + sudo chown gitlab-redis /var/opt/gitlab/redis + sudo chown gitlab-redis:gitlab-redis /var/opt/gitlab/redis/dump.rdb + sudo chown git:root /var/opt/gitlab/backups + sudo chown git:git /var/opt/gitlab/backups/your-backup.tar + ``` + +1. [Restore the GitLab backup](#restore-gitlab). +1. Verify that the Redis database restored correctly: + 1. On the top bar, select **Menu > Admin**. + 1. On the left sidebar, select **Monitoring > Background Jobs**. + 1. Under the Sidekiq dashboard, verify that the numbers + match with what was shown on the old server. + 1. While still under the Sidekiq dashboard, select **Cron** and then **Enable All** + to re-enable periodic background jobs. +1. Test that read-only operations on the GitLab instance work as expected. For example, browse through project repository files, merge requests, and issues. +1. Disable [Maintenance Mode](../administration/maintenance_mode/index.md) **(PREMIUM SELF)**, if previously enabled. +1. Test that the GitLab instance is working as expected. +1. If applicable, re-enable [incoming email](../administration/incoming_email.md) and test it is working as expected. +1. Update your DNS or load balancer to point at the new server. +1. Unblock new CI/CD jobs from starting by removing the custom NGINX config + you added previously: + + ```ruby + # The following line must be removed + nginx['custom_gitlab_server_config'] = "location /api/v4/jobs/request {\n deny all;\n return 503;\n}\n" + ``` + +1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. Remove the scheduled maintenance [broadcast message banner](../user/admin_area/broadcast_messages.md). + ## Additional notes This documentation is for GitLab Community and Enterprise Edition. We back up @@ -1308,9 +1507,11 @@ If you're using backup restore procedures, you may encounter the following warning messages: ```plaintext -psql:/var/opt/gitlab/backups/db/database.sql:22: ERROR: must be owner of extension plpgsql -psql:/var/opt/gitlab/backups/db/database.sql:2931: WARNING: no privileges could be revoked for "public" (two occurrences) -psql:/var/opt/gitlab/backups/db/database.sql:2933: WARNING: no privileges were granted for "public" (two occurrences) +ERROR: must be owner of extension pg_trgm +ERROR: must be owner of extension btree_gist +ERROR: must be owner of extension plpgsql +WARNING: no privileges could be revoked for "public" (two occurrences) +WARNING: no privileges were granted for "public" (two occurrences) ``` Be advised that the backup is successfully restored in spite of these warning @@ -1362,8 +1563,8 @@ Use the information in the following sections at your own risk. #### Verify that all values can be decrypted -You can determine if your database contains values that can't be decrypted by using the -[Secrets Doctor Rake task](../administration/raketasks/doctor.md). +You can determine if your database contains values that can't be decrypted by using a +[Rake task](../administration/raketasks/check.md#verify-database-values-can-be-decrypted-using-the-current-secrets). #### Take a backup @@ -1619,10 +1820,12 @@ There can be [risks when disabling released features](../administration/feature_flags.md#risks-when-disabling-released-features). Refer to this feature's version history for more details. -`gitaly-backup` is used by the backup Rake task to create and restore repository backups from Gitaly. +The `gitaly-backup` binary is used by the backup Rake task to create and restore repository backups from Gitaly. `gitaly-backup` replaces the previous backup method that directly calls RPCs on Gitaly from GitLab. -The backup Rake task must be able to find this executable. It can be configured in Omnibus GitLab packages: +The backup Rake task must be able to find this executable. In most cases, you don't need to change +the path to the binary as it should work fine with the default path `/opt/gitlab/embedded/bin/gitaly-backup`. +If you have a specific reason to change the path, it can be configured in Omnibus GitLab packages: 1. Add the following to `/etc/gitlab/gitlab.rb`: |