diff options
Diffstat (limited to 'doc/administration')
74 files changed, 2047 insertions, 2003 deletions
diff --git a/doc/administration/admin_area.md b/doc/administration/admin_area.md index ff18ac8ff3a..f9b26cd364d 100644 --- a/doc/administration/admin_area.md +++ b/doc/administration/admin_area.md @@ -470,3 +470,12 @@ The content of each log file is listed in chronological order. To minimize perfo ### Audit Events **(PREMIUM SELF)** The **Audit Events** page lists changes made within the GitLab server. With this information you can control, analyze, and track every change. + +### Statistics + +The **Instance overview** section of the Dashboard lists the current statistics of the GitLab instance. This information is retrieved using the [Application statistics API](../api/statistics.md#get-current-application-statistics). + +NOTE: +These statistics show exact counts for values less than 10,000. For values of 10,000 and higher, these statistics show approximate data +when [TablesampleCountStrategy](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/database/count/tablesample_count_strategy.rb?ref_type=heads#L16) and [ReltuplesCountStrategy](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/database/count/reltuples_count_strategy.rb?ref_type=heads) strategies are used for calculations. +. diff --git a/doc/administration/analytics/dev_ops_reports.md b/doc/administration/analytics/dev_ops_reports.md index 057dc5d48ee..313e99c1e57 100644 --- a/doc/administration/analytics/dev_ops_reports.md +++ b/doc/administration/analytics/dev_ops_reports.md @@ -49,18 +49,18 @@ feature is available. DevOps Adoption shows feature adoption for development, security, and operations. -| Category | Feature | -| --- | --- | -| Development | Approvals<br>Code owners<br>Issues<br>Merge requests | -| Security | DAST<br>Dependency Scanning<br>Fuzz Testing<br>SAST | -| Operations | Deployments<br>Pipelines<br>Runners | +| Category | Feature | +|-------------|---------| +| Development | Approvals<br>Code owners<br>Issues<br>Merge requests | +| Security | DAST<br>Dependency Scanning<br>Fuzz Testing<br>SAST | +| Operations | Deployments<br>Pipelines<br>Runners | You can use Group DevOps Adoption to: - Identify specific subgroups that are lagging in their adoption of GitLab features, so you can guide them on -their DevOps journey. + their DevOps journey. - Find subgroups that have adopted certain features, and provide guidance to other subgroups on -how to use those features. + how to use those features. - Verify if you are getting the return on investment that you expected from GitLab. ## Add or remove a group diff --git a/doc/administration/appearance.md b/doc/administration/appearance.md index 3599c444134..9ebc9a37407 100644 --- a/doc/administration/appearance.md +++ b/doc/administration/appearance.md @@ -6,107 +6,145 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GitLab Appearance **(FREE SELF)** -Several options are available for customizing the appearance of a self-managed instance -of GitLab. To access these settings: +You can update your settings to change the look and feel of your GitLab self-managed instance. + +To open the **Appearance** settings: 1. On the left sidebar, at the bottom, select **Admin Area**. 1. Select **Settings > Appearance**. -## Navigation bar +## Customize your homepage button + +Customize the appearance of your **Homepage** button. + +The **Homepage** button is located on the upper-left corner of the left sidebar. +Replace the default **GitLab logo** **{tanuki}** with any image. + +- The file should be less than 1 MB. +- The image should be 28 pixels high. Images more than 28 px high will be resized. + +To customize your **Homepage** icon image: -By default, the navigation bar has the GitLab logo, but this can be customized with -any image desired. It is optimized for images 28px high (any width), but any image can be -used (less than 1 MB) and it is automatically resized. +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Settings > Appearance**. +1. Under **Navigation bar**, select **Choose file**. +1. At the bottom of the page, select **Update appearance settings**. -After you select and upload an image, select **Update appearance settings** at the bottom -of the page to activate it in the GitLab instance. +Pipeline status emails also show your custom logo. However, some email applications do not support SVG images. If your custom image is in SVG format, pipeline emails show the default logo. -NOTE: -GitLab pipeline emails also display the custom logo, unless the logo is in SVG format. If the -custom logo is in SVG format, the default logo is used instead because the SVG format is not -supported by many email clients. +## Customize the favicon -## Favicon +Customize the appearance of the favicon. A favicon is the icon for a website that shows in your browser tabs. The **GitLab logo** **{tanuki}** is the default browser and CI/CD status favicon. Replace the default icon with any image that is `32 x 32` pixels and in `.png` or `.ico` format. -By default, the favicon (used by the browser as the tab icon and the CI status icon) -uses the GitLab logo. This can be customized with any icon desired. It must be a -32x32 `.png` or `.ico` image. +To change the favicon: -After you select and upload an icon, select **Update appearance settings** at the bottom -of the page to activate it in the GitLab instance. +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Settings > Appearance**. +1. Under **Favicon**, select **Choose file**. +1. At the bottom of the page, select **Update appearance settings**. -## System header and footer messages +## Add system header and footer messages > **Enable header and footer in emails** checkbox [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/344819) in GitLab 15.9. -You can add a small header message, a small footer message, or both, to the interface -of your GitLab instance. These messages appear on all projects and pages of the -instance, including the sign-in/sign-up page. The default color is white text on -an orange background, but this can be customized by selecting **Customize colors**. +Add a small header message, a small footer message, or both, to the interface of your GitLab instance. These messages show on all projects and pages of the instance, such as the sign-in and register pages. -Limited [Markdown](../user/markdown.md) is supported, such as bold, italics, and links, for -example. Other Markdown features, including lists, images, and quotes are not supported -as the header and footer messages can only be a single line. +- You can italicize, bold, or add links to your message with Markdown. +- Markdown lists, images, and quotes are not supported because system messages must be a single line. -You can select **Enable header and footer in emails** to have the text of -the header and footer added to all emails sent by the GitLab instance. +To add a system header, footer message, or both: -After you add a message, select **Update appearance settings** at the bottom of the page -to activate it in the GitLab instance. +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Settings > Appearance**. +1. Go to the **System header and footer** section. +1. Complete the fields. +1. Optional. Select the **Enable header and footer in emails** checkbox. Add your system messages to all emails sent by your GitLab instance. +1. At the bottom of the page, select **Update appearance settings**. -## Sign-in / Sign-up pages +By default, the system header and footer text is white text on an orange background. To customize the message colors: -You can replace the default message on the sign-in/sign-up page with your own message -and logo. You can make full use of [Markdown](../user/markdown.md) in the description. +- Go to the **System header and footer** section and select **Customize colors**. -The optimal size for the logo is 128 x 128 pixels, but any image can be used (below 1 MB) -and it is resized automatically. The logo image appears between the title and -the description, on the left of the sign-up page. +## Customize your sign-in and register pages -After you add a message, select **Update appearance settings** at the bottom of the page -to activate it in the GitLab instance. You can also select **Sign-in page**, -to review the saved appearance settings: +Customize the title, description, and logo on the sign-in and register page. By default, the register page logo is located on the left of the page, between the title and the description. -NOTE: -You can add also add a [customized help message](settings/help_page.md) below the sign-in message or add [a Sign-in text message](settings/sign_in_restrictions.md#sign-in-information). +To customize your sign-in and register page titles or descriptions: -## Progressive Web App +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Settings > Appearance**. +1. Go to the **Sign in/Sign up pages** section. +1. Complete the fields. You can format the page **Title** and **Description** with Markdown. +1. At the bottom of the page, select **Update appearance settings**. -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/375708) in GitLab 15.9. +To customize the logo on your sign-in and register pages: -GitLab can be installed as a [Progressive Web App](https://developer.mozilla.org/en-US/docs/Web/Progressive_web_apps) (PWA). -Use the Progressive Web App settings to customize its appearance, including its name, -description, and icon. +- The file should be less than 1 MB. +- The image should be 128 pixels high. Images more than 128 px high will be resized. -### Configure the PWA settings +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Settings > Appearance**. +1. Go to the **Sign in/Sign up pages** section. +1. Under **Logo**, select **Choose file**. +1. At the bottom of the page, select **Update appearance settings**. -To configure the PWA settings: +You can add also add a [customized help message](settings/help_page.md) below the sign-in message or add [a sign-in text message](settings/sign_in_restrictions.md#sign-in-information). + +## Customize the Progressive Web App + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/375708) in GitLab 15.9. + +Customize the icon, display name, short name, and description for your Progessive Web App (PWA). For more information, see [Progressive Web App](https://developer.mozilla.org/en-US/docs/Web/Progressive_web_apps). + +To add a Progressive Web App name and short name: 1. On the left sidebar, at the bottom, select **Admin Area**. 1. Select **Settings > Appearance**. -1. Scroll to the **Progressive Web App (PWA)** section. +1. Go to the **Progressive Web App (PWA)** section. 1. Complete the fields. - - **Icon**: If you use the standard GitLab icon, it is available in sizes 192x192 pixels, - 512x512 pixels, also as a maskable icon. If you use a custom icon, it must be in either size - 192x192 pixels, or 512x512 pixels. -1. Select **Update appearance settings**. + - **Name** is the display name of your PWA. + - **Short name** shows on mobile devices and small screens. +1. At the bottom of the page, select **Update appearance settings**. + +To add a Progressive Web App description: + +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Settings > Appearance**. +1. Go to the **Progressive Web App (PWA)** section. +1. Complete the fields. You can format the **Description** with Markdown. +1. At the bottom of the page, select **Update appearance settings**. + +To customize your Progressive Web App icon: + +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Settings > Appearance**. +1. Go to the **Progressive Web App (PWA)** section. +1. Under **Icon**, select **Choose file**. +1. At the bottom of the page, select **Update appearance settings**. + +## Add guidelines to the new project page -## New project pages +Add a guideline message to the **New project page**. You can format your message with Markdown. The guideline message shows under the **New Project** message and, on the left side of the **New project page**. -You can add a new project guidelines message to the **New project page** in GitLab. -You can make full use of [Markdown](../user/markdown.md) in the description: +To add a guideline message to the **New project page**: -The message is displayed below the **New Project** message, on the left side -of the **New project page**. +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Settings > Appearance**. +1. Go to the **New project pages** section. +1. Complete the fields. You can format your guidelines with Markdown. -After you add a message, select **Update appearance settings** at the bottom of the page -to activate it in the GitLab instance. You can also select **New project page**, -which brings you to the new project page so you can review the change. +## Add profile image guidelines + +Add guidelines for profile images. + +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Settings > Appearance**. +1. Go to the **Profile image guideline** section. +1. Complete the fields. You can format your text with Markdown. ## Libravatar -[Libravatar](https://www.libravatar.org) is supported by GitLab for avatar images, but you must -[manually enable Libravatar support on the GitLab instance](../administration/libravatar.md) to use the service. +GitLab supports [Libravatar](https://www.libravatar.org) is for avatar images, but you must manually enable Libravatar support on the GitLab instance. For more information, see [Libravatar](../administration/libravatar.md) to use the service. <!-- ## Troubleshooting diff --git a/doc/administration/audit_event_streaming/audit_event_types.md b/doc/administration/audit_event_streaming/audit_event_types.md index 28cab04553a..ff99fc1f6c6 100644 --- a/doc/administration/audit_event_streaming/audit_event_types.md +++ b/doc/administration/audit_event_streaming/audit_event_types.md @@ -64,12 +64,6 @@ Audit event types belong to the following product categories. | [`update_event_streaming_destination`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/74632) | Event triggered when an external audit event destination is updated| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [14.6](https://gitlab.com/gitlab-org/gitlab/-/issues/344664) | | [`update_instance_event_streaming_destination`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/125846) | Event triggered when an instance level external audit event destination is updated| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.2](https://gitlab.com/gitlab-org/gitlab/-/issues/404730) | -### Authorization - -| Name | Description | Saved to database | Streamed | Introduced in | -|:-----|:------------|:------------------|:---------|:--------------| -| [`member_role_created`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/137087) | Event triggered when a custom role is created.| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.7](https://gitlab.com/gitlab-org/gitlab/-/issues/388934) | - ### Code review | Name | Description | Saved to database | Streamed | Introduced in | @@ -181,7 +175,7 @@ Audit event types belong to the following product categories. | [`ci_group_variable_deleted`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91983) | Triggered when a group's CI variable is deleted| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [15.2](https://gitlab.com/gitlab-org/gitlab/-/issues/363090) | | [`ci_group_variable_updated`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91983) | Triggered when a group's CI variable is updated| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [15.2](https://gitlab.com/gitlab-org/gitlab/-/issues/363090) | | [`ci_instance_variable_created`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/131882) | When an instance level CI variable is created| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.5](https://gitlab.com/gitlab-org/gitlab/-/issues/8070) | -| [`ci_instance_variable_deleted`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/131882) | When an instance level CI varialbe is deleted| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.5](https://gitlab.com/gitlab-org/gitlab/-/issues/8070) | +| [`ci_instance_variable_deleted`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/131882) | When an instance level CI variable is deleted| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.5](https://gitlab.com/gitlab-org/gitlab/-/issues/8070) | | [`ci_instance_variable_updated`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/131882) | When an instance level CI variable is changed| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.5](https://gitlab.com/gitlab-org/gitlab/-/issues/8070) | | [`ci_variable_created`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91983) | Triggered when a CI variable is created at a project level| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [15.2](https://gitlab.com/gitlab-org/gitlab/-/issues/363090) | | [`ci_variable_deleted`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91983) | Triggered when a project's CI variable is deleted| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [15.2](https://gitlab.com/gitlab-org/gitlab/-/issues/363090) | @@ -233,6 +227,12 @@ Audit event types belong to the following product categories. | [`feature_flag_deleted`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/113453) | Triggered when a feature flag is deleted.| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [15.10](https://gitlab.com/gitlab-org/gitlab/-/issues/374109) | | [`feature_flag_updated`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/113453) | Triggered when a feature flag is updated.| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [15.10](https://gitlab.com/gitlab-org/gitlab/-/issues/374109) | +### Fleet visibility + +| Name | Description | Saved to database | Streamed | Introduced in | +|:-----|:------------|:------------------|:---------|:--------------| +| [`ci_runner_usage_export`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/139578) | Triggered when a runner usage report is generated.| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.8](https://gitlab.com/gitlab-org/gitlab/-/issues/426560) | + ### Fuzz testing | Name | Description | Saved to database | Streamed | Introduced in | @@ -326,6 +326,14 @@ Audit event types belong to the following product categories. |:-----|:------------|:------------------|:---------|:--------------| | [`experiment_features_enabled_updated`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/118222) | Event triggered on toggling setting for enabling experiment AI features| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.0](https://gitlab.com/gitlab-org/gitlab/-/issues/404856/) | +### Permissions + +| Name | Description | Saved to database | Streamed | Introduced in | +|:-----|:------------|:------------------|:---------|:--------------| +| [`member_role_created`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/137087) | Event triggered when a custom role is created.| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.7](https://gitlab.com/gitlab-org/gitlab/-/issues/388934) | +| [`member_role_deleted`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/141630) | Event triggered when a custom role is deleted.| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.9](https://gitlab.com/gitlab-org/gitlab/-/issues/437672) | +| [`member_role_updated`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/141630) | Event triggered when a custom role is updated.| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.9](https://gitlab.com/gitlab-org/gitlab/-/issues/437672) | + ### Portfolio management | Name | Description | Saved to database | Streamed | Introduced in | diff --git a/doc/administration/audit_event_streaming/index.md b/doc/administration/audit_event_streaming/index.md index 71ae33b3d87..2cfe71cd4a5 100644 --- a/doc/administration/audit_event_streaming/index.md +++ b/doc/administration/audit_event_streaming/index.md @@ -589,11 +589,8 @@ To delete Google Cloud Logging streaming destinations to an instance: ### AWS S3 destinations -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/138245) in GitLab 16.7 [with a flag](../feature_flags.md) named `allow_streaming_instance_audit_events_to_amazon_s3`. Disabled by default. - -FLAG: -On self-managed GitLab, by default this feature is not available. To enable the feature, an administrator can [enable the feature flag](../feature_flags.md) named -`allow_streaming_instance_audit_events_to_amazon_s3`. On GitLab.com, this feature is not available. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/138245) in GitLab 16.7 [with a flag](../feature_flags.md) named `allow_streaming_instance_audit_events_to_amazon_s3`. Disabled by default. +> - [Feature flag `allow_streaming_instance_audit_events_to_amazon_s3`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/137391) removed in GitLab 16.8. Manage AWS S3 destinations for entire instance. diff --git a/doc/administration/auth/index.md b/doc/administration/auth/index.md index 95268f6f39b..dd7f9cec77e 100644 --- a/doc/administration/auth/index.md +++ b/doc/administration/auth/index.md @@ -1,6 +1,7 @@ --- stage: Govern group: Authentication +description: Third-party authentication providers. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/administration/auth/jwt.md b/doc/administration/auth/jwt.md index f4e7ea09615..4e000576652 100644 --- a/doc/administration/auth/jwt.md +++ b/doc/administration/auth/jwt.md @@ -4,7 +4,7 @@ group: Authentication info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Use JWT as an OAuth 2.0 authentication provider **(FREE SELF)** +# Use JWT as an authentication provider **(FREE SELF)** To enable the JWT OmniAuth provider, you must register your application with JWT. JWT provides you with a secret key for you to use. diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index 62395ebdcd2..49841b5e1be 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -1210,10 +1210,9 @@ For more information on synchronizing users and groups between LDAP and GitLab, ## Move from LDAP to SAML -1. [Configure SAML](../../../integration/saml.md). Add `auto_link_ldap_user` to: +1. [Add SAML configuration](../../../integration/saml.md) to: - [`gitlab.rb` for Linux package installations](../../../integration/saml.html?tab=Linux+package+%28Omnibus%29). - [`values.yml` for Helm chart installations](../../../integration/saml.html?tab=Helm+chart+%28Kubernetes%29). - For more information, see the [initial settings for all providers](../../../integration/omniauth.md#configure-initial-settings). 1. Optional. [Disable the LDAP auth from the sign-in page](#disable-ldap-web-sign-in). diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md index 9cd306d979f..eb1ee203469 100644 --- a/doc/administration/auth/ldap/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap/ldap-troubleshooting.md @@ -774,10 +774,10 @@ If a connection can't be established, it is likely either because of a problem with your configuration or a firewall blocking the connection. - Ensure you don't have a firewall blocking the -connection, and that the LDAP server is accessible to the GitLab host. + connection, and that the LDAP server is accessible to the GitLab host. - Look for an error message in the Rake check output, which may lead to your LDAP configuration to -confirm that the configuration values (specifically `host`, `port`, `bind_dn`, and -`password`) are correct. + confirm that the configuration values (specifically `host`, `port`, `bind_dn`, and + `password`) are correct. - Look for [errors](#connection) in [the logs](#gitlab-logs) to further debug connection failures. If GitLab can successfully connect to LDAP but doesn't return any diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md index d20c9ee4412..80c3a21d953 100644 --- a/doc/administration/auth/oidc.md +++ b/doc/administration/auth/oidc.md @@ -4,7 +4,7 @@ group: Authentication info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Use OpenID Connect as an OAuth 2.0 authentication provider **(FREE SELF)** +# Use OpenID Connect as an authentication provider **(FREE SELF)** You can use GitLab as a client application with [OpenID Connect](https://openid.net/specs/openid-connect-core-1_0.html) as an OmniAuth provider. diff --git a/doc/administration/backup_restore/backup_gitlab.md b/doc/administration/backup_restore/backup_gitlab.md index f38358810e3..ac04709f8b2 100644 --- a/doc/administration/backup_restore/backup_gitlab.md +++ b/doc/administration/backup_restore/backup_gitlab.md @@ -136,13 +136,13 @@ For more information, see [Backup and restore Linux package (Omnibus) configurat :::TabTitle Docker - Back up the volume where the configuration files are stored. If you created -the GitLab container according to the documentation, it should be in the -`/srv/gitlab/config` directory. + the GitLab container according to the documentation, it should be in the + `/srv/gitlab/config` directory. :::TabTitle GitLab Helm chart - Follow the [Back up the secrets](https://docs.gitlab.com/charts/backup-restore/backup.html#back-up-the-secrets) -instructions. + instructions. ::EndTabs @@ -150,8 +150,8 @@ You may also want to back up any TLS keys and certificates (`/etc/gitlab/ssl`, ` [SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079) to avoid man-in-the-middle attack warnings if you have to perform a full machine restore. -In the unlikely event that the secrets file is lost, see the -[troubleshooting section](#when-the-secrets-file-is-lost). +In the unlikely event that the secrets file is lost, see +[When the secrets file is lost](../../administration/backup_restore/troubleshooting_backup_gitlab.md#when-the-secrets-file-is-lost). ### Other data @@ -656,7 +656,8 @@ sudo -u git -H bundle exec rake gitlab:backup:create REPOSITORIES_STORAGES=stora #### Back up specific repositories -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/88094) in GitLab 15.1. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/88094) in GitLab 15.1. +> - [Skipping specific repositories added](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121865) in GitLab 16.1. You can back up specific repositories using the `REPOSITORIES_PATHS` option. Similarly, you can use `SKIP_REPOSITORIES_PATHS` to skip certain repositories. @@ -1451,618 +1452,3 @@ There are a few possible downsides to this: There is an **experimental** script that attempts to automate this process in [the Geo team Runbooks project](https://gitlab.com/gitlab-org/geo-team/runbooks/-/tree/main/experimental-online-backup-through-rsync). - -## Troubleshooting - -The following are possible problems you might encounter, along with potential -solutions. - -### When the secrets file is lost - -If you didn't [back up the secrets file](#storing-configuration-files), you -must complete several steps to get GitLab working properly again. - -The secrets file is responsible for storing the encryption key for the columns -that contain required, sensitive information. If the key is lost, GitLab can't -decrypt those columns, preventing access to the following items: - -- [CI/CD variables](../../ci/variables/index.md) -- [Kubernetes / GCP integration](../../user/infrastructure/clusters/index.md) -- [Custom Pages domains](../../user/project/pages/custom_domains_ssl_tls_certification/index.md) -- [Project error tracking](../../operations/error_tracking.md) -- [Runner authentication](../../ci/runners/index.md) -- [Project mirroring](../../user/project/repository/mirror/index.md) -- [Integrations](../../user/project/integrations/index.md) -- [Web hooks](../../user/project/integrations/webhooks.md) - -In cases like CI/CD variables and runner authentication, you can experience -unexpected behaviors, such as: - -- Stuck jobs. -- 500 errors. - -In this case, you must reset all the tokens for CI/CD variables and -runner authentication, which is described in more detail in the following -sections. After resetting the tokens, you should be able to visit your project -and the jobs begin running again. - -WARNING: -The steps in this section can potentially lead to **data loss** on the above listed items. -Consider opening a [Support Request](https://support.gitlab.com/hc/en-us/requests/new) if you're a Premium or Ultimate customer. - -#### Verify that all values can be decrypted - -You can determine if your database contains values that can't be decrypted by using a -[Rake task](../raketasks/check.md#verify-database-values-can-be-decrypted-using-the-current-secrets). - -#### Take a backup - -You must directly modify GitLab data to work around your lost secrets file. - -WARNING: -Be sure to create a full database backup before attempting any changes. - -#### Disable user two-factor authentication (2FA) - -Users with 2FA enabled can't sign in to GitLab. In that case, you must -[disable 2FA for everyone](../../security/two_factor_authentication.md#for-all-users), -after which users must reactivate 2FA. - -#### Reset CI/CD variables - -1. Enter the database console: - - For the Linux package (Omnibus) GitLab 14.1 and earlier: - - ```shell - sudo gitlab-rails dbconsole - ``` - - For the Linux package (Omnibus) GitLab 14.2 and later: - - ```shell - sudo gitlab-rails dbconsole --database main - ``` - - For self-compiled installations, GitLab 14.1 and earlier: - - ```shell - sudo -u git -H bundle exec rails dbconsole -e production - ``` - - For self-compiled installations, GitLab 14.2 and later: - - ```shell - sudo -u git -H bundle exec rails dbconsole -e production --database main - ``` - -1. Examine the `ci_group_variables` and `ci_variables` tables: - - ```sql - SELECT * FROM public."ci_group_variables"; - SELECT * FROM public."ci_variables"; - ``` - - These are the variables that you need to delete. - -1. Delete all variables: - - ```sql - DELETE FROM ci_group_variables; - DELETE FROM ci_variables; - ``` - -1. If you know the specific group or project from which you wish to delete variables, you can include a `WHERE` statement to specify that in your `DELETE`: - - ```sql - DELETE FROM ci_group_variables WHERE group_id = <GROUPID>; - DELETE FROM ci_variables WHERE project_id = <PROJECTID>; - ``` - -You may need to reconfigure or restart GitLab for the changes to take effect. - -#### Reset runner registration tokens - -1. Enter the database console: - - For the Linux package (Omnibus) GitLab 14.1 and earlier: - - ```shell - sudo gitlab-rails dbconsole - ``` - - For the Linux package (Omnibus) GitLab 14.2 and later: - - ```shell - sudo gitlab-rails dbconsole --database main - ``` - - For self-compiled installations, GitLab 14.1 and earlier: - - ```shell - sudo -u git -H bundle exec rails dbconsole -e production - ``` - - For self-compiled installations, GitLab 14.2 and later: - - ```shell - sudo -u git -H bundle exec rails dbconsole -e production --database main - ``` - -1. Clear all tokens for projects, groups, and the entire instance: - - WARNING: - The final `UPDATE` operation stops the runners from being able to pick - up new jobs. You must register new runners. - - ```sql - -- Clear project tokens - UPDATE projects SET runners_token = null, runners_token_encrypted = null; - -- Clear group tokens - UPDATE namespaces SET runners_token = null, runners_token_encrypted = null; - -- Clear instance tokens - UPDATE application_settings SET runners_registration_token_encrypted = null; - -- Clear key used for JWT authentication - -- This may break the $CI_JWT_TOKEN job variable: - -- https://gitlab.com/gitlab-org/gitlab/-/issues/325965 - UPDATE application_settings SET encrypted_ci_jwt_signing_key = null; - -- Clear runner tokens - UPDATE ci_runners SET token = null, token_encrypted = null; - ``` - -#### Reset pending pipeline jobs - -1. Enter the database console: - - For the Linux package (Omnibus) GitLab 14.1 and earlier: - - ```shell - sudo gitlab-rails dbconsole - ``` - - For the Linux package (Omnibus) GitLab 14.2 and later: - - ```shell - sudo gitlab-rails dbconsole --database main - ``` - - For self-compiled installations, GitLab 14.1 and earlier: - - ```shell - sudo -u git -H bundle exec rails dbconsole -e production - ``` - - For self-compiled installations, GitLab 14.2 and later: - - ```shell - sudo -u git -H bundle exec rails dbconsole -e production --database main - ``` - -1. Clear all the tokens for pending jobs: - - For GitLab 15.3 and earlier: - - ```sql - -- Clear build tokens - UPDATE ci_builds SET token = null, token_encrypted = null; - ``` - - For GitLab 15.4 and later: - - ```sql - -- Clear build tokens - UPDATE ci_builds SET token_encrypted = null; - ``` - -A similar strategy can be employed for the remaining features. By removing the -data that can't be decrypted, GitLab can be returned to operation, and the -lost data can be manually replaced. - -#### Fix integrations and webhooks - -If you've lost your secrets, the [integrations settings](../../user/project/integrations/index.md) -and [webhooks settings](../../user/project/integrations/webhooks.md) pages might display `500` error messages. Lost secrets might also produce `500` errors when you try to access a repository in a project with a previously configured integration or webhook. - -The fix is to truncate the affected tables (those containing encrypted columns). -This deletes all your configured integrations, webhooks, and related metadata. -You should verify that the secrets are the root cause before deleting any data. - -1. Enter the database console: - - For the Linux package (Omnibus) GitLab 14.1 and earlier: - - ```shell - sudo gitlab-rails dbconsole - ``` - - For the Linux package (Omnibus) GitLab 14.2 and later: - - ```shell - sudo gitlab-rails dbconsole --database main - ``` - - For self-compiled installations, GitLab 14.1 and earlier: - - ```shell - sudo -u git -H bundle exec rails dbconsole -e production - ``` - - For self-compiled installations, GitLab 14.2 and later: - - ```shell - sudo -u git -H bundle exec rails dbconsole -e production --database main - ``` - -1. Truncate the following tables: - - ```sql - -- truncate web_hooks table - TRUNCATE integrations, chat_names, issue_tracker_data, jira_tracker_data, slack_integrations, web_hooks, zentao_tracker_data, web_hook_logs CASCADE; - ``` - -### Container registry push failures after restoring from a backup - -If you use the [container registry](../../user/packages/container_registry/index.md), -pushes to the registry may fail after restoring your backup on a Linux package (Omnibus) -instance after restoring the registry data. - -These failures mention permission issues in the registry logs, similar to: - -```plaintext -level=error -msg="response completed with error" -err.code=unknown -err.detail="filesystem: mkdir /var/opt/gitlab/gitlab-rails/shared/registry/docker/registry/v2/repositories/...: permission denied" -err.message="unknown error" -``` - -This issue is caused by the restore running as the unprivileged user `git`, -which is unable to assign the correct ownership to the registry files during -the restore process ([issue #62759](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/62759 "Incorrect permissions on registry filesystem after restore")). - -To get your registry working again: - -```shell -sudo chown -R registry:registry /var/opt/gitlab/gitlab-rails/shared/registry/docker -``` - -If you changed the default file system location for the registry, run `chown` -against your custom location, instead of `/var/opt/gitlab/gitlab-rails/shared/registry/docker`. - -### Backup fails to complete with Gzip error - -When running the backup, you may receive a Gzip error message: - -```shell -sudo /opt/gitlab/bin/gitlab-backup create -... -Dumping ... -... -gzip: stdout: Input/output error - -Backup failed -``` - -If this happens, examine the following: - -- Confirm there is sufficient disk space for the Gzip operation. It's not uncommon for backups that - use the [default strategy](#backup-strategy-option) to require half the instance size - in free disk space during backup creation. -- If NFS is being used, check if the mount option `timeout` is set. The - default is `600`, and changing this to smaller values results in this error. - -### Backup fails with `File name too long` error - -During backup, you can get the `File name too long` error ([issue #354984](https://gitlab.com/gitlab-org/gitlab/-/issues/354984)). For example: - -```plaintext -Problem: <class 'OSError: [Errno 36] File name too long: -``` - -This problem stops the backup script from completing. To fix this problem, you must truncate the file names causing the problem. A maximum of 246 characters, including the file extension, is permitted. - -WARNING: -The steps in this section can potentially lead to **data loss**. All steps must be followed strictly in the order given. -Consider opening a [Support Request](https://support.gitlab.com/hc/en-us/requests/new) if you're a Premium or Ultimate customer. - -Truncating file names to resolve the error involves: - -- Cleaning up remote uploaded files that aren't tracked in the database. -- Truncating the file names in the database. -- Rerunning the backup task. - -#### Clean up remote uploaded files - -A [known issue](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/45425) caused object store uploads to remain after a parent resource was deleted. This issue was [resolved](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/18698). - -To fix these files, you must clean up all remote uploaded files that are in the storage but not tracked in the `uploads` database table. - -1. List all the object store upload files that can be moved to a lost and found directory if they don't exist in the GitLab database: - - ```shell - bundle exec rake gitlab:cleanup:remote_upload_files RAILS_ENV=production - ``` - -1. If you are sure you want to delete these files and remove all non-referenced uploaded files, run: - - WARNING: - The following action is **irreversible**. - - ```shell - bundle exec rake gitlab:cleanup:remote_upload_files RAILS_ENV=production DRY_RUN=false - ``` - -#### Truncate the file names referenced by the database - -You must truncate the files referenced by the database that are causing the problem. The file names referenced by the database are stored: - -- In the `uploads` table. -- In the references found. Any reference found from other database tables and columns. -- On the file system. - -Truncate the file names in the `uploads` table: - -1. Enter the database console: - - For the Linux package (Omnibus) GitLab 14.2 and later: - - ```shell - sudo gitlab-rails dbconsole --database main - ``` - - For the Linux package (Omnibus) GitLab 14.1 and earlier: - - ```shell - sudo gitlab-rails dbconsole - ``` - - For self-compiled installations, GitLab 14.2 and later: - - ```shell - sudo -u git -H bundle exec rails dbconsole -e production --database main - ``` - - For self-compiled installations, GitLab 14.1 and earlier: - - ```shell - sudo -u git -H bundle exec rails dbconsole -e production - ``` - -1. Search the `uploads` table for file names longer than 246 characters: - - The following query selects the `uploads` records with file names longer than 246 characters in batches of 0 to 10000. This improves the performance on large GitLab instances with tables having thousand of records. - - ```sql - CREATE TEMP TABLE uploads_with_long_filenames AS - SELECT ROW_NUMBER() OVER(ORDER BY id) row_id, id, path - FROM uploads AS u - WHERE LENGTH((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1]) > 246; - - CREATE INDEX ON uploads_with_long_filenames(row_id); - - SELECT - u.id, - u.path, - -- Current file name - (regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1] AS current_filename, - -- New file name - CONCAT( - LEFT(SPLIT_PART((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1], '.', 1), 242), - COALESCE(SUBSTRING((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1] FROM '\.(?:.(?!\.))+$')) - ) AS new_filename, - -- New path - CONCAT( - COALESCE((regexp_match(u.path, '(.*\/).*'))[1], ''), - CONCAT( - LEFT(SPLIT_PART((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1], '.', 1), 242), - COALESCE(SUBSTRING((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1] FROM '\.(?:.(?!\.))+$')) - ) - ) AS new_path - FROM uploads_with_long_filenames AS u - WHERE u.row_id > 0 AND u.row_id <= 10000; - ``` - - Output example: - - ```postgresql - -[ RECORD 1 ]----+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - id | 34 - path | public/@hashed/loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisit.txt - current_filename | loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisit.txt - new_filename | loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelits.txt - new_path | public/@hashed/loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelits.txt - ``` - - Where: - - - `current_filename`: a file name that is currently more than 246 characters long. - - `new_filename`: a file name that has been truncated to 246 characters maximum. - - `new_path`: new path considering the `new_filename` (truncated). - - After you validate the batch results, you must change the batch size (`row_id`) using the following sequence of numbers (10000 to 20000). Repeat this process until you reach the last record in the `uploads` table. - -1. Rename the files found in the `uploads` table from long file names to new truncated file names. The following query rolls back the update so you can check the results safely in a transaction wrapper: - - ```sql - CREATE TEMP TABLE uploads_with_long_filenames AS - SELECT ROW_NUMBER() OVER(ORDER BY id) row_id, path, id - FROM uploads AS u - WHERE LENGTH((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1]) > 246; - - CREATE INDEX ON uploads_with_long_filenames(row_id); - - BEGIN; - WITH updated_uploads AS ( - UPDATE uploads - SET - path = - CONCAT( - COALESCE((regexp_match(updatable_uploads.path, '(.*\/).*'))[1], ''), - CONCAT( - LEFT(SPLIT_PART((regexp_match(updatable_uploads.path, '[^\\/:*?"<>|\r\n]+$'))[1], '.', 1), 242), - COALESCE(SUBSTRING((regexp_match(updatable_uploads.path, '[^\\/:*?"<>|\r\n]+$'))[1] FROM '\.(?:.(?!\.))+$')) - ) - ) - FROM - uploads_with_long_filenames AS updatable_uploads - WHERE - uploads.id = updatable_uploads.id - AND updatable_uploads.row_id > 0 AND updatable_uploads.row_id <= 10000 - RETURNING uploads.* - ) - SELECT id, path FROM updated_uploads; - ROLLBACK; - ``` - - After you validate the batch update results, you must change the batch size (`row_id`) using the following sequence of numbers (10000 to 20000). Repeat this process until you reach the last record in the `uploads` table. - -1. Validate that the new file names from the previous query are the expected ones. If you are sure you want to truncate the records found in the previous step to 246 characters, run the following: - - WARNING: - The following action is **irreversible**. - - ```sql - CREATE TEMP TABLE uploads_with_long_filenames AS - SELECT ROW_NUMBER() OVER(ORDER BY id) row_id, path, id - FROM uploads AS u - WHERE LENGTH((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1]) > 246; - - CREATE INDEX ON uploads_with_long_filenames(row_id); - - UPDATE uploads - SET - path = - CONCAT( - COALESCE((regexp_match(updatable_uploads.path, '(.*\/).*'))[1], ''), - CONCAT( - LEFT(SPLIT_PART((regexp_match(updatable_uploads.path, '[^\\/:*?"<>|\r\n]+$'))[1], '.', 1), 242), - COALESCE(SUBSTRING((regexp_match(updatable_uploads.path, '[^\\/:*?"<>|\r\n]+$'))[1] FROM '\.(?:.(?!\.))+$')) - ) - ) - FROM - uploads_with_long_filenames AS updatable_uploads - WHERE - uploads.id = updatable_uploads.id - AND updatable_uploads.row_id > 0 AND updatable_uploads.row_id <= 10000; - ``` - - After you finish the batch update, you must change the batch size (`updatable_uploads.row_id`) using the following sequence of numbers (10000 to 20000). Repeat this process until you reach the last record in the `uploads` table. - -Truncate the file names in the references found: - -1. Check if those records are referenced somewhere. One way to do this is to dump the database and search for the parent directory name and file name: - - 1. To dump your database, you can use the following command as an example: - - ```shell - pg_dump -h /var/opt/gitlab/postgresql/ -d gitlabhq_production > gitlab-dump.tmp - ``` - - 1. Then you can search for the references using the `grep` command. Combining the parent directory and the file name can be a good idea. For example: - - ```shell - grep public/alongfilenamehere.txt gitlab-dump.tmp - ``` - -1. Replace those long file names using the new file names obtained from querying the `uploads` table. - -Truncate the file names on the file system. You must manually rename the files in your file system to the new file names obtained from querying the `uploads` table. - -#### Re-run the backup task - -After following all the previous steps, re-run the backup task. - -### Restoring database backup fails when `pg_stat_statements` was previously enabled - -The GitLab backup of the PostgreSQL database includes all SQL statements required to enable extensions that were -previously enabled in the database. - -The `pg_stat_statements` extension can only be enabled or disabled by a PostgreSQL user with `superuser` role. -As the restore process uses a database user with limited permissions, it can't execute the following SQL statements: - -```sql -DROP EXTENSION IF EXISTS pg_stat_statements; -CREATE EXTENSION IF NOT EXISTS pg_stat_statements WITH SCHEMA public; -``` - -When trying to restore the backup in a PostgreSQL instance that doesn't have the `pg_stats_statements` extension, -the following error message is displayed: - -```plaintext -ERROR: permission denied to create extension "pg_stat_statements" -HINT: Must be superuser to create this extension. -ERROR: extension "pg_stat_statements" does not exist -``` - -When trying to restore in an instance that has the `pg_stats_statements` extension enabled, the cleaning up step -fails with an error message similar to the following: - -```plaintext -rake aborted! -ActiveRecord::StatementInvalid: PG::InsufficientPrivilege: ERROR: must be owner of view pg_stat_statements -/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:42:in `block (4 levels) in <top (required)>' -/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:41:in `each' -/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:41:in `block (3 levels) in <top (required)>' -/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:71:in `block (3 levels) in <top (required)>' -/opt/gitlab/embedded/bin/bundle:23:in `load' -/opt/gitlab/embedded/bin/bundle:23:in `<main>' -Caused by: -PG::InsufficientPrivilege: ERROR: must be owner of view pg_stat_statements -/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:42:in `block (4 levels) in <top (required)>' -/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:41:in `each' -/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:41:in `block (3 levels) in <top (required)>' -/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:71:in `block (3 levels) in <top (required)>' -/opt/gitlab/embedded/bin/bundle:23:in `load' -/opt/gitlab/embedded/bin/bundle:23:in `<main>' -Tasks: TOP => gitlab:db:drop_tables -(See full trace by running task with --trace) -``` - -#### Prevent the dump file to include `pg_stat_statements` - -To prevent the inclusion of the extension in the PostgreSQL dump file that is part of the backup bundle, -enable the extension in any schema except the `public` schema: - -```sql -CREATE SCHEMA adm; -CREATE EXTENSION pg_stat_statements SCHEMA adm; -``` - -If the extension was previously enabled in the `public` schema, move it to a new one: - -```sql -CREATE SCHEMA adm; -ALTER EXTENSION pg_stat_statements SET SCHEMA adm; -``` - -To query the `pg_stat_statements` data after changing the schema, prefix the view name with the new schema: - -```sql -SELECT * FROM adm.pg_stat_statements limit 0; -``` - -To make it compatible with third-party monitoring solutions that expect it to be enabled in the `public` schema, -you need to include it in the `search_path`: - -```sql -set search_path to public,adm; -``` - -#### Fix an existing dump file to remove references to `pg_stat_statements` - -To fix an existing backup file, do the following changes: - -1. Extract from the backup the following file: `db/database.sql.gz`. -1. Decompress the file or use an editor that is capable of handling it compressed. -1. Remove the following lines, or similar ones: - - ```sql - CREATE EXTENSION IF NOT EXISTS pg_stat_statements WITH SCHEMA public; - ``` - - ```sql - COMMENT ON EXTENSION pg_stat_statements IS 'track planning and execution statistics of all SQL statements executed'; - ``` - -1. Save the changes and recompress the file. -1. Update the backup file with the modified `db/database.sql.gz`. diff --git a/doc/administration/backup_restore/restore_gitlab.md b/doc/administration/backup_restore/restore_gitlab.md index 2dd85602f99..0b5bf3cc0ff 100644 --- a/doc/administration/backup_restore/restore_gitlab.md +++ b/doc/administration/backup_restore/restore_gitlab.md @@ -38,8 +38,7 @@ before restoring the backup. To restore a backup, **you must also restore the GitLab secrets**. These include the database encryption key, [CI/CD variables](../../ci/variables/index.md), and variables used for [two-factor authentication](../../user/profile/account/two_factor_authentication.md). -Without the keys, [multiple issues occur](backup_gitlab.md#when-the-secrets-file-is-lost), -including loss of access by users with [two-factor authentication enabled](../../user/profile/account/two_factor_authentication.md), +Without the keys, [multiple issues occur](../../administration/backup_restore/troubleshooting_backup_gitlab.md#when-the-secrets-file-is-lost), including loss of access by users with [two-factor authentication enabled](../../user/profile/account/two_factor_authentication.md), and GitLab Runners cannot log in. Restore: @@ -282,7 +281,8 @@ project or group from there: the backed-up instance from which you want to restore. 1. [Restore the backup](#restore-gitlab) into this new instance, then export your [project](../../user/project/settings/import_export.md) - or [group](../../user/group/import/index.md#migrate-groups-by-uploading-an-export-file-deprecated). For more information about what is and isn't exported, see the export feature's documentation. + or [group](../../user/project/settings/import_export.md#migrate-groups-by-uploading-an-export-file-deprecated). For + more information about what is and isn't exported, see the export feature's documentation. 1. After the export is complete, go to the old instance and then import it. 1. After importing the projects or groups that you wanted is complete, you may delete the new, temporary GitLab instance. diff --git a/doc/administration/backup_restore/troubleshooting_backup_gitlab.md b/doc/administration/backup_restore/troubleshooting_backup_gitlab.md new file mode 100644 index 00000000000..83a02b52741 --- /dev/null +++ b/doc/administration/backup_restore/troubleshooting_backup_gitlab.md @@ -0,0 +1,619 @@ +--- +stage: Systems +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Troubleshooting GitLab backups + +When you back up GitLab, you might encounter the following issues. + +## When the secrets file is lost + +If you didn't [back up the secrets file](../../administration/backup_restore/backup_gitlab.md#storing-configuration-files), you +must complete several steps to get GitLab working properly again. + +The secrets file is responsible for storing the encryption key for the columns +that contain required, sensitive information. If the key is lost, GitLab can't +decrypt those columns, preventing access to the following items: + +- [CI/CD variables](../../ci/variables/index.md) +- [Kubernetes / GCP integration](../../user/infrastructure/clusters/index.md) +- [Custom Pages domains](../../user/project/pages/custom_domains_ssl_tls_certification/index.md) +- [Project error tracking](../../operations/error_tracking.md) +- [Runner authentication](../../ci/runners/index.md) +- [Project mirroring](../../user/project/repository/mirror/index.md) +- [Integrations](../../user/project/integrations/index.md) +- [Web hooks](../../user/project/integrations/webhooks.md) + +In cases like CI/CD variables and runner authentication, you can experience +unexpected behaviors, such as: + +- Stuck jobs. +- 500 errors. + +In this case, you must reset all the tokens for CI/CD variables and +runner authentication, which is described in more detail in the following +sections. After resetting the tokens, you should be able to visit your project +and the jobs begin running again. + +WARNING: +The steps in this section can potentially lead to **data loss** on the above listed items. +Consider opening a [Support Request](https://support.gitlab.com/hc/en-us/requests/new) if you're a Premium or Ultimate customer. + +### Verify that all values can be decrypted + +You can determine if your database contains values that can't be decrypted by using a +[Rake task](../raketasks/check.md#verify-database-values-can-be-decrypted-using-the-current-secrets). + +### Take a backup + +You must directly modify GitLab data to work around your lost secrets file. + +WARNING: +Be sure to create a full database backup before attempting any changes. + +### Disable user two-factor authentication (2FA) + +Users with 2FA enabled can't sign in to GitLab. In that case, you must +[disable 2FA for everyone](../../security/two_factor_authentication.md#for-all-users), +after which users must reactivate 2FA. + +### Reset CI/CD variables + +1. Enter the database console: + + For the Linux package (Omnibus) GitLab 14.1 and earlier: + + ```shell + sudo gitlab-rails dbconsole + ``` + + For the Linux package (Omnibus) GitLab 14.2 and later: + + ```shell + sudo gitlab-rails dbconsole --database main + ``` + + For self-compiled installations, GitLab 14.1 and earlier: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production + ``` + + For self-compiled installations, GitLab 14.2 and later: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production --database main + ``` + +1. Examine the `ci_group_variables` and `ci_variables` tables: + + ```sql + SELECT * FROM public."ci_group_variables"; + SELECT * FROM public."ci_variables"; + ``` + + These are the variables that you need to delete. + +1. Delete all variables: + + ```sql + DELETE FROM ci_group_variables; + DELETE FROM ci_variables; + ``` + +1. If you know the specific group or project from which you wish to delete variables, you can include a `WHERE` statement to specify that in your `DELETE`: + + ```sql + DELETE FROM ci_group_variables WHERE group_id = <GROUPID>; + DELETE FROM ci_variables WHERE project_id = <PROJECTID>; + ``` + +You may need to reconfigure or restart GitLab for the changes to take effect. + +### Reset runner registration tokens + +1. Enter the database console: + + For the Linux package (Omnibus) GitLab 14.1 and earlier: + + ```shell + sudo gitlab-rails dbconsole + ``` + + For the Linux package (Omnibus) GitLab 14.2 and later: + + ```shell + sudo gitlab-rails dbconsole --database main + ``` + + For self-compiled installations, GitLab 14.1 and earlier: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production + ``` + + For self-compiled installations, GitLab 14.2 and later: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production --database main + ``` + +1. Clear all tokens for projects, groups, and the entire instance: + + WARNING: + The final `UPDATE` operation stops the runners from being able to pick + up new jobs. You must register new runners. + + ```sql + -- Clear project tokens + UPDATE projects SET runners_token = null, runners_token_encrypted = null; + -- Clear group tokens + UPDATE namespaces SET runners_token = null, runners_token_encrypted = null; + -- Clear instance tokens + UPDATE application_settings SET runners_registration_token_encrypted = null; + -- Clear key used for JWT authentication + -- This may break the $CI_JWT_TOKEN job variable: + -- https://gitlab.com/gitlab-org/gitlab/-/issues/325965 + UPDATE application_settings SET encrypted_ci_jwt_signing_key = null; + -- Clear runner tokens + UPDATE ci_runners SET token = null, token_encrypted = null; + ``` + +### Reset pending pipeline jobs + +1. Enter the database console: + + For the Linux package (Omnibus) GitLab 14.1 and earlier: + + ```shell + sudo gitlab-rails dbconsole + ``` + + For the Linux package (Omnibus) GitLab 14.2 and later: + + ```shell + sudo gitlab-rails dbconsole --database main + ``` + + For self-compiled installations, GitLab 14.1 and earlier: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production + ``` + + For self-compiled installations, GitLab 14.2 and later: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production --database main + ``` + +1. Clear all the tokens for pending jobs: + + For GitLab 15.3 and earlier: + + ```sql + -- Clear build tokens + UPDATE ci_builds SET token = null, token_encrypted = null; + ``` + + For GitLab 15.4 and later: + + ```sql + -- Clear build tokens + UPDATE ci_builds SET token_encrypted = null; + ``` + +A similar strategy can be employed for the remaining features. By removing the +data that can't be decrypted, GitLab can be returned to operation, and the +lost data can be manually replaced. + +### Fix integrations and webhooks + +If you've lost your secrets, the [integrations settings](../../user/project/integrations/index.md) +and [webhooks settings](../../user/project/integrations/webhooks.md) pages might display `500` error messages. Lost secrets might also produce `500` errors when you try to access a repository in a project with a previously configured integration or webhook. + +The fix is to truncate the affected tables (those containing encrypted columns). +This deletes all your configured integrations, webhooks, and related metadata. +You should verify that the secrets are the root cause before deleting any data. + +1. Enter the database console: + + For the Linux package (Omnibus) GitLab 14.1 and earlier: + + ```shell + sudo gitlab-rails dbconsole + ``` + + For the Linux package (Omnibus) GitLab 14.2 and later: + + ```shell + sudo gitlab-rails dbconsole --database main + ``` + + For self-compiled installations, GitLab 14.1 and earlier: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production + ``` + + For self-compiled installations, GitLab 14.2 and later: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production --database main + ``` + +1. Truncate the following tables: + + ```sql + -- truncate web_hooks table + TRUNCATE integrations, chat_names, issue_tracker_data, jira_tracker_data, slack_integrations, web_hooks, zentao_tracker_data, web_hook_logs CASCADE; + ``` + +## Container registry push failures after restoring from a backup + +If you use the [container registry](../../user/packages/container_registry/index.md), +pushes to the registry may fail after restoring your backup on a Linux package (Omnibus) +instance after restoring the registry data. + +These failures mention permission issues in the registry logs, similar to: + +```plaintext +level=error +msg="response completed with error" +err.code=unknown +err.detail="filesystem: mkdir /var/opt/gitlab/gitlab-rails/shared/registry/docker/registry/v2/repositories/...: permission denied" +err.message="unknown error" +``` + +This issue is caused by the restore running as the unprivileged user `git`, +which is unable to assign the correct ownership to the registry files during +the restore process ([issue #62759](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/62759 "Incorrect permissions on registry filesystem after restore")). + +To get your registry working again: + +```shell +sudo chown -R registry:registry /var/opt/gitlab/gitlab-rails/shared/registry/docker +``` + +If you changed the default file system location for the registry, run `chown` +against your custom location, instead of `/var/opt/gitlab/gitlab-rails/shared/registry/docker`. + +## Backup fails to complete with Gzip error + +When running the backup, you may receive a Gzip error message: + +```shell +sudo /opt/gitlab/bin/gitlab-backup create +... +Dumping ... +... +gzip: stdout: Input/output error + +Backup failed +``` + +If this happens, examine the following: + +- Confirm there is sufficient disk space for the Gzip operation. It's not uncommon for backups that + use the [default strategy](../../administration/backup_restore/backup_gitlab.md#backup-strategy-option) to require half the instance size + in free disk space during backup creation. +- If NFS is being used, check if the mount option `timeout` is set. The + default is `600`, and changing this to smaller values results in this error. + +## Backup fails with `File name too long` error + +During backup, you can get the `File name too long` error ([issue #354984](https://gitlab.com/gitlab-org/gitlab/-/issues/354984)). For example: + +```plaintext +Problem: <class 'OSError: [Errno 36] File name too long: +``` + +This problem stops the backup script from completing. To fix this problem, you must truncate the file names causing the problem. A maximum of 246 characters, including the file extension, is permitted. + +WARNING: +The steps in this section can potentially lead to **data loss**. All steps must be followed strictly in the order given. +Consider opening a [Support Request](https://support.gitlab.com/hc/en-us/requests/new) if you're a Premium or Ultimate customer. + +Truncating file names to resolve the error involves: + +- Cleaning up remote uploaded files that aren't tracked in the database. +- Truncating the file names in the database. +- Rerunning the backup task. + +### Clean up remote uploaded files + +A [known issue](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/45425) caused object store uploads to remain after a parent resource was deleted. This issue was [resolved](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/18698). + +To fix these files, you must clean up all remote uploaded files that are in the storage but not tracked in the `uploads` database table. + +1. List all the object store upload files that can be moved to a lost and found directory if they don't exist in the GitLab database: + + ```shell + bundle exec rake gitlab:cleanup:remote_upload_files RAILS_ENV=production + ``` + +1. If you are sure you want to delete these files and remove all non-referenced uploaded files, run: + + WARNING: + The following action is **irreversible**. + + ```shell + bundle exec rake gitlab:cleanup:remote_upload_files RAILS_ENV=production DRY_RUN=false + ``` + +### Truncate the file names referenced by the database + +You must truncate the files referenced by the database that are causing the problem. The file names referenced by the database are stored: + +- In the `uploads` table. +- In the references found. Any reference found from other database tables and columns. +- On the file system. + +Truncate the file names in the `uploads` table: + +1. Enter the database console: + + For the Linux package (Omnibus) GitLab 14.2 and later: + + ```shell + sudo gitlab-rails dbconsole --database main + ``` + + For the Linux package (Omnibus) GitLab 14.1 and earlier: + + ```shell + sudo gitlab-rails dbconsole + ``` + + For self-compiled installations, GitLab 14.2 and later: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production --database main + ``` + + For self-compiled installations, GitLab 14.1 and earlier: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production + ``` + +1. Search the `uploads` table for file names longer than 246 characters: + + The following query selects the `uploads` records with file names longer than 246 characters in batches of 0 to 10000. This improves the performance on large GitLab instances with tables having thousand of records. + + ```sql + CREATE TEMP TABLE uploads_with_long_filenames AS + SELECT ROW_NUMBER() OVER(ORDER BY id) row_id, id, path + FROM uploads AS u + WHERE LENGTH((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1]) > 246; + + CREATE INDEX ON uploads_with_long_filenames(row_id); + + SELECT + u.id, + u.path, + -- Current file name + (regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1] AS current_filename, + -- New file name + CONCAT( + LEFT(SPLIT_PART((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1], '.', 1), 242), + COALESCE(SUBSTRING((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1] FROM '\.(?:.(?!\.))+$')) + ) AS new_filename, + -- New path + CONCAT( + COALESCE((regexp_match(u.path, '(.*\/).*'))[1], ''), + CONCAT( + LEFT(SPLIT_PART((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1], '.', 1), 242), + COALESCE(SUBSTRING((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1] FROM '\.(?:.(?!\.))+$')) + ) + ) AS new_path + FROM uploads_with_long_filenames AS u + WHERE u.row_id > 0 AND u.row_id <= 10000; + ``` + + Output example: + + ```postgresql + -[ RECORD 1 ]----+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + id | 34 + path | public/@hashed/loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisit.txt + current_filename | loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisit.txt + new_filename | loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelits.txt + new_path | public/@hashed/loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelits.txt + ``` + + Where: + + - `current_filename`: a file name that is currently more than 246 characters long. + - `new_filename`: a file name that has been truncated to 246 characters maximum. + - `new_path`: new path considering the `new_filename` (truncated). + + After you validate the batch results, you must change the batch size (`row_id`) using the following sequence of numbers (10000 to 20000). Repeat this process until you reach the last record in the `uploads` table. + +1. Rename the files found in the `uploads` table from long file names to new truncated file names. The following query rolls back the update so you can check the results safely in a transaction wrapper: + + ```sql + CREATE TEMP TABLE uploads_with_long_filenames AS + SELECT ROW_NUMBER() OVER(ORDER BY id) row_id, path, id + FROM uploads AS u + WHERE LENGTH((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1]) > 246; + + CREATE INDEX ON uploads_with_long_filenames(row_id); + + BEGIN; + WITH updated_uploads AS ( + UPDATE uploads + SET + path = + CONCAT( + COALESCE((regexp_match(updatable_uploads.path, '(.*\/).*'))[1], ''), + CONCAT( + LEFT(SPLIT_PART((regexp_match(updatable_uploads.path, '[^\\/:*?"<>|\r\n]+$'))[1], '.', 1), 242), + COALESCE(SUBSTRING((regexp_match(updatable_uploads.path, '[^\\/:*?"<>|\r\n]+$'))[1] FROM '\.(?:.(?!\.))+$')) + ) + ) + FROM + uploads_with_long_filenames AS updatable_uploads + WHERE + uploads.id = updatable_uploads.id + AND updatable_uploads.row_id > 0 AND updatable_uploads.row_id <= 10000 + RETURNING uploads.* + ) + SELECT id, path FROM updated_uploads; + ROLLBACK; + ``` + + After you validate the batch update results, you must change the batch size (`row_id`) using the following sequence of numbers (10000 to 20000). Repeat this process until you reach the last record in the `uploads` table. + +1. Validate that the new file names from the previous query are the expected ones. If you are sure you want to truncate the records found in the previous step to 246 characters, run the following: + + WARNING: + The following action is **irreversible**. + + ```sql + CREATE TEMP TABLE uploads_with_long_filenames AS + SELECT ROW_NUMBER() OVER(ORDER BY id) row_id, path, id + FROM uploads AS u + WHERE LENGTH((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1]) > 246; + + CREATE INDEX ON uploads_with_long_filenames(row_id); + + UPDATE uploads + SET + path = + CONCAT( + COALESCE((regexp_match(updatable_uploads.path, '(.*\/).*'))[1], ''), + CONCAT( + LEFT(SPLIT_PART((regexp_match(updatable_uploads.path, '[^\\/:*?"<>|\r\n]+$'))[1], '.', 1), 242), + COALESCE(SUBSTRING((regexp_match(updatable_uploads.path, '[^\\/:*?"<>|\r\n]+$'))[1] FROM '\.(?:.(?!\.))+$')) + ) + ) + FROM + uploads_with_long_filenames AS updatable_uploads + WHERE + uploads.id = updatable_uploads.id + AND updatable_uploads.row_id > 0 AND updatable_uploads.row_id <= 10000; + ``` + + After you finish the batch update, you must change the batch size (`updatable_uploads.row_id`) using the following sequence of numbers (10000 to 20000). Repeat this process until you reach the last record in the `uploads` table. + +Truncate the file names in the references found: + +1. Check if those records are referenced somewhere. One way to do this is to dump the database and search for the parent directory name and file name: + + 1. To dump your database, you can use the following command as an example: + + ```shell + pg_dump -h /var/opt/gitlab/postgresql/ -d gitlabhq_production > gitlab-dump.tmp + ``` + + 1. Then you can search for the references using the `grep` command. Combining the parent directory and the file name can be a good idea. For example: + + ```shell + grep public/alongfilenamehere.txt gitlab-dump.tmp + ``` + +1. Replace those long file names using the new file names obtained from querying the `uploads` table. + +Truncate the file names on the file system. You must manually rename the files in your file system to the new file names obtained from querying the `uploads` table. + +### Re-run the backup task + +After following all the previous steps, re-run the backup task. + +## Restoring database backup fails when `pg_stat_statements` was previously enabled + +The GitLab backup of the PostgreSQL database includes all SQL statements required to enable extensions that were +previously enabled in the database. + +The `pg_stat_statements` extension can only be enabled or disabled by a PostgreSQL user with `superuser` role. +As the restore process uses a database user with limited permissions, it can't execute the following SQL statements: + +```sql +DROP EXTENSION IF EXISTS pg_stat_statements; +CREATE EXTENSION IF NOT EXISTS pg_stat_statements WITH SCHEMA public; +``` + +When trying to restore the backup in a PostgreSQL instance that doesn't have the `pg_stats_statements` extension, +the following error message is displayed: + +```plaintext +ERROR: permission denied to create extension "pg_stat_statements" +HINT: Must be superuser to create this extension. +ERROR: extension "pg_stat_statements" does not exist +``` + +When trying to restore in an instance that has the `pg_stats_statements` extension enabled, the cleaning up step +fails with an error message similar to the following: + +```plaintext +rake aborted! +ActiveRecord::StatementInvalid: PG::InsufficientPrivilege: ERROR: must be owner of view pg_stat_statements +/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:42:in `block (4 levels) in <top (required)>' +/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:41:in `each' +/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:41:in `block (3 levels) in <top (required)>' +/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:71:in `block (3 levels) in <top (required)>' +/opt/gitlab/embedded/bin/bundle:23:in `load' +/opt/gitlab/embedded/bin/bundle:23:in `<main>' +Caused by: +PG::InsufficientPrivilege: ERROR: must be owner of view pg_stat_statements +/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:42:in `block (4 levels) in <top (required)>' +/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:41:in `each' +/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:41:in `block (3 levels) in <top (required)>' +/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:71:in `block (3 levels) in <top (required)>' +/opt/gitlab/embedded/bin/bundle:23:in `load' +/opt/gitlab/embedded/bin/bundle:23:in `<main>' +Tasks: TOP => gitlab:db:drop_tables +(See full trace by running task with --trace) +``` + +### Prevent the dump file to include `pg_stat_statements` + +To prevent the inclusion of the extension in the PostgreSQL dump file that is part of the backup bundle, +enable the extension in any schema except the `public` schema: + +```sql +CREATE SCHEMA adm; +CREATE EXTENSION pg_stat_statements SCHEMA adm; +``` + +If the extension was previously enabled in the `public` schema, move it to a new one: + +```sql +CREATE SCHEMA adm; +ALTER EXTENSION pg_stat_statements SET SCHEMA adm; +``` + +To query the `pg_stat_statements` data after changing the schema, prefix the view name with the new schema: + +```sql +SELECT * FROM adm.pg_stat_statements limit 0; +``` + +To make it compatible with third-party monitoring solutions that expect it to be enabled in the `public` schema, +you need to include it in the `search_path`: + +```sql +set search_path to public,adm; +``` + +### Fix an existing dump file to remove references to `pg_stat_statements` + +To fix an existing backup file, do the following changes: + +1. Extract from the backup the following file: `db/database.sql.gz`. +1. Decompress the file or use an editor that is capable of handling it compressed. +1. Remove the following lines, or similar ones: + + ```sql + CREATE EXTENSION IF NOT EXISTS pg_stat_statements WITH SCHEMA public; + ``` + + ```sql + COMMENT ON EXTENSION pg_stat_statements IS 'track planning and execution statistics of all SQL statements executed'; + ``` + +1. Save the changes and recompress the file. +1. Update the backup file with the modified `db/database.sql.gz`. diff --git a/doc/administration/configure.md b/doc/administration/configure.md index 8b4f8a9abc6..7ae088cd783 100644 --- a/doc/administration/configure.md +++ b/doc/administration/configure.md @@ -1,6 +1,7 @@ --- stage: Systems group: Distribution +description: Installation settings. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/administration/dedicated/index.md b/doc/administration/dedicated/index.md index ef9e53c8fb7..005017c6178 100644 --- a/doc/administration/dedicated/index.md +++ b/doc/administration/dedicated/index.md @@ -1,8 +1,8 @@ --- stage: SaaS Platforms group: GitLab Dedicated -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments" -description: 'Learn how to configure your GitLab Dedicated instance.' +description: IP allowlists, SAML, maintenance. +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Configure GitLab Dedicated **(ULTIMATE)** @@ -14,12 +14,15 @@ The instructions on this page guide you through: 1. Onboarding and initial setup of your GitLab Dedicated instance using [Switchboard](https://about.gitlab.com/direction/saas-platforms/switchboard/), the GitLab Dedicated portal. 1. Configuring your GitLab Dedicated instance including enabling and updating the settings for [available functionality](../../subscriptions/gitlab_dedicated/index.md#available-features). -Any functionality in the GitLab application that is not controlled by the SaaS environment can be configured by using the [Admin Panel](../../administration/admin_area.md). +Any functionality in the GitLab application that is not controlled by the SaaS environment can be configured by using the [Admin Area](../../administration/admin_area.md). Examples of SaaS environment settings include `gitlab.rb` configurations and access to shell, Rails console, and PostgreSQL console. These environment settings cannot be changed by tenants. GitLab Dedicated Engineers also don't have direct access to tenant environments, except for [break glass situations](../../subscriptions/gitlab_dedicated/index.md#access-controls). +NOTE: +An instance refers to a GitLab Dedicated deployment, whereas a tenant refers to a customer. + ## Onboarding to GitLab Dedicated using Switchboard To create a new GitLab Dedicated environment for your organization, provide the following information to your account team: @@ -29,31 +32,33 @@ To create a new GitLab Dedicated environment for your organization, provide the - Email addresses of the users who are responsible to complete the onboarding and create your GitLab Dedicated instance using [Switchboard](https://about.gitlab.com/direction/saas-platforms/switchboard/). If you've been granted access to Switchboard, you receive an email invitation with temporary credentials to sign in. +Your invitation to Switchboard is valid for seven days. If you are having issues accessing +Switchboard, or if your invitation has expired, please [submit a support ticket](https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=4414917877650). NOTE: The credentials for Switchboard are separate from any other GitLab credentials you may already have to sign in to a GitLab self-managed or GitLab.com instance. -After you first sign in to Switchboard, you must update your password and set up MFA before you can complete your onboarding to create a tenant. +After you first sign in to Switchboard, you must update your password and set up MFA before you can complete your onboarding to create a new instance. -The following stages guide you through a series of four steps to provide the information required to create your GitLab Dedicated tenant. +The following stages guide you through a series of four steps to provide the information required to create your GitLab Dedicated instance. 1. Confirm account details: Confirm key attributes of your GitLab Dedicated account: - Reference architecture: Corresponds with the number of users you provided to your account team when beginning the onboarding process. For more information, see [reference architectures](../../subscriptions/gitlab_dedicated/index.md#availability-and-scalability). - Total repository storage size: Corresponds with the storage size you provided to your account team when beginning the onboarding process. - If you need to make changes to these attributes, [submit a support ticket](https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=4414917877650). -1. Tenant configuration: Provides the minimum required information needed to create your GitLab Dedicated tenant: +1. Tenant configuration: Provides the minimum required information needed to create your GitLab Dedicated instance: - Desired instance subdomain: The main domain for GitLab Dedicated instances is `gitlab-dedicated.com`. You choose the subdomain name where your instance is accessible from. For example, `customer_name.gitlab-dedicated.com`. - Desired primary region: Primary AWS region in which your data is stored. Note the [available AWS regions](../../subscriptions/gitlab_dedicated/index.md#available-aws-regions). - Desired secondary region: Secondary AWS region in which your data is stored. This region is used to recover your GitLab Dedicated instance in case of a disaster. - Desired backup region: An AWS region where the primary backups of your data are replicated. This can be the same as the primary or secondary region, or different. - Desired maintenance window: A weekly four-hour time slot that GitLab uses to perform routine maintenance and upgrade operations on all tenant instances. For more information, see [maintenance windows](#maintenance-window). 1. Security: You can provide your own [KMS keys](https://docs.aws.amazon.com/kms/latest/developerguide/overview.html) for encrypted AWS services. If you choose not to provide KMS keys, encryption keys are generated for your instance when it is created. For more information, see [encrypting your data at rest](#encrypted-data-at-rest-byok). -1. Summary: You confirm that the information you've provided in the previous steps is accurate before initiating the creation of your tenant. +1. Summary: You confirm that the information you've provided in the previous steps is accurate before initiating the creation of your instance. NOTE: -Some configuration settings (like the option to bring your own keys and your tenant name) are permanent and cannot be changed once your tenant has been created. +Some configuration settings (like the option to bring your own keys and your tenant name) are permanent and cannot be changed once your instance has been created. -It can take up to 3 hours to create the GitLab Dedicated tenant. When the setup is complete, you will receive a confirmation email with further instructions on how to access your tenant. +It can take up to 3 hours to create the GitLab Dedicated instance. When the setup is complete, you will receive a confirmation email with further instructions on how to access your instance. ### Maintenance window @@ -80,10 +85,11 @@ In an event of a platform outage, degradation or a security event requiring urge emergency maintenance will be carried out per [the emergency change processes](https://about.gitlab.com/handbook/engineering/infrastructure/emergency-change-processes/). -The emergency maintenance is initiated urgently when urgent actions need to be executed by GitLab -on a Dedicated tenant instance. -Communication with the customer will be provided on best effort basis prior to commencing the -maintenance, and full communication will follow after the immediate action is carried out. +The emergency maintenance is initiated when urgent actions need to be executed by GitLab on a +Dedicated tenant instance. Communication with the customer will be provided on best effort basis +prior to commencing the maintenance, and full communication will follow after the immediate action +is carried out. The GitLab Support Team will create a new ticket and send a message to the email +addresses of the users listed in Switchboard during [onboarding](#onboarding-to-gitlab-dedicated-using-switchboard). For example, when a critical security process is initiated to address an S1 vulnerability in GitLab, emergency maintenance is carried out to upgrade GitLab to the non-vulnerable version and that @@ -218,7 +224,7 @@ Make sure the AWS KMS keys are replicated to your desired primary, secondary and Configuration changes requested with a [support ticket](https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=4414917877650) are batched up and applied during your environment's weekly four-hour maintenance window. -This policy does not apply to configuration changes made by a GitLab Dedicated tenant admin [using Switchboard](#configuration-changes-in-switchboard). +This policy does not apply to configuration changes made by a GitLab Dedicated instance admin [using Switchboard](#configuration-changes-in-switchboard). To have a change considered for an upcoming weekly maintenance window, all required information must be submitted in full two business days before the start of the window. @@ -232,7 +238,7 @@ Changes requested with a support ticket cannot be applied outside of a weekly ma ### Configuration changes in Switchboard -Switchboard empowers the user to make limited configuration changes to their Dedicated Tenant Instance. As Switchboard matures further configuration changes will be made available. +Switchboard empowers the user to make limited configuration changes to their GitLab Dedicated instance. As Switchboard matures further configuration changes will be made available. To change or update the configuration of your GitLab Dedicated instance, use Switchboard following the instructions in the relevant section or open a [support ticket](https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=4414917877650) with your request. @@ -271,12 +277,14 @@ Consider the following when using Outbound Private Links: and repositories, or use any other GitLab functionality to access private services. - You can only establish Private Links between VPCs in the same region. Therefore, you can only establish a connection in the regions you selected for your Dedicated instance. -- The Network Load Balancer (NLB) that backs the Endpoint Service at your end must be enabled in at least one of the Availability Zones to which your Dedicated tenant was +- The Network Load Balancer (NLB) that backs the Endpoint Service at your end must be enabled in at least one of the Availability Zones to which your Dedicated instance was deployed. This is not the user-facing name such as `us-east-1a`, but the underlying [Availability Zone ID](https://docs.aws.amazon.com/ram/latest/userguide/working-with-az-ids.html). If you did not specify these during onboarding to Dedicated, you must either: - Ask for the Availability Zone IDs in the ticket you raise to enable the link and ensure the NLB is enabled in those AZs, or - Ensure the NLB has is enabled in every Availability Zone in the region. +You can view the `Reverse Private Link IAM Principal` attribute in the **Tenant Details** section of Switchboard. + To enable an Outbound Private Link: 1. [Create the Endpoint service](https://docs.aws.amazon.com/vpc/latest/privatelink/create-endpoint-service.html) through which your internal service @@ -300,7 +308,7 @@ To enable an Outbound Private Link: provide a list of DNS names that should resolve to the Private Link Endpoint. This list can be updated as needed in future. GitLab then configures the tenant instance to create the necessary Endpoint Interfaces based on the service names you provided. Any matching outbound -connections made from the tenant GitLab instance are directed through the PrivateLink into your VPC. +connections made from the tenant instance are directed through the PrivateLink into your VPC. ### Custom certificates @@ -348,7 +356,7 @@ Specify a comma separated list of IP addresses that can access your GitLab Dedic ### SAML NOTE: -GitLab Dedicated supports a limited number of SAML parameters. Parameters not shown in the configuration below are unavailable for GitLab Dedicated tenant instances. +GitLab Dedicated supports a limited number of SAML parameters. Parameters not shown in the configuration below are unavailable for GitLab Dedicated instances. Prerequisites: @@ -415,7 +423,7 @@ To activate SAML for your GitLab Dedicated instance: #### Request signing -If [SAML request signing](../../integration/saml.md#sign-saml-authentication-requests-optional) is desired, a certificate must be obtained. This certificate can be self-signed which has the advantage of not having to prove ownership of an arbitrary Common Name (CN) to a public Certificate Authority (CA)). +If [SAML request signing](../../integration/saml.md#sign-saml-authentication-requests-optional) is desired, a certificate must be obtained. This certificate can be self-signed which has the advantage of not having to prove ownership of an arbitrary Common Name (CN) to a public Certificate Authority (CA). If you choose to enable SAML request signing, the manual steps below will need to be completed before you are able to use SAML, since it requires certificate signing to happen. To enable SAML request signing, indicate on your SAML [support ticket](https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=4414917877650) that you want request signing enabled. GitLab works with you on sending the Certificate Signing Request (CSR) for you to sign. Alternatively, the CSR can be signed with a public CA. After the certificate is signed, GitLab adds the certificate and its associated private key to the `security` section of the SAML configuration. Authentication requests from GitLab to your identity provider can then be signed. @@ -434,12 +442,12 @@ To enable group sync: 1. Add the [required elements](../../user/group/saml_sso/group_sync.md#configure-saml-group-sync) to the SAML configuration block you provide in your [support ticket](https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=4414917877650). 1. Configure the [Group Links](../../user/group/saml_sso/group_sync.md#configure-saml-group-links). -### Add users to a tenant instance +### Add users to an instance -Tenant administrators can add Switchboard users to their tenant instance. There are two types of users: +Administrators can add Switchboard users to their GitLab Dedicated instance. There are two types of users: -- **Read only**: Users can only view tenant data. -- **Admin**: Users can edit the tenant configuration and manage users. +- **Read only**: Users can only view instance data. +- **Admin**: Users can edit the instance configuration and manage users. To add a new user to your GitLab Dedicated instance: diff --git a/doc/administration/docs_self_host.md b/doc/administration/docs_self_host.md index 5b9bfe82294..30ec21ef660 100644 --- a/doc/administration/docs_self_host.md +++ b/doc/administration/docs_self_host.md @@ -175,7 +175,7 @@ documentation URL requests as needed. For example, if your GitLab version is - The GitLab documentation URL becomes `http://0.0.0.0:4000/14.5/`. - The link in GitLab displays as `<instance_url>/help/administration/settings/help_page#destination-requirements`. - When you select the link, you are redirected to -`http://0.0.0.0:4000/14.5/ee/administration/settings/help_page/#destination-requirements`. + `http://0.0.0.0:4000/14.5/ee/administration/settings/help_page/#destination-requirements`. To test the setting, in GitLab, select a **Learn more** link. For example: diff --git a/doc/administration/external_users.md b/doc/administration/external_users.md index d8951631b72..b6744e56f4f 100644 --- a/doc/administration/external_users.md +++ b/doc/administration/external_users.md @@ -15,12 +15,8 @@ External users: - Cannot create project, groups, and snippets in their personal namespaces. - Can only create projects (including forks), subgroups, and snippets within top-level groups to which they are explicitly granted access. -- Can only access public projects and projects to which they are explicitly granted access, - thus hiding all other internal or private ones from them (like being - logged out). -- Can only access public groups and groups to which they are explicitly granted access, - thus hiding all other internal or private ones from them (like being - logged out). +- Can access public groups and public projects. +- Can only access projects and groups to which they are explicitly granted access. External users cannot access internal or private projects or groups that they are not granted access to. - Can only access public snippets. Access can be granted by adding the user as member to the project or group. diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index f97c892c3eb..7949bd498a7 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -71,12 +71,6 @@ must disable the **primary** site. ### Step 3. Promoting a **secondary** site -WARNING: -In GitLab 13.2 and 13.3, promoting a secondary site to a primary while the -secondary is paused fails. Do not pause replication before promoting a -secondary. If the secondary site is paused, be sure to resume before promoting. -This issue has been fixed in GitLab 13.4 and later. - Note the following when promoting a secondary: - If the secondary site [has been paused](../../geo/index.md#pausing-and-resuming-replication), the promotion @@ -90,7 +84,7 @@ Note the following when promoting a secondary: [troubleshooting advice](../replication/troubleshooting.md#fixing-errors-during-a-failover-or-when-promoting-a-secondary-to-a-primary-site). - You should [point the primary domain DNS at the newly promoted site](#step-4-optional-updating-the-primary-domain-dns-record). Otherwise, runners must be registered again with the newly promoted site, and all Git remotes, bookmarks, and external integrations must be updated. -#### Promoting a **secondary** site running on a single node running GitLab 14.5 and later +#### Promoting a **secondary** site running on a single node 1. SSH in to your **secondary** site and execute: @@ -110,63 +104,7 @@ Note the following when promoting a secondary: previously for the **secondary** site. 1. If successful, the **secondary** site is now promoted to the **primary** site. -#### Promoting a **secondary** site running on a single node running GitLab 14.4 and earlier - -WARNING: -The `gitlab-ctl promote-to-primary-node` and `gitlab-ctl promoted-db` commands are -deprecated in GitLab 14.5 and later, and [removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/345207). -Use `gitlab-ctl geo promote` instead. - -1. SSH in to your **secondary** site and login as root: - - ```shell - sudo -i - ``` - -1. If you're using GitLab 13.5 and later, skip this step. If not, edit - `/etc/gitlab/gitlab.rb` and remove any of the following lines that - might be present: - - ```ruby - geo_secondary_role['enable'] = true - roles ['geo_secondary_role'] - ``` - -1. Promote the **secondary** site to the **primary** site: - - - To promote the secondary site to primary along with [preflight checks](planned_failover.md#preflight-checks): - - ```shell - gitlab-ctl promote-to-primary-node - ``` - - - If you have already run the preflight checks separately or don't want to run them, - you can skip them with: - - ```shell - gitlab-ctl promote-to-primary-node --skip-preflight-checks - ``` - - NOTE: - In GitLab 13.7 and earlier, if you have a data type with zero items to sync - and don't skip the preflight checks, promoting the secondary reports - `ERROR - Replication is not up-to-date` even if replication is actually - up-to-date. If replication and verification output - shows that it is complete, you can skip the preflight checks to make the - command complete promotion. This bug was fixed in GitLab 13.8 and later. - - - To promote the secondary site to primary **without any further confirmation**, - even when preflight checks fail: - - ```shell - gitlab-ctl promote-to-primary-node --force - ``` - -1. Verify you can connect to the newly-promoted **primary** site using the URL used - previously for the **secondary** site. -1. If successful, the **secondary** site is now promoted to the **primary** site. - -#### Promoting a **secondary** site with multiple nodes running GitLab 14.5 and later +#### Promoting a **secondary** site with multiple nodes 1. SSH to every Sidekiq, PostgreSQL, and Gitaly node in the **secondary** site and run one of the following commands: @@ -200,49 +138,7 @@ Use `gitlab-ctl geo promote` instead. previously for the **secondary** site. 1. If successful, the **secondary** site is now promoted to the **primary** site. -#### Promoting a **secondary** site with multiple nodes running GitLab 14.4 and earlier - -WARNING: -The `gitlab-ctl promote-to-primary-node` and `gitlab-ctl promoted-db` commands are -deprecated in GitLab 14.5 and later, and [removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/345207). -Use `gitlab-ctl geo promote` instead. - -The `gitlab-ctl promote-to-primary-node` command cannot be used yet in -conjunction with multiple nodes, as it can only perform changes on -a **secondary** with only a single node. Instead, you must -do this manually. - -1. SSH in to the database node in the **secondary** site and trigger PostgreSQL to - promote to read-write: - - ```shell - sudo gitlab-ctl promote-db - ``` - -1. Edit `/etc/gitlab/gitlab.rb` on every node in the **secondary** site to - reflect its new status as **primary** by removing any of the following - lines that might be present: - - ```ruby - geo_secondary_role['enable'] = true - roles ['geo_secondary_role'] - ``` - - After making these changes, [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) - on each machine so the changes take effect. - -1. Promote the **secondary** to **primary**. SSH into a single application - server and execute: - - ```shell - sudo gitlab-rake geo:set_secondary_as_primary - ``` - -1. Verify you can connect to the newly-promoted **primary** using the URL used - previously for the **secondary**. -1. If successful, the **secondary** site is now promoted to the **primary** site. - -#### Promoting a **secondary** site with a Patroni standby cluster running GitLab 14.5 and later +#### Promoting a **secondary** site with a Patroni standby cluster 1. SSH to every Sidekiq, PostgreSQL, and Gitaly node in the **secondary** site and run one of the following commands: @@ -276,54 +172,7 @@ do this manually. previously for the **secondary** site. 1. If successful, the **secondary** site is now promoted to the **primary** site. -#### Promoting a **secondary** site with a Patroni standby cluster running GitLab 14.4 and earlier - -WARNING: -The `gitlab-ctl promote-to-primary-node` and `gitlab-ctl promoted-db` commands are -deprecated in GitLab 14.5 and later, and [removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/345207). -Use `gitlab-ctl geo promote` instead. - -The `gitlab-ctl promote-to-primary-node` command cannot be used yet in -conjunction with a Patroni standby cluster, as it can only perform changes on -a **secondary** with only a single node. Instead, you must do this manually. - -1. SSH in to the Standby Leader database node in the **secondary** site and trigger PostgreSQL to - promote to read-write: - - ```shell - sudo gitlab-ctl promote-db - ``` - -1. Edit `/etc/gitlab/gitlab.rb` on every application and Sidekiq nodes in the secondary to reflect its new status as primary by removing any of the following lines that might be present: - - ```ruby - geo_secondary_role['enable'] = true - roles ['geo_secondary_role'] - ``` - -1. Edit `/etc/gitlab/gitlab.rb` on every Patroni node in the secondary to disable the standby cluster: - - ```ruby - patroni['standby_cluster']['enable'] = false - ``` - -1. Reconfigure GitLab on each machine for the changes to take effect: - - ```shell - sudo gitlab-ctl reconfigure - ``` - -1. Promote the **secondary** to **primary**. SSH into a single application server and execute: - - ```shell - sudo gitlab-rake geo:set_secondary_as_primary - ``` - -1. Verify you can connect to the newly-promoted **primary** using the URL used - previously for the **secondary**. -1. If successful, the **secondary** site is now promoted to the **primary** site. - -#### Promoting a **secondary** site with an external PostgreSQL database running GitLab 14.5 and later +#### Promoting a **secondary** site with an external PostgreSQL database The `gitlab-ctl geo promote` command can be used in conjunction with an external PostgreSQL database. In this case, you must first manually promote the replica database associated @@ -388,68 +237,6 @@ with the **secondary** site: previously for the **secondary** site. 1. If successful, the **secondary** site is now promoted to the **primary** site. -#### Promoting a **secondary** site with an external PostgreSQL database running GitLab 14.4 and earlier - -WARNING: -The `gitlab-ctl promote-to-primary-node` and `gitlab-ctl promoted-db` commands are -deprecated in GitLab 14.5 and later, and [removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/345207). -Use `gitlab-ctl geo promote` instead. - -The `gitlab-ctl promote-to-primary-node` command cannot be used in conjunction with -an external PostgreSQL database, as it can only perform changes on a **secondary** -node with GitLab and the database on the same machine. As a result, a manual process is -required: - -1. Promote the replica database associated with the **secondary** site. This - sets the database to read-write. The instructions vary depending on where your database is hosted: - - [Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.Promote) - - [Azure PostgreSQL](https://learn.microsoft.com/en-us/azure/postgresql/single-server/how-to-read-replicas-portal#stop-replication) - - [Google Cloud SQL](https://cloud.google.com/sql/docs/mysql/replication/manage-replicas#promote-replica) - - For other external PostgreSQL databases, save the following script in your - secondary site, for example `/tmp/geo_promote.sh`, and modify the connection - parameters to match your environment. Then, execute it to promote the replica: - - ```shell - #!/bin/bash - - PG_SUPERUSER=postgres - - # The path to your pg_ctl binary. You may need to adjust this path to match - # your PostgreSQL installation - PG_CTL_BINARY=/usr/lib/postgresql/10/bin/pg_ctl - - # The path to your PostgreSQL data directory. You may need to adjust this - # path to match your PostgreSQL installation. You can also run - # `SHOW data_directory;` from PostgreSQL to find your data directory - PG_DATA_DIRECTORY=/etc/postgresql/10/main - - # Promote the PostgreSQL database and allow read/write operations - sudo -u $PG_SUPERUSER $PG_CTL_BINARY -D $PG_DATA_DIRECTORY promote - ``` - -1. Edit `/etc/gitlab/gitlab.rb` on every node in the **secondary** site to - reflect its new status as **primary** by removing any of the following - lines that might be present: - - ```ruby - geo_secondary_role['enable'] = true - roles ['geo_secondary_role'] - ``` - - After making these changes [Reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) - on each node so the changes take effect. - -1. Promote the **secondary** to **primary**. SSH into a single secondary application - node and execute: - - ```shell - sudo gitlab-rake geo:set_secondary_as_primary - ``` - -1. Verify you can connect to the newly-promoted **primary** using the URL used - previously for the **secondary**. -1. If successful, the **secondary** site is now promoted to the **primary** site. - ### Step 4. (Optional) Updating the primary domain DNS record Update DNS records for the primary domain to point to the **secondary** site. @@ -627,12 +414,6 @@ When updating a cloud-native Geo deployment, the process for updating any node t The following sections assume you are using the `gitlab` namespace. If you used a different namespace when setting up your cluster, you should also replace `--namespace gitlab` with your namespace. -WARNING: -In GitLab 13.2 and 13.3, promoting a secondary site to a primary while the -secondary is paused fails. Do not pause replication before promoting a -secondary. If the site is paused, be sure to resume before promoting. This -issue has been fixed in GitLab 13.4 and later. - ### Step 1. Permanently disable the **primary** cluster WARNING: @@ -671,10 +452,6 @@ If the secondary site [has been paused](../../geo/index.md#pausing-and-resuming- a point-in-time recovery to the last known state. Data that was created on the primary while the secondary was paused is lost. -::Tabs - -:::TabTitle For GitLab 14.5 and later - 1. For each node (such as PostgreSQL or Gitaly) outside of the **secondary** Kubernetes cluster using the Linux package, SSH into the node and run one of the following commands: @@ -709,46 +486,6 @@ Data that was created on the primary while the secondary was paused is lost. | ---- | ------------- | ------- | | `ENABLE_SILENT_MODE` | `false` | If `true`, enables [Silent Mode](../../silent_mode/index.md) before promotion (GitLab 16.4 and later) | -:::TabTitle For GitLab 14.4 and earlier - -1. SSH in to the database node in the **secondary** site and trigger PostgreSQL to - promote to read-write: - - ```shell - sudo gitlab-ctl promote-db - ``` - -1. Edit `/etc/gitlab/gitlab.rb` on the database node in the **secondary** site to - reflect its new status as **primary** by removing any lines that enabled the - `geo_secondary_role`: - - NOTE: - Depending on your architecture, these steps need to run on any GitLab node that is external to the **secondary** Kubernetes cluster. - - ```ruby - ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. - geo_secondary_role['enable'] = true - - ## In 11.5+ documentation, the role was enabled as follows. Remove this line. - roles ['geo_secondary_role'] - ``` - - After making these changes, [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) on the database node. - -1. Find the task runner pod: - - ```shell - kubectl --namespace gitlab get pods -lapp=task-runner - ``` - -1. Promote the secondary: - - ```shell - kubectl --namespace gitlab exec -ti gitlab-geo-task-runner-XXX -- gitlab-rake geo:set_secondary_as_primary - ``` - -::EndTabs - ### Step 3. Promote the **secondary** cluster 1. Update the existing cluster configuration. diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index 01e5c33a726..70bbc962ba0 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -68,19 +68,13 @@ site: ``` 1. Copy the backup tarball generated from your primary site to the `/var/opt/gitlab/backups` folder -on your secondary site. + on your secondary site. 1. On your secondary site, restore the registry following the [Restore GitLab](../../../administration/backup_restore/index.md#restore-gitlab) -documentation. + documentation. ## Preflight checks -NOTE: -In GitLab 13.7 and earlier, if you have a data type with zero items to sync, -this command reports `ERROR - Replication is not up-to-date` even if -replication is actually up-to-date. This bug was fixed in GitLab 13.8 and -later. - Run this command to list out all preflight checks and automatically check if replication and verification are complete before scheduling a planned failover to ensure the process goes smoothly: ```shell diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index 6609bbed5a6..811ac92de27 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -337,18 +337,9 @@ For answers to common questions, see the [Geo FAQ](replication/faq.md). ## Log files -Geo stores structured log messages in a `geo.log` file. For Linux package -installations, this file is at `/var/log/gitlab/gitlab-rails/geo.log`. +Geo stores structured log messages in a `geo.log` file. -This file contains information about when Geo attempts to sync repositories and files. Each line in the file contains a separate JSON entry that can be ingested into. For example, Elasticsearch or Splunk. - -For example: - -```json -{"severity":"INFO","time":"2017-08-06T05:40:16.104Z","message":"Repository update","project_id":1,"source":"repository","resync_repository":true,"resync_wiki":true,"class":"Gitlab::Geo::LogCursor::Daemon","cursor_delay_s":0.038} -``` - -This message shows that Geo detected that a repository update was needed for project `1`. +For more information on how to access and consume Geo logs, see the [Geo section in the log system documentation](../logs/index.md#geolog). ## Troubleshooting diff --git a/doc/administration/geo/replication/container_registry.md b/doc/administration/geo/replication/container_registry.md index b9bee753089..1eacca32902 100644 --- a/doc/administration/geo/replication/container_registry.md +++ b/doc/administration/geo/replication/container_registry.md @@ -170,3 +170,70 @@ To verify container registry replication is working, on the **secondary** site: The initial replication, or "backfill", is probably still in progress. You can monitor the synchronization process on each Geo site from the **primary** site's **Geo Nodes** dashboard in your browser. + +## Troubleshooting + +### Confirm that container registry replication is enabled + +This can be done with a check using the [Rails console](../../operations/rails_console.md#starting-a-rails-console-session): + +```ruby +Geo::ContainerRepositoryRegistry.replication_enabled? +``` + +### Missing container registry notification event + +1. When an image is pushed to the primary site's container registry, it should trigger a [Container Registry notification](../../../administration/packages/container_registry.md#configure-container-registry-notifications) +1. The primary site's container registry calls the primary site's API on `https://<example.com>/api/v4/container_registry_event/events` +1. The primary site inserts a record to the `geo_events` table with `replicable_name: 'container_repository', model_record_id: <ID of the container repository>`. +1. The record gets replicated by PostgreSQL to the secondary site's database. +1. The Geo Log Cursor service processes the new event and enqueues a Sidekiq job `Geo::EventWorker` + +To verify this is working correctly, push an image to the registry on the primary site, and run the following command on the Rails console to verify that the notification was received, and processed into an event: + +```ruby +Geo::Event.where(replicable_name: 'container_repository') +``` + +You can further verify this by checking `geo.log` for entries from `Geo::ContainerRepositorySyncService`. + +### Registry events logs response status 401 Unauthorized unaccepted + +`401 Unauthorized` errors indicate that the primary site's container registry notification is not accepted by the Rails application, preventing it from notifying GitLab that something was pushed. + +To fix this, make sure that the authorization headers being sent with the registry notification match what's configured on the primary site, as should be done during step [Configure primary site](#configure-primary-site). + +#### Registry error: `token from untrusted issuer: "<token>"` + +To replicate a container image, Sidekiq uses JWT to authenticate itself towards the container registry. Geo replication takes it as a prerequisite that the [container registry configuration](../../../administration/packages/container_registry.md) has been done correctly. + +Make sure that both sites share a single signing key pair, as instructed under [Configure secondary site](#configure-secondary-site), and that both container registries, plus primary and secondary sites are [all configured to use the same token issuer](../../../administration/packages/container_registry.md#configure-gitlab-and-registry-to-run-on-separate-nodes-linux-package-installations). + +On multinode deployments, make sure that the issuer configured on the Sidekiq node matches the value configured on the registries. + +### Manually trigger a container registry sync event + +To help with troubleshooting, you can manually trigger the container registry replication process by running the following commands on the secondary's Rails console: + +```ruby +registry = Geo::ContainerRepositoryRegistry.first # Choose a Geo registry entry +registry.replicator.sync_repository # Resync the container repository +pp registry.reload # Look at replication state fields + +#<Geo::ContainerRepositoryRegistry:0x00007f54c2a36060 + id: 1, + container_repository_id: 1, + state: "2", + retry_count: 0, + last_sync_failure: nil, + retry_at: nil, + last_synced_at: Thu, 28 Sep 2023 19:38:05.823680000 UTC +00:00, + created_at: Mon, 11 Sep 2023 15:38:06.262490000 UTC +00:00> +``` + +The `state` field represents sync state: + +- `"0"`: pending sync (usually means it was never synced) +- `"1"`: started sync (a sync job is currently running) +- `"2"`: successfully synced +- `"3"`: failed to sync diff --git a/doc/administration/geo/replication/location_aware_git_url.md b/doc/administration/geo/replication/location_aware_git_url.md index 6849a648991..2491b7c8bc8 100644 --- a/doc/administration/geo/replication/location_aware_git_url.md +++ b/doc/administration/geo/replication/location_aware_git_url.md @@ -51,8 +51,8 @@ In a Route53 Hosted Zone, traffic policies can be used to set up a variety of routing configurations. 1. Go to the -[Route53 dashboard](https://console.aws.amazon.com/route53/home) and select -**Traffic policies**. + [Route53 dashboard](https://console.aws.amazon.com/route53/home) and select + **Traffic policies**.  diff --git a/doc/administration/geo/replication/upgrading_the_geo_sites.md b/doc/administration/geo/replication/upgrading_the_geo_sites.md index aae35d45bde..4a72dcf0145 100644 --- a/doc/administration/geo/replication/upgrading_the_geo_sites.md +++ b/doc/administration/geo/replication/upgrading_the_geo_sites.md @@ -15,7 +15,7 @@ to Geo secondaries. This may result in a larger than expected downtime. Upgrading Geo sites involves performing: -1. [Version-specific upgrade steps](version_specific_upgrades.md), depending on the +1. [Version-specific upgrade steps](../../../update/index.md#version-specific-upgrading-instructions), depending on the version being upgraded to or from. 1. [General upgrade steps](#general-upgrade-steps), for all upgrades. diff --git a/doc/administration/geo/replication/version_specific_upgrades.md b/doc/administration/geo/replication/version_specific_upgrades.md deleted file mode 100644 index 13d00ced3cd..00000000000 --- a/doc/administration/geo/replication/version_specific_upgrades.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../../../update/index.md#version-specific-upgrading-instructions' -remove_date: '2023-12-01' ---- - -This document was moved to [another location](../../../update/index.md#version-specific-upgrading-instructions). - -<!-- This redirect file can be deleted after <2023-12-01>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/geo/secondary_proxy/index.md b/doc/administration/geo/secondary_proxy/index.md index 22bb7dd721d..376f3f3958e 100644 --- a/doc/administration/geo/secondary_proxy/index.md +++ b/doc/administration/geo/secondary_proxy/index.md @@ -79,11 +79,6 @@ In Kubernetes, you can [use the same domain under `global.hosts.domain` as for t NOTE: The feature flag described in this section is planned to be deprecated and removed in a future release. Support for read-only Geo secondary sites is proposed in [issue 366810](https://gitlab.com/gitlab-org/gitlab/-/issues/366810), you can upvote and share your use cases in that issue. -There are minor known issues linked in the -["Geo secondary proxying with separate URLs" epic](https://gitlab.com/groups/gitlab-org/-/epics/6865). -You can also add feedback in the epic about any use-cases that -are not possible anymore with proxying enabled. - If you run into issues, to disable this feature, disable the `geo_secondary_proxy_separate_urls` feature flag. 1. SSH into one node running Rails on your primary Geo site and run: @@ -121,7 +116,7 @@ for details. To use TLS certificates with Let's Encrypt, you can manually point the domain to one of the Geo sites, generate the certificate, then copy it to all other sites. -- Using Geo secondary sites to accelerate runners is not officially supported. Support for this functionality is planned and can be tracked in [epic 9779](https://gitlab.com/groups/gitlab-org/-/epics/9779). If a replication lag occurs between the primary and secondary site, and the pipeline ref is not available on the secondary site when the job is executed, the job will fail. +- Using Geo secondary sites to accelerate runners is experimental and is not recommended for production. It can be configured and tested by following the steps in [secondary proxy runners](runners.md). Progress toward general availability can be tracked in [epic 9779](https://gitlab.com/groups/gitlab-org/-/epics/9779). - When secondary proxying is used together with separate URLs, [signing in the secondary site using SAML](../replication/single_sign_on.md#saml-with-separate-url-with-proxying-enabled) @@ -150,25 +145,21 @@ It does not cover all data types. In this context, accelerated reads refer to read requests served from the secondary site, provided that the data is up to date for the component on the secondary site. If the data on the secondary site is determined to be out of date, the request is forwarded to the primary site. Read requests for components not listed in the table below are always automatically forwarded to the primary site. -| Feature / component | Accelerated reads? | -|:----------------------------------------------------|:-----------------------| -| Project, wiki, design repository (using the web UI) | **{dotted-circle}** No | -| Project, wiki repository (using Git) | **{check-circle}** Yes <sup>1</sup> | -| Project, Personal Snippet (using the web UI) | **{dotted-circle}** No | -| Project, Personal Snippet (using Git) | **{check-circle}** Yes <sup>1</sup> | -| Group wiki repository (using the web UI) | **{dotted-circle}** No | -| Group wiki repository (using Git) | **{check-circle}** Yes <sup>1</sup> | -| User uploads | **{dotted-circle}** No | -| LFS objects (using the web UI) | **{dotted-circle}** No | -| LFS objects (using Git) | **{check-circle}** Yes | -| Pages | **{dotted-circle}** No <sup>2</sup> | -| Advanced search (using the web UI) | **{dotted-circle}** No | -| Container registry | **{dotted-circle}** No <sup>3</sup>| - -1. Git reads are served from the local secondary while pushes get proxied to the primary. - Selective sync or cases where repositories don't exist locally on the Geo secondary throw a "not found" error. -1. Pages can use the same URL (without access control), but must be configured separately and are not proxied. -1. The container registry is only recommended for Disaster Recovery scenarios. If the secondary site's container registry is not up to date, the read request is served with old data as the request is not forwarded to the primary site. +| Feature / component | Accelerated reads? | Notes | +|:----------------------------------------------------|:------------------------------------| ------------| +| Project, wiki, design repository (using the web UI) | **{dotted-circle}** No | | +| Project, wiki repository (using Git) | **{check-circle}** Yes | Git reads are served from the local secondary while pushes get proxied to the primary. Selective sync or cases where repositories don't exist locally on the Geo secondary throw a "not found" error.| +| Project, Personal Snippet (using the web UI) | **{dotted-circle}** No | | +| Project, Personal Snippet (using Git) | **{check-circle}** Yes | Git reads are served from the local secondary while pushes get proxied to the primary. Selective sync or cases where repositories don't exist locally on the Geo secondary throw a "not found" error. | +| Group wiki repository (using the web UI) | **{dotted-circle}** No | | +| Group wiki repository (using Git) | **{check-circle}** Yes | Git reads are served from the local secondary while pushes get proxied to the primary. Selective sync or cases where repositories don't exist locally on the Geo secondary throw a "not found" error. | +| User uploads | **{dotted-circle}** No | | +| LFS objects (using the web UI) | **{dotted-circle}** No | | +| LFS objects (using Git) | **{check-circle}** Yes | | +| Pages | **{dotted-circle}** No | Pages can use the same URL (without access control), but must be configured separately and are not proxied.| +| Advanced search (using the web UI) | **{dotted-circle}** No | | +| Container registry | **{dotted-circle}** No | The container registry is only recommended for Disaster Recovery scenarios. If the secondary site's container registry is not up to date, the read request is served with old data as the request is not forwarded to the primary site. Accelerating the container registry is planned, please upvote or comment in the [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/365864) to indicate your interest or ask your GitLab representative to do so on your behalf. | +| Dependency Proxy | **{dotted-circle}** No | Read requests to a Geo secondary site's Dependency Proxy are always proxied to the primary site. | ## Disable Geo proxying diff --git a/doc/administration/geo/secondary_proxy/location_aware_external_url.md b/doc/administration/geo/secondary_proxy/location_aware_external_url.md index 4ff349d8741..2f55aa27dff 100644 --- a/doc/administration/geo/secondary_proxy/location_aware_external_url.md +++ b/doc/administration/geo/secondary_proxy/location_aware_external_url.md @@ -47,8 +47,8 @@ In a Route53 Hosted Zone, traffic policies can be used to set up a variety of routing configurations. To create a traffic policy: 1. Go to the -[Route53 dashboard](https://console.aws.amazon.com/route53/home) and select -**Traffic policies**. + [Route53 dashboard](https://console.aws.amazon.com/route53/home) and select + **Traffic policies**. 1. Select **Create traffic policy**. 1. Fill in the **Policy Name** field with `Single Git Host` and select **Next**. diff --git a/doc/administration/geo/secondary_proxy/runners.md b/doc/administration/geo/secondary_proxy/runners.md new file mode 100644 index 00000000000..6dfd61778e7 --- /dev/null +++ b/doc/administration/geo/secondary_proxy/runners.md @@ -0,0 +1,40 @@ +--- +stage: Systems +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Secondary runners **(PREMIUM SELF EXPERIMENT)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/415179) in GitLab 16.7 [with a flag](../../feature_flags.md) named `geo_proxy_check_pipeline_refs`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. +To make it available, an administrator can [enable the feature flag](../../feature_flags.md) named `geo_proxy_check_pipeline_refs`. On GitLab.com, this feature is not available. + +With [Geo proxying for secondary sites](index.md), it is possible to register a `gitlab-runner` with a secondary site. This offloads load from the primary instance. + +## Enable or disable secondary runners + +To enable secondary runners, SSH into a Rails node on the **primary** Geo site and run: + +```ruby +sudo gitlab-rails runner 'Feature.enable(:geo_proxy_check_pipeline_refs)' +``` + +To disable secondary runners, SSH into a Rails node on the **primary** Geo site and run: + +```ruby +sudo gitlab-rails runner `Feature.disable(:geo_proxy_check_pipeline_refs)` +``` + +## Use secondary runners with a Location Aware public URL (Unified URL) + +Using a [Location Aware public URL](location_aware_external_url.md), with the feature flag enabled works with no extra configuration. After you install and register a runner in the same location as a secondary site, it automatically talks to the closest site, and only proxies to the primary if the secondary is out of date. + +## Use secondary runners with separate URLs + +Using separate secondary URLs, the runners should be: + +1. Registered with the secondary external URL. +1. Configured with [`clone_url`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#how-clone_url-works) set to the `external_url` of the secondary instance. diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index 13615825a14..b9c2a69eaf7 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -622,7 +622,7 @@ If you still haven't [migrated from repmgr to Patroni](#migrating-from-repmgr-to 1. Before migrating, you should ensure there is no replication lag between the **primary** and **secondary** sites and that replication is paused. In GitLab 13.2 and later, you can pause and resume replication with `gitlab-ctl geo-replication-pause` and `gitlab-ctl geo-replication-resume` on a Geo secondary database node. 1. Follow the [instructions to migrate repmgr to Patroni](../../postgresql/replication_and_failover.md#switching-from-repmgr-to-patroni). When configuring Patroni on each **primary** site database node, add `patroni['replication_slots'] = { '<slot_name>' => 'physical' }` -to `gitlab.rb` where `<slot_name>` is the name of the replication slot for your **secondary** site. This ensures that Patroni recognizes the replication slot as permanent and doesn't drop it upon restarting. + to `gitlab.rb` where `<slot_name>` is the name of the replication slot for your **secondary** site. This ensures that Patroni recognizes the replication slot as permanent and doesn't drop it upon restarting. 1. If database replication to the **secondary** site was paused before migration, resume replication after Patroni is confirmed as working on the **primary** site. ### Migrating a single PostgreSQL node to Patroni diff --git a/doc/administration/get_started.md b/doc/administration/get_started.md index 336ed743a47..41783d55377 100644 --- a/doc/administration/get_started.md +++ b/doc/administration/get_started.md @@ -1,7 +1,8 @@ --- -info: For assistance with this tutorial, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-other-projects-and-subjects. stage: none group: Tutorials +description: Administration overview. +info: For assistance with this tutorial, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-other-projects-and-subjects. --- # Get started administering GitLab **(FREE SELF)** @@ -150,7 +151,7 @@ Backups of GitLab databases and file systems are taken every 24 hours, and are k - You can use the project export option in: - [The UI](../user/project/settings/import_export.md#export-a-project-and-its-data). - [The API](../api/project_import_export.md#schedule-an-export). -- [Group export by uploading a file export](../user/group/import/index.md#migrate-groups-by-uploading-an-export-file-deprecated) +- [Group export by uploading a file export](../user/project/settings/import_export.md#migrate-groups-by-uploading-an-export-file-deprecated) does **not** export the projects in it, but does export: - Epics - Milestones diff --git a/doc/administration/gitaly/concurrency_limiting.md b/doc/administration/gitaly/concurrency_limiting.md index 321bb9efe20..2b1a2bbef64 100644 --- a/doc/administration/gitaly/concurrency_limiting.md +++ b/doc/administration/gitaly/concurrency_limiting.md @@ -18,7 +18,7 @@ Enabling limits on your environment should be done with caution and only in select circumstances, such as to protect against unexpected traffic. When reached, limits _do_ result in disconnects that negatively impact users. For consistent and stable performance, you should first explore other options such as -adjusting node specifications, and [reviewing large repositories](../../user/project/repository/managing_large_repositories.md) or workloads. +adjusting node specifications, and [reviewing large repositories](../../user/project/repository/monorepos/index.md) or workloads. ## Limit RPC concurrency @@ -30,7 +30,7 @@ When cloning or pulling repositories, various RPCs run in the background. In par These RPCs can consume a large amount of resources, which can have a significant impact in situations such as: - Unexpectedly high traffic. -- Running against [large repositories](../../user/project/repository/managing_large_repositories.md) that don't follow best practices. +- Running against [large repositories](../../user/project/repository/monorepos/index.md) that don't follow best practices. You can limit these processes from overwhelming your Gitaly server in these scenarios using the concurrency limits in the Gitaly configuration file. For example: @@ -94,7 +94,7 @@ number of in-flight pack-object processes per remote IP address. WARNING: Only enable these limits on your environment with caution and only in select circumstances, such as to protect against unexpected traffic. When reached, these limits disconnect users. For consistent and stable performance, you should first explore other options such as adjusting node specifications, and -[reviewing large repositories](../../user/project/repository/managing_large_repositories.md) or workloads. +[reviewing large repositories](../../user/project/repository/monorepos/index.md) or workloads. Example configuration: diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md index d89413b2cf4..eb620ff7413 100644 --- a/doc/administration/gitaly/configure_gitaly.md +++ b/doc/administration/gitaly/configure_gitaly.md @@ -13,7 +13,7 @@ Configure Gitaly in one of two ways: :::TabTitle Linux package (Omnibus) 1. Edit `/etc/gitlab/gitlab.rb` and add or change the - [Gitaly settings](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/1dd07197c7e5ae23626aad5a4a070a800b670380/files/gitlab-config-template/gitlab.rb.template#L1622-1676). + [Gitaly settings](https://gitlab.com/gitlab-org/gitaly/-/blob/master/config.toml.example). 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). :::TabTitle Self-compiled (source) @@ -574,7 +574,7 @@ Enabling limits on your environment should be done with caution and only in select circumstances, such as to protect against unexpected traffic. When reached, limits _do_ result in disconnects that negatively impact users. For consistent and stable performance, you should first explore other options such as -adjusting node specifications, and [reviewing large repositories](../../user/project/repository/managing_large_repositories.md) or workloads. +adjusting node specifications, and [reviewing large repositories](../../user/project/repository/monorepos/index.md) or workloads. When enabling cgroups for memory, you should ensure that no swap is configured on the Gitaly nodes as processes may switch to using that instead of being terminated. This situation could lead to notably compromised @@ -1019,9 +1019,9 @@ meaning that unique requests do not get written into the cache. If you: - Increase this number, your cache hit rate goes down and the -cache uses less disk space. + cache uses less disk space. - Decrease this number, your cache hit -rate goes up and the cache uses more disk space. + rate goes up and the cache uses more disk space. You should set `min_occurrences` to `1`. On GitLab.com, going from 0 to 1 saved us 50% cache disk space while barely affecting @@ -1089,6 +1089,33 @@ Example: } ``` +## `cat-file` cache + +A lot of Gitaly RPCs need to look up Git objects from repositories. +Most of the time we use `git cat-file --batch` processes for that. For +better performance, Gitaly can re-use these `git cat-file` processes +across RPC calls. Previously used processes are kept around in a +["Git cat-file cache"](https://about.gitlab.com/blog/2019/07/08/git-performance-on-nfs/#enter-cat-file-cache). +To control how much system resources this uses, we have a maximum number of +cat-file processes that can go into the cache. + +The default limit is 100 `cat-file`s, which constitute a pair of +`git cat-file --batch` and `git cat-file --batch-check` processes. If +you see errors about "too many open files", or an +inability to create new processes, you may want to lower this limit. + +Ideally, the number should be large enough to handle standard +traffic. If you raise the limit, you should measure the cache hit ratio +before and after. If the hit ratio does not improve, the higher limit is +probably not making a meaningful difference. Here is an example +Prometheus query to see the hit rate: + +```plaintext +sum(rate(gitaly_catfile_cache_total{type="hit"}[5m])) / sum(rate(gitaly_catfile_cache_total{type=~"(hit)|(miss)"}[5m])) +``` + +Configure the `cat-file` cache in the [Gitaly configuration file](reference.md). + ## Repository consistency checks Gitaly runs repository consistency checks: diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index 66e220dacc2..112c441164d 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -949,8 +949,8 @@ You can also appoint an authoritative name server by setting it in this format: 1. Save the file and [reconfigure](../restart_gitlab.md#reconfigure-a-linux-package-installation). 1. On the Praefect clients (except Gitaly servers), edit `git_data_dirs` in -`/etc/gitlab/gitlab.rb` as follows. Replace `PRAEFECT_SERVICE_DISCOVERY_ADDRESS` -with Praefect service discovery address, such as `praefect.service.consul`. + `/etc/gitlab/gitlab.rb` as follows. Replace `PRAEFECT_SERVICE_DISCOVERY_ADDRESS` + with Praefect service discovery address, such as `praefect.service.consul`. ```ruby git_data_dirs({ diff --git a/doc/administration/gitaly/reference.md b/doc/administration/gitaly/reference.md index 946be6c8af3..169b6e00813 100644 --- a/doc/administration/gitaly/reference.md +++ b/doc/administration/gitaly/reference.md @@ -4,256 +4,24 @@ group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Gitaly reference **(FREE SELF)** +# Example configuration files **(FREE SELF)** -Gitaly is configured via a [TOML](https://github.com/toml-lang/toml) -configuration file. Unlike self-compiled installations, in Linux package installations you -would not edit this file directly. For Linux package installations, the default file location is `/var/opt/gitlab/gitaly/config.toml`. +Gitaly and Gitaly Cluster are configured by using configuration files. The default location of the configuration files +depends on the type of installation you have: -The configuration file is passed as an argument to the `gitaly` executable, which is usually done by either your Linux -package installation or your [init](https://en.wikipedia.org/wiki/Init) script. +- For Linux package installations, the default location for Gitaly and Gitaly Cluster configuration is in the + `/etc/gitlab/gitlab.rb` Ruby file. +- For self-compiled, the default location for Gitaly and Gitaly Cluster configuration is in the + `/home/git/gitaly/config.toml` and `/home/git/gitaly/config.prafect.toml` TOML files. -An [example configuration file](https://gitlab.com/gitlab-org/gitaly/blob/master/config.toml.example) -can be found in the Gitaly project. +You can find example TOML configuration files in the `gitaly` project for: -## Format +- Gitaly: <https://gitlab.com/gitlab-org/gitaly/-/blob/master/config.toml.example> +- Gitaly Cluster: <https://gitlab.com/gitlab-org/gitaly/-/blob/master/config.praefect.toml.example> -At the top level, `config.toml` defines the items described on the table below. +If you are configuring a Linux package installation, you must convert the examples into Ruby to use them. -| Name | Type | Required | Description | -| ---- | ---- | -------- | ----------- | -| `socket_path` | string | yes (if `listen_addr` is not set) | A path which Gitaly should open a Unix socket. | -| `listen_addr` | string | yes (if `socket_path` is not set) | TCP address for Gitaly to listen on. | -| `tls_listen_addr` | string | no | TCP over TLS address for Gitaly to listen on. | -| `bin_dir` | string | yes | Directory containing Gitaly executables. | -| `prometheus_listen_addr` | string | no | TCP listen address for Prometheus metrics. If not set, no Prometheus listener is started. | +For more information on: -For example: - -```toml -socket_path = "/home/git/gitlab/tmp/sockets/private/gitaly.socket" -listen_addr = "localhost:9999" -tls_listen_addr = "localhost:8888" -bin_dir = "/home/git/gitaly" -prometheus_listen_addr = "localhost:9236" -``` - -### Authentication - -Gitaly can be configured to reject requests that do not contain a -specific bearer token in their headers, which is a security measure to -be used when serving requests over TCP: - -```toml -[auth] -# A non-empty token enables authentication. -token = "the secret token" -``` - -Authentication is disabled when the token setting in `config.toml` is absent or -an empty string. - -It is possible to temporarily disable authentication with the `transitioning` -setting. This allows you to monitor if all clients are -authenticating correctly without causing a service outage for clients -that are still to be configured correctly: - -```toml -[auth] -token = "the secret token" -transitioning = true -``` - -WARNING: -Remember to disable `transitioning` when you are done -changing your token settings. - -All authentication attempts are counted in Prometheus under -the [`gitaly_authentications_total` metric](monitoring.md#queries). - -### TLS - -Gitaly supports TLS encryption. You need to bring your own certificates as -this isn't provided automatically. - -| Name | Type | Required | Description | -| ---- | ---- | -------- | ----------- | -| `certificate_path` | string | no | Path to the certificate. | -| `key_path` | string | no | Path to the key. | - -```toml -tls_listen_addr = "localhost:8888" - -[tls] -certificate_path = '/home/git/cert.cert' -key_path = '/home/git/key.pem' -``` - -[Read more](tls_support.md) about TLS in Gitaly. - -### Storage - -GitLab repositories are grouped into directories known as storages, such as -`/home/git/repositories`. They contain bare repositories managed -by GitLab with names, such as `default`. - -These names and paths are also defined in the `gitlab.yml` configuration file of -GitLab. When you run Gitaly on the same machine as GitLab (the default -and recommended configuration) storage paths defined in the Gitaly `config.toml` -must match those in `gitlab.yml`. - -| Name | Type | Required | Description | -| ---- | ---- | -------- | ----------- | -| `storage` | array | yes | An array of storage shards. | -| `path` | string | yes | The path to the storage shard. | -| `name` | string | yes | The name of the storage shard. | - -For example: - -```toml -[[storage]] -path = "/path/to/storage/repositories" -name = "my_shard" - -[[storage]] -path = "/path/to/other/repositories" -name = "other_storage" -``` - -### Git - -The following values can be set in the `[git]` section of the configuration file. - -| Name | Type | Required | Description | -| ---- | ---- | -------- | ----------- | -| `bin_path` | string | no | Path to Git binary. If not set, is resolved using `PATH`. | -| `catfile_cache_size` | integer | no | Maximum number of cached [cat-file processes](#cat-file-cache). Default is `100`. | -| `signing_key` | string | no | Path to [GPG signing key](configure_gitaly.md#configure-commit-signing-for-gitlab-ui-commits). If not set, Gitaly doesn't sign commits made using the UI. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/19185) in GitLab 15.4. | - -#### `cat-file` cache - -A lot of Gitaly RPCs need to look up Git objects from repositories. -Most of the time we use `git cat-file --batch` processes for that. For -better performance, Gitaly can re-use these `git cat-file` processes -across RPC calls. Previously used processes are kept around in a -["Git cat-file cache"](https://about.gitlab.com/blog/2019/07/08/git-performance-on-nfs/#enter-cat-file-cache). -To control how much system resources this uses, we have a maximum number of -cat-file processes that can go into the cache. - -The default limit is 100 `cat-file`s, which constitute a pair of -`git cat-file --batch` and `git cat-file --batch-check` processes. If -you are seeing errors complaining about "too many open files", or an -inability to create new processes, you may want to lower this limit. - -Ideally, the number should be large enough to handle standard -traffic. If you raise the limit, you should measure the cache hit ratio -before and after. If the hit ratio does not improve, the higher limit is -probably not making a meaningful difference. Here is an example -Prometheus query to see the hit rate: - -```plaintext -sum(rate(gitaly_catfile_cache_total{type="hit"}[5m])) / sum(rate(gitaly_catfile_cache_total{type=~"(hit)|(miss)"}[5m])) -``` - -#### Custom Hooks - -> Method of configuring custom hooks directory documented here is preferred from GitLab 16.4. `[gitlab-shell] dir` configuration is no longer required. - -Gitaly supports [custom Git hooks](../server_hooks.md) that are -used to perform tasks based on changes performed in any repository. The custom -hooks directory is configured in the `[hooks]` section: - -| Name | Type | Required | Description | -| ---- | ---- | -------- | ----------- | -| `custom_hooks_dir` | string | no | The directory where custom Git hooks are installed. | - -Example: - -```toml -[hooks] -custom_hooks_dir = "/home/git/custom-hooks" -``` - -If left unset, no custom hooks are used. - -### GitLab - -> Method of configuring custom hooks directory documented here is preferred from GitLab 16.4. `[gitlab-shell] dir` configuration is no longer required. - -Gitaly must connect to the GitLab application to perform access -checks when a user performs a change. The parameters to connect to GitLab must -be configured in the `[gitlab]` section. - -| Name | Type | Required | Description | -| ---- | ---- | -------- | ----------- | -| `url` | string | yes | The URL of the GitLab server. -| `secret` | string | no | The secret token used to authenticate with GitLab. | -| `secret_file` | string | no | The path of the file containing the secret token used to authenticate with GitLab. | - -Only one of `secret` or `secret_file` can be configured. - -### Prometheus - -You can optionally configure Gitaly to record histogram latencies on GRPC method -calls in Prometheus. - -| Name | Type | Required | Description | -| ---- | ---- | -------- | ----------- | -| `grpc_latency_buckets` | array | no | Prometheus stores each observation in a bucket, which means you'd get an approximation of latency. Optimizing the buckets gives more control over the accuracy of the approximation. | - -Example: - -```toml -prometheus_listen_addr = "localhost:9236" - -[prometheus] -grpc_latency_buckets = [0.001, 0.005, 0.025, 0.1, 0.5, 1.0, 10.0, 30.0, 60.0, 300.0, 1500.0] -``` - -### Logging - -The following values configure logging in Gitaly under the `[logging]` section. - -| Name | Type | Required | Description | -| ---- | ---- | -------- | ----------- | -| `format` | string | no | Log format: `text` or `json`. Default: `text`. | -| `level` | string | no | Log level: `debug`, `info`, `warn`, `error`, `fatal`, or `panic`. Default: `info`. | -| `sentry_dsn` | string | no | Sentry DSN (Data Source Name) for exception monitoring. | -| `sentry_environment` | string | no | [Sentry Environment](https://docs.sentry.io/product/sentry-basics/environments/) for exception monitoring. | - -While the main Gitaly application logs go to `stdout`, there are some extra log -files that go to a configured directory, like the GitLab Shell logs. -GitLab Shell does not support `panic` or `trace` level logs: - -- `panic` falls back to `error`. -- `trace` falls back to `debug`. -- Any other invalid log levels default to `info`. - -Example: - -```toml -[logging] -level = "warn" -dir = "/home/gitaly/logs" -format = "json" -sentry_dsn = "https://<key>:<secret>@sentry.io/<project>" -ruby_sentry_dsn = "https://<key>:<secret>@sentry.io/<project>" -``` - -## Concurrency - -You can adjust the `concurrency` of each RPC endpoint. - -| Name | Type | Required | Description | -| ---- | ---- | -------- | ----------- | -| `concurrency` | array | yes | An array of RPC endpoints. | -| `rpc` | string | no | The name of the RPC endpoint (`/gitaly.RepositoryService/GarbageCollect`). | -| `max_per_repo` | integer | no | Concurrency per RPC per repository. | - -Example: - -```toml -[[concurrency]] -rpc = "/gitaly.RepositoryService/GarbageCollect" -max_per_repo = 1 -``` +- Configuring Gitaly, see [Configure Gitaly](configure_gitaly.md). +- Configuring Gitaly Cluster, see [Configure Gitaly Cluster](praefect.md). diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index 7d0255a3efb..6c355f79191 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -224,7 +224,7 @@ the application might be fetching this secret from a different file. Your Gitaly If that setting is missing, GitLab defaults to using `.gitlab_shell_secret` under `/opt/gitlab/embedded/service/gitlab-rails/.gitlab_shell_secret`. -### Repository pushes fail +### Repository pushes fail with `401 Unauthorized` and `JWT::VerificationError` When attempting `git push`, you can see: @@ -240,7 +240,7 @@ When attempting `git push`, you can see: } ``` -This error occurs when the GitLab server has been upgraded to GitLab 15.5 or later but Gitaly has not yet been upgraded. +This combination of errors occurs when the GitLab server has been upgraded to GitLab 15.5 or later but Gitaly has not yet been upgraded. From GitLab 15.5, GitLab [authenticates with GitLab Shell using a JWT token instead of a shared secret](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/86148). You should follow the [recommendations on upgrading external Gitaly](../../update/plan_your_upgrade.md#external-gitaly) and upgrade Gitaly before the GitLab @@ -441,6 +441,23 @@ To resolve this, remove the `noexec` option from the file system mount. An alter Because Gitaly commit signing is headless and not associated with a specific user, the GPG signing key must be created without a passphrase, or the passphrase must be removed before export. +### Gitaly logs show errors in `info` messages + +Because of a bug [introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6201) in GitLab 16.3, additional entries were written to the +[Gitaly logs](../logs/index.md#gitaly-logs). These log entries contained `"level":"info"` but the `msg` string appeared to contain an error. + +For example: + +```json +{"level":"info","msg":"[core] [Server #3] grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"ServerHandshake(\\\"x.x.x.x:x\\\") failed: wrapped server handshake: EOF\"","pid":6145,"system":"system","time":"2023-12-14T21:20:39.999Z"} +``` + +The reason for this log entry is that the underlying gRPC library sometimes output verbose transportation logs. These log entries appear to be errors but are, in general, +safe to ignore. + +This bug was [fixed](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6513/) in GitLab 16.4.5, 16.5.5, and 16.6.0, which prevents these types of messages from +being written to the Gitaly logs. + ## Troubleshoot Praefect (Gitaly Cluster) The following sections provide possible solutions to Gitaly Cluster errors. diff --git a/doc/administration/integration/plantuml.md b/doc/administration/integration/plantuml.md index fbd3e636123..9e8bd323f5a 100644 --- a/doc/administration/integration/plantuml.md +++ b/doc/administration/integration/plantuml.md @@ -91,7 +91,7 @@ server to generate the diagrams: To run a PlantUML container in Docker, run this command: ```shell -docker run -d --name plantuml -p 8080:8080 plantuml/plantuml-server:tomcat +docker run -d --name plantuml -p 8005:8080 plantuml/plantuml-server:tomcat ``` The **PlantUML URL** is the hostname of the server running the container. @@ -99,7 +99,7 @@ The **PlantUML URL** is the hostname of the server running the container. When running GitLab in Docker, it must have access to the PlantUML container. To achieve that, use [Docker Compose](https://docs.docker.com/compose/). In this basic `docker-compose.yml` file, PlantUML is accessible to GitLab at the URL -`http://plantuml:8080/`: +`http://plantuml:8005/`: ```yaml version: "3" @@ -108,7 +108,7 @@ services: image: 'gitlab/gitlab-ee:12.2.5-ee.0' environment: GITLAB_OMNIBUS_CONFIG: | - nginx['custom_gitlab_server_config'] = "location /-/plantuml/ { \n rewrite ^/-/plantuml/(.*) /$1 break;\n proxy_cache off; \n proxy_pass http://plantuml:8080/; \n}\n" + nginx['custom_gitlab_server_config'] = "location /-/plantuml/ { \n rewrite ^/-/plantuml/(.*) /$1 break;\n proxy_cache off; \n proxy_pass http://plantuml:8005/; \n}\n" plantuml: image: 'plantuml/plantuml-server:tomcat' @@ -125,6 +125,8 @@ following: - `http://plantuml:8080/` - `http://localhost:8080/plantuml/` +- `http://plantuml:8005/` +- `http://localhost:8005/plantuml/` If you're running [GitLab with TLS](https://docs.gitlab.com/omnibus/settings/ssl/index.html) you must configure this redirection, because PlantUML uses the insecure HTTP protocol. @@ -137,7 +139,7 @@ To enable this redirection: ```ruby # Docker deployment - nginx['custom_gitlab_server_config'] = "location /-/plantuml/ { \n rewrite ^/-/plantuml/(.*) /$1 break;\n proxy_cache off; \n proxy_pass http://plantuml:8080/; \n}\n" + nginx['custom_gitlab_server_config'] = "location /-/plantuml/ { \n rewrite ^/-/plantuml/(.*) /$1 break;\n proxy_cache off; \n proxy_pass http://plantuml:8005/; \n}\n" ``` 1. To activate the changes, run the following command: diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 33dc2020331..0658af2360b 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -282,7 +282,7 @@ To migrate back to local storage, you must If [`artifacts:expire_in`](../ci/yaml/index.md#artifactsexpire_in) is used to set an expiry for the artifacts, they are marked for deletion right after that date passes. -Otherwise, they expire per the [default artifacts expiration setting](../administration/settings/continuous_integration.md). +Otherwise, they expire per the [default artifacts expiration setting](../administration/settings/continuous_integration.md#default-artifacts-expiration). Artifacts are deleted by the `expire_build_artifacts_worker` cron job which Sidekiq runs every 7 minutes (`*/7 * * * *` in [Cron](../topics/cron/index.md) syntax). diff --git a/doc/administration/job_artifacts_troubleshooting.md b/doc/administration/job_artifacts_troubleshooting.md index 138803a2675..6cdde87cc1d 100644 --- a/doc/administration/job_artifacts_troubleshooting.md +++ b/doc/administration/job_artifacts_troubleshooting.md @@ -24,7 +24,7 @@ reasons are: [Rake task for _orphaned_ artifact files](../raketasks/cleanup.md#remove-orphan-artifact-files) to remove these. This script should always find work to do, as it also removes empty directories (see above). - [Artifact housekeeping was changed significantly](#housekeeping-disabled-in-gitlab-146-to-152), - and you might need to enable a feature flag to used the updated system. + and you might need to enable a feature flag to use the updated system. - The [keep latest artifacts from most recent success jobs](../ci/jobs/job_artifacts.md#keep-artifacts-from-most-recent-successful-jobs) feature is enabled. @@ -220,7 +220,7 @@ end ### List projects by total size of job artifacts stored List the top 20 projects, sorted by the total size of job artifacts stored, by -running the following code in the Rails console (`sudo gitlab-rails console`): +running the following code in the [Rails console](operations/rails_console.md): ```ruby include ActionView::Helpers::NumberHelper @@ -235,7 +235,7 @@ number you want. ### List largest artifacts in a single project List the 50 largest job artifacts in a single project by running the following -code in the Rails console (`sudo gitlab-rails console`): +code in the [Rails console](operations/rails_console.md): ```ruby include ActionView::Helpers::NumberHelper @@ -272,8 +272,11 @@ To change the number of job artifacts listed, change the number in `limit(50)`. WARNING: These commands remove data permanently from database and storage. Before running them, we highly recommend seeking guidance from a Support Engineer, or running them in a test environment with a backup of the instance ready to be restored, just in case. -If you need to manually remove job artifacts associated with multiple jobs while -**retaining their job logs**, this can be done from the Rails console (`sudo gitlab-rails console`): +You can manually remove job artifacts associated with multiple completed jobs while +**retaining their job logs** from the [Rails console](operations/rails_console.md). +A completed job is any job with the status of success, failed, canceled, or skipped. + +To delete jobs completed before a specific date: 1. Select jobs to be deleted: @@ -297,7 +300,7 @@ If you need to manually remove job artifacts associated with multiple jobs while ["keep"](../ci/jobs/job_artifacts.md#download-job-artifacts). ```ruby - builds_to_clear = builds_with_artifacts.where("finished_at < ?", 1.week.ago) + builds_to_clear = builds_with_artifacts.where("finished_at < ?", 1.year.ago) builds_to_clear.find_each do |build| Ci::JobArtifacts::DeleteService.new(build).execute build.update!(artifacts_expire_at: Time.now) @@ -307,19 +310,16 @@ If you need to manually remove job artifacts associated with multiple jobs while In [GitLab 15.3 and earlier](https://gitlab.com/gitlab-org/gitlab/-/issues/372537), use the following instead: ```ruby - builds_to_clear = builds_with_artifacts.where("finished_at < ?", 1.week.ago) + builds_to_clear = builds_with_artifacts.where("finished_at < ?", 1.year.ago) builds_to_clear.find_each do |build| build.artifacts_expire_at = Time.now build.erase_erasable_artifacts! end ``` - `1.week.ago` is a Rails `ActiveSupport::Duration` method which calculates a new - date or time in the past. Other valid examples are: - - - `7.days.ago` - - `3.months.ago` - - `1.year.ago` + `1.year.ago` is a Rails [`ActiveSupport::Duration`](https://api.rubyonrails.org/classes/ActiveSupport/Duration.html) method. + Start with a long duration to reduce the risk of accidentally deleting artifacts that are still in use. + Rerun the deletion with shorter durations as needed, for example `3.months.ago`, `2.weeks.ago`, or `7.days.ago`. `erase_erasable_artifacts!` is a synchronous method, and upon execution the artifacts are immediately removed; they are not scheduled by a background queue. @@ -329,8 +329,11 @@ If you need to manually remove job artifacts associated with multiple jobs while WARNING: These commands remove data permanently from both the database and from disk. Before running them, we highly recommend seeking guidance from a Support Engineer, or running them in a test environment with a backup of the instance ready to be restored, just in case. -If you need to manually remove **all** job artifacts associated with multiple jobs, -**including job logs**, this can be done from the Rails console (`sudo gitlab-rails console`): +You can manually remove job artifacts associated with multiple completed jobs while +**retaining their job logs** from the [Rails console](operations/rails_console.md). +A completed job is any job with the status of success, failed, canceled, or skipped. + +To delete jobs completed before a specific date: 1. Select the jobs to be deleted: @@ -371,7 +374,7 @@ If you need to manually remove **all** job artifacts associated with multiple jo 1. Erase the job artifacts and logs older than a specific date: ```ruby - builds_to_clear = builds_with_artifacts.where("finished_at < ?", 1.week.ago) + builds_to_clear = builds_with_artifacts.where("finished_at < ?", 1.year.ago) builds_to_clear.find_each do |build| print "Ci::Build ID #{build.id}... " @@ -387,12 +390,9 @@ If you need to manually remove **all** job artifacts associated with multiple jo In [GitLab 15.3 and earlier](https://gitlab.com/gitlab-org/gitlab/-/issues/369132), replace `Ci::BuildEraseService.new(build, admin_user).execute` with `build.erase(erased_by: admin_user)`. - `1.week.ago` is a Rails `ActiveSupport::Duration` method which calculates a new - date or time in the past. Other valid examples are: - - - `7.days.ago` - - `3.months.ago` - - `1.year.ago` + `1.year.ago` is a Rails [`ActiveSupport::Duration`](https://api.rubyonrails.org/classes/ActiveSupport/Duration.html) method. + Start with a long duration to reduce the risk of accidentally deleting artifacts that are still in use. + Rerun the deletion with shorter durations as needed, for example `3.months.ago`, `2.weeks.ago`, or `7.days.ago`. ## Job artifact upload fails with error 500 diff --git a/doc/administration/labels.md b/doc/administration/labels.md index 25a86b8c2fc..dcecbb84c3d 100644 --- a/doc/administration/labels.md +++ b/doc/administration/labels.md @@ -6,7 +6,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Labels administration **(FREE SELF)** -To manage labels for the GitLab instance, in the Admin Area, on the left sidebar, select **Labels**. For more details on how to manage labels, see [Labels](../user/project/labels.md). +To manage labels for the GitLab instance: + +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Labels**. + +For more details on how to manage labels, see [Labels](../user/project/labels.md). Labels created in the Admin Area are automatically added to new projects. They are not available to new groups. diff --git a/doc/administration/license.md b/doc/administration/license.md index 1dc39d9ac23..9b95af28eed 100644 --- a/doc/administration/license.md +++ b/doc/administration/license.md @@ -36,6 +36,22 @@ To activate your instance with an activation code: The subscription is activated. +### Using one activation code for multiple instances + +You can use one activation code or license key for multiple self-managed instances if the users on +these instances are the same or are a subset of your licensed production instance. This means that if +you have a licensed production instance of GitLab, and other instances with the same list of users, the +production activation code applies, even if these users are configured in different groups and projects. + +### Uploading licenses for scaled architectures + +In a scaled architecture, upload the license file to one application instance only. The license is stored in the +database and is replicated to all your application instances so that you do not need to upload the license to all instances. + +### Uploading licenses for GitLab Geo + +When using GitLab Geo, you only need to upload the license to your primary Geo instance. The license is stored in the database and is replicated to all instances. + If you have an offline environment, [activate GitLab EE with a license file or key](license_file.md) instead. diff --git a/doc/administration/logs/index.md b/doc/administration/logs/index.md index 1b1591a36d9..08f31439b26 100644 --- a/doc/administration/logs/index.md +++ b/doc/administration/logs/index.md @@ -15,8 +15,8 @@ This guide talks about how to read and use these system log files. Read more about the log system and using the logs: - [Customize logging on Linux package installations](https://docs.gitlab.com/omnibus/settings/logs.html) -including adjusting log retention, log forwarding, -switching logs from JSON to plain text logging, and more. + including adjusting log retention, log forwarding, + switching logs from JSON to plain text logging, and more. - [How to parse and analyze JSON logs](../logs/log_parsing.md). ## Log Levels @@ -811,7 +811,7 @@ GraphQL queries are recorded in the file. For example: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/133371) in GitLab 16.5. The `clickhouse.log` file logs information related to the -ClickHouse database client in GitLab. +[ClickHouse database client](../../integration/clickhouse.md) in GitLab. ## `migrations.log` diff --git a/doc/administration/logs/tracing_correlation_id.md b/doc/administration/logs/tracing_correlation_id.md index f6b341ff114..a5f6b103d1f 100644 --- a/doc/administration/logs/tracing_correlation_id.md +++ b/doc/administration/logs/tracing_correlation_id.md @@ -36,7 +36,7 @@ To locate a relevant request and view its correlation ID: 1. To help isolate the requests you are looking for, you can filter for `document` requests. 1. Select the request of interest to view further detail. 1. Go to the **Headers** section and look for **Response Headers**. There you should find an `x-request-id` header with a -value that was randomly generated by GitLab for the request. + value that was randomly generated by GitLab for the request. See the following example: diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index 80121c7c235..f1d1b504c9d 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -180,6 +180,8 @@ The following metrics are available: | `gitlab_connection_pool_size` | Gauge | 16.7 | Size of connection pool | | `gitlab_connection_pool_available_count` | Gauge | 16.7 | Number of available connections in the pool | | `gitlab_security_policies_scan_result_process_duration_seconds` | Histogram | 16.7 | The amount of time to process scan result policies | +| `gitlab_highlight_usage` | Counter | 16.8 | The number of times `Gitlab::Highlight` is used | `used_on` | +| `dependency_linker_usage` | Counter | 16.8 | The number of times dependency linker is used | `used_on` | ## Metrics controlled by a feature flag diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index bd3f2a20006..1fe05ed7538 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -246,14 +246,14 @@ To use an external Prometheus server: ``` 1. To allow the Prometheus server to fetch from the [GitLab metrics](#gitlab-metrics) endpoint, add the Prometheus -server IP address to the [monitoring IP allowlist](../ip_allowlist.md): + server IP address to the [monitoring IP allowlist](../ip_allowlist.md): ```ruby gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '192.168.0.1'] ``` 1. As we are setting each bundled service's [exporter](#bundled-software-metrics) to listen on a network address, -update the firewall on the instance to only allow traffic from your Prometheus IP for the exporters enabled. A full reference list of exporter services and their respective ports can be found [here](../../package_information/defaults.md#ports). + update the firewall on the instance to only allow traffic from your Prometheus IP for the exporters enabled. A full reference list of exporter services and their respective ports can be found [here](../../package_information/defaults.md#ports). 1. [Reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) to apply the changes. 1. Edit the Prometheus server's configuration file. 1. Add each node's exporters to the Prometheus server's diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index 9a9b2811cf0..0862921f0d7 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -345,7 +345,7 @@ gitlab_rails['object_store']['connection'] = { If you use ADC, be sure that: - The service account that you use has the -[`iam.serviceAccounts.signBlob` permission](https://cloud.google.com/iam/docs/reference/credentials/rest/v1/projects.serviceAccounts/signBlob). + [`iam.serviceAccounts.signBlob` permission](https://cloud.google.com/iam/docs/reference/credentials/rest/v1/projects.serviceAccounts/signBlob). Typically this is done by granting the `Service Account Token Creator` role to the service account. - Your virtual machines have the [correct access scopes to access Google Cloud APIs](https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances#changeserviceaccountandscopes). If the machines do not have the right scope, the error logs may show: @@ -431,6 +431,29 @@ gitlab_rails['object_store']['connection'] = { The signature version must be `2`. Using v4 results in a HTTP 411 Length Required error. For more information, see [issue #4419](https://gitlab.com/gitlab-org/gitlab/-/issues/4419). +### Hitachi Vantara HCP + +NOTE: +Connections to HCP may return an error stating `SigntureDoesNotMatch - The request signature we calculated does not match the signature you provided. Check your HCP Secret Access key and signing method.` In these cases, set the `endpoint` to the URL of the tenant instead of the namespace, and ensure bucket paths are configured as `<namespace_name>/<bucket_name>`. + +[HCP](https://knowledge.hitachivantara.com/Documents/Storage/HCP_for_Cloud_Scale/1.0.0/Adminstering_HCP_for_cloud_scale/Getting_started/02_Support_for_Amazon_S3_API) provides an S3-compatible API. Use the following configuration example: + +```ruby +gitlab_rails['object_store']['connection'] = { + 'provider' => 'AWS', + 'endpoint' => 'https://<tenant_endpoint>', + 'path_style' => true, + 'region' => 'eu1', + 'aws_access_key_id' => 'ACCESS_KEY', + 'aws_secret_access_key' => 'SECRET_KEY', + 'aws_signature_version' => 4, + 'enable_signature_v4_streaming' => false +} + +# Example of <namespace_name/bucket_name> formatting +gitlab_rails['object_store']['objects']['artifacts']['bucket'] = '<namespace_name>/<bucket_name>' +``` + ## Full example using the consolidated form and Amazon S3 The following example uses AWS S3 to enable object storage for all supported services: @@ -874,28 +897,28 @@ When not proxying files, GitLab returns an This can result in some of the following problems: - If GitLab is using non-secure HTTP to access the object storage, clients may generate -`https->http` downgrade errors and refuse to process the redirect. The solution to this -is for GitLab to use HTTPS. LFS, for example, generates this error: + `https->http` downgrade errors and refuse to process the redirect. The solution to this + is for GitLab to use HTTPS. LFS, for example, generates this error: - ```plaintext - LFS: lfsapi/client: refusing insecure redirect, https->http - ``` + ```plaintext + LFS: lfsapi/client: refusing insecure redirect, https->http + ``` - Clients need to trust the certificate authority that issued the object storage -certificate, or may return common TLS errors such as: + certificate, or may return common TLS errors such as: - ```plaintext - x509: certificate signed by unknown authority - ``` + ```plaintext + x509: certificate signed by unknown authority + ``` - Clients need network access to the object storage. -Network firewalls could block access. -Errors that might result -if this access is not in place include: + Network firewalls could block access. + Errors that might result + if this access is not in place include: - ```plaintext - Received status code 403 from server: Forbidden - ``` + ```plaintext + Received status code 403 from server: Forbidden + ``` - Object storage buckets need to allow Cross-Origin Resource Sharing (CORS) access from the URL of the GitLab instance. Attempting to load diff --git a/doc/administration/operations/gitlab_sshd.md b/doc/administration/operations/gitlab_sshd.md index 86c6d9d7fba..a14f556af84 100644 --- a/doc/administration/operations/gitlab_sshd.md +++ b/doc/administration/operations/gitlab_sshd.md @@ -48,7 +48,7 @@ The following instructions enable `gitlab-sshd` on a different port than OpenSSH ``` 1. Optional. By default, Linux package installations generate SSH host keys for `gitlab-sshd` if -they do not exist in `/var/opt/gitlab/gitlab-sshd`. If you wish to disable this automatic generation, add this line: + they do not exist in `/var/opt/gitlab/gitlab-sshd`. If you wish to disable this automatic generation, add this line: ```ruby gitlab_sshd['generate_host_keys'] = false diff --git a/doc/administration/operations/index.md b/doc/administration/operations/index.md index 33f407b2dbf..d1ec818a2a5 100644 --- a/doc/administration/operations/index.md +++ b/doc/administration/operations/index.md @@ -1,6 +1,7 @@ --- stage: Systems group: Distribution +description: Backup and restore, move repos, maintenance tasks. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/administration/package_information/supported_os.md b/doc/administration/package_information/supported_os.md index 89b2f78f6d9..b85d6b8dd07 100644 --- a/doc/administration/package_information/supported_os.md +++ b/doc/administration/package_information/supported_os.md @@ -28,7 +28,6 @@ architecture. | Debian 10 | GitLab CE / GitLab EE 12.2.0 | amd64, arm64 | [Debian Install Documentation](https://about.gitlab.com/install/#debian) | 2024 | <https://wiki.debian.org/LTS> | | Debian 11 | GitLab CE / GitLab EE 14.6.0 | amd64, arm64 | [Debian Install Documentation](https://about.gitlab.com/install/#debian) | 2026 | <https://wiki.debian.org/LTS> | | Debian 12 | GitLab CE / GitLab EE 16.1.0 | amd64, arm64 | [Debian Install Documentation](https://about.gitlab.com/install/#debian) | TBD | <https://wiki.debian.org/LTS> | -| OpenSUSE 15.4 | GitLab CE / GitLab EE 15.7.0 | x86_64, aarch64 | [OpenSUSE Install Documentation](https://about.gitlab.com/install/#opensuse-leap) | Nov 2023 | <https://en.opensuse.org/Lifetime> | | OpenSUSE 15.5 | GitLab CE / GitLab EE 16.4.0 | x86_64, aarch64 | [OpenSUSE Install Documentation](https://about.gitlab.com/install/#opensuse-leap) | Dec 2024 | <https://en.opensuse.org/Lifetime> | | RHEL 8 | GitLab CE / GitLab EE 12.8.1 | x86_64, arm64 | [Use CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | May 2029 | [RHEL Details](https://access.redhat.com/support/policy/updates/errata/#Life_Cycle_Dates) | | RHEL 9 | GitLab CE / GitLab EE 16.0.0 | x86_64, arm64 | [Use CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | May 2032 | [RHEL Details](https://access.redhat.com/support/policy/updates/errata/#Life_Cycle_Dates) | @@ -106,12 +105,13 @@ release for them can be found below: | Raspbian Stretch | [June 2020](https://downloads.raspberrypi.org/raspbian/images/raspbian-2019-04-09/) | [GitLab CE](https://packages.gitlab.com/app/gitlab/raspberry-pi2/search?q=gitlab-ce_13.3&dist=raspbian%2Fstretch) 13.3 | | Debian Jessie | [June 2020](https://www.debian.org/News/2020/20200709) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_13.2&dist=debian%2Fjessie) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_13.2&dist=debian%2Fjessie) 13.3 | | CentOS 6 | [November 2020](https://wiki.centos.org/About/Product) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=13.6&filter=all&filter=all&dist=el%2F6) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=13.6&filter=all&filter=all&dist=el%2F6) 13.6 | -| CentOS 8 | [December 2021](https://wiki.centos.org/About/Product) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=14.6&filter=all&filter=all&dist=el%2F8) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=14.6&filter=all&filter=all&dist=el%2F8) 14.6 | +| CentOS 8 | [December 2021](https://wiki.centos.org/About/Product) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=14.6&filter=all&filter=all&dist=el%2F8) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=14.6&filter=all&filter=all&dist=el%2F8) 14.6 | | OpenSUSE 15.1 | [November 2020](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-13.12&dist=opensuse%2F15.1) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-13.12&dist=opensuse%2F15.1) 13.12 | | Ubuntu 16.04 | [April 2021](https://ubuntu.com/info/release-end-of-life) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_13.12&dist=ubuntu%2Fxenial) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_13.12&dist=ubuntu%2Fxenial) 13.12 | | OpenSUSE 15.2 | [December 2021](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-14.7&dist=opensuse%2F15.2) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-14.7&dist=opensuse%2F15.2) 14.7 | | Debian 9 "Stretch" | [June 2022](https://lists.debian.org/debian-lts-announce/2022/07/msg00002.html) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_15.2&dist=debian%2Fstretch) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_15.2&dist=debian%2Fstretch) 15.2 | -| OpenSUSE 15.3 | [December 2022](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-15.10&dist=opensuse%2F15.3) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-15.10&dist=opensuse%2F15.3) 15.10 | +| OpenSUSE 15.3 | [December 2022](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-15.10&dist=opensuse%2F15.3) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-15.10&dist=opensuse%2F15.3) 15.10 | +| OpenSUSE 15.4 | [December 2023](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-16.7&dist=opensuse%2F15.4) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-16.7&dist=opensuse%2F15.4) 16.7 | NOTE: An exception to this deprecation policy is when we are unable to provide diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index 0594c4b93bd..150e0115eb7 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -249,7 +249,7 @@ outside world. ``` 1. If you haven't named your certificate and key `example.io.crt` and `example.io.key`, -you must also add the full paths as shown below: + you must also add the full paths as shown below: ```ruby pages_nginx['ssl_certificate'] = "/etc/gitlab/ssl/pages-nginx.crt" @@ -258,8 +258,8 @@ you must also add the full paths as shown below: 1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). 1. If you're using [Pages Access Control](#access-control), update the redirect URI in the GitLab Pages -[System OAuth application](../../integration/oauth_provider.md#create-an-instance-wide-application) -to use the HTTPS protocol. + [System OAuth application](../../integration/oauth_provider.md#create-an-instance-wide-application) + to use the HTTPS protocol. WARNING: Multiple wildcards for one instance is not supported. Only one wildcard per instance can be assigned. @@ -517,7 +517,7 @@ world. Custom domains and TLS are supported. If you don't have IPv6, you can omit the IPv6 address. 1. If you haven't named your certificate `example.io.crt` and your key `example.io.key`, -then you need to also add the full paths as shown below: + then you need to also add the full paths as shown below: ```ruby gitlab_pages['cert'] = "/etc/gitlab/ssl/example.io.crt" @@ -526,8 +526,8 @@ then you need to also add the full paths as shown below: 1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). 1. If you're using [Pages Access Control](#access-control), update the redirect URI in the GitLab Pages -[System OAuth application](../../integration/oauth_provider.md#create-an-instance-wide-application) -to use the HTTPS protocol. + [System OAuth application](../../integration/oauth_provider.md#create-an-instance-wide-application) + to use the HTTPS protocol. ### Custom domain verification @@ -1011,25 +1011,20 @@ Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. Examples: - Increasing `gitlab_cache_expiry` allows items to exist in the cache longer. -This setting might be useful if the communication between GitLab Pages and GitLab Rails -is not stable. - + This setting might be useful if the communication between GitLab Pages and GitLab Rails + is not stable. - Increasing `gitlab_cache_refresh` reduces the frequency at which GitLab Pages -requests a domain's configuration from GitLab Rails. This setting might be useful -GitLab Pages generates too many requests to GitLab API and content does not change frequently. - + requests a domain's configuration from GitLab Rails. This setting might be useful + GitLab Pages generates too many requests to GitLab API and content does not change frequently. - Decreasing `gitlab_cache_cleanup` removes expired items from the cache more frequently, -reducing the memory usage of your Pages node. - + reducing the memory usage of your Pages node. - Decreasing `gitlab_retrieval_timeout` allows you to stop the request to GitLab Rails -more quickly. Increasing it allows more time to receive a response from the API, -useful in slow networking environments. - + more quickly. Increasing it allows more time to receive a response from the API, + useful in slow networking environments. - Decreasing `gitlab_retrieval_interval` makes requests to the API more frequently, -only when there is an error response from the API, for example a connection timeout. - + only when there is an error response from the API, for example a connection timeout. - Decreasing `gitlab_retrieval_retries` reduces the number of times a domain's -configuration is tried to be resolved automatically before reporting an error. + configuration is tried to be resolved automatically before reporting an error. ## Object storage settings @@ -1212,6 +1207,8 @@ TLS connection-based rate limits are enforced using the following: - `rate_limit_tls_domain`: Set the maximum threshold in number of TLS connections per hosted pages domain per second. Set to 0 to disable this feature. - `rate_limit_tls_domain_burst`: Sets the maximum threshold of number of TLS connections allowed in an initial outburst of TLS connections per hosted pages domain. +An IPv6 address receives a large prefix in the 128-bit address space. The prefix is typically at least size /64. Because of the large number of possible addresses, if the client's IP address is IPv6, the limit is applied to the IPv6 prefix with a length of 64, rather than the entire IPv6 address. + #### Enable HTTP requests rate limits by source-IP > [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/631) in GitLab 14.5. diff --git a/doc/administration/postgresql/multiple_databases.md b/doc/administration/postgresql/multiple_databases.md index 2caa4f147c4..b91b0e573f0 100644 --- a/doc/administration/postgresql/multiple_databases.md +++ b/doc/administration/postgresql/multiple_databases.md @@ -1,4 +1,5 @@ --- + stage: Data Stores group: Tenant Scale info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments @@ -35,8 +36,95 @@ databases. Some examples: ## Known issues - Once data is migrated to the `ci` database, you cannot migrate it back. +- HA setups or PgBouncer setups are not yet supported by this procedure. + +## Migrate existing installations using a script + +> A script for migrating existing Linux package installations was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/368729) in GitLab 16.8. + +NOTE: +If something unexpected happens during the migration, it is safe to start over. + +### Existing Linux package installations + +#### Preparation + +1. Verify available disk space: + + - The database node that will store the `gitlabhq_production_ci` database needs enough space to store a copy of the existing database: we _duplicate_ `gitlabhq_production`. Run the following SQL query to find out how much space is needed. Add 25%, to ensure you will not run out of disk space. + + ```shell + sudo gitlab-psql -c "SELECT pg_size_pretty( pg_database_size('gitlabhq_production') );" + ``` -## Migrate existing installations + - During the process, a dump of the `gitlabhq_production` database needs to be temporarily stored on the filesystem of the node that will run the migration. Execute the following SQL statement to find out how much local disk space will be used. Add 25%, to ensure you will not run out of disk space. + + ```shell + sudo gitlab-psql -c "select sum(pg_table_size(concat(table_schema,'.',table_name))) from information_schema.tables where table_catalog = 'gitlabhq_production' and table_type = 'BASE TABLE'" + ``` + +1. Plan for downtime. The downtime is dependent on the size of the `gitlabhq_production` database. + + - We dump `gitlabhq_production` and restore it into a new `gitlabhq_production_ci` database. Database sizes below 100 GB should be done within 30 minutes. + - We advise to also plan some time for smaller tasks like modifying the configuration. + +1. Create the new `gitlabhq_production_ci` database: + + ```shell + sudo gitlab-psql -c "CREATE DATABASE gitlabhq_production_ci WITH OWNER 'gitlab'" + ``` + +#### Migration + +This process includes downtime. Running the migration script will stop the GitLab instance. After the migration has been finished, the instance is restarted. + +1. Create a backup of the configuration: + + ```shell + sudo cp /etc/gitlab/gitlab.rb /etc/gitlab/gitlab.rb.org + ``` + +1. Edit `/etc/gitlab/gitlab.rb` and save the changes. Do **not** run the reconfigure command, the migration script will run that for you. + + ```ruby + gitlab_rails['env'] = { 'GITLAB_ALLOW_SEPARATE_CI_DATABASE' => 'true' } + gitlab_rails['databases']['ci']['enable'] = true + gitlab_rails['databases']['ci']['db_database'] = 'gitlabhq_production_ci' + ``` + +1. Run the migration script: + + ```shell + sudo gitlab-ctl pg-decomposition-migration + ``` + +At this point, the GitLab instance should start and be functional. + +If you want to abort the procedure and you want to start GitLab without changing anything, run the following commands: + +```shell +sudo cp /etc/gitlab/gitlab.rb.org /etc/gitlab/gitlab.rb +sudo gitlab-ctl reconfigure +sudo gitlab-ctl restart +``` + +#### Cleaning up + +If everything works as expected, we can clean up unneeded data: + +- Delete the CI data in Main database: + +```shell +sudo gitlab-rake gitlab:db:truncate_legacy_tables:main +``` + +- Delete the Main data in CI database: + +```shell +sudo gitlab-rake gitlab:db:truncate_legacy_tables:ci +``` + +## Migrate existing installations (manual procedure) To migrate existing data from the `main` database to the `ci` database, you can copy the database across. @@ -46,9 +134,9 @@ If something unexpected happens during the migration, it is safe to start over. ### Existing self-compiled installation -1. [Disable background migrations](../../development/database/batched_background_migrations.md#enable-or-disable-background-migrations) +1. [Disable background migrations](../../development/database/batched_background_migrations.md#enable-or-disable-background-migrations). -1. [Ensure all background migrations are finished](../../update/background_migrations.md#check-the-status-of-batched-background-migrations) +1. [Ensure all background migrations are finished](../../update/background_migrations.md#check-the-status-of-batched-background-migrations). 1. Stop GitLab, except for PostgreSQL: @@ -112,7 +200,7 @@ the other way around. ### Self-compiled installations 1. For existing installations, - [migrate the data](#migrate-existing-installations) first. + [migrate the data](#migrate-existing-installations-manual-procedure) first. 1. [Back up GitLab](../../administration/backup_restore/index.md) in case of unforeseen issues. @@ -168,7 +256,7 @@ the other way around. ### Linux package installations 1. For existing installations, - [migrate the data](#migrate-existing-installations) first. + [migrate the data](#migrate-existing-installations-manual-procedure) first. 1. [Back up GitLab](../../administration/backup_restore/index.md) in case of unforeseen issues. diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index f4ed9d99b45..e93dfc8336e 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -563,8 +563,7 @@ gitlab-rake gitlab:db:configure > **Note**: If you encounter a `rake aborted!` error stating that PgBouncer is failing to connect to PostgreSQL it may be that your PgBouncer node's IP address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` on your database nodes. See -[PgBouncer error `ERROR: pgbouncer cannot connect to server`](#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) -in the Troubleshooting section before proceeding. +[PgBouncer error `ERROR: pgbouncer cannot connect to server`](../../administration/postgresql/replication_and_failover_troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) before you proceed. ### Backups @@ -575,8 +574,7 @@ Do not backup or restore GitLab through a PgBouncer connection: this causes a Gi ### Ensure GitLab is running At this point, your GitLab instance should be up and running. Verify you're able -to sign in, and create issues and merge requests. If you encounter issues, see -the [Troubleshooting section](#troubleshooting). +to sign in, and create issues and merge requests. For more information, see [Troubleshooting replication and failover](../../administration/postgresql/replication_and_failover_troubleshooting.md). ## Example configuration @@ -901,7 +899,7 @@ Stopping or restarting the Patroni service on the leader node triggers an automa WARNING: In GitLab 16.5 and earlier, PgBouncer nodes do not automatically fail over alongside Patroni nodes. PgBouncer services -[must be restarted manually](#pgbouncer-errors-error-running-command-gitlabctlerrorsexecutionerror-and-error-database-gitlabhq_production-is-not-paused) +[must be restarted manually](../../administration/postgresql/replication_and_failover_troubleshooting.md#pgbouncer-errors-error-running-command-gitlabctlerrorsexecutionerror-and-error-database-gitlabhq_production-is-not-paused) for a successful switchover. While Patroni supports automatic failover, you also have the ability to perform @@ -1085,8 +1083,7 @@ Considering these, you should carefully plan your PostgreSQL upgrade: ``` If issues are encountered upgrading the replicas, -[there is a troubleshooting section](#postgresql-major-version-upgrade-fails-on-a-patroni-replica) -that might be the solution. +[there is a troubleshooting section](../../administration/postgresql/replication_and_failover_troubleshooting.md#postgresql-major-version-upgrade-fails-on-a-patroni-replica) that might be the solution. NOTE: Reverting the PostgreSQL upgrade with `gitlab-ctl revert-pg-upgrade` has the same considerations as @@ -1274,468 +1271,3 @@ After completing these steps, then you can clean up the resources of the old Pat They are no longer needed. However, before removing the resources, remove the logical replication subscription on the new leader by running `DROP SUBSCRIPTION patroni_upgrade` with `gitlab-psql`. - -## Troubleshooting - -### Consul and PostgreSQL changes not taking effect - -Due to the potential impacts, `gitlab-ctl reconfigure` only reloads Consul and PostgreSQL, it does not restart the services. However, not all changes can be activated by reloading. - -To restart either service, run `gitlab-ctl restart SERVICE` - -For PostgreSQL, it is usually safe to restart the leader node by default. Automatic failover defaults to a 1 minute timeout. Provided the database returns before then, nothing else needs to be done. - -On the Consul server nodes, it is important to [restart the Consul service](../consul.md#restart-consul) in a controlled manner. - -### PgBouncer error `ERROR: pgbouncer cannot connect to server` - -You may get this error when running `gitlab-rake gitlab:db:configure` or you -may see the error in the PgBouncer log file. - -```plaintext -PG::ConnectionBad: ERROR: pgbouncer cannot connect to server -``` - -The problem may be that your PgBouncer node's IP address is not included in the -`trust_auth_cidr_addresses` setting in `/etc/gitlab/gitlab.rb` on the database nodes. - -You can confirm that this is the issue by checking the PostgreSQL log on the leader -database node. If you see the following error then `trust_auth_cidr_addresses` -is the problem. - -```plaintext -2018-03-29_13:59:12.11776 FATAL: no pg_hba.conf entry for host "123.123.123.123", user "pgbouncer", database "gitlabhq_production", SSL off -``` - -To fix the problem, add the IP address to `/etc/gitlab/gitlab.rb`. - -```ruby -postgresql['trust_auth_cidr_addresses'] = %w(123.123.123.123/32 <other_cidrs>) -``` - -[Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. - -### PgBouncer errors `Error running command: GitlabCtl::Errors::ExecutionError` and `ERROR: database gitlabhq_production is not paused` - -Due to a [known issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/8166) that -affects versions of GitLab prior to 16.5.0, the automatic failover of PgBouncer nodes does not -happen after a [Patroni switchover](#manual-failover-procedure-for-patroni). In this -example, GitLab failed to detect a paused database, then attempted to `RESUME` a -not-paused database: - -```plaintext -INFO -- : Running: gitlab-ctl pgb-notify --pg-database gitlabhq_production --newhost database7.example.com --user pgbouncer --hostuser gitlab-consul -ERROR -- : STDERR: Error running command: GitlabCtl::Errors::ExecutionError -ERROR -- : STDERR: ERROR: ERROR: database gitlabhq_production is not paused -``` - -To ensure a [Patroni switchover](#manual-failover-procedure-for-patroni) succeeds, -you must manually restart the PgBouncer service on all PgBouncer nodes with this command: - -```shell -gitlab-ctl restart pgbouncer -``` - -### Reinitialize a replica - -If a replica cannot start or rejoin the cluster, or when it lags behind and cannot catch up, it might be necessary to reinitialize the replica: - -1. [Check the replication status](#check-replication-status) to confirm which server - needs to be reinitialized. For example: - - ```plaintext - + Cluster: postgresql-ha (6970678148837286213) ------+---------+--------------+----+-----------+ - | Member | Host | Role | State | TL | Lag in MB | - +-------------------------------------+--------------+---------+--------------+----+-----------+ - | gitlab-database-1.example.com | 172.18.0.111 | Replica | running | 55 | 0 | - | gitlab-database-2.example.com | 172.18.0.112 | Replica | start failed | | unknown | - | gitlab-database-3.example.com | 172.18.0.113 | Leader | running | 55 | | - +-------------------------------------+--------------+---------+--------------+----+-----------+ - ``` - -1. Sign in to the broken server and reinitialize the database and replication. Patroni shuts - down PostgreSQL on that server, remove the data directory, and reinitialize it from scratch: - - ```shell - sudo gitlab-ctl patroni reinitialize-replica --member gitlab-database-2.example.com - ``` - - This can be run on any Patroni node, but be aware that `sudo gitlab-ctl patroni - reinitialize-replica` without `--member` restarts the server it is run on. - You should run it locally on the broken server to reduce the risk of - unintended data loss. -1. Monitor the logs: - - ```shell - sudo gitlab-ctl tail patroni - ``` - -### Reset the Patroni state in Consul - -WARNING: -Resetting the Patroni state in Consul is a potentially destructive process. Make sure that you have a healthy database backup first. - -As a last resort you can reset the Patroni state in Consul completely. - -This may be required if your Patroni cluster is in an unknown or bad state and no node can start: - -```plaintext -+ Cluster: postgresql-ha (6970678148837286213) ------+---------+---------+----+-----------+ -| Member | Host | Role | State | TL | Lag in MB | -+-------------------------------------+--------------+---------+---------+----+-----------+ -| gitlab-database-1.example.com | 172.18.0.111 | Replica | stopped | | unknown | -| gitlab-database-2.example.com | 172.18.0.112 | Replica | stopped | | unknown | -| gitlab-database-3.example.com | 172.18.0.113 | Replica | stopped | | unknown | -+-------------------------------------+--------------+---------+---------+----+-----------+ -``` - -**Before deleting the Patroni state in Consul**, -[try and resolve the `gitlab-ctl` errors](#errors-running-gitlab-ctl) on the Patroni nodes. - -This process results in a reinitialized Patroni cluster when -the first Patroni node starts. - -To reset the Patroni state in Consul: - -1. Take note of the Patroni node that was the leader, or that the application thinks is the current leader, - if the current state shows more than one, or none: - - Look on the PgBouncer nodes in `/var/opt/gitlab/consul/databases.ini`, - which contains the hostname of the current leader. - - Look in the Patroni logs `/var/log/gitlab/patroni/current` (or the older rotated and - compressed logs `/var/log/gitlab/patroni/@40000*`) on **all** database nodes to see - which server was most recently identified as the leader by the cluster: - - ```plaintext - INFO: no action. I am a secondary (database1.local) and following a leader (database2.local) - ``` - -1. Stop Patroni on all nodes: - - ```shell - sudo gitlab-ctl stop patroni - ``` - -1. Reset the state in Consul: - - ```shell - /opt/gitlab/embedded/bin/consul kv delete -recurse /service/postgresql-ha/ - ``` - -1. Start one Patroni node, which initializes the Patroni cluster to elect as a leader. - It's highly recommended to start the previous leader (noted in the first step), - so as to not lose existing writes that may have not been replicated because - of the broken cluster state: - - ```shell - sudo gitlab-ctl start patroni - ``` - -1. Start all other Patroni nodes that join the Patroni cluster as replicas: - - ```shell - sudo gitlab-ctl start patroni - ``` - -If you are still seeing issues, the next step is restoring the last healthy backup. - -### Errors in the Patroni log about a `pg_hba.conf` entry for `127.0.0.1` - -The following log entry in the Patroni log indicates the replication is not working -and a configuration change is needed: - -```plaintext -FATAL: no pg_hba.conf entry for replication connection from host "127.0.0.1", user "gitlab_replicator" -``` - -To fix the problem, ensure the loopback interface is included in the CIDR addresses list: - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - postgresql['trust_auth_cidr_addresses'] = %w(<other_cidrs> 127.0.0.1/32) - ``` - -1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. -1. Check that [all the replicas are synchronized](#check-replication-status) - -### Errors in Patroni logs: the requested start point is ahead of the Write Ahead Log (WAL) flush position - -This error indicates that the database is not replicating: - -```plaintext -FATAL: could not receive data from WAL stream: ERROR: requested starting point 0/5000000 is ahead of the WAL flush position of this server 0/4000388 -``` - -This example error is from a replica that was initially misconfigured, and had never replicated. - -Fix it [by reinitializing the replica](#reinitialize-a-replica). - -### Patroni fails to start with `MemoryError` - -Patroni may fail to start, logging an error and stack trace: - -```plaintext -MemoryError -Traceback (most recent call last): - File "/opt/gitlab/embedded/bin/patroni", line 8, in <module> - sys.exit(main()) -[..] - File "/opt/gitlab/embedded/lib/python3.7/ctypes/__init__.py", line 273, in _reset_cache - CFUNCTYPE(c_int)(lambda: None) -``` - -If the stack trace ends with `CFUNCTYPE(c_int)(lambda: None)`, this code triggers `MemoryError` -if the Linux server has been hardened for security. - -The code causes Python to write temporary executable files, and if it cannot find a file system in which to do this. For example, if `noexec` is set on the `/tmp` file system, it fails with `MemoryError` ([read more in the issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6184)). - -Workarounds: - -- Remove `noexec` from the mount options for filesystems like `/tmp` and `/var/tmp`. -- If set to enforcing, SELinux may also prevent these operations. Verify the issue is fixed by setting - SELinux to permissive. - -Patroni first shipped in the Linux package for GitLab 13.1, along with a build of Python 3.7. -The code which causes this was removed in Python 3.8: this fix shipped in -[the Linux package for GitLab 14.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5547) -and later, removing the need for a workaround. - -### Errors running `gitlab-ctl` - -Patroni nodes can get into a state where `gitlab-ctl` commands fail -and `gitlab-ctl reconfigure` cannot fix the node. - -If this co-incides with a version upgrade of PostgreSQL, [follow a different procedure](#postgresql-major-version-upgrade-fails-on-a-patroni-replica) - -One common symptom is that `gitlab-ctl` cannot determine -information it needs about the installation if the database server is failing to start: - -```plaintext -Malformed configuration JSON file found at /opt/gitlab/embedded/nodes/<HOSTNAME>.json. -This usually happens when your last run of `gitlab-ctl reconfigure` didn't complete successfully. -``` - -```plaintext -Error while reinitializing replica on the current node: Attributes not found in -/opt/gitlab/embedded/nodes/<HOSTNAME>.json, has reconfigure been run yet? -``` - -Similarly, the nodes file (`/opt/gitlab/embedded/nodes/<HOSTNAME>.json`) should contain a lot of information, -but might get created with only: - -```json -{ - "name": "<HOSTNAME>" -} -``` - -The following process for fixing this includes reinitializing this replica: -the current state of PostgreSQL on this node is discarded: - -1. Shut down the Patroni and (if present) PostgreSQL services: - - ```shell - sudo gitlab-ctl status - sudo gitlab-ctl stop patroni - sudo gitlab-ctl stop postgresql - ``` - -1. Remove `/var/opt/gitlab/postgresql/data` in case its state prevents - PostgreSQL from starting: - - ```shell - cd /var/opt/gitlab/postgresql - sudo rm -rf data - ``` - - **Take care with this step to avoid data loss**. - This step can be also achieved by renaming `data/`: - make sure there's enough free disk for a new copy of the primary database, - and remove the extra directory when the replica is fixed. - -1. With PostgreSQL not running, the nodes file now gets created successfully: - - ```shell - sudo gitlab-ctl reconfigure - ``` - -1. Start Patroni: - - ```shell - sudo gitlab-ctl start patroni - ``` - -1. Monitor the logs and check the cluster state: - - ```shell - sudo gitlab-ctl tail patroni - sudo gitlab-ctl patroni members - ``` - -1. Re-run `reconfigure` again: - - ```shell - sudo gitlab-ctl reconfigure - ``` - -1. Reinitialize the replica if `gitlab-ctl patroni members` indicates this is needed: - - ```shell - sudo gitlab-ctl patroni reinitialize-replica - ``` - -If this procedure doesn't work **and** if the cluster is unable to elect a leader, -[there is a another fix](#reset-the-patroni-state-in-consul) which should only be -used as a last resort. - -### PostgreSQL major version upgrade fails on a Patroni replica - -A Patroni **replica** can get stuck in a loop during `gitlab-ctl pg-upgrade`, and -the upgrade fails. - -An example set of symptoms is as follows: - -1. A `postgresql` service is defined, - which shouldn't usually be present on a Patroni node. It is present because - `gitlab-ctl pg-upgrade` adds it to create a new empty database: - - ```plaintext - run: patroni: (pid 1972) 1919s; run: log: (pid 1971) 1919s - down: postgresql: 1s, normally up, want up; run: log: (pid 1973) 1919s - ``` - -1. PostgreSQL generates `PANIC` log entries in - `/var/log/gitlab/postgresql/current` as Patroni is removing - `/var/opt/gitlab/postgresql/data` as part of reinitializing the replica: - - ```plaintext - DETAIL: Could not open file "pg_xact/0000": No such file or directory. - WARNING: terminating connection because of crash of another server process - LOG: all server processes terminated; reinitializing - PANIC: could not open file "global/pg_control": No such file or directory - ``` - -1. In `/var/log/gitlab/patroni/current`, Patroni logs the following. - The local PostgreSQL version is different from the cluster leader: - - ```plaintext - INFO: trying to bootstrap from leader 'HOSTNAME' - pg_basebackup: incompatible server version 12.6 - pg_basebackup: removing data directory "/var/opt/gitlab/postgresql/data" - ERROR: Error when fetching backup: pg_basebackup exited with code=1 - ``` - -**Important**: This workaround applies when the Patroni cluster is in the following state: - -- The [leader has been successfully upgraded to the new major version](#upgrading-postgresql-major-version-in-a-patroni-cluster). -- The step to upgrade PostgreSQL on replicas is failing. - -This workaround completes the PostgreSQL upgrade on a Patroni replica -by setting the node to use the new PostgreSQL version, and then reinitializing -it as a replica in the new cluster that was created -when the leader was upgraded: - -1. Check the cluster status on all nodes to confirm which is the leader - and what state the replicas are in - - ```shell - sudo gitlab-ctl patroni members - ``` - -1. Replica: check which version of PostgreSQL is active: - - ```shell - sudo ls -al /opt/gitlab/embedded/bin | grep postgres - ``` - -1. Replica: ensure the nodes file is correct and `gitlab-ctl` can run. This resolves - the [errors running `gitlab-ctl`](#errors-running-gitlab-ctl) issue if the replica - has any of those errors as well: - - ```shell - sudo gitlab-ctl stop patroni - sudo gitlab-ctl reconfigure - ``` - -1. Replica: relink the PostgreSQL binaries to the required version - to fix the `incompatible server version` error: - - 1. Edit `/etc/gitlab/gitlab.rb` and specify the required version: - - ```ruby - postgresql['version'] = 13 - ``` - - 1. Reconfigure GitLab: - - ```shell - sudo gitlab-ctl reconfigure - ``` - - 1. Check the binaries are relinked. The binaries distributed for - PostgreSQL vary between major releases, it's typical to - have a small number of incorrect symbolic links: - - ```shell - sudo ls -al /opt/gitlab/embedded/bin | grep postgres - ``` - -1. Replica: ensure PostgreSQL is fully reinitialized for the specified version: - - ```shell - cd /var/opt/gitlab/postgresql - sudo rm -rf data - sudo gitlab-ctl reconfigure - ``` - -1. Replica: optionally monitor the database in two additional terminal sessions: - - - Disk use increases as `pg_basebackup` runs. Track progress of the - replica initialization with: - - ```shell - cd /var/opt/gitlab/postgresql - watch du -sh data - ``` - - - Monitor the process in the logs: - - ```shell - sudo gitlab-ctl tail patroni - ``` - -1. Replica: Start Patroni to reinitialize the replica: - - ```shell - sudo gitlab-ctl start patroni - ``` - -1. Replica: After it completes, remove the hardcoded version from `/etc/gitlab/gitlab.rb`: - - 1. Edit `/etc/gitlab/gitlab.rb` and remove `postgresql['version']`. - 1. Reconfigure GitLab: - - ```shell - sudo gitlab-ctl reconfigure - ``` - - 1. Check the correct binaries are linked: - - ```shell - sudo ls -al /opt/gitlab/embedded/bin | grep postgres - ``` - -1. Check the cluster status on all nodes: - - ```shell - sudo gitlab-ctl patroni members - ``` - -Repeat this procedure on the other replica if required. - -### Issues with other components - -If you're running into an issue with a component not outlined here, be sure to check the troubleshooting section of their specific documentation page: - -- [Consul](../consul.md#troubleshooting-consul) -- [PostgreSQL](https://docs.gitlab.com/omnibus/settings/database.html#troubleshooting) diff --git a/doc/administration/postgresql/replication_and_failover_troubleshooting.md b/doc/administration/postgresql/replication_and_failover_troubleshooting.md new file mode 100644 index 00000000000..1b5aa5fc3b5 --- /dev/null +++ b/doc/administration/postgresql/replication_and_failover_troubleshooting.md @@ -0,0 +1,472 @@ +--- +stage: Data Stores +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Troubleshooting PostgreSQL replication and failover for Linux package installations **(PREMIUM SELF)** + +When working with PostgreSQL replication and failover, you might encounter the following issues. + +## Consul and PostgreSQL changes not taking effect + +Due to the potential impacts, `gitlab-ctl reconfigure` only reloads Consul and PostgreSQL, it does not restart the services. However, not all changes can be activated by reloading. + +To restart either service, run `gitlab-ctl restart SERVICE` + +For PostgreSQL, it is usually safe to restart the leader node by default. Automatic failover defaults to a 1 minute timeout. Provided the database returns before then, nothing else needs to be done. + +On the Consul server nodes, it is important to [restart the Consul service](../consul.md#restart-consul) in a controlled manner. + +## PgBouncer error `ERROR: pgbouncer cannot connect to server` + +You may get this error when running `gitlab-rake gitlab:db:configure` or you +may see the error in the PgBouncer log file. + +```plaintext +PG::ConnectionBad: ERROR: pgbouncer cannot connect to server +``` + +The problem may be that your PgBouncer node's IP address is not included in the +`trust_auth_cidr_addresses` setting in `/etc/gitlab/gitlab.rb` on the database nodes. + +You can confirm that this is the issue by checking the PostgreSQL log on the leader +database node. If you see the following error then `trust_auth_cidr_addresses` +is the problem. + +```plaintext +2018-03-29_13:59:12.11776 FATAL: no pg_hba.conf entry for host "123.123.123.123", user "pgbouncer", database "gitlabhq_production", SSL off +``` + +To fix the problem, add the IP address to `/etc/gitlab/gitlab.rb`. + +```ruby +postgresql['trust_auth_cidr_addresses'] = %w(123.123.123.123/32 <other_cidrs>) +``` + +[Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. + +## PgBouncer errors `Error running command: GitlabCtl::Errors::ExecutionError` and `ERROR: database gitlabhq_production is not paused` + +Due to a [known issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/8166) that +affects versions of GitLab prior to 16.5.0, the automatic failover of PgBouncer nodes does not +happen after a [Patroni switchover](../../administration/postgresql/replication_and_failover.md#manual-failover-procedure-for-patroni). In this +example, GitLab failed to detect a paused database, then attempted to `RESUME` a +not-paused database: + +```plaintext +INFO -- : Running: gitlab-ctl pgb-notify --pg-database gitlabhq_production --newhost database7.example.com --user pgbouncer --hostuser gitlab-consul +ERROR -- : STDERR: Error running command: GitlabCtl::Errors::ExecutionError +ERROR -- : STDERR: ERROR: ERROR: database gitlabhq_production is not paused +``` + +To ensure a [Patroni switchover](../../administration/postgresql/replication_and_failover.md#manual-failover-procedure-for-patroni) succeeds, +you must manually restart the PgBouncer service on all PgBouncer nodes with this command: + +```shell +gitlab-ctl restart pgbouncer +``` + +## Reinitialize a replica + +If a replica cannot start or rejoin the cluster, or when it lags behind and cannot catch up, it might be necessary to reinitialize the replica: + +1. [Check the replication status](../../administration/postgresql/replication_and_failover.md#check-replication-status) to confirm which server + needs to be reinitialized. For example: + + ```plaintext + + Cluster: postgresql-ha (6970678148837286213) ------+---------+--------------+----+-----------+ + | Member | Host | Role | State | TL | Lag in MB | + +-------------------------------------+--------------+---------+--------------+----+-----------+ + | gitlab-database-1.example.com | 172.18.0.111 | Replica | running | 55 | 0 | + | gitlab-database-2.example.com | 172.18.0.112 | Replica | start failed | | unknown | + | gitlab-database-3.example.com | 172.18.0.113 | Leader | running | 55 | | + +-------------------------------------+--------------+---------+--------------+----+-----------+ + ``` + +1. Sign in to the broken server and reinitialize the database and replication. Patroni shuts + down PostgreSQL on that server, remove the data directory, and reinitialize it from scratch: + + ```shell + sudo gitlab-ctl patroni reinitialize-replica --member gitlab-database-2.example.com + ``` + + This can be run on any Patroni node, but be aware that `sudo gitlab-ctl patroni + reinitialize-replica` without `--member` restarts the server it is run on. + You should run it locally on the broken server to reduce the risk of + unintended data loss. +1. Monitor the logs: + + ```shell + sudo gitlab-ctl tail patroni + ``` + +## Reset the Patroni state in Consul + +WARNING: +Resetting the Patroni state in Consul is a potentially destructive process. Make sure that you have a healthy database backup first. + +As a last resort you can reset the Patroni state in Consul completely. + +This may be required if your Patroni cluster is in an unknown or bad state and no node can start: + +```plaintext ++ Cluster: postgresql-ha (6970678148837286213) ------+---------+---------+----+-----------+ +| Member | Host | Role | State | TL | Lag in MB | ++-------------------------------------+--------------+---------+---------+----+-----------+ +| gitlab-database-1.example.com | 172.18.0.111 | Replica | stopped | | unknown | +| gitlab-database-2.example.com | 172.18.0.112 | Replica | stopped | | unknown | +| gitlab-database-3.example.com | 172.18.0.113 | Replica | stopped | | unknown | ++-------------------------------------+--------------+---------+---------+----+-----------+ +``` + +**Before deleting the Patroni state in Consul**, +[try and resolve the `gitlab-ctl` errors](#errors-running-gitlab-ctl) on the Patroni nodes. + +This process results in a reinitialized Patroni cluster when +the first Patroni node starts. + +To reset the Patroni state in Consul: + +1. Take note of the Patroni node that was the leader, or that the application thinks is the current leader, + if the current state shows more than one, or none: + - Look on the PgBouncer nodes in `/var/opt/gitlab/consul/databases.ini`, + which contains the hostname of the current leader. + - Look in the Patroni logs `/var/log/gitlab/patroni/current` (or the older rotated and + compressed logs `/var/log/gitlab/patroni/@40000*`) on **all** database nodes to see + which server was most recently identified as the leader by the cluster: + + ```plaintext + INFO: no action. I am a secondary (database1.local) and following a leader (database2.local) + ``` + +1. Stop Patroni on all nodes: + + ```shell + sudo gitlab-ctl stop patroni + ``` + +1. Reset the state in Consul: + + ```shell + /opt/gitlab/embedded/bin/consul kv delete -recurse /service/postgresql-ha/ + ``` + +1. Start one Patroni node, which initializes the Patroni cluster to elect as a leader. + It's highly recommended to start the previous leader (noted in the first step), + so as to not lose existing writes that may have not been replicated because + of the broken cluster state: + + ```shell + sudo gitlab-ctl start patroni + ``` + +1. Start all other Patroni nodes that join the Patroni cluster as replicas: + + ```shell + sudo gitlab-ctl start patroni + ``` + +If you are still seeing issues, the next step is restoring the last healthy backup. + +## Errors in the Patroni log about a `pg_hba.conf` entry for `127.0.0.1` + +The following log entry in the Patroni log indicates the replication is not working +and a configuration change is needed: + +```plaintext +FATAL: no pg_hba.conf entry for replication connection from host "127.0.0.1", user "gitlab_replicator" +``` + +To fix the problem, ensure the loopback interface is included in the CIDR addresses list: + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + postgresql['trust_auth_cidr_addresses'] = %w(<other_cidrs> 127.0.0.1/32) + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. +1. Check that [all the replicas are synchronized](../../administration/postgresql/replication_and_failover.md#check-replication-status) + +## Errors in Patroni logs: the requested start point is ahead of the Write Ahead Log (WAL) flush position + +This error indicates that the database is not replicating: + +```plaintext +FATAL: could not receive data from WAL stream: ERROR: requested starting point 0/5000000 is ahead of the WAL flush position of this server 0/4000388 +``` + +This example error is from a replica that was initially misconfigured, and had never replicated. + +Fix it [by reinitializing the replica](#reinitialize-a-replica). + +## Patroni fails to start with `MemoryError` + +Patroni may fail to start, logging an error and stack trace: + +```plaintext +MemoryError +Traceback (most recent call last): + File "/opt/gitlab/embedded/bin/patroni", line 8, in <module> + sys.exit(main()) +[..] + File "/opt/gitlab/embedded/lib/python3.7/ctypes/__init__.py", line 273, in _reset_cache + CFUNCTYPE(c_int)(lambda: None) +``` + +If the stack trace ends with `CFUNCTYPE(c_int)(lambda: None)`, this code triggers `MemoryError` +if the Linux server has been hardened for security. + +The code causes Python to write temporary executable files, and if it cannot find a file system in which to do this. For example, if `noexec` is set on the `/tmp` file system, it fails with `MemoryError` ([read more in the issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6184)). + +Workarounds: + +- Remove `noexec` from the mount options for filesystems like `/tmp` and `/var/tmp`. +- If set to enforcing, SELinux may also prevent these operations. Verify the issue is fixed by setting + SELinux to permissive. + +Patroni first shipped in the Linux package for GitLab 13.1, along with a build of Python 3.7. +The code which causes this was removed in Python 3.8: this fix shipped in +[the Linux package for GitLab 14.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5547) +and later, removing the need for a workaround. + +## Errors running `gitlab-ctl` + +Patroni nodes can get into a state where `gitlab-ctl` commands fail +and `gitlab-ctl reconfigure` cannot fix the node. + +If this co-incides with a version upgrade of PostgreSQL, [follow a different procedure](#postgresql-major-version-upgrade-fails-on-a-patroni-replica) + +One common symptom is that `gitlab-ctl` cannot determine +information it needs about the installation if the database server is failing to start: + +```plaintext +Malformed configuration JSON file found at /opt/gitlab/embedded/nodes/<HOSTNAME>.json. +This usually happens when your last run of `gitlab-ctl reconfigure` didn't complete successfully. +``` + +```plaintext +Error while reinitializing replica on the current node: Attributes not found in +/opt/gitlab/embedded/nodes/<HOSTNAME>.json, has reconfigure been run yet? +``` + +Similarly, the nodes file (`/opt/gitlab/embedded/nodes/<HOSTNAME>.json`) should contain a lot of information, +but might get created with only: + +```json +{ + "name": "<HOSTNAME>" +} +``` + +The following process for fixing this includes reinitializing this replica: +the current state of PostgreSQL on this node is discarded: + +1. Shut down the Patroni and (if present) PostgreSQL services: + + ```shell + sudo gitlab-ctl status + sudo gitlab-ctl stop patroni + sudo gitlab-ctl stop postgresql + ``` + +1. Remove `/var/opt/gitlab/postgresql/data` in case its state prevents + PostgreSQL from starting: + + ```shell + cd /var/opt/gitlab/postgresql + sudo rm -rf data + ``` + + **Take care with this step to avoid data loss**. + This step can be also achieved by renaming `data/`: + make sure there's enough free disk for a new copy of the primary database, + and remove the extra directory when the replica is fixed. + +1. With PostgreSQL not running, the nodes file now gets created successfully: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. Start Patroni: + + ```shell + sudo gitlab-ctl start patroni + ``` + +1. Monitor the logs and check the cluster state: + + ```shell + sudo gitlab-ctl tail patroni + sudo gitlab-ctl patroni members + ``` + +1. Re-run `reconfigure` again: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. Reinitialize the replica if `gitlab-ctl patroni members` indicates this is needed: + + ```shell + sudo gitlab-ctl patroni reinitialize-replica + ``` + +If this procedure doesn't work **and** if the cluster is unable to elect a leader, +[there is a another fix](#reset-the-patroni-state-in-consul) which should only be +used as a last resort. + +## PostgreSQL major version upgrade fails on a Patroni replica + +A Patroni **replica** can get stuck in a loop during `gitlab-ctl pg-upgrade`, and +the upgrade fails. + +An example set of symptoms is as follows: + +1. A `postgresql` service is defined, + which shouldn't usually be present on a Patroni node. It is present because + `gitlab-ctl pg-upgrade` adds it to create a new empty database: + + ```plaintext + run: patroni: (pid 1972) 1919s; run: log: (pid 1971) 1919s + down: postgresql: 1s, normally up, want up; run: log: (pid 1973) 1919s + ``` + +1. PostgreSQL generates `PANIC` log entries in + `/var/log/gitlab/postgresql/current` as Patroni is removing + `/var/opt/gitlab/postgresql/data` as part of reinitializing the replica: + + ```plaintext + DETAIL: Could not open file "pg_xact/0000": No such file or directory. + WARNING: terminating connection because of crash of another server process + LOG: all server processes terminated; reinitializing + PANIC: could not open file "global/pg_control": No such file or directory + ``` + +1. In `/var/log/gitlab/patroni/current`, Patroni logs the following. + The local PostgreSQL version is different from the cluster leader: + + ```plaintext + INFO: trying to bootstrap from leader 'HOSTNAME' + pg_basebackup: incompatible server version 12.6 + pg_basebackup: removing data directory "/var/opt/gitlab/postgresql/data" + ERROR: Error when fetching backup: pg_basebackup exited with code=1 + ``` + +**Important**: This workaround applies when the Patroni cluster is in the following state: + +- The [leader has been successfully upgraded to the new major version](../../administration/postgresql/replication_and_failover.md#upgrading-postgresql-major-version-in-a-patroni-cluster). +- The step to upgrade PostgreSQL on replicas is failing. + +This workaround completes the PostgreSQL upgrade on a Patroni replica +by setting the node to use the new PostgreSQL version, and then reinitializing +it as a replica in the new cluster that was created +when the leader was upgraded: + +1. Check the cluster status on all nodes to confirm which is the leader + and what state the replicas are in + + ```shell + sudo gitlab-ctl patroni members + ``` + +1. Replica: check which version of PostgreSQL is active: + + ```shell + sudo ls -al /opt/gitlab/embedded/bin | grep postgres + ``` + +1. Replica: ensure the nodes file is correct and `gitlab-ctl` can run. This resolves + the [errors running `gitlab-ctl`](#errors-running-gitlab-ctl) issue if the replica + has any of those errors as well: + + ```shell + sudo gitlab-ctl stop patroni + sudo gitlab-ctl reconfigure + ``` + +1. Replica: relink the PostgreSQL binaries to the required version + to fix the `incompatible server version` error: + + 1. Edit `/etc/gitlab/gitlab.rb` and specify the required version: + + ```ruby + postgresql['version'] = 13 + ``` + + 1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + + 1. Check the binaries are relinked. The binaries distributed for + PostgreSQL vary between major releases, it's typical to + have a small number of incorrect symbolic links: + + ```shell + sudo ls -al /opt/gitlab/embedded/bin | grep postgres + ``` + +1. Replica: ensure PostgreSQL is fully reinitialized for the specified version: + + ```shell + cd /var/opt/gitlab/postgresql + sudo rm -rf data + sudo gitlab-ctl reconfigure + ``` + +1. Replica: optionally monitor the database in two additional terminal sessions: + + - Disk use increases as `pg_basebackup` runs. Track progress of the + replica initialization with: + + ```shell + cd /var/opt/gitlab/postgresql + watch du -sh data + ``` + + - Monitor the process in the logs: + + ```shell + sudo gitlab-ctl tail patroni + ``` + +1. Replica: Start Patroni to reinitialize the replica: + + ```shell + sudo gitlab-ctl start patroni + ``` + +1. Replica: After it completes, remove the hardcoded version from `/etc/gitlab/gitlab.rb`: + + 1. Edit `/etc/gitlab/gitlab.rb` and remove `postgresql['version']`. + 1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + + 1. Check the correct binaries are linked: + + ```shell + sudo ls -al /opt/gitlab/embedded/bin | grep postgres + ``` + +1. Check the cluster status on all nodes: + + ```shell + sudo gitlab-ctl patroni members + ``` + +Repeat this procedure on the other replica if required. + +## Issues with other components + +If you're running into an issue with a component not outlined here, be sure to check the troubleshooting section of their specific documentation page: + +- [Consul](../consul.md#troubleshooting-consul) +- [PostgreSQL](https://docs.gitlab.com/omnibus/settings/database.html#troubleshooting) diff --git a/doc/administration/raketasks/check.md b/doc/administration/raketasks/check.md index 207dbc7b509..39d7cae5dde 100644 --- a/doc/administration/raketasks/check.md +++ b/doc/administration/raketasks/check.md @@ -215,7 +215,7 @@ secrets file (`gitlab-secrets.json`). Automatic resolution is not yet implemented. If you have values that cannot be decrypted, you can follow steps to reset them, see our -documentation on what to do [when the secrets file is lost](../../administration/backup_restore/backup_gitlab.md#when-the-secrets-file-is-lost). +documentation on what to do [when the secrets file is lost](../../administration/backup_restore/troubleshooting_backup_gitlab.md#when-the-secrets-file-is-lost). This can take a very long time, depending on the size of your database, as it checks all rows in all tables. diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md index 3a136e773d9..21a7ba258c1 100644 --- a/doc/administration/raketasks/maintenance.md +++ b/doc/administration/raketasks/maintenance.md @@ -245,7 +245,7 @@ clear Redis' cache. To do this, run: Sometimes during version upgrades you might end up with some wrong CSS or missing some icons. In that case, try to precompile the assets again. -This Rake task only applies to self-compiled installations. [Read more](../../update/package/index.md#missing-asset-files) +This Rake task only applies to self-compiled installations. [Read more](../../update/package/package_troubleshooting.md#missing-asset-files) about troubleshooting this problem when running the Linux package. The guidance for Linux package might be applicable for Kubernetes and Docker deployments of GitLab, though in general, container-based installations @@ -400,15 +400,13 @@ To manually rebuild a database index: ### Notes - Rebuilding database indexes is a disk-intensive task, so you should perform the -task during off-peak hours. Running the task during peak hours can lead to -_increased_ bloat, and can also cause certain queries to perform slowly. - + task during off-peak hours. Running the task during peak hours can lead to + _increased_ bloat, and can also cause certain queries to perform slowly. - The task requires free disk space for the index being restored. The created -indexes are appended with `_ccnew`. If the reindexing task fails, re-running the -task cleans up the temporary indexes. - + indexes are appended with `_ccnew`. If the reindexing task fails, re-running the + task cleans up the temporary indexes. - The time it takes for database index rebuilding to complete depends on the size -of the target database. It can take between several hours and several days. + of the target database. It can take between several hours and several days. ## Dump the database schema diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index defc74b7e9e..41897a999ec 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -43,7 +43,7 @@ specifically the [Before you start](index.md#before-you-start) and [Deciding whi <!-- markdownlint-disable MD029 --> 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance) and [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Provide your own Redis instances](#provide-your-own-redis-instances) and [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - Redis is primarily single threaded. It's strongly recommended separating out the instances as specified into Cache and Persistent data to achieve optimum performance at this scale. + - Redis is primarily single threaded and doesn't significantly benefit from an increase in CPU cores. For this size of architecture it's strongly recommended having separate Cache and Persistent instances as specified to achieve optimum performance. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. 4. Should be run on reputable Cloud Provider or Self Managed solutions. See [Configure the object storage](#configure-the-object-storage) for more information. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. @@ -707,7 +707,7 @@ SSH in to any of the Patroni nodes on the **primary site**: ``` If the 'State' column for any node doesn't say "running", check the -[PostgreSQL replication and failover troubleshooting section](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) +[PostgreSQL replication and failover troubleshooting section](../../administration/postgresql/replication_and_failover_troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) before proceeding. <div align="right"> @@ -721,6 +721,10 @@ before proceeding. Now that the PostgreSQL servers are all set up, let's configure PgBouncer for tracking and handling reads/writes to the primary database. +NOTE: +PgBouncer is single threaded and doesn't significantly benefit from an increase in CPU cores. +Refer to the [scaling documentation](index.md#scaling-an-environment) for more information. + The following IPs will be used as an example: - `10.6.0.31`: PgBouncer 1 @@ -820,7 +824,8 @@ NOTE: Redis clusters must each be deployed in an odd number of 3 nodes or more. This is to ensure Redis Sentinel can take votes as part of a quorum. This does not apply when configuring Redis externally, such as a cloud provider service. NOTE: -Redis is primarily single threaded. It's strongly recommended separating out the instances as specified into Cache and Persistent data to achieve optimum performance at this scale. +Redis is primarily single threaded and doesn't significantly benefit from an increase in CPU cores. For this size of architecture it's strongly recommended having separate Cache and Persistent instances as specified to achieve optimum performance. +Refer to the [scaling documentation](index.md#scaling-an-environment) for more information. Redis requires authentication if used with Sentinel. See [Redis Security](https://redis.io/docs/manual/security/) documentation for more @@ -1765,6 +1770,10 @@ NOTE: [Because it's recommended to use Object storage](../object_storage.md) instead of NFS for data objects, the following examples include the Object storage configuration. +NOTE: +If you find that the environment's Sidekiq job processing is slow with long queues +you can scale it accordingly. Refer to the [scaling documentation](index.md#scaling-an-environment) for more information. + - `10.6.0.101`: Sidekiq 1 - `10.6.0.102`: Sidekiq 2 - `10.6.0.103`: Sidekiq 3 @@ -1880,6 +1889,15 @@ Updates to example must be made at: 'google_json_key_location' => '<path-to-gcp-service-account-key>' } gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + + gitlab_rails['ci_secure_files_object_store_enabled'] = true + gitlab_rails['ci_secure_files_object_store_remote_directory'] = "gcp-ci_secure_files-bucket-name" + + gitlab_rails['ci_secure_files_object_store_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Linux package node you configured and add or replace @@ -1896,11 +1914,6 @@ Updates to example must be made at: 1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. -NOTE: -If you find that the environment's Sidekiq job processing is slow with long queues, -more nodes can be added as required. You can also tune your Sidekiq nodes to -run [multiple Sidekiq processes](../sidekiq/extra_sidekiq_processes.md). - <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> @@ -2023,6 +2036,14 @@ On each node perform the following: 'google_json_key_location' => '<path-to-gcp-service-account-key>' } gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + gitlab_rails['ci_secure_files_object_store_enabled'] = true + gitlab_rails['ci_secure_files_object_store_remote_directory'] = "gcp-ci_secure_files-bucket-name" + + gitlab_rails['ci_secure_files_object_store_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } ``` 1. If you're using [Gitaly with TLS support](#gitaly-cluster-tls-support), make sure the @@ -2266,7 +2287,7 @@ the overall makeup as desired as long as the minimum CPU and Memory requirements | Supporting services | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | 7.75 vCPU, 25 GB memory | - For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) -[Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. + [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. - Nodes configuration is shown as it is forced to ensure pod vCPU / memory ratios and avoid scaling during **performance testing**. - In production deployments, there is no need to assign pods to specific nodes. A minimum of three nodes per node group in three different availability zones is strongly recommended to align with resilient cloud architecture practices. @@ -2290,7 +2311,7 @@ services where applicable): <!-- markdownlint-disable MD029 --> 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance) and [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Provide your own Redis instances](#provide-your-own-redis-instances) and [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - Redis is primarily single threaded. It's strongly recommended separating out the instances as specified into Cache and Persistent data to achieve optimum performance at this scale. + - Redis is primarily single threaded and doesn't significantly benefit from an increase in CPU cores. For this size of architecture it's strongly recommended having separate Cache and Persistent instances as specified to achieve optimum performance. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. 4. Should be run on reputable Cloud Provider or Self Managed solutions. See [Configure the object storage](#configure-the-object-storage) for more information. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 6520f63957b..2f0b51e6662 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -43,7 +43,7 @@ specifically the [Before you start](index.md#before-you-start) and [Deciding whi <!-- markdownlint-disable MD029 --> 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance) for more information. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Provide your own Redis instances](#provide-your-own-redis-instances) for more information. - - Redis is primarily single threaded. It's strongly recommended separating out the instances as specified into Cache and Persistent data to achieve optimum performance at this scale. + - Redis is primarily single threaded and doesn't significantly benefit from an increase in CPU cores. For this size of architecture it's strongly recommended having separate Cache and Persistent instances as specified to achieve optimum performance. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. 4. Should be run on reputable Cloud Provider or Self Managed solutions. See [Configure the object storage](#configure-the-object-storage) for more information. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. @@ -715,7 +715,7 @@ SSH in to any of the Patroni nodes on the **primary site**: ``` If the 'State' column for any node doesn't say "running", check the -[PostgreSQL replication and failover troubleshooting section](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) +[PostgreSQL replication and failover troubleshooting section](../../administration/postgresql/replication_and_failover_troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) before proceeding. <div align="right"> @@ -729,6 +729,10 @@ before proceeding. Now that the PostgreSQL servers are all set up, let's configure PgBouncer for tracking and handling reads/writes to the primary database. +NOTE: +PgBouncer is single threaded and doesn't significantly benefit from an increase in CPU cores. +Refer to the [scaling documentation](index.md#scaling-an-environment) for more information. + The following IPs will be used as an example: - `10.6.0.31`: PgBouncer 1 @@ -828,7 +832,8 @@ NOTE: Redis clusters must each be deployed in an odd number of 3 nodes or more. This is to ensure Redis Sentinel can take votes as part of a quorum. This does not apply when configuring Redis externally, such as a cloud provider service. NOTE: -Redis is primarily single threaded. It's strongly recommended separating out the instances as specified into Cache and Persistent data to achieve optimum performance at this scale. +Redis is primarily single threaded and doesn't significantly benefit from an increase in CPU cores. For this size of architecture it's strongly recommended having separate Cache and Persistent instances as specified to achieve optimum performance. +Refer to the [scaling documentation](index.md#scaling-an-environment) for more information. Redis requires authentication if used with Sentinel. See [Redis Security](https://redis.io/docs/manual/security/) documentation for more @@ -1500,7 +1505,7 @@ Updates to example must be made at: ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Linux package node you configured and add or replace -the file of the same name on this server. If this is the first Linux package node you are configuring then you can skip this step. + the file of the same name on this server. If this is the first Linux package node you are configuring then you can skip this step. 1. Praefect requires to run some database migrations, much like the main GitLab application. For this you should select **one Praefect node only to run the migrations**, AKA the _Deploy Node_. This node @@ -1773,6 +1778,10 @@ NOTE: [Because it's recommended to use Object storage](../object_storage.md) instead of NFS for data objects, the following examples include the Object storage configuration. +NOTE: +If you find that the environment's Sidekiq job processing is slow with long queues +you can scale it accordingly. Refer to the [scaling documentation](index.md#scaling-an-environment) for more information. + - `10.6.0.101`: Sidekiq 1 - `10.6.0.102`: Sidekiq 2 - `10.6.0.103`: Sidekiq 3 @@ -1888,6 +1897,15 @@ Updates to example must be made at: 'google_json_key_location' => '<path-to-gcp-service-account-key>' } gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + + gitlab_rails['ci_secure_files_object_store_enabled'] = true + gitlab_rails['ci_secure_files_object_store_remote_directory'] = "gcp-ci_secure_files-bucket-name" + + gitlab_rails['ci_secure_files_object_store_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Linux package node you configured and add or replace @@ -1904,11 +1922,6 @@ Updates to example must be made at: 1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. -NOTE: -If you find that the environment's Sidekiq job processing is slow with long queues, -more nodes can be added as required. You can also tune your Sidekiq nodes to -run [multiple Sidekiq processes](../sidekiq/extra_sidekiq_processes.md). - <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> @@ -2033,6 +2046,14 @@ On each node perform the following: 'google_json_key_location' => '<path-to-gcp-service-account-key>' } gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + gitlab_rails['ci_secure_files_object_store_enabled'] = true + gitlab_rails['ci_secure_files_object_store_remote_directory'] = "gcp-ci_secure_files-bucket-name" + + gitlab_rails['ci_secure_files_object_store_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } ``` 1. If you're using [Gitaly with TLS support](#gitaly-cluster-tls-support), make sure the @@ -2276,7 +2297,7 @@ the overall makeup as desired as long as the minimum CPU and Memory requirements | Supporting services | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | 7.75 vCPU, 25 GB memory | - For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) -[Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. + [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. - Nodes configuration is shown as it is forced to ensure pod vCPU / memory ratios and avoid scaling during **performance testing**. - In production deployments, there is no need to assign pods to specific nodes. A minimum of three nodes per node group in three different availability zones is strongly recommended to align with resilient cloud architecture practices. @@ -2300,7 +2321,7 @@ services where applicable): <!-- markdownlint-disable MD029 --> 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance) for more information. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Provide your own Redis instances](#provide-your-own-redis-instances) for more information. - - Redis is primarily single threaded. It's strongly recommended separating out the instances as specified into Cache and Persistent data to achieve optimum performance at this scale. + - Redis is primarily single threaded and doesn't significantly benefit from an increase in CPU cores. For this size of architecture it's strongly recommended having separate Cache and Persistent instances as specified to achieve optimum performance. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. 4. Should be run on reputable Cloud Provider or Self Managed solutions. See [Configure the object storage](#configure-the-object-storage) for more information. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index d3f53ce3e14..dd71190b76d 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -24,7 +24,7 @@ For a full list of reference architectures, see | Load balancer<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | PostgreSQL<sup>1</sup> | 1 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | | Redis<sup>2</sup> | 1 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `m5.large` | `D2s v3` | -| Gitaly | 1 | 4 vCPU, 15 GB memory<sup>5</sup> | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Gitaly<sup>5</sup> | 1 | 4 vCPU, 15 GB memory<sup>5</sup> | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | Sidekiq<sup>6</sup> | 1 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | GitLab Rails<sup>6</sup> | 2 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | | Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | @@ -35,7 +35,6 @@ For a full list of reference architectures, see 2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Provide your own Redis instance](#provide-your-own-redis-instance) and [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. 4. Should be run on reputable Cloud Provider or Self Managed solutions. See [Configure the object storage](#configure-the-object-storage) for more information. -4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. 5. Gitaly specifications are based on the use of normal-sized repositories in good health. However, if you have large monorepos (larger than several gigabytes) this can **significantly** impact Git and Gitaly performance and an increase of specifications will likely be required. Refer to [large monorepos](index.md#large-monorepos) for more information. @@ -52,7 +51,7 @@ skinparam linetype ortho card "**External Load Balancer**" as elb #6a9be7 -collections "**GitLab Rails** x3" as gitlab #32CD32 +collections "**GitLab Rails** x2" as gitlab #32CD32 card "**Prometheus**" as monitor #7FFFD4 card "**Gitaly**" as gitaly #FF8C00 card "**PostgreSQL**" as postgres #4EA7FF @@ -340,6 +339,10 @@ are supported and can be added if needed. In this section, you'll be guided through configuring an external Redis instance to be used with GitLab. +NOTE: +Redis is primarily single threaded and doesn't significantly benefit from an increase in CPU cores. +Refer to the [scaling documentation](index.md#scaling-an-environment) for more information. + ### Provide your own Redis instance You can optionally use a [third party external service for the Redis instance](../redis/replication_and_failover_external.md#redis-as-a-managed-service-in-a-cloud-provider) with the following guidance: @@ -603,6 +606,10 @@ Sidekiq requires connection to the [Redis](#configure-redis), [PostgreSQL](#configure-postgresql) and [Gitaly](#configure-gitaly) instances. It also requires a connection to [Object Storage](#configure-the-object-storage) as recommended. +NOTE: +If you find that the environment's Sidekiq job processing is slow with long queues +you can scale it accordingly. Refer to the [scaling documentation](index.md#scaling-an-environment) for more information. + To configure the Sidekiq server, on the server node you want to use for Sidekiq: 1. SSH in to the Sidekiq server. @@ -687,6 +694,14 @@ Updates to example must be made at: 'google_json_key_location' => '<path-to-gcp-service-account-key>' } gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + gitlab_rails['ci_secure_files_object_store_enabled'] = true + gitlab_rails['ci_secure_files_object_store_remote_directory'] = "gcp-ci_secure_files-bucket-name" + + gitlab_rails['ci_secure_files_object_store_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Linux package node you configured and add or replace @@ -717,11 +732,6 @@ Updates to example must be made at: run: sidekiq: (pid 26870) 92996s; run: log: (pid 26391) 93042s ``` -NOTE: -If you find that the environment's Sidekiq job processing is slow with long queues, -more nodes can be added as required. You can also tune your Sidekiq nodes to -run [multiple Sidekiq processes](../sidekiq/extra_sidekiq_processes.md). - <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> @@ -820,6 +830,15 @@ On each node perform the following: } gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + gitlab_rails['ci_secure_files_object_store_enabled'] = true + gitlab_rails['ci_secure_files_object_store_remote_directory'] = "gcp-ci_secure_files-bucket-name" + + gitlab_rails['ci_secure_files_object_store_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } + ## Uncomment and edit the following options if you have set up NFS ## ## Prevent GitLab from starting if NFS data mounts are not available @@ -1098,7 +1117,7 @@ the overall makeup as desired as long as the minimum CPU and Memory requirements | Supporting services | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | 1.9 vCPU, 5.5 GB memory | - For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) -[Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. + [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. - Nodes configuration is shown as it is forced to ensure pod vCPU / memory ratios and avoid scaling during **performance testing**. - In production deployments, there is no need to assign pods to specific nodes. A minimum of three nodes per node group in three different availability zones is strongly recommended to align with resilient cloud architecture practices. diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index c01564a3e7a..5d0ab62d4a2 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -433,6 +433,10 @@ NOTE: Multi-node Redis must be deployed in an odd number of 3 nodes or more to ensure Redis Sentinel can take votes as part of a quorum. This does not apply when configuring Redis externally, such as a cloud provider service. +NOTE: +Redis is primarily single threaded and doesn't significantly benefit from an increase in CPU cores. +Refer to the [scaling documentation](index.md#scaling-an-environment) for more information. + Redis requires authentication if used with Sentinel. See [Redis Security](https://redis.io/docs/manual/security/) documentation for more information. We recommend using a combination of a Redis password and tight @@ -977,7 +981,7 @@ SSH in to any of the Patroni nodes on the **primary site**: ``` If the 'State' column for any node doesn't say "running", check the -[PostgreSQL replication and failover troubleshooting section](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) +[PostgreSQL replication and failover troubleshooting section](../../administration/postgresql/replication_and_failover_troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) before proceeding. <div align="right"> @@ -991,6 +995,10 @@ before proceeding. Now that the PostgreSQL servers are all set up, let's configure PgBouncer for tracking and handling reads/writes to the primary database. +NOTE: +PgBouncer is single threaded and doesn't significantly benefit from an increase in CPU cores. +Refer to the [scaling documentation](index.md#scaling-an-environment) for more information. + The following IPs will be used as an example: - `10.6.0.31`: PgBouncer 1 @@ -1415,7 +1423,7 @@ Updates to example must be made at: ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Linux package node you configured and add or replace -the file of the same name on this server. If this is the first Linux package node you are configuring then you can skip this step. + the file of the same name on this server. If this is the first Linux package node you are configuring then you can skip this step. 1. Praefect requires to run some database migrations, much like the main GitLab application. For this you should select **one Praefect node only to run the migrations**, AKA the _Deploy Node_. This node @@ -1692,6 +1700,10 @@ NOTE: [Because it's recommended to use Object storage](../object_storage.md) instead of NFS for data objects, the following examples include the Object storage configuration. +NOTE: +If you find that the environment's Sidekiq job processing is slow with long queues +you can scale it accordingly. Refer to the [scaling documentation](index.md#scaling-an-environment) for more information. + The following IPs will be used as an example: - `10.6.0.71`: Sidekiq 1 @@ -1800,6 +1812,15 @@ Updates to example must be made at: 'google_json_key_location' => '<path-to-gcp-service-account-key>' } gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + + gitlab_rails['ci_secure_files_object_store_enabled'] = true + gitlab_rails['ci_secure_files_object_store_remote_directory'] = "gcp-ci_secure_files-bucket-name" + + gitlab_rails['ci_secure_files_object_store_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Linux package node you configured and add or replace @@ -1831,11 +1852,6 @@ Updates to example must be made at: run: sidekiq: (pid 30142) 77351s; run: log: (pid 29638) 77386s ``` -NOTE: -If you find that the environment's Sidekiq job processing is slow with long queues, -more nodes can be added as required. You can also tune your Sidekiq nodes to -run [multiple Sidekiq processes](../sidekiq/extra_sidekiq_processes.md). - <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> @@ -1969,6 +1985,14 @@ On each node perform the following: 'google_json_key_location' => '<path-to-gcp-service-account-key>' } gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + gitlab_rails['ci_secure_files_object_store_enabled'] = true + gitlab_rails['ci_secure_files_object_store_remote_directory'] = "gcp-ci_secure_files-bucket-name" + + gitlab_rails['ci_secure_files_object_store_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } ``` 1. If you're using [Gitaly with TLS support](#gitaly-cluster-tls-support), make sure the @@ -2250,7 +2274,7 @@ the overall makeup as desired as long as the minimum CPU and Memory requirements | Supporting services | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | 3.9 vCPU, 11.8 GB memory | - For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) -[Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. + [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. - Nodes configuration is shown as it is forced to ensure pod vCPU / memory ratios and avoid scaling during **performance testing**. - In production deployments, there is no need to assign pods to specific nodes. A minimum of three nodes per node group in three different availability zones is strongly recommended to align with resilient cloud architecture practices. diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 92421a35273..ed66620119e 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -43,7 +43,7 @@ specifically the [Before you start](index.md#before-you-start) and [Deciding whi <!-- markdownlint-disable MD029 --> 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance) for more information. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Provide your own Redis instances](#provide-your-own-redis-instances) for more information. - - Redis is primarily single threaded. It's strongly recommended separating out the instances as specified into Cache and Persistent data to achieve optimum performance at this scale. + - Redis is primarily single threaded and doesn't significantly benefit from an increase in CPU cores. For this size of architecture it's strongly recommended having separate Cache and Persistent instances as specified to achieve optimum performance. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. 4. Should be run on reputable Cloud Provider or Self Managed solutions. See [Configure the object storage](#configure-the-object-storage) for more information. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. @@ -717,7 +717,7 @@ SSH in to any of the Patroni nodes on the **primary site**: ``` If the 'State' column for any node doesn't say "running", check the -[PostgreSQL replication and failover troubleshooting section](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) +[PostgreSQL replication and failover troubleshooting section](../../administration/postgresql/replication_and_failover_troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) before proceeding. <div align="right"> @@ -731,6 +731,10 @@ before proceeding. Now that the PostgreSQL servers are all set up, let's configure PgBouncer for tracking and handling reads/writes to the primary database. +NOTE: +PgBouncer is single threaded and doesn't significantly benefit from an increase in CPU cores. +Refer to the [scaling documentation](index.md#scaling-an-environment) for more information. + The following IPs will be used as an example: - `10.6.0.31`: PgBouncer 1 @@ -830,7 +834,9 @@ NOTE: Redis clusters must each be deployed in an odd number of 3 nodes or more. This is to ensure Redis Sentinel can take votes as part of a quorum. This does not apply when configuring Redis externally, such as a cloud provider service. NOTE: -Redis is primarily single threaded. It's strongly recommended separating out the instances as specified into Cache and Persistent data to achieve optimum performance at this scale. +Redis is primarily single threaded and doesn't significantly benefit from increasing CPU cores. +For this size of architecture it's strongly recommended having separate Cache and Persistent instances as specified to achieve optimum performance at this scale. +Refer to the [scaling documentation](index.md#scaling-an-environment) for more information. Redis requires authentication if used with Sentinel. See [Redis Security](https://redis.io/docs/manual/security/) documentation for more @@ -1505,7 +1511,7 @@ Updates to example must be made at: ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Linux package node you configured and add or replace -the file of the same name on this server. If this is the first Linux package node you are configuring then you can skip this step. + the file of the same name on this server. If this is the first Linux package node you are configuring then you can skip this step. 1. Praefect requires to run some database migrations, much like the main GitLab application. For this you should select **one Praefect node only to run the migrations**, AKA the _Deploy Node_. This node @@ -1778,6 +1784,11 @@ NOTE: [Because it's recommended to use Object storage](../object_storage.md) instead of NFS for data objects, the following examples include the Object storage configuration. +NOTE: +If you find that the environment's Sidekiq job processing is slow with long queues +you can scale it accordingly. +Refer to the [scaling documentation](index.md#scaling-an-environment) for more information. + - `10.6.0.101`: Sidekiq 1 - `10.6.0.102`: Sidekiq 2 - `10.6.0.103`: Sidekiq 3 @@ -1893,6 +1904,15 @@ Updates to example must be made at: 'google_json_key_location' => '<path-to-gcp-service-account-key>' } gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + + gitlab_rails['ci_secure_files_object_store_enabled'] = true + gitlab_rails['ci_secure_files_object_store_remote_directory'] = "gcp-ci_secure_files-bucket-name" + + gitlab_rails['ci_secure_files_object_store_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Linux package node you configured and add or replace @@ -1909,11 +1929,6 @@ Updates to example must be made at: 1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. -NOTE: -If you find that the environment's Sidekiq job processing is slow with long queues, -more nodes can be added as required. You can also tune your Sidekiq nodes to -run [multiple Sidekiq processes](../sidekiq/extra_sidekiq_processes.md). - <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> @@ -2044,7 +2059,16 @@ On each node perform the following: 'google_project' => '<gcp-project-name>', 'google_json_key_location' => '<path-to-gcp-service-account-key>' } + gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + gitlab_rails['ci_secure_files_object_store_enabled'] = true + gitlab_rails['ci_secure_files_object_store_remote_directory'] = "gcp-ci_secure_files-bucket-name" + + gitlab_rails['ci_secure_files_object_store_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } ``` 1. If you're using [Gitaly with TLS support](#gitaly-cluster-tls-support), make sure the @@ -2284,7 +2308,7 @@ the overall makeup as desired as long as the minimum CPU and Memory requirements | Supporting services | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | 7.75 vCPU, 25 GB memory | - For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) -[Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. + [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. - Nodes configuration is shown as it is forced to ensure pod vCPU / memory ratios and avoid scaling during **performance testing**. - In production deployments, there is no need to assign pods to specific nodes. A minimum of three nodes per node group in three different availability zones is strongly recommended to align with resilient cloud architecture practices. @@ -2308,7 +2332,7 @@ services where applicable): <!-- markdownlint-disable MD029 --> 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance) for more information. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Provide your own Redis instances](#provide-your-own-redis-instances) for more information. - - Redis is primarily single threaded. It's strongly recommended separating out the instances as specified into Cache and Persistent data to achieve optimum performance at this scale. + - Redis is primarily single threaded and doesn't significantly benefit from an increase in CPU cores. For this size of architecture it's strongly recommended having separate Cache and Persistent instances as specified to achieve optimum performance. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. 4. Should be run on reputable Cloud Provider or Self Managed solutions. See [Configure the object storage](#configure-the-object-storage) for more information. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 16a92944984..65b46474f7a 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -429,6 +429,14 @@ Using [Redis](https://redis.io/) in scalable environment is possible using a **P topology with a [Redis Sentinel](https://redis.io/docs/manual/sentinel/) service to watch and automatically start the failover procedure. +NOTE: +Multi-node Redis must be deployed in an odd number of 3 nodes or more to ensure Redis Sentinel can take votes as part of a quorum. This does not apply when configuring Redis externally, +such as a cloud provider service. + +NOTE: +Redis is primarily single threaded and doesn't significantly benefit from an increase in CPU cores. +Refer to the [scaling documentation](index.md#scaling-an-environment) for more information. + Redis requires authentication if used with Sentinel. See [Redis Security](https://redis.io/docs/manual/security/) documentation for more information. We recommend using a combination of a Redis password and tight @@ -972,7 +980,7 @@ SSH in to any of the Patroni nodes on the **primary site**: ``` If the 'State' column for any node doesn't say "running", check the -[PostgreSQL replication and failover troubleshooting section](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) +[PostgreSQL replication and failover troubleshooting section](../../administration/postgresql/replication_and_failover_troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) before proceeding. <div align="right"> @@ -986,6 +994,10 @@ before proceeding. Now that the PostgreSQL servers are all set up, let's configure PgBouncer for tracking and handling reads/writes to the primary database. +NOTE: +PgBouncer is single threaded and doesn't significantly benefit from an increase in CPU cores. +Refer to the [scaling documentation](index.md#scaling-an-environment) for more information. + The following IPs are used as an example: - `10.6.0.31`: PgBouncer 1 @@ -1411,7 +1423,7 @@ Updates to example must be made at: ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Linux package node you configured and add or replace -the file of the same name on this server. If this is the first Linux package node you are configuring then you can skip this step. + the file of the same name on this server. If this is the first Linux package node you are configuring then you can skip this step. 1. Praefect requires to run some database migrations, much like the main GitLab application. For this you should select **one Praefect node only to run the migrations**, AKA the _Deploy Node_. This node @@ -1687,7 +1699,7 @@ examples include the Object storage configuration. - `10.6.0.71`: Sidekiq 1 - `10.6.0.72`: Sidekiq 2 -To configure the Sidekiq nodes, one each one: +To configure the Sidekiq nodes, on each one: 1. SSH in to the Sidekiq server. 1. [Download and install](https://about.gitlab.com/install/) the Linux package @@ -1791,6 +1803,15 @@ Updates to example must be made at: 'google_json_key_location' => '<path-to-gcp-service-account-key>' } gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + + gitlab_rails['ci_secure_files_object_store_enabled'] = true + gitlab_rails['ci_secure_files_object_store_remote_directory'] = "gcp-ci_secure_files-bucket-name" + + gitlab_rails['ci_secure_files_object_store_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Linux package node you configured and add or replace @@ -1948,7 +1969,15 @@ On each node perform the following: 'google_json_key_location' => '<path-to-gcp-service-account-key>' } gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + gitlab_rails['ci_secure_files_object_store_enabled'] = true + gitlab_rails['ci_secure_files_object_store_remote_directory'] = "gcp-ci_secure_files-bucket-name" + gitlab_rails['ci_secure_files_object_store_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } + ## Uncomment and edit the following options if you have set up NFS ## ## Prevent GitLab from starting if NFS data mounts are not available @@ -2220,7 +2249,7 @@ the overall makeup as desired as long as the minimum CPU and Memory requirements | Supporting services | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | 3.9 vCPU, 11.8 GB memory | - For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) -[Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. + [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. - Nodes configuration is shown as it is forced to ensure pod vCPU / memory ratios and avoid scaling during **performance testing**. - In production deployments, there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices. diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index c4827695716..7e4e929f80d 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -1,6 +1,7 @@ --- stage: Systems group: Distribution +description: Recommended deployments at scale. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -13,8 +14,7 @@ GitLab Quality Engineering and Support teams to provide recommended deployments The following Reference Architectures are available as recommended starting points for your environment. -The architectures are named in terms of user count, which in this case means the architecture is designed against -the _total_ load that comes with such a user count based on real data along with substantial headroom added to cover most scenarios such as CI or other automated workloads. +The architectures are named in terms of _total_ load, both manual and automated, correlated to user count and based on real data along with substantial headroom added to add additional coverage for most scenarios. However, it should be noted that in some cases, known heavy scenarios such as [large monorepos](#large-monorepos) or notable [additional workloads](#additional-workloads) may require adjustments to be made. @@ -679,32 +679,44 @@ You should upgrade a Reference Architecture in the same order as you created it. ### Scaling an environment -Scaling a GitLab environment is designed to be as flexible and seamless as possible. +The Reference Architectures have been designed to support scaling in various ways depending on your use case and circumstances. +This can be done iteratively or wholesale to the next size of architecture depending on if metrics suggest a component is being exhausted. -This can be done iteratively or wholesale to the next size of architecture depending on your circumstances. -For example, if any of your GitLab Rails, Sidekiq, Gitaly, Redis or PostgreSQL nodes are consistently oversaturated, then increase their resources accordingly while leaving the rest of the environment as is. +NOTE: +If you're seeing a component continuously exhausting it's given resources it's strongly recommended for you to reach out to our [Support team](https://about.gitlab.com/support/) before performing any scaling. This is especially so if you're planning to scale any component significantly. -If expecting a large increase in users, you may elect to scale up the whole environment to the next -size of architecture. +For most components vertical and horizontal scaling can be applied as normal. However, before doing so though please be aware of the below caveats: -If the overall design is being followed, you can scale the environment vertically as required. +- When scaling Puma or Sidekiq vertically the amount of workers will need to be adjusted to use the additional specs. Puma will be scaled automatically on the next reconfigure but Sidekiq will need [its configuration changed beforehand](../sidekiq/extra_sidekiq_processes.md#start-multiple-processes). +- Redis and PgBouncer are primarily single threaded. If these components are seeing CPU exhaustion they may need to be scaled out horizontally. +- Scaling certain components significantly can result in notable knock on effects that affect the performance of the environment. [Refer to the dedicated section below for more guidance](#scaling-knock-on-effects). -If robust metrics are in place that show the environment is over-provisioned, you can apply the same process for -scaling downwards. You should take an iterative approach when scaling downwards to ensure there are no issues. +Conversely, if you have robust metrics in place that show the environment is over-provisioned, you can scale downwards similarly. +You should take an iterative approach when scaling downwards, however, to ensure there are no issues. -#### Scaling from a non-HA to an HA architecture +#### Scaling knock on effects -While in most cases vertical scaling is only required to increase an environment's resources, if you are moving to an HA environment, -there may be some additional steps required as shown below: +In some cases scaling a component significantly may result in knock on effects for downstream components, impacting performance. The Reference Architectures were designed with balance in mind to ensure components that depend on each other are congruent in terms of specs. As such you may find when notably scaling a component that it's increase may result in additional throughput being passed to the other components it depends on and that they, in turn, may need to be scaled as well. -- If you're scaling from a non-HA environment to an HA environment, various components are recommended to be deployed in their HA forms: - - [Redis to multi-node Redis w/ Redis Sentinel](../redis/replication_and_failover.md#switching-from-an-existing-single-machine-installation) - - [Postgres to multi-node Postgres w/ Consul + PgBouncer](../postgresql/moving.md) - - [Gitaly to Gitaly Cluster w/ Praefect](../gitaly/index.md#migrate-to-gitaly-cluster) -- From 10k users and higher, Redis is recommended to be split into multiple HA servers as it's single threaded. +NOTE: +As a general rule most components have good headroom to accommodate an upstream component being scaled, so this is typically on a case by case basis and specific to what has been changed. It's recommended for you to reach out to our [Support team](https://about.gitlab.com/support/) before you make any significant changes to the environment. -Conversely, if you have robust metrics in place that show the environment is over-provisioned, you can apply the same process for -scaling downwards. You should take an iterative approach when scaling downwards, however, to ensure there are no issues. +The following components can impact others when they have been significantly scaled: + +- Puma and Sidekiq - Notable scale ups of either Puma or Sidekiq workers will result in higher concurrent connections to the internal load balancer, PostgreSQL (via PgBouncer if present), Gitaly (via Praefect if present) and Redis respectively. + - Redis is primarily single threaded and in some cases may need to be split up into different instances (Cache / Persistent) if the increased throughput causes CPU exhaustion if a combined cluster is currently being used. + - PgBouncer is also single threaded but note that a scale out will result in a new pool being added that in turn will increase total connections to Postgres. It's strongly recommended to only do this if you have experience in managing Postgres connections and to seek assistance if in doubt. +- Gitaly Cluster / PostgreSQL - A notable scale out of additional nodes can have a detrimental effect on the HA system and performance due to increased replication calls to the primary node. + +#### Scaling from a non-HA to an HA architecture + +While in most cases vertical scaling is only required to increase an environment's resources, if you are moving to an HA environment +additional steps will be required for the following components to switch over to their HA forms respectively by following the given +documentation for each as follows + +- [Redis to multi-node Redis w/ Redis Sentinel](../redis/replication_and_failover.md#switching-from-an-existing-single-machine-installation) +- [Postgres to multi-node Postgres w/ Consul + PgBouncer](../postgresql/moving.md) +- [Gitaly to Gitaly Cluster w/ Praefect](../gitaly/index.md#migrate-to-gitaly-cluster) ### Monitoring @@ -721,6 +733,7 @@ You can find a full history of changes [on the GitLab project](https://gitlab.co **2023:** - [2023-12-12](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/133457): Updated notes on Load Balancers to be more reflective that any reputable offering is expected to work. +- [2023-12-12](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/133457): Updated notes on Load Balancers to be more reflective that any reputable offering is expected to work. - [2023-11-03](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/133457): Expanded details on what each Reference Architecture is designed for, the testing methodology used as well as added details on how to scale environments. - [2023-11-03](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/134518): Added expanded notes on disk types, object storage and monitoring. - [2023-10-25](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/134518): Adjusted Sidekiq configuration example to use Linux Package role. diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md deleted file mode 100644 index 844dc76b23d..00000000000 --- a/doc/administration/repository_storage_types.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'repository_storage_paths.md' -remove_date: '2023-11-29' ---- - -This document was moved to [another location](repository_storage_paths.md). - -<!-- This redirect file can be deleted after <2023-11-29>. --> diff --git a/doc/administration/settings/email.md b/doc/administration/settings/email.md index cc80b082139..a010a855ba1 100644 --- a/doc/administration/settings/email.md +++ b/doc/administration/settings/email.md @@ -10,7 +10,7 @@ You can customize some of the content in emails sent from your GitLab instance. ## Custom logo -The logo in the header of some emails can be customized, see the [logo customization section](../../administration/appearance.md#navigation-bar). +The logo in the header of some emails can be customized, see the [logo customization section](../../administration/appearance.md#customize-your-homepage-button). ## Include author name in email notification email body **(PREMIUM SELF)** diff --git a/doc/administration/settings/help_page.md b/doc/administration/settings/help_page.md index 1077bbec316..bb7f214a106 100644 --- a/doc/administration/settings/help_page.md +++ b/doc/administration/settings/help_page.md @@ -55,7 +55,7 @@ GitLab marketing-related entries are occasionally shown on the Help page. To hid You can specify a custom URL to which users are directed when they: -- Select **Support** from the Help dropdown list. +- Select **Help > Support**. - Select **See our website for help** on the Help page. 1. On the left sidebar, at the bottom, select **Admin Area**. diff --git a/doc/administration/settings/import_and_export_settings.md b/doc/administration/settings/import_and_export_settings.md index ddb31e49483..54995cdc686 100644 --- a/doc/administration/settings/import_and_export_settings.md +++ b/doc/administration/settings/import_and_export_settings.md @@ -121,7 +121,7 @@ To modify this setting: > - **Maximum decompressed file size for archives from imports** field [renamed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/130081) from **Maximum decompressed size** in GitLab 16.4. When you import a project using [file exports](../../user/project/settings/import_export.md) or -[direct transfer](../../user/group/import/index.md#migrate-groups-by-direct-transfer-recommended), you can specify the +[direct transfer](../../user/group/import/index.md), you can specify the maximum decompressed file size for imported archives. The default value is 25 GiB. When you import a compressed file, the decompressed size cannot exceed the maximum decompressed file size limit. If the diff --git a/doc/administration/settings/index.md b/doc/administration/settings/index.md index 531073887c5..2df9d1cd52d 100644 --- a/doc/administration/settings/index.md +++ b/doc/administration/settings/index.md @@ -1,6 +1,7 @@ --- stage: none group: unassigned +description: Product settings. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/administration/settings/jira_cloud_app_troubleshooting.md b/doc/administration/settings/jira_cloud_app_troubleshooting.md index 1b6f88d43ae..cc17c620724 100644 --- a/doc/administration/settings/jira_cloud_app_troubleshooting.md +++ b/doc/administration/settings/jira_cloud_app_troubleshooting.md @@ -10,7 +10,7 @@ When administering the GitLab for Jira Cloud app for self-managed instances, you For GitLab.com, see [GitLab for Jira Cloud app](../../integration/jira/connect-app.md#troubleshooting). -## Browser displays a sign-in message when already signed in +## Sign-in message displayed when already signed in You might get the following message prompting you to sign in to GitLab.com when you're already signed in: @@ -26,12 +26,17 @@ To resolve this issue, set up [OAuth authentication](jira_cloud_app.md#set-up-oa ## Manual installation fails -You might get an error if you have installed the GitLab for Jira Cloud app from the official marketplace listing and replaced it with [manual installation](jira_cloud_app.md#install-the-gitlab-for-jira-cloud-app-manually): +You might get one of the following errors if you've installed the GitLab for Jira Cloud app +from the official marketplace listing and replaced it with [manual installation](jira_cloud_app.md#install-the-gitlab-for-jira-cloud-app-manually): ```plaintext The app "gitlab-jira-connect-gitlab.com" could not be installed as a local app as it has previously been installed from Atlassian Marketplace ``` +```plaintext +The app host returned HTTP response code 401 when we tried to contact it during installation. Please try again later or contact the app vendor. +``` + To resolve this issue, disable the **Jira Connect Proxy URL** setting. - In GitLab 15.7: @@ -47,7 +52,7 @@ To resolve this issue, disable the **Jira Connect Proxy URL** setting. 1. Clear the **Jira Connect Proxy URL** text box. 1. Select **Save changes**. -## Data sync fails with `Invalid JWT` error +## Data sync fails with `Invalid JWT` If the GitLab for Jira Cloud app continuously fails to sync data, it may be due to an outdated secret token. Atlassian can send new secret tokens that must be processed and stored by GitLab. If GitLab fails to store the token or misses the new token request, an `Invalid JWT` error occurs. @@ -101,13 +106,16 @@ due to a [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/388943). To ### Debugging Jira Connect Proxy issues -If you are using a self-managed GitLab instance and you have configured `https://gitlab.com` for the Jira Connect Proxy URL when -[setting up the OAuth authentication](jira_cloud_app.md#set-up-oauth-authentication), you can inspect the network traffic in your browser's development -tools while reproducing the `Failed to update the GitLab instance` error to see a more precise error. +If you set **Jira Connect Proxy URL** to `https://gitlab.com` when you +[set up your instance](jira_cloud_app.md#set-up-your-instance), you can: + +- Inspect the network traffic in your browser's development tools. +- Reproduce the `Failed to update the GitLab instance` error for more information. You should see a `GET` request to `https://gitlab.com/-/jira_connect/installations`. -This request should return a `200` status code, but it can return a `422` status code if there was a problem. The response body can be checked for the error. +This request should return a `200 OK`, but it might return a `422 Unprocessable Entity` if there was a problem. +You can check the response body for the error. If you cannot resolve the problem and you are a GitLab customer, contact [GitLab Support](https://about.gitlab.com/support/) for assistance. Provide GitLab Support with: @@ -119,7 +127,7 @@ GitLab Support with: The GitLab Support team can then look up why this is failing in the GitLab.com server logs. -#### Process for GitLab Support +#### GitLab Support NOTE: These steps can only be completed by GitLab Support. @@ -138,12 +146,12 @@ For the second log: - `json.message` is `Proxy lifecycle event received error response` or similar. - `json.jira_status_code` and `json.jira_body` might contain details on why GitLab.com wasn't able to connect back to the self-managed instance. -- If `json.jira_status_code` is `401` and `json.jira_body` is empty, this might indicate that the [**Jira Connect Proxy URL**](jira_cloud_app.md#set-up-your-instance) is not set to +- If `json.jira_status_code` is `401` and `json.jira_body` is empty, [**Jira Connect Proxy URL**](jira_cloud_app.md#set-up-your-instance) might not be set to `https://gitlab.com`. -## `Failed to link group` +## Error when connecting the app -After you connect the GitLab for Jira Cloud app for self-managed instances, you might get one of these errors: +When you connect the GitLab for Jira Cloud app, you might get one of these errors: ```plaintext Failed to load Jira Connect Application ID. Please try again. @@ -159,6 +167,7 @@ When you check the browser console, you might see the following message: Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://gitlab.example.com/-/jira_connect/oauth_application_id. (Reason: CORS header 'Access-Control-Allow-Origin' missing). Status code: 403. ``` -`403` status code is returned if the user information cannot be fetched from Jira because of insufficient permissions. +A `403 Forbidden` is returned if the user information cannot be fetched from Jira because of insufficient permissions. -To resolve this issue, ensure that the Jira user that installs and configures the GitLab for Jira Cloud app meets certain [requirements](jira_cloud_app.md#jira-user-requirements). +To resolve this issue, ensure the Jira user that installs and configures the app meets certain +[requirements](jira_cloud_app.md#jira-user-requirements). diff --git a/doc/administration/settings/rate_limit_on_members_api.md b/doc/administration/settings/rate_limit_on_members_api.md new file mode 100644 index 00000000000..3e8868adc91 --- /dev/null +++ b/doc/administration/settings/rate_limit_on_members_api.md @@ -0,0 +1,33 @@ +--- +stage: Data Stores +group: Tenant Scale +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Rate limit on Members API **(FREE SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/140633) in GitLab 16.9. + +You can configure the rate limit per group (or project) per user to the +[delete members API](../../api/members.md#remove-a-member-from-a-group-or-project). + +To change the rate limit: + +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Settings > Network**. +1. Expand **Members API rate limit**. +1. In the **Maximum requests per minute per group / project** text box, enter the new value. +1. Select **Save changes**. + +The rate limit: + +- Applies per group or project per user. +- Can be set to 0 to disable rate limiting. + +The default value of the rate limit is `60`. + +Requests over the rate limit are logged into the `auth.log` file. + +For example, if you set a limit of 60, requests sent to the +[delete members API](../../api/members.md#remove-a-member-from-a-group-or-project) exceeding a rate of 300 per minute +are blocked. Access to the endpoint is allowed after one minute. diff --git a/doc/administration/settings/scim_setup.md b/doc/administration/settings/scim_setup.md index 52061150fa7..2e1b40d58b8 100644 --- a/doc/administration/settings/scim_setup.md +++ b/doc/administration/settings/scim_setup.md @@ -33,6 +33,51 @@ To configure GitLab SCIM: - Token from the **Your SCIM token** field. - URL from the **SCIM API endpoint URL** field. +## Configure an identity provider + +You can configure the following as an identity provider: + +- [Okta](#configure-okta). + +NOTE: +Other identity providers can work with GitLab but they have not been tested and are not supported. You should contact the provider for support. GitLab support can assist by reviewing related log entries. + +### Configure Okta + +The SAML application created during [single sign-on](index.md) set up for Okta must be set up for SCIM. + +Prerequisites: + +- You must use the [Okta Lifecycle Management](https://www.okta.com/products/lifecycle-management/) product. This + product tier is required to use SCIM on Okta. +- [GitLab is configured](#configure-gitlab) for SCIM. +- The SAML application for [Okta](https://developer.okta.com/docs/guides/build-sso-integration/saml2/main/) set up as + described in the [Okta setup notes](../../integration/saml.md#set-up-okta). +- Your Okta SAML setup matches the [configuration steps](index.md), especially the NameID configuration. + +To configure Okta for SCIM: + +1. Sign in to Okta. +1. In the upper-right corner, select **Admin**. The button is not visible from the Admin Area. +1. In the **Application** tab, select **Browse App Catalog**. +1. Find and select the **GitLab** application. +1. On the GitLab application overview page, select **Add Integration**. +1. Under **Application Visibility**, select both checkboxes. The GitLab application does not support SAML + authentication so the icon should not be shown to users. +1. Select **Done** to finish adding the application. +1. In the **Provisioning** tab, select **Configure API integration**. +1. Select **Enable API integration**. + - For **Base URL**, paste the URL you copied from **SCIM API endpoint URL** on the GitLab SCIM configuration page. + - For **API Token**, paste the SCIM token you copied from **Your SCIM token** on the GitLab SCIM + configuration page. +1. To verify the configuration, select **Test API Credentials**. +1. Select **Save**. +1. After saving the API integration details, new settings tabs appear on the left. Select **To App**. +1. Select **Edit**. +1. Select the **Enable** checkbox for both **Create Users** and **Deactivate Users**. +1. Select **Save**. +1. Assign users in the **Assignments** tab. Assigned users are created and managed in your GitLab group. + ## Remove access Removing or deactivating a user on the identity provider blocks the user on diff --git a/doc/administration/settings/sign_in_restrictions.md b/doc/administration/settings/sign_in_restrictions.md index 60994bc6e43..6888994580c 100644 --- a/doc/administration/settings/sign_in_restrictions.md +++ b/doc/administration/settings/sign_in_restrictions.md @@ -85,7 +85,7 @@ To enable Admin Mode through the UI: To turn on Admin Mode for your current session and access potentially dangerous resources: -1. On the left sidebar, select **Search or go to**. +1. On the left sidebar, select your avatar. 1. Select **Enter Admin Mode**. 1. Try to access any part of the UI with `/admin` in the URL (which requires administrator access). @@ -103,7 +103,7 @@ authentication are supported by Admin Mode. Admin Mode status is stored in the c To turn off Admin Mode for your current session: -1. On the left sidebar, select **Search or go to**. +1. On the left sidebar, select your avatar. 1. Select **Leave Admin Mode**. ### Limitations of Admin Mode diff --git a/doc/administration/settings/slack_app.md b/doc/administration/settings/slack_app.md index 5e54294b720..48352a74060 100644 --- a/doc/administration/settings/slack_app.md +++ b/doc/administration/settings/slack_app.md @@ -106,9 +106,11 @@ When administering the GitLab for Slack app for self-managed instances, you migh For GitLab.com, see [GitLab for Slack app](../../user/project/integrations/gitlab_slack_app_troubleshooting.md). -### Slash commands return an error in Slack +### Slash commands return `dispatch_failed` in Slack -Slash commands might return `/gitlab failed with the error "dispatch_failed"` in Slack. To resolve this issue, ensure: +Slash commands might return `/gitlab failed with the error "dispatch_failed"` in Slack. + +To resolve this issue, ensure: - The GitLab for Slack app is properly [configured](#configure-the-settings) and the **Enable GitLab for Slack app** checkbox is selected. - Your GitLab instance [allows requests to and from Slack](#connectivity-requirements). diff --git a/doc/administration/settings/usage_statistics.md b/doc/administration/settings/usage_statistics.md index 20479a7dd8a..b661bcd6746 100644 --- a/doc/administration/settings/usage_statistics.md +++ b/doc/administration/settings/usage_statistics.md @@ -100,15 +100,6 @@ for all authenticated users, and on the Admin Area pages. The statuses are:  -GitLab Inc. collects your instance's version and hostname (through the HTTP -referer) as part of the version check. No other information is collected. - -This information is used, among other things, to identify to which versions -patches must be backported, making sure active GitLab instances remain -secure. - -If you disable version check, this information isn't collected. - ### Enable or disable version check 1. On the left sidebar, at the bottom, select **Admin Area**. diff --git a/doc/administration/sidekiq/sidekiq_memory_killer.md b/doc/administration/sidekiq/sidekiq_memory_killer.md index 6dcc97f2f7a..ff823bb6c56 100644 --- a/doc/administration/sidekiq/sidekiq_memory_killer.md +++ b/doc/administration/sidekiq/sidekiq_memory_killer.md @@ -28,7 +28,7 @@ run as a process group leader (for example, using `chpst -P`). If using a Linux ## Configuring the limits -Sidekiq memory limits are controlled using environment variables. +Sidekiq memory limits are controlled using [environment variables](https://docs.gitlab.com/omnibus/settings/environment-variables.html#setting-custom-environment-variables) - `SIDEKIQ_MEMORY_KILLER_MAX_RSS` (KB): defines the Sidekiq process soft limit for allowed RSS. If the Sidekiq process RSS (expressed in kilobytes) exceeds `SIDEKIQ_MEMORY_KILLER_MAX_RSS`, diff --git a/doc/administration/troubleshooting/postgresql.md b/doc/administration/troubleshooting/postgresql.md index 315a68fb126..75fd807643c 100644 --- a/doc/administration/troubleshooting/postgresql.md +++ b/doc/administration/troubleshooting/postgresql.md @@ -103,7 +103,7 @@ This section is for links to information elsewhere in the GitLab documentation. - Managing PostgreSQL versions on Linux package installations [from the development docs](https://docs.gitlab.com/omnibus/development/managing-postgresql-versions.html). - [PostgreSQL scaling](../postgresql/replication_and_failover.md) - - Including [troubleshooting](../postgresql/replication_and_failover.md#troubleshooting) + - Including [troubleshooting](../../administration/postgresql/replication_and_failover_troubleshooting.md) `gitlab-ctl patroni check-leader` and PgBouncer errors. - [Developer database documentation](../../development/feature_development.md#database-guides), diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index 31d0781ee79..95454ef629c 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -19,7 +19,7 @@ This is the default configuration. To change the location where the uploads are stored locally, use the steps in this section based on your installation method: NOTE: -For historical reasons, uploads for the whole instance (for example the [favicon](../administration/appearance.md#favicon)) are stored in a base directory, +For historical reasons, uploads for the whole instance (for example the [favicon](../administration/appearance.md#customize-the-favicon)) are stored in a base directory, which by default is `uploads/-/system`. Changing the base directory on an existing GitLab installation is strongly discouraged. |