diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2021-08-19 12:08:42 +0300 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2021-08-19 12:08:42 +0300 |
| commit | b76ae638462ab0f673e5915986070518dd3f9ad3 (patch) | |
| tree | bdab0533383b52873be0ec0eb4d3c66598ff8b91 /doc/administration | |
| parent | 434373eabe7b4be9593d18a585fb763f1e5f1a6f (diff) | |
Add latest changes from gitlab-org/gitlab@14-2-stable-eev14.2.0-rc42
Diffstat (limited to 'doc/administration')
90 files changed, 2019 insertions, 1402 deletions
diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index 7a871caf658..48bd812c7f2 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -123,6 +123,9 @@ From there, you can see the following actions: - Created, updated, or deleted DAST profiles, DAST scanner profiles, and DAST site profiles ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217872) in GitLab 14.1) - Changed a project's compliance framework ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/329362) in GitLab 14.1) +- User password required for approvals was updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336211) in GitLab 14.2) +- Permission to modify merge requests approval rules in merge requests was updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336211) in GitLab 14.2) +- New approvals requirement when new commits are added to an MR was updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336211) in GitLab 14.2) Project events can also be accessed via the [Project Audit Events API](../api/audit_events.md#project-audit-events). @@ -205,9 +208,6 @@ It may make the user interface for your project or audit events very busy, and t `audit_events` PostgreSQL table may increase considerably. It's disabled by default to prevent performance degradations on GitLab instances with very high Git write traffic. -In an upcoming release, Audit Events for Git push events are planned to be enabled -by default. Follow our [Partitioning strategy for Audit Events epic](https://gitlab.com/groups/gitlab-org/-/epics/3206) for updates. - If you still wish to enable **Repository push** events in your instance, follow the steps below. diff --git a/doc/administration/auth/atlassian.md b/doc/administration/auth/atlassian.md index b3892f8f5d9..868482148e5 100644 --- a/doc/administration/auth/atlassian.md +++ b/doc/administration/auth/atlassian.md @@ -12,22 +12,14 @@ To enable the Atlassian OmniAuth provider for passwordless authentication you mu ## Atlassian application registration 1. Go to <https://developer.atlassian.com/console/myapps/> and sign-in with the Atlassian - account that will administer the application. - + account to administer the application. 1. Click **Create a new app**. - 1. Choose an App Name, such as 'GitLab', and click **Create**. - 1. Note the `Client ID` and `Secret` for the [GitLab configuration](#gitlab-configuration) steps. - 1. In the left sidebar under **APIS AND FEATURES**, click **OAuth 2.0 (3LO)**. - 1. Enter the GitLab callback URL using the format `https://gitlab.example.com/users/auth/atlassian_oauth2/callback` and click **Save changes**. - 1. Click **+ Add** in the left sidebar under **APIS AND FEATURES**. - 1. Click **Add** for **Jira platform REST API** and then **Configure**. - 1. Click **Add** next to the following scopes: - **View Jira issue data** - **View user profiles** @@ -50,7 +42,6 @@ To enable the Atlassian OmniAuth provider for passwordless authentication you mu ``` 1. See [Initial OmniAuth Configuration](../../integration/omniauth.md#initial-omniauth-configuration) for initial settings to enable single sign-on and add `atlassian_oauth2` as an OAuth provider. - 1. Add the provider configuration for Atlassian: For Omnibus GitLab installations: @@ -76,9 +67,7 @@ To enable the Atlassian OmniAuth provider for passwordless authentication you mu ``` 1. Change `YOUR_CLIENT_ID` and `YOUR_CLIENT_SECRET` to the Client credentials you received in [application registration](#atlassian-application-registration) steps. - 1. Save the configuration file. - 1. [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) or [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect if you installed GitLab via Omnibus or from source respectively. On the sign-in page there should now be an Atlassian icon below the regular sign in form. Click the icon to begin the authentication process. diff --git a/doc/administration/auth/cognito.md b/doc/administration/auth/cognito.md index de5fa991abe..41e77c10e27 100644 --- a/doc/administration/auth/cognito.md +++ b/doc/administration/auth/cognito.md @@ -12,8 +12,7 @@ The following documentation enables Cognito as an OAuth2 provider. ## Configure AWS Cognito -To enable the [AWS Cognito](https://aws.amazon.com/cognito/) OAuth2 OmniAuth provider, register your application with Cognito, -where it will generate a Client ID and Client Secret for your application. +To enable the [AWS Cognito](https://aws.amazon.com/cognito/) OAuth2 OmniAuth provider, register your application with Cognito. This process generates a Client ID and Client Secret for your application. Any settings you configure in the following procedure can be modified later. The following steps enable AWS Cognito as an authentication provider: @@ -24,7 +23,7 @@ The following steps enable AWS Cognito as an authentication provider: 1. Under **How do you want your end users to sign in?**, select **Email address or phone number** and **Allow email addresses**. 1. Under **Which standard attributes do you want to require?**, select **email**. 1. Go to the next steps of configuration and set the rest of the settings to suit your needs - in the basic setup they are not related to GitLab configuration. -1. In the **App clients** settings, click **Add an app client**, add **App client name** and select the **Enable username password based authentication** check box. +1. In the **App clients** settings, click **Add an app client**, add **App client name** and select the **Enable username password based authentication** checkbox. 1. Click **Create app client**. 1. In the next step, you can set up AWS Lambda functions for sending emails. You can then finish creating the pool. 1. After creating the user pool, go to **App client settings** and provide the required information: @@ -86,7 +85,7 @@ Include the code block in the `/etc/gitlab/gitlab.rb` file: 1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. Your sign-in page should now display a Cognito button below the regular sign-in form. -To begin the authentication process, click the icon, and AWS Cognito will ask the user to sign in and authorize the GitLab application. -If successful, the user will be redirected and signed in to your GitLab instance. +To begin the authentication process, click the icon, and AWS Cognito asks the user to sign in and authorize the GitLab application. +If successful, the user is redirected and signed in to your GitLab instance. For more information, see the [Initial OmniAuth Configuration](../../integration/omniauth.md#initial-omniauth-configuration). diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index a9d59ca0983..63e3a0a3686 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -197,8 +197,8 @@ Some examples of the `user_filter` field syntax: LDAP attributes that GitLab uses to create an account for the LDAP user. The specified attribute can either be the attribute name as a string (for example, `'mail'`), or an -array of attribute names to try in order (for example, `['mail', 'email']`). Note that -the user's LDAP sign-in is the attribute specified as `uid` above. +array of attribute names to try in order (for example, `['mail', 'email']`). +The user's LDAP sign-in is the attribute specified as `uid` above. | Setting | Description | Required | Examples | |--------------|-------------|----------|----------| @@ -654,7 +654,7 @@ NOTE: Administrators are not synced unless `group_base` is also specified alongside `admin_group`. Also, only specify the CN of the `admin_group`, as opposed to the full DN. -Additionally, note that if an LDAP user has an `admin` role, but is not a member of the `admin_group` +Additionally, if an LDAP user has an `admin` role, but is not a member of the `admin_group` group, GitLab revokes their `admin` role when syncing. **Omnibus configuration** diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md index 5e6c3443e44..15e8496e915 100644 --- a/doc/administration/auth/ldap/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap/ldap-troubleshooting.md @@ -426,7 +426,7 @@ Started syncing 'ldapmain' provider for 'my_group' group ``` The following entry shows an array of all user DNs GitLab sees in the LDAP server. -Note that these are the users for a single LDAP group, not a GitLab group. If +These are the users for a single LDAP group, not a GitLab group. If you have multiple LDAP groups linked to this GitLab group, you see multiple log entries like this - one for each LDAP group. If you don't see an LDAP user DN in this log entry, LDAP is not returning the user when we do the lookup. @@ -666,7 +666,7 @@ ldapsearch -D "cn=admin,dc=ldap-testing,dc=example,dc=com" \ -b "dc=ldap-testing,dc=example,dc=com" ``` -Note that the `bind_dn`, `password`, `port`, `host`, and `base` are all +The `bind_dn`, `password`, `port`, `host`, and `base` are all identical to what's configured in the `gitlab.rb`. For more information, see the [official `ldapsearch` documentation](https://linux.die.net/man/1/ldapsearch). diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md index 951c7df26ef..6a037e75f54 100644 --- a/doc/administration/auth/oidc.md +++ b/doc/administration/auth/oidc.md @@ -280,7 +280,7 @@ but `LocalAccounts` works for authenticating against local, Active Directory acc The policy name is lowercased in the URL. For example, `B2C_1A_signup_signin` policy appears as `b2c_1a_signup_sigin`. - Note that the trailing forward slash is required. +The trailing forward slash is required. 1. Verify the operation of the OIDC discovery URL and issuer URL, append `.well-known/openid-configuration` to the issuer URL: @@ -336,7 +336,7 @@ but `LocalAccounts` works for authenticating against local, Active Directory acc - `"accessTokenAcceptedVersion": null` - `"signInAudience": "AzureADMyOrg"` - Note that this configuration corresponds with the `Supported account types` setting used when +This configuration corresponds with the `Supported account types` setting used when creating the `IdentityExperienceFramework` app. #### Keycloak diff --git a/doc/administration/compliance.md b/doc/administration/compliance.md index 742d23105a9..e356ef0366b 100644 --- a/doc/administration/compliance.md +++ b/doc/administration/compliance.md @@ -30,4 +30,4 @@ relevant compliance standards. |**Separation of Duties using [Protected branches](../user/project/protected_branches.md#require-code-owner-approval-on-a-protected-branch) and [custom CI Configuration Paths](../ci/pipelines/settings.md#specify-a-custom-cicd-configuration-file)**<br> GitLab Premium users can leverage the GitLab cross-project YAML configurations to define deployers of code and developers of code. View the [Separation of Duties Deploy Project](https://gitlab.com/guided-explorations/separation-of-duties-deploy/blob/master/README.md) and [Separation of Duties Project](https://gitlab.com/guided-explorations/separation-of-duties/blob/master/README.md) to see how to use this set up to define these roles. | Premium+ | **{check-circle}** Yes | Project | |**[Compliance frameworks](../user/project/settings/index.md#compliance-frameworks)**<br>Create a custom compliance framework at the group level to describe the type of compliance requirements any child project needs to follow. | Premium+ | **{check-circle}** Yes | Group | |**[Compliance pipelines](../user/project/settings/index.md#compliance-pipeline-configuration)**<br>Define a pipeline configuration to run for any projects with a given compliance framework. | Ultimate | **{check-circle}** Yes | Group | -|**[Compliance dashboard](../user/compliance/compliance_dashboard/index.md)**<br>Quickly get visibility into the compliance posture of your organization. | Ultimate | **{check-circle}** Yes | Group | +|**[Compliance report](../user/compliance/compliance_report/index.md)**<br>Quickly get visibility into the compliance posture of your organization. | Ultimate | **{check-circle}** Yes | Group | diff --git a/doc/administration/encrypted_configuration.md b/doc/administration/encrypted_configuration.md index ff48005e427..8afe30d20ab 100644 --- a/doc/administration/encrypted_configuration.md +++ b/doc/administration/encrypted_configuration.md @@ -12,6 +12,7 @@ type: reference GitLab can read settings for certain features from encrypted settings files. The supported features are: - [LDAP `user_bn` and `password`](auth/ldap/index.md#using-encrypted-credentials) +- [SMTP `user_name` and `password`](raketasks/smtp.md#secrets) In order to enable the encrypted configuration settings, a new base key needs to be generated for `encrypted_settings_key_base`. The secret can be generated in the following ways: diff --git a/doc/administration/feature_flags.md b/doc/administration/feature_flags.md index 44abf4a875d..575fa9eb229 100644 --- a/doc/administration/feature_flags.md +++ b/doc/administration/feature_flags.md @@ -31,16 +31,27 @@ These features can be enabled and disabled to allow or disallow users to use them. It can be done by GitLab administrators with access to GitLab Rails console. +When you disable a feature flag, the feature is hidden from users and all of the functionality is turned off. +For example, data is not recorded and services do not run. + If you used a certain feature and identified a bug, a misbehavior, or an error, it's very important that you [**provide feedback**](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issue[title]=Docs%20-%20feature%20flag%20feedback%3A%20Feature%20Name&issue[description]=Describe%20the%20problem%20you%27ve%20encountered.%0A%0A%3C!--%20Don%27t%20edit%20below%20this%20line%20--%3E%0A%0A%2Flabel%20~%22docs%5C-comments%22%20) to GitLab as soon as possible so we can improve or fix it while behind a flag. When you upgrade GitLab to an earlier version, the feature flag status may change. -WARNING: -Features deployed behind feature flags may not be ready for -production use. However, disabling features behind flags that were deployed -enabled by default may also present a risk. If they're enabled, we recommend -you leave them as-is. +## Risks when enabling features still in development + +Features that are disabled by default may change or be removed without notice in a future version of GitLab. + +Data corruption, stability degradation, or performance degradation might occur if +you enable a feature that's disabled by default. Problems caused by using a default +disabled feature aren't covered by GitLab support, unless you were directed by GitLab +to enable the feature. + +## Risks when disabling released features + +In most cases, the feature flag code is removed in a future version of GitLab. +If and when that occurs, from that point onward you can't keep the feature in a disabled state. ## How to enable and disable features behind flags diff --git a/doc/administration/geo/disaster_recovery/background_verification.md b/doc/administration/geo/disaster_recovery/background_verification.md index f03cd64c14e..65e0ffd4366 100644 --- a/doc/administration/geo/disaster_recovery/background_verification.md +++ b/doc/administration/geo/disaster_recovery/background_verification.md @@ -161,7 +161,7 @@ If the **primary** and **secondary** nodes have a checksum verification mismatch  1. Go to the project's repository directory on both **primary** and **secondary** nodes - (the path is usually `/var/opt/gitlab/git-data/repositories`). Note that if `git_data_dirs` + (the path is usually `/var/opt/gitlab/git-data/repositories`). If `git_data_dirs` is customized, check the directory layout on your server to be sure: ```shell @@ -198,7 +198,7 @@ nodes, and comparing the output between them. In GitLab EE 12.1, Geo calculates checksums for attachments, LFS objects, and archived traces on secondary nodes after the transfer, compares it with the -stored checksums, and rejects transfers if mismatched. Please note that Geo +stored checksums, and rejects transfers if mismatched. Geo currently does not support an automatic way to verify these data if they have been synced before GitLab EE 12.1. diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index 5c15523ac78..8b55bebb42b 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -118,7 +118,7 @@ On the **secondary** node: objects aren't yet replicated (shown in gray), consider giving the node more time to complete -  +  If any objects are failing to replicate, this should be investigated before scheduling the maintenance window. Following a planned failover, anything that diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md index 16ae5bde062..27990748071 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md @@ -72,7 +72,7 @@ On the **secondary** node: objects aren't yet replicated (shown in gray), consider giving the node more time to complete. -  +  If any objects are failing to replicate, this should be investigated before scheduling the maintenance window. After a planned failover, anything that @@ -94,7 +94,7 @@ follow these steps to avoid unnecessary data loss: 1. Until a [read-only mode](https://gitlab.com/gitlab-org/gitlab/-/issues/14609) is implemented, updates must be prevented from happening manually to the - **primary**. Note that your **secondary** node still needs read-only + **primary**. Your **secondary** node still needs read-only access to the **primary** node during the maintenance window: 1. At the scheduled time, using your cloud provider or your node's firewall, block diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md index 36c9d46d650..9d5e65cd194 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md @@ -57,7 +57,7 @@ and there should be no failures (shown in red). If a large proportion of objects aren't yet replicated (shown in gray), consider giving the node more time to complete. - + If any objects are failing to replicate, this should be investigated before scheduling the maintenance window. After a planned failover, anything that @@ -79,7 +79,7 @@ follow these steps to avoid unnecessary data loss: 1. Until a [read-only mode](https://gitlab.com/gitlab-org/gitlab/-/issues/14609) is implemented, updates must be prevented from happening manually to the - **primary**. Note that your **secondary** node still needs read-only + **primary**. Your **secondary** node still needs read-only access to the **primary** node during the maintenance window: 1. At the scheduled time, using your cloud provider or your node's firewall, block diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index 295a448c432..7175d41abd8 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -101,12 +101,12 @@ From the perspective of a user performing Git operations: - The **primary** site behaves as a full read-write GitLab instance. - **Secondary** sites are read-only but proxy Git push operations to the **primary** site. This makes **secondary** sites appear to support push operations themselves. -To simplify the diagram, some necessary components are omitted. Note that: +To simplify the diagram, some necessary components are omitted. - Git over SSH requires [`gitlab-shell`](https://gitlab.com/gitlab-org/gitlab-shell) and OpenSSH. - Git over HTTPS required [`gitlab-workhorse`](https://gitlab.com/gitlab-org/gitlab-workhorse). -Note that a **secondary** site needs two different PostgreSQL databases: +A **secondary** site needs two different PostgreSQL databases: - A read-only database instance that streams data from the main GitLab database. - [Another database instance](#geo-tracking-database) used internally by the **secondary** site to record what data has been replicated. @@ -193,11 +193,8 @@ This list of limitations only reflects the latest version of GitLab. If you are - The **primary** site has to be online for OAuth login to happen. Existing sessions and Git are not affected. Support for the **secondary** site to use an OAuth provider independent from the primary is [being planned](https://gitlab.com/gitlab-org/gitlab/-/issues/208465). - The installation takes multiple manual steps that together can take about an hour depending on circumstances. We are working on improving this experience. See [Omnibus GitLab issue #2978](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/2978) for details. - Real-time updates of issues/merge requests (for example, via long polling) doesn't work on the **secondary** site. -- [Selective synchronization](replication/configuration.md#selective-synchronization) applies only to files and repositories. Other datasets are replicated to the **secondary** site in full, making it inappropriate for use as an access control mechanism. -- Object pools for forked project deduplication work only on the **primary** site, and are duplicated on the **secondary** site. - GitLab Runners cannot register with a **secondary** site. Support for this is [planned for the future](https://gitlab.com/gitlab-org/gitlab/-/issues/3294). -- Configuring Geo **secondary** sites to [use high-availability configurations of PostgreSQL](https://gitlab.com/groups/gitlab-org/-/epics/2536) is currently in **alpha** support. -- [Selective synchronization](replication/configuration.md#selective-synchronization) only limits what repositories are replicated. The entire PostgreSQL data is still replicated. Selective synchronization is not built to accommodate compliance / export control use cases. +- [Selective synchronization](replication/configuration.md#selective-synchronization) only limits what repositories and files are replicated. The entire PostgreSQL data is still replicated. Selective synchronization is not built to accommodate compliance / export control use cases. ### Limitations on replication/verification diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index e8ffa1ae91a..5b22741f578 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -7,35 +7,35 @@ type: howto # Geo configuration **(PREMIUM SELF)** -## Configuring a new **secondary** node +## Configuring a new **secondary** site NOTE: -This is the final step in setting up a **secondary** Geo node. Stages of the +This is the final step in setting up a **secondary** Geo site. Stages of the setup process must be completed in the documented order. Before attempting the steps in this stage, [complete all prior stages](../setup/index.md#using-omnibus-gitlab). -The basic steps of configuring a **secondary** node are to: +The basic steps of configuring a **secondary** site are to: -- Replicate required configurations between the **primary** node and the **secondary** nodes. -- Configure a tracking database on each **secondary** node. -- Start GitLab on each **secondary** node. +- Replicate required configurations between the **primary** site and the **secondary** sites. +- Configure a tracking database on each **secondary** site. +- Start GitLab on each **secondary** site. You are encouraged to first read through all the steps before executing them in your testing/production environment. NOTE: -**Do not** set up any custom authentication for the **secondary** nodes. This is handled by the **primary** node. +**Do not** set up any custom authentication for the **secondary** sites. This is handled by the **primary** site. Any change that requires access to the **Admin Area** needs to be done in the -**primary** node because the **secondary** node is a read-only replica. +**primary** site because the **secondary** site is a read-only replica. ### Step 1. Manually replicate secret GitLab values GitLab stores a number of secret values in the `/etc/gitlab/gitlab-secrets.json` -file which *must* be the same on all nodes. Until there is -a means of automatically replicating these between nodes (see [issue #3789](https://gitlab.com/gitlab-org/gitlab/-/issues/3789)), -they must be manually replicated to the **secondary** node. +file which *must* be the same on all of a site's nodes. Until there is +a means of automatically replicating these between sites (see [issue #3789](https://gitlab.com/gitlab-org/gitlab/-/issues/3789)), +they must be manually replicated to **all nodes of the secondary site**. -1. SSH into the **primary** node, and execute the command below: +1. SSH into a **Rails node on your primary** site, and execute the command below: ```shell sudo cat /etc/gitlab/gitlab-secrets.json @@ -43,7 +43,7 @@ they must be manually replicated to the **secondary** node. This displays the secrets that need to be replicated, in JSON format. -1. SSH into the **secondary** node and login as the `root` user: +1. SSH **into each node on your secondary Geo site** and login as the `root` user: ```shell sudo -i @@ -55,7 +55,7 @@ they must be manually replicated to the **secondary** node. mv /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.`date +%F` ``` -1. Copy `/etc/gitlab/gitlab-secrets.json` from the **primary** node to the **secondary** node, or +1. Copy `/etc/gitlab/gitlab-secrets.json` from the **Rails node on your primary** site to **each node on your secondary** site, or copy-and-paste the file contents between nodes: ```shell @@ -72,28 +72,28 @@ they must be manually replicated to the **secondary** node. chmod 0600 /etc/gitlab/gitlab-secrets.json ``` -1. Reconfigure the **secondary** node for the change to take effect: +1. Reconfigure **each Rails, Sidekiq and Gitaly nodes on your secondary** site for the change to take effect: ```shell gitlab-ctl reconfigure gitlab-ctl restart ``` -### Step 2. Manually replicate the **primary** node's SSH host keys +### Step 2. Manually replicate the **primary** site's SSH host keys GitLab integrates with the system-installed SSH daemon, designating a user (typically named `git`) through which all access requests are handled. In a [Disaster Recovery](../disaster_recovery/index.md) situation, GitLab system -administrators promote a **secondary** node to the **primary** node. DNS records for the -**primary** domain should also be updated to point to the new **primary** node -(previously a **secondary** node). Doing so avoids the need to update Git remotes and API URLs. +administrators promote a **secondary** site to the **primary** site. DNS records for the +**primary** domain should also be updated to point to the new **primary** site +(previously a **secondary** site). Doing so avoids the need to update Git remotes and API URLs. -This causes all SSH requests to the newly promoted **primary** node to +This causes all SSH requests to the newly promoted **primary** site to fail due to SSH host key mismatch. To prevent this, the primary SSH host -keys must be manually replicated to the **secondary** node. +keys must be manually replicated to the **secondary** site. -1. SSH into the **secondary** node and login as the `root` user: +1. SSH into **each node on your secondary** site and login as the `root` user: ```shell sudo -i @@ -105,34 +105,34 @@ keys must be manually replicated to the **secondary** node. find /etc/ssh -iname ssh_host_* -exec cp {} {}.backup.`date +%F` \; ``` -1. Copy OpenSSH host keys from the **primary** node: +1. Copy OpenSSH host keys from the **primary** site: - If you can access your **primary** node using the **root** user: + If you can access one of the **nodes on your primary** site serving SSH traffic (usually, the main GitLab Rails application nodes) using the **root** user: ```shell - # Run this from the secondary node, change `<primary_node_fqdn>` for the IP or FQDN of the server + # Run this from the secondary site, change `<primary_site_fqdn>` for the IP or FQDN of the server scp root@<primary_node_fqdn>:/etc/ssh/ssh_host_*_key* /etc/ssh ``` If you only have access through a user with `sudo` privileges: ```shell - # Run this from your primary node: + # Run this from the node on your primary site: sudo tar --transform 's/.*\///g' -zcvf ~/geo-host-key.tar.gz /etc/ssh/ssh_host_*_key* - # Run this from your secondary node: - scp <user_with_sudo>@<primary_node_fqdn>:geo-host-key.tar.gz . + # Run this on each node on your secondary site: + scp <user_with_sudo>@<primary_site_fqdn>:geo-host-key.tar.gz . tar zxvf ~/geo-host-key.tar.gz -C /etc/ssh ``` -1. On your **secondary** node, ensure the file permissions are correct: +1. On **each node on your secondary** site, ensure the file permissions are correct: ```shell chown root:root /etc/ssh/ssh_host_*_key* chmod 0600 /etc/ssh/ssh_host_*_key* ``` -1. To verify key fingerprint matches, execute the following command on both nodes: +1. To verify key fingerprint matches, execute the following command on both primary and secondary nodes on each site: ```shell for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done @@ -160,7 +160,7 @@ keys must be manually replicated to the **secondary** node. NOTE: The output for private keys and public keys command should generate the same fingerprint. -1. Restart `sshd` on your **secondary** node: +1. Restart `sshd` on **each node on your secondary** site: ```shell # Debian or Ubuntu installations @@ -175,31 +175,34 @@ keys must be manually replicated to the **secondary** node. SSH into your GitLab **secondary** server in a new terminal. If you are unable to connect, verify the permissions are correct according to the previous steps. -### Step 3. Add the **secondary** node +### Step 3. Add the **secondary** site -1. SSH into your GitLab **secondary** server and login as root: +1. SSH into **each Rails and Sidekiq node on your secondary** site and login as root: ```shell sudo -i ``` -1. Edit `/etc/gitlab/gitlab.rb` and add a **unique** name for your node. You need this in the next steps: +1. Edit `/etc/gitlab/gitlab.rb` and add a **unique** name for your site. You need this in the next steps: ```ruby - # The unique identifier for the Geo node. - gitlab_rails['geo_node_name'] = '<node_name_here>' + ## + ## The unique identifier for the Geo site. See + ## https://docs.gitlab.com/ee/user/admin_area/geo_nodes.html#common-settings + ## + gitlab_rails['geo_node_name'] = '<site_name_here>' ``` -1. Reconfigure the **secondary** node for the change to take effect: +1. Reconfigure **each Rails and Sidekiq node on your secondary** site for the change to take effect: ```shell gitlab-ctl reconfigure ``` -1. On the top bar of the primary node, select **Menu >** **{admin}** **Admin**. -1. On the left sidebar, select **Geo > Nodes**. -1. Select **Add site**. -  +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Geo > Sites**. +1. Select **New site**. +  1. Fill in **Name** with the `gitlab_rails['geo_node_name']` in `/etc/gitlab/gitlab.rb`. These values must always match *exactly*, character for character. @@ -207,10 +210,10 @@ keys must be manually replicated to the **secondary** node. values must always match, but it doesn't matter if one ends with a `/` and the other doesn't. 1. Optionally, choose which groups or storage shards should be replicated by the - **secondary** node. Leave blank to replicate all. Read more in + **secondary** site. Leave blank to replicate all. Read more in [selective synchronization](#selective-synchronization). -1. Select **Add node** to add the **secondary** node. -1. SSH into your GitLab **secondary** server and restart the services: +1. Select **Add site** to add the **secondary** site. +1. SSH into **each Rails, and Sidekiq node on your secondary** site and restart the services: ```shell gitlab-ctl restart @@ -222,58 +225,58 @@ keys must be manually replicated to the **secondary** node. gitlab-rake gitlab:geo:check ``` -1. SSH into your **primary** server and login as root to verify the - **secondary** node is reachable or there are any common issue with your Geo setup: +1. SSH into a **Rails or Sidekiq server on your primary** site and login as root to verify the + **secondary** site is reachable or there are any common issue with your Geo setup: ```shell gitlab-rake gitlab:geo:check ``` -Once added to the Geo administration page and restarted, the **secondary** node automatically starts -replicating missing data from the **primary** node in a process known as **backfill**. -Meanwhile, the **primary** node starts to notify each **secondary** node of any changes, so -that the **secondary** node can act on those notifications immediately. +Once added to the Geo administration page and restarted, the **secondary** site automatically starts +replicating missing data from the **primary** site in a process known as **backfill**. +Meanwhile, the **primary** site starts to notify each **secondary** site of any changes, so +that the **secondary** site can act on those notifications immediately. -Be sure the _secondary_ node is running and accessible. You can sign in to the -_secondary_ node with the same credentials as were used with the _primary_ node. +Be sure the _secondary_ site is running and accessible. You can sign in to the +_secondary_ site with the same credentials as were used with the _primary_ site. -### Step 4. (Optional) Configuring the **secondary** node to trust the **primary** node +### Step 4. (Optional) Configuring the **secondary** site to trust the **primary** site -You can safely skip this step if your **primary** node uses a CA-issued HTTPS certificate. +You can safely skip this step if your **primary** site uses a CA-issued HTTPS certificate. -If your **primary** node is using a self-signed certificate for *HTTPS* support, you -need to add that certificate to the **secondary** node's trust store. Retrieve the -certificate from the **primary** node and follow +If your **primary** site is using a self-signed certificate for *HTTPS* support, you +need to add that certificate to the **secondary** site's trust store. Retrieve the +certificate from the **primary** site and follow [these instructions](https://docs.gitlab.com/omnibus/settings/ssl.html) -on the **secondary** node. +on the **secondary** site. ### Step 5. Enable Git access over HTTP/HTTPS Geo synchronizes repositories over HTTP/HTTPS, and therefore requires this clone -method to be enabled. This is enabled by default, but if converting an existing node to Geo it should be checked: +method to be enabled. This is enabled by default, but if converting an existing site to Geo it should be checked: -On the **primary** node: +On the **primary** site: 1. On the top bar, select **Menu >** **{admin}** **Admin**. 1. On the left sidebar, select **Settings > General**. 1. Expand **Visibility and access controls**. 1. Ensure "Enabled Git access protocols" is set to either "Both SSH and HTTP(S)" or "Only HTTP(S)". -### Step 6. Verify proper functioning of the **secondary** node +### Step 6. Verify proper functioning of the **secondary** site -You can sign in to the **secondary** node with the same credentials you used with -the **primary** node. After you sign in: +You can sign in to the **secondary** site with the same credentials you used with +the **primary** site. After you sign in: 1. On the top bar, select **Menu >** **{admin}** **Admin**. -1. On the left sidebar, select **Geo > Nodes**. -1. Verify that it's correctly identified as a **secondary** Geo node, and that +1. On the left sidebar, select **Geo > Sites**. +1. Verify that it's correctly identified as a **secondary** Geo site, and that Geo is enabled. The initial replication, or 'backfill', is probably still in progress. You -can monitor the synchronization process on each Geo node from the **primary** -node's **Geo Nodes** dashboard in your browser. +can monitor the synchronization process on each Geo site from the **primary** +site's **Geo Sites** dashboard in your browser. - + If your installation isn't working properly, check the [troubleshooting document](troubleshooting.md). @@ -286,10 +289,10 @@ The two most obvious issues that can become apparent in the dashboard are: - You are using a custom certificate or custom CA (see the [troubleshooting document](troubleshooting.md)). - The instance is firewalled (check your firewall rules). -Please note that disabling a **secondary** node stops the synchronization process. +Disabling a **secondary** site stops the synchronization process. -Please note that if `git_data_dirs` is customized on the **primary** node for multiple -repository shards you must duplicate the same configuration on each **secondary** node. +If `git_data_dirs` is customized on the **primary** site for multiple +repository shards you must duplicate the same configuration on each **secondary** site. Point your users to the [Using a Geo Site guide](usage.md). @@ -304,7 +307,7 @@ Currently, this is what is synced: ## Selective synchronization Geo supports selective synchronization, which allows administrators to choose -which projects should be synchronized by **secondary** nodes. +which projects should be synchronized by **secondary** sites. A subset of projects can be chosen, either by group or by storage shard. The former is ideal for replicating data belonging to a subset of users, while the latter is more suited to progressively rolling out Geo to a large GitLab @@ -312,22 +315,22 @@ instance. It is important to note that selective synchronization: -1. Does not restrict permissions from **secondary** nodes. -1. Does not hide project metadata from **secondary** nodes. +1. Does not restrict permissions from **secondary** sites. +1. Does not hide project metadata from **secondary** sites. - Since Geo currently relies on PostgreSQL replication, all project metadata - gets replicated to **secondary** nodes, but repositories that have not been + gets replicated to **secondary** sites, but repositories that have not been selected are empty. 1. Does not reduce the number of events generated for the Geo event log. - - The **primary** node generates events as long as any **secondary** nodes are present. - Selective synchronization restrictions are implemented on the **secondary** nodes, - not the **primary** node. + - The **primary** site generates events as long as any **secondary** sites are present. + Selective synchronization restrictions are implemented on the **secondary** sites, + not the **primary** site. ### Git operations on unreplicated repositories > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2562) in GitLab 12.10 for HTTP(S) and in GitLab 13.0 for SSH. Git clone, pull, and push operations over HTTP(S) and SSH are supported for repositories that -exist on the **primary** node but not on **secondary** nodes. This situation can occur +exist on the **primary** site but not on **secondary** sites. This situation can occur when: - Selective synchronization does not include the project attached to the repository. @@ -335,7 +338,7 @@ when: ## Upgrading Geo -See the [updating the Geo nodes document](updating_the_geo_nodes.md). +See the [updating the Geo sites document](updating_the_geo_nodes.md). ## Troubleshooting diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index a56d9dc813c..a696e5410e5 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -32,7 +32,6 @@ verification methods: | Git | Project repository | Geo with Gitaly | Gitaly Checksum | | Git | Project wiki repository | Geo with Gitaly | Gitaly Checksum | | Git | Project designs repository | Geo with Gitaly | Gitaly Checksum | -| Git | Object pools for forked project deduplication | Geo with Gitaly | _Not implemented_ | | Git | Project Snippets | Geo with Gitaly | Gitaly Checksum | | Git | Personal Snippets | Geo with Gitaly | Gitaly Checksum | | Git | Group wiki repository | Geo with Gitaly | _Not implemented_ | @@ -47,8 +46,8 @@ verification methods: | Blobs | Container registry _(file system)_ | Geo with API/Docker API | _Not implemented_ | | Blobs | Container registry _(object storage)_ | Geo with API/Managed/Docker API (*2*) | _Not implemented_ | | Blobs | Package registry _(file system)_ | Geo with API | SHA256 checksum | -| Blobs | Package registry _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | -| Blobs | Versioned Terraform State _(file system)_ | Geo with API | SHA256 checksum | +| Blobs | Package registry _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | Versioned Terraform State _(file system)_ | Geo with API | SHA256 checksum | | Blobs | Versioned Terraform State _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | External Merge Request Diffs _(file system)_ | Geo with API | _Not implemented_ | | Blobs | External Merge Request Diffs _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | @@ -69,6 +68,8 @@ or using LVM. It requires no special file system and can work with NFS or a mounted Storage Appliance (there may be performance limitations when using a remote file system). +Geo will trigger garbage collection in Gitaly to [deduplicate forked repositories](../../../development/git_object_deduplication.md#git-object-deduplication-and-gitlab-geo) on Geo secondary sites. + Communication is done via Gitaly's own gRPC API. There are three possible ways of synchronization: - Using regular Git clone/fetch from one Geo site to another (with special authentication). @@ -186,7 +187,6 @@ successfully, you must replicate their data using some other means. |[CI job artifacts (other than Job Logs)](../../../ci/pipelines/job_artifacts.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. | |[CI Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/pipeline_artifact.rb) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | Via Object Storage provider if supported. Native Geo support (Beta). | Persists additional artifacts after a pipeline completes | |[Job logs](../../job_logs.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. | -|[Object pools for forked project deduplication](../../../development/git_object_deduplication.md) | **Yes** | No | No | | |[Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | No | Disabled by default. See [instructions](docker_registry.md) to enable. | |[Content in object storage (beta)](object_storage.md) | **Yes** (12.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) | No | | |[Infrastructure Registry for Terraform Module](../../../user/packages/terraform_module_registry/index.md) | **Yes** (14.0) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (14.0) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | @@ -200,11 +200,11 @@ successfully, you must replicate their data using some other means. |[Package Registry for generic packages](../../../user/packages/generic_packages/index.md) | **Yes** (13.5) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | |[Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.12) | Via Object Storage provider if supported. Native Geo support (Beta). | Replication is behind the feature flag `geo_terraform_state_version_replication`, enabled by default. Verification was behind the feature flag `geo_terraform_state_version_verification`, which was removed in 14.0| |[External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Replication is behind the feature flag `geo_merge_request_diff_replication`, enabled by default. Verification is under development, behind the feature flag `geo_merge_request_diff_verification`, introduced in 14.0.| -|[Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2810) | No | | +|[Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [**Yes** (14.2)](https://gitlab.com/groups/gitlab-org/-/epics/2810) | No | Verification was implemented behind the feature flag `geo_snippet_repository_verification` in 13.11, and the feature flag was removed in 14.2. | |[Server-side Git hooks](../../server_hooks.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1867) | No | No | | |[Elasticsearch integration](../../../integration/elasticsearch.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | No | | |[GitLab Pages](../../pages/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | Via Object Storage provider if supported. **No** native Geo support (Beta). | | -|[Dependency proxy images](../../../user/packages/dependency_proxy/index.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/259694) | No | No | Blocked on [Geo: Secondary Mimicry](https://gitlab.com/groups/gitlab-org/-/epics/1528). Note that replication of this cache is not needed for Disaster Recovery purposes because it can be recreated from external sources. | +|[Dependency proxy images](../../../user/packages/dependency_proxy/index.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/259694) | No | No | Blocked on [Geo: Secondary Mimicry](https://gitlab.com/groups/gitlab-org/-/epics/1528). Replication of this cache is not needed for Disaster Recovery purposes because it can be recreated from external sources. | |[Vulnerability Export](../../../user/application_security/vulnerability_report/#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | | Not planned because they are ephemeral and sensitive. They can be regenerated on demand. | #### Limitation of verification for files in Object Storage diff --git a/doc/administration/geo/replication/img/adding_a_secondary_node_v13_3.png b/doc/administration/geo/replication/img/adding_a_secondary_v13_3.png Binary files differindex e43ace99666..e43ace99666 100644 --- a/doc/administration/geo/replication/img/adding_a_secondary_node_v13_3.png +++ b/doc/administration/geo/replication/img/adding_a_secondary_v13_3.png diff --git a/doc/administration/geo/replication/img/geo_node_dashboard_v14_0.png b/doc/administration/geo/replication/img/geo_dashboard_v14_0.png Binary files differindex 6d183fc6bd2..6d183fc6bd2 100644 --- a/doc/administration/geo/replication/img/geo_node_dashboard_v14_0.png +++ b/doc/administration/geo/replication/img/geo_dashboard_v14_0.png diff --git a/doc/administration/geo/replication/img/geo_node_health_v14_0.png b/doc/administration/geo/replication/img/geo_site_health_v14_0.png Binary files differindex 4c640522569..4c640522569 100644 --- a/doc/administration/geo/replication/img/geo_node_health_v14_0.png +++ b/doc/administration/geo/replication/img/geo_site_health_v14_0.png diff --git a/doc/administration/geo/replication/location_aware_git_url.md b/doc/administration/geo/replication/location_aware_git_url.md index a80c293149e..ce1af27611e 100644 --- a/doc/administration/geo/replication/location_aware_git_url.md +++ b/doc/administration/geo/replication/location_aware_git_url.md @@ -20,7 +20,7 @@ as well. NOTE: You can also use a load balancer to distribute web UI or API traffic to -[multiple Geo **secondary** sites](../../../user/admin_area/geo_nodes.md#multiple-secondary-nodes-behind-a-load-balancer). +[multiple Geo **secondary** sites](../../../user/admin_area/geo_nodes.md#multiple-secondary-sites-behind-a-load-balancer). Importantly, the **primary** site cannot yet be included. See the feature request [Support putting the **primary** behind a Geo node load balancer](https://gitlab.com/gitlab-org/gitlab/-/issues/10888). @@ -107,7 +107,7 @@ You can customize the: - SSH remote URL to use the location-aware `git.example.com`. To do so, change the SSH remote URL's host by setting `gitlab_rails['gitlab_ssh_host']` in `gitlab.rb` of web nodes. - HTTP remote URL as shown in - [Custom Git clone URL for HTTP(S)](../../../user/admin_area/settings/visibility_and_access_controls.md#custom-git-clone-url-for-https). + [Custom Git clone URL for HTTP(S)](../../../user/admin_area/settings/visibility_and_access_controls.md#customize-git-clone-url-for-https). ## Example Git request handling behavior diff --git a/doc/administration/geo/replication/multiple_servers.md b/doc/administration/geo/replication/multiple_servers.md index ea2488b65fb..7db210d31f4 100644 --- a/doc/administration/geo/replication/multiple_servers.md +++ b/doc/administration/geo/replication/multiple_servers.md @@ -17,71 +17,59 @@ described, it is possible to adapt these instructions to your needs. _[diagram source - GitLab employees only](https://docs.google.com/drawings/d/1z0VlizKiLNXVVVaERFwgsIOuEgjcUqDTWPdQYsE7Z4c/edit)_ -The topology above assumes the **primary** and **secondary** Geo clusters +The topology above assumes the **primary** and **secondary** Geo sites are located in two separate locations, on their own virtual network with private IP addresses. The network is configured such that all machines in one geographic location can communicate with each other using their private IP addresses. The IP addresses given are examples and may be different depending on the network topology of your deployment. -The only external way to access the two Geo deployments is by HTTPS at +The only external way to access the two Geo sites is by HTTPS at `gitlab.us.example.com` and `gitlab.eu.example.com` in the example above. NOTE: -The **primary** and **secondary** Geo deployments must be able to communicate to each other over HTTPS. +The **primary** and **secondary** Geo sites must be able to communicate to each other over HTTPS. ## Redis and PostgreSQL for multiple nodes -Geo supports: - -- Redis and PostgreSQL on the **primary** node configured for multiple nodes. -- Redis on **secondary** nodes configured for multiple nodes. - -NOTE: -Support for PostgreSQL on **secondary** nodes in multi-node configuration -[is planned](https://gitlab.com/groups/gitlab-org/-/epics/2536). - Because of the additional complexity involved in setting up this configuration for PostgreSQL and Redis, it is not covered by this Geo multi-node documentation. For more information on setting up a multi-node PostgreSQL cluster and Redis cluster using the Omnibus GitLab package, see: -- [PostgreSQL multi-node documentation](../../postgresql/replication_and_failover.md) +- [Geo multi-node database replication](../setup/database.md#multi-node-database-replication) - [Redis multi-node documentation](../../redis/replication_and_failover.md) NOTE: It is possible to use cloud hosted services for PostgreSQL and Redis, but this is beyond the scope of this document. -## Prerequisites: Two working GitLab multi-node clusters +## Prerequisites: Two independently working GitLab multi-node sites -One cluster serves as the **primary** node. Use the -[GitLab multi-node documentation](../../reference_architectures/index.md) to set this up. If +One GitLab site serves as the Geo **primary** site. Use the +[GitLab reference architectures documentation](../../reference_architectures/index.md) +to set this up. You can use different reference architecture sizes for each Geo site. If you already have a working GitLab instance that is in-use, it can be used as a -**primary**. +**primary** site. -The second cluster serves as the **secondary** node. Again, use the -[GitLab multi-node documentation](../../reference_architectures/index.md) to set this up. +The second GitLab site serves as the Geo **secondary** site. Again, use the +[GitLab reference architectures documentation](../../reference_architectures/index.md) to set this up. It's a good idea to log in and test it. However, be aware that its data is -wiped out as part of the process of replicating from the **primary** node. +wiped out as part of the process of replicating from the **primary** site. -## Configure the GitLab cluster to be the **primary** node +## Configure a GitLab site to be the Geo **primary** site -The following steps enable a GitLab cluster to serve as the **primary** node. +The following steps enable a GitLab site to serve as the Geo **primary** site. -### Step 1: Configure the **primary** frontend servers +### Step 1: Configure the **primary** frontend nodes 1. Edit `/etc/gitlab/gitlab.rb` and add the following: ```ruby ## - ## Enable the Geo primary role + ## The unique identifier for the Geo site. See + ## https://docs.gitlab.com/ee/user/admin_area/geo_nodes.html#common-settings ## - roles ['geo_primary_role'] - - ## - ## The unique identifier for the Geo node. - ## - gitlab_rails['geo_node_name'] = '<node_name_here>' + gitlab_rails['geo_node_name'] = '<site_name_here>' ## ## Disable automatic migrations @@ -93,155 +81,63 @@ After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus NOTE: PostgreSQL and Redis should have already been disabled on the -application servers during normal GitLab multi-node setup. Connections -from the application servers to services on the backend servers should +application nodes during normal GitLab multi-node setup. Connections +from the application nodes to services on the backend nodes should have also been configured. See multi-node configuration documentation for [PostgreSQL](../../postgresql/replication_and_failover.md#configuring-the-application-nodes) and [Redis](../../redis/replication_and_failover.md#example-configuration-for-the-gitlab-application). -### Step 2: Configure the **primary** database - -1. Edit `/etc/gitlab/gitlab.rb` and add the following: - - ```ruby - ## - ## Configure the Geo primary role and the PostgreSQL role - ## - roles ['geo_primary_role', 'postgres_role'] - ``` +## Configure the other GitLab site to be a Geo **secondary** site -## Configure a **secondary** node - -A **secondary** cluster is similar to any other GitLab multi-node cluster, with two +A **secondary** site is similar to any other GitLab multi-node site, with three major differences: -- The main PostgreSQL database is a read-only replica of the **primary** node's +- The main PostgreSQL database is a read-only replica of the Geo **primary** site's PostgreSQL database. -- There is also a single PostgreSQL database for the **secondary** cluster, - called the "tracking database", which tracks the synchronization state of - various resources. +- There is an additional PostgreSQL database for each Geo **secondary** site, + called the "Geo tracking database", which tracks the replication and verification + state of various resources. +- There is an additional GitLab service [`geo-logcursor`](../index.md#geo-log-cursor) Therefore, we set up the multi-node components one by one and include deviations from the normal multi-node setup. However, we highly recommend configuring a -brand-new cluster first, as if it were not part of a Geo setup. This allows -verifying that it is a working cluster. And only then should it be modified -for use as a Geo **secondary**. This helps to separate Geo setup problems from -unrelated problems. +brand-new GitLab site first, as if it were not part of a Geo setup. This allows +verifying that it is a working GitLab site. And only then should it be modified +for use as a Geo **secondary** site. This helps to separate Geo setup problems from +unrelated multi-node configuration problems. -### Step 1: Configure the Redis and Gitaly services on the **secondary** node +### Step 1: Configure the Redis and Gitaly services on the Geo **secondary** site Configure the following services, again using the non-Geo multi-node documentation: - [Configuring Redis for GitLab](../../redis/replication_and_failover.md#example-configuration-for-the-gitlab-application) for multiple nodes. - [Gitaly](../../gitaly/index.md), which stores data that is - synchronized from the **primary** node. + synchronized from the Geo **primary** site. NOTE: [NFS](../../nfs.md) can be used in place of Gitaly but is not recommended. -### Step 2: Configure the main read-only replica PostgreSQL database on the **secondary** node - -NOTE: -The following documentation assumes the database runs on -a single node only. Multi-node PostgreSQL on **secondary** nodes is -[not currently supported](https://gitlab.com/groups/gitlab-org/-/epics/2536). - -Configure the [**secondary** database](../setup/database.md) as a read-only replica of -the **primary** database. Use the following as a guide. - -1. Generate an MD5 hash of the desired password for the database user that the - GitLab application uses to access the read-replica database: - - Note that the username (`gitlab` by default) is incorporated into the hash. - - ```shell - gitlab-ctl pg-password-md5 gitlab - # Enter password: <your_password_here> - # Confirm password: <your_password_here> - # fca0b89a972d69f00eb3ec98a5838484 - ``` - - Use this hash to fill in `<md5_hash_of_your_password>` in the next step. +### Step 2: Configure Postgres streaming replication -1. Edit `/etc/gitlab/gitlab.rb` in the replica database machine, and add the - following: - - ```ruby - ## - ## Configure the Geo secondary role and the PostgreSQL role - ## - roles ['geo_secondary_role', 'postgres_role'] - - ## - ## The unique identifier for the Geo node. - ## This should match the secondary's application node. - ## - gitlab_rails['geo_node_name'] = '<node_name_here>' - - ## - ## Secondary address - ## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node - ## - replace '<tracking_database_ip>' with the public or VPC address of your Geo tracking database node - ## - postgresql['listen_address'] = '<secondary_node_ip>' - postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32', '<tracking_database_ip>/32'] - - ## - ## Database credentials password (defined previously in primary node) - ## - replicate same values here as defined in primary node - ## - postgresql['sql_user_password'] = '<md5_hash_of_your_password>' - gitlab_rails['db_password'] = '<your_password_here>' - - ## - ## When running the Geo tracking database on a separate machine, disable it - ## here and allow connections from the tracking database host. And ensure - ## the tracking database IP is in postgresql['md5_auth_cidr_addresses'] above. - ## - geo_postgresql['enable'] = false - - ## - ## Disable all other services that aren't needed. Note that we had to enable - ## geo_secondary_role to cause some configuration changes to postgresql, but - ## the role enables single-node services by default. - ## - alertmanager['enable'] = false - consul['enable'] = false - geo_logcursor['enable'] = false - gitaly['enable'] = false - gitlab_exporter['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = false - node_exporter['enable'] = false - pgbouncer_exporter['enable'] = false - prometheus['enable'] = false - redis['enable'] = false - redis_exporter['enable'] = false - patroni['enable'] = false - sidekiq['enable'] = false - sidekiq_cluster['enable'] = false - puma['enable'] = false - ``` - -After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. +Follow the [Geo database replication instructions](../setup/database.md). If using an external PostgreSQL instance, refer also to [Geo with external PostgreSQL instances](../setup/external_database.md). -### Step 3: Configure the tracking database on the **secondary** node +### Step 3: Configure the Geo tracking database on the Geo **secondary** site -NOTE: -This documentation assumes the tracking database runs on -only a single machine, rather than as a PostgreSQL cluster. +If you want to run the Geo tracking database in a multi-node PostgreSQL cluster, +then follow [Configuring Patroni cluster for the tracking PostgreSQL database](../setup/database.md#configuring-patroni-cluster-for-the-tracking-postgresql-database). -Configure the tracking database. +If you want to run the Geo tracking database on a single node, then follow +the instructions below. 1. Generate an MD5 hash of the desired password for the database user that the GitLab application uses to access the tracking database: - Note that the username (`gitlab_geo` by default) is incorporated into the + The username (`gitlab_geo` by default) is incorporated into the hash. ```shell @@ -254,8 +150,8 @@ Configure the tracking database. Use this hash to fill in `<tracking_database_password_md5_hash>` in the next step. -1. Edit `/etc/gitlab/gitlab.rb` in the tracking database machine, and add the - following: +1. On the machine where the Geo tracking database is intended to run, add the + following to `/etc/gitlab/gitlab.rb`: ```ruby ## @@ -271,29 +167,8 @@ Configure the tracking database. geo_postgresql['md5_auth_cidr_addresses'] = ['<replica_database_ip>/32'] gitlab_rails['db_host'] = '<replica_database_ip>' - # Prevent reconfigure from attempting to run migrations on the replica DB + # Prevent reconfigure from attempting to run migrations on the replica database gitlab_rails['auto_migrate'] = false - - ## - ## Ensure unnecessary services are disabled - ## - alertmanager['enable'] = false - consul['enable'] = false - geo_logcursor['enable'] = false - gitaly['enable'] = false - gitlab_exporter['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = false - node_exporter['enable'] = false - pgbouncer_exporter['enable'] = false - postgresql['enable'] = false - prometheus['enable'] = false - redis['enable'] = false - redis_exporter['enable'] = false - patroni['enable'] = false - sidekiq['enable'] = false - sidekiq_cluster['enable'] = false - puma['enable'] = false ``` After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. @@ -301,29 +176,42 @@ After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus If using an external PostgreSQL instance, refer also to [Geo with external PostgreSQL instances](../setup/external_database.md). -### Step 4: Configure the frontend application servers on the **secondary** node +### Step 4: Configure the frontend application nodes on the Geo **secondary** site -In the architecture overview, there are two machines running the GitLab -application services. These services are enabled selectively in the -configuration. +In the minimal [architecture diagram](#architecture-overview) above, there are two +machines running the GitLab application services. These services are enabled +selectively in the configuration. -Configure the GitLab Rails application servers following the relevant steps +Configure the GitLab Rails application nodes following the relevant steps outlined in the [reference architectures](../../reference_architectures/index.md), then make the following modifications: -1. Edit `/etc/gitlab/gitlab.rb` on each application server in the **secondary** - cluster, and add the following: +1. Edit `/etc/gitlab/gitlab.rb` on each application node in the Geo **secondary** + site, and add the following: ```ruby ## - ## Enable the Geo secondary role + ## Enable GitLab application services. The application_role enables many services. + ## Alternatively, you can choose to enable or disable specific services on + ## different nodes to aid in horizontal scaling and separation of concerns. ## - roles ['geo_secondary_role', 'application_role'] + roles ['application_role'] + + ## `application_role` already enables this. You only need this line if + ## you selectively enable individual services that depend on Rails, like + ## `puma`, `sidekiq`, `geo-logcursor`, etc. + gitlab_rails['enable'] = true ## - ## The unique identifier for the Geo node. + ## Enable Geo Log Cursor service ## - gitlab_rails['geo_node_name'] = '<node_name_here>' + geo_logcursor['enable'] = true + + ## + ## The unique identifier for the Geo site. See + ## https://docs.gitlab.com/ee/user/admin_area/geo_nodes.html#common-settings + ## + gitlab_rails['geo_node_name'] = '<site_name_here>' ## ## Disable automatic migrations @@ -331,12 +219,11 @@ then make the following modifications: gitlab_rails['auto_migrate'] = false ## - ## Configure the connection to the tracking DB. And disable application - ## servers from running tracking databases. + ## Configure the connection to the tracking database ## + geo_secondary['enable'] = true geo_secondary['db_host'] = '<geo_tracking_db_host>' geo_secondary['db_password'] = '<geo_tracking_db_password>' - geo_postgresql['enable'] = false ## ## Configure connection to the streaming replica database, if you haven't @@ -353,7 +240,7 @@ then make the following modifications: ## ## If you are using custom users not managed by Omnibus, you need to specify - ## UIDs and GIDs like below, and ensure they match between servers in a + ## UIDs and GIDs like below, and ensure they match between nodes in a ## cluster to avoid permissions issues ## user['uid'] = 9000 @@ -369,14 +256,17 @@ If you had set up PostgreSQL cluster using the omnibus package and had set `postgresql['sql_user_password'] = 'md5 digest of secret'`, keep in mind that `gitlab_rails['db_password']` and `geo_secondary['db_password']` contains the plaintext passwords. This is used to let the Rails -servers connect to the databases. +nodes connect to the databases. NOTE: -Make sure that current node IP is listed in `postgresql['md5_auth_cidr_addresses']` setting of your remote database. +Make sure that current node's IP is listed in +`postgresql['md5_auth_cidr_addresses']` setting of the read-replica database to +allow Rails on this node to connect to Postgres. After making these changes [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. -On the secondary the following GitLab frontend services are enabled: +In the [architecture overview](#architecture-overview) topology, the following GitLab +services are enabled on the "frontend" nodes: - `geo-logcursor` - `gitlab-pages` @@ -388,62 +278,41 @@ On the secondary the following GitLab frontend services are enabled: - `sidekiq` - `puma` -Verify these services by running `sudo gitlab-ctl status` on the frontend -application servers. +Verify these services exist by running `sudo gitlab-ctl status` on the frontend +application nodes. -### Step 5: Set up the LoadBalancer for the **secondary** node +### Step 5: Set up the LoadBalancer for the Geo **secondary** site -In this topology, a load balancer is required at each geographic location to -route traffic to the application servers. +The minimal [architecture diagram](#architecture-overview) above shows a load +balancer at each geographic location to route traffic to the application nodes. See [Load Balancer for GitLab with multiple nodes](../../load_balancer.md) for more information. -### Step 6: Configure the backend application servers on the **secondary** node +### Step 6: Configure the backend application nodes on the Geo **secondary** site -The minimal reference architecture diagram above shows all application services +The minimal [architecture diagram](#architecture-overview) above shows all application services running together on the same machines. However, for multiple nodes we [strongly recommend running all services separately](../../reference_architectures/index.md). -For example, a Sidekiq server could be configured similarly to the frontend -application servers above, with some changes to run only the `sidekiq` service: +For example, a Sidekiq node could be configured similarly to the frontend +application nodes above, with some changes to run only the `sidekiq` service: -1. Edit `/etc/gitlab/gitlab.rb` on each Sidekiq server in the **secondary** - cluster, and add the following: +1. Edit `/etc/gitlab/gitlab.rb` on each Sidekiq node in the Geo **secondary** + site, and add the following: ```ruby ## - ## Enable the Geo secondary role - ## - roles ['geo_secondary_role'] - - ## ## Enable the Sidekiq service ## sidekiq['enable'] = true + gitlab_rails['enable'] = true ## - ## Ensure unnecessary services are disabled - ## - alertmanager['enable'] = false - consul['enable'] = false - gitaly['enable'] = false - gitlab_exporter['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = false - node_exporter['enable'] = false - pgbouncer_exporter['enable'] = false - postgresql['enable'] = false - prometheus['enable'] = false - redis['enable'] = false - redis_exporter['enable'] = false - patroni['enable'] = false - puma['enable'] = false - - ## - ## The unique identifier for the Geo node. + ## The unique identifier for the Geo site. See + ## https://docs.gitlab.com/ee/user/admin_area/geo_nodes.html#common-settings ## - gitlab_rails['geo_node_name'] = '<node_name_here>' + gitlab_rails['geo_node_name'] = '<site_name_here>' ## ## Disable automatic migrations @@ -451,12 +320,11 @@ application servers above, with some changes to run only the `sidekiq` service: gitlab_rails['auto_migrate'] = false ## - ## Configure the connection to the tracking DB. And disable application - ## servers from running tracking databases. + ## Configure the connection to the tracking database ## + geo_secondary['enable'] = true geo_secondary['db_host'] = '<geo_tracking_db_host>' geo_secondary['db_password'] = '<geo_tracking_db_password>' - geo_postgresql['enable'] = false ## ## Configure connection to the streaming replica database, if you haven't @@ -473,7 +341,7 @@ application servers above, with some changes to run only the `sidekiq` service: ## ## If you are using custom users not managed by Omnibus, you need to specify - ## UIDs and GIDs like below, and ensure they match between servers in a + ## UIDs and GIDs like below, and ensure they match between nodes in a ## cluster to avoid permissions issues ## user['uid'] = 9000 @@ -484,8 +352,8 @@ application servers above, with some changes to run only the `sidekiq` service: registry['gid'] = 9002 ``` - You can similarly configure a server to run only the `geo-logcursor` service + You can similarly configure a node to run only the `geo-logcursor` service with `geo_logcursor['enable'] = true` and disabling Sidekiq with `sidekiq['enable'] = false`. - These servers do not need to be attached to the load balancer. + These nodes do not need to be attached to the load balancer. diff --git a/doc/administration/geo/replication/object_storage.md b/doc/administration/geo/replication/object_storage.md index 90a41ed3e1c..d3931c0ba80 100644 --- a/doc/administration/geo/replication/object_storage.md +++ b/doc/administration/geo/replication/object_storage.md @@ -9,6 +9,11 @@ type: howto Geo can be used in combination with Object Storage (AWS S3, or other compatible object storage). +The storage method for files is recorded in the database, and the database is replicated +from the **primary** Geo site to the **secondary** Geo site, so the **secondary** Geo site +must match the storage method of the **primary** Geo site. +Therefore, if the **primary** Geo site uses object storage, the **secondary** Geo site must use it too. + Currently, **secondary** sites can use either: - The same storage bucket as the **primary** site. diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index d63e927627a..e072696b37c 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -39,7 +39,7 @@ to help identify if something is wrong: - Is the node's secondary tracking database connected? - Is the node's secondary tracking database up-to-date? - + For information on how to resolve common errors reported from the UI, see [Fixing Common Errors](#fixing-common-errors). @@ -159,6 +159,27 @@ This machine's Geo node name matches a database record ... no doc/administration/geo/replication/troubleshooting.md#can-geo-detect-the-current-node-correctly ``` +### Message: `WARNING: oldest xmin is far in the past` and `pg_wal` size growing + +If a replication slot is inactive, +the `pg_wal` logs corresponding to the slot are reserved forever +(or until the slot is active again). This causes continuous disk usage growth +and the following messages appear repeatedly in the +[PostgreSQL logs](../../logs.md#postgresql-logs): + +```plaintext +WARNING: oldest xmin is far in the past +HINT: Close open transactions soon to avoid wraparound problems. +You might also need to commit or roll back old prepared transactions, or drop stale replication slots. +``` + +To fix this, do the following: + +1. [Connect to the primary database](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database). +1. Run `SELECT * FROM pg_replication_slots;`. +1. Note the `slot_name` that reports `active` as `f` (false). +1. Follow [all these steps to remove that Geo site](remove_geo_site.md). + ## Fixing errors found when running the Geo check Rake task When running this Rake task, you may see errors if the nodes are not properly configured: @@ -299,7 +320,7 @@ sudo gitlab-ctl \ --backup-timeout=21600 ``` -This will give the initial replication up to six hours to complete, rather than +This gives the initial replication up to six hours to complete, rather than the default thirty minutes. Adjust as required for your installation. ### Message: "PANIC: could not write to file `pg_xlog/xlogtemp.123`: No space left on device" @@ -314,7 +335,7 @@ log data to build up in `pg_xlog`. Removing the unused slots can reduce the amou ``` NOTE: - Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. + Using `gitlab-rails dbconsole` does not work, because managing replication slots requires superuser permissions. 1. View your replication slots with: @@ -325,7 +346,8 @@ log data to build up in `pg_xlog`. Removing the unused slots can reduce the amou Slots where `active` is `f` are not active. - When this slot should be active, because you have a **secondary** node configured using that slot, - log in to that **secondary** node and check the PostgreSQL logs why the replication is not running. + log in to that **secondary** node and check the [PostgreSQL logs](../../logs.md#postgresql-logs) + to view why the replication is not running. - If you are no longer using the slot (for example, you no longer have Geo enabled), you can remove it with in the PostgreSQL console session: @@ -336,13 +358,12 @@ Slots where `active` is `f` are not active. ### Message: "ERROR: canceling statement due to conflict with recovery" -This error may rarely occur under normal usage, and the system is resilient +This error occurs infrequently under normal usage, and the system is resilient enough to recover. However, under certain conditions, some database queries on secondaries may run -excessively long, which increases the frequency of this error. At some point, -some of these queries will never be able to complete due to being canceled -every time. +excessively long, which increases the frequency of this error. This can lead to a situation +where some queries never complete due to being canceled on every replication. These long-running queries are [planned to be removed in the future](https://gitlab.com/gitlab-org/gitlab/-/issues/34269), @@ -483,7 +504,7 @@ Then reconfigure GitLab: sudo gitlab-ctl reconfigure ``` -This will increase the timeout to four hours (14400 seconds). Choose a time +This increases the timeout to four hours (14400 seconds). Choose a time long enough to accommodate a full clone of your largest repositories. ### New LFS objects are never replicated @@ -521,7 +542,7 @@ to start again from scratch, there are a few steps that can help you: gitlab-ctl stop geo-logcursor ``` - You can watch Sidekiq logs to know when Sidekiq jobs processing have finished: + You can watch the [Sidekiq logs](../../logs.md#sidekiq-logs) to know when Sidekiq jobs processing has finished: ```shell gitlab-ctl tail sidekiq @@ -542,9 +563,8 @@ to start again from scratch, there are a few steps that can help you: 1. _(Optional)_ Rename other data folders and create new ones WARNING: - You may still have files on the **secondary** node that have been removed from **primary** node but - removal have not been reflected. If you skip this step, they will never be removed - from this Geo node. + You may still have files on the **secondary** node that have been removed from the **primary** node, but this + removal has not been reflected. If you skip this step, these files are not removed at all from the Geo node. Any uploaded content like file attachments, avatars or LFS objects are stored in a subfolder in one of the two paths below: @@ -639,12 +659,12 @@ Counts: #### If you are promoting a Geo secondary site running on a single server -`gitlab-ctl promotion-preflight-checks` will fail due to the existence of +`gitlab-ctl promotion-preflight-checks` fails due to the existence of `failed` rows in the `geo_design_registry` table. Use the [previous snippet](#design-repository-failures-on-mirrored-projects-and-project-imports) to determine the actual replication status of Design repositories. -`gitlab-ctl promote-to-primary-node` will fail since it runs preflight checks. +`gitlab-ctl promote-to-primary-node` fails since it runs preflight checks. If the [previous snippet](#design-repository-failures-on-mirrored-projects-and-project-imports) shows that all designs are synced, then you can use the `--skip-preflight-checks` option or the `--force` option to move forward with @@ -652,7 +672,7 @@ promotion. #### If you are promoting a Geo secondary site running on multiple servers -`gitlab-ctl promotion-preflight-checks` will fail due to the existence of +`gitlab-ctl promotion-preflight-checks` fails due to the existence of `failed` rows in the `geo_design_registry` table. Use the [previous snippet](#design-repository-failures-on-mirrored-projects-and-project-imports) to determine the actual replication status of Design repositories. @@ -744,7 +764,7 @@ When you need to run the `gitlab-pg-ctl` command to promote the PostgreSQL read-replica database. -In GitLab 12.8 and earlier, this command will fail with the message: +In GitLab 12.8 and earlier, this command fails with the message: ```plaintext sudo: gitlab-pg-ctl: command not found @@ -803,7 +823,7 @@ If you notice for some reason there are more artifacts on the Geo secondary node than on the Geo primary node, you can use the Rake task to [cleanup orphan artifact files](../../../raketasks/cleanup.md#remove-orphan-artifact-files). -On a Geo **secondary** node, this command will also clean up all Geo +On a Geo **secondary** node, this command also cleans up all Geo registry record related to the orphan files on disk. ## Fixing sign in errors diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index bc4128deb4a..fa343f7eb40 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -69,11 +69,14 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o sudo -i ``` -1. Edit `/etc/gitlab/gitlab.rb` and add a **unique** name for your node: +1. Edit `/etc/gitlab/gitlab.rb` and add a **unique** name for your site: ```ruby - # The unique identifier for the Geo node. - gitlab_rails['geo_node_name'] = '<node_name_here>' + ## + ## The unique identifier for the Geo site. See + ## https://docs.gitlab.com/ee/user/admin_area/geo_nodes.html#common-settings + ## + gitlab_rails['geo_node_name'] = '<site_name_here>' ``` 1. Reconfigure the **primary** node for the change to take effect: @@ -207,7 +210,12 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ```ruby ## ## Geo Primary role - ## - configure dependent flags automatically to enable Geo + ## - Configures Postgres settings for replication + ## - Prevents automatic upgrade of Postgres since it requires downtime of + ## streaming replication to Geo secondary sites + ## - Enables standard single-node GitLab services like NGINX, Puma, Redis, + ## Sidekiq, etc. If you are segregating services, then you will need to + ## explicitly disable unwanted services. ## roles(['geo_primary_role']) @@ -571,6 +579,13 @@ Leader instance**: patroni['password'] = 'PATRONI_API_PASSWORD' patroni['replication_password'] = 'PLAIN_TEXT_POSTGRESQL_REPLICATION_PASSWORD' + # Add all patroni nodes to the allowlist + patroni['allowlist'] = %w[ + 127.0.0.1/32 + PATRONI_PRIMARY1_IP/32 PATRONI_PRIMARY2_IP/32 PATRONI_PRIMARY3_IP/32 + PATRONI_SECONDARY1_IP/32 PATRONI_SECONDARY2_IP/32 PATRONI_SECONDARY3_IP/32 + ] + # We list all secondary instances as they can all become a Standby Leader postgresql['md5_auth_cidr_addresses'] = %w[ PATRONI_PRIMARY1_IP/32 PATRONI_PRIMARY2_IP/32 PATRONI_PRIMARY3_IP/32 PATRONI_PRIMARY_PGBOUNCER/32 @@ -725,6 +740,13 @@ For each Patroni instance on the secondary site: # Any other instance that needs access to the database as per documentation ] + + # Add patroni nodes to the allowlist + patroni['allowlist'] = %w[ + 127.0.0.1/32 + PATRONI_SECONDARY1_IP/32 PATRONI_SECONDARY2_IP/32 PATRONI_SECONDARY3_IP/32 + ] + patroni['standby_cluster']['enable'] = true patroni['standby_cluster']['host'] = 'INTERNAL_LOAD_BALANCER_PRIMARY_IP' patroni['standby_cluster']['port'] = INTERNAL_LOAD_BALANCER_PRIMARY_PORT @@ -903,6 +925,12 @@ For each Patroni instance on the secondary site for the tracking database: # Any other instance that needs access to the database as per documentation ] + # Add patroni nodes to the allowlist + patroni['allowlist'] = %w[ + 127.0.0.1/32 + PATRONI_TRACKINGDB1_IP/32 PATRONI_TRACKINGDB2_IP/32 PATRONI_TRACKINGDB3_IP/32 + ] + # Patroni configuration patroni['username'] = 'PATRONI_API_USERNAME' patroni['password'] = 'PATRONI_API_PASSWORD' diff --git a/doc/administration/geo/setup/external_database.md b/doc/administration/geo/setup/external_database.md index 3ec84f1268b..56d196b54ec 100644 --- a/doc/administration/geo/setup/external_database.md +++ b/doc/administration/geo/setup/external_database.md @@ -34,9 +34,10 @@ developed and tested. We aim to be compatible with most external roles ['geo_primary_role'] ## - ## The unique identifier for the Geo site. + ## The unique identifier for the Geo site. See + ## https://docs.gitlab.com/ee/user/admin_area/geo_nodes.html#common-settings ## - gitlab_rails['geo_node_name'] = '<geo_site_name_here>' + gitlab_rails['geo_node_name'] = '<site_name_here>' ``` 1. Reconfigure the **primary** node for the change to take effect: @@ -242,7 +243,7 @@ the tracking database on port 5432. 1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) -1. The reconfigure should automatically create the database. If needed, you can perform this task manually. Note that this task (whether run by itself or during reconfigure) requires the database user to be a superuser. +1. The reconfigure should automatically create the database. If needed, you can perform this task manually. This task (whether run by itself or during reconfigure) requires the database user to be a superuser. ```shell gitlab-rake geo:db:create diff --git a/doc/administration/get_started.md b/doc/administration/get_started.md index a9ac8b279de..6fe66aa1642 100644 --- a/doc/administration/get_started.md +++ b/doc/administration/get_started.md @@ -1,7 +1,5 @@ --- -stage: -group: -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: For assistance with this TAM Onboarding page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-other-projects-and-subjects. --- # Get started administering GitLab **(FREE)** @@ -85,6 +83,8 @@ While this isn't an exhaustive list, following these steps gives you a solid sta - Configure [user and IP rate limits](https://about.gitlab.com/blog/2020/05/20/gitlab-instance-security-best-practices/#user-and-ip-rate-limits). - Limit [webhooks local access](https://about.gitlab.com/blog/2020/05/20/gitlab-instance-security-best-practices/#webhooks). - Set [rate limits for protected paths](../user/admin_area/settings/protected_paths.md). +- Sign up for [Security Alerts](https://about.gitlab.com/company/preference-center/) from the Communication Preference Center. +- Keep track of security best practices on our [blog page](https://about.gitlab.com/blog/2020/05/20/gitlab-instance-security-best-practices/). ## Monitor GitLab performance @@ -146,7 +146,7 @@ Backups of GitLab databases and filesystems are taken every 24 hours, and are ke - GitLab SaaS creates backups to ensure your data is secure, but you can't use these methods to export or back up your data yourself. - Issues are stored in the database. They can't be stored in Git itself. - You can use the project export option in: - - [The UI](../user/project/settings/import_export.md#exporting-a-project-and-its-data). + - [The UI](../user/project/settings/import_export.md#export-a-project-and-its-data). - [The API](../api/project_import_export.md#schedule-an-export). - [Group export](../user/group/settings/import_export.md) does *not* export the projects in it, but does export: - Epics diff --git a/doc/administration/git_annex.md b/doc/administration/git_annex.md deleted file mode 100644 index d7fb8a37b9c..00000000000 --- a/doc/administration/git_annex.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'index.md' -remove_date: '2021-07-22' ---- - -This document was moved to [another location](index.md). - -<!-- This redirect file can be deleted after <2021-07-22>. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/git_protocol.md b/doc/administration/git_protocol.md index 6e391cb459e..e3e2db81fb0 100644 --- a/doc/administration/git_protocol.md +++ b/doc/administration/git_protocol.md @@ -113,14 +113,5 @@ URL to use SSH. ### Observe Git protocol version of connections -To observe what Git protocol versions are being used in a -production environment, you can use the following Prometheus query: - -```prometheus -sum(rate(gitaly_git_protocol_requests_total[1m])) by (grpc_method,git_protocol,grpc_service) -``` - -<!-- This link sporadically returns a 503 during automated link checking but is correct --> - -You can view what Git protocol versions are being used on GitLab.com at -<https://dashboards.gitlab.com/d/pqlQq0xik/git-protocol-versions>. +For information on observing the Git protocol versions are being used in a production environment, +see the [relevant documentation](gitaly/index.md#useful-queries). diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md index 0b22df5a115..5e8cbac42c1 100644 --- a/doc/administration/gitaly/configure_gitaly.md +++ b/doc/administration/gitaly/configure_gitaly.md @@ -217,10 +217,7 @@ disable enforcement. For more information, see the documentation on configuring 1. Edit `/etc/gitlab/gitlab.rb`: - <!-- - updates to following example must also be made at - https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab - --> + <!-- Updates to following example must also be made at https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab --> ```ruby # Avoid running unnecessary services on the Gitaly server @@ -267,10 +264,7 @@ disable enforcement. For more information, see the documentation on configuring 1. Append the following to `/etc/gitlab/gitlab.rb` for each respective Gitaly server: - <!-- - updates to following example must also be made at - https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab - --> + <!-- Updates to following example must also be made at https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab --> On `gitaly1.internal`: @@ -595,10 +589,7 @@ To configure Gitaly with TLS: 1. Edit `/etc/gitlab/gitlab.rb` and add: - <!-- - updates to following example must also be made at - https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab - --> + <!-- Updates to following example must also be made at https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab --> ```ruby gitaly['tls_listen_addr'] = "0.0.0.0:9999" @@ -693,12 +684,8 @@ To configure Gitaly with TLS: ### Observe type of Gitaly connections -[Prometheus](../monitoring/prometheus/index.md) can be used observe what type of connections Gitaly -is serving a production environment. Use the following Prometheus query: - -```prometheus -sum(rate(gitaly_connections_total[5m])) by (type) -``` +For information on observing the type of Gitaly connections being served, see the +[relevant documentation](index.md#useful-queries). ## `gitaly-ruby` @@ -790,20 +777,8 @@ repository. In the example above: - If another request comes in for a repository that has used up its 20 slots, that request gets queued. -You can observe the behavior of this queue using the Gitaly logs and Prometheus: - -- In the Gitaly logs, look for the string (or structured log field) `acquire_ms`. Messages that have - this field are reporting about the concurrency limiter. -- In Prometheus, look for the following metrics: - - - `gitaly_rate_limiting_in_progress`. - - `gitaly_rate_limiting_queued`. - - `gitaly_rate_limiting_seconds`. - -NOTE: -Although the name of the Prometheus metric contains `rate_limiting`, it's a concurrency limiter, not -a rate limiter. If a Gitaly client makes 1,000 requests in a row very quickly, concurrency doesn't -exceed 1, and the concurrency limiter has no effect. +You can observe the behavior of this queue using the Gitaly logs and Prometheus. For more +information, see the [relevant documentation](index.md#monitor-gitaly). ## Background Repository Optimization @@ -857,30 +832,11 @@ server" and "Gitaly client" refers to the same machine. ### Verify authentication monitoring -Before rotating a Gitaly authentication token, verify that you can monitor the authentication -behavior of your GitLab installation using Prometheus. Use the following Prometheus query: +Before rotating a Gitaly authentication token, verify that you can +[monitor the authentication behavior](index.md#useful-queries) of your GitLab installation using +Prometheus. -```prometheus -sum(rate(gitaly_authentications_total[5m])) by (enforced, status) -``` - -In a system where authentication is configured correctly and where you have live traffic, you -see something like this: - -```prometheus -{enforced="true",status="ok"} 4424.985419441742 -``` - -There may also be other numbers with rate 0. We care only about the non-zero numbers. - -The only non-zero number should have `enforced="true",status="ok"`. If you have other non-zero -numbers, something is wrong in your configuration. - -The `status="ok"` number reflects your current request rate. In the example above, Gitaly is -handling about 4000 requests per second. - -Now that you have established that you can monitor the Gitaly authentication behavior of your GitLab -installation, you can begin the rest of the procedure. +You can then continue the rest of the procedure. ### Enable "auth transitioning" mode @@ -955,7 +911,7 @@ result as you did at the start. For example: {enforced="true",status="ok"} 4424.985419441742 ``` -Note that `enforced="true"` means that authentication is being enforced. +`enforced="true"` means that authentication is being enforced. ## Pack-objects cache **(FREE SELF)** @@ -1079,7 +1035,7 @@ cache hit and the average amount of storage used by cache files. Entries older than `max_age` get evicted from the in-memory metadata store, and deleted from disk. -Note that eviction does not interfere with ongoing requests, so it is OK +Eviction does not interfere with ongoing requests, so it is OK for `max_age` to be less than the time it takes to do a fetch over a slow connection. This is because Unix filesystems do not truly delete a file until all processes that are reading the deleted file have @@ -1087,9 +1043,8 @@ closed it. ### Observe the cache -The cache can be observed in logs and using metrics. - -#### Logs +The cache can be observed [using metrics](index.md#monitor-gitaly) and in the following logged +information: |Message|Fields|Description| |:---|:---|:---| @@ -1149,33 +1104,3 @@ Example: "time":"2021-03-25T14:57:53.543Z" } ``` - -#### Metrics - -The following cache metrics are available. - -|Metric|Type|Labels|Description| -|:---|:---|:---|:---| -|`gitaly_pack_objects_cache_enabled`|gauge|`dir`,`max_age`|Set to `1` when the cache is enabled via the Gitaly configuration file| -|`gitaly_pack_objects_cache_lookups_total`|counter|`result`|Hit/miss counter for cache lookups| -|`gitaly_pack_objects_generated_bytes_total`|counter||Number of bytes written into the cache| -|`gitaly_pack_objects_served_bytes_total`|counter||Number of bytes read from the cache| -|`gitaly_streamcache_filestore_disk_usage_bytes`|gauge|`dir`|Total size of cache files| -|`gitaly_streamcache_index_entries`|gauge|`dir`|Number of entries in the cache| - -Some of these metrics start with `gitaly_streamcache` -because they are generated by the "streamcache" internal library -package in Gitaly. - -Example: - -```plaintext -gitaly_pack_objects_cache_enabled{dir="/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache",max_age="300"} 1 -gitaly_pack_objects_cache_lookups_total{result="hit"} 2 -gitaly_pack_objects_cache_lookups_total{result="miss"} 1 -gitaly_pack_objects_generated_bytes_total 2.618649e+07 -gitaly_pack_objects_served_bytes_total 7.855947e+07 -gitaly_streamcache_filestore_disk_usage_bytes{dir="/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache"} 2.6200152e+07 -gitaly_streamcache_filestore_removed_total{dir="/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache"} 1 -gitaly_streamcache_index_entries{dir="/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache"} 1 -``` diff --git a/doc/administration/gitaly/faq.md b/doc/administration/gitaly/faq.md index a5964b7a2eb..c7ecaa020e0 100644 --- a/doc/administration/gitaly/faq.md +++ b/doc/administration/gitaly/faq.md @@ -25,7 +25,7 @@ The following table outlines the major differences between Gitaly Cluster and Ge | Tool | Nodes | Locations | Latency tolerance | Failover | Consistency | Provides redundancy for | |:---------------|:---------|:----------|:-------------------|:----------------------------------------------------------------------------|:-----------------------------------------|:------------------------| -| Gitaly Cluster | Multiple | Single | Approximately 1 ms | [Automatic](praefect.md#automatic-failover-and-primary-election-strategies) | [Strong](praefect.md#strong-consistency) | Data storage in Git | +| Gitaly Cluster | Multiple | Single | Approximately 1 ms | [Automatic](praefect.md#automatic-failover-and-primary-election-strategies) | [Strong](index.md#strong-consistency) | Data storage in Git | | Geo | Multiple | Multiple | Up to one minute | [Manual](../geo/disaster_recovery/index.md) | Eventual | Entire GitLab instance | For more information, see: @@ -35,12 +35,12 @@ For more information, see: ## Are there instructions for migrating to Gitaly Cluster? -Yes! For more information, see [Migrate to Gitaly Cluster](praefect.md#migrate-to-gitaly-cluster). +Yes! For more information, see [Migrate to Gitaly Cluster](index.md#migrate-to-gitaly-cluster). ## What are some repository storage recommendations? The size of the required storage can vary between instances and depends on the set -[replication factor](praefect.md#replication-factor). You might want to include implementing +[replication factor](index.md#replication-factor). You might want to include implementing repository storage redundancy. For a replication factor: diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 0af248e0573..bca83e903ac 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -30,8 +30,8 @@ repository storage is either: - A Gitaly storage with direct access to repositories using [storage paths](../repository_storage_paths.md), where each repository is stored on a single Gitaly node. All requests are routed to this node. -- A virtual storage provided by [Gitaly Cluster](#gitaly-cluster), where each repository can be - stored on multiple Gitaly nodes for fault tolerance. In a Gitaly Cluster: +- A [virtual storage](#virtual-storage) provided by [Gitaly Cluster](#gitaly-cluster), where each + repository can be stored on multiple Gitaly nodes for fault tolerance. In a Gitaly Cluster: - Read requests are distributed between multiple Gitaly nodes, which can improve performance. - Write requests are broadcast to repository replicas. @@ -39,32 +39,6 @@ WARNING: Engineering support for NFS for Git repositories is deprecated. Read the [deprecation notice](#nfs-deprecation-notice). -## Virtual storage - -Virtual storage makes it viable to have a single repository storage in GitLab to simplify repository -management. - -Virtual storage with Gitaly Cluster can usually replace direct Gitaly storage configurations. -However, this is at the expense of additional storage space needed to store each repository on multiple -Gitaly nodes. The benefit of using Gitaly Cluster virtual storage over direct Gitaly storage is: - -- Improved fault tolerance, because each Gitaly node has a copy of every repository. -- Improved resource utilization, reducing the need for over-provisioning for shard-specific peak - loads, because read loads are distributed across Gitaly nodes. -- Manual rebalancing for performance is not required, because read loads are distributed across - Gitaly nodes. -- Simpler management, because all Gitaly nodes are identical. - -The number of repository replicas can be configured using a -[replication factor](praefect.md#replication-factor). - -It can -be uneconomical to have the same replication factor for all repositories. -[Variable replication factor](https://gitlab.com/groups/gitlab-org/-/epics/3372) is planned to -provide greater flexibility for extremely large GitLab instances. - -As with normal Gitaly storages, virtual storages can be sharded. - ## Gitaly The following shows GitLab set up to use direct access to Gitaly: @@ -160,7 +134,7 @@ In this example: - Repositories are stored on a virtual storage called `storage-1`. - Three Gitaly nodes provide `storage-1` access: `gitaly-1`, `gitaly-2`, and `gitaly-3`. - The three Gitaly nodes share data in three separate hashed storage locations. -- The [replication factor](praefect.md#replication-factor) is `3`. There are three copies maintained +- The [replication factor](#replication-factor) is `3`. There are three copies maintained of each repository. The availability objectives for Gitaly clusters are: @@ -170,7 +144,7 @@ The availability objectives for Gitaly clusters are: Writes are replicated asynchronously. Any writes that have not been replicated to the newly promoted primary are lost. - [Strong consistency](praefect.md#strong-consistency) can be used to avoid loss in some + [Strong consistency](#strong-consistency) can be used to avoid loss in some circumstances. - **Recovery Time Objective (RTO):** Less than 10 seconds. @@ -178,20 +152,34 @@ The availability objectives for Gitaly clusters are: second. Failover requires ten consecutive failed health checks on each Praefect node. - [Faster outage detection](https://gitlab.com/gitlab-org/gitaly/-/issues/2608) - is planned to improve this to less than 1 second. + Faster outage detection, to improve this speed to less than 1 second, + is tracked [in this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/2608). + +### Virtual storage + +Virtual storage makes it viable to have a single repository storage in GitLab to simplify repository +management. + +Virtual storage with Gitaly Cluster can usually replace direct Gitaly storage configurations. +However, this is at the expense of additional storage space needed to store each repository on multiple +Gitaly nodes. The benefit of using Gitaly Cluster virtual storage over direct Gitaly storage is: -Gitaly Cluster supports: +- Improved fault tolerance, because each Gitaly node has a copy of every repository. +- Improved resource utilization, reducing the need for over-provisioning for shard-specific peak + loads, because read loads are distributed across Gitaly nodes. +- Manual rebalancing for performance is not required, because read loads are distributed across + Gitaly nodes. +- Simpler management, because all Gitaly nodes are identical. -- [Strong consistency](praefect.md#strong-consistency) of the secondary replicas. -- [Automatic failover](praefect.md#automatic-failover-and-primary-election-strategies) from the primary to the secondary. -- Reporting of possible data loss if replication queue is non-empty. -- From GitLab 13.0 to GitLab 14.0, marking repositories as [read-only](praefect.md#read-only-mode) - if data loss is detected to prevent data inconsistencies. +The number of repository replicas can be configured using a +[replication factor](#replication-factor). + +It can +be uneconomical to have the same replication factor for all repositories. +To provide greater flexibility for extremely large GitLab instances, +variable replication factor is tracked in [this issue](https://gitlab.com/groups/gitlab-org/-/epics/3372). -Follow the [Gitaly Cluster epic](https://gitlab.com/groups/gitlab-org/-/epics/1489) -for improvements including -[horizontally distributing reads](https://gitlab.com/groups/gitlab-org/-/epics/2013). +As with normal Gitaly storages, virtual storages can be sharded. ### Moving beyond NFS @@ -220,7 +208,7 @@ Further reading: - Blog post: [The road to Gitaly v1.0 (aka, why GitLab doesn't require NFS for storing Git data anymore)](https://about.gitlab.com/blog/2018/09/12/the-road-to-gitaly-1-0/) - Blog post: [How we spent two weeks hunting an NFS bug in the Linux kernel](https://about.gitlab.com/blog/2018/11/14/how-we-spent-two-weeks-hunting-an-nfs-bug/) -### Components of Gitaly Cluster +### Components Gitaly Cluster consists of multiple components: @@ -240,10 +228,227 @@ component for running a Gitaly Cluster. For more information, see [Gitaly High Availability (HA) Design](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/design_ha.md). +### Features + +Gitaly Cluster provides the following features: + +- [Distributed reads](#distributed-reads) among Gitaly nodes. +- [Strong consistency](#strong-consistency) of the secondary replicas. +- [Replication factor](#replication-factor) of repositories for increased redundancy. +- [Automatic failover](praefect.md#automatic-failover-and-primary-election-strategies) from the + primary Gitaly node to secondary Gitaly nodes. +- Reporting of possible [data loss](praefect.md#check-for-data-loss) if replication queue is + non-empty. + +Follow the [Gitaly Cluster epic](https://gitlab.com/groups/gitlab-org/-/epics/1489) for improvements +including [horizontally distributing reads](https://gitlab.com/groups/gitlab-org/-/epics/2013). + +#### Distributed reads + +> - Introduced in GitLab 13.1 in [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga) with feature flag `gitaly_distributed_reads` set to disabled. +> - [Made generally available and enabled by default](https://gitlab.com/gitlab-org/gitaly/-/issues/2951) in GitLab 13.3. +> - [Disabled by default](https://gitlab.com/gitlab-org/gitaly/-/issues/3178) in GitLab 13.5. +> - [Enabled by default](https://gitlab.com/gitlab-org/gitaly/-/issues/3334) in GitLab 13.8. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitaly/-/issues/3383) in GitLab 13.11. + +Gitaly Cluster supports distribution of read operations across Gitaly nodes that are configured for +the [virtual storage](#virtual-storage). + +All RPCs marked with the `ACCESSOR` option are redirected to an up to date and healthy Gitaly node. +For example, [`GetBlob`](https://gitlab.com/gitlab-org/gitaly/-/blob/v12.10.6/proto/blob.proto#L16). + +_Up to date_ in this context means that: + +- There is no replication operations scheduled for this Gitaly node. +- The last replication operation is in _completed_ state. + +The primary node is chosen to serve the request if: + +- There are no up to date nodes. +- Any other error occurs during node selection. + +You can [monitor distribution of reads](#monitor-gitaly-cluster) using Prometheus. + +#### Strong consistency + +> - Introduced in GitLab 13.1 in [alpha](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga), disabled by default. +> - Entered [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga) in GitLab 13.2, disabled by default. +> - In GitLab 13.3, disabled unless primary-wins voting strategy is disabled. +> - From GitLab 13.4, enabled by default. +> - From GitLab 13.5, you must use Git v2.28.0 or higher on Gitaly nodes to enable strong consistency. +> - From GitLab 13.6, primary-wins voting strategy and `gitaly_reference_transactions_primary_wins` feature flag were removed from the source code. + +By default, Gitaly Cluster guarantees eventual consistency by replicating all writes to secondary +Gitaly nodes after the write to the primary Gitaly node has happened. + +Praefect can instead provide strong consistency by creating a transaction and writing changes to all +Gitaly nodes at once. + +If enabled, transactions are only available for a subset of RPCs. For more information, see the +[strong consistency epic](https://gitlab.com/groups/gitlab-org/-/epics/1189). + +For configuration information, see [Configure strong consistency](praefect.md#configure-strong-consistency). + +#### Replication factor + +Replication factor is the number of copies Gitaly Cluster maintains of a given repository. A higher +replication factor: + +- Offers better redundancy and distribution of read workload. +- Results in higher storage cost. + +By default, Gitaly Cluster replicates repositories to every storage in a +[virtual storage](#virtual-storage). + +For configuration information, see [Configure replication factor](praefect.md#configure-replication-factor). + ### Configure Gitaly Cluster For more information on configuring Gitaly Cluster, see [Configure Gitaly Cluster](praefect.md). +### Migrate to Gitaly Cluster + +Whether migrating to Gitaly Cluster because of [NFS support deprecation](index.md#nfs-deprecation-notice) +or to move from single Gitaly nodes, the basic process involves: + +1. Create the required storage. Refer to + [repository storage recommendations](faq.md#what-are-some-repository-storage-recommendations). +1. Create and configure [Gitaly Cluster](praefect.md). +1. [Move the repositories](../operations/moving_repositories.md#move-repositories). To migrate to + Gitaly Cluster, existing repositories stored outside Gitaly Cluster must be moved. There is no + automatic migration but the moves can be scheduled with the GitLab API. + +## Monitor Gitaly and Gitaly Cluster + +You can use the available logs and [Prometheus metrics](../monitoring/prometheus/index.md) to +monitor Gitaly and Gitaly Cluster (Praefect). + +Metric definitions are available: + +- Directly from Prometheus `/metrics` endpoint configured for Gitaly. +- Using [Grafana Explore](https://grafana.com/docs/grafana/latest/explore/) on a + Grafana instance configured against Prometheus. + +### Monitor Gitaly + +You can observe the behavior of [queued requests](configure_gitaly.md#limit-rpc-concurrency) using +the Gitaly logs and Prometheus: + +- In the [Gitaly logs](../logs.md#gitaly-logs), look for the string (or structured log field) + `acquire_ms`. Messages that have this field are reporting about the concurrency limiter. +- In Prometheus, look for the following metrics: + - `gitaly_rate_limiting_in_progress`. + - `gitaly_rate_limiting_queued`. + - `gitaly_rate_limiting_seconds`. + + Although the name of the Prometheus metric contains `rate_limiting`, it's a concurrency limiter, + not a rate limiter. If a Gitaly client makes 1,000 requests in a row very quickly, concurrency + doesn't exceed 1, and the concurrency limiter has no effect. + +The following [pack-objects cache](configure_gitaly.md#pack-objects-cache) metrics are available: + +- `gitaly_pack_objects_cache_enabled`, a gauge set to `1` when the cache is enabled. Available + labels: `dir` and `max_age`. +- `gitaly_pack_objects_cache_lookups_total`, a counter for cache lookups. Available label: `result`. +- `gitaly_pack_objects_generated_bytes_total`, a counter for the number of bytes written into the + cache. +- `gitaly_pack_objects_served_bytes_total`, a counter for the number of bytes read from the cache. +- `gitaly_streamcache_filestore_disk_usage_bytes`, a gauge for the total size of cache files. + Available label: `dir`. +- `gitaly_streamcache_index_entries`, a gauge for the number of entries in the cache. Available + label: `dir`. + +Some of these metrics start with `gitaly_streamcache` because they are generated by the +`streamcache` internal library package in Gitaly. + +Example: + +```plaintext +gitaly_pack_objects_cache_enabled{dir="/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache",max_age="300"} 1 +gitaly_pack_objects_cache_lookups_total{result="hit"} 2 +gitaly_pack_objects_cache_lookups_total{result="miss"} 1 +gitaly_pack_objects_generated_bytes_total 2.618649e+07 +gitaly_pack_objects_served_bytes_total 7.855947e+07 +gitaly_streamcache_filestore_disk_usage_bytes{dir="/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache"} 2.6200152e+07 +gitaly_streamcache_filestore_removed_total{dir="/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache"} 1 +gitaly_streamcache_index_entries{dir="/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache"} 1 +``` + +#### Useful queries + +The following are useful queries for monitoring Gitaly: + +- Use the following Prometheus query to observe the + [type of connections](configure_gitaly.md#enable-tls-support) Gitaly is serving a production + environment: + + ```prometheus + sum(rate(gitaly_connections_total[5m])) by (type) + ``` + +- Use the following Prometheus query to monitor the + [authentication behavior](configure_gitaly.md#observe-type-of-gitaly-connections) of your GitLab + installation: + + ```prometheus + sum(rate(gitaly_authentications_total[5m])) by (enforced, status) + ``` + + In a system where authentication is configured correctly and where you have live traffic, you + see something like this: + + ```prometheus + {enforced="true",status="ok"} 4424.985419441742 + ``` + + There may also be other numbers with rate 0, but you only have to take note of the non-zero numbers. + + The only non-zero number should have `enforced="true",status="ok"`. If you have other non-zero + numbers, something is wrong in your configuration. + + The `status="ok"` number reflects your current request rate. In the example above, Gitaly is + handling about 4000 requests per second. + +- Use the following Prometheus query to observe the [Git protocol versions](../git_protocol.md) + being used in a production environment: + + ```prometheus + sum(rate(gitaly_git_protocol_requests_total[1m])) by (grpc_method,git_protocol,grpc_service) + ``` + +### Monitor Gitaly Cluster + +To monitor Gitaly Cluster (Praefect), you can use these Prometheus metrics: + +- `gitaly_praefect_read_distribution`, a counter to track [distribution of reads](#distributed-reads). + It has two labels: + + - `virtual_storage`. + - `storage`. + + They reflect configuration defined for this instance of Praefect. + +- `gitaly_praefect_replication_latency_bucket`, a histogram measuring the amount of time it takes + for replication to complete once the replication job starts. Available in GitLab 12.10 and later. +- `gitaly_praefect_replication_delay_bucket`, a histogram measuring how much time passes between + when the replication job is created and when it starts. Available in GitLab 12.10 and later. +- `gitaly_praefect_node_latency_bucket`, a histogram measuring the latency in Gitaly returning + health check information to Praefect. This indicates Praefect connection saturation. Available in + GitLab 12.10 and later. + +To monitor [strong consistency](#strong-consistency), you can use the following Prometheus metrics: + +- `gitaly_praefect_transactions_total`, the number of transactions created and voted on. +- `gitaly_praefect_subtransactions_per_transaction_total`, the number of times nodes cast a vote for + a single transaction. This can happen multiple times if multiple references are getting updated in + a single transaction. +- `gitaly_praefect_voters_per_transaction_total`: the number of Gitaly nodes taking part in a + transaction. +- `gitaly_praefect_transactions_delay_seconds`, the server-side delay introduced by waiting for the + transaction to be committed. +- `gitaly_hook_transaction_voting_delay_seconds`, the client-side delay introduced by waiting for + the transaction to be committed. + ## Do not bypass Gitaly GitLab doesn't advise directly accessing Gitaly repositories stored on disk with a Git client, @@ -253,8 +458,8 @@ your assumptions, resulting in performance degradation, instability, and even da - Gitaly has optimizations such as the [`info/refs` advertisement cache](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/design_diskcache.md), that rely on Gitaly controlling and monitoring access to repositories by using the official gRPC interface. -- [Gitaly Cluster](praefect.md) has optimizations, such as fault tolerance and - [distributed reads](praefect.md#distributed-reads), that depend on the gRPC interface and database +- [Gitaly Cluster](#gitaly-cluster) has optimizations, such as fault tolerance and + [distributed reads](#distributed-reads), that depend on the gRPC interface and database to determine repository state. WARNING: @@ -367,7 +572,7 @@ Additional information: GitLab recommends: - Creating a [Gitaly Cluster](#gitaly-cluster) as soon as possible. -- [Moving your repositories](praefect.md#migrate-to-gitaly-cluster) from NFS-based storage to Gitaly +- [Moving your repositories](#migrate-to-gitaly-cluster) from NFS-based storage to Gitaly Cluster. We welcome your feedback on this process. You can: diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index e483bcc944a..4af7f1a58a5 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -24,7 +24,7 @@ NOTE: Upgrade instructions for Omnibus GitLab installations [are available](https://docs.gitlab.com/omnibus/update/#gitaly-cluster). -## Requirements for configuring a Gitaly Cluster +## Requirements The minimum recommended configuration for a Gitaly Cluster requires: @@ -33,14 +33,33 @@ The minimum recommended configuration for a Gitaly Cluster requires: - 3 Praefect nodes - 3 Gitaly nodes (1 primary, 2 secondary) -See the [design -document](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/design_ha.md) +See the [design document](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/design_ha.md) for implementation details. NOTE: If not set in GitLab, feature flags are read as false from the console and Praefect uses their default value. The default value depends on the GitLab version. +### Network connectivity + +Gitaly Cluster [components](index.md#components) need to communicate with each other over many +routes. Your firewall rules must allow the following for Gitaly Cluster to function properly: + +| From | To | Default port | TLS port | +|:-----------------------|:-----------------------|:-------------|:---------| +| GitLab | Praefect load balancer | `2305` | `3305` | +| Praefect load balancer | Praefect | `2305` | `3305` | +| Praefect | Gitaly | `8075` | `9999` | +| Gitaly | GitLab (internal API) | `80` | `443` | +| Gitaly | Praefect load balancer | `2305` | `3305` | +| Gitaly | Praefect | `2305` | `3305` | +| Gitaly | Gitaly | `8075` | `9999` | + +NOTE: +Gitaly does not directly connect to Praefect. However, requests from Gitaly to the Praefect +load balancer may still be blocked unless firewalls on the Praefect nodes allow traffic from +the Gitaly nodes. + ## Setup Instructions If you [installed](https://about.gitlab.com/install/) GitLab using the Omnibus GitLab package @@ -129,7 +148,7 @@ The following options are available: - For non-Geo installations, either: - Use one of the documented [PostgreSQL setups](../postgresql/index.md). - - Use your own third-party database setup. This will require [manual setup](#manual-database-setup). + - Use your own third-party database setup. This requires [manual setup](#manual-database-setup). - For Geo instances, either: - Set up a separate [PostgreSQL instance](https://www.postgresql.org/docs/11/high-availability.html). - Use a cloud-managed PostgreSQL service. AWS @@ -176,7 +195,7 @@ instructions only work on Omnibus-provided PostgreSQL: ``` Replace `<PRAEFECT_SQL_PASSWORD_HASH>` with the hash of the password you generated in the - preparation step. Note that it is prefixed with `md5` literal. + preparation step. It is prefixed with `md5` literal. 1. The PgBouncer that is shipped with Omnibus is configured to use [`auth_query`](https://www.pgbouncer.org/config.html#generic-settings) and uses `pg_shadow_lookup` function. You need to create this function in `praefect_production` @@ -413,7 +432,7 @@ On the **Praefect** node: WARNING: If you have data on an already existing storage called `default`, you should configure the virtual storage with another name and - [migrate the data to the Gitaly Cluster storage](#migrate-to-gitaly-cluster) + [migrate the data to the Gitaly Cluster storage](index.md#migrate-to-gitaly-cluster) afterwards. Replace `PRAEFECT_INTERNAL_TOKEN` with a strong secret, which is used by @@ -457,7 +476,7 @@ On the **Praefect** node: In [GitLab 13.8 and earlier](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4988), Gitaly nodes were configured directly under the virtual storage, and not under the `nodes` key. -1. [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2013) in GitLab 13.1 and later, enable [distribution of reads](#distributed-reads). +1. [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2013) in GitLab 13.1 and later, enable [distribution of reads](index.md#distributed-reads). 1. Save the changes to `/etc/gitlab/gitlab.rb` and [reconfigure Praefect](../restart_gitlab.md#omnibus-gitlab-reconfigure): @@ -877,7 +896,7 @@ Particular attention should be shown to: WARNING: If you have existing data stored on the default Gitaly storage, - you should [migrate the data your Gitaly Cluster storage](#migrate-to-gitaly-cluster) + you should [migrate the data your Gitaly Cluster storage](index.md#migrate-to-gitaly-cluster) first. ```ruby @@ -1053,75 +1072,9 @@ To get started quickly: Congratulations! You've configured an observable fault-tolerant Praefect cluster. -## Network connectivity requirements - -Gitaly Cluster components need to communicate with each other over many routes. -Your firewall rules must allow the following for Gitaly Cluster to function properly: - -| From | To | Default port / TLS port | -|:-----------------------|:------------------------|:------------------------| -| GitLab | Praefect load balancer | `2305` / `3305` | -| Praefect load balancer | Praefect | `2305` / `3305` | -| Praefect | Gitaly | `8075` / `9999` | -| Gitaly | GitLab (internal API) | `80` / `443` | -| Gitaly | Praefect load balancer | `2305` / `3305` | -| Gitaly | Praefect | `2305` / `3305` | -| Gitaly | Gitaly | `8075` / `9999` | - -NOTE: -Gitaly does not directly connect to Praefect. However, requests from Gitaly to the Praefect -load balancer may still be blocked unless firewalls on the Praefect nodes allow traffic from -the Gitaly nodes. - -## Distributed reads - -> - Introduced in GitLab 13.1 in [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga) with feature flag `gitaly_distributed_reads` set to disabled. -> - [Made generally available and enabled by default](https://gitlab.com/gitlab-org/gitaly/-/issues/2951) in GitLab 13.3. -> - [Disabled by default](https://gitlab.com/gitlab-org/gitaly/-/issues/3178) in GitLab 13.5. -> - [Enabled by default](https://gitlab.com/gitlab-org/gitaly/-/issues/3334) in GitLab 13.8. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitaly/-/issues/3383) in GitLab 13.11. - -Praefect supports distribution of read operations across Gitaly nodes that are -configured for the virtual node. - -All RPCs marked with `ACCESSOR` option like -[GetBlob](https://gitlab.com/gitlab-org/gitaly/-/blob/v12.10.6/proto/blob.proto#L16) -are redirected to an up to date and healthy Gitaly node. - -_Up to date_ in this context means that: +## Configure strong consistency -- There is no replication operations scheduled for this node. -- The last replication operation is in _completed_ state. - -If there is no such nodes, or any other error occurs during node selection, the primary -node is chosen to serve the request. - -To track distribution of read operations, you can use the `gitaly_praefect_read_distribution` -Prometheus counter metric. It has two labels: - -- `virtual_storage`. -- `storage`. - -They reflect configuration defined for this instance of Praefect. - -## Strong consistency - -> - Introduced in GitLab 13.1 in [alpha](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga), disabled by default. -> - Entered [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga) in GitLab 13.2, disabled by default. -> - In GitLab 13.3, disabled unless primary-wins voting strategy is disabled. -> - From GitLab 13.4, enabled by default. -> - From GitLab 13.5, you must use Git v2.28.0 or higher on Gitaly nodes to enable strong consistency. -> - From GitLab 13.6, primary-wins voting strategy and `gitaly_reference_transactions_primary_wins` feature flag were removed from the source code. - -Praefect guarantees eventual consistency by replicating all writes to secondary nodes -after the write to the primary Gitaly node has happened. - -Praefect can instead provide strong consistency by creating a transaction and writing -changes to all Gitaly nodes at once. -If enabled, transactions are only available for a subset of RPCs. For more -information, see the [strong consistency epic](https://gitlab.com/groups/gitlab-org/-/epics/1189). - -To enable strong consistency: +To enable [strong consistency](index.md#strong-consistency): - In GitLab 13.5, you must use Git v2.28.0 or higher on Gitaly nodes to enable strong consistency. - In GitLab 13.4 and later, the strong consistency voting strategy has been improved and enabled by default. @@ -1141,28 +1094,10 @@ Feature.enable(:gitaly_reference_transactions) Feature.disable(:gitaly_reference_transactions_primary_wins) ``` -To monitor strong consistency, you can use the following Prometheus metrics: - -- `gitaly_praefect_transactions_total`: Number of transactions created and - voted on. -- `gitaly_praefect_subtransactions_per_transaction_total`: Number of times - nodes cast a vote for a single transaction. This can happen multiple times if - multiple references are getting updated in a single transaction. -- `gitaly_praefect_voters_per_transaction_total`: Number of Gitaly nodes taking - part in a transaction. -- `gitaly_praefect_transactions_delay_seconds`: Server-side delay introduced by - waiting for the transaction to be committed. -- `gitaly_hook_transaction_voting_delay_seconds`: Client-side delay introduced - by waiting for the transaction to be committed. +For information on monitoring strong consistency, see the +[relevant documentation](index.md#monitor-gitaly-cluster). -## Replication factor - -Replication factor is the number of copies Praefect maintains of a given repository. A higher -replication factor offers better redundancy and distribution of read workload, but also results -in a higher storage cost. By default, Praefect replicates repositories to every storage in a -virtual storage. - -### Configure replication factor +## Configure replication factor WARNING: Configurable replication factors require [repository-specific primary nodes](#repository-specific-primary-nodes) to be used. @@ -1639,128 +1574,3 @@ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.t - Replace the placeholder `<virtual-storage>` with the virtual storage containing the Gitaly node storage to be checked. - Replace the placeholder `<up-to-date-storage>` with the Gitaly storage name containing up to date repositories. - Replace the placeholder `<outdated-storage>` with the Gitaly storage name containing outdated repositories. - -## Migrate to Gitaly Cluster - -Whether migrating to Gitaly Cluster because of [NFS support deprecation](index.md#nfs-deprecation-notice) -or to move from single Gitaly nodes, the basic process involves: - -1. Create the required storage. -1. Create and configure Gitaly Cluster. -1. [Move the repositories](#move-repositories). - -When creating the storage, see some -[repository storage recommendations](faq.md#what-are-some-repository-storage-recommendations). - -### Move Repositories - -To migrate to Gitaly Cluster, existing repositories stored outside Gitaly Cluster must be -moved. There is no automatic migration but the moves can be scheduled with the GitLab API. - -GitLab repositories can be associated with projects, groups, and snippets. Each of these types -have a separate API to schedule the respective repositories to move. To move all repositories -on a GitLab instance, each of these types must be scheduled to move for each storage. - -Each repository is made read-only for the duration of the move. The repository is not writable -until the move has completed. - -After creating and configuring Gitaly Cluster: - -1. Ensure all storages are accessible to the GitLab instance. In this example, these are - `<original_storage_name>` and `<cluster_storage_name>`. -1. [Configure repository storage weights](../repository_storage_paths.md#configure-where-new-repositories-are-stored) - so that the Gitaly Cluster receives all new projects. This stops new projects from being created - on existing Gitaly nodes while the migration is in progress. -1. Schedule repository moves for: - - [Projects](#bulk-schedule-project-moves). - - [Snippets](#bulk-schedule-snippet-moves). - - [Groups](#bulk-schedule-group-moves). **(PREMIUM SELF)** - -#### Bulk schedule project moves - -1. [Schedule repository storage moves for all projects on a storage shard](../../api/project_repository_storage_moves.md#schedule-repository-storage-moves-for-all-projects-on-a-storage-shard) using the API. For example: - - ```shell - curl --request POST --header "Private-Token: <your_access_token>" \ - --header "Content-Type: application/json" \ - --data '{"source_storage_name":"<original_storage_name>","destination_storage_name":"<cluster_storage_name>"}' \ - "https://gitlab.example.com/api/v4/project_repository_storage_moves" - ``` - -1. [Query the most recent repository moves](../../api/project_repository_storage_moves.md#retrieve-all-project-repository-storage-moves) - using the API. The query indicates either: - - The moves have completed successfully. The `state` field is `finished`. - - The moves are in progress. Re-query the repository move until it completes successfully. - - The moves have failed. Most failures are temporary and are solved by rescheduling the move. - -1. After the moves are complete, [query projects](../../api/projects.md#list-all-projects) - using the API to confirm that all projects have moved. No projects should be returned - with `repository_storage` field set to the old storage. - - ```shell - curl --header "Private-Token: <your_access_token>" --header "Content-Type: application/json" \ - "https://gitlab.example.com/api/v4/projects?repository_storage=<original_storage_name>" - ``` - - Alternatively use [the rails console](../operations/rails_console.md) to - confirm that all projects have moved. Run the following in the rails console: - - ```ruby - ProjectRepository.for_repository_storage('<original_storage_name>') - ``` - -1. Repeat for each storage as required. - -#### Bulk schedule snippet moves - -1. [Schedule repository storage moves for all snippets on a storage shard](../../api/snippet_repository_storage_moves.md#schedule-repository-storage-moves-for-all-snippets-on-a-storage-shard) using the API. For example: - - ```shell - curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ - --header "Content-Type: application/json" \ - --data '{"source_storage_name":"<original_storage_name>","destination_storage_name":"<cluster_storage_name>"}' \ - "https://gitlab.example.com/api/v4/snippet_repository_storage_moves" - ``` - -1. [Query the most recent repository moves](../../api/snippet_repository_storage_moves.md#retrieve-all-snippet-repository-storage-moves) - using the API. The query indicates either: - - The moves have completed successfully. The `state` field is `finished`. - - The moves are in progress. Re-query the repository move until it completes successfully. - - The moves have failed. Most failures are temporary and are solved by rescheduling the move. - -1. After the moves are complete, use [the rails console](../operations/rails_console.md) to - confirm that all snippets have moved. No snippets should be returned for the original - storage. Run the following in the rails console: - - ```ruby - SnippetRepository.for_repository_storage('<original_storage_name>') - ``` - -1. Repeat for each storage as required. - -#### Bulk schedule group moves **(PREMIUM SELF)** - -1. [Schedule repository storage moves for all groups on a storage shard](../../api/group_repository_storage_moves.md#schedule-repository-storage-moves-for-all-groups-on-a-storage-shard) using the API. - - ```shell - curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ - --header "Content-Type: application/json" \ - --data '{"source_storage_name":"<original_storage_name>","destination_storage_name":"<cluster_storage_name>"}' \ - "https://gitlab.example.com/api/v4/group_repository_storage_moves" - ``` - -1. [Query the most recent repository moves](../../api/group_repository_storage_moves.md#retrieve-all-group-repository-storage-moves) - using the API. The query indicates either: - - The moves have completed successfully. The `state` field is `finished`. - - The moves are in progress. Re-query the repository move until it completes successfully. - - The moves have failed. Most failures are temporary and are solved by rescheduling the move. - -1. After the moves are complete, use [the rails console](../operations/rails_console.md) to - confirm that all groups have moved. No groups should be returned for the original - storage. Run the following in the rails console: - - ```ruby - GroupWikiRepository.for_repository_storage('<original_storage_name>') - ``` - -1. Repeat for each storage as required. diff --git a/doc/administration/gitaly/reference.md b/doc/administration/gitaly/reference.md index ec5a8d47ae2..9fe09be10a3 100644 --- a/doc/administration/gitaly/reference.md +++ b/doc/administration/gitaly/reference.md @@ -71,7 +71,7 @@ Remember to disable `transitioning` when you are done changing your token settings. All authentication attempts are counted in Prometheus under -the `gitaly_authentications_total` metric. +the [`gitaly_authentications_total` metric](index.md#useful-queries). ### TLS diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index ab6f493cf0f..3dd700968f9 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -223,6 +223,28 @@ on the Gitaly server matches the one on Gitaly client. If it doesn't match, update the secrets file on the Gitaly server to match the Gitaly client, then [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). +### Repository pushes fail with a `deny updating a hidden ref` error + +Due to [a change](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3426) +introduced in GitLab 13.12, Gitaly has read-only, internal GitLab references that users are not +permitted to update. If you attempt to update internal references with `git push --mirror`, Git +returns the rejection error, `deny updating a hidden ref`. + +The following references are read-only: + +- refs/environments/ +- refs/keep-around/ +- refs/merge-requests/ +- refs/pipelines/ + +To mirror-push branches and tags only, and avoid attempting to mirror-push protected refs, run: + +```shell +git push origin +refs/heads/*:refs/heads/* +refs/tags/*:refs/tags/* +``` + +Any other namespaces that the admin wants to push can be included there as well via additional patterns. + ### Command line tools cannot connect to Gitaly gRPC cannot reach your Gitaly server if: @@ -370,3 +392,11 @@ the `praefect` command: $ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migrate praefect sql-migrate: OK (applied 21 migrations) ``` + +### Requests fail with 'repo scoped: invalid Repository' errors + +This indicates that the virtual storage name used in the +[Praefect configuration](praefect.md#praefect) does not match the storage name used in +[`git_data_dirs` setting](praefect.md#gitaly) for GitLab. + +Resolve this by matching the virtual storage names used in Praefect and GitLab configuration. diff --git a/doc/administration/img/time_zone_settings.png b/doc/administration/img/time_zone_settings.png Binary files differdeleted file mode 100644 index e735a8bc5ec..00000000000 --- a/doc/administration/img/time_zone_settings.png +++ /dev/null diff --git a/doc/administration/index.md b/doc/administration/index.md index 74c89b4d5c0..46624ab39a3 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -121,7 +121,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Creating users](../user/profile/account/create_accounts.md): Create users manually or through authentication integrations. - [Libravatar](libravatar.md): Use Libravatar instead of Gravatar for user avatars. - [Sign-up restrictions](../user/admin_area/settings/sign_up_restrictions.md): block email addresses of specific domains, or whitelist only specific domains. -- [Access restrictions](../user/admin_area/settings/visibility_and_access_controls.md#enabled-git-access-protocols): Define which Git access protocols can be used to talk to GitLab (SSH, HTTP, HTTPS). +- [Access restrictions](../user/admin_area/settings/visibility_and_access_controls.md#configure-enabled-git-access-protocols): Define which Git access protocols can be used to talk to GitLab (SSH, HTTP, HTTPS). - [Authentication and Authorization](auth/index.md): Configure external authentication with LDAP, SAML, CAS, and additional providers. - [Sync LDAP](auth/ldap/index.md) - [Kerberos authentication](../integration/kerberos.md) @@ -166,7 +166,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. ## Continuous Integration settings -- [Enable/disable GitLab CI/CD](../ci/enable_or_disable_ci.md#site-wide-admin-setting): Enable or disable GitLab CI/CD for your instance. +- [Enable/disable GitLab CI/CD](../ci/enable_or_disable_ci.md#make-gitlab-cicd-disabled-by-default-in-new-projects): Enable or disable GitLab CI/CD for your instance. - [GitLab CI/CD administration settings](../user/admin_area/settings/continuous_integration.md): Enable or disable Auto DevOps site-wide and define the artifacts' max size and expiration time. - [External Pipeline Validation](external_pipeline_validation.md): Enable, disable, and configure external pipeline validation. - [Job artifacts](job_artifacts.md): Enable, disable, and configure job artifacts (a set of files and directories which are outputted by a job when it completes successfully). diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index 5e0d87cd7b6..1ea2ec4c904 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -230,8 +230,10 @@ Set the limit to `0` to disable it. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/237891) in GitLab 13.7. -The [minimum time between pull refreshes](../user/project/repository/repository_mirroring.md) -defaults to 300 seconds (5 minutes). +The [minimum wait time between pull refreshes](../user/project/repository/repository_mirroring.md) +defaults to 300 seconds (5 minutes). For example, by default a pull refresh will only run once in a given 300 second period regardless of how many times you try to trigger it. + +This setting applies in the context of pull refreshes invoked via the [projects API](../api/projects.md#start-the-pull-mirroring-process-for-a-project), or when forcing an update by selecting the **Update now** (**{retry}**) button within **Settings > Repository > Mirroring repositories**. This setting has no effect on the automatic 30 minute interval schedule used by Sidekiq for [pull mirroring](../user/project/repository/repository_mirroring.md#how-it-works). To change this limit for a self-managed installation, run the following in the [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): @@ -457,6 +459,21 @@ installation, run the following in the [GitLab Rails console](operations/rails_c Plan.default.actual_limits.update!(ci_max_artifact_size_junit: 10) ``` +### Number of files per GitLab Pages web-site + +The total number of file entries (including directories and symlinks) is limited to `100000` per +GitLab Pages website. + +This is the default limit for all [GitLab self-managed and SaaS plans](https://about.gitlab.com/pricing/). + +You can update the limit in your self-managed instance using the +[GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session). +For example, to change the limit to `100`: + +```ruby +Plan.default.actual_limits.update!(pages_file_entries: 100) +``` + ### Number of registered runners per scope > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/321368) in GitLab 13.12. @@ -484,15 +501,8 @@ A runner's registration fails if it exceeds the limit for the scope determined b ### Maximum file size for job logs -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276192) in GitLab 14.1. -> - [Deployed behind a feature flag](../user/feature_flags.md), disabled by default. -> - Disabled on GitLab.com. -> - Not recommended for production use. -> - To use in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-job-log-limits). **(FREE SELF)** - -This in-development feature might not be available for your use. There can be -[risks when enabling features still in development](../user/feature_flags.md#risks-when-enabling-features-still-in-development). -Refer to this feature's version history for more details. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276192) in GitLab 14.1, disabled by default. +> - Enabled by default and [feature flag `ci_jobs_trace_size_limit` removed](https://gitlab.com/gitlab-org/gitlab/-/issues/335259) in GitLab 14.2. The job log file size limit is 100 megabytes by default. Any job that exceeds this value is dropped. @@ -503,25 +513,6 @@ Update `ci_jobs_trace_size_limit` with the new value in megabytes: Plan.default.actual_limits.update!(ci_jobs_trace_size_limit: 125) ``` -#### Enable or disable job log limits **(FREE SELF)** - -This feature is under development and not ready for production use. It is -deployed behind a feature flag that is **disabled by default**. -[GitLab administrators with access to the GitLab Rails console](feature_flags.md) -can enable it. - -To enable it: - -```ruby -Feature.enable(:ci_jobs_trace_size_limit) -``` - -To disable it: - -```ruby -Feature.disable(:ci_jobs_trace_size_limit) -``` - ## Instance monitoring and metrics ### Incident Management inbound alert limits @@ -626,8 +617,8 @@ Reports that go over the 20 MB limit won't be loaded. Affected reports: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8638) in GitLab 13.3. You can set a limit on the content of repository files that are indexed in -Elasticsearch. Any files larger than this limit is neither indexed -nor searchable. +Elasticsearch. Any files larger than this limit only index the file name. +The file content is neither indexed nor searchable. Setting a limit helps reduce the memory usage of the indexing processes and the overall index size. This value defaults to `1024 KiB` (1 MiB) as any diff --git a/doc/administration/integration/kroki.md b/doc/administration/integration/kroki.md index e36b8a0be9d..729894052b2 100644 --- a/doc/administration/integration/kroki.md +++ b/doc/administration/integration/kroki.md @@ -1,6 +1,6 @@ --- -stage: Create -group: Ecosystem +stage: Ecosystem +group: Integrations info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/integration/plantuml.md b/doc/administration/integration/plantuml.md index 2b18efde95d..9f83ba8e972 100644 --- a/doc/administration/integration/plantuml.md +++ b/doc/administration/integration/plantuml.md @@ -208,7 +208,7 @@ After configuring your local PlantUML server, you're ready to enable the PlantUM 1. Sign in to GitLab as an [Administrator](../../user/permissions.md) user. 1. On the top bar, select **Menu >** **{admin}** **Admin**. 1. In the left sidebar, go to **Settings > General** and expand the **PlantUML** section. -1. Select the **Enable PlantUML** check box. +1. Select the **Enable PlantUML** checkbox. 1. Set the PlantUML instance as `https://gitlab.example.com/-/plantuml/`, and click **Save changes**. diff --git a/doc/administration/integration/terminal.md b/doc/administration/integration/terminal.md index 9302e9a1edc..1be234c2771 100644 --- a/doc/administration/integration/terminal.md +++ b/doc/administration/integration/terminal.md @@ -1,6 +1,6 @@ --- -stage: Create -group: Ecosystem +stage: Ecosystem +group: Integrations info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/job_logs.md b/doc/administration/job_logs.md index 87dd365769f..a4fdf734c2e 100644 --- a/doc/administration/job_logs.md +++ b/doc/administration/job_logs.md @@ -101,7 +101,7 @@ Archived logs are considered as [job artifacts](job_artifacts.md). Therefore, when you [set up the object storage integration](job_artifacts.md#object-storage-settings), job logs are automatically migrated to it along with the other job artifacts. -See "Phase 4: uploading" in [Data flow](#data-flow) to learn about the process. +See "Phase 3: uploading" in [Data flow](#data-flow) to learn about the process. ## Prevent local disk usage diff --git a/doc/administration/lfs/index.md b/doc/administration/lfs/index.md index edf0e324a5c..682352d8f59 100644 --- a/doc/administration/lfs/index.md +++ b/doc/administration/lfs/index.md @@ -323,7 +323,7 @@ See more information in [!19581](https://gitlab.com/gitlab-org/gitlab-foss/-/mer ### LFS commands fail on TLS v1.3 server -If you configure GitLab to [disable TLS v1.2](https://docs.gitlab.com/omnibus/settings/nginx.md) +If you configure GitLab to [disable TLS v1.2](https://docs.gitlab.com/omnibus/settings/nginx.html) and only enable TLS v1.3 connections, LFS operations require a [Git LFS client](https://git-lfs.github.com) version 2.11.0 or later. If you use a Git LFS client earlier than version 2.11.0, GitLab displays an error: @@ -334,7 +334,7 @@ error: failed to fetch some objects from 'https://username:[MASKED]@gitlab.examp ``` When using GitLab CI over a TLS v1.3 configured GitLab server, you must -[upgrade to GitLab Runner](https://docs.gitlab.com/runner/install/index.md) 13.2.0 +[upgrade to GitLab Runner](https://docs.gitlab.com/runner/install/index.html) 13.2.0 or later to receive an updated Git LFS client version via the included [GitLab Runner Helper image](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#helper-image). diff --git a/doc/administration/libravatar.md b/doc/administration/libravatar.md index b62e2ed4313..142cf2f3685 100644 --- a/doc/administration/libravatar.md +++ b/doc/administration/libravatar.md @@ -82,7 +82,7 @@ which case the URL would look like: `plain_url: "http://cdn.libravatar.org/avata ## Usage examples for Microsoft Office 365 If your users are Office 365 users, the `GetPersonaPhoto` service can be used. -Note that this service requires a login, so this use case is most useful in a +This service requires a login, so this use case is most useful in a corporate installation where all users have access to Office 365. ```ruby diff --git a/doc/administration/logs.md b/doc/administration/logs.md index b1605604df5..883f1db8e09 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -97,7 +97,8 @@ Line breaks were added to examples for legibility: "cpu_s":17.50, "db_duration_s":0.08, "view_duration_s":2.39, - "duration_s":20.54 + "duration_s":20.54, + "pid": 81836 } ``` @@ -120,6 +121,7 @@ seconds: - `redis_<instance>_duration_s`: Total time to retrieve data from a Redis instance - `redis_<instance>_read_bytes`: Total bytes read from a Redis instance - `redis_<instance>_write_bytes`: Total bytes written to a Redis instance +- `pid`: Process ID of the Puma worker User clone and fetch activity using HTTP transport appears in the log as `action: git_upload_pack`. @@ -190,7 +192,8 @@ Starting with GitLab 12.5, if an error occurs, an "cpu_s":17.50, "db_duration_s":0.08, "view_duration_s":2.39, - "duration_s":20.54 + "duration_s":20.54, + "pid": 81836, "exception.class": "NameError", "exception.message": "undefined local variable or method `adsf' for #<Admin::DashboardController:0x00007ff3c9648588>", "exception.backtrace": [ @@ -342,7 +345,7 @@ Depending on your installation method, this file is located at: - Installations from source: `/home/git/gitlab/log/integrations_json.log` It contains information about [integration](../user/project/integrations/overview.md) -activities, such as Jira, Asana, and Irker services. It uses JSON format, +activities, such as Jira, Asana, and irker services. It uses JSON format, like this example: ```json diff --git a/doc/administration/maintenance_mode/index.md b/doc/administration/maintenance_mode/index.md index 37415468517..7664c7c1751 100644 --- a/doc/administration/maintenance_mode/index.md +++ b/doc/administration/maintenance_mode/index.md @@ -60,7 +60,7 @@ There are three ways to disable Maintenance Mode: - [**Rails console**](../operations/rails_console.md#starting-a-rails-console-session): ```ruby - ::Gitlab::CurrentSettings.update_attributes!(maintenance_mode: false) + ::Gitlab::CurrentSettings.update!(maintenance_mode: false) ``` ## Behavior of GitLab features in Maintenance Mode diff --git a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md index b1ec74c2f40..f8764468256 100644 --- a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md +++ b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md @@ -4,43 +4,44 @@ group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab self monitoring project **(FREE SELF)** +# Self monitoring project **(FREE SELF)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32351) in GitLab 12.7, behind a disabled feature flag (`self_monitoring_project`). -> - The feature flag was removed and the Self Monitoring Project was [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/198511) in GitLab 12.8. +> - The feature flag was removed and the self monitoring project was [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/198511) in GitLab 12.8. -GitLab has been adding the ability for administrators to see insights into the -health of their GitLab instance. To surface this experience in a native way -(similar to how you would interact with an application deployed using GitLab), -a base project called "GitLab self monitoring" with -[internal visibility](../../../public_access/public_access.md#internal-projects) -is added under a group called "GitLab Instance Administrators" -specifically created for visualizing and configuring the monitoring of your -GitLab instance. +GitLab provides administrators insights into the health of their GitLab instance. -All administrators at the time of creation of the project and group are -added as maintainers of the group and project, and as an administrator, you can -add new members to the group to give them the [Maintainer role](../../../user/permissions.md) for -the project. +To provide a native experience (similar interacting with an application deployed using GitLab), a +project called **Monitoring** is created: -This project is used to self monitor your GitLab instance. The metrics dashboard -of the project shows some basic resource usage charts, such as CPU and memory usage -of each server in [Omnibus GitLab](https://docs.gitlab.com/omnibus/) installations. +- With [internal visibility](../../../public_access/public_access.md#internal-projects). +- Under a group called **GitLab Instance**. -You can also use the project to configure your own -[custom metrics](../../../operations/metrics/index.md#adding-custom-metrics) using -metrics exposed by the [GitLab exporter](../prometheus/gitlab_metrics.md#metrics-available). +The project is created specifically for visualizing and configuring the monitoring of your GitLab +instance. -## Creating the self monitoring project +When the project and group are created, all administrators are added as maintainers. As an +administrator, you can add new members to the group to give them the +[Maintainer role](../../../user/permissions.md) for the project. + +This project can be used to: + +- Self monitor your GitLab instance. The metrics dashboard of the project shows some basic resource + usage charts, such as CPU and memory usage of each server in + [Omnibus GitLab](https://docs.gitlab.com/omnibus/) installations. +- Also configure your own [custom metrics](../../../operations/metrics/index.md#adding-custom-metrics) + using metrics exposed by the [GitLab exporter](../prometheus/gitlab_metrics.md#metrics-available). + +## Create the self monitoring project 1. On the top bar, select **Menu >** **{admin}** **Admin**. 1. On the left sidebar, select **Settings > Metrics and profiling** and expand **Self monitoring**. -1. Toggle the **Create Project** button on. +1. Toggle **Self monitoring** on. 1. After your GitLab instance creates the project, GitLab displays a link to the - project in the text above the **Create Project** toggle. You can also find it + project in the text above the **Self monitoring** toggle. You can also find it from the top bar by selecting **Menu > Project**, then selecting **Your projects**. -## Deleting the self monitoring project +## Delete the self monitoring project WARNING: Deleting the self monitoring project removes any changes made to the project. If @@ -48,8 +49,8 @@ you create the project again, it's created in its default state. 1. On the top bar, select **Menu >** **{admin}** **Admin**. 1. On the left sidebar, go to **Settings > Metrics and profiling** and expand **Self monitoring**. -1. Toggle the **Create Project** button off. -1. In the confirmation dialog that opens, click **Delete project**. +1. Toggle **Self monitoring** off. +1. In the confirmation dialog that opens, click **Delete self monitoring project**. It can take a few seconds for it to be deleted. 1. After the project is deleted, GitLab displays a message confirming your action. @@ -66,27 +67,24 @@ The dashboard uses metrics available in You can also [create your own dashboards](../../../operations/metrics/dashboards/index.md). -## Connection to Prometheus +## Connect to Prometheus The project is automatically configured to connect to the -[internal Prometheus](../prometheus/index.md) instance if the Prometheus -instance is present (should be the case if GitLab was installed via Omnibus -and you haven't disabled it). +[internal Prometheus](../prometheus/index.md) instance if the Prometheus instance is present. +This should be the case if GitLab was installed using Omnibus GitLab and you haven't disabled it. -If that's not the case or if you have an external Prometheus instance or a customized setup, -you should -[configure it manually](../../../user/project/integrations/prometheus.md#manual-configuration-of-prometheus). +If that's not the case, or if you have an external Prometheus instance or a customized setup, +you [configure it manually](../../../user/project/integrations/prometheus.md#manual-configuration-of-prometheus). -## Taking action on Prometheus alerts **(ULTIMATE)** +## Take action on Prometheus alerts **(ULTIMATE)** You can [add a webhook](../../../operations/metrics/alerts.md#external-prometheus-instances) -to the Prometheus configuration for GitLab to receive notifications of any -alerts. +to the Prometheus configuration for GitLab to receive notifications of any alerts. Once the webhook is setup, you can [take action on incoming alerts](../../../operations/metrics/alerts.md#trigger-actions-from-alerts). -## Adding custom metrics to the self monitoring project +## Add custom metrics to the self monitoring project You can add custom metrics in the self monitoring project by: @@ -95,11 +93,10 @@ You can add custom metrics in the self monitoring project by: ## Troubleshooting -### Getting error message in logs: `Could not create instance administrators group. Errors: ["You don't have permission to create groups."]` +### Error message in logs: `Could not create instance administrators group. Errors: ["You don't have permission to create groups."]` -There is [a bug](https://gitlab.com/gitlab-org/gitlab/-/issues/208676) which causes -project creation to fail with the following error (which appears in the log file) -when the first administrator user is an +A [bug](https://gitlab.com/gitlab-org/gitlab/-/issues/208676) causes project creation to fail with +the following error in the log file when the first administrator user is an [external user](../../../user/permissions.md#external-users): ```plaintext diff --git a/doc/administration/monitoring/ip_whitelist.md b/doc/administration/monitoring/ip_whitelist.md index 20c97a0df8f..b6df176ea87 100644 --- a/doc/administration/monitoring/ip_whitelist.md +++ b/doc/administration/monitoring/ip_whitelist.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > Introduced in GitLab 9.4. NOTE: -We intend to [rename IP whitelist as `IP allowlist`](https://gitlab.com/gitlab-org/gitlab/-/issues/7554). +We intend to [rename IP whitelist as `IP allowlist`](https://gitlab.com/groups/gitlab-org/-/epics/3478). GitLab provides some [monitoring endpoints](../../user/admin_area/monitoring/health_check.md) that provide health check information when probed. diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md index 3b82b0e4bb8..0f40fd3c0e7 100644 --- a/doc/administration/monitoring/performance/grafana_configuration.md +++ b/doc/administration/monitoring/performance/grafana_configuration.md @@ -65,15 +65,41 @@ GitLab sidebar: 1. On the top bar, select **Menu >** **{admin}** **Admin**. 1. On the left sidebar, select **Settings > Metrics and profiling** and expand **Metrics - Grafana**. -1. Check the **Enable access to Grafana** checkbox. +1. Select the **Add a link to Grafana** checkbox. 1. Configure the **Grafana URL**: - *If Grafana is enabled through Omnibus GitLab and on the same server,* leave **Grafana URL** unchanged. It should be `/-/grafana`. - *Otherwise,* enter the full URL of the Grafana instance. -1. Click **Save changes**. +1. Select **Save changes**. GitLab displays your link in the **Menu > Admin > Monitoring > Metrics Dashboard**. +## Required Scopes + +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5822) in GitLab 13.10. + +When setting up Grafana through the process above, no scope shows in the screen at +**Menu >** **{admin}** **Admin > Applications > GitLab Grafana**. However, the `read_user` scope is +required and is provided to the application automatically. Setting any scope other than +`read_user` without also including `read_user` leads to this error when you try to log in using +GitLab as the OAuth provider: + +```plaintext +The requested scope is invalid, unknown, or malformed. +``` + +If you see this error, make sure that one of the following is true in the GitLab Grafana +configuration screen: + +- No scopes appear. +- The `read_user` scope is included. + +> Versions of GitLab prior 13.10 use the API scope instead of `read_user`. In versions of GitLab +> prior to 13.10, the API scope: +> +> - Is required to access Grafana through the GitLab OAuth provider. +> - Is set by enabling the Grafana application as shown in [Integration with GitLab UI](#integration-with-gitlab-ui). + ## Security Update Users running GitLab version 12.0 or later should immediately upgrade to one of the diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index 5a7e8e12a38..6d9418133d8 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -4,22 +4,37 @@ group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Performance Bar **(FREE SELF)** +# Performance bar **(FREE SELF)** -> The **Stats** field [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/271551) in GitLab 13.9. -> The **Memory** field [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/330736) in GitLab 14.0. +> - The **Stats** field [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/271551) in GitLab 13.9. +> - The **Memory** field [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/330736) in GitLab 14.0. -You can display the GitLab Performance Bar to see statistics for the performance -of a page. When activated, it looks as follows: +You can display the performance bar to see statistics for the performance of a GitLab UI page. +For example: - + -From left to right, it displays: +## Available information + +From left to right, the performance bar displays: - **Current Host**: the current host serving the page. - **Database queries**: the time taken (in milliseconds) and the total number of database queries, displayed in the format `00ms / 00 (00 cached) pg`. Click to display - a modal window with more details. + a modal window with more details. You can use this to see the following + details for each query: + - **In a transaction**: shows up below the query if it was executed in + the context of a transaction + - **Role**: shows up when [database load + balancing](../../database_load_balancing.md) is enabled. It shows + which server role was used for the query. "Primary" means that the query + was sent to the read/write primary server. "Replica" means it was sent + to a read-only replica. + - **Config name**: shows up only when the + `GITLAB_MULTIPLE_DATABASE_METRICS` environment variable is set. This is + used to distinguish between different databases configured for different + GitLab features. The name shown is the same name used to configure database + connections in GitLab. - **Gitaly calls**: the time taken (in milliseconds) and the total number of [Gitaly](../../gitaly/index.md) calls. Click to display a modal window with more details. @@ -57,8 +72,17 @@ From left to right, it displays: NOTE: Not all indicators are available in all environments. For instance, the memory view -requires to run Ruby with [specific patches](https://gitlab.com/gitlab-org/gitlab-build-images/-/blob/master/patches/ruby/2.7.2/thread-memory-allocations-2.7.patch) applied. -When running GitLab locally using the GDK this is typically not the case and the memory view cannot be used. +requires running Ruby with [specific patches](https://gitlab.com/gitlab-org/gitlab-build-images/-/blob/master/patches/ruby/2.7.2/thread-memory-allocations-2.7.patch) +applied. When running GitLab locally using [GDK](../../../development/contributing/index.md#gitlab-development-kit), +this is typically not the case and the memory view cannot be used. + +## Keyboard shortcut + +Press the [<kbd>p</kbd> + <kbd>b</kbd> keyboard shortcut](../../../user/shortcuts.md) to display +the performance bar, and again to hide it. + +For non-administrators to display the performance bar, it must be +[enabled for them](#enable-the-performance-bar-for-non-administrators). ## Request warnings @@ -74,23 +98,17 @@ appears next to requests with warnings.  -## Enable the Performance Bar via the Admin Area +## Enable the performance bar for non-administrators -The GitLab Performance Bar is disabled by default for non-administrators. To enable it +The performance bar is disabled by default for non-administrators. To enable it for a given group: -1. Sign in as a user with Administrator [permissions](../../../user/permissions.md). +1. Sign in as a user with Administrator [role](../../../user/permissions.md). 1. On the top bar, select **Menu >** **{admin}** **Admin**. 1. On the left sidebar, select **Settings > Metrics and profiling** (`admin/application_settings/metrics_and_profiling`), and expand **Profiling - Performance bar**. -1. Click **Enable access to the Performance Bar**. -1. In the **Allowed group** field, provide the full path of the group allowed - to access the GitLab Performance Bar. +1. Click **Allow non-administrators to access to the performance bar**. +1. In the **Allow access to members of the following group** field, provide the full path of the + group allowed to access the performance. 1. Click **Save changes**. - -## Keyboard shortcut for the Performance Bar - -After enabling the GitLab Performance Bar, press the [<kbd>p</kbd> + -<kbd>b</kbd> keyboard shortcut](../../../user/shortcuts.md) to display it, and -again to hide it. diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index 2aa95a2b0f1..459eb259498 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -8,10 +8,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w To enable the GitLab Prometheus metrics: -1. Log into GitLab as a user with [administrator permissions](../../../user/permissions.md). +1. Log in to GitLab as a user with Administrator [role](../../../user/permissions.md). 1. On the top bar, select **Menu >** **{admin}** **Admin**. 1. On the left sidebar, select **Settings > Metrics and profiling**. -1. Find the **Metrics - Prometheus** section, and click **Enable Prometheus Metrics**. +1. Find the **Metrics - Prometheus** section, and select **Add link to Prometheus**. 1. [Restart GitLab](../../restart_gitlab.md#omnibus-gitlab-restart) for the changes to take effect. For installations from source you must configure it yourself. @@ -45,6 +45,7 @@ The following metrics are available: | `gitlab_ci_pipeline_size_builds` | Histogram | 13.1 | Total number of builds within a pipeline grouped by a pipeline source | `source` | | `job_waiter_started_total` | Counter | 12.9 | Number of batches of jobs started where a web request is waiting for the jobs to complete | `worker` | | `job_waiter_timeouts_total` | Counter | 12.9 | Number of batches of jobs that timed out where a web request is waiting for the jobs to complete | `worker` | +| `gitlab_ci_active_jobs` | Histogram | 14.2 | Count of active jobs when pipeline is created | | | `gitlab_database_transaction_seconds` | Histogram | 12.1 | Time spent in database transactions, in seconds | | | `gitlab_method_call_duration_seconds` | Histogram | 10.2 | Method calls real duration | `controller`, `action`, `module`, `method` | | `gitlab_page_out_of_bounds` | Counter | 12.8 | Counter for the PageLimiter pagination limit being hit | `controller`, `action`, `bot` | @@ -152,15 +153,8 @@ The following metrics can be controlled by feature flags: ## Praefect metrics -You can [configure Praefect to report metrics](../../gitaly/praefect.md#praefect). -These are some of the Praefect metrics served from the `/metrics` path on the [configured port](index.md#changing-the-port-and-address-prometheus-listens-on) -(9652 by default). - -| Metric | Type | Since | Description | Labels | -| :----- | :--- | ----: | :---------- | :----- | -| `gitaly_praefect_replication_latency_bucket` | Histogram | 12.10 | The amount of time it takes for replication to complete once the replication job starts. | | -| `gitaly_praefect_replication_delay_bucket` | Histogram | 12.10 | A measure of how much time passes between when the replication job is created and when it starts. | | -| `gitaly_praefect_node_latency_bucket` | Histogram | 12.10 | The latency in Gitaly returning health check information to Praefect. This indicates Praefect connection saturation. | | +You can [configure Praefect](../../gitaly/praefect.md#praefect) to report metrics. For information +on available metrics, see the [relevant documentation](../../gitaly/index.md#monitor-gitaly-cluster). ## Sidekiq metrics diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index dd402f800e3..dd81f71d418 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -8,18 +8,19 @@ info: To determine the technical writer assigned to the Stage/Group associated w [Prometheus](https://prometheus.io) is a powerful time-series monitoring service, providing a flexible platform for monitoring GitLab and other software products. + GitLab provides out-of-the-box monitoring with Prometheus, providing easy access to high quality time-series monitoring of GitLab services. -> **Notes:** -> -> - Prometheus and the various exporters listed in this page are bundled in the -> Omnibus GitLab package. Check each exporter's documentation for the timeline -> they got added. For installations from source you must install them -> yourself. Over subsequent releases additional GitLab metrics are captured. -> - Prometheus services are on by default with GitLab 9.0. -> - Prometheus and its exporters don't authenticate users, and are available -> to anyone who can access them. +Prometheus and the various exporters listed in this page are bundled in the +Omnibus GitLab package. Check each exporter's documentation for the timeline +they got added. For installations from source you must install them +yourself. Over subsequent releases additional GitLab metrics are captured. + +Prometheus services are on by default. + +Prometheus and its exporters don't authenticate users, and are available to anyone who can access +them. ## Overview @@ -33,7 +34,7 @@ dashboard tool like [Grafana](https://grafana.com). For installations from source, you must install and configure it yourself. -Prometheus and its exporters are on by default, starting with GitLab 9.0. +Prometheus and its exporters are on by default. Prometheus runs as the `gitlab-prometheus` user and listen on `http://localhost:9090`. By default, Prometheus is only accessible from the GitLab server itself. Each exporter is automatically set up as a @@ -294,8 +295,8 @@ To use an external Prometheus server: You can visit `http://localhost:9090` for the dashboard that Prometheus offers by default. If SSL has been enabled on your GitLab instance, you may not be able to access -Prometheus on the same browser as GitLab if using the same FQDN due to [HSTS](https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security). We plan to -[provide access via GitLab](https://gitlab.com/gitlab-org/multi-user-prometheus), but in the interim there are +Prometheus on the same browser as GitLab if using the same FQDN due to [HSTS](https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security). +[A test project exists](https://gitlab.com/gitlab-org/multi-user-prometheus) to provide access via GitLab, but in the interim there are some workarounds: using a separate FQDN, using server IP, using a separate browser for Prometheus, resetting HSTS, or having [NGINX proxy it](https://docs.gitlab.com/omnibus/settings/nginx.html#inserting-custom-nginx-settings-into-the-gitlab-server-block). diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md index c4ff19ec3ea..0cab46a95c9 100644 --- a/doc/administration/nfs.md +++ b/doc/administration/nfs.md @@ -86,7 +86,7 @@ options: users. However, in this case only GitLab uses the NFS share so it is safe. GitLab recommends the `no_root_squash` setting because we need to manage file permissions automatically. Without the setting you may receive - errors when the Omnibus package tries to alter permissions. Note that GitLab + errors when the Omnibus package tries to alter permissions. GitLab and other bundled components do **not** run as `root` but as non-privileged users. The recommendation for `no_root_squash` is to allow the Omnibus package to set ownership and permissions on files, as needed. In some cases where the diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index 525b41359cf..b8a6f2acc56 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -241,7 +241,7 @@ The connection settings match those provided by [fog-aws](https://github.com/fog #### Oracle Cloud S3 connection settings -Note that Oracle Cloud S3 must be sure to use the following settings: +Oracle Cloud S3 must be sure to use the following settings: | Setting | Value | |---------------------------------|---------| @@ -365,7 +365,7 @@ settings are populated from the previous settings. azure_storage_access_key = "<AZURE STORAGE ACCESS KEY>" ``` -If you are using a custom Azure storage domain, note that +If you are using a custom Azure storage domain, `azure_storage_domain` does **not** have to be set in the Workhorse configuration. This information is exchanged in an API call between GitLab Rails and Workhorse. @@ -690,7 +690,7 @@ When configured either with an instance profile or with the consolidated object configuration, GitLab Workhorse properly uploads files to S3 buckets that have [SSE-S3 or SSE-KMS encryption enabled by default](https://docs.aws.amazon.com/kms/latest/developerguide/services-s3.html). -Note that customer master keys (CMKs) and SSE-C encryption are [not +Customer master keys (CMKs) and SSE-C encryption are [not supported since this requires sending the encryption keys in every request](https://gitlab.com/gitlab-org/gitlab/-/issues/226006). ##### Server-side encryption headers diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md index 1f195bcc378..2cc4e3a4551 100644 --- a/doc/administration/operations/extra_sidekiq_processes.md +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -95,14 +95,14 @@ To view the Sidekiq processes in GitLab: ## Negate settings -To have the additional Sidekiq processes work on every queue **except** the ones +To have the Sidekiq process work on every queue **except** the ones you list. In this example, we exclude all import-related jobs from a Sidekiq node: -1. After you follow the steps for [starting extra processes](#start-multiple-processes), - edit `/etc/gitlab/gitlab.rb` and add: +1. Edit `/etc/gitlab/gitlab.rb` and add: ```ruby sidekiq['negate'] = true + sidekiq['queue_selector'] = true sidekiq['queue_groups'] = [ "feature_category=importers" ] @@ -140,50 +140,6 @@ sidekiq['queue_groups'] = [ ] ``` -### Disable Sidekiq cluster - -WARNING: -Sidekiq cluster is [scheduled](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/240) -to be the only way to start Sidekiq in GitLab 14.0. - -By default, the Sidekiq service runs `sidekiq-cluster`. To disable this behavior, -add the following to the Sidekiq configuration: - -```ruby -sidekiq['enable'] = true -sidekiq['cluster'] = false -``` - -All of the aforementioned configuration options for `sidekiq` -are available. By default, they are configured as follows: - -```ruby -sidekiq['queue_selector'] = false -sidekiq['interval'] = nil -sidekiq['max_concurrency'] = 50 -sidekiq['min_concurrency'] = nil -sidekiq['negate'] = false -sidekiq['queue_groups'] = ['*'] -sidekiq['shutdown_timeout'] = 25 -``` - -`sidekiq_cluster` must be disabled if you decide to configure the -cluster as above. - -When disabling `sidekiq_cluster`, you must copy your configuration for -`sidekiq_cluster`over to `sidekiq`. Anything configured for -`sidekiq_cluster` is overridden by the options for `sidekiq` when -setting `sidekiq['cluster'] = true`. - -When using this feature, the service called `sidekiq` is now -running `sidekiq-cluster`. - -The [concurrency](#manage-concurrency) and other options configured -for Sidekiq are respected. - -By default, logs for `sidekiq-cluster` go to `/var/log/gitlab/sidekiq` -like regular Sidekiq logs. - ## Ignore all import queues When [importing from GitHub](../../user/project/import/github.md) or @@ -264,7 +220,7 @@ being equal to `max_concurrency`. Running a single Sidekiq process is the default in GitLab 12.10 and earlier. WARNING: -Running Sidekiq directly is scheduled to be removed in GitLab +Running Sidekiq directly was removed in GitLab [14.0](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/240). 1. Edit `/etc/gitlab/gitlab.rb` and add: diff --git a/doc/administration/operations/extra_sidekiq_routing.md b/doc/administration/operations/extra_sidekiq_routing.md index 80540b7ba46..6938f8a7012 100644 --- a/doc/administration/operations/extra_sidekiq_routing.md +++ b/doc/administration/operations/extra_sidekiq_routing.md @@ -103,9 +103,11 @@ based on a subset of worker attributes: - `worker_name` - the worker name. The other attributes are typically more useful as they are more general, but this is available in case a particular worker needs to be selected. -- `name` - the queue name. The other attributes are typically more useful as - they are more general, but this is available in case a particular queue needs - to be selected. +- `name` - the queue name generated from the worker name. The other attributes + are typically more useful as they are more general, but this is available in + case a particular queue needs to be selected. Because this is generated from + the worker name, it does not change based on the result of other routing + rules. - `resource_boundary` - if the queue is bound by `cpu`, `memory`, or `unknown`. For example, the `ProjectExportWorker` is memory bound as it has to load data in memory before saving it for export. diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index bb0756cf948..5c1271486c0 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -48,7 +48,7 @@ lookups via database lookup. As part of [setting up Geo](../geo/index.md#setup-instructions), you are required to follow the steps outlined below for both the primary and -secondary nodes, but note that the `Write to "authorized keys" file` checkbox +secondary nodes, but the `Write to "authorized keys" file` checkbox only needs to be unchecked on the primary node since it is reflected automatically on the secondary if database replication is working. diff --git a/doc/administration/operations/moving_repositories.md b/doc/administration/operations/moving_repositories.md index fba20da9aea..765cf64e735 100644 --- a/doc/administration/operations/moving_repositories.md +++ b/doc/administration/operations/moving_repositories.md @@ -7,8 +7,7 @@ type: reference # Moving repositories managed by GitLab **(FREE SELF)** -Sometimes you need to move all repositories managed by GitLab to -another file system or another server. +You can move all repositories managed by GitLab to another file system or another server. ## Moving data within a GitLab instance @@ -28,7 +27,128 @@ For more information, see: querying and scheduling snippet repository moves. - [The API documentation](../../api/group_repository_storage_moves.md) details the endpoints for querying and scheduling group repository moves **(PREMIUM SELF)**. -- [Migrate to Gitaly Cluster](../gitaly/praefect.md#migrate-to-gitaly-cluster). +- [Migrate to Gitaly Cluster](../gitaly/index.md#migrate-to-gitaly-cluster). + +### Move Repositories + +GitLab repositories can be associated with projects, groups, and snippets. Each of these types +have a separate API to schedule the respective repositories to move. To move all repositories +on a GitLab instance, each of these types must be scheduled to move for each storage. + +WARNING: +To move repositories into a [Gitaly Cluster](../gitaly/index.md#gitaly-cluster) in GitLab versions +13.12 to 14.1, you must [enable the `gitaly_replicate_repository_direct_fetch` feature flag](../feature_flags.md). + +Each repository is made read-only for the duration of the move. The repository is not writable +until the move has completed. + +To move repositories: + +1. Ensure all storages are accessible to the GitLab instance. In this example, these are + `<original_storage_name>` and `<cluster_storage_name>`. +1. [Configure repository storage weights](../repository_storage_paths.md#configure-where-new-repositories-are-stored) + so that the new storages receives all new projects. This stops new projects from being created + on existing storages while the migration is in progress. +1. Schedule repository moves for: + - [Projects](#bulk-schedule-project-moves). + - [Snippets](#bulk-schedule-snippet-moves). + - [Groups](#bulk-schedule-group-moves). **(PREMIUM SELF)** + +### Bulk schedule project moves + +Use the API to schedule project moves: + +1. [Schedule repository storage moves for all projects on a storage shard](../../api/project_repository_storage_moves.md#schedule-repository-storage-moves-for-all-projects-on-a-storage-shard) + using the API. For example: + + ```shell + curl --request POST --header "Private-Token: <your_access_token>" \ + --header "Content-Type: application/json" \ + --data '{"source_storage_name":"<original_storage_name>","destination_storage_name":"<cluster_storage_name>"}' \ + "https://gitlab.example.com/api/v4/project_repository_storage_moves" + ``` + +1. [Query the most recent repository moves](../../api/project_repository_storage_moves.md#retrieve-all-project-repository-storage-moves) + using the API. The response indicates either: + - The moves have completed successfully. The `state` field is `finished`. + - The moves are in progress. Re-query the repository move until it completes successfully. + - The moves have failed. Most failures are temporary and are solved by rescheduling the move. + +1. After the moves are complete, use the API to [query projects](../../api/projects.md#list-all-projects) and confirm that all projects have moved. None of the projects should be returned with the + `repository_storage` field set to the old storage. For example: + + ```shell + curl --header "Private-Token: <your_access_token>" --header "Content-Type: application/json" \ + "https://gitlab.example.com/api/v4/projects?repository_storage=<original_storage_name>" + ``` + + Alternatively use [the rails console](../operations/rails_console.md) to confirm that all + projects have moved. Run the following in the rails console: + + ```ruby + ProjectRepository.for_repository_storage('<original_storage_name>') + ``` + +1. Repeat for each storage as required. + +### Bulk schedule snippet moves + +Use the API to schedule snippet moves: + +1. [Schedule repository storage moves for all snippets on a storage shard](../../api/snippet_repository_storage_moves.md#schedule-repository-storage-moves-for-all-snippets-on-a-storage-shard). For example: + + ```shell + curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --header "Content-Type: application/json" \ + --data '{"source_storage_name":"<original_storage_name>","destination_storage_name":"<cluster_storage_name>"}' \ + "https://gitlab.example.com/api/v4/snippet_repository_storage_moves" + ``` + +1. [Query the most recent repository moves](../../api/snippet_repository_storage_moves.md#retrieve-all-snippet-repository-storage-moves) +The response indicates either: + - The moves have completed successfully. The `state` field is `finished`. + - The moves are in progress. Re-query the repository move until it completes successfully. + - The moves have failed. Most failures are temporary and are solved by rescheduling the move. + +1. After the moves are complete, use [the rails console](../operations/rails_console.md) to confirm + that all snippets have moved. No snippets should be returned for the original storage. Run the + following in the rails console: + + ```ruby + SnippetRepository.for_repository_storage('<original_storage_name>') + ``` + +1. Repeat for each storage as required. + +### Bulk schedule group moves **(PREMIUM SELF)** + +Use the API to schedule group moves: + +1. [Schedule repository storage moves for all groups on a storage shard](../../api/group_repository_storage_moves.md#schedule-repository-storage-moves-for-all-groups-on-a-storage-shard) +. For example: + + ```shell + curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --header "Content-Type: application/json" \ + --data '{"source_storage_name":"<original_storage_name>","destination_storage_name":"<cluster_storage_name>"}' \ + "https://gitlab.example.com/api/v4/group_repository_storage_moves" + ``` + +1. [Query the most recent repository moves](../../api/group_repository_storage_moves.md#retrieve-all-group-repository-storage-moves) +. The response indicates either: + - The moves have completed successfully. The `state` field is `finished`. + - The moves are in progress. Re-query the repository move until it completes successfully. + - The moves have failed. Most failures are temporary and are solved by rescheduling the move. + +1. After the moves are complete, use [the rails console](../operations/rails_console.md) to confirm + that all groups have moved. No groups should be returned for the original storage. Run the + following in the rails console: + + ```ruby + GroupWikiRepository.for_repository_storage('<original_storage_name>') + ``` + +1. Repeat for each storage as required. ## Migrating to another GitLab instance diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index 74483b65c4d..eb118709f94 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -580,6 +580,54 @@ However, this behavior is undesirable for registries used by internal hosts that 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. +#### Encrypted S3 buckets + +You can use server-side encryption with AWS KMS for S3 buckets that have +[SSE-S3 or SSE-KMS encryption enabled by default](https://docs.aws.amazon.com/kms/latest/developerguide/services-s3.html). +Customer master keys (CMKs) and SSE-C encryption aren't supported since this requires sending the +encryption keys in every request. + +For SSE-S3, you must enable the `encrypt` option in the registry settings. How you do this depends +on how you installed GitLab. Follow the instructions here that match your installation method. + +For Omnibus GitLab installations: + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + registry['storage'] = { + 's3' => { + 'accesskey' => 's3-access-key', + 'secretkey' => 's3-secret-key-for-access-key', + 'bucket' => 'your-s3-bucket', + 'region' => 'your-s3-region', + 'regionendpoint' => 'your-s3-regionendpoint', + 'encrypt' => true + } + } + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect. + +For installations from source: + +1. Edit your registry configuration YML file: + + ```yaml + storage: + s3: + accesskey: 'AKIAKIAKI' + secretkey: 'secret123' + bucket: 'gitlab-registry-bucket-AKIAKIAKI' + region: 'your-s3-region' + regionendpoint: 'your-s3-regionendpoint' + encrypt: true + ``` + +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) + for the changes to take effect. + ### Storage limitations Currently, there is no storage limitation, which means a user can upload an @@ -617,8 +665,7 @@ In the examples below we set the Registry's port to `5001`. ## Disable Container Registry per project If Registry is enabled in your GitLab instance, but you don't need it for your -project, you can disable it from your project's settings. Read the user guide -on how to achieve that. +project, you can [disable it from your project's settings](../../user/project/settings/index.md#sharing-and-permissions). ## Use an external container registry with GitLab as an auth endpoint @@ -777,6 +824,13 @@ notifications: ## Run the Cleanup policy now +WARNING: +If you're using a distributed architecture and Sidekiq is running on a different node, the cleanup +policies don't work. To fix this, you must configure the `gitlab.rb` file on the Sidekiq nodes to +point to the correct registry URL and copy the `registry.key` file to each Sidekiq node. For more +information, see the [Sidekiq configuration](../sidekiq.md) +page. + To reduce the amount of [Container Registry disk space used by a given project](../troubleshooting/gitlab_rails_cheat_sheet.md#registry-disk-space-usage-by-project), administrators can clean up image tags and [run garbage collection](#container-registry-garbage-collection). @@ -892,26 +946,6 @@ understand the implications. WARNING: This is a destructive operation. -When you run `registry-garbage-collect` with the -m flag, garbage collection unlinks manifests that -are part of a multi-arch manifest, unless they're tagged in the same repository. -See [this issue](https://gitlab.com/gitlab-org/container-registry/-/issues/149) for details. - -To work around this issue, instead of: - -```plaintext -myrepo/multiarchmanifest:latest -myrepo/manifest/amd-64:latest -myrepo/manifest/arm:latest -``` - -Use: - -```plaintext -myrepo/multiarchmanifest:latest -myrepo/manifest:amd-64-latest -myrepo/manifest:arm-latest -``` - The GitLab Container Registry follows the same default workflow as Docker Distribution: retain untagged manifests and all layers, even ones that are not referenced directly. All content can be accessed by using context addressable identifiers. diff --git a/doc/administration/pages/img/lets_encrypt_integration_v12_1.png b/doc/administration/pages/img/lets_encrypt_integration_v12_1.png Binary files differdeleted file mode 100644 index 0f3ca25ce55..00000000000 --- a/doc/administration/pages/img/lets_encrypt_integration_v12_1.png +++ /dev/null diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index ea1e99524b8..5aeb3eaef7f 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -41,7 +41,7 @@ which you can set it up: the Pages daemon is installed, so you must share it through the network. - Run the Pages daemon in the same server as GitLab, listening on the same IP but on different ports. In that case, you must proxy the traffic with - a load balancer. If you choose that route note that you should use TCP load + a load balancer. If you choose that route, you should use TCP load balancing for HTTPS. If you use TLS-termination (HTTPS-load balancing), the pages can't be served with user-provided certificates. For HTTP it's OK to use HTTP or TCP load balancing. @@ -407,17 +407,15 @@ verification requirement: allows users to add Let's Encrypt SSL certificates for GitLab Pages sites served under a custom domain. -To enable it, you must: +To enable it: 1. Choose an email address on which you want to receive notifications about expiring domains. 1. On the top bar, select **Menu >** **{admin}** **Admin**. 1. On the left sidebar, select **Settings > Preferences**. 1. Expand **Pages**. -1. Enter the email address for receiving notifications and accept Let's Encrypt's Terms of Service as shown below. +1. Enter the email address for receiving notifications and accept Let's Encrypt's Terms of Service. 1. Select **Save changes**. - - ### Access control > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/33422) in GitLab 11.5. @@ -466,6 +464,7 @@ The scope to use for authentication must match the GitLab Pages OAuth applicatio pre-existing applications must modify the GitLab Pages OAuth application. Follow these steps to do this: +1. Enable [access control](#access-control). 1. On the top bar, select **Menu >** **{admin}** **Admin**. 1. On the left sidebar, select **Settings > Applications**. 1. Expand **GitLab Pages**. @@ -473,7 +472,7 @@ this: `read_api`). 1. Select **Save changes**. -#### Disabling public access to all Pages websites +#### Disable public access to all Pages sites > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32095) in GitLab 12.7. @@ -662,6 +661,16 @@ Follow the steps below to configure the proxy listener of GitLab Pages. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +## Set global maximum pages size per project **(FREE SELF)** + +To set the global maximum pages size for a project: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > Preferences**. +1. Expand **Pages**. +1. Edit the **Maximum size of pages**. +1. Select **Save changes**. + ## Override maximum pages size per project or group **(PREMIUM SELF)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16610) in GitLab 12.7. diff --git a/doc/administration/pages/source.md b/doc/administration/pages/source.md index 4aaf430db97..b5c3330b2ce 100644 --- a/doc/administration/pages/source.md +++ b/doc/administration/pages/source.md @@ -41,7 +41,7 @@ which you can set it up: the Pages daemon is installed, so you must share it through the network. 1. Run the Pages daemon in the same server as GitLab, listening on the same IP but on different ports. In that case, you must proxy the traffic with - a load balancer. If you choose that route, note that you should use TCP load + a load balancer. If you choose that route, you should use TCP load balancing for HTTPS. If you use TLS-termination (HTTPS-load balancing), the pages aren't able to be served with user-provided certificates. For HTTP, it's OK to use HTTP or TCP load balancing. @@ -127,17 +127,43 @@ The Pages daemon doesn't listen to the outside world. # path: shared/pages host: example.io - port: 80 + access_control: false + port: 8090 https: false + artifacts_server: false + external_http: ["127.0.0.1:8090"] + secret_file: /home/git/gitlab/gitlab-pages/gitlab-pages-secret + ``` + +1. Add the following configuration file to + `/home/git/gitlab/gitlab-pages/gitlab-pages.conf`, and be sure to change + `example.io` to the FQDN from which you want to serve GitLab Pages and + `gitlab.example.com` to the URL of your GitLab instance: + + ```ini + listen-http=:8090 + pages-root=/home/git/gitlab/shared/pages + api-secret-key=/home/git/gitlab/gitlab-pages-secret + pages-domain=example.io + internal-gitlab-server=https://gitlab.example.com + ``` + + You may use an `http` address, when running GitLab Pages and GitLab on the + same host. If you use `https` and use a self-signed certificate, be sure to + make your custom CA available to GitLab Pages, for example by setting the + `SSL_CERT_DIR` environment variable. + +1. Add the secret API key: + + ```shell + sudo -u git -H openssl rand -base64 32 > /home/git/gitlab/gitlab-pages-secret ``` 1. Edit `/etc/default/gitlab` and set `gitlab_pages_enabled` to `true` in - order to enable the pages daemon. In `gitlab_pages_options` the - `-pages-domain` must match the `host` setting that you set above. + order to enable the pages daemon: ```ini gitlab_pages_enabled=true - gitlab_pages_options="-pages-domain example.io -pages-root $app_root/shared/pages -listen-proxy 127.0.0.1:8090" ``` 1. Copy the `gitlab-pages` NGINX configuration file: diff --git a/doc/administration/polling.md b/doc/administration/polling.md index ec5d6cd45d8..5c4ee837057 100644 --- a/doc/administration/polling.md +++ b/doc/administration/polling.md @@ -4,29 +4,31 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Polling configuration **(FREE SELF)** +# Polling interval multiplier **(FREE SELF)** -The GitLab UI polls for updates for different resources (issue notes, issue -titles, pipeline statuses, and so on) on a schedule appropriate to the resource. +The GitLab UI polls for updates for different resources (issue notes, issue titles, pipeline +statuses, and so on) on a schedule appropriate to the resource. -To configure the polling interval multiplier: +Adjust the multiplier on these schedules to adjust how frequently the GitLab UI polls for updates. If +you set the multiplier to: + +- A value greater than `1`, UI polling slows down. If you see issues with database load from lots of + clients polling for updates, increasing the multiplier can be a good alternative to disabling + polling completely. For example, if you set the value to `2`, all polling intervals + are multiplied by 2, which means that polling happens half as frequently. +- A value between `0` and `1`, the UI polls more frequently so updates occur more frequently. + **Not recommended**. +- `0`, all polling is disabled. On the next poll, clients stop polling for updates. + +The default value (`1`) is recommended for the majority of GitLab installations. + +## Configure + +To adjust the polling interval multiplier: 1. On the top bar, select **Menu >** **{admin}** **Admin**. 1. On the left sidebar, select **Settings > Preferences**. -1. Expand **Real-time features**. -1. Set a value for the polling interval multiplier. This multiplier is applied - to all resources at once, and decimal values are supported: - - - `1.0` is the default, and recommended for most installations. - - `0` disables UI polling completely. On the next poll, clients stop - polling for updates. - - A value greater than `1` slows polling down. If you see issues with - database load from lots of clients polling for updates, increasing the - multiplier from 1 can be a good compromise, rather than disabling polling - completely. For example, if you set the value to `2`, all polling intervals - are multiplied by 2, which means that polling happens half as frequently. - - A value between `0` and `1` makes the UI poll more frequently (so updates - show in other sessions faster), but is **not recommended**. `1` should be - fast enough. - +1. Expand **Polling interval multiplier**. +1. Set a value for the polling interval multiplier. This multiplier is applied to all resources at + once. 1. Select **Save changes**. diff --git a/doc/administration/postgresql/pgbouncer.md b/doc/administration/postgresql/pgbouncer.md index 4f9056b9b50..7171e90949e 100644 --- a/doc/administration/postgresql/pgbouncer.md +++ b/doc/administration/postgresql/pgbouncer.md @@ -200,6 +200,45 @@ Once you've performed the tasks or procedure, switch back to using PgBouncer: sudo gitlab-ctl reconfigure ``` +## Fine tuning + +PgBouncer's default settings suit the majority of installations. +In specific cases you may want to change the performance-specific and resource-specific variables to either increase possible +throughput or to limit resource utilization that could cause memory exhaustion on the database. + +You can find the parameters and respective documentation on the [official PgBouncer documentation](https://www.pgbouncer.org/config.html). +Listed below are the most relevant ones and their defaults on an Omnibus GitLab installation: + +- `pgbouncer['max_client_conn']` (default: `2048`, depends on server file descriptor limits) + This is the "frontend" pool in PgBouncer: connections from Rails to PgBouncer. +- `pgbouncer['default_pool_size']` (default: `100`) + This is the "backend" pool in PgBouncer: connections from PgBouncer to the database. + +The ideal number for `default_pool_size` must be enough to handle all provisioned services that need to access +the database. Each of the listed services below use the following formula to define database pool size: + +- `puma` : `max_threads + headroom` (default `14`) + - `max_threads` is configured via: `gitlab['puma']['max_threads']` (default: `4`) + - `headroom` can be configured via `DB_POOL_HEADROOM` env variable (default to `10`) +- `sidekiq` : `max_concurrency + 1 + headroom` (default: `61`) + - `max_concurrency` is configured via: `sidekiq['max_concurrency']` (default: `50`) + - `headroom` can be configured via `DB_POOL_HEADROOM` env variable (default to `10`) +- `geo-logcursor`: `1+headroom` (default: `11`) + - `headroom` can be configured via `DB_POOL_HEADROOM` env variable (default to `10`) + +To calculate the `default_pool_size`, multiply the number of instances of `puma`, `sidekiq` and `geo-logcursor` by the +number of connections each can consume as per listed above. The total will be the suggested `default_pool_size`. + +If you are using more than one PgBouncer with an internal Load Balancer, you may be able to divide the +`default_pool_size` by the number of instances to guarantee an evenly distributed load between them. + +The `pgbouncer['max_client_conn']` is the hard-limit of connections PgBouncer can accept. It's unlikely you will need +to change this. If you are hitting that limit, you may want to consider adding additional PgBouncers with an internal +Load Balancer. + +When setting up the limits for a PgBouncer that points to the Geo Tracking Database, +you can likely ignore `puma` from the equation, as it is only accessing that database sporadically. + ## Troubleshooting In case you are experiencing any issues connecting through PgBouncer, the first diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index d1dd233f08b..1308697c16e 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -57,7 +57,7 @@ consists of: ### Consul server node -The Consul server node runs the Consul server service. These nodes must have reached the quorum and elected a leader _before_ Patroni cluster bootstrap otherwise database nodes will wait until such Consul leader is elected. +The Consul server node runs the Consul server service. These nodes must have reached the quorum and elected a leader _before_ Patroni cluster bootstrap otherwise database nodes wait until such Consul leader is elected. ### PgBouncer node @@ -67,7 +67,7 @@ Each PgBouncer node runs two services: `Consul` agent - Watches the status of the PostgreSQL service definition on the Consul cluster. If that status changes, Consul runs a script which updates the -PgBouncer configuration to point to the new PostgreSQL master node and reloads +PgBouncer configuration to point to the new PostgreSQL leader node and reloads the PgBouncer service. ### Connection flow @@ -84,7 +84,7 @@ Each service in the package comes with a set of [default ports](https://docs.git ### Required information -Before proceeding with configuration, you will need to collect all the necessary +Before proceeding with configuration, you need to collect all the necessary information. #### Network information @@ -93,12 +93,12 @@ PostgreSQL doesn't listen on any network interface by default. It needs to know which IP address to listen on to be accessible to other services. Similarly, PostgreSQL access is controlled based on the network source. -This is why you will need: +This is why you need: -- IP address of each nodes network interface. This can be set to `0.0.0.0` to +- The IP address of each node's network interface. This can be set to `0.0.0.0` to listen on all interfaces. It cannot be set to the loopback address `127.0.0.1`. - Network Address. This can be in subnet (that is, `192.168.0.0/255.255.255.0`) - or CIDR (that is, `192.168.0.0/24`) form. + or Classless Inter-Domain Routing (CIDR) (`192.168.0.0/24`) form. #### Consul information @@ -141,12 +141,12 @@ patroni['postgresql']['max_replication_slots'] = 6 patroni['postgresql']['max_wal_senders'] = 7 ``` -As previously mentioned, you'll have to prepare the network subnets that will -be allowed to authenticate with the database. -You'll also need to supply the IP addresses or DNS records of Consul -server nodes. +As previously mentioned, prepare the network subnets that need permission +to authenticate with the database. +You also need to have the IP addresses or DNS records of Consul +server nodes on hand. -We will need the following password information for the application's database user: +You need the following password information for the application's database user: - `POSTGRESQL_USERNAME`. The default user for Omnibus GitLab is `gitlab` - `POSTGRESQL_USER_PASSWORD`. The password for the database user @@ -159,14 +159,14 @@ We will need the following password information for the application's database u #### Patroni information -We will need the following password information for the Patroni API: +You need the following password information for the Patroni API: - `PATRONI_API_USERNAME`. A username for basic auth to the API - `PATRONI_API_PASSWORD`. A password for basic auth to the API #### PgBouncer information -When using default setup, minimum configuration requires: +When using a default setup, the minimum configuration requires: - `PGBOUNCER_USERNAME`. The default user for Omnibus GitLab is `pgbouncer` - `PGBOUNCER_PASSWORD`. This is a password for PgBouncer service. @@ -179,11 +179,11 @@ When using default setup, minimum configuration requires: - `PGBOUNCER_NODE`, is the IP address or a FQDN of the node running PgBouncer. -Few notes on the service itself: +Few things to remember about the service itself: - The service runs as the same system account as the database - In the package, this is by default `gitlab-psql` -- If you use a non-default user account for PgBouncer service (by default `pgbouncer`), you will have to specify this username. We will refer to this requirement with `PGBOUNCER_USERNAME`. +- If you use a non-default user account for PgBouncer service (by default `pgbouncer`), you need to specify this username. - Passwords are stored in the following locations: - `/etc/gitlab/gitlab.rb`: hashed, and in plain text - `/var/opt/gitlab/pgbouncer/pg_auth`: hashed @@ -207,7 +207,7 @@ When installing the GitLab package, do not supply `EXTERNAL_URL` value. You must enable Patroni explicitly to be able to use it (with `patroni['enable'] = true`). Any PostgreSQL configuration item that controls replication, for example `wal_level`, `max_wal_senders`, etc, are strictly -controlled by Patroni and will override the original settings that you make with the `postgresql[...]` configuration key. +controlled by Patroni. These configurations override the original settings that you make with the `postgresql[...]` configuration key. Hence, they are all separated and placed under `patroni['postgresql'][...]`. This behavior is limited to replication. Patroni honours any other PostgreSQL configuration that was made with the `postgresql[...]` configuration key. For example, `max_wal_senders` by default is set to `5`. If you wish to change this you must set it with the `patroni['postgresql']['max_wal_senders']` @@ -257,6 +257,9 @@ patroni['postgresql']['max_replication_slots'] = X # available database connections. patroni['postgresql']['max_wal_senders'] = X+1 +# Replace XXX.XXX.XXX.XXX/YY with Network Addresses for your other patroni nodes +patroni['allowlist'] = %w(XXX.XXX.XXX.XXX/YY 127.0.0.1/32) + # Replace XXX.XXX.XXX.XXX/YY with Network Address postgresql['trust_auth_cidr_addresses'] = %w(XXX.XXX.XXX.XXX/YY 127.0.0.1/32) @@ -274,12 +277,12 @@ consul['configuration'] = { All database nodes use the same configuration. The leader node is not determined in configuration, and there is no additional or different configuration for either leader or replica nodes. -Once the configuration of a node is done, you must [reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) +After the configuration of a node is complete, you must [reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) on each node for the changes to take effect. Generally, when Consul cluster is ready, the first node that [reconfigures](../restart_gitlab.md#omnibus-gitlab-reconfigure) becomes the leader. You do not need to sequence the nodes reconfiguration. You can run them in parallel or in any order. -If you choose an arbitrary order you do not have any predetermined master. +If you choose an arbitrary order you do not have any predetermined leader. #### Enable Monitoring @@ -300,6 +303,63 @@ If you enable Monitoring, it must be enabled on **all** database servers. 1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. +#### Enable TLS support for the Patroni API + +By default, Patroni's [REST API](https://patroni.readthedocs.io/en/latest/rest_api.html#rest-api) is served over HTTP. +You have the option to enable TLS and use HTTPS over the same [port](https://docs.gitlab.com/omnibus/package-information/defaults.html#patroni). + +To enable TLS, you need PEM-formatted certificate and private key files. Both files must be readable by the PostgreSQL user (`gitlab-psql` by default, or the one set by `postgresql['username']`): + +```ruby +patroni['tls_certificate_file'] = '/path/to/server/certificate.pem' +patroni['tls_key_file'] = '/path/to/server/key.pem' +``` + +If the server's private key is encrypted, specify the password to decrypt it: + +```ruby +patroni['tls_key_password'] = 'private-key-password' # This is the plain-text password. +``` + +If you are using a self-signed certificate or an internal CA, you need to either disable the TLS verification or pass the certificate of the +internal CA, otherwise you may run into an unexpected error when using the `gitlab-ctl patroni ....` commands. Omnibus ensures that Patroni API +clients honor this configuration. + +TLS certificate verification is enabled by default. To disable it: + +```ruby +patroni['tls_verify'] = false +``` + +Alternatively, you can pass a PEM-formatted certificate of the internal CA. Again, the file must be readable by the PostgreSQL user: + +```ruby +patroni['tls_ca_file'] = '/path/to/ca.pem' +``` + +When TLS is enabled, mutual authentication of the API server and client is possible for all endpoints, the extent of which depends on +the `patroni['tls_client_mode']` attribute: + +- `none` (default): the API will not check for any client certificates. +- `optional`: client certificates are required for all [unsafe](https://patroni.readthedocs.io/en/latest/security.html#protecting-the-rest-api) API calls. +- `required`: client certificates are required for all API calls. + +The client certificates are verified against the CA certificate that is specified with the `patroni['tls_ca_file']` attribute. Therefore, +this attribute is required for mutual TLS authentication. You also need to specify PEM-formatted client certificate and private key files. +Both files must be readable by the PostgreSQL user: + +```ruby +patroni['tls_client_mode'] = 'required' +patroni['tls_ca_file'] = '/path/to/ca.pem' + +patroni['tls_client_certificate_file'] = '/path/to/client/certificate.pem' +patroni['tls_client_key_file'] = '/path/to/client/key.pem' +``` + +You can use different certificates and keys for both API server and client on different Patroni nodes as long as they can be verified. +However, the CA certificate (`patroni['tls_ca_file']`), TLS certificate verification (`patroni['tls_verify']`), and client TLS +authentication mode (`patroni['tls_client_mode']`), must each have the same value on all nodes. + ### Configuring the PgBouncer node 1. Make sure you collect [`CONSUL_SERVER_NODES`](#consul-information), [`CONSUL_PASSWORD_HASH`](#consul-information), and [`PGBOUNCER_PASSWORD_HASH`](#pgbouncer-information) before executing the next step. @@ -355,19 +415,19 @@ If you enable Monitoring, it must be enabled on **all** database servers. #### PgBouncer Checkpoint -1. Ensure each node is talking to the current master: +1. Ensure each node is talking to the current node leader: ```shell gitlab-ctl pgb-console # Supply PGBOUNCER_PASSWORD when prompted ``` If there is an error `psql: ERROR: Auth failed` after typing in the - password, ensure you previously generated the MD5 password hashes with the correct + password, ensure you have previously generated the MD5 password hashes with the correct format. The correct format is to concatenate the password and the username: `PASSWORDUSERNAME`. For example, `Sup3rS3cr3tpgbouncer` would be the text needed to generate an MD5 password hash for the `pgbouncer` user. -1. Once the console prompt is available, run the following queries: +1. After the console prompt has become available, run the following queries: ```shell show databases ; show clients ; @@ -390,7 +450,7 @@ If you enable Monitoring, it must be enabled on **all** database servers. #### Configure the internal load balancer -If you're running more than one PgBouncer node as recommended, then at this time you'll need to set up a TCP internal load balancer to serve each correctly. This can be done with any reputable TCP load balancer. +If you're running more than one PgBouncer node as recommended, then you need to set up a TCP internal load balancer to serve each correctly. This can be accomplished with any reputable TCP load balancer. As an example here's how you could do it with [HAProxy](https://www.haproxy.org/): @@ -450,8 +510,7 @@ Ensure that all migrations ran: gitlab-rake gitlab:db:configure ``` -> **Note**: If you encounter a `rake aborted!` error stating that PgBouncer is failing to connect to -PostgreSQL it may be that your PgBouncer node's IP address is missing from +> **Note**: If you encounter a `rake aborted!` error stating that PgBouncer is failing to connect to PostgreSQL it may be that your PgBouncer node's IP address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` on your database nodes. See [PgBouncer error `ERROR: pgbouncer cannot connect to server`](#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) in the Troubleshooting section before proceeding. @@ -497,7 +556,7 @@ Here is a list and description of each machine and the assigned IP: All passwords are set to `toomanysecrets`, please do not use this password or derived hashes and the `external_url` for GitLab is `http://gitlab.example.com`. -Please note that after the initial configuration, if a failover occurs, the PostgresSQL master will change to one of the available secondaries until it is failed back. +After the initial configuration, if a failover occurs, the PostgresSQL leader node changes to one of the available secondaries until it is failed back. #### Example recommended setup for Consul servers @@ -572,6 +631,7 @@ patroni['password'] = 'PATRONI_API_PASSWORD' patroni['postgresql']['max_replication_slots'] = 6 patroni['postgresql']['max_wal_senders'] = 7 +patroni['allowlist'] = = %w(10.6.0.0/16 127.0.0.1/32) postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/16 127.0.0.1/32) # Configure the Consul agent @@ -628,7 +688,7 @@ All passwords are set to `toomanysecrets`, please do not use this password or de The `external_url` for GitLab is `http://gitlab.example.com` -Please note that after the initial configuration, if a failover occurs, the PostgresSQL master will change to one of the available secondaries until it is failed back. +After the initial configuration, if a failover occurs, the PostgresSQL leader node changes to one of the available secondaries until it is failed back. #### Example minimal configuration for database servers @@ -664,6 +724,7 @@ patroni['password'] = 'PATRONI_API_PASSWORD' # available database connections. patroni['postgresql']['max_wal_senders'] = 7 +patroni['allowlist'] = = %w(10.6.0.0/16 127.0.0.1/32) postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/16 127.0.0.1/32) consul['configuration'] = { @@ -721,21 +782,14 @@ The manual steps for this configuration are the same as for the [example recomme NOTE: Using Patroni instead of Repmgr is supported for PostgreSQL 11 and required for PostgreSQL 12. Starting with GitLab 14.0, only PostgreSQL 12 is available and hence Patroni is mandatory to achieve failover and replication. -Patroni is an opinionated solution for PostgreSQL high-availability. It takes the control of PostgreSQL, overrides its -configuration and manages its lifecycle (start, stop, restart). Patroni is the only option for PostgreSQL 12 clustering and for cascading replication for Geo deployments. +Patroni is an opinionated solution for PostgreSQL high-availability. It takes the control of PostgreSQL, overrides its configuration, and manages its lifecycle (start, stop, restart). Patroni is the only option for PostgreSQL 12 clustering and for cascading replication for Geo deployments. -The [architecture](#example-recommended-setup-manual-steps) (that was mentioned above) does not change for Patroni. -You do not need any special consideration for Patroni while provisioning your database nodes. Patroni heavily relies on -Consul to store the state of the cluster and elect a leader. Any failure in Consul cluster and its leader election will -propagate to Patroni cluster as well. +The fundamental [architecture](#example-recommended-setup-manual-steps) (mentioned above) does not change for Patroni. +You do not need any special consideration for Patroni while provisioning your database nodes. Patroni heavily relies on Consul to store the state of the cluster and elect a leader. Any failure in Consul cluster and its leader election propagates to the Patroni cluster as well. -Patroni monitors the cluster and handles failover. When the primary node fails it works with Consul to notify PgBouncer. On failure, Patroni handles the transitioning of the old primary to a replica and rejoins it to the cluster automatically. +Patroni monitors the cluster and handles any failover. When the primary node fails it works with Consul to notify PgBouncer. On failure, Patroni handles the transitioning of the old primary to a replica and rejoins it to the cluster automatically. -With Patroni the connection flow is slightly different. Patroni on each node connects to Consul agent to join the -cluster. Only after this point it decides if the node is the primary or a replica. Based on this decision, it configures -and starts PostgreSQL which it communicates with directly over a Unix socket. This implies that if Consul cluster is not -functional or does not have a leader, Patroni and by extension PostgreSQL will not start. Patroni also exposes a REST -API which can be accessed via its [default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#patroni) +With Patroni, the connection flow is slightly different. Patroni on each node connects to Consul agent to join the cluster. Only after this point it decides if the node is the primary or a replica. Based on this decision, it configures and starts PostgreSQL which it communicates with directly over a Unix socket. This means that if the Consul cluster is not functional or does not have a leader, Patroni and by extension PostgreSQL does not start. Patroni also exposes a REST API which can be accessed via its [default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#patroni) on each node. ### Check replication status @@ -758,7 +812,7 @@ To verify the status of replication: echo 'select * from pg_stat_wal_receiver\x\g\x \n select * from pg_stat_replication\x\g\x' | gitlab-psql ``` -The same command can be run on all three database servers, and will return any information +The same command can be run on all three database servers. It returns any information about replication available depending on the role the server is performing. The leader should return one record per replica: @@ -858,7 +912,7 @@ patroni['remove_data_directory_on_rewind_failure'] = false patroni['remove_data_directory_on_diverged_timelines'] = false ``` -[The upstream documentation will always be more up to date](https://patroni.readthedocs.io/en/latest/SETTINGS.html#postgresql), but the table below should provide a minimal overview of functionality. +[The upstream documentation is always more up to date](https://patroni.readthedocs.io/en/latest/SETTINGS.html#postgresql), but the table below should provide a minimal overview of functionality. |Setting|Overview| |-|-| @@ -868,11 +922,11 @@ patroni['remove_data_directory_on_diverged_timelines'] = false ### Database authorization for Patroni -Patroni uses Unix socket to manage PostgreSQL instance. Therefore, the connection from the `local` socket must be trusted. +Patroni uses a Unix socket to manage the PostgreSQL instance. Therefore, a connection from the `local` socket must be trusted. Also, replicas use the replication user (`gitlab_replicator` by default) to communicate with the leader. For this user, you can choose between `trust` and `md5` authentication. If you set `postgresql['sql_replication_password']`, -Patroni will use `md5` authentication, otherwise it falls back to `trust`. You must to specify the cluster CIDR in +Patroni uses `md5` authentication, and otherwise falls back to `trust`. You must to specify the cluster CIDR in `postgresql['md5_auth_cidr_addresses']` or `postgresql['trust_auth_cidr_addresses']` respectively. ### Interacting with Patroni cluster @@ -881,18 +935,14 @@ You can use `gitlab-ctl patroni members` to check the status of the cluster memb `gitlab-ctl patroni` provides two additional sub-commands, `check-leader` and `check-replica` which indicate if a node is the primary or a replica. -When Patroni is enabled, you don't have direct control over `postgresql` service. Patroni will signal PostgreSQL's startup, -shutdown, and restart. For example, for shutting down PostgreSQL on a node, you must shutdown Patroni on the same node -with: +When Patroni is enabled, it exclusively controls PostgreSQL's startup, +shutdown, and restart. This means, to shut down PostgreSQL on a certain node you must shutdown Patroni on the same node with: ```shell sudo gitlab-ctl stop patroni ``` -Note that stopping or restarting Patroni service on the leader node will trigger the automatic failover. If you -want to signal Patroni to reload its configuration or restart PostgreSQL process without triggering the failover, you -must use the `reload` or `restart` sub-commands of `gitlab-ctl patroni` instead. These two sub-commands are wrappers of -the same `patronictl` commands. +Stopping or restarting the Patroni service on the leader node triggers an automatic failover. If you need Patroni to reload its configuration or restart the PostgreSQL process without triggering the failover, you must use the `reload` or `restart` sub-commands of `gitlab-ctl patroni` instead. These two sub-commands are wrappers of the same `patronictl` commands. ### Manual failover procedure for Patroni @@ -918,17 +968,17 @@ For further details on this subject, see the #### Geo secondary site considerations -When a Geo secondary site is replicating from a primary site that uses `Patroni` and `PgBouncer`, [replicating through PgBouncer is not supported](https://github.com/pgbouncer/pgbouncer/issues/382#issuecomment-517911529) and the secondary must replicate directly from the leader node in the `Patroni` cluster. Therefore, when there is an automatic or manual failover in the `Patroni` cluster, you will need to manually re-point your secondary site to replicate from the new leader with: +When a Geo secondary site is replicating from a primary site that uses `Patroni` and `PgBouncer`, [replicating through PgBouncer is not supported](https://github.com/pgbouncer/pgbouncer/issues/382#issuecomment-517911529). The secondary *must* replicate directly from the leader node in the `Patroni` cluster. When there is an automatic or manual failover in the `Patroni` cluster, you can manually re-point your secondary site to replicate from the new leader with: ```shell sudo gitlab-ctl replicate-geo-database --host=<new_leader_ip> --replication-slot=<slot_name> ``` -Otherwise, the replication will not happen anymore, even if the original node gets re-added as a follower node. This will re-sync your secondary site database and may take a long time depending on the amount of data to sync. You may also need to run `gitlab-ctl reconfigure` if replication is still not working after re-syncing. +Otherwise, the replication will not happen, even if the original node gets re-added as a follower node. This re-syncs your secondary site database and may take a long time depending on the amount of data to sync. You may also need to run `gitlab-ctl reconfigure` if replication is still not working after re-syncing. ### Recovering the Patroni cluster -To recover the old primary and rejoin it to the cluster as a replica, you can simply start Patroni with: +To recover the old primary and rejoin it to the cluster as a replica, you can start Patroni with: ```shell sudo gitlab-ctl start patroni @@ -938,14 +988,13 @@ No further configuration or intervention is needed. ### Maintenance procedure for Patroni -With Patroni enabled, you can run a planned maintenance. If you want to do some maintenance work on one node and you -don't want Patroni to manage it, you can use put it into maintenance mode: +With Patroni enabled, you can run planned maintenance on your nodes. To perform maintenance on one node without Patroni, you can put it into maintenance mode with: ```shell sudo gitlab-ctl patroni pause ``` -When Patroni runs in a paused mode, it does not change the state of PostgreSQL. Once you are done you can resume Patroni: +When Patroni runs in a paused mode, it does not change the state of PostgreSQL. After you are done, you can resume Patroni: ```shell sudo gitlab-ctl patroni resume @@ -956,9 +1005,7 @@ For further details, see [Patroni documentation on this subject](https://patroni ### Switching from repmgr to Patroni WARNING: -Although switching from repmgr to Patroni is fairly straightforward the other way around is not. Rolling back from -Patroni to repmgr can be complicated and may involve deletion of data directory. If you need to do that, please contact -GitLab support. +Switching from repmgr to Patroni is straightforward, the other way around is *not*. Rolling back from Patroni to repmgr can be complicated and may involve deletion of data directory. If you need to do that, please contact GitLab support. You can switch an exiting database cluster to use Patroni instead of repmgr with the following steps: @@ -989,7 +1036,7 @@ You can switch an exiting database cluster to use Patroni instead of repmgr with 1. Repeat the last two steps for all replica nodes. `gitlab.rb` should look the same on all nodes. 1. If present, remove the `gitlab_repmgr` database and role on the primary. If you don't delete the `gitlab_repmgr` - database, upgrading PostgreSQL 11 to 12 will fail with: + database, upgrading PostgreSQL 11 to 12 fails with: ```plaintext could not load library "$libdir/repmgr_funcs": ERROR: could not access file "$libdir/repmgr_funcs": No such file or directory @@ -997,9 +1044,7 @@ You can switch an exiting database cluster to use Patroni instead of repmgr with ### Upgrading PostgreSQL major version in a Patroni cluster -As of GitLab 13.3, PostgreSQL 11.7 and 12.3 are both shipped with Omnibus GitLab, and as of GitLab 13.7 -PostgreSQL 12 is used by default. If you want to upgrade to PostgreSQL 12 in versions prior to GitLab 13.7, -you must ask for it explicitly. +As of GitLab 13.3, PostgreSQL 11.7 and 12.3 are both shipped with Omnibus GitLab by default. As of GitLab 13.7, PostgreSQL 12 is the default. If you want to upgrade to PostgreSQL 12 in versions prior to GitLab 13.7, you must ask for it explicitly. WARNING: The procedure for upgrading PostgreSQL in a Patroni cluster is different than when upgrading using repmgr. @@ -1008,20 +1053,16 @@ upgrading PostgreSQL. Here are a few key facts that you must consider before upgrading PostgreSQL: -- The main point is that you will have to **shut down the Patroni cluster**. This means that your +- The main point is that you have to **shut down the Patroni cluster**. This means that your GitLab deployment is down for the duration of database upgrade or, at least, as long as your leader node is upgraded. This can be **a significant downtime depending on the size of your database**. -- Upgrading PostgreSQL creates a new data directory with a new control data. From Patroni's perspective - this is a new cluster that needs to be bootstrapped again. Therefore, as part of the upgrade procedure, - the cluster state (stored in Consul) is wiped out. Once the upgrade is completed, Patroni - bootstraps a new cluster. **Note that this changes your _cluster ID_**. +- Upgrading PostgreSQL creates a new data directory with a new control data. From Patroni's perspective this is a new cluster that needs to be bootstrapped again. Therefore, as part of the upgrade procedure, the cluster state (stored in Consul) is wiped out. After the upgrade is complete, Patroni bootstraps a new cluster. **This changes your _cluster ID_**. -- The procedures for upgrading leader and replicas are not the same. That is why it is important to use the - right procedure on each node. +- The procedures for upgrading leader and replicas are not the same. That is why it is important to use the right procedure on each node. - Upgrading a replica node **deletes the data directory and resynchronizes it** from the leader using the - configured replication method (currently `pg_basebackup` is the only available option). It might take some + configured replication method (`pg_basebackup` is the only available option). It might take some time for replica to catch up with the leader, depending on the size of your database. - An overview of the upgrade procedure is outlined in [Patoni's documentation](https://patroni.readthedocs.io/en/latest/existing_data.html#major-upgrade-of-postgresql-version). @@ -1036,9 +1077,7 @@ Considering these, you should carefully plan your PostgreSQL upgrade: ``` NOTE: - `gitlab-ctl pg-upgrade` tries to detect the role of the node. If for any reason the auto-detection - does not work or you believe it did not detect the role correctly, you can use the `--leader` or `--replica` - arguments to manually override it. + `gitlab-ctl pg-upgrade` tries to detect the role of the node. If for any reason the auto-detection does not work or you believe it did not detect the role correctly, you can use the `--leader` or `--replica` arguments to manually override it. 1. Stop Patroni **only on replicas**. @@ -1081,7 +1120,7 @@ Considering these, you should carefully plan your PostgreSQL upgrade: ``` NOTE: -Reverting PostgreSQL upgrade with `gitlab-ctl revert-pg-upgrade` has the same considerations as +Reverting the PostgreSQL upgrade with `gitlab-ctl revert-pg-upgrade` has the same considerations as `gitlab-ctl pg-upgrade`. You should follow the same procedure by first stopping the replicas, then reverting the leader, and finally reverting the replicas. @@ -1089,11 +1128,11 @@ then reverting the leader, and finally reverting the replicas. ### Consul and PostgreSQL changes not taking effect -Due to the potential impacts, `gitlab-ctl reconfigure` only reloads Consul and PostgreSQL, it will not restart the services. However, not all changes can be activated by reloading. +Due to the potential impacts, `gitlab-ctl reconfigure` only reloads Consul and PostgreSQL, it does not restart the services. However, not all changes can be activated by reloading. To restart either service, run `gitlab-ctl restart SERVICE` -For PostgreSQL, it is usually safe to restart the master node by default. Automatic failover defaults to a 1 minute timeout. Provided the database returns before then, nothing else needs to be done. +For PostgreSQL, it is usually safe to restart the leader node by default. Automatic failover defaults to a 1 minute timeout. Provided the database returns before then, nothing else needs to be done. On the Consul server nodes, it is important to [restart the Consul service](../consul.md#restart-consul) in a controlled manner. @@ -1109,7 +1148,7 @@ PG::ConnectionBad: ERROR: pgbouncer cannot connect to server The problem may be that your PgBouncer node's IP address is not included in the `trust_auth_cidr_addresses` setting in `/etc/gitlab/gitlab.rb` on the database nodes. -You can confirm that this is the issue by checking the PostgreSQL log on the master +You can confirm that this is the issue by checking the PostgreSQL log on the leader database node. If you see the following error then `trust_auth_cidr_addresses` is the problem. @@ -1172,16 +1211,16 @@ To reset the Patroni state in Consul: /opt/gitlab/embedded/bin/consul kv delete -recurse /service/postgresql-ha/ ``` -1. Start one Patroni node, which will initialize the Patroni cluster and be elected as a leader. +1. Start one Patroni node, which initializes the Patroni cluster to elect as a leader. It's highly recommended to start the previous leader (noted in the first step), - in order to not lose existing writes that may have not been replicated because + so as to not lose existing writes that may have not been replicated because of the broken cluster state: ```shell sudo gitlab-ctl start patroni ``` -1. Start all other Patroni nodes that will join the Patroni cluster as replicas: +1. Start all other Patroni nodes that join the Patroni cluster as replicas: ```shell sudo gitlab-ctl start patroni @@ -1238,9 +1277,7 @@ Traceback (most recent call last): If the stack trace ends with `CFUNCTYPE(c_int)(lambda: None)`, this code triggers `MemoryError` if the Linux server has been hardened for security. -The code causes Python to write temporary executable files, and if it cannot find a filesystem -in which to do this, for example if `noexec` is set on the `/tmp` filesystem, it fails with -`MemoryError` ([read more in the issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6184)). +The code causes Python to write temporary executable files, and if it cannot find a file system in which to do this. For example, if `noexec` is set on the `/tmp` file system, it fails with `MemoryError` ([read more in the issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6184)). Workarounds: @@ -1248,7 +1285,7 @@ Workarounds: - If set to enforcing, SELinux may also prevent these operations. Verify the issue is fixed by setting SELinux to permissive. -Omnibus GitLab has shipped with Patroni since 13.1 along with a build of Python 3.7. +Patroni has been shipping with Omnibus GitLab since 13.1, along with a build of Python 3.7. Workarounds should stop being required when GitLab 14.x starts shipping with [a later version of Python](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6164) as the code which causes this was removed from Python 3.8. diff --git a/doc/administration/raketasks/check.md b/doc/administration/raketasks/check.md index bcc2f838565..56bf711f187 100644 --- a/doc/administration/raketasks/check.md +++ b/doc/administration/raketasks/check.md @@ -21,7 +21,7 @@ There are 3 things that are checked to determine integrity. 1. Check for `config.lock` in the repository directory. 1. Check for any branch/references lock files in `refs/heads`. -It's important to note that the existence of `config.lock` or reference locks +The existence of `config.lock` or reference locks alone do not necessarily indicate a problem. Lock files are routinely created and removed as Git and GitLab perform operations on the repository. They serve to prevent data integrity issues. However, if a Git operation is interrupted these diff --git a/doc/administration/raketasks/doctor.md b/doc/administration/raketasks/doctor.md index ec3f7835b9c..02d1557b6a4 100644 --- a/doc/administration/raketasks/doctor.md +++ b/doc/administration/raketasks/doctor.md @@ -70,6 +70,7 @@ bundle exec rake gitlab:doctor:secrets RAILS_ENV=production VERBOSE=1 **Example verbose output** <!-- vale gitlab.SentenceSpacing = NO --> + ```plaintext I, [2020-06-11T17:17:54.951815 #27148] INFO -- : Checking encrypted values in the database I, [2020-06-11T17:18:12.677708 #27148] INFO -- : - ApplicationSetting failures: 0 @@ -83,4 +84,5 @@ I, [2020-06-11T17:18:15.575533 #27148] INFO -- : - ScimOauthAccessToken failure I, [2020-06-11T17:18:15.575678 #27148] INFO -- : Total: 1 row(s) affected I, [2020-06-11T17:18:15.575711 #27148] INFO -- : Done! ``` + <!-- vale gitlab.SentenceSpacing = YES --> diff --git a/doc/administration/raketasks/smtp.md b/doc/administration/raketasks/smtp.md new file mode 100644 index 00000000000..c738f1dcd00 --- /dev/null +++ b/doc/administration/raketasks/smtp.md @@ -0,0 +1,103 @@ +--- +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# SMTP Rake tasks **(FREE SELF)** + +[Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67802) in GitLab 14.2. + +The following are SMTP-related Rake tasks. + +## Secrets + +GitLab can use SMTP configuration secrets to read from an encrypted file. The following Rake tasks are provided for updating the contents of the encrypted file. + +### Show secret + +Show the contents of the current SMTP secrets. + +**Omnibus Installation** + +```shell +sudo gitlab-rake gitlab:smtp:secret:show +``` + +**Source Installation** + +```shell +bundle exec rake gitlab:smtp:secret:show RAILS_ENV=production +``` + +**Example output:** + +```plaintext +password: '123' +user_name: 'gitlab-inst' +``` + +### Edit secret + +Opens the secret contents in your editor, and writes the resulting content to the encrypted secret file when you exit. + +**Omnibus Installation** + +```shell +sudo gitlab-rake gitlab:smtp:secret:edit EDITOR=vim +``` + +**Source Installation** + +```shell +bundle exec rake gitlab:smtp:secret:edit RAILS_ENV=production EDITOR=vim +``` + +### Write raw secret + +Write new secret content by providing it on STDIN. + +**Omnibus Installation** + +```shell +echo -e "password: '123'" | sudo gitlab-rake gitlab:smtp:secret:write +``` + +**Source Installation** + +```shell +echo -e "password: '123'" | bundle exec rake gitlab:smtp:secret:write RAILS_ENV=production +``` + +### Secrets examples + +**Editor example** + +The write task can be used in cases where the edit command does not work with your editor: + +```shell +# Write the existing secret to a plaintext file +sudo gitlab-rake gitlab:smtp:secret:show > smtp.yaml +# Edit the smtp file in your editor +... +# Re-encrypt the file +cat smtp.yaml | sudo gitlab-rake gitlab:smtp:secret:write +# Remove the plaintext file +rm smtp.yaml +``` + +**KMS integration example** + +It can also be used as a receiving application for content encrypted with a KMS: + +```shell +gcloud kms decrypt --key my-key --keyring my-test-kms --plaintext-file=- --ciphertext-file=my-file --location=us-west1 | sudo gitlab-rake gitlab:smtp:secret:write +``` + +**Google Cloud secret integration example** + +It can also be used as a receiving application for secrets out of Google Cloud: + +```shell +gcloud secrets versions access latest --secret="my-test-secret" > $1 | sudo gitlab-rake gitlab:smtp:secret:write +``` diff --git a/doc/administration/raketasks/uploads/migrate.md b/doc/administration/raketasks/uploads/migrate.md index ab0a51ba8d6..0628e351b63 100644 --- a/doc/administration/raketasks/uploads/migrate.md +++ b/doc/administration/raketasks/uploads/migrate.md @@ -133,9 +133,9 @@ migrate your data out of object storage and back into your local storage. WARNING: **Extended downtime is required** so no new files are created in object storage during -the migration. A configuration setting is planned to allow migrating -from object storage to local files with only a brief moment of downtime for configuration changes. -To follow progress, see the [relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/30979). +the migration. A configuration setting to allow migrating +from object storage to local files with only a brief moment of downtime for configuration changes +is tracked [in this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/30979). ### All-in-one Rake task diff --git a/doc/administration/read_only_gitlab.md b/doc/administration/read_only_gitlab.md index c8931eb79df..cf45d3e15cf 100644 --- a/doc/administration/read_only_gitlab.md +++ b/doc/administration/read_only_gitlab.md @@ -76,7 +76,7 @@ the database is read-only: -h /var/opt/gitlab/postgresql gitlabhq_production ``` -1. Create the `gitlab_read_only` user. Note that the password is set to `mypassword`, +1. Create the `gitlab_read_only` user. The password is set to `mypassword`, change that to your liking: ```sql diff --git a/doc/administration/redis/replication_and_failover.md b/doc/administration/redis/replication_and_failover.md index 37d586b1d32..48db734b1af 100644 --- a/doc/administration/redis/replication_and_failover.md +++ b/doc/administration/redis/replication_and_failover.md @@ -106,7 +106,7 @@ is not initiated. As an example, if you have `6` Redis nodes, a maximum of `3` can be simultaneously down. -Please note that there are different requirements for Sentinel nodes. +There are different requirements for Sentinel nodes. If you host them in the same Redis machines, you may need to take that restrictions into consideration when calculating the amount of nodes to be provisioned. See [Sentinel setup overview](#sentinel-setup-overview) @@ -539,7 +539,7 @@ Here is a list and description of each **machine** and the assigned **IP**: - `10.0.0.3`: Redis Replica 2 + Sentinel 3 - `10.0.0.4`: GitLab application -Please note that after the initial configuration, if a failover is initiated +After the initial configuration, if a failover is initiated by the Sentinel nodes, the Redis nodes are reconfigured and the **Primary** changes permanently (including in `redis.conf`) from one node to the other, until a new failover is initiated again. @@ -683,8 +683,6 @@ To make this work with Sentinel: ] ``` - Note that: - - Redis URLs should be in the format: `redis://:PASSWORD@SENTINEL_PRIMARY_NAME`, where: - `PASSWORD` is the plaintext password for the Redis instance. - `SENTINEL_PRIMARY_NAME` is the Sentinel primary name set with `redis['master_name']`, @@ -731,7 +729,7 @@ redis_master_role['enable'] = true # enable only one of them redis_replica_role['enable'] = true # enable only one of them # When Redis primary or Replica role are enabled, the following services are -# enabled/disabled. Note that if Redis and Sentinel roles are combined, both +# enabled/disabled. If Redis and Sentinel roles are combined, both # services are enabled. # The following services are disabled diff --git a/doc/administration/redis/replication_and_failover_external.md b/doc/administration/redis/replication_and_failover_external.md index 65ec8eb50e5..234b1aa7fb9 100644 --- a/doc/administration/redis/replication_and_failover_external.md +++ b/doc/administration/redis/replication_and_failover_external.md @@ -260,7 +260,7 @@ Here is a list and description of each **machine** and the assigned **IP**: - `10.0.0.3`: Redis Replica 2 + Sentinel 3 - `10.0.0.4`: GitLab application -Please note that after the initial configuration, if a failover is initiated +After the initial configuration, if a failover is initiated by the Sentinel nodes, the Redis nodes are reconfigured and the **Primary** changes permanently (including in `redis.conf`) from one node to the other, until a new failover is initiated again. diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 1fc3483fbd4..f9397e6dbca 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -1,5 +1,4 @@ --- -reading_time: true stage: Enablement group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments @@ -14,33 +13,34 @@ full list of reference architectures, see > - **Supported users (approximate):** 10,000 > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) > - **Test requests per second (RPS) rates:** API: 200 RPS, Web: 20 RPS, Git (Pull): 20 RPS, Git (Push): 4 RPS - -| Service | Nodes | Configuration | GCP | AWS | Azure | -|--------------------------------------------|-------------|-------------------------|------------------|--------------|-----------| -| External load balancing node(3) | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Consul(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| PostgreSQL(1) | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | `D8s v3` | -| PgBouncer(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Internal load balancing node(3) | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Redis - Cache(2) | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis - Queues / Shared State(2) | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis Sentinel - Cache(2) | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | -| Redis Sentinel - Queues / Shared State(2) | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | -| Gitaly | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | `D16s v3` | -| Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Praefect PostgreSQL(1) | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| GitLab Rails | 3 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | -| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | -| Object storage(4) | n/a | n/a | n/a | n/a | n/a | -| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +> - **[Latest 10k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k)** + +| Service | Nodes | Configuration | GCP | AWS | Azure | +|-----------------------------------------------------|-------------|-------------------------|------------------|--------------|-----------| +| External load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| PostgreSQL<sup>1</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | `D8s v3` | +| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | +| Redis Sentinel - Queues / Shared State<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | +| Gitaly | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | `D16s v3` | +| Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| GitLab Rails | 3 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | +| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | n/a | +| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. -4. Should be run on reputable third party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. <!-- markdownlint-enable MD029 --> NOTE: @@ -141,7 +141,7 @@ is recommended instead of using NFS. Using an object storage service also doesn't require you to provision and maintain a node. It's also worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and -that to achieve full High Availability a third party PostgreSQL database solution will be required. +that to achieve full High Availability a third-party PostgreSQL database solution will be required. We hope to offer a built in solutions for these restrictions in the future but in the meantime a non HA PostgreSQL server can be set up via Omnibus GitLab, which the above specs reflect. Refer to the following issues for more information: [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398) @@ -345,7 +345,7 @@ Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`. The Internal Load Balancer is used to balance any internal connections the GitLab environment requires such as connections to [PgBouncer](#configure-pgbouncer) and [Praefect](#configure-praefect) (Gitaly Cluster). -Note that it's a separate node from the External Load Balancer and shouldn't have any access externally. +It's a separate node from the External Load Balancer and shouldn't have any access externally. The following IP will be used as an example: @@ -569,12 +569,12 @@ in the second step, do not supply the `EXTERNAL_URL` value. # Sets `max_replication_slots` to double the number of database nodes. # Patroni uses one extra slot per node when initiating the replication. - patroni['postgresql']['max_replication_slots'] = 8 + patroni['postgresql']['max_replication_slots'] = 6 # Set `max_wal_senders` to one more than the number of replication slots in the cluster. # This is used to prevent replication from using up all of the # available database connections. - patroni['postgresql']['max_wal_senders'] = 9 + patroni['postgresql']['max_wal_senders'] = 7 # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. patroni['postgresql']['max_connections'] = 500 @@ -601,7 +601,10 @@ in the second step, do not supply the `EXTERNAL_URL` value. patroni['username'] = '<patroni_api_username>' patroni['password'] = '<patroni_api_password>' - # Replace XXX.XXX.XXX.XXX/YY with Network Address + # Replace 10.6.0.0/24 with Network Addresses for your other patroni nodes + patroni['allowlist'] = %w(10.6.0.0/24 127.0.0.1/32) + + # Replace 10.6.0.0/24 with Network Address postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24 127.0.0.1/32) # Set the network addresses that the exporters will listen on for monitoring @@ -2368,7 +2371,7 @@ in Kubernetes via our official [Helm Charts](https://docs.gitlab.com/charts/). In this setup, we support running the equivalent of GitLab Rails and Sidekiq nodes in a Kubernetes cluster, named Webservice and Sidekiq respectively. In addition, the following other supporting services are supported: NGINX, Task Runner, Migrations, -Prometheus and Grafana. +Prometheus, and Grafana. Hybrid installations leverage the benefits of both cloud native and traditional compute deployments. With this, _stateless_ components can benefit from cloud native @@ -2379,23 +2382,23 @@ NOTE: This is an **advanced** setup. Running services in Kubernetes is well known to be complex. **This setup is only recommended** if you have strong working knowledge and experience in Kubernetes. The rest of this -section will assume this. +section assumes this. ### Cluster topology -The following tables and diagram details the hybrid environment using the same formats +The following tables and diagram detail the hybrid environment using the same formats as the normal environment above. -First starting with the components that run in Kubernetes. The recommendations at this -time use Google Cloud’s Kubernetes Engine (GKE) and associated machine types, but the memory +First are the components that run in Kubernetes. The recommendation at this time is to +use Google Cloud’s Kubernetes Engine (GKE) and associated machine types, but the memory and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. -| Service | Nodes(1) | Configuration | GCP | Allocatable CPUs and Memory | -|-------------------------------------------------------|----------|-------------------------|------------------|-----------------------------| -| Webservice | 4 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | 127.5 vCPU, 118 GB memory | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | 15.5 vCPU, 50 GB memory | -| Supporting services such as NGINX or Prometheus | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | 7.75 vCPU, 25 GB memory | +| Service | Nodes<sup>1</sup> | Configuration | GCP | Allocatable CPUs and Memory | +|-------------------------------------------------------|-------------------|-------------------------|------------------|-----------------------------| +| Webservice | 4 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | 127.5 vCPU, 118 GB memory | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | 15.5 vCPU, 50 GB memory | +| Supporting services such as NGINX, Prometheus | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | 7.75 vCPU, 25 GB memory | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> @@ -2406,27 +2409,27 @@ future with further specific cloud provider details. Next are the backend components that run on static compute VMs via Omnibus (or External PaaS services where applicable): -| Service | Nodes | Configuration | GCP | -|--------------------------------------------|-------|-------------------------|------------------| -| Consul(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| PostgreSQL(1) | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | -| PgBouncer(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Internal load balancing node(3) | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Redis - Cache(2) | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis - Queues / Shared State(2) | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis Sentinel - Cache(2) | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | -| Redis Sentinel - Queues / Shared State(2) | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | -| Gitaly | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | -| Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Praefect PostgreSQL(1) | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Object storage(4) | n/a | n/a | n/a | +| Service | Nodes | Configuration | GCP | +|-----------------------------------------------------|-------|-------------------------|------------------| +| Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| PostgreSQL<sup>1</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | +| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | +| Redis Sentinel - Queues / Shared State<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | +| Gitaly | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | +| Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Object storage<sup>4</sup> | n/a | n/a | n/a | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. -4. Should be run on reputable third party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. <!-- markdownlint-enable MD029 --> NOTE: @@ -2520,11 +2523,11 @@ documents how to apply the calculated configuration to the Helm Chart. #### Webservice Webservice pods typically need about 1 vCPU and 1.25 GB of memory _per worker_. -Each Webservice pod will consume roughly 4 vCPUs and 5 GB of memory using +Each Webservice pod consumes roughly 4 vCPUs and 5 GB of memory using the [recommended topology](#cluster-topology) because four worker processes are created by default and each pod has other small processes running. -For 10k users we recommend a total Puma worker count of around 80. +For 10,000 users we recommend a total Puma worker count of around 80. With the [provided recommendations](#cluster-topology) this allows the deployment of up to 20 Webservice pods with 4 workers per pod and 5 pods per node. Expand available resources using the ratio of 1 vCPU to 1.25 GB of memory _per each worker process_ for each additional diff --git a/doc/administration/reference_architectures/1k_users.md b/doc/administration/reference_architectures/1k_users.md index 6ad4e0f05e3..18e34711953 100644 --- a/doc/administration/reference_architectures/1k_users.md +++ b/doc/administration/reference_architectures/1k_users.md @@ -19,6 +19,7 @@ many organizations . > - **High Availability:** No. For a highly-available environment, you can > follow a modified [3K reference architecture](3k_users.md#supported-modifications-for-lower-user-counts-ha). > - **Test requests per second (RPS) rates:** API: 20 RPS, Web: 2 RPS, Git (Pull): 2 RPS, Git (Push): 1 RPS +> - **[Latest 1k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/1k)** | Users | Configuration | GCP | AWS | Azure | |--------------|-------------------------|----------------|--------------|----------| diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index e45a8f6963c..65d59422da8 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -1,5 +1,4 @@ --- -reading_time: true stage: Enablement group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments @@ -14,33 +13,34 @@ full list of reference architectures, see > - **Supported users (approximate):** 25,000 > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) > - **Test requests per second (RPS) rates:** API: 500 RPS, Web: 50 RPS, Git (Pull): 50 RPS, Git (Push): 10 RPS - -| Service | Nodes | Configuration | GCP | AWS | Azure | -|------------------------------------------|-------------|-------------------------|------------------|--------------|-----------| -| External load balancing node(3) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | -| Consul(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| PostgreSQL(1) | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | `D16s v3` | -| PgBouncer(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Internal load balancing node(3) | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | `c5.large` | `F2s v2` | -| Redis - Cache(2) | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis - Queues / Shared State(2) | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis Sentinel - Cache(2) | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | -| Redis Sentinel - Queues / Shared State(2)| 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | -| Gitaly | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` | -| Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | -| Praefect PostgreSQL(1) | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| GitLab Rails | 5 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | -| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | -| Object storage(4) | n/a | n/a | n/a | n/a | n/a | -| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +> - **[Latest 25k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/25k)** + +| Service | Nodes | Configuration | GCP | AWS | Azure | +|---------------------------------------------------|-------------|-------------------------|------------------|--------------|-----------| +| External load balancing node<sup>3</sup> | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| PostgreSQL<sup>1</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | `D16s v3` | +| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Internal load balancing node<sup>3</sup> | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | `c5.large` | `F2s v2` | +| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | +| Redis Sentinel - Queues / Shared State<sup>2</sup>| 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | +| Gitaly | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` | +| Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| GitLab Rails | 5 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | +| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | n/a | +| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. -4. Should be run on reputable third party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. <!-- markdownlint-enable MD029 --> NOTE: @@ -141,7 +141,7 @@ is recommended instead of using NFS. Using an object storage service also doesn't require you to provision and maintain a node. It's also worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and -that to achieve full High Availability a third party PostgreSQL database solution will be required. +that to achieve full High Availability a third-party PostgreSQL database solution will be required. We hope to offer a built in solutions for these restrictions in the future but in the meantime a non HA PostgreSQL server can be set up via Omnibus GitLab, which the above specs reflect. Refer to the following issues for more information: [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398) @@ -347,7 +347,7 @@ Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`. The Internal Load Balancer is used to balance any internal connections the GitLab environment requires such as connections to [PgBouncer](#configure-pgbouncer) and [Praefect](#configure-praefect) (Gitaly Cluster). -Note that it's a separate node from the External Load Balancer and shouldn't have any access externally. +It's a separate node from the External Load Balancer and shouldn't have any access externally. The following IP will be used as an example: @@ -571,12 +571,12 @@ in the second step, do not supply the `EXTERNAL_URL` value. # Sets `max_replication_slots` to double the number of database nodes. # Patroni uses one extra slot per node when initiating the replication. - patroni['postgresql']['max_replication_slots'] = 8 + patroni['postgresql']['max_replication_slots'] = 6 # Set `max_wal_senders` to one more than the number of replication slots in the cluster. # This is used to prevent replication from using up all of the # available database connections. - patroni['postgresql']['max_wal_senders'] = 9 + patroni['postgresql']['max_wal_senders'] = 7 # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. patroni['postgresql']['max_connections'] = 500 @@ -603,7 +603,10 @@ in the second step, do not supply the `EXTERNAL_URL` value. patroni['username'] = '<patroni_api_username>' patroni['password'] = '<patroni_api_password>' - # Replace XXX.XXX.XXX.XXX/YY with Network Address + # Replace 10.6.0.0/24 with Network Addresses for your other patroni nodes + patroni['allowlist'] = %w(10.6.0.0/24 127.0.0.1/32) + + # Replace 10.6.0.0/24 with Network Address postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24 127.0.0.1/32) # Set the network addresses that the exporters will listen on for monitoring @@ -2380,7 +2383,7 @@ in Kubernetes via our official [Helm Charts](https://docs.gitlab.com/charts/). In this setup, we support running the equivalent of GitLab Rails and Sidekiq nodes in a Kubernetes cluster, named Webservice and Sidekiq respectively. In addition, the following other supporting services are supported: NGINX, Task Runner, Migrations, -Prometheus and Grafana. +Prometheus, and Grafana. Hybrid installations leverage the benefits of both cloud native and traditional compute deployments. With this, _stateless_ components can benefit from cloud native @@ -2391,23 +2394,23 @@ NOTE: This is an **advanced** setup. Running services in Kubernetes is well known to be complex. **This setup is only recommended** if you have strong working knowledge and experience in Kubernetes. The rest of this -section will assume this. +section assumes this. ### Cluster topology -The following tables and diagram details the hybrid environment using the same formats +The following tables and diagram detail the hybrid environment using the same formats as the normal environment above. -First starting with the components that run in Kubernetes. The recommendations at this -time use Google Cloud’s Kubernetes Engine (GKE) and associated machine types, but the memory +First are the components that run in Kubernetes. The recommendation at this time is to +use Google Cloud’s Kubernetes Engine (GKE) and associated machine types, but the memory and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. -| Service | Nodes(1) | Configuration | GCP | Allocatable CPUs and Memory | -|-------------------------------------------------------|----------|-------------------------|------------------|-----------------------------| -| Webservice | 7 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | 223 vCPU, 206.5 GB memory | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | 15.5 vCPU, 50 GB memory | -| Supporting services such as NGINX, Prometheus, etc. | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | 7.75 vCPU, 25 GB memory | +| Service | Nodes<sup>1</sup> | Configuration | GCP | Allocatable CPUs and Memory | +|-------------------------------------------------------|-------------------|-------------------------|------------------|-----------------------------| +| Webservice | 7 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | 223 vCPU, 206.5 GB memory | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | 15.5 vCPU, 50 GB memory | +| Supporting services such as NGINX, Prometheus | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | 7.75 vCPU, 25 GB memory | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> @@ -2418,27 +2421,27 @@ future with further specific cloud provider details. Next are the backend components that run on static compute VMs via Omnibus (or External PaaS services where applicable): -| Service | Nodes | Configuration | GCP | -|--------------------------------------------|-------|-------------------------|------------------| -| Consul(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| PostgreSQL(1) | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | -| PgBouncer(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Internal load balancing node(3) | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | -| Redis - Cache(2) | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis - Queues / Shared State(2) | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis Sentinel - Cache(2) | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | -| Redis Sentinel - Queues / Shared State(2) | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | -| Gitaly | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | -| Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | -| Praefect PostgreSQL(1) | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Object storage(4) | n/a | n/a | n/a | +| Service | Nodes | Configuration | GCP | +|-----------------------------------------------------|-------|-------------------------|------------------| +| Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| PostgreSQL<sup>1</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | +| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Internal load balancing node<sup>3</sup> | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | +| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | +| Redis Sentinel - Queues / Shared State<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | +| Gitaly | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | +| Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | +| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Object storage<sup>4</sup> | n/a | n/a | n/a | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. -4. Should be run on reputable third party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. <!-- markdownlint-enable MD029 --> NOTE: @@ -2532,11 +2535,11 @@ documents how to apply the calculated configuration to the Helm Chart. #### Webservice Webservice pods typically need about 1 vCPU and 1.25 GB of memory _per worker_. -Each Webservice pod will consume roughly 4 vCPUs and 5 GB of memory using +Each Webservice pod consumes roughly 4 vCPUs and 5 GB of memory using the [recommended topology](#cluster-topology) because four worker processes are created by default and each pod has other small processes running. -For 25k users we recommend a total Puma worker count of around 140. +For 25,000 users we recommend a total Puma worker count of around 140. With the [provided recommendations](#cluster-topology) this allows the deployment of up to 35 Webservice pods with 4 workers per pod and 5 pods per node. Expand available resources using the ratio of 1 vCPU to 1.25 GB of memory _per each worker process_ for each additional diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index ff3db877553..0af4dbc8a7f 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -1,5 +1,4 @@ --- -reading_time: true stage: Enablement group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments @@ -15,23 +14,24 @@ For a full list of reference architectures, see > - **High Availability:** No. For a highly-available environment, you can > follow a modified [3K reference architecture](3k_users.md#supported-modifications-for-lower-user-counts-ha). > - **Test requests per second (RPS) rates:** API: 40 RPS, Web: 4 RPS, Git (Pull): 4 RPS, Git (Push): 1 RPS +> - **[Latest 2k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/2k)** | Service | Nodes | Configuration | GCP | AWS | Azure | |------------------------------------------|--------|-------------------------|-----------------|--------------|----------| -| Load balancer(3) | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| PostgreSQL(1) | 1 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | -| Redis(2) | 1 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `m5.large` | `D2s v3` | +| Load balancer<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| PostgreSQL<sup>1</sup> | 1 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | +| Redis<sup>2</sup> | 1 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `m5.large` | `D2s v3` | | Gitaly | 1 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | GitLab Rails | 2 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | | Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Object storage(4) | n/a | n/a | n/a | n/a | n/a | +| Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | n/a | | NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. -2. Can be optionally run as reputable third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. -3. Can be optionally run as reputable third party load balancing services (LB PaaS). AWS ELB is known to work. -4. Should be run on reputable third party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +2. Can be optionally run as reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +3. Can be optionally run as reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. +4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. <!-- markdownlint-enable MD029 --> NOTE: @@ -444,10 +444,7 @@ To configure the Gitaly server, on the server node you want to use for Gitaly: 1. Edit the Gitaly server node's `/etc/gitlab/gitlab.rb` file to configure storage paths, enable the network listener, and to configure the token: - <!-- - updates to following example must also be made at - https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab - --> + <!-- Updates to following example must also be made at https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab --> ```ruby # /etc/gitlab/gitlab.rb @@ -557,10 +554,7 @@ To configure Gitaly with TLS: 1. Edit `/etc/gitlab/gitlab.rb` and add: - <!-- - updates to following example must also be made at - https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab - --> + <!-- Updates to following example must also be made at https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab --> ```ruby gitaly['tls_listen_addr'] = "0.0.0.0:9999" @@ -973,6 +967,143 @@ Read: - The [Gitaly and NFS deprecation notice](../gitaly/index.md#nfs-deprecation-notice). - About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). +## Cloud Native Hybrid reference architecture with Helm Charts (alternative) + +As an alternative approach, you can also run select components of GitLab as Cloud Native +in Kubernetes via our official [Helm Charts](https://docs.gitlab.com/charts/). +In this setup, we support running the equivalent of GitLab Rails and Sidekiq nodes +in a Kubernetes cluster, named Webservice and Sidekiq respectively. In addition, +the following other supporting services are supported: NGINX, Task Runner, Migrations, +Prometheus, and Grafana. + +Hybrid installations leverage the benefits of both cloud native and traditional +compute deployments. With this, _stateless_ components can benefit from cloud native +workload management benefits while _stateful_ components are deployed in compute VMs +with Omnibus to benefit from increased permanence. + +The 2,000 reference architecture is not a highly-available setup. To achieve HA, you can follow a modified [3K reference architecture](3k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative). + +NOTE: +This is an **advanced** setup. Running services in Kubernetes is well known +to be complex. **This setup is only recommended** if you have strong working +knowledge and experience in Kubernetes. The rest of this +section assumes this. + +### Cluster topology + +The following tables and diagram detail the hybrid environment using the same formats +as the normal environment above. + +First are the components that run in Kubernetes. The recommendation at this time is to +use Google Cloud’s Kubernetes Engine (GKE) and associated machine types, but the memory +and CPU requirements should translate to most other providers. We hope to update this in the +future with further specific cloud provider details. + +| Service | Nodes<sup>1</sup> | Configuration | GCP | Allocatable CPUs and Memory | +|-------------------------------------------------------|-------------------|------------------------|-----------------|-----------------------------| +| Webservice | 3 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | 23.7 vCPU, 16.9 GB memory | +| Sidekiq | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | 3.9 vCPU, 11.8 GB memory | +| Supporting services such as NGINX, Prometheus | 2 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | 1.9 vCPU, 5.5 GB memory | + +<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> +<!-- markdownlint-disable MD029 --> +1. Nodes configuration is shown as it is forced to ensure pod vcpu / memory ratios and avoid scaling during **performance testing**. + In production deployments there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices. +<!-- markdownlint-enable MD029 --> + +Next are the backend components that run on static compute VMs via Omnibus (or External PaaS +services where applicable): + +| Service | Nodes | Configuration | GCP | +|--------------------------------------------|-------|-------------------------|------------------| +| PostgreSQL<sup>1</sup> | 1 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | +| Redis<sup>2</sup> | 1 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | +| Gitaly | 1 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Object storage<sup>3</sup> | n/a | n/a | n/a | + +<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> +<!-- markdownlint-disable MD029 --> +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +3. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +<!-- markdownlint-enable MD029 --> + +NOTE: +For all PaaS solutions that involve configuring instances, it is strongly recommended to implement a minimum of three nodes in three different availability zones to align with resilient cloud architecture practices. + +```plantuml +@startuml 2k + +card "Kubernetes via Helm Charts" as kubernetes { + card "**External Load Balancer**" as elb #6a9be7 + + together { + collections "**Webservice** x3" as gitlab #32CD32 + collections "**Sidekiq** x2" as sidekiq #ff8dd1 + } + + card "**Prometheus + Grafana**" as monitor #7FFFD4 + card "**Supporting Services**" as support +} + +card "**Gitaly**" as gitaly #FF8C00 +card "**PostgreSQL**" as postgres #4EA7FF +card "**Redis**" as redis #FF6347 +cloud "**Object Storage**" as object_storage #white + +elb -[#6a9be7]-> gitlab +elb -[#6a9be7]--> monitor + +gitlab -[#32CD32]--> gitaly +gitlab -[#32CD32]--> postgres +gitlab -[#32CD32]-> object_storage +gitlab -[#32CD32]--> redis + +sidekiq -[#ff8dd1]--> gitaly +sidekiq -[#ff8dd1]-> object_storage +sidekiq -[#ff8dd1]---> postgres +sidekiq -[#ff8dd1]---> redis + +monitor .[#7FFFD4]u-> gitlab +monitor .[#7FFFD4]-> gitaly +monitor .[#7FFFD4]-> postgres +monitor .[#7FFFD4,norank]--> redis +monitor .[#7FFFD4,norank]u--> elb + +@enduml +``` + +### Resource usage settings + +The following formulas help when calculating how many pods may be deployed within resource constraints. +The [2k reference architecture example values file](https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/examples/ref/2k.yaml) +documents how to apply the calculated configuration to the Helm Chart. + +#### Webservice + +Webservice pods typically need about 1 vCPU and 1.25 GB of memory _per worker_. +Each Webservice pod consumes roughly 2 vCPUs and 2.5 GB of memory using +the [recommended topology](#cluster-topology) because two worker processes +are created by default and each pod has other small processes running. + +For 2,000 users we recommend a total Puma worker count of around 12. +With the [provided recommendations](#cluster-topology) this allows the deployment of up to 6 +Webservice pods with 2 workers per pod and 2 pods per node. Expand available resources using +the ratio of 1 vCPU to 1.25 GB of memory _per each worker process_ for each additional +Webservice pod. + +For further information on resource usage, see the [Webservice resources](https://docs.gitlab.com/charts/charts/gitlab/webservice/#resources). + +#### Sidekiq + +Sidekiq pods should generally have 1 vCPU and 2 GB of memory. + +[The provided starting point](#cluster-topology) allows the deployment of up to +2 Sidekiq pods. Expand available resources using the 1 vCPU to 2GB memory +ratio for each additional pod. + +For further information on resource usage, see the [Sidekiq resources](https://docs.gitlab.com/charts/charts/gitlab/sidekiq/#resources). + <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index ef58e69ee27..f4ae01c7442 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -1,5 +1,4 @@ --- -reading_time: true stage: Enablement group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments @@ -24,22 +23,23 @@ For a full list of reference architectures, see > - **Supported users (approximate):** 3,000 > - **High Availability:** Yes, although [Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution > - **Test requests per second (RPS) rates:** API: 60 RPS, Web: 6 RPS, Git (Pull): 6 RPS, Git (Push): 1 RPS +> - **[Latest 3k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/3k)** | Service | Nodes | Configuration | GCP | AWS | Azure | |--------------------------------------------|-------------|-----------------------|-----------------|--------------|----------| -| External load balancing node(3) | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Redis(2) | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | -| Consul(1) + Sentinel(2) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| PostgreSQL(1) | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | -| PgBouncer(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Internal load balancing node(3) | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| External load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Redis<sup>2</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | +| Consul<sup>1</sup> + Sentinel<sup>2</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| PostgreSQL<sup>1</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | +| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Gitaly | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Praefect PostgreSQL(1) | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Sidekiq | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | | GitLab Rails | 3 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | | Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Object storage(4) | n/a | n/a | n/a | n/a | n/a | +| Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | n/a | | NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> @@ -47,7 +47,7 @@ For a full list of reference architectures, see 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. -4. Should be run on reputable third party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. <!-- markdownlint-enable MD029 --> NOTE: @@ -63,10 +63,7 @@ together { collections "**Sidekiq** x4" as sidekiq #ff8dd1 } -together { - card "**Prometheus + Grafana**" as monitor #7FFFD4 - collections "**Consul** x3" as consul #e76a9b -} +card "**Prometheus + Grafana**" as monitor #7FFFD4 card "Gitaly Cluster" as gitaly_cluster { collections "**Praefect** x3" as praefect #FF8C00 @@ -86,14 +83,15 @@ card "Database" as database { postgres_primary .[#4EA7FF]> postgres_secondary } -card "redis" as redis { - collections "**Redis Persistent** x3" as redis_persistent #FF6347 - collections "**Redis Cache** x3" as redis_cache #FF6347 - collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347 - collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347 +card "**Consul + Sentinel**" as consul_sentinel { + collections "**Consul** x3" as consul #e76a9b + collections "**Redis Sentinel** x3" as sentinel #e6e727 +} - redis_persistent <.[#FF6347]- redis_persistent_sentinel - redis_cache <.[#FF6347]- redis_cache_sentinel +card "Redis" as redis { + collections "**Redis** x3" as redis_nodes #FF6347 + + redis_nodes <.[#FF6347]- sentinel } cloud "**Object Storage**" as object_storage #white @@ -348,7 +346,7 @@ Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`. The Internal Load Balancer is used to balance any internal connections the GitLab environment requires such as connections to [PgBouncer](#configure-pgbouncer) and [Praefect](#configure-praefect) (Gitaly Cluster). -Note that it's a separate node from the External Load Balancer and shouldn't have any access externally. +It's a separate node from the External Load Balancer and shouldn't have any access externally. The following IP will be used as an example: @@ -885,7 +883,10 @@ in the second step, do not supply the `EXTERNAL_URL` value. patroni['username'] = '<patroni_api_username>' patroni['password'] = '<patroni_api_password>' - # Replace XXX.XXX.XXX.XXX/YY with Network Address + # Replace 10.6.0.0/24 with Network Addresses for your other patroni nodes + patroni['allowlist'] = %w(10.6.0.0/24 127.0.0.1/32) + + # Replace 10.6.0.0/24 with Network Address postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24 127.0.0.1/32) # Set the network addresses that the exporters will listen on for monitoring @@ -2088,12 +2089,193 @@ but with smaller performance requirements, several modifications can be consider - Combining select nodes: Some nodes can be combined to reduce complexity at the cost of some performance: - GitLab Rails and Sidekiq: Sidekiq nodes can be removed and the component instead enabled on the GitLab Rails nodes. - PostgreSQL and PgBouncer: PgBouncer nodes can be removed and the component instead enabled on PostgreSQL with the Internal Load Balancer pointing to them instead. +- Reducing the node counts: Some node types do not need consensus and can run with fewer nodes (but more than one for redundancy). This will also lead to reduced performance. + - GitLab Rails and Sidekiq: Stateless services don't have a minimum node count. Two are enough for redundancy. + - Gitaly and Praefect: A quorum is not strictly necessary. Two Gitaly nodes and two Praefect nodes are enough for redundancy. - Running select components in reputable Cloud PaaS solutions: Select components of the GitLab setup can instead be run on Cloud Provider PaaS solutions. By doing this, additional dependent components can also be removed: - PostgreSQL: Can be run on reputable Cloud PaaS solutions such as Google Cloud SQL or AWS RDS. In this setup, the PgBouncer and Consul nodes are no longer required: - Consul may still be desired if [Prometheus](../monitoring/prometheus/index.md) auto discovery is a requirement, otherwise you would need to [manually add scrape configurations](../monitoring/prometheus/index.md#adding-custom-scrape-configurations) for all nodes. - As Redis Sentinel runs on the same box as Consul in this architecture, it may need to be run on a separate box if Redis is still being run via Omnibus. - Redis: Can be run on reputable Cloud PaaS solutions such as Google Memorystore and AWS ElastiCache. In this setup, the Redis Sentinel is no longer required. +## Cloud Native Hybrid reference architecture with Helm Charts (alternative) + +As an alternative approach, you can also run select components of GitLab as Cloud Native +in Kubernetes via our official [Helm Charts](https://docs.gitlab.com/charts/). +In this setup, we support running the equivalent of GitLab Rails and Sidekiq nodes +in a Kubernetes cluster, named Webservice and Sidekiq respectively. In addition, +the following other supporting services are supported: NGINX, Task Runner, Migrations, +Prometheus, and Grafana. + +Hybrid installations leverage the benefits of both cloud native and traditional +compute deployments. With this, _stateless_ components can benefit from cloud native +workload management benefits while _stateful_ components are deployed in compute VMs +with Omnibus to benefit from increased permanence. + +NOTE: +This is an **advanced** setup. Running services in Kubernetes is well known +to be complex. **This setup is only recommended** if you have strong working +knowledge and experience in Kubernetes. The rest of this +section assumes this. + +### Cluster topology + +The following tables and diagram detail the hybrid environment using the same formats +as the normal environment above. + +First are the components that run in Kubernetes. The recommendation at this time is to +use Google Cloud’s Kubernetes Engine (GKE) and associated machine types, but the memory +and CPU requirements should translate to most other providers. We hope to update this in the +future with further specific cloud provider details. + +| Service | Nodes<sup>1</sup> | Configuration | GCP | Allocatable CPUs and Memory | +|-------------------------------------------------------|-------------------|-------------------------|------------------|-----------------------------| +| Webservice | 2 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | 31.8 vCPU, 24.8 GB memory | +| Sidekiq | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | 11.8 vCPU, 38.9 GB memory | +| Supporting services such as NGINX, Prometheus | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | 3.9 vCPU, 11.8 GB memory | + +<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> +<!-- markdownlint-disable MD029 --> +1. Nodes configuration is shown as it is forced to ensure pod vcpu / memory ratios and avoid scaling during **performance testing**. + In production deployments there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices. +<!-- markdownlint-enable MD029 --> + +Next are the backend components that run on static compute VMs via Omnibus (or External PaaS +services where applicable): + +| Service | Nodes | Configuration | GCP | +|--------------------------------------------|-------|-------------------------|------------------| +| Redis<sup>2</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | +| Consul<sup>1</sup> + Sentinel<sup>2</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| PostgreSQL<sup>1</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | +| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Gitaly | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Object storage<sup>4</sup> | n/a | n/a | n/a | + +<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> +<!-- markdownlint-disable MD029 --> +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. +4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +<!-- markdownlint-enable MD029 --> + +NOTE: +For all PaaS solutions that involve configuring instances, it is strongly recommended to implement a minimum of three nodes in three different availability zones to align with resilient cloud architecture practices. + +```plantuml +@startuml 3k + +card "Kubernetes via Helm Charts" as kubernetes { + card "**External Load Balancer**" as elb #6a9be7 + + together { + collections "**Webservice** x2" as gitlab #32CD32 + collections "**Sidekiq** x3" as sidekiq #ff8dd1 + } + + card "**Prometheus + Grafana**" as monitor #7FFFD4 + card "**Supporting Services**" as support +} + +card "**Internal Load Balancer**" as ilb #9370DB + +card "**Consul + Sentinel**" as consul_sentinel { + collections "**Consul** x3" as consul #e76a9b + collections "**Redis Sentinel** x3" as sentinel #e6e727 +} + +card "Gitaly Cluster" as gitaly_cluster { + collections "**Praefect** x3" as praefect #FF8C00 + collections "**Gitaly** x3" as gitaly #FF8C00 + card "**Praefect PostgreSQL***\n//Non fault-tolerant//" as praefect_postgres #FF8C00 + + praefect -[#FF8C00]-> gitaly + praefect -[#FF8C00]> praefect_postgres +} + +card "Database" as database { + collections "**PGBouncer** x3" as pgbouncer #4EA7FF + card "**PostgreSQL** (Primary)" as postgres_primary #4EA7FF + collections "**PostgreSQL** (Secondary) x2" as postgres_secondary #4EA7FF + + pgbouncer -[#4EA7FF]-> postgres_primary + postgres_primary .[#4EA7FF]> postgres_secondary +} + +card "Redis" as redis { + collections "**Redis** x3" as redis_nodes #FF6347 + + redis_nodes <.[#FF6347]- sentinel +} + +cloud "**Object Storage**" as object_storage #white + +elb -[#6a9be7]-> gitlab +elb -[#6a9be7]-> monitor +elb -[hidden]-> support + +gitlab -[#32CD32]--> ilb +gitlab -[#32CD32]-> object_storage +gitlab -[#32CD32]---> redis +gitlab -[hidden]--> consul + +sidekiq -[#ff8dd1]--> ilb +sidekiq -[#ff8dd1]-> object_storage +sidekiq -[#ff8dd1]---> redis +sidekiq -[hidden]--> consul + +ilb -[#9370DB]-> gitaly_cluster +ilb -[#9370DB]-> database + +consul .[#e76a9b]-> database +consul .[#e76a9b]-> gitaly_cluster +consul .[#e76a9b,norank]--> redis + +monitor .[#7FFFD4]> consul +monitor .[#7FFFD4]-> database +monitor .[#7FFFD4]-> gitaly_cluster +monitor .[#7FFFD4,norank]--> redis +monitor .[#7FFFD4]> ilb +monitor .[#7FFFD4,norank]u--> elb + +@enduml +``` + +### Resource usage settings + +The following formulas help when calculating how many pods may be deployed within resource constraints. +The [3k reference architecture example values file](https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/examples/ref/3k.yaml) +documents how to apply the calculated configuration to the Helm Chart. + +#### Webservice + +Webservice pods typically need about 1 vCPU and 1.25 GB of memory _per worker_. +Each Webservice pod consumes roughly 4 vCPUs and 5 GB of memory using +the [recommended topology](#cluster-topology) because four worker processes +are created by default and each pod has other small processes running. + +For 3,000 users we recommend a total Puma worker count of around 16. +With the [provided recommendations](#cluster-topology) this allows the deployment of up to 4 +Webservice pods with 4 workers per pod and 2 pods per node. Expand available resources using +the ratio of 1 vCPU to 1.25 GB of memory _per each worker process_ for each additional +Webservice pod. + +For further information on resource usage, see the [Webservice resources](https://docs.gitlab.com/charts/charts/gitlab/webservice/#resources). + +#### Sidekiq + +Sidekiq pods should generally have 1 vCPU and 2 GB of memory. + +[The provided starting point](#cluster-topology) allows the deployment of up to +8 Sidekiq pods. Expand available resources using the 1 vCPU to 2GB memory +ratio for each additional pod. + +For further information on resource usage, see the [Sidekiq resources](https://docs.gitlab.com/charts/charts/gitlab/sidekiq/#resources). + <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 766f94f6c53..b262545b27d 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -1,5 +1,4 @@ --- -reading_time: true stage: Enablement group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments @@ -14,33 +13,34 @@ full list of reference architectures, see > - **Supported users (approximate):** 50,000 > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) > - **Test requests per second (RPS) rates:** API: 1000 RPS, Web: 100 RPS, Git (Pull): 100 RPS, Git (Push): 20 RPS - -| Service | Nodes | Configuration | GCP | AWS | Azure | -|------------------------------------------|-------------|-------------------------|------------------|---------------|-----------| -| External load balancing node(3) | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | -| Consul(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| PostgreSQL(1) | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` | -| PgBouncer(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Internal load balancing node(3) | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | -| Redis - Cache(2) | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis - Queues / Shared State(2) | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis Sentinel - Cache(2) | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | -| Redis Sentinel - Queues / Shared State(2)| 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | -| Gitaly | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | `m5.16xlarge` | `D64s v3` | -| Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | -| Praefect PostgreSQL(1) | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| GitLab Rails | 12 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | -| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | -| Object storage(4) | n/a | n/a | n/a | n/a | n/a | -| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +> - **[Latest 50k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/50k)** + +| Service | Nodes | Configuration | GCP | AWS | Azure | +|---------------------------------------------------|-------------|-------------------------|------------------|---------------|-----------| +| External load balancing node<sup>3</sup> | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | +| Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| PostgreSQL<sup>1</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` | +| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Internal load balancing node<sup>3</sup> | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | +| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | +| Redis Sentinel - Queues / Shared State<sup>2</sup>| 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | +| Gitaly | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | `m5.16xlarge` | `D64s v3` | +| Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| GitLab Rails | 12 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | +| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | n/a | +| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. -4. Should be run on reputable third party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. <!-- markdownlint-enable MD029 --> NOTE: @@ -141,7 +141,7 @@ is recommended instead of using NFS. Using an object storage service also doesn't require you to provision and maintain a node. It's also worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and -that to achieve full High Availability a third party PostgreSQL database solution will be required. +that to achieve full High Availability a third-party PostgreSQL database solution will be required. We hope to offer a built in solutions for these restrictions in the future but in the meantime a non HA PostgreSQL server can be set up via Omnibus GitLab, which the above specs reflect. Refer to the following issues for more information: [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398) @@ -354,7 +354,7 @@ Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`. The Internal Load Balancer is used to balance any internal connections the GitLab environment requires such as connections to [PgBouncer](#configure-pgbouncer) and [Praefect](#configure-praefect) (Gitaly Cluster). -Note that it's a separate node from the External Load Balancer and shouldn't have any access externally. +It's a separate node from the External Load Balancer and shouldn't have any access externally. The following IP will be used as an example: @@ -578,12 +578,12 @@ in the second step, do not supply the `EXTERNAL_URL` value. # Sets `max_replication_slots` to double the number of database nodes. # Patroni uses one extra slot per node when initiating the replication. - patroni['postgresql']['max_replication_slots'] = 8 + patroni['postgresql']['max_replication_slots'] = 6 # Set `max_wal_senders` to one more than the number of replication slots in the cluster. # This is used to prevent replication from using up all of the # available database connections. - patroni['postgresql']['max_wal_senders'] = 9 + patroni['postgresql']['max_wal_senders'] = 7 # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. patroni['postgresql']['max_connections'] = 500 @@ -611,7 +611,10 @@ in the second step, do not supply the `EXTERNAL_URL` value. patroni['username'] = '<patroni_api_username>' patroni['password'] = '<patroni_api_password>' - # Replace XXX.XXX.XXX.XXX/YY with Network Address + # Replace 10.6.0.0/24 with Network Addresses for your other patroni nodes + patroni['allowlist'] = %w(10.6.0.0/24 127.0.0.1/32) + + # Replace 10.6.0.0/24 with Network Address postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24 127.0.0.1/32) # Set the network addresses that the exporters will listen on for monitoring @@ -2391,7 +2394,7 @@ in Kubernetes via our official [Helm Charts](https://docs.gitlab.com/charts/). In this setup, we support running the equivalent of GitLab Rails and Sidekiq nodes in a Kubernetes cluster, named Webservice and Sidekiq respectively. In addition, the following other supporting services are supported: NGINX, Task Runner, Migrations, -Prometheus and Grafana. +Prometheus, and Grafana. Hybrid installations leverage the benefits of both cloud native and traditional compute deployments. With this, _stateless_ components can benefit from cloud native @@ -2402,23 +2405,23 @@ NOTE: This is an **advanced** setup. Running services in Kubernetes is well known to be complex. **This setup is only recommended** if you have strong working knowledge and experience in Kubernetes. The rest of this -section will assume this. +section assumes this. ### Cluster topology -The following tables and diagram details the hybrid environment using the same formats +The following tables and diagram detail the hybrid environment using the same formats as the normal environment above. -First starting with the components that run in Kubernetes. The recommendations at this -time use Google Cloud’s Kubernetes Engine (GKE) and associated machine types, but the memory +First are the components that run in Kubernetes. The recommendation at this time is to +use Google Cloud’s Kubernetes Engine (GKE) and associated machine types, but the memory and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. -| Service | Nodes(1) | Configuration | GCP | Allocatable CPUs and Memory | -|-------------------------------------------------------|----------|-------------------------|------------------|-----------------------------| -| Webservice | 16 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | 510 vCPU, 472 GB memory | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | 15.5 vCPU, 50 GB memory | -| Supporting services such as NGINX, Prometheus, etc. | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | 7.75 vCPU, 25 GB memory | +| Service | Nodes<sup>1</sup> | Configuration | GCP | Allocatable CPUs and Memory | +|-------------------------------------------------------|-------------------|-------------------------|------------------|-----------------------------| +| Webservice | 16 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | 510 vCPU, 472 GB memory | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | 15.5 vCPU, 50 GB memory | +| Supporting services such as NGINX, Prometheus | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | 7.75 vCPU, 25 GB memory | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> @@ -2429,27 +2432,27 @@ future with further specific cloud provider details. Next are the backend components that run on static compute VMs via Omnibus (or External PaaS services where applicable): -| Service | Nodes | Configuration | GCP | -|--------------------------------------------|-------|-------------------------|------------------| -| Consul(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| PostgreSQL(1) | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | -| PgBouncer(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Internal load balancing node(3) | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | -| Redis - Cache(2) | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis - Queues / Shared State(2) | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis Sentinel - Cache(2) | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | -| Redis Sentinel - Queues / Shared State(2) | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | -| Gitaly | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | -| Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | -| Praefect PostgreSQL(1) | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Object storage(4) | n/a | n/a | n/a | +| Service | Nodes | Configuration | GCP | +|-----------------------------------------------------|-------|-------------------------|------------------| +| Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| PostgreSQL<sup>1</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | +| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Internal load balancing node<sup>3</sup> | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | +| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | +| Redis Sentinel - Queues / Shared State<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | +| Gitaly | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | +| Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | +| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Object storage<sup>4</sup> | n/a | n/a | n/a | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. -4. Should be run on reputable third party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. <!-- markdownlint-enable MD029 --> NOTE: @@ -2543,11 +2546,11 @@ documents how to apply the calculated configuration to the Helm Chart. #### Webservice Webservice pods typically need about 1 vCPU and 1.25 GB of memory _per worker_. -Each Webservice pod will consume roughly 4 vCPUs and 5 GB of memory using +Each Webservice pod consumes roughly 4 vCPUs and 5 GB of memory using the [recommended topology](#cluster-topology) because four worker processes are created by default and each pod has other small processes running. -For 50k users we recommend a total Puma worker count of around 320. +For 50,000 users we recommend a total Puma worker count of around 320. With the [provided recommendations](#cluster-topology) this allows the deployment of up to 80 Webservice pods with 4 workers per pod and 5 pods per node. Expand available resources using the ratio of 1 vCPU to 1.25 GB of memory _per each worker process_ for each additional diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index e57c4545b13..666d18a66fc 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -1,5 +1,4 @@ --- -reading_time: true stage: Enablement group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments @@ -21,22 +20,23 @@ costly-to-operate environment by using the > - **Supported users (approximate):** 5,000 > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) > - **Test requests per second (RPS) rates:** API: 100 RPS, Web: 10 RPS, Git (Pull): 10 RPS, Git (Push): 2 RPS +> - **[Latest 5k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/5k)** | Service | Nodes | Configuration | GCP | AWS | Azure | |--------------------------------------------|-------------|-------------------------|-----------------|--------------|----------| -| External load balancing node(3) | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Redis(2) | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | -| Consul(1) + Sentinel(2) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| PostgreSQL(1) | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| PgBouncer(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Internal load balancing node(3) | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| External load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Redis<sup>2</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | +| Consul<sup>1</sup> + Sentinel<sup>2</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| PostgreSQL<sup>1</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Gitaly | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | `D8s v3` | | Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Praefect PostgreSQL(1) | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Sidekiq | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | | GitLab Rails | 3 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | `c5.4xlarge` | `F16s v2`| | Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Object storage(4) | n/a | n/a | n/a | n/a | n/a | +| Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | n/a | | NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> @@ -44,7 +44,7 @@ costly-to-operate environment by using the 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. -4. Should be run on reputable third party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. <!-- markdownlint-enable MD029 --> NOTE: @@ -80,9 +80,9 @@ card "Database" as database { postgres_primary .[#4EA7FF]> postgres_secondary } -node "**Consul + Sentinel** x3" as consul_sentinel { - component Consul as consul #e76a9b - component Sentinel as sentinel #e6e727 +card "**Consul + Sentinel**" as consul_sentinel { + collections "**Consul** x3" as consul #e76a9b + collections "**Redis Sentinel** x3" as sentinel #e6e727 } card "Redis" as redis { @@ -143,7 +143,7 @@ is recommended instead of using NFS. Using an object storage service also doesn't require you to provision and maintain a node. It's also worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and -that to achieve full High Availability a third party PostgreSQL database solution will be required. +that to achieve full High Availability a third-party PostgreSQL database solution will be required. We hope to offer a built in solutions for these restrictions in the future but in the meantime a non HA PostgreSQL server can be set up via Omnibus GitLab, which the above specs reflect. Refer to the following issues for more information: [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398) @@ -338,7 +338,7 @@ Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`. The Internal Load Balancer is used to balance any internal connections the GitLab environment requires such as connections to [PgBouncer](#configure-pgbouncer) and [Praefect](#configure-praefect) (Gitaly Cluster). -Note that it's a separate node from the External Load Balancer and shouldn't have any access externally. +It's a separate node from the External Load Balancer and shouldn't have any access externally. The following IP will be used as an example: @@ -842,12 +842,12 @@ in the second step, do not supply the `EXTERNAL_URL` value. # Sets `max_replication_slots` to double the number of database nodes. # Patroni uses one extra slot per node when initiating the replication. - patroni['postgresql']['max_replication_slots'] = 8 + patroni['postgresql']['max_replication_slots'] = 6 # Set `max_wal_senders` to one more than the number of replication slots in the cluster. # This is used to prevent replication from using up all of the # available database connections. - patroni['postgresql']['max_wal_senders'] = 9 + patroni['postgresql']['max_wal_senders'] = 7 # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. patroni['postgresql']['max_connections'] = 500 @@ -874,7 +874,10 @@ in the second step, do not supply the `EXTERNAL_URL` value. patroni['username'] = '<patroni_api_username>' patroni['password'] = '<patroni_api_password>' - # Replace XXX.XXX.XXX.XXX/YY with Network Address + # Replace 10.6.0.0/24 with Network Addresses for your other patroni nodes + patroni['allowlist'] = %w(10.6.0.0/24 127.0.0.1/32) + + # Replace 10.6.0.0/24 with Network Address postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24 127.0.0.1/32) # Set the network addresses that the exporters will listen on for monitoring @@ -2072,7 +2075,7 @@ in Kubernetes via our official [Helm Charts](https://docs.gitlab.com/charts/). In this setup, we support running the equivalent of GitLab Rails and Sidekiq nodes in a Kubernetes cluster, named Webservice and Sidekiq respectively. In addition, the following other supporting services are supported: NGINX, Task Runner, Migrations, -Prometheus and Grafana. +Prometheus, and Grafana. Hybrid installations leverage the benefits of both cloud native and traditional compute deployments. With this, _stateless_ components can benefit from cloud native @@ -2083,23 +2086,23 @@ NOTE: This is an **advanced** setup. Running services in Kubernetes is well known to be complex. **This setup is only recommended** if you have strong working knowledge and experience in Kubernetes. The rest of this -section will assume this. +section assumes this. ### Cluster topology -The following tables and diagram details the hybrid environment using the same formats +The following tables and diagram detail the hybrid environment using the same formats as the normal environment above. -First starting with the components that run in Kubernetes. The recommendations at this -time use Google Cloud’s Kubernetes Engine (GKE) and associated machine types, but the memory +First are the components that run in Kubernetes. The recommendation at this time is to +use Google Cloud’s Kubernetes Engine (GKE) and associated machine types, but the memory and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. -| Service | Nodes(1) | Configuration | GCP | Allocatable CPUs and Memory | -|-------------------------------------------------------|----------|-------------------------|------------------|-----------------------------| -| Webservice | 5 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | 79.5 vCPU, 62 GB memory | -| Sidekiq | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | 11.8 vCPU, 38.9 GB memory | -| Supporting services such as NGINX, Prometheus, etc. | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | 3.9 vCPU, 11.8 GB memory | +| Service | Nodes<sup>1</sup> | Configuration | GCP | Allocatable CPUs and Memory | +|-------------------------------------------------------|-------------------|-------------------------|------------------|-----------------------------| +| Webservice | 5 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | 79.5 vCPU, 62 GB memory | +| Sidekiq | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | 11.8 vCPU, 38.9 GB memory | +| Supporting services such as NGINX, Prometheus | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | 3.9 vCPU, 11.8 GB memory | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> @@ -2112,22 +2115,22 @@ services where applicable): | Service | Nodes | Configuration | GCP | |--------------------------------------------|-------|-------------------------|------------------| -| Redis(2) | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | -| Consul(1) + Sentinel(2) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| PostgreSQL(1) | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| PgBouncer(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Internal load balancing node(3) | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Redis<sup>2</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | +| Consul<sup>1</sup> + Sentinel<sup>2</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| PostgreSQL<sup>1</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Gitaly | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | | Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Praefect PostgreSQL(1) | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Object storage(4) | n/a | n/a | n/a | +| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Object storage<sup>4</sup> | n/a | n/a | n/a | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. -4. Should be run on reputable third party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. <!-- markdownlint-enable MD029 --> NOTE: @@ -2150,9 +2153,9 @@ card "Kubernetes via Helm Charts" as kubernetes { card "**Internal Load Balancer**" as ilb #9370DB -node "**Consul + Sentinel** x3" as consul_sentinel { - component Consul as consul #e76a9b - component Sentinel as sentinel #e6e727 +card "**Consul + Sentinel**" as consul_sentinel { + collections "**Consul** x3" as consul #e76a9b + collections "**Redis Sentinel** x3" as sentinel #e6e727 } card "Gitaly Cluster" as gitaly_cluster { @@ -2221,11 +2224,11 @@ documents how to apply the calculated configuration to the Helm Chart. #### Webservice Webservice pods typically need about 1 vCPU and 1.25 GB of memory _per worker_. -Each Webservice pod will consume roughly 4 vCPUs and 5 GB of memory using +Each Webservice pod consumes roughly 4 vCPUs and 5 GB of memory using the [recommended topology](#cluster-topology) because four worker processes are created by default and each pod has other small processes running. -For 5k users we recommend a total Puma worker count of around 40. +For 5,000 users we recommend a total Puma worker count of around 40. With the [provided recommendations](#cluster-topology) this allows the deployment of up to 10 Webservice pods with 4 workers per pod and 2 pods per node. Expand available resources using the ratio of 1 vCPU to 1.25 GB of memory _per each worker process_ for each additional diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index 22871f6ea8d..74d8bf39d03 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -71,6 +71,8 @@ The following reference architectures are available: The following Cloud Native Hybrid reference architectures, where select recommended components can be run in Kubernetes, are available: +- [Up to 2,000 users](2k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) +- [Up to 3,000 users](3k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) - [Up to 5,000 users](5k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) - [Up to 10,000 users](10k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) - [Up to 25,000 users](25k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md index 68f351e737a..f4aed0f6a0f 100644 --- a/doc/administration/repository_storage_paths.md +++ b/doc/administration/repository_storage_paths.md @@ -120,17 +120,23 @@ For compatibility reasons `gitlab.yml` has a different structure than Omnibus Gi 1. [Restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. +1. [Configure where new repositories are stored](#configure-where-new-repositories-are-stored). + **For Omnibus installations** -Edit `/etc/gitlab/gitlab.rb` by appending the rest of the paths to the default one: +1. Edit `/etc/gitlab/gitlab.rb` by appending the rest of the paths to the default one: + + ```ruby + git_data_dirs({ + "default" => { "path" => "/var/opt/gitlab/git-data" }, + "storage1" => { "path" => "/mnt/storage1/git-data" }, + "storage2" => { "path" => "/mnt/storage2/git-data" } + }) + ``` + +1. [Restart GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -```ruby -git_data_dirs({ - "default" => { "path" => "/var/opt/gitlab/git-data" }, - "storage1" => { "path" => "/mnt/storage1/git-data" }, - "storage2" => { "path" => "/mnt/storage2/git-data" } -}) -``` +1. [Configure where new repositories are stored](#configure-where-new-repositories-are-stored). NOTE: Omnibus stores the repositories in a `repositories` subdirectory of the `git-data` directory. @@ -153,7 +159,18 @@ The higher the weight of a given repository storage path relative to other repos paths, the more often it is chosen. That is, `(storage weight) / (sum of all weights) * 100 = chance %`. +By default, if repository weights have not been configured earlier: + +- `default` is weighted `100`. +- All other storages are weighted `0`. + +NOTE: +If all storage weights are `0` (for example, when `default` does not exist), GitLab attempts to +create new repositories on `default`, regardless of the configuration or if `default` exists. +See [the tracking issue](https://gitlab.com/gitlab-org/gitlab/-/issues/36175) for more information. +information. + ## Move repositories To move a repository to a different repository storage (for example, from `default` to `storage2`), use the -same process as [migrating to Gitaly Cluster](gitaly/praefect.md#migrate-to-gitaly-cluster). +same process as [migrating to Gitaly Cluster](gitaly/index.md#migrate-to-gitaly-cluster). diff --git a/doc/administration/sidekiq.md b/doc/administration/sidekiq.md index 95fef1255af..e753832f2c3 100644 --- a/doc/administration/sidekiq.md +++ b/doc/administration/sidekiq.md @@ -94,6 +94,16 @@ you want using steps 1 and 2 from the GitLab downloads page. gitlab_exporter['enable'] = false ``` +1. If you're using the Container Registry and it's running on a different node than Sidekiq, then + configure the registry URL: + + ```ruby + registry_external_url 'https://registry.example.com' + gitlab_rails['registry_api_url'] = "https://registry.example.com" + ``` + + You must also copy the `registry.key` file to each Sidekiq node. + 1. Run `gitlab-ctl reconfigure`. You will need to restart the Sidekiq nodes after an update has occurred and database @@ -180,6 +190,10 @@ node_exporter['listen_address'] = '10.10.1.48:9100' # Rails Status for prometheus gitlab_rails['monitoring_whitelist'] = ['10.10.1.42', '127.0.0.1'] + +# Container Registry URL for cleanup jobs +registry_external_url 'https://registry.example.com' +gitlab_rails['registry_api_url'] = "https://registry.example.com" ``` ## Further reading diff --git a/doc/administration/static_objects_external_storage.md b/doc/administration/static_objects_external_storage.md index 48b98156b4f..2f19a2e5058 100644 --- a/doc/administration/static_objects_external_storage.md +++ b/doc/administration/static_objects_external_storage.md @@ -5,22 +5,23 @@ info: "To determine the technical writer assigned to the Stage/Group associated type: reference --- -# Static objects external storage **(FREE)** +# External storage for static objects **(FREE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/31025) in GitLab 12.3. -You can configure GitLab to serve repository static objects, like archives or raw blobs, -from an external storage, such as a Content Delivery Network (CDN). +Configure GitLab to serve repository static objects (such as archives or raw blobs) from external +storage such as a content delivery network (CDN). -## Configuring +## Configure external storage To configure external storage for static objects: 1. On the top bar, select **Menu >** **{admin}** **Admin**. 1. In the left sidebar, select **Settings > Repository**. -1. Expand the **Repository static objects** section. +1. Expand the **External storage for repository static objects** section. 1. Enter the base URL and an arbitrary token. When you [set up external storage](#set-up-external-storage), use a script that sets these values as `ORIGIN_HOSTNAME` and `STORAGE_TOKEN`. +1. Select **Save changes**. The token is required to distinguish requests coming from the external storage, so users don't circumvent the external storage and access the application directly. GitLab expects @@ -29,18 +30,23 @@ originating from the external storage. ## Serving private static objects -GitLab appends a user-specific token for static object URLs belonging to private projects, -so an external storage can be authenticated on the user's behalf. When processing requests originating -from the external storage, GitLab checks the following places to confirm the user may -access the requested object: +GitLab appends a user-specific token for static object URLs belonging to private projects so +external storage can be authenticated on the user's behalf. + +When processing requests originating +from the external storage, GitLab checks the following to confirm the user may access the requested +object: - The `token` query parameter. - The `X-Gitlab-Static-Object-Token` header. ## Requests flow example -The following example shows a sequence of requests and responses between the user, -GitLab, and the CDN: +The following example shows a sequence of requests and responses between: + +- The user. +- GitLab. +- The content delivery network. ```mermaid sequenceDiagram @@ -72,7 +78,7 @@ other CDNs or Function as a Service (FaaS) systems should work using the same pr - `ORIGIN_HOSTNAME`: the hostname of your GitLab installation. - `STORAGE_TOKEN`: any arbitrary secure token. You can get a token by running `pwgen -cn1 64` on a UNIX machine. Save this token for the Admin Area, as - described in the [configuring](#configuring) section. + described in the [configuring](#configure-external-storage) section. ```javascript const ORIGIN_HOSTNAME = 'gitlab.installation.com' // FIXME: SET CORRECT VALUE @@ -229,4 +235,4 @@ other CDNs or Function as a Service (FaaS) systems should work using the same pr 1. Create a new worker with this script. 1. Copy your values for `ORIGIN_HOSTNAME` and `STORAGE_TOKEN`. - Use those values [to configure external storage for static objects](#configuring). + Use those values [to configure external storage for static objects](#configure-external-storage). diff --git a/doc/administration/timezone.md b/doc/administration/timezone.md index bc94110c5b6..b6076c8ed26 100644 --- a/doc/administration/timezone.md +++ b/doc/administration/timezone.md @@ -44,15 +44,12 @@ gitlab-ctl restart ## Changing time zone per user -To allow users to change the time zone in their profile, the feature flag `user_time_settings` should be enabled: +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/57654) in GitLab 11.11, disabled by default behind `user_time_settings` [feature flag](feature_flags.md). +> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/29669) in GitLab 13.9. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/29669) in GitLab 14.1. -1. [Start a Rails console session](operations/rails_console.md). -1. Enable the feature flag: +A user can set their time zone in their profile. If a user has not set their time zone, it defaults +to the time zone [configured at the instance level](#changing-your-time-zone). On GitLab.com, the +default time zone is UTC. - ```ruby - Feature.enable(:user_time_settings) - ``` - -1. You should now be able to see the timezone dropdown in the users' **Settings > Profile** page. - -  +For more information, see [Set your time zone](../user/profile/index.md#set-your-time-zone). diff --git a/doc/administration/troubleshooting/debug.md b/doc/administration/troubleshooting/debug.md index 031f44b1f9f..aea891b8a77 100644 --- a/doc/administration/troubleshooting/debug.md +++ b/doc/administration/troubleshooting/debug.md @@ -45,7 +45,7 @@ session by running: ActiveRecord::Base.connection.execute('SET statement_timeout TO 0') ``` -Note that this change only affects the current Rails console session and will +This change only affects the current Rails console session and will not be persisted in the GitLab production environment or in the next Rails console session. @@ -213,7 +213,7 @@ downtime. Otherwise skip to the next section. exit ``` -Note that if the Puma process terminates before you are able to run these +If the Puma process terminates before you are able to run these commands, GDB will report an error. To buy more time, you can always raise the Puma worker timeout. For omnibus users, you can edit `/etc/gitlab/gitlab.rb` and increase it from 60 seconds to 600: @@ -285,5 +285,5 @@ The output in `/tmp/puma.txt` may help diagnose the root cause. ## More information -- [Debugging Stuck Ruby Processes](https://newrelic.com/blog/engineering/debugging-stuck-ruby-processes-what-to-do-before-you-kill-9/) +- [Debugging Stuck Ruby Processes](https://newrelic.com/blog/best-practices/debugging-stuck-ruby-processes-what-to-do-before-you-kill-9) - [Cheat sheet of using GDB and Ruby processes](gdb-stuck-ruby.txt) diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 08755dd3285..c55efe78216 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -16,13 +16,13 @@ and your [support options](https://about.gitlab.com/support/), before attempting this information. WARNING: -Please note that some of these scripts could be damaging if not run correctly, +Some of these scripts could be damaging if not run correctly, or under the right conditions. We highly recommend running them under the guidance of a Support Engineer, or running them in a test environment with a backup of the instance ready to be restored, just in case. WARNING: -Please also note that as GitLab changes, changes to the code are inevitable, +As GitLab changes, changes to the code are inevitable, and so some scripts may not work as they once used to. These are not kept up-to-date as these scripts/commands were added as they were found/needed. As mentioned above, we recommend running these scripts under the supervision of a @@ -466,7 +466,7 @@ sudo gitlab-rake cache:clear ### Export a repository -It's typically recommended to export a project through [the web interface](../../user/project/settings/import_export.md#exporting-a-project-and-its-data) or through [the API](../../api/project_import_export.md). In situations where this is not working as expected, it may be preferable to export a project directly via the Rails console: +It's typically recommended to export a project through [the web interface](../../user/project/settings/import_export.md#export-a-project-and-its-data) or through [the API](../../api/project_import_export.md). In situations where this is not working as expected, it may be preferable to export a project directly via the Rails console: ```ruby user = User.find_by_username('USERNAME') @@ -554,13 +554,28 @@ end ## Users +### Create new user + +```ruby +u = User.new(username: 'test_user', email: 'test@example.com', name: 'Test User', password: 'password', password_confirmation: 'password') +u.skip_confirmation! # Use it only if you wish user to be automatically confirmed. If skipped, user will recieve confirmation e-mail +u.save! +``` + ### Skip reconfirmation ```ruby -user = User.find_by_username '<username>' +user = User.find_by_username('<username>') user.skip_reconfirmation! ``` +### Disable 2fa for single user + +```ruby +user = User.find_by_username('<username>') +user.disable_two_factor! +``` + ### Active users & Historical users ```ruby diff --git a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md index bb0287df596..abeba514e4b 100644 --- a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md +++ b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md @@ -135,7 +135,7 @@ and they will assist you with any issues you are having. # source-style commands should also work cd /srv/gitlab && bundle exec rake gitlab:check RAILS_ENV=production - # run GitLab check. Note that the output can be confusing and invalid because of the specific structure of GitLab installed via helm chart + # run GitLab check. The output can be confusing and invalid because of the specific structure of GitLab installed via helm chart /usr/local/bin/gitlab-rake gitlab:check # open console without entering pod diff --git a/doc/administration/troubleshooting/log_parsing.md b/doc/administration/troubleshooting/log_parsing.md index c16302dd251..0a73c61ae94 100644 --- a/doc/administration/troubleshooting/log_parsing.md +++ b/doc/administration/troubleshooting/log_parsing.md @@ -60,7 +60,7 @@ zcat some_json.log.25.gz | (head -1; tail -1) | jq '.time' #### Find all requests with a 5XX status code ```shell -jq 'select(status >= 500)' <FILE> +jq 'select(.status >= 500)' <FILE> ``` #### Top 10 slowest requests diff --git a/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md b/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md index e55118d7309..57d64a2323e 100644 --- a/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md +++ b/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md @@ -265,7 +265,7 @@ user.save!(validate: false) This is not recommended, as validations are usually put in place to ensure the integrity and consistency of user-provided data. -Note that a validation error will prevent the entire object from being saved to +A validation error will prevent the entire object from being saved to the database. We'll see a little of this in the next section. If you're getting a mysterious red banner in the GitLab UI when submitting a form, this can often be the fastest way to get to the root of the problem. diff --git a/doc/administration/troubleshooting/ssl.md b/doc/administration/troubleshooting/ssl.md index 5ce4c7f0fb2..80be30a6509 100644 --- a/doc/administration/troubleshooting/ssl.md +++ b/doc/administration/troubleshooting/ssl.md @@ -199,7 +199,7 @@ To fix this problem: git config --global http.sslCAInfo ~/.ssl/gitlab.domain.tld.crt ``` - - Disable SSL verification in your Git client. Note that this intended as a + - Disable SSL verification in your Git client. This is intended as a temporary measure, as it could be considered a security risk. ```shell diff --git a/doc/administration/wikis/index.md b/doc/administration/wikis/index.md index bf6ff457ad3..01c175e014e 100644 --- a/doc/administration/wikis/index.md +++ b/doc/administration/wikis/index.md @@ -74,6 +74,14 @@ You can also use the API to [retrieve the current value](../../api/settings.md#g curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/application/settings" ``` +### Reduce wiki repository size + +The wiki counts as part of the [namespace storage size](../../user/admin_area/settings/account_and_limit_settings.md), +so you should keep your wiki repositories as compact as possible. + +For more information about tools to compact repositories, +read the documentation on [reducing repository size](../../user/project/repository/reducing_the_repo_size_using_git.md). + ## Related topics - [User documentation for wikis](../../user/project/wiki/index.md) |
