diff options
Diffstat (limited to 'doc/administration')
90 files changed, 1346 insertions, 1723 deletions
diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index 3ff5fb2635d..572c341f2b2 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -137,8 +137,6 @@ Project event queries are limited to a maximum of 30 days. ### Instance events **(PREMIUM SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2336) in GitLab 9.3. - Server-wide audit events introduce the ability to observe user actions across the entire instance of your GitLab server, making it easy to understand who changed what and when for audit purposes. diff --git a/doc/administration/auth/README.md b/doc/administration/auth/README.md deleted file mode 100644 index 5ab8653dc35..00000000000 --- a/doc/administration/auth/README.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'index.md' ---- - -This document was moved to [another location](index.md). - -<!-- This redirect file can be deleted after 2021-09-28. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/auth/index.md b/doc/administration/auth/index.md index a072cc73c43..959c5218554 100644 --- a/doc/administration/auth/index.md +++ b/doc/administration/auth/index.md @@ -31,7 +31,7 @@ providers: - [Salesforce](../../integration/salesforce.md) - [SAML](../../integration/saml.md) - [SAML for GitLab.com groups](../../user/group/saml_sso/index.md) **(PREMIUM SAAS)** -- [Shibboleth](../../integration/shibboleth.md) +- [Shibboleth](../../integration/saml.md) - [Smartcard](smartcard.md) **(PREMIUM SELF)** - [Twitter](../../integration/twitter.md) @@ -45,7 +45,7 @@ For more information, see the links shown on this page for each external provide | Capability | SaaS | Self-Managed | |-------------------------------------------------|-----------------------------------------|------------------------------------| -| **User Provisioning** | SCIM<br>JIT Provisioning | LDAP Sync | +| **User Provisioning** | SCIM<br>Just-In-Time (JIT) Provisioning | LDAP Sync | | **User Detail Updating** (not group management) | Not Available | LDAP Sync | | **Authentication** | SAML at top-level group (1 provider) | LDAP (multiple providers)<br>Generic OAuth2<br>SAML (only 1 permitted per unique provider)<br>Kerberos<br>JWT<br>Smartcard<br>OmniAuth Providers (only 1 permitted per unique provider) | | **Provider-to-GitLab Role Sync** | SAML Group Sync | LDAP Group Sync | diff --git a/doc/administration/auth/ldap/google_secure_ldap.md b/doc/administration/auth/ldap/google_secure_ldap.md index 137f35986ac..e5af8e8256a 100644 --- a/doc/administration/auth/ldap/google_secure_ldap.md +++ b/doc/administration/auth/ldap/google_secure_ldap.md @@ -15,7 +15,7 @@ LDAP service that can be configured with GitLab for authentication and group syn Secure LDAP requires a slightly different configuration than standard LDAP servers. The steps below cover: -- Configuring the Secure LDAP Client in the Google Admin console. +- Configuring the Secure LDAP Client in the Google administrator console. - Required GitLab configuration. ## Configuring Google LDAP client diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index 1992b450338..92815f10b92 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -5,7 +5,7 @@ group: Access info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# General LDAP setup **(FREE SELF)** +# Integrate LDAP with GitLab **(FREE SELF)** GitLab integrates with [LDAP](https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol) to support user authentication. @@ -39,7 +39,9 @@ the LDAP server, or share email addresses. ### User deletion Users deleted from the LDAP server are immediately blocked from signing in -to GitLab. However, there's an LDAP check cache time of one hour (which is +to GitLab and [no longer consumes a +license](../../../user/admin_area/moderate_users.md). +However, there's an LDAP check cache time of one hour (which is [configurable](#adjust-ldap-user-sync-schedule) for GitLab Premium users). This means users already signed-in or who are using Git over SSH can access GitLab for up to one hour. Manually block the user in the GitLab Admin Area @@ -70,15 +72,15 @@ LDAP email address, and then sign into GitLab by using their LDAP credentials. LDAP service that can be configured with GitLab for authentication and group sync. See [Google Secure LDAP](google_secure_ldap.md) for detailed configuration instructions. -## Configuration +## Configure LDAP -To enable LDAP integration you must add your LDAP server settings in -`/etc/gitlab/gitlab.rb` or `/home/git/gitlab/config/gitlab.yml` for Omnibus -GitLab and installations from source respectively. +To configure LDAP integration, add your LDAP server settings in: -There is a Rake task to check LDAP configuration. After configuring LDAP -using the documentation below, see [LDAP check Rake task](../../raketasks/check.md#ldap-check) -for information on the LDAP check Rake task. +- `/etc/gitlab/gitlab.rb` for Omnibus GitLab instances. +- `/home/git/gitlab/config/gitlab.yml` for source install instances. + +After configuring LDAP, to test the configuration, use the +[LDAP check Rake task](../../raketasks/check.md#ldap-check). NOTE: The `encryption` value `simple_tls` corresponds to 'Simple TLS' in the LDAP @@ -90,9 +92,9 @@ with `start_tls` and `ssl` was replaced with `simple_tls`. LDAP users must have a set email address, regardless of whether or not it's used to sign in. -### Example Configurations +### Example Omnibus GitLab configuration -**Omnibus Configuration** +This example shows configuration for Omnibus GitLab instances: ```ruby gitlab_rails['ldap_enabled'] = true @@ -139,7 +141,9 @@ gitlab_rails['ldap_servers'] = { } ``` -**Source Configuration** +### Example source install configuration + +This example shows configuration for source install instances: ```yaml production: @@ -155,6 +159,8 @@ production: ### Basic configuration settings +These configuration settings are available: + | Setting | Description | Required | Examples | |--------------------|-------------|----------|----------| | `label` | A human-friendly name for your LDAP server. It is displayed on your sign-in page. | **{check-circle}** Yes | `'Paris'` or `'Acme, Ltd.'` | @@ -183,6 +189,8 @@ Some examples of the `user_filter` field syntax: ### SSL configuration settings +These SSL configuration settings are available: + | Setting | Description | Required | Examples | |---------------|-------------|----------|----------| | `ca_file` | Specifies the path to a file containing a PEM-format CA certificate, for example, if you need an internal CA. | **{dotted-circle}** No | `'/etc/ca.pem'` | @@ -193,69 +201,72 @@ Some examples of the `user_filter` field syntax: ### Attribute configuration settings -LDAP attributes that GitLab uses to create an account for the LDAP user. The specified -attribute can either be the attribute name as a string (for example, `'mail'`), or an -array of attribute names to try in order (for example, `['mail', 'email']`). -The user's LDAP sign-in is the attribute specified as `uid` above. +GitLab uses these LDAP attributes to create an account for the LDAP user. The specified +attribute can be either: + +- The attribute name as a string. For example, `'mail'`. +- An array of attribute names to try in order. For example, `['mail', 'email']`. + +The user's LDAP sign in is the LDAP attribute [specified as `uid`](#basic-configuration-settings). | Setting | Description | Required | Examples | |--------------|-------------|----------|----------| -| `username` | The username is used in paths for the user's own projects (like `gitlab.example.com/username/project`) and when mentioning them in issues, merge request and comments (like `@username`). If the attribute specified for `username` contains an email address, the GitLab username is part of the email address before the `@`. | **{dotted-circle}** No | `['uid', 'userid', 'sAMAccountName']` | +| `username` | Used in paths for the user's own projects (for example, `gitlab.example.com/username/project`) and when mentioning them in issues, merge request and comments (for example, `@username`). If the attribute specified for `username` contains an email address, the GitLab username is part of the email address before the `@`. | **{dotted-circle}** No | `['uid', 'userid', 'sAMAccountName']` | | `email` | LDAP attribute for user email. | **{dotted-circle}** No | `['mail', 'email', 'userPrincipalName']` | | `name` | LDAP attribute for user display name. If `name` is blank, the full name is taken from the `first_name` and `last_name`. | **{dotted-circle}** No | Attributes `'cn'`, or `'displayName'` commonly carry full names. Alternatively, you can force the use of `first_name` and `last_name` by specifying an absent attribute such as `'somethingNonExistent'`. | | `first_name` | LDAP attribute for user first name. Used when the attribute configured for `name` does not exist. | **{dotted-circle}** No | `'givenName'` | | `last_name` | LDAP attribute for user last name. Used when the attribute configured for `name` does not exist. | **{dotted-circle}** No | `'sn'` | -### LDAP Sync configuration settings **(PREMIUM SELF)** +### LDAP sync configuration settings **(PREMIUM SELF)** + +These LDAP sync configuration settings are available: | Setting | Description | Required | Examples | |-------------------|-------------|----------|----------| | `group_base` | Base used to search for groups. | **{dotted-circle}** No | `'ou=groups,dc=gitlab,dc=example'` | -| `admin_group` | The CN of a group containing GitLab administrators. Note: Not `cn=administrators` or the full DN. | **{dotted-circle}** No | `'administrators'` | -| `external_groups` | An array of CNs of groups containing users that should be considered external. Note: Not `cn=interns` or the full DN. | **{dotted-circle}** No | `['interns', 'contractors']` | +| `admin_group` | The CN of a group containing GitLab administrators. Not `cn=administrators` or the full DN. | **{dotted-circle}** No | `'administrators'` | +| `external_groups` | An array of CNs of groups containing users that should be considered external. Not `cn=interns` or the full DN. | **{dotted-circle}** No | `['interns', 'contractors']` | | `sync_ssh_keys` | The LDAP attribute containing a user's public SSH key. | **{dotted-circle}** No | `'sshPublicKey'` or false if not set | ### Set up LDAP user filter -If you want to limit all GitLab access to a subset of the LDAP users on your -LDAP server, the first step should be to narrow the configured `base`. However, -it's sometimes necessary to further filter users. In this case, you can set -up an LDAP user filter. The filter must comply with -[RFC 4515](https://tools.ietf.org/search/rfc4515). +To limit all GitLab access to a subset of the LDAP users on your LDAP server, first narrow the +configured `base`. However, to further filter users if +necessary, you can set up an LDAP user filter. The filter must comply with [RFC 4515](https://tools.ietf.org/search/rfc4515). -**Omnibus configuration** +- Example user filter for Omnibus GitLab instances: -```ruby -gitlab_rails['ldap_servers'] = { -'main' => { - # snip... - 'user_filter' => '(employeeType=developer)' + ```ruby + gitlab_rails['ldap_servers'] = { + 'main' => { + # snip... + 'user_filter' => '(employeeType=developer)' + } } -} -``` + ``` -**Source configuration** +- Example user filter for source install instances: -```yaml -production: - ldap: - servers: - main: - # snip... - user_filter: '(employeeType=developer)' -``` + ```yaml + production: + ldap: + servers: + main: + # snip... + user_filter: '(employeeType=developer)' + ``` -If you want to limit access to the nested members of an Active Directory -group, use the following syntax: +To limit access to the nested members of an Active Directory group, use the following syntax: ```plaintext (memberOf:1.2.840.113556.1.4.1941:=CN=My Group,DC=Example,DC=com) ``` -For more information about this "LDAP_MATCHING_RULE_IN_CHAIN" filter, see the following -[Microsoft Search Filter Syntax](https://docs.microsoft.com/en-us/windows/win32/adsi/search-filter-syntax) document. +For more information about `LDAP_MATCHING_RULE_IN_CHAIN` filters, see +[Search Filter Syntax](https://docs.microsoft.com/en-us/windows/win32/adsi/search-filter-syntax). + Support for nested members in the user filter shouldn't be confused with -[group sync nested groups support](#supported-ldap-group-typesattributes). **(PREMIUM SELF)** +[group sync nested groups](#supported-ldap-group-typesattributes) support. GitLab does not support the custom filter syntax used by OmniAuth LDAP. @@ -451,7 +462,7 @@ If initially your LDAP configuration looked like: ### TLS server authentication -There are two encryption methods, `simple_tls` and `start_tls`. +`simple_tls` and `start_tls` are the two available encryption methods. For either encryption method, if setting `verify_certificates: false`, TLS encryption is established with the LDAP server before any LDAP-protocol data is @@ -463,9 +474,9 @@ exchanged but no validation of the LDAP server's SSL certificate is performed. Not implemented by `Net::LDAP`. -You should disable anonymous LDAP authentication and enable simple or SASL -authentication. The TLS client authentication setting in your LDAP server cannot -be mandatory and clients cannot be authenticated with the TLS protocol. +You should disable anonymous LDAP authentication and enable simple or Simple Authentication +and Security Layer (SASL) authentication. The TLS client authentication setting in your LDAP server +cannot be mandatory and clients cannot be authenticated with the TLS protocol. ## Multiple LDAP servers **(PREMIUM SELF)** @@ -474,7 +485,7 @@ connects to. To add another LDAP server: -1. Duplicate the settings under [the main configuration](#configuration). +1. Duplicate the settings under [the main configuration](#configure-ldap). 1. Edit them to match the additional LDAP server. Be sure to choose a different provider ID made of letters a-z and numbers 0-9. @@ -526,7 +537,7 @@ The process executes the following access checks: - Ensure the user is still present in LDAP. - If the LDAP server is Active Directory, ensure the user is active (not - blocked/disabled state). This is checked only if + blocked/disabled state). This check is performed only if `active_directory: true` is set in the LDAP configuration. In Active Directory, a user is marked as disabled/blocked if the user @@ -702,7 +713,7 @@ When enabled, the following applies: To enable it, you must: -1. [Enable LDAP](#configuration) +1. [Configure LDAP](#configure-ldap). 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > General**. 1. Expand the **Visibility and access controls** section. @@ -716,7 +727,7 @@ The values shown are in cron format. If needed, you can use a WARNING: Do not start the sync process too frequently as this -could lead to multiple syncs running concurrently. This is primarily a concern +could lead to multiple syncs running concurrently. This concern is primarily for installations with a large number of LDAP users. Review the [LDAP group sync benchmark metrics](#benchmarks) to see how your installation compares before proceeding. @@ -850,7 +861,7 @@ LDAP group links each: - Subsequent syncs (checking membership, no writes) took 15 minutes These metrics are meant to provide a baseline and performance may vary based on -any number of factors. This was an extreme benchmark and most instances don't +any number of factors. This benchmark was extreme and most instances don't have near this many users or groups. Disk speed, database performance, network and LDAP server response time affects these metrics. diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md index 1952e8afa97..4757725d0bd 100644 --- a/doc/administration/auth/ldap/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap/ldap-troubleshooting.md @@ -55,9 +55,8 @@ main: # 'main' is the GitLab 'provider ID' of this LDAP server #### Query LDAP **(PREMIUM SELF)** The following allows you to perform a search in LDAP using the rails console. -Depending on what you're trying to do, it may make more sense to query [a -user](#query-a-user-in-ldap) or [a group](#query-a-group-in-ldap) directly, or -even [use `ldapsearch`](#ldapsearch) instead. +Depending on what you're trying to do, it may make more sense to query [a user](#query-a-user-in-ldap) +or [a group](#query-a-group-in-ldap) directly, or even [use `ldapsearch`](#ldapsearch) instead. ```ruby adapter = Gitlab::Auth::Ldap::Adapter.new('ldapmain') @@ -90,7 +89,7 @@ established but GitLab doesn't show you LDAP users in the output, one of the following is most likely true: - The `bind_dn` user doesn't have enough permissions to traverse the user tree. -- The user(s) don't fall under the [configured `base`](index.md#configuration). +- The user(s) don't fall under the [configured `base`](index.md#configure-ldap). - The [configured `user_filter`](index.md#set-up-ldap-user-filter) blocks access to the user(s). In this case, you con confirm which of the above is true using @@ -102,7 +101,7 @@ In this case, you con confirm which of the above is true using A user can have trouble signing in for any number of reasons. To get started, here are some questions to ask yourself: -- Does the user fall under the [configured `base`](index.md#configuration) in +- Does the user fall under the [configured `base`](index.md#configure-ldap) in LDAP? The user must fall under this `base` to sign in. - Does the user pass through the [configured `user_filter`](index.md#set-up-ldap-user-filter)? If one is not configured, this question can be ignored. If it is, then the @@ -355,11 +354,10 @@ things to check to debug the situation. 1. Select the **Identities** tab. There should be an LDAP identity with an LDAP DN as the 'Identifier'. If not, this user hasn't signed in with LDAP yet and must do so first. -- You've waited an hour or [the configured - interval](index.md#adjust-ldap-group-sync-schedule) for the group to - sync. To speed up the process, either go to the GitLab group **Group information > Members** - and press **Sync now** (sync one group) or [run the group sync Rake - task](../../raketasks/ldap.md#run-a-group-sync) (sync all groups). +- You've waited an hour or [the configured interval](index.md#adjust-ldap-group-sync-schedule) for + the group to sync. To speed up the process, either go to the GitLab group **Group information > Members** + and press **Sync now** (sync one group) or [run the group sync Rake task](../../raketasks/ldap.md#run-a-group-sync) + (sync all groups). If all of the above looks good, jump in to a little more advanced debugging in the rails console. @@ -371,8 +369,8 @@ the rails console. 1. Look through the output of the sync. See [example log output](#example-console-output-after-a-group-sync) for how to read the output. -1. If you still aren't able to see why the user isn't being added, [query the - LDAP group directly](#query-a-group-in-ldap) to see what members are listed. +1. If you still aren't able to see why the user isn't being added, [query the LDAP group directly](#query-a-group-in-ldap) + to see what members are listed. 1. Is the user's DN or UID in one of the lists from the above output? One of the DNs or UIDs here should match the 'Identifier' from the LDAP identity checked earlier. If it doesn't, the user does not appear to be in the LDAP group. @@ -387,7 +385,7 @@ the following are true: - The configured `admin_group` in the `gitlab.rb` is a CN, rather than a DN or an array. - This CN falls under the scope of the configured `group_base`. - The members of the `admin_group` have already signed into GitLab with their LDAP - credentials. GitLab only grants this administrator access to the users whose + credentials. GitLab only grants the Administrator role to the users whose accounts are already connected to LDAP. If all the above are true and the users are still not getting access, [run a manual @@ -398,8 +396,8 @@ GitLab syncs the `admin_group`. #### Sync all groups NOTE: -To sync all groups manually when debugging is unnecessary, [use the Rake -task](../../raketasks/ldap.md#run-a-group-sync) instead. +To sync all groups manually when debugging is unnecessary, +[use the Rake task](../../raketasks/ldap.md#run-a-group-sync) instead. The output from a manual [group sync](index.md#group-sync) can show you what happens when GitLab syncs its LDAP group memberships against LDAP. diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md index 6a037e75f54..12729f2050b 100644 --- a/doc/administration/auth/oidc.md +++ b/doc/administration/auth/oidc.md @@ -228,8 +228,7 @@ Azure B2C [offers two ways of defining the business logic for logging in a user] While cumbersome to configure, custom policies are required because standard Azure B2C user flows [do not send the OpenID `email` claim](https://github.com/MicrosoftDocs/azure-docs/issues/16566). In -other words, they do not work with the [`allow_single_sign_on` or `auto_link_user` -parameters](../../integration/omniauth.md#initial-omniauth-configuration). +other words, they do not work with the [`allow_single_sign_on` or `auto_link_user` parameters](../../integration/omniauth.md#initial-omniauth-configuration). With a standard Azure B2C policy, GitLab cannot create a new account or link to an existing one with an email address. diff --git a/doc/administration/auth/smartcard.md b/doc/administration/auth/smartcard.md index 7e2699d5eb3..d79837776b2 100644 --- a/doc/administration/auth/smartcard.md +++ b/doc/administration/auth/smartcard.md @@ -126,7 +126,7 @@ more information, see [the relevant issue](https://gitlab.com/gitlab-org/gitlab/ gitlab_rails['smartcard_client_certificate_required_port'] = 3444 ``` - NOTE: **Note** + NOTE: Assign a value to at least one of the following variables: `gitlab_rails['smartcard_client_certificate_required_host']` or `gitlab_rails['smartcard_client_certificate_required_port']`. diff --git a/doc/administration/clusters/kas.md b/doc/administration/clusters/kas.md index 6afaff73396..226710a8911 100644 --- a/doc/administration/clusters/kas.md +++ b/doc/administration/clusters/kas.md @@ -104,7 +104,7 @@ In Omnibus GitLab, find the logs in `/var/log/gitlab/gitlab-kas/`. See also the [user documentation](../../user/clusters/agent/index.md#troubleshooting) for troubleshooting problems with individual agents. -### KAS logs - GitOps: failed to get project info +### KAS logs - GitOps: failed to get project information If you get the following error message: diff --git a/doc/administration/configure.md b/doc/administration/configure.md index d3e37b4a0ee..822acc1a74e 100644 --- a/doc/administration/configure.md +++ b/doc/administration/configure.md @@ -7,10 +7,44 @@ type: reference # Configure your GitLab installation **(FREE SELF)** -Customize and configure your self-managed GitLab installation. +Customize and configure your self-managed GitLab installation. Here are some quick links to get you started: - [Authentication](auth/index.md) - [Configuration](../user/admin_area/index.md) - [Repository storage](repository_storage_paths.md) - [Geo](geo/index.md) - [Packages](packages/index.md) + +The following tables are intended to guide you to choose the right combination of capabilties based on your requirements. It is common to want the most +available, quickly recoverable, highly performant and fully data resilient solution. However, there are tradeoffs. + +The tables lists features on the left and provides their capabilities to the right along with known trade-offs. + +## Gitaly Capabilities + +| | Availability | Recoverability | Data Resiliency | Performance | Risks/Trade-offs| +|-|--------------|----------------|-----------------|-------------|-----------------| +|Gitaly Cluster | Very high - tolerant of node failures | RTO for a single node of 10s with no manual intervention | Data is stored on multiple nodes | Good - While writes may take slightly longer due to voting, read distribution improves read speeds | **Trade-off** - Slight decrease in write speed for redundant, strongly-consistent storage solution. **Risks** - [Does not currently support snapshot backups](gitaly/index.md#snapshot-backup-and-recovery-limitations), GitLab backup task can be slow for large data sets | +|Gitaly Shards | Single storage location is a single point of failure | Would need to restore only shards which failed | Single point of failure | Good - can allocate repositories to shards to spread load | **Trade-off** - Need to manually configure repositories into different shards to balance loads / storage space **Risks** - Single point of failure relies on recovery process when single-node failure occurs | +|Gitaly + NFS | Single storage location is a single point of failure | Single node failure requires restoration from backup | Single point of failure | Average - NFS is not ideally suited to large quantities of small reads / writes which can have a detrimental impact on performance | **Trade-off** - Easy and familiar administration though NFS is not ideally suited to Git demands **Risks** - Many instances of NFS compatibility issues which provide very poor customer experiences | + +## Geo Capabilities + +If your availabity needs to span multiple zones or multiple locations, please read about [Geo](geo/index.md). + +| | Availability | Recoverability | Data Resiliency | Performance | Risks/Trade-offs| +|-|--------------|----------------|-----------------|-------------|-----------------| +|Geo| Depends on the architecture of the Geo site. It is possible to deploy secondaries in single and multiple node configurations. | Eventually consistent. Recovery point depends on replication lag, which depends on a number of factors such as network speeds. Geo supports failover from a primary to secondary site using manual commands that are scriptable. | Geo currently replicates 100% of planned data types and verifies 50%. See [limitations table](geo/replication/datatypes.md#limitations-on-replicationverification) for more detail. | Improves read/clone times for users of a secondary. | Geo is not intended to replace other backup/restore solutions. Because of replication lag and the possibility of replicating bad data from a primary, we recommend that customers also take regular backups of their primary site and test the restore process. | + +## Scenarios for failure modes and available mitigation paths + +The following table outlines failure modes and mitigation paths for the product offerings detailed in the tables above. Note - Gitaly Cluster install assumes an odd number replication factor of 3 or greater + +| Gitaly Mode | Loss of Single Gitaly Node | Application / Data Corruption | Regional Outage (Loss of Instance) | Notes | +| ----------- | -------------------------- | ----------------------------- | ---------------------------------- | ----- | +| Single Gitaly Node | Downtime - Must restore from backup | Downtime - Must restore from Backup | Downtime - Must wait for outage to end | | +| Single Gitaly Node + Geo Secondary | Downtime - Must restore from backup, can perform a manual failover to secondary | Downtime - Must restore from Backup, errors could have propagated to secondary | Manual intervention - failover to Geo secondary | | +| Sharded Gitaly Install | Partial Downtime - Only repos on impacted node affected, must restore from backup | Partial Downtime - Only repos on impacted node affected, must restore from backup | Downtime - Must wait for outage to end | | +| Sharded Gitaly Install + Geo Secondary | Partial Downtime - Only repos on impacted node affected, must restore from backup, could perform manual failover to secondary for impacted repos | Partial Downtime - Only repos on impacted node affected, must restore from backup, errors could have propagated to secondary | Manual intervention - failover to Geo secondary | | +| Gitaly Cluster Install* | No Downtime - will swap repository primary to another node after 10 seconds | N/A - All writes are voted on by multiple Gitaly Cluster nodes | Downtime - Must wait for outage to end | Snapshot backups for Gitaly Cluster nodes not supported at this time | +| Gitaly Cluster Install* + Geo Secondary | No Downtime - will swap repository primary to another node after 10 seconds | N/A - All writes are voted on by multiple Gitaly Cluster nodes | Manual intervention - failover to Geo secondary | Snapshot backups for Gitaly Cluster nodes not supported at this time | diff --git a/doc/administration/environment_variables.md b/doc/administration/environment_variables.md index 057abce0ed5..3af80363916 100644 --- a/doc/administration/environment_variables.md +++ b/doc/administration/environment_variables.md @@ -21,6 +21,7 @@ You can use the following environment variables to override certain values: |--------------------------------------------|---------|---------------------------------------------------------------------------------------------------------| | `DATABASE_URL` | string | The database URL; is of the form: `postgresql://localhost/blog_development`. | | `ENABLE_BOOTSNAP` | string | Enables Bootsnap for speeding up initial Rails boot (`1` to enable). | +| `EXTERNAL_URL` | string | Specify the external URL at the [time of installation](https://docs.gitlab.com/omnibus/settings/configuration.html#specifying-the-external-url-at-the-time-of-installation). | | `EXTERNAL_VALIDATION_SERVICE_TIMEOUT` | integer | Timeout, in seconds, for an [external CI/CD pipeline validation service](external_pipeline_validation.md). Default is `5`. | | `EXTERNAL_VALIDATION_SERVICE_URL` | string | URL to an [external CI/CD pipeline validation service](external_pipeline_validation.md). | | `EXTERNAL_VALIDATION_SERVICE_TOKEN` | string | The `X-Gitlab-Token` for authentication with an [external CI/CD pipeline validation service](external_pipeline_validation.md). | diff --git a/doc/administration/external_pipeline_validation.md b/doc/administration/external_pipeline_validation.md index 738cf591210..a4ed287cc3b 100644 --- a/doc/administration/external_pipeline_validation.md +++ b/doc/administration/external_pipeline_validation.md @@ -76,7 +76,8 @@ required number of seconds. "email": { "type": "string" }, "created_at": { "type": ["string", "null"], "format": "date-time" }, "current_sign_in_ip": { "type": ["string", "null"] }, - "last_sign_in_ip": { "type": ["string", "null"] } + "last_sign_in_ip": { "type": ["string", "null"] }, + "sign_in_count": { "type": "integer" } } }, "pipeline": { diff --git a/doc/administration/feature_flags.md b/doc/administration/feature_flags.md index 575fa9eb229..f2067e7a2d1 100644 --- a/doc/administration/feature_flags.md +++ b/doc/administration/feature_flags.md @@ -1,8 +1,7 @@ --- -stage: none -group: Development -info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" -type: reference +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments description: "GitLab administrator: enable and disable GitLab features deployed behind feature flags" --- diff --git a/doc/administration/file_hooks.md b/doc/administration/file_hooks.md index f73c961f541..e041f1e11c6 100644 --- a/doc/administration/file_hooks.md +++ b/doc/administration/file_hooks.md @@ -7,16 +7,15 @@ type: reference # File hooks **(FREE SELF)** -> - Introduced in GitLab 10.6. -> - Until GitLab 12.8, the feature name was Plugins. +> Renamed feature from Plugins to File hooks in GitLab 12.8. With custom file hooks, GitLab administrators can introduce custom integrations without modifying the GitLab source code. -NOTE: -Instead of writing and supporting your own file hook you can make changes -directly to the GitLab source code and contribute back upstream. This way we can -ensure functionality is preserved across versions and covered by tests. +A file hook runs on each event. You can filter events or projects +in a file hook's code, and create many file hooks as you need. Each file hook is +triggered by GitLab asynchronously in case of an event. For a list of events +see the [system hooks](../system_hooks/system_hooks.md) documentation. NOTE: File hooks must be configured on the file system of the GitLab server. Only GitLab @@ -24,10 +23,9 @@ server administrators can complete these tasks. Explore [system hooks](../system_hooks/system_hooks.md) or [webhooks](../user/project/integrations/webhooks.md) as an option if you do not have file system access. -A file hook runs on each event. You can filter events or projects -in a file hook's code, and create many file hooks as you need. Each file hook is -triggered by GitLab asynchronously in case of an event. For a list of events -see the [system hooks](../system_hooks/system_hooks.md) documentation. +Instead of writing and supporting your own file hook, you can also make changes +directly to the GitLab source code and contribute back upstream. In this way, we can +ensure functionality is preserved across versions and covered by tests. ## Setup @@ -67,7 +65,7 @@ message is logged to: - `log/file_hook.log` in a source installation. NOTE: -Before 14.0 release, the file name was `plugin.log` +In GitLab 13.12 and earlier, the filename was `plugin.log` ## Creating file hooks diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index a7a64701cbd..6312ed669ae 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -162,6 +162,9 @@ be disabled on the **primary** site: ## Finish replicating and verifying all data +NOTE: +GitLab 13.9 through GitLab 14.3 are affected by a bug in which the Geo secondary site statuses will appear to stop updating and become unhealthy. For more information, see [Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](../replication/troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). + 1. If you are manually replicating any data not managed by Geo, trigger the final replication process now. 1. On the **primary** node: @@ -192,12 +195,13 @@ At this point, your **secondary** node contains an up-to-date copy of everything ## Promote the **secondary** node -Finally, follow the [Disaster Recovery docs](index.md) to promote the -**secondary** node to a **primary** node. This process causes a brief outage on the **secondary** node, and users may need to log in again. +After the replication is finished, [promote the **secondary** node to a **primary** node](index.md). This process causes a brief outage on the **secondary** node, and users may need to log in again. If you follow the steps correctly, the old primary Geo site should still be disabled and user traffic should go to the newly-promoted site instead. -Once it is completed, the maintenance window is over! Your new **primary** node, now -begin to diverge from the old one. If problems do arise at this point, failing +When the promotion is completed, the maintenance window is over, and your new **primary** node now +begins to diverge from the old one. If problems do arise at this point, failing back to the old **primary** node [is possible](bring_primary_back.md), but likely to result in the loss of any data uploaded to the new **primary** in the meantime. -Don't forget to remove the broadcast message after failover is complete. +Don't forget to remove the broadcast message after the failover is complete. + +Finally, you can bring the [old site back as a secondary](bring_primary_back.md#configure-the-former-primary-node-to-be-a-secondary-node). diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md index 4255fba83f6..3eb7bc2a8e0 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md @@ -63,6 +63,9 @@ Before following any of those steps, make sure you have `root` access to the **secondary** to promote it, since there isn't provided an automated way to promote a Geo replica and perform a failover. +NOTE: +GitLab 13.9 through GitLab 14.3 are affected by a bug in which the Geo secondary site statuses will appear to stop updating and become unhealthy. For more information, see [Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](../../replication/troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). + On the **secondary** node: 1. On the top bar, select **Menu > Admin**. diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md index 18923da1056..d4782144df8 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md @@ -51,6 +51,9 @@ Before following any of those steps, make sure you have `root` access to the **secondary** to promote it, since there isn't provided an automated way to promote a Geo replica and perform a failover. +NOTE: +GitLab 13.9 through GitLab 14.3 are affected by a bug in which the Geo secondary site statuses will appear to stop updating and become unhealthy. For more information, see [Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](../../replication/troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). + On the **secondary** node, navigate to the **Admin Area > Geo** dashboard to review its status. Replicated objects (shown in green) should be close to 100%, and there should be no failures (shown in red). If a large proportion of diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index 3f38436429a..e8e87f92481 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -198,12 +198,12 @@ successfully, you must replicate their data using some other means. |[Package Registry](../../../user/packages/package_registry/index.md) | **Yes** (13.2) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | |[Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.12) | Via Object Storage provider if supported. Native Geo support (Beta). | Replication is behind the feature flag `geo_terraform_state_version_replication`, enabled by default. Verification was behind the feature flag `geo_terraform_state_version_verification`, which was removed in 14.0| |[External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Replication is behind the feature flag `geo_merge_request_diff_replication`, enabled by default. Verification is under development, behind the feature flag `geo_merge_request_diff_verification`, introduced in 14.0.| -|[Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [**Yes** (14.2)](https://gitlab.com/groups/gitlab-org/-/epics/2810) | No | Verification was implemented behind the feature flag `geo_snippet_repository_verification` in 13.11, and the feature flag was removed in 14.2. | -|[Server-side Git hooks](../../server_hooks.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1867) | No | No | | -|[Elasticsearch integration](../../../integration/elasticsearch.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | No | | +|[Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [**Yes** (14.2)](https://gitlab.com/groups/gitlab-org/-/epics/2810) | No | Verification was implemented behind the feature flag `geo_snippet_repository_verification` in 13.11, and the feature flag was removed in 14.2. | |[GitLab Pages](../../pages/index.md) | [**Yes** (14.3)](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_pages_deployment_replication`, enabled by default. | -|[Dependency proxy images](../../../user/packages/dependency_proxy/index.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/259694) | No | No | Blocked on [Geo: Secondary Mimicry](https://gitlab.com/groups/gitlab-org/-/epics/1528). Replication of this cache is not needed for Disaster Recovery purposes because it can be recreated from external sources. | -|[Vulnerability Export](../../../user/application_security/vulnerability_report/#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | | Not planned because they are ephemeral and sensitive. They can be regenerated on demand. | +|[Server-side Git hooks](../../server_hooks.md) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/1867) | No | No | Not planned because of current implementation complexity, low customer interest, and availability of alternatives to hooks. | +|[Elasticsearch integration](../../../integration/elasticsearch.md) | [Not planned](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | No | Not planned because further product discovery is required and Elasticsearch (ES) clusters can be rebuilt. Secondaries currently use the same ES cluster as the primary. | +|[Dependency proxy images](../../../user/packages/dependency_proxy/index.md) | [Not planned](https://gitlab.com/gitlab-org/gitlab/-/issues/259694) | No | No | Blocked by [Geo: Secondary Mimicry](https://gitlab.com/groups/gitlab-org/-/epics/1528). Replication of this cache is not needed for disaster recovery purposes because it can be recreated from external sources. | +|[Vulnerability Export](../../../user/application_security/vulnerability_report/#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | No | Not planned because they are ephemeral and sensitive information. They can be regenerated on demand. | #### Limitation of verification for files in Object Storage diff --git a/doc/administration/geo/replication/geo_validation_tests.md b/doc/administration/geo/replication/geo_validation_tests.md index c6b1078ddf0..a4c2f156216 100644 --- a/doc/administration/geo/replication/geo_validation_tests.md +++ b/doc/administration/geo/replication/geo_validation_tests.md @@ -114,6 +114,13 @@ The following are GitLab upgrade validation tests we performed. The following are PostgreSQL upgrade validation tests we performed. +### September 2021 + +[Verify Geo installation with PostgreSQL 13](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6131): + +- Description: With PostgreSQL 13 available as an opt-in version in GitLab 14.1, we tested fresh installations of GitLab with Geo when PostgreSQL 13 is enabled. +- Outcome: Successfully built an environment with Geo and PostgreSQL 13 using [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit) and performed Geo QA tests against the environment without failures. + ### September 2020 [Verify PostgreSQL 12 upgrade for Geo installations](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5454): diff --git a/doc/administration/geo/replication/multiple_servers.md b/doc/administration/geo/replication/multiple_servers.md index 7db210d31f4..87b1aa7fc44 100644 --- a/doc/administration/geo/replication/multiple_servers.md +++ b/doc/administration/geo/replication/multiple_servers.md @@ -199,7 +199,7 @@ then make the following modifications: ## `application_role` already enables this. You only need this line if ## you selectively enable individual services that depend on Rails, like - ## `puma`, `sidekiq`, `geo-logcursor`, etc. + ## `puma`, `sidekiq`, `geo-logcursor`, and so on. gitlab_rails['enable'] = true ## diff --git a/doc/administration/geo/replication/object_storage.md b/doc/administration/geo/replication/object_storage.md index 1f799b30125..3a10d3bad58 100644 --- a/doc/administration/geo/replication/object_storage.md +++ b/doc/administration/geo/replication/object_storage.md @@ -73,7 +73,7 @@ GitLab does not currently support the case where both: ## Third-party replication services When using Amazon S3, you can use -[CRR](https://docs.aws.amazon.com/AmazonS3/latest/dev/crr.html) to +[Cross-Region Replication (CRR)](https://docs.aws.amazon.com/AmazonS3/latest/dev/crr.html) to have automatic replication between the bucket used by the **primary** site and the bucket used by **secondary** sites. diff --git a/doc/administration/geo/replication/security_review.md b/doc/administration/geo/replication/security_review.md index 966902a3d74..df893298f85 100644 --- a/doc/administration/geo/replication/security_review.md +++ b/doc/administration/geo/replication/security_review.md @@ -26,7 +26,7 @@ from [owasp.org](https://owasp.org/). - Geo streams almost all data held by a GitLab instance between sites. This includes full database replication, most files (user-uploaded attachments, - etc) and repository + wiki data. In a typical configuration, this will + and so on) and repository + wiki data. In a typical configuration, this will happen across the public Internet, and be TLS-encrypted. - PostgreSQL replication is TLS-encrypted. - See also: [only TLSv1.2 should be supported](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/2948) diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 7b82d742bd5..b7370d32056 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -83,7 +83,7 @@ Checking Geo ... Finished #### Sync status Rake task Current sync information can be found manually by running this Rake task on any -**secondary** app node: +node running Rails (Puma, Sidekiq, or Geo Log Cursor) on the Geo **secondary** site: ```shell sudo gitlab-rake geo:status @@ -292,9 +292,8 @@ be set on the **primary** database. In GitLab 9.4, we have made this setting default to 1. You may need to increase this value if you have more **secondary** nodes. -Be sure to restart PostgreSQL for this to take -effect. See the [PostgreSQL replication -setup](../setup/database.md#postgresql-replication) guide for more details. +Be sure to restart PostgreSQL for this to take effect. See the +[PostgreSQL replication setup](../setup/database.md#postgresql-replication) guide for more details. ### Message: `FATAL: could not start WAL streaming: ERROR: replication slot "geo_secondary_my_domain_com" does not exist`? @@ -430,7 +429,7 @@ their resync may take a long time and cause significant load on your Geo nodes, storage and network systems. If you get the error `Synchronization failed - Error syncing repository` along with the following log messages, this indicates that the expected `geo` remote is not present in the `.git/config` file -of a repository on the secondary Geo node's filesystem: +of a repository on the secondary Geo node's file system: ```json { @@ -803,7 +802,7 @@ get_ctl_options': invalid option: --skip-preflight-checks (OptionParser::Invalid get_ctl_options': invalid option: --force (OptionParser::InvalidOption) ``` -This can happen with XFS or filesystems that list files in lexical order, because the +This can happen with XFS or file systems that list files in lexical order, because the load order of the Omnibus command files can be different than expected, and a global function would get redefined. More details can be found in [the related issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6076). @@ -923,6 +922,14 @@ To resolve this issue: If using a load balancer, ensure that the load balancer's URL is set as the `external_url` in the `/etc/gitlab/gitlab.rb` of the nodes behind the load balancer. +### Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode + +In GitLab 13.9 through GitLab 14.3, when [GitLab Maintenance Mode](../../maintenance_mode/index.md) is enabled, the status of Geo secondary sites will stop getting updated. After 10 minutes, the status will become `Unhealthy`. + +Geo secondary sites will continue to replicate and verify data, and the secondary sites should still be usable. You can use the [Sync status Rake task](#sync-status-rake-task) to determine the actual status of a secondary site during Maintenance Mode. + +This bug was [fixed in GitLab 14.4](https://gitlab.com/gitlab-org/gitlab/-/issues/292983). + ### GitLab Pages return 404 errors after promoting This is due to [Pages data not being managed by Geo](datatypes.md#limitations-on-replicationverification). diff --git a/doc/administration/geo/replication/version_specific_updates.md b/doc/administration/geo/replication/version_specific_updates.md index 84193e6baac..1b22a5f0991 100644 --- a/doc/administration/geo/replication/version_specific_updates.md +++ b/doc/administration/geo/replication/version_specific_updates.md @@ -13,6 +13,8 @@ for updating Geo nodes. ## Updating to 14.1, 14.2, 14.3 +### Multi-arch images + We found an [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/336013) where the Container Registry replication wasn't fully working if you used multi-arch images. In case of a multi-arch image, only the primary architecture (for example `amd64`) would be replicated to the secondary node. This has been [fixed in GitLab 14.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67624) and was backported to 14.2 and 14.1, but manual steps are required to force a re-sync. You can check if you are affected by running: @@ -46,18 +48,28 @@ Otherwise, on all your **secondary** nodes, in a [Rails console](../../operation If you are running a version prior to 14.1 and are using Geo and multi-arch containers in your Container Registry, we recommend [upgrading](updating_the_geo_sites.md) to at least GitLab 14.1. +### Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode + +GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) will cause Geo secondary site statuses to appear to stop updating and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). + ## Updating to GitLab 14.0/14.1 +### Primary sites can not be removed from the UI + We found an issue where [Primary sites can not be removed from the UI](https://gitlab.com/gitlab-org/gitlab/-/issues/338231). This bug only exists in the UI and does not block the removal of Primary sites using any other method. -### If you have already updated to an affected version and need to remove your Primary site +If you are running an affected version and need to remove your Primary site, you can manually remove the Primary site by using the [Geo Nodes API](../../../api/geo_nodes.md#delete-a-geo-node). -You can manually remove the Primary site by using the [Geo Nodes API](../../../api/geo_nodes.md#delete-a-geo-node). +### Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode + +GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) will cause Geo secondary site statuses to appear to stop updating and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). ## Updating to GitLab 13.12 +### Secondary nodes re-download all LFS files upon update + We found an issue where [secondary nodes re-download all LFS files](https://gitlab.com/gitlab-org/gitlab/-/issues/334550) upon update. This bug: - Only applies to Geo secondary sites that have replicated LFS objects. @@ -68,7 +80,7 @@ We found an issue where [secondary nodes re-download all LFS files](https://gitl If you don't have many LFS objects or can stand a bit of churn, then it is safe to let the secondary sites re-download LFS objects. If you do have many LFS objects, or many Geo secondary sites, or limited bandwidth, or a combination of them all, then we recommend you skip GitLab 13.12.0 through 13.12.6 and update to GitLab 13.12.7 or newer. -### If you have already updated to an affected version, and the re-sync is ongoing +#### If you have already updated to an affected version, and the re-sync is ongoing You can manually migrate the legacy sync state to the new state column by running the following command in a [Rails console](../../operations/rails_console.md). It should take under a minute: @@ -76,15 +88,31 @@ You can manually migrate the legacy sync state to the new state column by runnin Geo::LfsObjectRegistry.where(state: 0, success: true).update_all(state: 2) ``` +### Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode + +GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) will cause Geo secondary site statuses to appear to stop updating and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). + ## Updating to GitLab 13.11 We found an [issue with Git clone/pull through HTTP(s)](https://gitlab.com/gitlab-org/gitlab/-/issues/330787) on Geo secondaries and on any GitLab instance if maintenance mode is enabled. This was caused by a regression in GitLab Workhorse. This is fixed in the [GitLab 13.11.4 patch release](https://about.gitlab.com/releases/2021/05/14/gitlab-13-11-4-released/). To avoid this issue, upgrade to GitLab 13.11.4 or later. +### Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode + +GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) will cause Geo secondary site statuses to appear to stop updating and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). + +## Updating to GitLab 13.10 + +### Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode + +GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) will cause Geo secondary site statuses to appear to stop updating and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). + ## Updating to GitLab 13.9 +### Error during zero-downtime update: "cannot drop column asset_proxy_whitelist" + We've detected an issue [with a column rename](https://gitlab.com/gitlab-org/gitlab/-/issues/324160) that will prevent upgrades to GitLab 13.9.0, 13.9.1, 13.9.2 and 13.9.3 when following the zero-downtime steps. It is necessary -to perform the following additional steps for the zero-downtime upgrade: +to perform the following additional steps for the zero-downtime update: 1. Before running the final `sudo gitlab-rake db:migrate` command on the deploy node, execute the following queries using the PostgreSQL console (or `sudo gitlab-psql`) @@ -118,6 +146,10 @@ DETAIL: trigger trigger_0d588df444c8 on table application_settings depends on co To work around this bug, follow the previous steps to complete the update. More details are available [in this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/324160). +### Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode + +GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) will cause Geo secondary site statuses to appear to stop updating and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). + ## Updating to GitLab 13.7 We've detected an issue with the `FetchRemove` call used by Geo secondaries. diff --git a/doc/administration/get_started.md b/doc/administration/get_started.md index c0d5a45d8d1..2455aecafea 100644 --- a/doc/administration/get_started.md +++ b/doc/administration/get_started.md @@ -54,7 +54,7 @@ Get started: You may need to import projects from external sources like GitHub, Bitbucket, or another instance of GitLab. Many external sources can be imported into GitLab. - Review the [GitLab projects documentation](../user/project/index.md#project-integrations). -- Consider [repository mirroring](../user/project/repository/repository_mirroring.md)—an [alternative to project migrations](../ci/ci_cd_for_external_repos/index.md). +- Consider [repository mirroring](../user/project/repository/mirror/index.md)—an [alternative to project migrations](../ci/ci_cd_for_external_repos/index.md). - Check out our [migration index](../user/project/import/index.md) for documentation on common migration paths. - Schedule your project exports with our [import/export API](../api/project_import_export.md#schedule-an-export). @@ -128,7 +128,7 @@ The routine differs, depending on whether you deployed with Omnibus or the Helm When you backing up an Omnibus (single node) GitLab server, you can use a single Rake task. -Learn about [backing up Omnibus or Helm variations](../raketasks/backup_restore.md#back-up-gitlab). +Learn about [backing up Omnibus or Helm variations](../raketasks/backup_restore.md). This process backs up your entire instance, but does not back up the configuration files. Ensure those are backed up separately. Keep your configuration files and backup archives in a separate location to ensure the encryption keys are not kept with the encrypted data. @@ -214,7 +214,7 @@ If you use GitLab SaaS, you have several channels with which to get support and To get assistance for GitLab SaaS: -- Access [GitLab Docs](../README.md) for self-service support. +- Access [GitLab Docs](../index.md) for self-service support. - Join the [GitLab Forum](https://forum.gitlab.com/) for community support. - Gather [your subscription information](https://about.gitlab.com/support/#for-self-managed-users) before submitting a ticket. - Submit a support ticket for: diff --git a/doc/administration/gitaly/faq.md b/doc/administration/gitaly/faq.md index c7ecaa020e0..f79b9555c10 100644 --- a/doc/administration/gitaly/faq.md +++ b/doc/administration/gitaly/faq.md @@ -35,7 +35,7 @@ For more information, see: ## Are there instructions for migrating to Gitaly Cluster? -Yes! For more information, see [Migrate to Gitaly Cluster](index.md#migrate-to-gitaly-cluster). +Yes! For more information, see [Migrating to Gitaly Cluster](index.md#migrating-to-gitaly-cluster). ## What are some repository storage recommendations? diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 7a7aac884ed..c689530e12c 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -10,14 +10,22 @@ type: reference [Gitaly](https://gitlab.com/gitlab-org/gitaly) provides high-level RPC access to Git repositories. It is used by GitLab to read and write Git data. +Gitaly is present in every GitLab installation and coordinates Git repository +storage and retrieval. Gitaly can be: + +- A simple background service operating on a single instance Omnibus GitLab (all of + GitLab on one machine). +- Separated onto its own instance and configured in a full cluster configuration, + depending on scaling and availability requirements. + Gitaly implements a client-server architecture: - A Gitaly server is any node that runs Gitaly itself. -- A Gitaly client is any node that runs a process that makes requests of the Gitaly server. These - include, but are not limited to: - - [GitLab Rails application](https://gitlab.com/gitlab-org/gitlab). - - [GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell). - - [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse). +- A Gitaly client is any node that runs a process that makes requests of the Gitaly server. Gitaly clients are also known as _Gitaly consumers_ and include: + - [GitLab Rails application](https://gitlab.com/gitlab-org/gitlab) + - [GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell) + - [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) + - [GitLab Elasticsearch Indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer) Gitaly manages only Git repository access for GitLab. Other types of GitLab data aren't accessed using Gitaly. @@ -35,9 +43,49 @@ repository storage is either: - Read requests are distributed between multiple Gitaly nodes, which can improve performance. - Write requests are broadcast to repository replicas. -WARNING: -Engineering support for NFS for Git repositories is deprecated. Read the -[deprecation notice](#nfs-deprecation-notice). +## Guidance regarding Gitaly Cluster + +Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please review existing technical limitations and considerations prior to deploying Gitaly Cluster. + +- [Known issues](#known-issues) +- [Snapshot limitations](#snapshot-backup-and-recovery-limitations). + +Please also review the [configuration guidance](configure_gitaly.md) and [Repository storage options](../repository_storage_paths.md) to make sure that Gitaly Cluster is the best set-up for you. Finally, refer to the following guidance: + +- If you have not yet migrated to Gitaly Cluster and want to continue using NFS, remain on the + service you are using. NFS is supported in 14.x releases. +- If you have not yet migrated to Gitaly Cluster but want to migrate away from NFS, you have two options - a sharded Gitaly instance or Gitaly Cluster. +- If you have migrated to Gitaly Cluster and the limitations and tradeoffs are not suitable for your environment, your options are: + 1. [Migrate off Gitaly Cluster](#migrate-off-gitaly-cluster) back to your NFS solution + 1. [Migrate off Gitaly Cluster](#migrate-off-gitaly-cluster) to NFS solution or to a sharded Gitaly instance. + +Reach out to your Technical Account Manager or customer support if you have any questions. + +### Known issues + +The following table outlines current known issues impacting the use of Gitaly Cluster. For +the current status of these issues, please refer to the referenced issues and epics. + +| Issue | Summary | How to avoid | +|:--------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------| +| Gitaly Cluster + Geo - Issues retrying failed syncs | If Gitaly Cluster is used on a Geo secondary site, repositories that have failed to sync could continue to fail when Geo tries to resync them. Recovering from this state requires assistance from support to run manual steps. Work is in-progress to update Gitaly Cluster to [identify repositories with a unique and persistent identifier](https://gitlab.com/gitlab-org/gitaly/-/issues/3485), which is expected to resolve the issue. | No known solution at this time. | +| Database inconsistencies due to repository access outside of Gitaly Cluster's control | Operations that write to the repository storage that do not go through normal Gitaly Cluster methods can cause database inconsistencies. These can include (but are not limited to) snapshot restoration for cluster node disks, node upgrades which modify files under Git control, or any other disk operation that may touch repository storage external to GitLab. The Gitaly team is actively working to provide manual commands to [reconcile the Praefect database with the repository storage](https://gitlab.com/groups/gitlab-org/-/epics/6723). | Don't directly change repositories on any Gitaly Cluster node at this time. | +| Praefect unable to insert data into the database due to migrations not being applied after an upgrade | If the database is not kept up to date with completed migrations, then the Praefect node is unable to perform normal operation. | Make sure the Praefect database is up and running with all migrations completed (For example: `/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migrate-status` should show a list of all applied migrations). Consider [requesting live upgrade assistance](https://about.gitlab.com/support/scheduling-live-upgrade-assistance.html) so your upgrade plan can be reviewed by support. | +| Restoring a Gitaly Cluster node from a snapshot in a running cluster | Because the Gitaly Cluster runs with consistent state, introducing a single node that is behind will result in the cluster not being able to reconcile the nodes data and other nodes data | Don't restore a single Gitaly Cluster node from a backup snapshot. If you need to restore from backup, it's best to snapshot all Gitaly Cluster nodes at the same time and take a database dump of the Praefect database. | + +### Snapshot backup and recovery limitations + +Gitaly Cluster does not support snapshot backups because these can cause issues where the Praefect +database becomes out of sync with the disk storage. Because of how Praefect rebuilds the replication +metadata of Gitaly disk information during a restore, we recommend using the +[official backup and restore Rake tasks](../../raketasks/backup_restore.md). If you are unable to use this method, please contact customer support for restoration help. + +To track progress on work on a solution for manually re-synchronizing the Praefect database with +disk storage, see [this epic](https://gitlab.com/groups/gitlab-org/-/epics/6575). + +### What to do if you are on Gitaly Cluster experiencing an issue or limitation + +Please contact customer support for immediate help in restoration or recovery. ## Gitaly @@ -156,54 +204,6 @@ WARNING: If complete cluster failure occurs, disaster recovery plans should be executed. These can affect the RPO and RTO discussed above. -### Architecture and configuration recommendations - -The following table provides recommendations based on your -requirements. Users means concurrent users actively performing -simultaneous Git Operations. - -Gitaly services are present in every GitLab installation and always coordinates Git repository storage and -retrieval. Gitaly can be as simple as a single background service when operating on a single instance Omnibus -GitLab (All of GitLab on one machine). Gitaly can be separated into it's own instance and it can be configured in -a full cluster configuration depending on scaling and availability requirements. - -The GitLab Reference Architectures provide guidance for what Gitaly configuration is advisable at each of the scales. -The Gitaly configuration is noted by the architecture diagrams and the table of required resources. - -| User Scaling | Reference Architecture To Use | GitLab Instance Configuration | Gitaly Configuration | Git Repository Storage | Instances Dedicated to Gitaly Services | -| ------------------------------------------------------------ | ------------------------------------------------------------ | -------------------------------------------- | ------------------------------- | ---------------------------------- | -------------------------------------- | -| Up to 1000 Users | [1K](../reference_architectures/1k_users.md) | Single Instance for all of GitLab | Already Integrated as a Service | Local Disk | 0 | -| Up to 2999 Users | [2K](../reference_architectures/2k_users.md) | Horizontally Scaled GitLab Instance (Non-HA) | Single Gitaly Server | Local Disk of Gitaly Instance | 1 | -| 3000 Users and Over | [3K](../reference_architectures/1k_users.md) | Horizontally Scaled GitLab Instance with HA | Gitaly Cluster | Local Disk of Each Gitaly Instance | 8 | -| RTO/RPO Requirements for AZ Failure Tolerance Regardless of User Scale | [3K (with downscaling)](../reference_architectures/3k_users.md) | Custom (1) | Gitaly Cluster | Local Disk of Each Gitaly Instance | 8 | - -1. If you feel that you need AZ Failure Tolerance for user scaling lower than 3K, please contact Customer Success - to discuss your RTO and RPO needs and what options exist to meet those objectives. - -WARNING: -At present, some [known database inconsistency issues](#known-issues-impacting-gitaly-cluster) -exist in Gitaly Cluster. It is our recommendation that for now, you remain on your current service. -We will adjust the date for NFS support removal if this applies to you. - -### Known issues impacting Gitaly Cluster - -The following table outlines current known issues impacting the use of Gitaly Cluster. For -the most up to date status of these issues, please refer to the referenced issues / epics. - -| Issue | Summary | -| Gitaly Cluster + Geo can cause database inconsistencies | There are some conditions during Geo replication that can cause database inconsistencies with Gitaly Cluster. These have been identified and are being resolved by updating Gitaly Cluster to [identify repositories with a unique and persistent identifier](https://gitlab.com/gitlab-org/gitaly/-/issues/3485). | -| Database inconsistencies due to repository access outside of Gitaly Cluster's control | Operations that write to the repository storage which do not go through normal Gitaly Cluster methods can cause database inconsistencies. These can include (but are not limited to) snapshot restoration for cluster node disks, node upgrades which modify files under Git control, or any other disk operation that may touch repository storage external to GitLab. The Gitaly team is actively working to provide manual commands to [reconcile the Praefect database with the repository storage](https://gitlab.com/groups/gitlab-org/-/epics/6723). | - -### Snapshot backup and recovery limitations - -Gitaly Cluster does not support snapshot backups because these can cause issues where the -Praefect database becomes out of sync with the disk storage. Because of how Praefect rebuilds -the replication metadata of Gitaly disk information during a restore, we recommend using the -[official backup and restore Rake tasks](../../raketasks/backup_restore.md). - -To track progress on work on a solution for manually re-synchronizing the Praefect database -with disk storage, see [this epic](https://gitlab.com/groups/gitlab-org/-/epics/6575). - ### Virtual storage Virtual storage makes it viable to have a single repository storage in GitLab to simplify repository @@ -232,9 +232,7 @@ As with normal Gitaly storages, virtual storages can be sharded. ### Moving beyond NFS -WARNING: -Engineering support for NFS for Git repositories is deprecated. Technical support is planned to be -unavailable from GitLab 15.0. No further enhancements are planned for this feature. +Engineering support for NFS for Git repositories is deprecated. Technical support is planned to be unavailable starting GitLab 15.0. Please see our [statement of support](https://about.gitlab.com/support/statement-of-support.html#gitaly-and-nfs) for more details. [Network File System (NFS)](https://en.wikipedia.org/wiki/Network_File_System) is not well suited to Git workloads which are CPU and IOPS sensitive. @@ -355,13 +353,9 @@ For configuration information, see [Configure replication factor](praefect.md#co For more information on configuring Gitaly Cluster, see [Configure Gitaly Cluster](praefect.md). -## Migrate to Gitaly Cluster - -We recommend you migrate to Gitaly Cluster if your -[requirements recommend](#architecture-and-configuration-recommendations) Gitaly Cluster. +## Migrating to Gitaly Cluster -Whether migrating to Gitaly Cluster because of [NFS support deprecation](index.md#nfs-deprecation-notice) -or to move from single Gitaly nodes, the basic process involves: +Please see [current guidance on Gitaly Cluster](#guidance-regarding-gitaly-cluster). The basic process for migrating to Gitaly Cluster involves: 1. Create the required storage. Refer to [repository storage recommendations](faq.md#what-are-some-repository-storage-recommendations). @@ -371,9 +365,20 @@ or to move from single Gitaly nodes, the basic process involves: automatic migration but the moves can be scheduled with the GitLab API. WARNING: -At present, some [known database inconsistency issues](#known-issues-impacting-gitaly-cluster) -exist in Gitaly Cluster. It is our recommendation that for now, you remain on your current service. -We will adjust the date for NFS support removal if this applies to you. +Some [known database inconsistency issues](#known-issues) exist in Gitaly Cluster. We recommend you +remain on your current service for now. + +### Migrate off Gitaly Cluster + +If you have repositories stored on a Gitaly Cluster, but you'd like to migrate +them back to direct Gitaly storage: + +1. Create and configure a new + [Gitaly server](configure_gitaly.md#run-gitaly-on-its-own-server). +1. [Move the repositories](../operations/moving_repositories.md#move-repositories) + to the newly created storage. There are different possibilities to move them + by shard or by group, this gives you the opportunity to spread them over + multiple Gitaly servers. ## Monitor Gitaly and Gitaly Cluster @@ -615,20 +620,4 @@ The second facet presents the only real solution. For this, we developed ## NFS deprecation notice Engineering support for NFS for Git repositories is deprecated. Technical support is planned to be -unavailable from GitLab 15.0. No further enhancements are planned for this feature. - -Additional information: - -- [Recommended NFS mount options and known issues with Gitaly and NFS](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). -- [GitLab statement of support](https://about.gitlab.com/support/statement-of-support.html#gitaly-and-nfs). - -GitLab recommends: - -- Creating a [Gitaly Cluster](#gitaly-cluster) as soon as possible. -- [Moving your repositories](#migrate-to-gitaly-cluster) from NFS-based storage to Gitaly - Cluster. - -We welcome your feedback on this process. You can: - -- Raise a support ticket. -- [Comment on the epic](https://gitlab.com/groups/gitlab-org/-/epics/4916). +unavailable from GitLab 15.0. For further information, please see our [NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation) documentation. diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index eb666f1caf4..ce5fb1aaf88 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -265,8 +265,8 @@ praefect['database_direct_dbname'] = 'praefect_production' #praefect['database_direct_sslrootcert'] = '...' ``` -We recommend using PgBouncer with `session` pool mode instead. You can use the [bundled -PgBouncer](../postgresql/pgbouncer.md) or use an external PgBouncer and [configure it +We recommend using PgBouncer with `session` pool mode instead. You can use the +[bundled PgBouncer](../postgresql/pgbouncer.md) or use an external PgBouncer and [configure it manually](https://www.pgbouncer.org/config.html). The following example uses the bundled PgBouncer and sets up two separate connection pools, @@ -429,7 +429,7 @@ On the **Praefect** node: WARNING: If you have data on an already existing storage called `default`, you should configure the virtual storage with another name and - [migrate the data to the Gitaly Cluster storage](index.md#migrate-to-gitaly-cluster) + [migrate the data to the Gitaly Cluster storage](index.md#migrating-to-gitaly-cluster) afterwards. Replace `PRAEFECT_INTERNAL_TOKEN` with a strong secret, which is used by @@ -475,8 +475,8 @@ On the **Praefect** node: 1. [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2013) in GitLab 13.1 and later, enable [distribution of reads](index.md#distributed-reads). -1. Save the changes to `/etc/gitlab/gitlab.rb` and [reconfigure - Praefect](../restart_gitlab.md#omnibus-gitlab-reconfigure): +1. Save the changes to `/etc/gitlab/gitlab.rb` and + [reconfigure Praefect](../restart_gitlab.md#omnibus-gitlab-reconfigure): ```shell gitlab-ctl reconfigure @@ -499,16 +499,16 @@ On the **Praefect** node: running reconfigure automatically when running commands such as `apt-get update`. This way any additional configuration changes can be done and then reconfigure can be run manually. -1. Save the changes to `/etc/gitlab/gitlab.rb` and [reconfigure - Praefect](../restart_gitlab.md#omnibus-gitlab-reconfigure): +1. Save the changes to `/etc/gitlab/gitlab.rb` and + [reconfigure Praefect](../restart_gitlab.md#omnibus-gitlab-reconfigure): ```shell gitlab-ctl reconfigure ``` 1. To ensure that Praefect [has updated its Prometheus listen - address](https://gitlab.com/gitlab-org/gitaly/-/issues/2734), [restart - Praefect](../restart_gitlab.md#omnibus-gitlab-restart): + address](https://gitlab.com/gitlab-org/gitaly/-/issues/2734), + [restart Praefect](../restart_gitlab.md#omnibus-gitlab-restart): ```shell gitlab-ctl restart praefect @@ -695,8 +695,8 @@ Particular attention should be shown to: was set in the [previous section](#praefect). This document uses `gitaly-1`, `gitaly-2`, and `gitaly-3` as Gitaly storage names. -For more information on Gitaly server configuration, see our [Gitaly -documentation](configure_gitaly.md#configure-gitaly-servers). +For more information on Gitaly server configuration, see our +[Gitaly documentation](configure_gitaly.md#configure-gitaly-servers). 1. SSH into the **Gitaly** node and login as root: @@ -803,16 +803,16 @@ documentation](configure_gitaly.md#configure-gitaly-servers). }) ``` -1. Save the changes to `/etc/gitlab/gitlab.rb` and [reconfigure - Gitaly](../restart_gitlab.md#omnibus-gitlab-reconfigure): +1. Save the changes to `/etc/gitlab/gitlab.rb` and + [reconfigure Gitaly](../restart_gitlab.md#omnibus-gitlab-reconfigure): ```shell gitlab-ctl reconfigure ``` 1. To ensure that Gitaly [has updated its Prometheus listen - address](https://gitlab.com/gitlab-org/gitaly/-/issues/2734), [restart - Gitaly](../restart_gitlab.md#omnibus-gitlab-restart): + address](https://gitlab.com/gitlab-org/gitaly/-/issues/2734), + [restart Gitaly](../restart_gitlab.md#omnibus-gitlab-restart): ```shell gitlab-ctl restart gitaly @@ -893,7 +893,7 @@ Particular attention should be shown to: WARNING: If you have existing data stored on the default Gitaly storage, - you should [migrate the data your Gitaly Cluster storage](index.md#migrate-to-gitaly-cluster) + you should [migrate the data your Gitaly Cluster storage](index.md#migrating-to-gitaly-cluster) first. ```ruby @@ -1044,8 +1044,8 @@ To get started quickly: grafana['disable_login_form'] = false ``` -1. Save the changes to `/etc/gitlab/gitlab.rb` and [reconfigure - GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure): +1. Save the changes to `/etc/gitlab/gitlab.rb` and + [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure): ```shell gitlab-ctl reconfigure @@ -1309,12 +1309,7 @@ To minimize data loss in GitLab 13.0 to 14.0, Gitaly Cluster: new primary. If the failed primary contained unreplicated writes, [data loss can occur](#check-for-data-loss). > - Removed in GitLab 14.1. Instead, repositories [become unavailable](#unavailable-repositories). -In GitLab 13.0 to 14.0, when Gitaly Cluster switches to a new primary, repositories enter -read-only mode if they are out of date. This can happen after failing over to an outdated -secondary. Read-only mode eases data recovery efforts by preventing writes that may conflict -with the unreplicated writes on other nodes. - -When Gitaly Cluster switches to a new primary In GitLab 13.0 to 14.0, repositories enter +When Gitaly Cluster switches to a new primary in GitLab 13.0 to 14.0, repositories enter read-only mode if they are out of date. This can happen after failing over to an outdated secondary. Read-only mode eases data recovery efforts by preventing writes that may conflict with the unreplicated writes on other nodes. @@ -1596,3 +1591,26 @@ because of: - An in-flight RPC call targeting the repository. If this occurs, run `remove-repository` again. + +### Manually list untracked repositories + +> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3926) in GitLab 14.4. + +The `list-untracked-repositories` Praefect sub-command lists repositories of the Gitaly Cluster that both: + +- Exist for at least one Gitaly storage. +- Aren't tracked in the Praefect database. + +The command outputs: + +- Result to `STDOUT` and the command's logs. +- Errors to `STDERR`. + +Each entry is a complete JSON string with a newline at the end (configurable using the +`-delimiter` flag). For example: + +```shell +sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml list-untracked-repositories +{"virtual_storage":"default","storage":"gitaly-1","relative_path":"@hashed/ab/cd/abcd123456789012345678901234567890123456789012345678901234567890.git"} +{"virtual_storage":"default","storage":"gitaly-1","relative_path":"@hashed/ab/cd/abcd123456789012345678901234567890123456789012345678901234567891.git"} +``` diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index 1b53a0994f5..1f90ebb7565 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -243,7 +243,7 @@ To mirror-push branches and tags only, and avoid attempting to mirror-push prote git push origin +refs/heads/*:refs/heads/* +refs/tags/*:refs/tags/* ``` -Any other namespaces that the admin wants to push can be included there as well via additional patterns. +Any other namespaces that the administrator wants to push can be included there as well via additional patterns. ### Command line tools cannot connect to Gitaly @@ -365,6 +365,15 @@ To determine the current primary Gitaly node for a specific Praefect node: curl localhost:9652/metrics | grep gitaly_praefect_primaries` ``` +### Check that repositories are in sync + +Is [some cases](index.md#known-issues) the Praefect database can get out of sync with the underlying Gitaly nodes. To check that +a given repository is fully synced on all nodes, run the [`gitlab:praefect:replicas` Rake task](../raketasks/praefect.md#replica-checksums) +that checksums the repository on all Gitaly nodes. + +The [Praefect dataloss](praefect.md#check-for-data-loss) command only checks the state of the repo in the Praefect database, and cannot +be relied to detect sync problems in this scenario. + ### Relation does not exist errors By default Praefect database tables are created automatically by `gitlab-ctl reconfigure` task. @@ -393,7 +402,7 @@ $ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config praefect sql-migrate: OK (applied 21 migrations) ``` -### Requests fail with 'repo scoped: invalid Repository' errors +### Requests fail with 'repository scoped: invalid Repository' errors This indicates that the virtual storage name used in the [Praefect configuration](praefect.md#praefect) does not match the storage name used in diff --git a/doc/administration/index.md b/doc/administration/index.md index 9412994edb7..ee17edc35fc 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -41,7 +41,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. ### Configuring GitLab -- [Adjust your instance's timezone](timezone.md): Customize the default time zone of GitLab. +- [Adjust your instance's time zone](timezone.md): Customize the default time zone of GitLab. - [System hooks](../system_hooks/system_hooks.md): Notifications when users, projects and keys are changed. - [Security](../security/index.md): Learn what you can do to further secure your GitLab instance. - [Usage statistics, version check, and Service Ping](../user/admin_area/settings/usage_statistics.md): Enable or disable information about your instance to be sent to GitLab, Inc. diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index 24ffee088f3..a2729e60545 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -88,6 +88,29 @@ requests per user. For more information, read - **Default rate limit**: Disabled by default. +### Files API + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68561) in GitLab 14.3. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, +ask an administrator to [enable the `files_api_throttling` flag](../administration/feature_flags.md). On GitLab.com, this feature is available but can be configured by GitLab.com administrators only. +The feature is not ready for production use. + +This setting limits the request rate on the Packages API per user or IP address. For more information, read +[Files API rate limits](../user/admin_area/settings/files_api_rate_limits.md). + +- **Default rate limit**: Disabled by default. + +### Deprecated API endpoints + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68645) in GitLab 14.4. + +This setting limits the request rate on deprecated API endpoints per user or IP address. For more information, read +[Deprecated API rate limits](../user/admin_area/settings/deprecated_api_rate_limits.md). + +- **Default rate limit**: Disabled by default. + ### Import/Export > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35728) in GitLab 13.2. @@ -212,7 +235,7 @@ Activity history for projects and individuals' profiles was limited to one year > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14939) in GitLab 12.7. -There is a limit when embedding metrics in GFM for performance reasons. +There is a limit when embedding metrics in GitLab Flavored Markdown (GFM) for performance reasons. - **Max limit**: 100 embeds. @@ -240,10 +263,10 @@ Set the limit to `0` to disable it. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/237891) in GitLab 13.7. -The [minimum wait time between pull refreshes](../user/project/repository/repository_mirroring.md) +The [minimum wait time between pull refreshes](../user/project/repository/mirror/index.md) defaults to 300 seconds (5 minutes). For example, by default a pull refresh will only run once in a given 300 second period regardless of how many times you try to trigger it. -This setting applies in the context of pull refreshes invoked via the [projects API](../api/projects.md#start-the-pull-mirroring-process-for-a-project), or when forcing an update by selecting the **Update now** (**{retry}**) button within **Settings > Repository > Mirroring repositories**. This setting has no effect on the automatic 30 minute interval schedule used by Sidekiq for [pull mirroring](../user/project/repository/repository_mirroring.md#how-it-works). +This setting applies in the context of pull refreshes invoked via the [projects API](../api/projects.md#start-the-pull-mirroring-process-for-a-project), or when forcing an update by selecting the **Update now** (**{retry}**) button within **Settings > Repository > Mirroring repositories**. This setting has no effect on the automatic 30 minute interval schedule used by Sidekiq for [pull mirroring](../user/project/repository/mirror/pull.md). To change this limit for a self-managed installation, run the following in the [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): @@ -511,7 +534,11 @@ Plan.default.actual_limits.update!(pages_file_entries: 100) ### Number of registered runners per scope -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/321368) in GitLab 13.12. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/321368) in GitLab 13.12. Disabled by default. +> - Enabled on GitLab.com in GitLab 14.3. +> - Enabled on self-managed in GitLab 14.4. +> - Feature flag `ci_runner_limits` removed in GitLab 14.4. You can still use `ci_runner_limits_override` + to remove limits for a given scope. The total number of registered runners is limited at the group and project levels. Each time a new runner is registered, GitLab checks these limits against runners that have been active in the last 3 months. @@ -749,7 +776,7 @@ than the specified limit, hooks won't be executed. More information can be found in these docs: -- [Webhooks push events](../user/project/integrations/webhooks.md#push-events) +- [Webhooks push events](../user/project/integrations/webhook_events.md#push-events) - [Project services push hooks limit](../user/project/integrations/overview.md#push-hooks-limit) ### Activities diff --git a/doc/administration/instance_review.md b/doc/administration/instance_review.md index b166bb32aa1..62897651166 100644 --- a/doc/administration/instance_review.md +++ b/doc/administration/instance_review.md @@ -12,7 +12,7 @@ If you run a medium-sized self-managed instance (50+ users) of a free version of [either Community Edition or unlicensed Enterprise Edition](https://about.gitlab.com/install/ce-or-ee/), you qualify for a free Instance Review. -1. Sign in as a user with administrator [permissions](../user/permissions.md). +1. Sign in as a user with Administrator [role](../user/permissions.md). 1. In the top menu, click your user icon, and select **Get a free instance review**: diff --git a/doc/administration/integration/terminal.md b/doc/administration/integration/terminal.md index 882580b35b6..45b94781adc 100644 --- a/doc/administration/integration/terminal.md +++ b/doc/administration/integration/terminal.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Web terminals **(FREE)** -With the introduction of the [Kubernetes integration](../../user/project/clusters/index.md), +With the introduction of the [Kubernetes integration](../../user/infrastructure/clusters/index.md), GitLab can store and use credentials for a Kubernetes cluster. GitLab uses these credentials to provide access to [web terminals](../../ci/environments/index.md#web-terminals) for environments. @@ -50,8 +50,8 @@ detail below. ## Enabling and disabling terminal support NOTE: -AWS Elastic Load Balancers (ELBs) do not support web sockets. -If you want web terminals to work, use AWS Application Load Balancers (ALBs). +AWS Classic Load Balancers (CLBs) do not support web sockets. +If you want web terminals to work, use AWS Network Load Balancers (NLBs). Read [AWS Elastic Load Balancing Product Comparison](https://aws.amazon.com/elasticloadbalancing/features/#compare) for more information. diff --git a/doc/administration/invalidate_markdown_cache.md b/doc/administration/invalidate_markdown_cache.md index 7a880c81843..855910fec6a 100644 --- a/doc/administration/invalidate_markdown_cache.md +++ b/doc/administration/invalidate_markdown_cache.md @@ -15,8 +15,8 @@ in the cached text would still refer to the old URL. To avoid this problem, the administrator can invalidate the existing cache by increasing the `local_markdown_version` setting in application settings. This can -be done by [changing the application settings through -the API](../api/settings.md#change-application-settings): +be done by changing the application settings +[through the API](../api/settings.md#change-application-settings): ```shell curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/application/settings?local_markdown_version=<increased_number>" diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index b4c16e007cc..46a9ee11679 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -446,10 +446,10 @@ List the artifacts for a single project, sorted by artifact size. The output inc - on-disk location of the artifact ```ruby -p = Project.find_by_id(:project ID) +p = Project.find_by_id(<project_id>) arts = Ci::JobArtifact.where(project: p) -list = arts.order('sort DESC').limit(50).each do |art| +list = arts.order(size: :desc).limit(50).each do |art| puts "Job ID: #{art.job_id} - Size: #{art.size}b - Type: #{art.file_type} - Created: #{art.created_at} - File loc: #{art.file}" end ``` diff --git a/doc/administration/job_logs.md b/doc/administration/job_logs.md index 64d9248cb16..f2748305c24 100644 --- a/doc/administration/job_logs.md +++ b/doc/administration/job_logs.md @@ -146,9 +146,9 @@ a background job archives the job log. The log is moved to `/var/opt/gitlab/gitl by default, or to object storage if configured. In a [scaled-out architecture](reference_architectures/index.md) with Rails and Sidekiq running on more than one -server, these two locations on the filesystem have to be shared using NFS. +server, these two locations on the file system have to be shared using NFS. -To eliminate both filesystem requirements: +To eliminate both file system requirements: - [Enable the incremental logging feature](#enable-or-disable-incremental-logging), which uses Redis instead of disk space for temporary caching of job logs. - Configure [object storage](job_artifacts.md#object-storage-settings) for storing archived job logs. diff --git a/doc/administration/lfs/index.md b/doc/administration/lfs/index.md index 682352d8f59..d2f220e3795 100644 --- a/doc/administration/lfs/index.md +++ b/doc/administration/lfs/index.md @@ -2,18 +2,15 @@ stage: Create group: Source Code info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" -type: reference, howto disqus_identifier: 'https://docs.gitlab.com/ee/workflow/lfs/lfs_administration.html' --- # GitLab Git Large File Storage (LFS) Administration **(FREE SELF)** -> - Git LFS is supported in GitLab starting with version 8.2. -> - Support for object storage, such as AWS S3, was introduced in 10.0. -> - LFS is enabled in GitLab self-managed instances by default. - Documentation about how to use Git LFS are under [Managing large binary files with Git LFS doc](../../topics/git/lfs/index.md). +LFS is enabled in GitLab self-managed instances by default. + ## Requirements - Users need to install [Git LFS client](https://git-lfs.github.com) version 1.0.1 or later. @@ -346,8 +343,6 @@ git lfs version ## Known limitations -- Support for removing unreferenced LFS objects was added in 8.14 onward. -- LFS authentications via SSH was added with GitLab 8.12. - Only compatible with the Git LFS client versions 1.1.0 and later, or 1.0.2. - The storage statistics count each LFS object for every project linking to it. diff --git a/doc/administration/logs.md b/doc/administration/logs.md index 990287e3907..a9fd698a525 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -1069,6 +1069,14 @@ For Omnibus GitLab installations, Praefect logs are in `/var/log/gitlab/praefect GitLab also tracks [Prometheus metrics for Praefect](gitaly/#monitor-gitaly-cluster). +## Backup log + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63832) in GitLab 14.1. + +For Omnibus installations, the backup log is located at `/var/log/gitlab/gitlab-rails/backup_json.log`. + +This log is populated when a [GitLab backup is created](../raketasks/backup_restore.md). You can use this log to understand how the backup process performed. + ## Performance bar stats > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/48149) in GitLab 13.7. @@ -1082,7 +1090,7 @@ Performance bar statistics (currently only duration of SQL queries) are recorded in that file. For example: ```json -{"severity":"INFO","time":"2020-12-04T09:29:44.592Z","correlation_id":"33680b1490ccd35981b03639c406a697","filename":"app/models/ci/pipeline.rb","method_path":"app/models/ci/pipeline.rb:each_with_object","request_id":"rYHomD0VJS4","duration_ms":26.889,"count":2,"type": "sql"} +{"severity":"INFO","time":"2020-12-04T09:29:44.592Z","correlation_id":"33680b1490ccd35981b03639c406a697","filename":"app/models/ci/pipeline.rb","method_path":"app/models/ci/pipeline.rb:each_with_object","request_id":"rYHomD0VJS4","duration_ms":26.889,"count":2,"query_type": "active-record"} ``` These statistics are logged on .com only, disabled on self-deployments. diff --git a/doc/administration/maintenance_mode/index.md b/doc/administration/maintenance_mode/index.md index 39ee357cc2f..d5bcd132665 100644 --- a/doc/administration/maintenance_mode/index.md +++ b/doc/administration/maintenance_mode/index.md @@ -75,7 +75,7 @@ An error is displayed when a user tries to perform a write operation that isn't NOTE: In some cases, the visual feedback from an action could be misleading, for example when starring a project, the **Star** button changes to show the **Unstar** action, however, this is only the frontend update, and it doesn't take into account the failed status of the POST request. These visual bugs are to be fixed [in follow-up iterations](https://gitlab.com/gitlab-org/gitlab/-/issues/295197). -### Admin functions +### Administrator functions Systems administrators can edit the application settings. This allows them to disable Maintenance Mode after it's been enabled. @@ -111,18 +111,18 @@ For most JSON requests, POST, PUT, PATCH, and DELETE are blocked, and the API re |HTTP request | Allowed routes | Notes | |:----:|:--------------------------------------:|:----:| -| POST | `/admin/application_settings/general` | To allow updating application settings in the admin UI | +| POST | `/admin/application_settings/general` | To allow updating application settings in the administrator UI | | PUT | `/api/v4/application/settings` | To allow updating application settings with the API | | POST | `/users/sign_in` | To allow users to log in. | | POST | `/users/sign_out`| To allow users to log out. | | POST | `/oauth/token` | To allow users to log in to a Geo secondary for the first time. | -| POST | `/admin/session`, `/admin/session/destroy` | To allow [Admin mode for GitLab administrators](https://gitlab.com/groups/gitlab-org/-/epics/2158) | +| POST | `/admin/session`, `/admin/session/destroy` | To allow [Administrator mode for GitLab administrators](https://gitlab.com/groups/gitlab-org/-/epics/2158) | | POST | Paths ending with `/compare`| Git revision routes. | | POST | `.git/git-upload-pack` | To allow Git pull/clone. | | POST | `/api/v4/internal` | [internal API routes](../../development/internal_api.md) | -| POST | `/admin/sidekiq` | To allow management of background jobs in the admin UI | -| POST | `/admin/geo` | To allow updating Geo Nodes in the admin UI | -| POST | `/api/v4/geo_replication`| To allow certain Geo-specific admin UI actions on secondary sites | +| POST | `/admin/sidekiq` | To allow management of background jobs in the Admin UI | +| POST | `/admin/geo` | To allow updating Geo Nodes in the administrator UI | +| POST | `/api/v4/geo_replication`| To allow certain Geo-specific administrator UI actions on secondary sites | ### GraphQL API diff --git a/doc/administration/monitoring/performance/img/performance_bar_v14_0.png b/doc/administration/monitoring/performance/img/performance_bar_v14_0.png Binary files differdeleted file mode 100644 index 42261ddd720..00000000000 --- a/doc/administration/monitoring/performance/img/performance_bar_v14_0.png +++ /dev/null diff --git a/doc/administration/monitoring/performance/img/performance_bar_v14_4.png b/doc/administration/monitoring/performance/img/performance_bar_v14_4.png Binary files differnew file mode 100644 index 00000000000..388d628c7c4 --- /dev/null +++ b/doc/administration/monitoring/performance/img/performance_bar_v14_4.png diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index ef4db93d5fc..0befd9eac5b 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -8,11 +8,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - The **Stats** field [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/271551) in GitLab 13.9. > - The **Memory** field [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/330736) in GitLab 14.0. +> - The **Flamegraph** field [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30275) in GitLab 14.4. You can display the performance bar to see statistics for the performance of a GitLab UI page. For example: - + ## Available information @@ -64,6 +65,11 @@ From left to right, the performance bar displays: can be added by its full URL (authenticated as the current user), or by the value of its `X-Request-Id` header. - **Download**: a link to download the raw JSON used to generate the Performance Bar reports. +- **Flamegraph** with mode: a link to generate a [flamegraph](../../../development/profiling.md#speedscope-flamegraphs) + of the current URL with the selected [Stackprof mode](https://github.com/tmm1/stackprof#sampling): + - The **Wall** mode samples every *interval* of the time on a clock on a wall. The interval is set to `10100` microseconds. + - The **CPU** mode samples every *interval* of CPU activity. The interval is set to `10100` microseconds. + - The **Object** mode samples every *interval*. The interval is set to `100` allocations. - **Request Selector**: a select box displayed on the right-hand side of the Performance Bar which enables you to view these metrics for any requests made while the current page was open. Only the first two requests per unique URL are captured. diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index c36d2b0f7a4..1d275010556 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -119,12 +119,15 @@ The following metrics are available: | `action_cable_pool_largest_size` | Gauge | 13.4 | Largest number of worker threads observed so far in ActionCable thread pool | `server_mode` | | `action_cable_pool_pending_tasks` | Gauge | 13.4 | Number of tasks waiting to be executed in ActionCable thread pool | `server_mode` | | `action_cable_pool_tasks_total` | Gauge | 13.4 | Total number of tasks executed in ActionCable thread pool | `server_mode` | +| `gitlab_ci_trace_operations_total` | Counter | 13.4 | Total amount of different operations on a build trace | `operation` | +| `gitlab_ci_trace_bytes_total` | Counter | 13.4 | Total amount of build trace bytes transferred | | | `action_cable_single_client_transmissions_total` | Counter | 13.10 | The number of ActionCable messages transmitted to any client in any channel | `server_mode` | | `action_cable_subscription_confirmations_total` | Counter | 13.10 | The number of ActionCable subscriptions from clients confirmed | `server_mode` | | `action_cable_subscription_rejections_total` | Counter | 13.10 | The number of ActionCable subscriptions from clients rejected | `server_mode` | -| `action_cable_transmitted_bytes` | Histogram | 14.1 | Message size, in bytes, transmitted over action cable | `operation`, `channel` | +| `action_cable_transmitted_bytes` | Histogram | 14.1 | Message size, in bytes, transmitted over action cable | `operation`, `channel` | | `gitlab_issuable_fast_count_by_state_total` | Counter | 13.5 | Total number of row count operations on issue/merge request list pages | | | `gitlab_issuable_fast_count_by_state_failures_total` | Counter | 13.5 | Number of soft-failed row count operations on issue/merge request list pages | | +| `gitlab_ci_trace_finalize_duration_seconds` | Histogram | 13.6 | Duration of build trace chunks migration to object storage | | | `gitlab_external_http_total` | Counter | 13.8 | Total number of HTTP calls to external systems | `controller`, `action` | | `gitlab_external_http_duration_seconds` | Counter | 13.8 | Duration in seconds spent on each HTTP call to external systems | | | `gitlab_external_http_exception_total` | Counter | 13.8 | Total number of exceptions raised when making external HTTP calls | | @@ -132,15 +135,15 @@ The following metrics are available: | `pipeline_graph_link_calculation_duration_seconds` | Histogram | 13.9 | Total time spent calculating links, in seconds | | | `pipeline_graph_links_total` | Histogram | 13.9 | Number of links per graph | | | `pipeline_graph_links_per_job_ratio` | Histogram | 13.9 | Ratio of links to job per graph | | -| `gitlab_ci_pipeline_security_orchestration_policy_processing_duration_seconds` | Histogram | 13.12 | Time in seconds it takes to process Security Policies in CI/CD pipeline | | -| `gitlab_ci_difference_live_vs_actual_minutes` | Histogram | 13.12 | Difference between CI minute consumption counted while jobs were running (live) vs when jobs are complete (actual). Used to enforce CI minute consumption limits on long running jobs. | `plan` | -| `gitlab_spamcheck_request_duration_seconds` | Histogram | 13.12 | The duration for requests between Rails and the anti-spam engine | | +| `gitlab_ci_pipeline_security_orchestration_policy_processing_duration_seconds` | Histogram | 13.12 | Time in seconds it takes to process Security Policies in CI/CD pipeline | | +| `gitlab_spamcheck_request_duration_seconds` | Histogram | 13.12 | The duration for requests between Rails and the anti-spam engine | | | `service_desk_thank_you_email` | Counter | 14.0 | Total number of email responses to new service desk emails | | | `service_desk_new_note_email` | Counter | 14.0 | Total number of email notifications on new service desk comment | | | `email_receiver_error` | Counter | 14.1 | Total number of errors when processing incoming emails | | | `gitlab_snowplow_events_total` | Counter | 14.1 | Total number of GitLab Snowplow product intelligence events emitted | | | `gitlab_snowplow_failed_events_total` | Counter | 14.1 | Total number of GitLab Snowplow product intelligence events emission failures | | | `gitlab_snowplow_successful_events_total` | Counter | 14.1 | Total number of GitLab Snowplow product intelligence events emission successes | | +| `gitlab_ci_build_trace_errors_total` | Counter | 14.4 | Total amount of different error types on a build trace | `type` | ## Metrics controlled by a feature flag diff --git a/doc/administration/monitoring/prometheus/puma_exporter.md b/doc/administration/monitoring/prometheus/puma_exporter.md new file mode 100644 index 00000000000..c348e74afaf --- /dev/null +++ b/doc/administration/monitoring/prometheus/puma_exporter.md @@ -0,0 +1,27 @@ +--- +stage: Monitor +group: Monitor +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Puma exporter **(FREE SELF)** + +You can use the [Puma exporter](https://github.com/sapcc/puma-exporter) +to measure various Puma metrics. + +To enable the Puma exporter: + +1. [Enable Prometheus](index.md#configuring-prometheus). +1. Edit `/etc/gitlab/gitlab.rb` to add (or find and uncomment) the following lines. Make sure + `puma['exporter_enabled']` is set to `true`: + + ```ruby + puma['exporter_enabled'] = true + puma['exporter_address'] = "127.0.0.1" + puma['exporter_port'] = 8083 + ``` + +1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect. + +Prometheus begins collecting performance data from the Puma exporter exposed at `localhost:8083`. diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md index 0cab46a95c9..f18c39af24a 100644 --- a/doc/administration/nfs.md +++ b/doc/administration/nfs.md @@ -20,12 +20,46 @@ file system performance, see ## Gitaly and NFS deprecation +Starting with GitLab version 14.0, support for NFS to store Git repository data will be deprecated. Technical customer support and engineering support will be available for the 14.x releases. Engineering will fix bugs and security vulnerabilities consistent with our [release and maintenance policy](../policy/maintenance.md#security-releases). + +At the end of the 14.12 milestone (tenatively June 22nd, 2022) technical and engineering support for using NFS to store Git repository data will be officially at end-of-life. There will be no product changes or troubleshooting provided via Engineering, Security or Paid Support channels. + +For those customers still running earlier versions of GitLab, [our support eligibility and maintenance policy applies](https://about.gitlab.com/support/statement-of-support.html#version-support). + +For the 14.x releases, we will continue to help with Git related tickets from customers running one or more Gitaly servers with its data stored on NFS. Examples may include: + +- Performance issues or timeouts accessing Git data +- Commits or branches vanish +- GitLab intermittently returns the wrong Git data (such as reporting that a repository has no branches) + +Assistance will be limited to activities like: + +- Verifying developers' workflow uses features like protected branches +- Reviewing GitLab event data from the database to advise if it looks like a force push over-wrote branches +- Verifying that NFS client mount options match our [documented recommendations](#mount-options) +- Analyzing the GitLab Workhorse and Rails logs, and determining that `500` errors being seen in the environment are caused by slow responses from Gitaly + +GitLab support will be unable to continue with the investigation if: + +- The date of the request is on or after the release of GitLab version 15.0, and +- Support Engineers and Management determine that all reasonable non-NFS root causes have been exhausted + +If the issue is reproducible, or if it happens intermittently but regularly, GitLab Support will investigate providing the issue reproduces without the use of NFS. In order to reproduce without NFS, the affected repositories should be migrated to a different Gitaly shard, such as Gitaly cluster or a standalone Gitaly VM, backed with block storage. + +### Why remove NFS for Git repository data + +{:.no-toc} + +NFS is not well-suited to a workload consisting of many small files, like Git repositories. NFS does provide a number of configuration options designed to improve performance. However, over time, a number of these mount options have proven to result in inconsistencies across multiple nodes mounting the NFS volume, up to and including data loss. Addressing these inconsistencies consume extraordinary development and support engineer time that hamper our ability to develop [Gitaly Cluster](gitaly/praefect.md), our purpose-built solution to addressing the deficiencies of NFS in this environment. + +Please note that Gitaly Cluster provides highly-available Git repository storage. If this is not a requirement, single-node Gitaly backed by block storage is a suitable substitute. + Engineering support for NFS for Git repositories is deprecated. Technical support is planned to be unavailable from GitLab 15.0. No further enhancements are planned for this feature. Read: -- The [Gitaly and NFS deprecation notice](gitaly/index.md#nfs-deprecation-notice). +- [Moving beyond NFS](gitaly/index.md#moving-beyond-nfs). - About the [correct mount options to use](#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). ## Known kernel version incompatibilities @@ -153,7 +187,7 @@ If the Rugged feature flag is explicitly set to either `true` or `false`, GitLab NOTE: From GitLab 12.7, Rugged is not automatically enabled if Puma thread count is greater than `1`. -If you want to use Rugged with Puma, [set Puma thread count to `1`](https://docs.gitlab.com/omnibus/settings/puma.html#puma-settings). +If you want to use Rugged with Puma, [set Puma thread count to `1`](../install/requirements.md#puma-settings). If you want to use Rugged with Puma thread count more than `1`, Rugged can be enabled using the [feature flag](../development/gitaly.md#legacy-rugged-code). @@ -217,7 +251,7 @@ use the `hard` option, because (from the man page): > use the soft option only when client responsiveness is more important than data integrity Other vendors make similar recommendations, including -[SAP](http://wiki.scn.sap.com/wiki/x/PARnFQ) and NetApp's +[System Applications and Products in Data Processing (SAP)](http://wiki.scn.sap.com/wiki/x/PARnFQ) and NetApp's [knowledge base](https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Software/ONTAP_OS/What_are_the_differences_between_hard_mount_and_soft_mount), they highlight that if the NFS client driver caches data, `soft` means there is no certainty if writes by GitLab are actually on disk. @@ -351,7 +385,7 @@ Any `Operation not permitted` errors means you should investigate your NFS serve If the traffic between your NFS server and NFS client(s) is subject to port filtering by a firewall, then you need to reconfigure that firewall to allow NFS communication. -[This guide from TDLP](https://tldp.org/HOWTO/NFS-HOWTO/security.html#FIREWALLS) +[This guide from The Linux Documentation Project (TDLP)](https://tldp.org/HOWTO/NFS-HOWTO/security.html#FIREWALLS) covers the basics of using NFS in a firewalled environment. Additionally, we encourage you to search for and review the specific documentation for your operating system or distribution and your firewall software. @@ -370,8 +404,8 @@ sudo ufw allow from <client_ip_address> to any port nfs ### Upgrade to Gitaly Cluster or disable caching if experiencing data loss WARNING: -Engineering support for NFS for Git repositories is deprecated. Read the -[Gitaly and NFS deprecation notice](gitaly/index.md#nfs-deprecation-notice). +Engineering support for NFS for Git repositories is deprecated. Read about +[moving beyond NFS](gitaly/index.md#moving-beyond-nfs). Customers and users have reported data loss on high-traffic repositories when using NFS for Git repositories. For example, we have seen: diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index e30ad4d8248..8aa5af4c2bf 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -104,12 +104,12 @@ In the case of lookup failures (which are common), the `authorized_keys` file is still scanned. So Git SSH performance would still be slow for many users as long as a large file exists. -To disable any more writes to the `authorized_keys` file: +To disable writes to the `authorized_keys` file: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Network**. 1. Expand **Performance optimization**. -1. Clear the **Write to "authorized_keys" file** checkbox. +1. Clear the **Use authorized_keys file to authenticate SSH keys** checkbox. 1. Select **Save changes**. Again, confirm that SSH is working by removing your user's SSH key in the UI, @@ -123,10 +123,14 @@ or for asking users to re-add their keys. This is a brief overview. Please refer to the above instructions for more context. -1. [Rebuild the `authorized_keys` file](../raketasks/maintenance.md#rebuild-authorized_keys-file) -1. Enable writes to the `authorized_keys` file in Application Settings +1. [Rebuild the `authorized_keys` file](../raketasks/maintenance.md#rebuild-authorized_keys-file). +1. Enable writes to the `authorized_keys` file. + 1. On the top bar, select **Menu > Admin**. + 1. On the left sidebar, select **Settings > Network**. + 1. Expand **Performance optimization**. + 1. Select the **Use authorized_keys file to authenticate SSH keys** checkbox. 1. Remove the `AuthorizedKeysCommand` lines from `/etc/ssh/sshd_config` or from `/assets/sshd_config` if you are using Omnibus Docker. -1. Reload `sshd`: `sudo service sshd reload` +1. Reload `sshd`: `sudo service sshd reload`. ## Compiling a custom version of OpenSSH for CentOS 6 diff --git a/doc/administration/operations/moving_repositories.md b/doc/administration/operations/moving_repositories.md index 61a9ec8e7d4..8aeaadc17e9 100644 --- a/doc/administration/operations/moving_repositories.md +++ b/doc/administration/operations/moving_repositories.md @@ -27,7 +27,7 @@ For more information, see: querying and scheduling snippet repository moves. - [The API documentation](../../api/group_repository_storage_moves.md) details the endpoints for querying and scheduling group repository moves **(PREMIUM SELF)**. -- [Migrate to Gitaly Cluster](../gitaly/index.md#migrate-to-gitaly-cluster). +- [Migrating to Gitaly Cluster](../gitaly/index.md#migrating-to-gitaly-cluster). ### Move Repositories diff --git a/doc/administration/operations/ssh_certificates.md b/doc/administration/operations/ssh_certificates.md index 814e742b026..77dc4eb180b 100644 --- a/doc/administration/operations/ssh_certificates.md +++ b/doc/administration/operations/ssh_certificates.md @@ -21,8 +21,7 @@ upload the new keys to GitLab. WARNING: OpenSSH version 6.9+ is required because that version introduced the `AuthorizedPrincipalsCommand` configuration option. If -using CentOS 6, you can [follow these -instructions](fast_ssh_key_lookup.html#compiling-a-custom-version-of-openssh-for-centos-6) +using CentOS 6, you can [follow these instructions](fast_ssh_key_lookup.md#compiling-a-custom-version-of-openssh-for-centos-6) to compile an up-to-date version. ## Why use OpenSSH certificates? @@ -132,8 +131,8 @@ requirement for it, we effectively only care about the "key ID" being correct. Once that's extracted GitLab enforces its own ACLs for that user (for example, what projects the user can access). -So it's OK to e.g. be overly generous in what you accept, since if the -user e.g. has no access to GitLab at all it just errors out with a +It's therefore fine to be overly generous in what you accept. For example, if the user has no access +to GitLab, an error is produced with a message about an invalid user. message about this being an invalid user. ## Interaction with the `authorized_keys` file diff --git a/doc/administration/package_information/defaults.md b/doc/administration/package_information/defaults.md index 45bea065995..95d6135c28c 100644 --- a/doc/administration/package_information/defaults.md +++ b/doc/administration/package_information/defaults.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Package defaults +# Package defaults **(FREE SELF)** Unless configuration is specified in the `/etc/gitlab/gitlab.rb` file, the package will assume the defaults as noted below. @@ -14,42 +14,42 @@ the package will assume the defaults as noted below. See the table below for the list of ports that the Omnibus GitLab assigns by default: -| Component | On by default | Communicates via | Alternative | Connection port | -| :----------------------------------------------------: | :------------: | :--------------: | :---------: | :------------------------------------: | -| <a name="gitlab-rails"></a> GitLab Rails | Yes | Port | X | 80 or 443 | -| <a name="gitlab-shell"></a> GitLab Shell | Yes | Port | X | 22 | -| <a name="postgresql"></a> PostgreSQL | Yes | Socket | Port (5432) | X | -| <a name="redis"></a> Redis | Yes | Socket | Port (6379) | X | -| <a name="puma"></a> Puma | Yes | Socket | Port (8080) | X | -| <a name="gitlab-workhorse"></a> GitLab Workhorse | Yes | Socket | Port (8181) | X | -| <a name="nginx-status"></a> NGINX status | Yes | Port | X | 8060 | -| <a name="prometheus"></a> Prometheus | Yes | Port | X | 9090 | -| <a name="node-exporter"></a> Node exporter | Yes | Port | X | 9100 | -| <a name="redis-exporter"></a> Redis exporter | Yes | Port | X | 9121 | -| <a name="postgres-exporter"></a> PostgreSQL exporter | Yes | Port | X | 9187 | -| <a name="pgbouncer-exporter"></a> PgBouncer exporter | No | Port | X | 9188 | -| <a name="gitlab-exporter"></a> GitLab Exporter | Yes | Port | X | 9168 | -| <a name="sidekiq-exporter"></a> Sidekiq exporter | Yes | Port | X | 8082 | -| <a name="puma-exporter"></a> Puma exporter | No | Port | X | 8083 | -| <a name="geo-postgresql"></a> Geo PostgreSQL | No | Socket | Port (5431) | X | -| <a name="redis-sentinel"></a> Redis Sentinel | No | Port | X | 26379 | -| <a name="incoming-email"></a> Incoming email | No | Port | X | 143 | -| <a name="elasticsearch"></a> Elastic search | No | Port | X | 9200 | -| <a name="gitlab-pages"></a> GitLab Pages | No | Port | X | 80 or 443 | -| <a name="gitlab-registry-web"></a> GitLab Registry | No* | Port | X | 80, 443 or 5050 | -| <a name="gitlab-registry"></a> GitLab Registry | No | Port | X | 5000 | -| <a name="ldap"></a> LDAP | No | Port | X | Depends on the component configuration | -| <a name="kerberos"></a> Kerberos | No | Port | X | 8443 or 8088 | -| <a name="omniauth"></a> OmniAuth | Yes | Port | X | Depends on the component configuration | -| <a name="smtp"></a> SMTP | No | Port | X | 465 | -| <a name="remote-syslog"></a> Remote syslog | No | Port | X | 514 | -| <a name="mattermost"></a> Mattermost | No | Port | X | 8065 | -| <a name="mattermost-web"></a> Mattermost | No | Port | X | 80 or 443 | -| <a name="pgbouncer"></a> PgBouncer | No | Port | X | 6432 | -| <a name="consul"></a> Consul | No | Port | X | 8300, 8301(UDP), 8500, 8600[^Consul-notes] | -| <a name="patroni"></a> Patroni | No | Port | X | 8008 | -| <a name="gitlab-kas"></a> GitLab KAS | No | Port | X | 8150 | -| <a name="gitaly"></a> Gitaly | No | Port | X | 8075 | +| Component | On by default | Communicates via | Alternative | Connection port | +|:-------------------:|:-------------:|:----------------:|:-----------:|:------------------------------------------:| +| GitLab Rails | Yes | Port | X | 80 or 443 | +| GitLab Shell | Yes | Port | X | 22 | +| PostgreSQL | Yes | Socket | Port (5432) | X | +| Redis | Yes | Socket | Port (6379) | X | +| Puma | Yes | Socket | Port (8080) | X | +| GitLab Workhorse | Yes | Socket | Port (8181) | X | +| NGINX status | Yes | Port | X | 8060 | +| Prometheus | Yes | Port | X | 9090 | +| Node exporter | Yes | Port | X | 9100 | +| Redis exporter | Yes | Port | X | 9121 | +| PostgreSQL exporter | Yes | Port | X | 9187 | +| PgBouncer exporter | No | Port | X | 9188 | +| GitLab Exporter | Yes | Port | X | 9168 | +| Sidekiq exporter | Yes | Port | X | 8082 | +| Puma exporter | No | Port | X | 8083 | +| Geo PostgreSQL | No | Socket | Port (5431) | X | +| Redis Sentinel | No | Port | X | 26379 | +| Incoming email | No | Port | X | 143 | +| Elastic search | No | Port | X | 9200 | +| GitLab Pages | No | Port | X | 80 or 443 | +| GitLab Registry | No* | Port | X | 80, 443 or 5050 | +| GitLab Registry | No | Port | X | 5000 | +| LDAP | No | Port | X | Depends on the component configuration | +| Kerberos | No | Port | X | 8443 or 8088 | +| OmniAuth | Yes | Port | X | Depends on the component configuration | +| SMTP | No | Port | X | 465 | +| Remote syslog | No | Port | X | 514 | +| Mattermost | No | Port | X | 8065 | +| Mattermost | No | Port | X | 80 or 443 | +| PgBouncer | No | Port | X | 6432 | +| Consul | No | Port | X | 8300, 8301(UDP), 8500, 8600[^Consul-notes] | +| Patroni | No | Port | X | 8008 | +| GitLab KAS | No | Port | X | 8150 | +| Gitaly | No | Port | X | 8075 | Legend: @@ -59,7 +59,7 @@ Legend: - `Alternative` - If it is possible to configure the component to use different type of communication. The type is listed with default port used in that case. - `Connection port` - Port on which the component communicates. -GitLab also expects a filesystem to be ready for the storage of Git repositories +GitLab also expects a file system to be ready for the storage of Git repositories and various other files. Note that if you are using NFS (Network File System), files will be carried diff --git a/doc/administration/package_information/deprecated_os.md b/doc/administration/package_information/deprecated_os.md index 251dbe1e20e..35b333241b2 100644 --- a/doc/administration/package_information/deprecated_os.md +++ b/doc/administration/package_information/deprecated_os.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# OS Versions that are no longer supported +# OS Versions that are no longer supported **(FREE SELF)** GitLab provides omnibus packages for operating systems only until their EOL (End-Of-Life). After the EOL date of the OS, GitLab will stop releasing diff --git a/doc/administration/package_information/deprecation_policy.md b/doc/administration/package_information/deprecation_policy.md index cc16661442a..80ce72d54f5 100644 --- a/doc/administration/package_information/deprecation_policy.md +++ b/doc/administration/package_information/deprecation_policy.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Deprecation policy +# Deprecation policy **(FREE SELF)** The Omnibus GitLab packages come with number of different libraries and services which offers users plethora of configuration options. @@ -60,7 +60,7 @@ We should aim to not remove sensitive configuration in the *next major* release See the table below for some examples: -| Config. type | Deprecation announced | Final minor release | Remove | +| Configuration type | Deprecation announced | Final minor release | Remove | | -------- | -------- | -------- | -------- | | Sensitive | 10.1.0 | 10.9.0 | 11.0.0 | | Sensitive | 10.7.0 | 10.9.0 | 12.0.0 | @@ -90,6 +90,6 @@ the feature will continue working the same way as if you had `gitlab_rails['bett However, setting the old version of configuration will print out a deprecation notice at the end of installation/upgrade/reconfigure run. -With GitLab 11, `gitlab_rails['configuration'] = true` will no longer work and you will have to manually change the configuration in `/etc/gitlab/gitlab.rb` to the new valid config. +With GitLab 11, `gitlab_rails['configuration'] = true` will no longer work and you will have to manually change the configuration in `/etc/gitlab/gitlab.rb` to the new valid configuration. **Note** If this configuration option is sensitive and can put integrity of the installation or data in danger, installation/upgrade will be aborted. diff --git a/doc/administration/package_information/index.md b/doc/administration/package_information/index.md index e18fb621b89..12f3274ecab 100644 --- a/doc/administration/package_information/index.md +++ b/doc/administration/package_information/index.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Package information +# Package information **(FREE SELF)** The Omnibus GitLab package is bundled with all dependencies required for GitLab to function correctly. More details can be found @@ -15,10 +15,10 @@ at [bundling dependencies document](omnibus_packages.md). The released package versions are in the format `MAJOR.MINOR.PATCH-EDITION.OMNIBUS_RELEASE` | Component | Meaning | Example | -| --------- | ------- | ------- | -| MAJOR.MINOR.PATCH | The GitLab version this corresponds to | 13.3.0 | -| EDITION | The edition of GitLab this corresponds to | ee | -| OMNIBUS_RELEASE | The omnibus release. Usually, this will be 0. This will be incremented if we need to build a new package without changing the GitLab version. | 0 | +|-------------------|---------|---------| +| MAJOR.MINOR.PATCH | The GitLab version this corresponds to. | 13.3.0 | +| EDITION | The edition of GitLab this corresponds to. | ee | +| OMNIBUS_RELEASE | The Omnibus GitLab release. Usually, this will be 0. This is incremented if we need to build a new package without changing the GitLab version. | 0 | ## Licenses @@ -26,23 +26,22 @@ See [licensing](licensing.md) ## Defaults -The Omnibus GitLab package requires various configuration to get the -components in working order. -If the configuration is not provided, the package will use the default -values assumed in the package. +The Omnibus GitLab package requires various configuration to get the components +in working order. If the configuration is not provided, the package will use +the default values assumed in the package. These defaults are noted in the package [defaults document](defaults.md). ## Checking the versions of bundled software -Once the Omnibus GitLab package is installed, all versions of the bundled +After the Omnibus GitLab package is installed, all versions of the bundled libraries are located in `/opt/gitlab/version-manifest.txt`. If you don't have the package installed, you can always check the Omnibus GitLab [source repository](https://gitlab.com/gitlab-org/omnibus-gitlab/tree/master), specifically the -[config directory](https://gitlab.com/gitlab-org/omnibus-gitlab/tree/master/config). +[configuration directory](https://gitlab.com/gitlab-org/omnibus-gitlab/tree/master/config). -For example, if you take a look at the `8-6-stable` branch, you can conclude that +For example, if you examine the `8-6-stable` branch, you can conclude that 8.6 packages were running [Ruby 2.1.8](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8-6-stable/config/projects/gitlab.rb#L48). Or, that 8.5 packages were bundled with [NGINX 1.9.0](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8-5-stable/config/software/nginx.rb#L20). @@ -70,11 +69,10 @@ To view a diff between your configuration file and the latest version, run: sudo gitlab-ctl diff-config ``` -_**Note:** This command is available from GitLab 8.17_ - -**Important:** If you are copy-pasting the output of this command into your -`/etc/gitlab/gitlab.rb` configuration file, make sure to omit leading `+` and `-` -on each line. +WARNING: +If you are pasting the output of this command into your +`/etc/gitlab/gitlab.rb` configuration file, omit any leading `+` and `-` +characters on each line. ## Init system detection diff --git a/doc/administration/package_information/licensing.md b/doc/administration/package_information/licensing.md index 8557a94bf93..02358c66993 100644 --- a/doc/administration/package_information/licensing.md +++ b/doc/administration/package_information/licensing.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Package Licensing +# Package Licensing **(FREE SELF)** ## License diff --git a/doc/administration/package_information/omnibus_packages.md b/doc/administration/package_information/omnibus_packages.md index aa73534fc55..115d6c394ad 100644 --- a/doc/administration/package_information/omnibus_packages.md +++ b/doc/administration/package_information/omnibus_packages.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Omnibus based packages and images +# Omnibus based packages and images **(FREE SELF)** Below you can find some basic information on why GitLab provides packages and a Docker image that come with bundled dependencies. diff --git a/doc/administration/package_information/postgresql_versions.md b/doc/administration/package_information/postgresql_versions.md index 89da4864872..252e0cad76d 100644 --- a/doc/administration/package_information/postgresql_versions.md +++ b/doc/administration/package_information/postgresql_versions.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# PostgreSQL versions shipped with Omnibus GitLab +# PostgreSQL versions shipped with Omnibus GitLab **(FREE SELF)** NOTE: This table lists only GitLab versions where a significant change happened in the diff --git a/doc/administration/package_information/signed_packages.md b/doc/administration/package_information/signed_packages.md index fb994809460..fb605f8d5be 100644 --- a/doc/administration/package_information/signed_packages.md +++ b/doc/administration/package_information/signed_packages.md @@ -4,9 +4,9 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Package Signatures +# Package Signatures **(FREE SELF)** -As of the release of GitLab 9.5 on August 22, 2017, GitLab provides signed Omnibus GitLab packages for RPM and DEB based distributions. This means that all packages provided on <https://packages.gitlab.com> are signed, starting with `9.5.0`, and all future versions of supported branches (e.g. `9.3.x` and `9.4.x` after August 22, 2017). Any package version prior to August 22, 2017, will not be signed. Please pass the appropriate argument to your package manager. (Example: `yum --nogpgcheck`) +As of the release of GitLab 9.5 on August 22, 2017, GitLab provides signed Omnibus GitLab packages for RPM and DEB based distributions. This means that all packages provided on <https://packages.gitlab.com> are signed, starting with `9.5.0`, and all future versions of supported branches (for example `9.3.x` and `9.4.x` after August 22, 2017). Any package version prior to August 22, 2017, will not be signed. Please pass the appropriate argument to your package manager. (Example: `yum --nogpgcheck`) Omnibus GitLab packages produced by GitLab are created via the [Omnibus](https://github.com/chef/omnibus) tool, for which GitLab has added DEB signing via `debsigs` in [our own fork](https://gitlab.com/gitlab-org/omnibus). This addition, combined with the existing functionality of RPM signing, allows GitLab to provide signed packages for all supported distributions using DEB or RPM. diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index 1067474c8b4..7e711bb5740 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -119,6 +119,11 @@ GitLab from source respectively. Ensure you choose a port different than the one that Registry listens to (`5000` by default), otherwise conflicts occur. +NOTE: +Host and container firewall rules must be configured to allow traffic in through the port listed +under the `registry_external_url` line, rather than the port listed under +`gitlab_rails['registry_port']` (default `5000`). + **Omnibus GitLab installations** 1. Your `/etc/gitlab/gitlab.rb` should contain the Registry URL as well as the @@ -151,6 +156,19 @@ otherwise conflicts occur. If your certificate provider provides the CA Bundle certificates, append them to the TLS certificate file. +An administrator may want the container registry listening on an arbitrary port such as `5678`. +However, the registry and application server are behind an AWS application load balancer that only +listens on ports `80` and `443`. The admin may simply remove the port number for +`registry_external_url`, so HTTP or HTTPS is assumed. Then, the rules apply that map the load +balancer to the registry from ports `80` or `443` to the arbitrary port. This is important if users +rely on the `docker login` example in the container registry. Here's an example: + +```ruby +registry_external_url 'https://registry-gitlab.example.com' +registry_nginx['redirect_http_to_https'] = true +registry_nginx['listen_port'] = 5678 +``` + **Installations from source** 1. Open `/home/git/gitlab/config/gitlab.yml`, find the `registry` entry and diff --git a/doc/administration/packages/dependency_proxy.md b/doc/administration/packages/dependency_proxy.md index 32e7e191011..a394a32fc18 100644 --- a/doc/administration/packages/dependency_proxy.md +++ b/doc/administration/packages/dependency_proxy.md @@ -29,8 +29,8 @@ To enable the dependency proxy feature: gitlab_rails['dependency_proxy_enabled'] = true ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab") for the changes to take effect. -1. Enable the [Puma web server](https://docs.gitlab.com/omnibus/settings/puma.html). +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Enable the [Puma web server](../operations/puma.md). **Helm chart installations** @@ -88,7 +88,7 @@ To change the local storage path: gitlab_rails['dependency_proxy_storage_path'] = "/mnt/dependency_proxy" ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab") for the changes to take effect. +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. **Installations from source** @@ -145,7 +145,7 @@ This section describes the earlier configuration format. } ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab") for the changes to take effect. +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. **Installations from source** diff --git a/doc/administration/packages/img/gitlab-registry-architecture.png b/doc/administration/packages/img/gitlab-registry-architecture.png Binary files differindex 742678d5e11..d6e8f935ad2 100644 --- a/doc/administration/packages/img/gitlab-registry-architecture.png +++ b/doc/administration/packages/img/gitlab-registry-architecture.png diff --git a/doc/administration/packages/index.md b/doc/administration/packages/index.md index 37473d35573..90f2d9127fe 100644 --- a/doc/administration/packages/index.md +++ b/doc/administration/packages/index.md @@ -68,7 +68,7 @@ To enable the Packages feature: gitlab_rails['packages_enabled'] = true ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab") for the changes to take effect. +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. **Installations from source** @@ -80,7 +80,7 @@ To enable the Packages feature: enabled: true ``` -1. [Restart GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab") for the changes to take effect. +1. [Restart GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. **Helm Chart installations** @@ -92,7 +92,7 @@ To enable the Packages feature: enabled: true ``` -1. [Restart GitLab](../restart_gitlab.md#helm-chart-installations "How to reconfigure Helm GitLab") for the changes to take effect. +1. [Restart GitLab](../restart_gitlab.md#helm-chart-installations) for the changes to take effect. ## Rate limits diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index 8b7af5ee170..8a0d3f552bf 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -7,12 +7,6 @@ description: 'Learn how to administer GitLab Pages.' # GitLab Pages administration **(FREE SELF)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/80) in GitLab EE 8.3. -> - Custom CNAMEs with TLS support were [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/173) in GitLab EE 8.5. -> - GitLab Pages [was ported](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/14605) to Community Edition in GitLab 8.17. -> - Support for subgroup project's websites was -> [introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/30548) in GitLab 11.8. - GitLab Pages allows for hosting of static sites. It must be configured by an administrator. Separate [user documentation](../../user/project/pages/index.md) is available. @@ -23,10 +17,10 @@ GitLab from source, see ## Overview -GitLab Pages makes use of the [GitLab Pages daemon](https://gitlab.com/gitlab-org/gitlab-pages), a simple HTTP server +GitLab Pages makes use of the [GitLab Pages daemon](https://gitlab.com/gitlab-org/gitlab-pages), a basic HTTP server written in Go that can listen on an external IP address and provide support for custom domains and custom certificates. It supports dynamic certificates through -SNI and exposes pages using HTTP2 by default. +Server Name Indication (SNI) and exposes pages using HTTP2 by default. You are encouraged to read its [README](https://gitlab.com/gitlab-org/gitlab-pages/blob/master/README.md) to fully understand how it works. @@ -89,7 +83,7 @@ added `gitlab.io` [in 2016](https://gitlab.com/gitlab-com/infrastructure/-/issue ### DNS configuration GitLab Pages expect to run on their own virtual host. In your DNS server/provider -you need to add a [wildcard DNS A record](https://en.wikipedia.org/wiki/Wildcard_DNS_record) pointing to the +add a [wildcard DNS A record](https://en.wikipedia.org/wiki/Wildcard_DNS_record) pointing to the host that GitLab runs. For example, an entry would look like this: ```plaintext @@ -99,9 +93,9 @@ host that GitLab runs. For example, an entry would look like this: Where `example.io` is the domain GitLab Pages is served from, `192.0.2.1` is the IPv4 address of your GitLab instance, and `2001:db8::1` is the -IPv6 address. If you don't have IPv6, you can omit the AAAA record. +IPv6 address. If you don't have IPv6, you can omit the `AAAA` record. -#### Custom domains +#### DNS configuration for custom domains If support for custom domains is needed, the Pages root domain and its subdomains should point to the secondary IP (which is dedicated for the Pages daemon). `<namespace>.<pages root domain>` should @@ -131,7 +125,7 @@ Depending on your needs, you can set up GitLab Pages in 4 different ways. The following examples are listed from the easiest setup to the most advanced one. The absolute minimum requirement is to set up the wildcard DNS -since that is needed in all configurations. +because that is needed in all configurations. ### Wildcard domains @@ -143,7 +137,7 @@ since that is needed in all configurations. URL scheme: `http://<namespace>.example.io/<project_slug>` -This is the minimum setup that you can use Pages with. It is the base for all +The following is the minimum setup that you can use Pages with. It is the base for all other setups as described below. NGINX proxies all requests to the daemon. The Pages daemon doesn't listen to the outside world. @@ -180,8 +174,8 @@ outside world. pages_nginx['redirect_http_to_https'] = true ``` -1. If you haven't named your certificate and key `example.io.crt` and `example.io.key` -then you'll need to also add the full paths as shown below: +1. If you haven't named your certificate and key `example.io.crt` and `example.io.key`, +you must also add the full paths as shown below: ```ruby pages_nginx['ssl_certificate'] = "/etc/gitlab/ssl/pages-nginx.crt" @@ -201,7 +195,7 @@ Multiple wildcards for one instance is not supported. Only one wildcard per inst Below is a table of all configuration settings known to Pages in Omnibus GitLab, and what they do. These options can be adjusted in `/etc/gitlab/gitlab.rb`, and take effect after you [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -Most of these settings don't need to be configured manually unless you need more granular +Most of these settings don't have to be configured manually unless you need more granular control over how the Pages daemon runs and serves content in your environment. | Setting | Description | @@ -382,8 +376,6 @@ To enable it: ### Access control -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/33422) in GitLab 11.5. - GitLab Pages access control can be configured per-project, and allows access to a Pages site to be controlled based on a user's membership to that project. @@ -444,7 +436,7 @@ You can enforce [Access Control](#access-control) for all GitLab Pages websites on your GitLab instance. By doing so, only logged-in users have access to them. This setting overrides Access Control set by users in individual projects. -This can be useful to preserve information published with Pages websites to the users +This can be helpful to restrict information published with Pages websites to the users of your instance only. To do that: @@ -491,7 +483,7 @@ For Omnibus, this is fixed by [installing a custom CA in Omnibus GitLab](https:/ > [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/392) in GitLab 13.7. WARNING: -These are advanced settings. The recommended default values are set inside GitLab Pages. You should +These instructions deal with some advanced settings of your GitLab instance. The recommended default values are set inside GitLab Pages. You should change these settings only if absolutely necessary. Use extreme caution. GitLab Pages can serve content from ZIP archives through object storage (an @@ -524,9 +516,6 @@ After an archive reaches `zip_cache_expiration`, it's marked as expired and remo ## Activate verbose logging for daemon -Verbose logging was [introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/2533) in -Omnibus GitLab 11.1. - Follow the steps below to configure verbose logging of GitLab Pages daemon. 1. By default the daemon only logs with `INFO` level. @@ -603,8 +592,6 @@ the below steps to do a no downtime transfer to a new storage location. ## Configure listener for reverse proxy requests -> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/2533) in Omnibus GitLab 11.1. - Follow the steps below to configure the proxy listener of GitLab Pages. 1. By default the listener is configured to listen for requests on `localhost:8090`. @@ -775,7 +762,7 @@ WARNING: From [GitLab 13.3](https://gitlab.com/gitlab-org/gitlab/-/issues/217912) to [GitLab 13.12](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5993) GitLab Pages can either use `disk` or `gitlab` domain configuration source. -We highly advise you to use `gitlab` configuration source as it will make transition to newer versions easier. +We highly advise you to use `gitlab` configuration source as it makes transitions to newer versions easier. To explicitly enable API source: @@ -1019,14 +1006,13 @@ Starting from GitLab 13.12, this setting also disables the [legacy storage](#mig In GitLab 14.0 a number of breaking changes were introduced which may require some user intervention. The steps below describe the best way to migrate without causing any downtime for your GitLab instance. -If you run GitLab on a single server, then most likely the upgrade process to 14.0 will go smoothly for you -and you will not notice any problem after upgrading. +A GitLab instance running on a single server typically upgrades to 14.0 smoothly, and there should be minimal issues after the upgrade is complete. Regardless, we recommend everyone follow the migration steps to ensure a successful upgrade. If at any point you run into issues, consult the [troubleshooting section](#troubleshooting). -If your current GitLab version is lower than 13.12, then you first need to update to 13.12. +If your current GitLab version is lower than 13.12, then you must first update to 13.12. Updating directly to 14.0 is [not supported](../../update/index.md#upgrade-paths) -and may cause downtime for some web-sites hosted on GitLab Pages. Once you update to 13.12, +and may cause downtime for some web-sites hosted on GitLab Pages. After you update to 13.12, migrate GitLab Pages to prepare them for GitLab 14.0: 1. Set [`domain_config_source` to `gitlab`](#domain-source-configuration-before-140), which @@ -1077,7 +1063,7 @@ This issue is fixed in GitLab 14.3 and above, try upgrading GitLab first. GitLab Pages runs inside a `chroot` jail, usually in a uniquely numbered directory like `/tmp/gitlab-pages-*`. -Within the jail, a bundle of trusted certificates is +In the jail, a bundle of trusted certificates is provided at `/etc/ssl/ca-bundle.pem`. It's [copied there](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/51) from `/opt/gitlab/embedded/ssl/certs/cacert.pem` @@ -1206,26 +1192,10 @@ To stop `systemd` from cleaning the Pages related content: sudo gitlab-ctl restart gitlab-pages ``` -### 404 error after transferring the project to a different group or user, or changing project path - -If you encounter a `404 Not Found` error a Pages site after transferring a project to -another group or user, or changing project path, you must trigger a domain configuration -update for Pages. To do so, write something in the `.update` file. The Pages daemon -monitors for changes to this file, and reloads the configuration when changes occur. - -Use this example to fix a `404 Not Found` error after transferring a project or changing -a project path with Pages: - -```shell -date > /var/opt/gitlab/gitlab-rails/shared/pages/.update -``` - -If you've customized the Pages storage path, adjust the command above to use your custom path. - ### 404 error after promoting a Geo secondary to a primary node -These are due to the Pages files not being among the -[supported data types](../geo/replication/datatypes.md#limitations-on-replicationverification). +Pages files are not among the +[supported data types](../geo/replication/datatypes.md#limitations-on-replicationverification) for replication in Geo. After a secondary node is promoted to a primary node, attempts to access a Pages site result in a `404 Not Found` error. It is possible to copy the subfolders and files in the [Pages path](#change-storage-path) to the new primary node to resolve this. @@ -1233,11 +1203,11 @@ For example, you can adapt the `rsync` strategy from the [moving repositories documentation](../operations/moving_repositories.md). Alternatively, run the CI pipelines of those projects that contain a `pages` job again. -## 404 or 500 error when accessing GitLab Pages in a Geo setup +### 404 or 500 error when accessing GitLab Pages in a Geo setup Pages sites are only available on the primary Geo site, while the codebase of the project is available on all sites. -If you try to access a Pages page on a secondary site, you will get a 404 or 500 HTTP code depending on the access control. +If you try to access a Pages page on a secondary site, a 404 or 500 HTTP code is returned depending on the access control. Read more which [features don't support Geo replication/verification](../geo/replication/datatypes.md#limitations-on-replicationverification). diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index 2e0820b69c9..dc569a81abf 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -23,10 +23,7 @@ replication and failover requires: - A minimum of three database nodes. - A minimum of three `Consul` server nodes. -- A minimum of one `pgbouncer` service node, but it's recommended to have one - per database node. - - An internal load balancer (TCP) is required when there is more than one - `pgbouncer` service node. +- A minimum of one `pgbouncer` service node, but it's recommended to have one per database node. An internal load balancer (TCP) is required when there is more than one `pgbouncer` service node.  @@ -35,40 +32,31 @@ sure you have redundant connectivity between all Database and GitLab instances to avoid the network becoming a single point of failure. NOTE: -As of GitLab 13.3, PostgreSQL 12 is shipped with Omnibus GitLab. Clustering for PostgreSQL 12 is only supported with +As of GitLab 13.3, PostgreSQL 12 is shipped with Omnibus GitLab. Clustering for PostgreSQL 12 is supported only with Patroni. See the [Patroni](#patroni) section for further details. Starting with GitLab 14.0, only PostgreSQL 12 is -shipped with Omnibus GitLab and thus Patroni becomes mandatory for replication and failover. +shipped with Omnibus GitLab, and thus Patroni becomes mandatory for replication and failover. ### Database node Each database node runs three services: -`PostgreSQL` - The database itself. - -`Patroni` - Communicates with other Patroni services in the cluster and handles -failover when issues with the leader server occurs. The failover procedure -consists of: - -- Selecting a new leader for the cluster. -- Promoting the new node to leader. -- Instructing remaining servers to follow the new leader node. - -`Consul` agent - To communicate with Consul cluster which stores the current Patroni state. The agent monitors the status of each node in the database cluster and tracks its health in a service definition on the Consul cluster. +- `PostgreSQL`: The database itself. +- `Patroni`: Communicates with other Patroni services in the cluster and handles failover when issues with the leader server occurs. The failover procedure consists of: + - Selecting a new leader for the cluster. + - Promoting the new node to leader. + - Instructing remaining servers to follow the new leader node. +- `Consul` agent: To communicate with Consul cluster which stores the current Patroni state. The agent monitors the status of each node in the database cluster and tracks its health in a service definition on the Consul cluster. ### Consul server node -The Consul server node runs the Consul server service. These nodes must have reached the quorum and elected a leader _before_ Patroni cluster bootstrap otherwise database nodes wait until such Consul leader is elected. +The Consul server node runs the Consul server service. These nodes must have reached the quorum and elected a leader _before_ Patroni cluster bootstrap; otherwise, database nodes wait until such Consul leader is elected. ### PgBouncer node Each PgBouncer node runs two services: -`PgBouncer` - The database connection pooler itself. - -`Consul` agent - Watches the status of the PostgreSQL service definition on the -Consul cluster. If that status changes, Consul runs a script which updates the -PgBouncer configuration to point to the new PostgreSQL leader node and reloads -the PgBouncer service. +- `PgBouncer`: The database connection pooler itself. +- `Consul` agent: Watches the status of the PostgreSQL service definition on the Consul cluster. If that status changes, Consul runs a script which updates the PgBouncer configuration to point to the new PostgreSQL leader node and reloads the PgBouncer service. ### Connection flow @@ -106,8 +94,7 @@ When using default setup, minimum configuration requires: - `CONSUL_USERNAME`. The default user for Omnibus GitLab is `gitlab-consul` - `CONSUL_DATABASE_PASSWORD`. Password for the database user. -- `CONSUL_PASSWORD_HASH`. This is a hash generated out of Consul username/password pair. - Can be generated with: +- `CONSUL_PASSWORD_HASH`. This is a hash generated out of Consul username/password pair. It can be generated with: ```shell sudo gitlab-ctl pg-password-md5 CONSUL_USERNAME @@ -118,8 +105,7 @@ When using default setup, minimum configuration requires: Few notes on the service itself: - The service runs under a system account, by default `gitlab-consul`. - - If you are using a different username, you have to specify it through the - `CONSUL_USERNAME` variable. +- If you are using a different username, you have to specify it through the `CONSUL_USERNAME` variable. - Passwords are stored in the following locations: - `/etc/gitlab/gitlab.rb`: hashed - `/var/opt/gitlab/pgbouncer/pg_auth`: hashed @@ -129,10 +115,8 @@ Few notes on the service itself: When configuring PostgreSQL, we do the following: -- Set `max_replication_slots` to double the number of database nodes. - Patroni uses one extra slot per node when initiating the replication. -- Set `max_wal_senders` to one more than the allocated number of replication slots in the cluster. - This prevents replication from using up all of the available database connections. +- Set `max_replication_slots` to double the number of database nodes. Patroni uses one extra slot per node when initiating the replication. +- Set `max_wal_senders` to one more than the allocated number of replication slots in the cluster. This prevents replication from using up all of the available database connections. In this document we are assuming 3 database nodes, which makes this configuration: @@ -151,7 +135,7 @@ You need the following password information for the application's database user: - `POSTGRESQL_USERNAME`. The default user for Omnibus GitLab is `gitlab` - `POSTGRESQL_USER_PASSWORD`. The password for the database user - `POSTGRESQL_PASSWORD_HASH`. This is a hash generated out of the username/password pair. - Can be generated with: + It can be generated with: ```shell sudo gitlab-ctl pg-password-md5 POSTGRESQL_USERNAME @@ -170,8 +154,7 @@ When using a default setup, the minimum configuration requires: - `PGBOUNCER_USERNAME`. The default user for Omnibus GitLab is `pgbouncer` - `PGBOUNCER_PASSWORD`. This is a password for PgBouncer service. -- `PGBOUNCER_PASSWORD_HASH`. This is a hash generated out of PgBouncer username/password pair. - Can be generated with: +- `PGBOUNCER_PASSWORD_HASH`. This is a hash generated out of PgBouncer username/password pair. It can be generated with: ```shell sudo gitlab-ctl pg-password-md5 PGBOUNCER_USERNAME @@ -181,8 +164,7 @@ When using a default setup, the minimum configuration requires: Few things to remember about the service itself: -- The service runs as the same system account as the database - - In the package, this is by default `gitlab-psql` +- The service runs as the same system account as the database. In the package, this is by default `gitlab-psql` - If you use a non-default user account for PgBouncer service (by default `pgbouncer`), you need to specify this username. - Passwords are stored in the following locations: - `/etc/gitlab/gitlab.rb`: hashed, and in plain text @@ -206,7 +188,7 @@ When installing the GitLab package, do not supply `EXTERNAL_URL` value. You must enable Patroni explicitly to be able to use it (with `patroni['enable'] = true`). -Any PostgreSQL configuration item that controls replication, for example `wal_level`, `max_wal_senders`, etc, are strictly +Any PostgreSQL configuration item that controls replication, for example `wal_level`, `max_wal_senders`, or others are strictly controlled by Patroni. These configurations override the original settings that you make with the `postgresql[...]` configuration key. Hence, they are all separated and placed under `patroni['postgresql'][...]`. This behavior is limited to replication. Patroni honours any other PostgreSQL configuration that was made with the `postgresql[...]` configuration key. For example, @@ -215,7 +197,7 @@ configuration key. NOTE: The configuration of a Patroni node is very similar to a repmgr but shorter. When Patroni is enabled, first you can ignore -any replication setting of PostgreSQL (it is overwritten anyway). Then you can remove any `repmgr[...]` or +any replication setting of PostgreSQL (which is overwritten). Then, you can remove any `repmgr[...]` or repmgr-specific configuration as well. Especially, make sure that you remove `postgresql['shared_preload_libraries'] = 'repmgr_funcs'`. Here is an example: @@ -282,7 +264,7 @@ on each node for the changes to take effect. Generally, when Consul cluster is ready, the first node that [reconfigures](../restart_gitlab.md#omnibus-gitlab-reconfigure) becomes the leader. You do not need to sequence the nodes reconfiguration. You can run them in parallel or in any order. -If you choose an arbitrary order you do not have any predetermined leader. +If you choose an arbitrary order, you do not have any predetermined leader. #### Enable Monitoring @@ -296,7 +278,7 @@ If you enable Monitoring, it must be enabled on **all** database servers. # Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true - # Set the network addresses that the exporters will listen on + # Set the network addresses that the exporters must listen on node_exporter['listen_address'] = '0.0.0.0:9100' postgres_exporter['listen_address'] = '0.0.0.0:9187' ``` @@ -340,9 +322,9 @@ patroni['tls_ca_file'] = '/path/to/ca.pem' When TLS is enabled, mutual authentication of the API server and client is possible for all endpoints, the extent of which depends on the `patroni['tls_client_mode']` attribute: -- `none` (default): the API will not check for any client certificates. -- `optional`: client certificates are required for all [unsafe](https://patroni.readthedocs.io/en/latest/security.html#protecting-the-rest-api) API calls. -- `required`: client certificates are required for all API calls. +- `none` (default): The API does not check for any client certificates. +- `optional`: Client certificates are required for all [unsafe](https://patroni.readthedocs.io/en/latest/security.html#protecting-the-rest-api) API calls. +- `required`: Client certificates are required for all API calls. The client certificates are verified against the CA certificate that is specified with the `patroni['tls_ca_file']` attribute. Therefore, this attribute is required for mutual TLS authentication. You also need to specify PEM-formatted client certificate and private key files. @@ -450,9 +432,9 @@ authentication mode (`patroni['tls_client_mode']`), must each have the same valu #### Configure the internal load balancer -If you're running more than one PgBouncer node as recommended, then you need to set up a TCP internal load balancer to serve each correctly. This can be accomplished with any reputable TCP load balancer. +If you're running more than one PgBouncer node as recommended, you must set up a TCP internal load balancer to serve each correctly. This can be accomplished with any reputable TCP load balancer. -As an example here's how you could do it with [HAProxy](https://www.haproxy.org/): +As an example, here's how you could do it with [HAProxy](https://www.haproxy.org/): ```plaintext global @@ -554,7 +536,7 @@ Here is a list and description of each machine and the assigned IP: - `10.6.0.33`: PostgreSQL 3 - `10.6.0.41`: GitLab application -All passwords are set to `toomanysecrets`, please do not use this password or derived hashes and the `external_url` for GitLab is `http://gitlab.example.com`. +All passwords are set to `toomanysecrets`. Please do not use this password or derived hashes and the `external_url` for GitLab is `http://gitlab.example.com`. After the initial configuration, if a failover occurs, the PostgresSQL leader node changes to one of the available secondaries until it is failed back. @@ -675,7 +657,7 @@ This example uses 3 PostgreSQL servers, and 1 application node (with PgBouncer s It differs from the [recommended setup](#example-recommended-setup) by moving the Consul servers into the same servers we use for PostgreSQL. The trade-off is between reducing server counts, against the increased operational complexity of needing to deal with PostgreSQL [failover](#manual-failover-procedure-for-patroni) procedures in addition to [Consul outage recovery](../consul.md#outage-recovery) on the same set of machines. -In this example we start with all servers on the same 10.6.0.0/16 private network range, they can connect to each freely other on those addresses. +In this example, we start with all servers on the same 10.6.0.0/16 private network range; they can connect to each freely other on those addresses. Here is a list and description of each machine and the assigned IP: @@ -684,7 +666,7 @@ Here is a list and description of each machine and the assigned IP: - `10.6.0.23`: PostgreSQL 3 - `10.6.0.31`: GitLab application -All passwords are set to `toomanysecrets`, please do not use this password or derived hashes. +All passwords are set to `toomanysecrets`. Please do not use this password or derived hashes. The `external_url` for GitLab is `http://gitlab.example.com` @@ -787,7 +769,7 @@ Patroni is an opinionated solution for PostgreSQL high-availability. It takes th The fundamental [architecture](#example-recommended-setup-manual-steps) (mentioned above) does not change for Patroni. You do not need any special consideration for Patroni while provisioning your database nodes. Patroni heavily relies on Consul to store the state of the cluster and elect a leader. Any failure in Consul cluster and its leader election propagates to the Patroni cluster as well. -Patroni monitors the cluster and handles any failover. When the primary node fails it works with Consul to notify PgBouncer. On failure, Patroni handles the transitioning of the old primary to a replica and rejoins it to the cluster automatically. +Patroni monitors the cluster and handles any failover. When the primary node fails, it works with Consul to notify PgBouncer. On failure, Patroni handles the transitioning of the old primary to a replica and rejoins it to the cluster automatically. With Patroni, the connection flow is slightly different. Patroni on each node connects to Consul agent to join the cluster. Only after this point it decides if the node is the primary or a replica. Based on this decision, it configures and starts PostgreSQL which it communicates with directly over a Unix socket. This means that if the Consul cluster is not functional or does not have a leader, Patroni and by extension PostgreSQL does not start. Patroni also exposes a REST API which can be accessed via its [default port](../package_information/defaults.md) on each node. @@ -847,7 +829,7 @@ Investigate further if: - `reply_time` is not current. The `lsn` fields relate to which write-ahead-log segments have been replicated. -Run the following on the leader to find out the current LSN: +Run the following on the leader to find out the current Log Sequence Number (LSN): ```shell echo 'SELECT pg_current_wal_lsn();' | gitlab-psql @@ -918,7 +900,7 @@ patroni['remove_data_directory_on_diverged_timelines'] = false |-|-| |`use_pg_rewind`|Try running `pg_rewind` on the former cluster leader before it rejoins the database cluster.| |`remove_data_directory_on_rewind_failure`|If `pg_rewind` fails, remove the local PostgreSQL data directory and re-replicate from the current cluster leader.| -|`remove_data_directory_on_diverged_timelines`|If `pg_rewind` cannot be used and the former leader's timeline has diverged from the current one, then delete the local data directory and re-replicate from the current cluster leader.| +|`remove_data_directory_on_diverged_timelines`|If `pg_rewind` cannot be used and the former leader's timeline has diverged from the current one, delete the local data directory and re-replicate from the current cluster leader.| ### Database authorization for Patroni @@ -936,7 +918,7 @@ You can use `gitlab-ctl patroni members` to check the status of the cluster memb is the primary or a replica. When Patroni is enabled, it exclusively controls PostgreSQL's startup, -shutdown, and restart. This means, to shut down PostgreSQL on a certain node you must shutdown Patroni on the same node with: +shutdown, and restart. This means, to shut down PostgreSQL on a certain node, you must shutdown Patroni on the same node with: ```shell sudo gitlab-ctl stop patroni @@ -974,7 +956,7 @@ When a Geo secondary site is replicating from a primary site that uses `Patroni` sudo gitlab-ctl replicate-geo-database --host=<new_leader_ip> --replication-slot=<slot_name> ``` -Otherwise, the replication will not happen, even if the original node gets re-added as a follower node. This re-syncs your secondary site database and may take a long time depending on the amount of data to sync. You may also need to run `gitlab-ctl reconfigure` if replication is still not working after re-syncing. +Otherwise, the replication does not happen, even if the original node gets re-added as a follower node. This re-syncs your secondary site database and may take a long time depending on the amount of data to sync. You may also need to run `gitlab-ctl reconfigure` if replication is still not working after re-syncing. ### Recovering the Patroni cluster @@ -1097,7 +1079,7 @@ Considering these, you should carefully plan your PostgreSQL upgrade: sudo gitlab-ctl pg-upgrade -V 12 ``` -1. Check the status of the leader and cluster. You can only proceed if you have a healthy leader: +1. Check the status of the leader and cluster. You can proceed only if you have a healthy leader: ```shell gitlab-ctl patroni check-leader @@ -1192,7 +1174,7 @@ If replication is not occurring, it may be necessary to reinitialize a replica. WARNING: This is a destructive process and may lead the cluster into a bad state. Make sure that you have a healthy backup before running this process. -As a last resort, if your Patroni cluster is in an unknown/bad state and no node can start, you can +As a last resort, if your Patroni cluster is in an unknown or bad state and no node can start, you can reset the Patroni state in Consul completely, resulting in a reinitialized Patroni cluster when the first Patroni node starts. @@ -1248,7 +1230,7 @@ To fix the problem, ensure the loopback interface is included in the CIDR addres 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Check that [all the replicas are synchronized](#check-replication-status) -### Errors in Patroni logs: the requested start point is ahead of the WAL flush position +### Errors in Patroni logs: the requested start point is ahead of the Write Ahead Log (WAL) flush position This error indicates that the database is not replicating: diff --git a/doc/administration/raketasks/check.md b/doc/administration/raketasks/check.md index 7cd7ecc26f7..1d60b8c6f50 100644 --- a/doc/administration/raketasks/check.md +++ b/doc/administration/raketasks/check.md @@ -14,7 +14,8 @@ Even though Git is very resilient and tries to prevent data integrity issues, there are times when things go wrong. The following Rake tasks intend to help GitLab administrators diagnose problem repositories so they can be fixed. -There are 3 things that are checked to determine integrity. +These Rake tasks use three different methods to determine the integrity of Git +repositories. 1. Git repository file system check ([`git fsck`](https://git-scm.com/docs/git-fsck)). This step verifies the connectivity and validity of objects in the repository. @@ -37,7 +38,7 @@ exactly which repositories are causing the trouble. ### Check project code repositories This task loops through the project code repositories and runs the integrity check -described previously. If a project uses a pool repository, that will also be checked. +described previously. If a project uses a pool repository, that is also checked. Other types of Git repositories [are not checked](https://gitlab.com/gitlab-org/gitaly/-/issues/3643). **Omnibus Installation** @@ -67,7 +68,7 @@ source repository. This task loops through all repositories on the GitLab server and outputs checksums in the format `<PROJECT ID>,<CHECKSUM>`. -- If a repository doesn't exist, the project ID will have a blank checksum. +- If a repository doesn't exist, the project ID is a blank checksum. - If a repository exists but is empty, the output checksum is `0000000000000000000000000000000000000000`. - Projects which don't exist are skipped. @@ -85,9 +86,9 @@ sudo -u git -H bundle exec rake gitlab:git:checksum_projects RAILS_ENV=productio For example, if: -- Project with ID#2 doesn't exist, it will be skipped. -- Project with ID#4 doesn't have a repository, its checksum will be blank. -- Project with ID#5 has an empty repository, its checksum will be `0000000000000000000000000000000000000000`. +- Project with ID#2 doesn't exist, it is skipped. +- Project with ID#4 doesn't have a repository, its checksum is blank. +- Project with ID#5 has an empty repository, its checksum is `0000000000000000000000000000000000000000`. The output would then look something like: @@ -105,7 +106,7 @@ Optionally, specific project IDs can be checksummed by setting an environment variable `CHECKSUM_PROJECT_IDS` with a list of comma-separated integers, for example: ```shell -CHECKSUM_PROJECT_IDS="1,3" sudo gitlab-rake gitlab:git:checksum_projects +sudo CHECKSUM_PROJECT_IDS="1,3" gitlab-rake gitlab:git:checksum_projects ``` ## Uploaded files integrity @@ -115,7 +116,7 @@ These integrity checks can detect missing files. Additionally, for locally stored files, checksums are generated and stored in the database upon upload, and these checks verify them against current files. -Currently, integrity checks are supported for the following types of file: +Integrity checks are supported for the following types of file: - CI artifacts (Available from version 10.7.0) - LFS objects (Available from version 10.6.0) @@ -206,7 +207,7 @@ above. ### Dangling objects -The `gitlab:git:fsck` task can find dangling objects such as: +The `gitlab-rake gitlab:git:fsck` task can find dangling objects such as: ```plaintext dangling blob a12... diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md index 5ddab999efe..d770361864e 100644 --- a/doc/administration/raketasks/maintenance.md +++ b/doc/administration/raketasks/maintenance.md @@ -115,7 +115,7 @@ component servers like [Gitaly](../gitaly/configure_gitaly.md#run-gitaly-on-its- You may also have a look at our troubleshooting guides for: - [GitLab](../index.md#troubleshooting) -- [Omnibus GitLab](https://docs.gitlab.com/omnibus/README.html#troubleshooting) +- [Omnibus GitLab](https://docs.gitlab.com/omnibus/index.html#troubleshooting) To run `gitlab:check`, run: @@ -247,7 +247,7 @@ have been corrupted, you should reinstall the omnibus package. Sometimes you need to know if your GitLab installation can connect to a TCP service on another machine (for example a PostgreSQL or web server) -in order to troubleshoot proxy issues. +to troubleshoot proxy issues. A Rake task is included to help you with this. **Omnibus Installation** @@ -334,13 +334,13 @@ This is an experimental feature that isn't enabled by default. It requires Postg Database indexes can be rebuilt regularly to reclaim space and maintain healthy levels of index bloat over time. -In order to rebuild the two indexes with the highest estimated bloat, use the following Rake task: +To rebuild the two indexes with the highest estimated bloat, use the following Rake task: ```shell sudo gitlab-rake gitlab:db:reindex ``` -In order to target a specific index, use the following Rake task: +To target a specific index, use the following Rake task: ```shell sudo gitlab-rake gitlab:db:reindex['public.a_specific_index'] @@ -352,7 +352,7 @@ The following index types are not supported: 1. Partitioned indexes 1. Expression indexes -Optionally, this Rake task sends annotations to a Grafana (4.6 or later) endpoint. Use the following custom environment variables in order to enable annotations: +Optionally, this Rake task sends annotations to a Grafana (4.6 or later) endpoint. Use the following custom environment variables to enable annotations: 1. `GRAFANA_API_URL` - Grafana's base URL, for example `http://some-host:3000`. 1. `GRAFANA_API_KEY` - Grafana API key with at least `Editor role`. diff --git a/doc/administration/read_only_gitlab.md b/doc/administration/read_only_gitlab.md index cf45d3e15cf..2fbcb2a62e7 100644 --- a/doc/administration/read_only_gitlab.md +++ b/doc/administration/read_only_gitlab.md @@ -66,7 +66,7 @@ sudo gitlab-ctl start puma If you want to allow users to use the GitLab UI, then you'll need to ensure that the database is read-only: -1. Take a [GitLab backup](../raketasks/backup_restore.md#back-up-gitlab) +1. Take a [GitLab backup](../raketasks/backup_restore.md) in case things don't go as expected. 1. Enter PostgreSQL on the console as an administrator user: diff --git a/doc/administration/redis/replication_and_failover.md b/doc/administration/redis/replication_and_failover.md index 48db734b1af..db652a80780 100644 --- a/doc/administration/redis/replication_and_failover.md +++ b/doc/administration/redis/replication_and_failover.md @@ -647,6 +647,7 @@ persistence classes. | `shared_state` | Store session-related and other persistent data. | | `actioncable` | Pub/Sub queue backend for ActionCable. | | `trace_chunks` | Store [CI trace chunks](../job_logs.md#enable-or-disable-incremental-logging) data. | +| `rate_limiting` | Store [rate limiting](../../user/admin_area/settings/user_and_ip_rate_limits.md) state. | To make this work with Sentinel: @@ -659,6 +660,7 @@ To make this work with Sentinel: gitlab_rails['redis_shared_state_instance'] = REDIS_SHARED_STATE_URL gitlab_rails['redis_actioncable_instance'] = REDIS_ACTIONCABLE_URL gitlab_rails['redis_trace_chunks_instance'] = REDIS_TRACE_CHUNKS_URL + gitlab_rails['redis_rate_limiting_instance'] = REDIS_RATE_LIMITING_URL # Configure the Sentinels gitlab_rails['redis_cache_sentinels'] = [ @@ -681,6 +683,10 @@ To make this work with Sentinel: { host: TRACE_CHUNKS_SENTINEL_HOST, port: 26379 }, { host: TRACE_CHUNKS_SENTINEL_HOST2, port: 26379 } ] + gitlab_rails['redis_rate_limiting_sentinels'] = [ + { host: RATE_LIMITING_SENTINEL_HOST, port: 26379 }, + { host: RATE_LIMITING_SENTINEL_HOST2, port: 26379 } + ] ``` - Redis URLs should be in the format: `redis://:PASSWORD@SENTINEL_PRIMARY_NAME`, where: diff --git a/doc/administration/redis/standalone.md b/doc/administration/redis/standalone.md index 42482475d26..8a3b88780a1 100644 --- a/doc/administration/redis/standalone.md +++ b/doc/administration/redis/standalone.md @@ -44,8 +44,7 @@ Omnibus GitLab: 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Note the Redis node's IP address or hostname, port, and - Redis password. These will be necessary when [configuring the GitLab - application servers](#set-up-the-gitlab-rails-application-instance). + Redis password. These will be necessary when [configuring the GitLab application servers](#set-up-the-gitlab-rails-application-instance). [Advanced configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 0fd597e6a2d..e731085b0ce 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -12,6 +12,7 @@ full list of reference architectures, see > - **Supported users (approximate):** 10,000 > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) +> - **Cloud Native Hybrid:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) > - **Test requests per second (RPS) rates:** API: 200 RPS, Web: 20 RPS, Git (Pull): 20 RPS, Git (Push): 4 RPS > - **[Latest 10k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k)** @@ -22,18 +23,16 @@ full list of reference architectures, see | PostgreSQL<sup>1</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | `D8s v3` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | -| Redis Sentinel - Queues / Shared State<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | -| Gitaly | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | `D16s v3` | -| Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Gitaly<sup>5</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | `D16s v3` | +| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | GitLab Rails | 3 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | n/a | -| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> @@ -41,6 +40,7 @@ full list of reference architectures, see 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. <!-- markdownlint-enable MD029 --> NOTE: @@ -82,11 +82,6 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 - collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347 - collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347 - - redis_persistent <.[#FF6347]- redis_persistent_sentinel - redis_cache <.[#FF6347]- redis_cache_sentinel } cloud "**Object Storage**" as object_storage #white @@ -137,8 +132,7 @@ our [Sysbench](https://github.com/akopytov/sysbench)-based Due to better performance and availability, for data objects (such as LFS, uploads, or artifacts), using an [object storage service](#configure-the-object-storage) -is recommended instead of using NFS. Using an object storage service also -doesn't require you to provision and maintain a node. +is recommended. It's also worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and that to achieve full High Availability a third-party PostgreSQL database solution will be required. @@ -169,10 +163,8 @@ To set up GitLab and its components to accommodate up to 10,000 users: used for shared data objects. 1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, more advanced code search across your entire GitLab instance. -1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) - to have shared disk storage service as an alternative to Gitaly or object - storage. You can skip this step if you're not using GitLab Pages (which - requires NFS). +1. [Configure NFS](#configure-nfs) + to have shared disk storage service for certain GitLab operations (non Gitaly or Object Storage). The servers start on the same 10.6.0.0/24 private network range, and can connect to each other freely on these addresses. @@ -193,15 +185,9 @@ The following list includes descriptions of each server and its assigned IP: - `10.6.0.51`: Redis - Cache Primary - `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.53`: Redis - Cache Replica 2 -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 -- `10.6.0.61`: Redis - Queues Primary -- `10.6.0.62`: Redis - Queues Replica 1 -- `10.6.0.63`: Redis - Queues Replica 2 -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 +- `10.6.0.61`: Redis - Persistent Primary +- `10.6.0.62`: Redis - Persistent Replica 1 +- `10.6.0.63`: Redis - Persistent Replica 2 - `10.6.0.91`: Gitaly 1 - `10.6.0.92`: Gitaly 2 - `10.6.0.93`: Gitaly 3 @@ -792,15 +778,9 @@ to be used with GitLab. The following IPs will be used as an example: - `10.6.0.51`: Redis - Cache Primary - `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.53`: Redis - Cache Replica 2 -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 -- `10.6.0.61`: Redis - Queues Primary -- `10.6.0.62`: Redis - Queues Replica 1 -- `10.6.0.63`: Redis - Queues Replica 2 -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 +- `10.6.0.61`: Redis - Persistent Primary +- `10.6.0.62`: Redis - Persistent Replica 1 +- `10.6.0.63`: Redis - Persistent Replica 2 ### Providing your own Redis instance @@ -812,7 +792,7 @@ optional count argument to SPOP, which is required for [Merge Trains](../../ci/p Note the Redis node's IP address or hostname, port, and password (if required). These will be necessary later when configuring the [GitLab application servers](#configure-gitlab-rails). -### Configure the Redis and Sentinel Cache cluster +### Configure the Redis Cache cluster This is the section where we install and set up the new Redis Cache instances. @@ -830,8 +810,12 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' and enable Consul agent - roles(['redis_master_role', 'consul_role']) + # Specify server roles as 'redis_master_role' with sentinel and the Consul agent + roles ['redis_sentinel_role', 'redis_master_role', 'consul_role'] + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -843,8 +827,19 @@ a node and change its status from primary to replica (and vice versa). # machines to connect to it. redis['port'] = 6379 - # Set up password authentication for Redis (use the same password in all nodes). + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + # Set up password authentication for Redis and replicas (use the same password in all nodes). redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-cache' + + ## The IP of this primary Redis node. + redis['master_ip'] = '10.6.0.51' # Set the Redis Cache instance as an LRU # 90% of available RAM in MB @@ -878,10 +873,6 @@ a node and change its status from primary to replica (and vice versa). 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - #### Configure the replica Redis Cache nodes 1. SSH in to the **replica** Redis server. @@ -889,11 +880,15 @@ You can specify multiple roles, like sentinel and Redis, as: package of your choice. Be sure to both follow _only_ installation steps 1 and 2 on the page, and to select the correct Omnibus GitLab package, with the same version and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: +1. Edit `/etc/gitlab/gitlab.rb` and add same contents as the primary node in the previous section replacing `redis_master_node` with `redis_replica_node`: ```ruby - # Specify server role as 'redis_replica_role' and enable Consul agent - roles(['redis_replica_role', 'consul_role']) + # Specify server roles as 'redis_sentinel_role' and 'redis_replica_role' + roles ['redis_sentinel_role', 'redis_replica_role', 'consul_role'] + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -905,15 +900,19 @@ You can specify multiple roles, like sentinel and Redis, as: # machines to connect to it. redis['port'] = 6379 - # The same password for Redis authentication you set up for the primary node. + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + # Set up password authentication for Redis and replicas (use the same password in all nodes). redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' - # The IP of the primary Redis node. - redis['master_ip'] = '10.6.0.51' + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-cache' - # Port of primary Redis server, uncomment to change to non default. Defaults - # to `6379`. - #redis['master_port'] = 6379 + ## The IP of the primary Redis node. + redis['master_ip'] = '10.6.0.51' # Set the Redis Cache instance as an LRU # 90% of available RAM in MB @@ -949,15 +948,6 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - -These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after -a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-cache-nodes), and even after a -`gitlab-ctl reconfigure`, they will get their configuration restored by -the same Sentinels. - Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. @@ -967,133 +957,15 @@ are supported and can be added if needed. </a> </div> -#### Configure the Sentinel Cache nodes - -Now that the Redis servers are all set up, let's configure the Sentinel -servers. The following IPs will be used as an example: - -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 - -NOTE: -If you're using an external Redis Sentinel instance, be sure to exclude the -`requirepass` parameter from the Sentinel configuration. This parameter causes -clients to report `NOAUTH Authentication required.`. -[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279). - -To configure the Sentinel Cache server: - -1. SSH in to the server that will host Consul/Sentinel. -1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab - package of your choice. Be sure to both follow _only_ installation steps 1 and 2 - on the page, and to select the correct Omnibus GitLab package, with the same version - and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: - - ```ruby - roles(['redis_sentinel_role', 'consul_role']) - - ## Must be the same in every sentinel node - redis['master_name'] = 'gitlab-redis-cache' - - ## The same password for Redis authentication you set up for the primary node. - redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' - - ## The IP of the primary Redis node. - redis['master_ip'] = '10.6.0.51' - - ## Define a port so Redis can listen for TCP requests which will allow other - ## machines to connect to it. - redis['port'] = 6379 - - ## Port of primary Redis server, uncomment to change to non default. Defaults - ## to `6379`. - #redis['master_port'] = 6379 - - ## Configure Sentinel's IP - sentinel['bind'] = '10.6.0.71' - - ## Port that Sentinel listens on, uncomment to change to non default. Defaults - ## to `26379`. - #sentinel['port'] = 26379 - - ## Quorum must reflect the amount of voting sentinels it take to start a failover. - ## Value must NOT be greater then the amount of sentinels. - ## - ## The quorum can be used to tune Sentinel in two ways: - ## 1. If a the quorum is set to a value smaller than the majority of Sentinels - ## we deploy, we are basically making Sentinel more sensible to primary failures, - ## triggering a failover as soon as even just a minority of Sentinels is no longer - ## able to talk with the primary. - ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are - ## making Sentinel able to failover only when there are a very large number (larger - ## than majority) of well connected Sentinels which agree about the primary being down.s - sentinel['quorum'] = 2 - - ## Consider unresponsive server down after x amount of ms. - #sentinel['down_after_milliseconds'] = 10000 - - ## Specifies the failover timeout in milliseconds. It is used in many ways: - ## - ## - The time needed to re-start a failover after a previous failover was - ## already tried against the same primary by a given Sentinel, is two - ## times the failover timeout. - ## - ## - The time needed for a replica replicating to a wrong primary according - ## to a Sentinel current configuration, to be forced to replicate - ## with the right primary, is exactly the failover timeout (counting since - ## the moment a Sentinel detected the misconfiguration). - ## - ## - The time needed to cancel a failover that is already in progress but - ## did not produced any configuration change (REPLICAOF NO ONE yet not - ## acknowledged by the promoted replica). - ## - ## - The maximum time a failover in progress waits for all the replica to be - ## reconfigured as replicas of the new primary. However even after this time - ## the replicas will be reconfigured by the Sentinels anyway, but not with - ## the exact parallel-syncs progression as specified. - #sentinel['failover_timeout'] = 60000 - - ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true - - ## The IPs of the Consul server nodes - ## You can also use FQDNs and intermix them with IPs - consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), - } - - # Set the network addresses that the exporters will listen on - node_exporter['listen_address'] = '0.0.0.0:9100' - redis_exporter['listen_address'] = '0.0.0.0:9121' - - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false - ``` - -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace - the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. - -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. Go through the steps again for all the other Consul/Sentinel nodes, and - make sure you set up the correct IPs. - -<div align="right"> - <a type="button" class="btn btn-default" href="#setup-components"> - Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> - -### Configure the Redis and Sentinel Queues cluster +### Configure the Redis Persistent cluster -This is the section where we install and set up the new Redis Queues instances. +This is the section where we install and set up the new Redis Persistent instances. Both the primary and replica Redis nodes need the same password defined in `redis['password']`. At any time during a failover, the Sentinels can reconfigure a node and change its status from primary to replica (and vice versa). -#### Configure the primary Redis Queues node +#### Configure the primary Redis Persistent node 1. SSH in to the **Primary** Redis server. 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab @@ -1103,8 +975,12 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' and enable Consul agent - roles(['redis_master_role', 'consul_role']) + # Specify server roles as 'redis_master_role' with Sentinel and the Consul agent + roles ['redis_sentinel_role', 'redis_master_role', 'consul_role'] + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1116,8 +992,19 @@ a node and change its status from primary to replica (and vice versa). # machines to connect to it. redis['port'] = 6379 - # Set up password authentication for Redis (use the same password in all nodes). - redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + # Set up password authentication for Redis and replicas (use the same password in all nodes). + redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-persistent' + + ## The IP of this primary Redis node. + redis['master_ip'] = '10.6.0.61' ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1141,13 +1028,9 @@ a node and change its status from primary to replica (and vice versa). 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - -#### Configure the replica Redis Queues nodes +#### Configure the replica Redis Persistent nodes -1. SSH in to the **replica** Redis Queue server. +1. SSH in to the **replica** Redis Persistent server. 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to both follow _only_ installation steps 1 and 2 on the page, and to select the correct Omnibus GitLab package, with the same version @@ -1155,8 +1038,12 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_replica_role' and enable Consul agent - roles(['redis_replica_role', 'consul_role']) + # Specify server roles as 'redis_sentinel_role' and 'redis_replica_role' + roles ['redis_sentinel_role', 'redis_replica_role', 'consul_role'] + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1168,16 +1055,20 @@ You can specify multiple roles, like sentinel and Redis, as: # machines to connect to it. redis['port'] = 6379 + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + # The same password for Redis authentication you set up for the primary node. redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-persistent' # The IP of the primary Redis node. redis['master_ip'] = '10.6.0.61' - # Port of primary Redis server, uncomment to change to non default. Defaults - # to `6379`. - #redis['master_port'] = 6379 - ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1202,15 +1093,6 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - -These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after -a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-queues-nodes), and even after a -`gitlab-ctl reconfigure`, they will get their configuration restored by -the same Sentinels. - Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. @@ -1220,129 +1102,15 @@ are supported and can be added if needed. </a> </div> -#### Configure the Sentinel Queues nodes - -Now that the Redis servers are all set up, let's configure the Sentinel -servers. The following IPs will be used as an example: - -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 - -NOTE: -If you're using an external Redis Sentinel instance, be sure to exclude the -`requirepass` parameter from the Sentinel configuration. This parameter causes -clients to report `NOAUTH Authentication required.`. -[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279). - -To configure the Sentinel Queues server: - -1. SSH in to the server that will host Sentinel. -1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab - package of your choice. Be sure to both follow _only_ installation steps 1 and 2 - on the page, and to select the correct Omnibus GitLab package, with the same version - and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: - - ```ruby - roles(['redis_sentinel_role', 'consul_role']) - - ## Must be the same in every sentinel node - redis['master_name'] = 'gitlab-redis-persistent' - - ## The same password for Redis authentication you set up for the primary node. - redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' - - ## The IP of the primary Redis node. - redis['master_ip'] = '10.6.0.61' - - ## Define a port so Redis can listen for TCP requests which will allow other - ## machines to connect to it. - redis['port'] = 6379 - - ## Port of primary Redis server, uncomment to change to non default. Defaults - ## to `6379`. - #redis['master_port'] = 6379 - - ## Configure Sentinel's IP - sentinel['bind'] = '10.6.0.81' - - ## Port that Sentinel listens on, uncomment to change to non default. Defaults - ## to `26379`. - #sentinel['port'] = 26379 - - ## Quorum must reflect the amount of voting sentinels it take to start a failover. - ## Value must NOT be greater then the amount of sentinels. - ## - ## The quorum can be used to tune Sentinel in two ways: - ## 1. If a the quorum is set to a value smaller than the majority of Sentinels - ## we deploy, we are basically making Sentinel more sensible to primary failures, - ## triggering a failover as soon as even just a minority of Sentinels is no longer - ## able to talk with the primary. - ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are - ## making Sentinel able to failover only when there are a very large number (larger - ## than majority) of well connected Sentinels which agree about the primary being down.s - sentinel['quorum'] = 2 - - ## Consider unresponsive server down after x amount of ms. - #sentinel['down_after_milliseconds'] = 10000 - - ## Specifies the failover timeout in milliseconds. It is used in many ways: - ## - ## - The time needed to re-start a failover after a previous failover was - ## already tried against the same primary by a given Sentinel, is two - ## times the failover timeout. - ## - ## - The time needed for a replica replicating to a wrong primary according - ## to a Sentinel current configuration, to be forced to replicate - ## with the right primary, is exactly the failover timeout (counting since - ## the moment a Sentinel detected the misconfiguration). - ## - ## - The time needed to cancel a failover that is already in progress but - ## did not produced any configuration change (REPLICAOF NO ONE yet not - ## acknowledged by the promoted replica). - ## - ## - The maximum time a failover in progress waits for all the replica to be - ## reconfigured as replicas of the new primary. However even after this time - ## the replicas will be reconfigured by the Sentinels anyway, but not with - ## the exact parallel-syncs progression as specified. - #sentinel['failover_timeout'] = 60000 - - ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true - - ## The IPs of the Consul server nodes - ## You can also use FQDNs and intermix them with IPs - consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), - } - - # Set the network addresses that the exporters will listen on - node_exporter['listen_address'] = '0.0.0.0:9100' - redis_exporter['listen_address'] = '0.0.0.0:9121' - - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false - ``` - -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace - the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. - -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. Go through the steps again for all the other Sentinel nodes, and - make sure you set up the correct IPs. - -<div align="right"> - <a type="button" class="btn btn-default" href="#setup-components"> - Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> - ## Configure Gitaly Cluster [Gitaly Cluster](../gitaly/praefect.md) is a GitLab provided and recommended fault tolerant solution for storing Git repositories. In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. +NOTE: +Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). +For implementations with Gitaly Sharded, the same Gitaly specs should be used. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. + The recommended cluster setup includes the following components: - 3 Gitaly nodes: Replicated storage of Git repositories. @@ -1509,7 +1277,7 @@ the details of each Gitaly node that makes up the cluster. Each storage is also and this name is used in several areas of the configuration. In this guide, the name of the storage will be `default`. Also, this guide is geared towards new installs, if upgrading an existing environment to use Gitaly Cluster, you may need to use a different name. -Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more info. +Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more information. The following IPs will be used as an example: @@ -1836,7 +1604,7 @@ To configure the Sidekiq nodes, on each one: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page. -1. Open `/etc/gitlab/gitlab.rb` with your editor: +1. Create or edit `/etc/gitlab/gitlab.rb` and use the following configuration: ```ruby # Avoid running unnecessary services on the Sidekiq server @@ -1851,36 +1619,40 @@ To configure the Sidekiq nodes, on each one: gitlab_exporter['enable'] = false nginx['enable'] = false + # External URL + ## This should match the URL of the external load balancer + external_url 'https://gitlab.example.com' + # Redis ## Redis connection details ## First cluster that will host the cache gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ - {host: '10.6.0.71', port: 26379}, - {host: '10.6.0.72', port: 26379}, - {host: '10.6.0.73', port: 26379}, + {host: '10.6.0.51', port: 26379}, + {host: '10.6.0.52', port: 26379}, + {host: '10.6.0.53', port: 26379}, ] - ## Second cluster that will host the queues, shared state, and actioncable + ## Second cluster that will host the persistent queues, shared state, and actioncable gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_actioncable_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] # Gitaly Cluster @@ -2032,30 +1804,30 @@ On each node perform the following: gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ - {host: '10.6.0.71', port: 26379}, - {host: '10.6.0.72', port: 26379}, - {host: '10.6.0.73', port: 26379}, + {host: '10.6.0.51', port: 26379}, + {host: '10.6.0.52', port: 26379}, + {host: '10.6.0.53', port: 26379}, ] - ## Second cluster that will host the queues, shared state, and actionable + ## Second cluster that will host the persistent queues, shared state, and actionable gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_actioncable_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] # Set the network addresses that the exporters used for monitoring will listen on @@ -2127,7 +1899,7 @@ On each node perform the following: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. If you're [using NFS](#configure-nfs-optional): +1. If you're [using NFS](#configure-nfs): 1. If necessary, install the NFS client utility packages using the following commands: @@ -2271,7 +2043,7 @@ To configure the Monitoring node: ## Configure the object storage GitLab supports using an object storage service for holding numerous types of data. -It's recommended over [NFS](#configure-nfs-optional) and in general it's better +It's recommended over [NFS](#configure-nfs) and in general it's better in larger setups as object storage is typically much more performant, reliable, and scalable. @@ -2295,6 +2067,9 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. +GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. +In GitLab 13.6 and later, it's recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. + For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based on what features you intend to use: @@ -2341,7 +2116,7 @@ cluster alongside your instance, read how to </a> </div> -## Configure NFS (optional) +## Configure NFS [Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) are recommended over NFS wherever possible for improved performance. If you intend @@ -2355,7 +2130,7 @@ unavailable from GitLab 15.0. No further enhancements are planned for this featu Read: -- The [Gitaly and NFS deprecation notice](../gitaly/index.md#nfs-deprecation-notice). +- [Gitaly and NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation). - About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). <div align="right"> @@ -2415,12 +2190,10 @@ services where applicable): | PostgreSQL<sup>1</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | -| Redis Sentinel - Queues / Shared State<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | -| Gitaly | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | -| Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Gitaly<sup>5</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | +| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Object storage<sup>4</sup> | n/a | n/a | n/a | @@ -2430,6 +2203,7 @@ services where applicable): 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. <!-- markdownlint-enable MD029 --> NOTE: @@ -2474,11 +2248,6 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 - collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347 - collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347 - - redis_persistent <.[#FF6347]- redis_persistent_sentinel - redis_cache <.[#FF6347]- redis_cache_sentinel } cloud "**Object Storage**" as object_storage #white diff --git a/doc/administration/reference_architectures/1k_users.md b/doc/administration/reference_architectures/1k_users.md index ea40e150e58..ae832c2226f 100644 --- a/doc/administration/reference_architectures/1k_users.md +++ b/doc/administration/reference_architectures/1k_users.md @@ -18,6 +18,8 @@ many organizations. > - **Supported users (approximate):** 1,000 > - **High Availability:** No. For a highly-available environment, you can > follow a modified [3K reference architecture](3k_users.md#supported-modifications-for-lower-user-counts-ha). +> - **Cloud Native Hybrid:** No. For a cloud native hybrid environment, you +> can follow a [modified hybrid reference architecture](#cloud-native-hybrid-reference-architecture-with-helm-charts). > - **Test requests per second (RPS) rates:** API: 20 RPS, Web: 2 RPS, Git (Pull): 2 RPS, Git (Push): 1 RPS > - **[Latest 1k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/1k)** @@ -58,3 +60,12 @@ Elasticsearch cluster design and requirements are dependent on your specific data. For recommended best practices about how to set up your Elasticsearch cluster alongside your instance, read how to [choose the optimal cluster configuration](../../integration/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). + +## Cloud Native Hybrid reference architecture with Helm Charts + +Cloud Native Hybrid Reference Architecture is an alternative approach where select _stateless_ +components are deployed in Kubernetes via our official [Helm Charts](https://docs.gitlab.com/charts/), +and _stateful_ components are deployed in compute VMs with Omnibus. + +The [2k GitLab Cloud Native Hybrid](2k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) (non HA) and [3k GitLab Cloud Native Hybrid](3k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) (HA) reference architectures are the smallest we recommend in Kubernetes. +For environments that need to serve less users, you can lower the node specs. Depending on your user count, you can lower all suggested node specs as desired. However, it's recommended that you don't go lower than the [general requirements](../../install/requirements.md). diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index f500434d75b..267f81efd91 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -12,6 +12,7 @@ full list of reference architectures, see > - **Supported users (approximate):** 25,000 > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) +> - **Cloud Native Hybrid:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) > - **Test requests per second (RPS) rates:** API: 500 RPS, Web: 50 RPS, Git (Pull): 50 RPS, Git (Push): 10 RPS > - **[Latest 25k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/25k)** @@ -22,18 +23,16 @@ full list of reference architectures, see | PostgreSQL<sup>1</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | `D16s v3` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Internal load balancing node<sup>3</sup> | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | `c5.large` | `F2s v2` | -| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | -| Redis Sentinel - Queues / Shared State<sup>2</sup>| 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | -| Gitaly | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` | -| Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.large` | `D4s v3` | +| Gitaly<sup>5</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` | +| Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | GitLab Rails | 5 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | n/a | -| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> @@ -41,6 +40,7 @@ full list of reference architectures, see 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. <!-- markdownlint-enable MD029 --> NOTE: @@ -82,11 +82,6 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 - collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347 - collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347 - - redis_persistent <.[#FF6347]- redis_persistent_sentinel - redis_cache <.[#FF6347]- redis_cache_sentinel } cloud "**Object Storage**" as object_storage #white @@ -137,8 +132,7 @@ our [Sysbench](https://github.com/akopytov/sysbench)-based Due to better performance and availability, for data objects (such as LFS, uploads, or artifacts), using an [object storage service](#configure-the-object-storage) -is recommended instead of using NFS. Using an object storage service also -doesn't require you to provision and maintain a node. +is recommended. It's also worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and that to achieve full High Availability a third-party PostgreSQL database solution will be required. @@ -169,10 +163,9 @@ To set up GitLab and its components to accommodate up to 25,000 users: used for shared data objects. 1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, more advanced code search across your entire GitLab instance. -1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) - to have shared disk storage service as an alternative to Gitaly or object - storage. You can skip this step if you're not using GitLab Pages (which - requires NFS). +1. [Configure NFS](#configure-nfs) + to have shared disk storage service for certain GitLab operations (non + Gitaly or Object Storage). The servers start on the same 10.6.0.0/24 private network range, and can connect to each other freely on these addresses. @@ -193,15 +186,9 @@ The following list includes descriptions of each server and its assigned IP: - `10.6.0.51`: Redis - Cache Primary - `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.53`: Redis - Cache Replica 2 -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 -- `10.6.0.61`: Redis - Queues Primary -- `10.6.0.62`: Redis - Queues Replica 1 -- `10.6.0.63`: Redis - Queues Replica 2 -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 +- `10.6.0.61`: Redis - Persistent Primary +- `10.6.0.62`: Redis - Persistent Replica 1 +- `10.6.0.63`: Redis - Persistent Replica 2 - `10.6.0.91`: Gitaly 1 - `10.6.0.92`: Gitaly 2 - `10.6.0.93`: Gitaly 3 @@ -794,15 +781,9 @@ to be used with GitLab. The following IPs will be used as an example: - `10.6.0.51`: Redis - Cache Primary - `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.53`: Redis - Cache Replica 2 -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 -- `10.6.0.61`: Redis - Queues Primary -- `10.6.0.62`: Redis - Queues Replica 1 -- `10.6.0.63`: Redis - Queues Replica 2 -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 +- `10.6.0.61`: Redis - Persistent Primary +- `10.6.0.62`: Redis - Persistent Replica 1 +- `10.6.0.63`: Redis - Persistent Replica 2 ### Providing your own Redis instance @@ -814,7 +795,7 @@ optional count argument to SPOP, which is required for [Merge Trains](../../ci/p Note the Redis node's IP address or hostname, port, and password (if required). These will be necessary later when configuring the [GitLab application servers](#configure-gitlab-rails). -### Configure the Redis and Sentinel Cache cluster +### Configure the Redis Cache cluster This is the section where we install and set up the new Redis Cache instances. @@ -832,10 +813,14 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' and enable Consul agent - roles(['redis_master_role', 'consul_role'] + # Specify server role as 'redis_sentinel_role' 'redis_master_role' + roles ['redis_sentinel_role', 'redis_master_role', 'consul_role'] - # IP address pointing to a local IP that the other machines can reach to. + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 + + # IP address pointing to a local IP that the other machines can reach. # You can also set bind to '0.0.0.0' which listen in all interfaces. # If you really need to bind to an external accessible IP, make # sure you add extra firewall rules to prevent unauthorized access. @@ -845,8 +830,18 @@ a node and change its status from primary to replica (and vice versa). # machines to connect to it. redis['port'] = 6379 + # Port of primary Redis server for Sentinel, uncomment to change to non default. + # Defaults to `6379`. + # redis['master_port'] = 6379 + # Set up password authentication for Redis (use the same password in all nodes). - redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + + # Must be the same in every Redis node. + redis['master_name'] = 'gitlab-redis-cache' + + # The IP of this primary Redis node. + redis['master_ip'] = '10.6.0.51' # Set the Redis Cache instance as an LRU # 90% of available RAM in MB @@ -880,10 +875,6 @@ a node and change its status from primary to replica (and vice versa). 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - #### Configure the replica Redis Cache nodes 1. SSH in to the **replica** Redis server. @@ -891,14 +882,18 @@ You can specify multiple roles, like sentinel and Redis, as: package of your choice. Be sure to both follow _only_ installation steps 1 and 2 on the page, and to select the correct Omnibus GitLab package, with the same version and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: +1. Edit `/etc/gitlab/gitlab.rb` and add the same contents as the primary node in the previous section replacing `redis_master_node` with `redis_replica_node`: ```ruby # Specify server role as 'redis_replica_role' and enable Consul agent - roles(['redis_replica_role', 'consul_role'] + roles ['redis_sentinel_role', 'redis_replica_role', 'consul_role'] - # IP address pointing to a local IP that the other machines can reach to. - # You can also set bind to '0.0.0.0' which listen in all interfaces. + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = `0.0.0.0` + sentinel['quorum'] = 2 + + # IP address pointing to a local IP that the other machines can reach. + # Set bind to '0.0.0.0' to listen on all interfaces. # If you really need to bind to an external accessible IP, make # sure you add extra firewall rules to prevent unauthorized access. redis['bind'] = '10.6.0.52' @@ -907,16 +902,19 @@ You can specify multiple roles, like sentinel and Redis, as: # machines to connect to it. redis['port'] = 6379 - # The same password for Redis authentication you set up for the primary node. - redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + # Set up password authentication for Redis and replicas (use the same password in all nodes). + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + + # Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-cache' # The IP of the primary Redis node. redis['master_ip'] = '10.6.0.51' - # Port of primary Redis server, uncomment to change to non default. Defaults - # to `6379`. - #redis['master_port'] = 6379 - # Set the Redis Cache instance as an LRU # 90% of available RAM in MB redis['maxmemory'] = '13500mb' @@ -952,17 +950,7 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - -These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after -a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-cache-nodes), and even after a -`gitlab-ctl reconfigure`, they will get their configuration restored by -the same Sentinels. - -Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) -are supported and can be added if needed. + Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -970,134 +958,15 @@ are supported and can be added if needed. </a> </div> -#### Configure the Sentinel Cache nodes - -Now that the Redis servers are all set up, let's configure the Sentinel -servers. The following IPs will be used as an example: - -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 - -NOTE: -If you're using an external Redis Sentinel instance, be sure to exclude the -`requirepass` parameter from the Sentinel configuration. This parameter causes -clients to report `NOAUTH Authentication required.`. -[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279). - -To configure the Sentinel Cache server: +### Configure the Redis Persistent cluster -1. SSH in to the server that will host Consul/Sentinel. -1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab - package of your choice. Be sure to both follow _only_ installation steps 1 and 2 - on the page, and to select the correct Omnibus GitLab package, with the same version - and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: - - ```ruby - roles(['redis_sentinel_role', 'consul_role']) - - ## Must be the same in every sentinel node - redis['master_name'] = 'gitlab-redis-cache' - - ## The same password for Redis authentication you set up for the primary node. - redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' - - ## The IP of the primary Redis node. - redis['master_ip'] = '10.6.0.51' - - ## Define a port so Redis can listen for TCP requests which will allow other - ## machines to connect to it. - redis['port'] = 6379 - - ## Port of primary Redis server, uncomment to change to non default. Defaults - ## to `6379`. - #redis['master_port'] = 6379 - - ## Configure Sentinel's IP - sentinel['bind'] = '10.6.0.71' - - ## Port that Sentinel listens on, uncomment to change to non default. Defaults - ## to `26379`. - #sentinel['port'] = 26379 - - ## Quorum must reflect the amount of voting sentinels it take to start a failover. - ## Value must NOT be greater then the amount of sentinels. - ## - ## The quorum can be used to tune Sentinel in two ways: - ## 1. If a the quorum is set to a value smaller than the majority of Sentinels - ## we deploy, we are basically making Sentinel more sensible to primary failures, - ## triggering a failover as soon as even just a minority of Sentinels is no longer - ## able to talk with the primary. - ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are - ## making Sentinel able to failover only when there are a very large number (larger - ## than majority) of well connected Sentinels which agree about the primary being down.s - sentinel['quorum'] = 2 - - ## Consider unresponsive server down after x amount of ms. - #sentinel['down_after_milliseconds'] = 10000 - - ## Specifies the failover timeout in milliseconds. It is used in many ways: - ## - ## - The time needed to re-start a failover after a previous failover was - ## already tried against the same primary by a given Sentinel, is two - ## times the failover timeout. - ## - ## - The time needed for a replica replicating to a wrong primary according - ## to a Sentinel current configuration, to be forced to replicate - ## with the right primary, is exactly the failover timeout (counting since - ## the moment a Sentinel detected the misconfiguration). - ## - ## - The time needed to cancel a failover that is already in progress but - ## did not produced any configuration change (REPLICAOF NO ONE yet not - ## acknowledged by the promoted replica). - ## - ## - The maximum time a failover in progress waits for all the replica to be - ## reconfigured as replicas of the new primary. However even after this time - ## the replicas will be reconfigured by the Sentinels anyway, but not with - ## the exact parallel-syncs progression as specified. - #sentinel['failover_timeout'] = 60000 - - ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true - - ## The IPs of the Consul server nodes - ## You can also use FQDNs and intermix them with IPs - consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), - } - - # Set the network addresses that the exporters will listen on - node_exporter['listen_address'] = '0.0.0.0:9100' - redis_exporter['listen_address'] = '0.0.0.0:9121' - - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false - ``` - -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace - the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. - -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -1. Go through the steps again for all the other Consul/Sentinel nodes, and - make sure you set up the correct IPs. - -<div align="right"> - <a type="button" class="btn btn-default" href="#setup-components"> - Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> - -### Configure the Redis and Sentinel Queues cluster - -This is the section where we install and set up the new Redis Queues instances. +This is the section where we install and set up the new Redis Persistent instances. Both the primary and replica Redis nodes need the same password defined in `redis['password']`. At any time during a failover, the Sentinels can reconfigure a node and change its status from primary to replica (and vice versa). -#### Configure the primary Redis Queues node +#### Configure the primary Redis Persistent node 1. SSH in to the **Primary** Redis server. 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab @@ -1107,8 +976,12 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' and enable Consul agent - roles(['redis_master_role', 'consul_role']) + # Specify server roles as 'redis_sentinel_role' and 'redis_master_role' + roles ['redis_sentinel_role', 'redis_master_role', 'consul_role'] + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1120,8 +993,19 @@ a node and change its status from primary to replica (and vice versa). # machines to connect to it. redis['port'] = 6379 - # Set up password authentication for Redis (use the same password in all nodes). - redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + # Set up password authentication for Redis and replicas (use the same password in all nodes). + redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-persistent' + + ## The IP of this primary Redis node. + redis['master_ip'] = '10.6.0.61' ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1145,13 +1029,9 @@ a node and change its status from primary to replica (and vice versa). 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - -#### Configure the replica Redis Queues nodes +#### Configure the replica Redis Persistent nodes -1. SSH in to the **replica** Redis Queue server. +1. SSH in to the **replica** Redis Persistent server. 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to both follow _only_ installation steps 1 and 2 on the page, and to select the correct Omnibus GitLab package, with the same version @@ -1160,7 +1040,11 @@ You can specify multiple roles, like sentinel and Redis, as: ```ruby # Specify server role as 'redis_replica_role' and enable Consul agent - roles(['redis_replica_role', 'consul_role']) + roles ['redis_sentinel_role', 'redis_replica_role', 'consul_role'] + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1172,16 +1056,19 @@ You can specify multiple roles, like sentinel and Redis, as: # machines to connect to it. redis['port'] = 6379 + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + # The same password for Redis authentication you set up for the primary node. - redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-persistent' # The IP of the primary Redis node. redis['master_ip'] = '10.6.0.61' - # Port of primary Redis server, uncomment to change to non default. Defaults - # to `6379`. - #redis['master_port'] = 6379 - ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1211,15 +1098,6 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - -These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after -a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-queues-nodes), and even after a -`gitlab-ctl reconfigure`, they will get their configuration restored by -the same Sentinels. - Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. @@ -1229,138 +1107,16 @@ are supported and can be added if needed. </a> </div> -#### Configure the Sentinel Queues nodes - -Now that the Redis servers are all set up, let's configure the Sentinel -servers. The following IPs will be used as an example: - -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 - -NOTE: -If you're using an external Redis Sentinel instance, be sure to exclude the -`requirepass` parameter from the Sentinel configuration. This parameter causes -clients to report `NOAUTH Authentication required.`. -[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279). - -To configure the Sentinel Queues server: - -1. SSH in to the server that will host Sentinel. -1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab - package of your choice. Be sure to both follow _only_ installation steps 1 and 2 - on the page, and to select the correct Omnibus GitLab package, with the same version - and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: - - ```ruby - roles(['redis_sentinel_role', 'consul_role']) - - ## Must be the same in every sentinel node - redis['master_name'] = 'gitlab-redis-persistent' - - ## The same password for Redis authentication you set up for the primary node. - redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' - - ## The IP of the primary Redis node. - redis['master_ip'] = '10.6.0.61' - - ## Define a port so Redis can listen for TCP requests which will allow other - ## machines to connect to it. - redis['port'] = 6379 - - ## Port of primary Redis server, uncomment to change to non default. Defaults - ## to `6379`. - #redis['master_port'] = 6379 - - ## Configure Sentinel's IP - sentinel['bind'] = '10.6.0.81' - - ## Port that Sentinel listens on, uncomment to change to non default. Defaults - ## to `26379`. - #sentinel['port'] = 26379 - - ## Quorum must reflect the amount of voting sentinels it take to start a failover. - ## Value must NOT be greater then the amount of sentinels. - ## - ## The quorum can be used to tune Sentinel in two ways: - ## 1. If a the quorum is set to a value smaller than the majority of Sentinels - ## we deploy, we are basically making Sentinel more sensible to primary failures, - ## triggering a failover as soon as even just a minority of Sentinels is no longer - ## able to talk with the primary. - ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are - ## making Sentinel able to failover only when there are a very large number (larger - ## than majority) of well connected Sentinels which agree about the primary being down.s - sentinel['quorum'] = 2 - - ## Consider unresponsive server down after x amount of ms. - #sentinel['down_after_milliseconds'] = 10000 - - ## Specifies the failover timeout in milliseconds. It is used in many ways: - ## - ## - The time needed to re-start a failover after a previous failover was - ## already tried against the same primary by a given Sentinel, is two - ## times the failover timeout. - ## - ## - The time needed for a replica replicating to a wrong primary according - ## to a Sentinel current configuration, to be forced to replicate - ## with the right primary, is exactly the failover timeout (counting since - ## the moment a Sentinel detected the misconfiguration). - ## - ## - The time needed to cancel a failover that is already in progress but - ## did not produced any configuration change (REPLICAOF NO ONE yet not - ## acknowledged by the promoted replica). - ## - ## - The maximum time a failover in progress waits for all the replica to be - ## reconfigured as replicas of the new primary. However even after this time - ## the replicas will be reconfigured by the Sentinels anyway, but not with - ## the exact parallel-syncs progression as specified. - #sentinel['failover_timeout'] = 60000 - - ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true - - ## The IPs of the Consul server nodes - ## You can also use FQDNs and intermix them with IPs - consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), - } - - # Set the network addresses that the exporters will listen on - node_exporter['listen_address'] = '0.0.0.0:9100' - redis_exporter['listen_address'] = '0.0.0.0:9121' - - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false - ``` - -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace - the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. - -1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: - - ```shell - sudo touch /etc/gitlab/skip-auto-reconfigure - ``` - - Only the primary GitLab application server should handle migrations. - -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -1. Go through the steps again for all the other Sentinel nodes, and - make sure you set up the correct IPs. - -<div align="right"> - <a type="button" class="btn btn-default" href="#setup-components"> - Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> - ## Configure Gitaly Cluster -[Gitaly Cluster](../gitaly/praefect.md) is a GitLab provided and recommended fault tolerant solution for storing Git repositories. +[Gitaly Cluster](../gitaly/praefect.md) is a GitLab-provided and recommended +fault tolerant solution for storing Git repositories. In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. +NOTE: +Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). +For implementations with Gitaly Sharded, the same Gitaly specs should be used. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. + The recommended cluster setup includes the following components: - 3 Gitaly nodes: Replicated storage of Git repositories. @@ -1527,7 +1283,7 @@ the details of each Gitaly node that makes up the cluster. Each storage is also and this name is used in several areas of the configuration. In this guide, the name of the storage will be `default`. Also, this guide is geared towards new installs, if upgrading an existing environment to use Gitaly Cluster, you may need to use a different name. -Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more info. +Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more information. The following IPs will be used as an example: @@ -1854,7 +1610,7 @@ To configure the Sidekiq nodes, on each one: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page. -1. Open `/etc/gitlab/gitlab.rb` with your editor: +1. Create or edit `/etc/gitlab/gitlab.rb` and use the following configuration: ```ruby # Avoid running unnecessary services on the Sidekiq server @@ -1869,36 +1625,40 @@ To configure the Sidekiq nodes, on each one: gitlab_exporter['enable'] = false nginx['enable'] = false + # External URL + ## This should match the URL of the external load balancer + external_url 'https://gitlab.example.com' + # Redis ## Redis connection details ## First cluster that will host the cache gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ - {host: '10.6.0.71', port: 26379}, - {host: '10.6.0.72', port: 26379}, - {host: '10.6.0.73', port: 26379}, + {host: '10.6.0.51', port: 26379}, + {host: '10.6.0.52', port: 26379}, + {host: '10.6.0.53', port: 26379}, ] - ## Second cluster that will host the queues, shared state, and actioncable + ## Second cluster that will host the persistent queues, shared state, and actioncable gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_actioncable_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] # Gitaly Cluster @@ -2052,9 +1812,9 @@ On each node perform the following: gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ - {host: '10.6.0.71', port: 26379}, - {host: '10.6.0.72', port: 26379}, - {host: '10.6.0.73', port: 26379}, + {host: '10.6.0.51', port: 26379}, + {host: '10.6.0.52', port: 26379}, + {host: '10.6.0.53', port: 26379}, ] ## Second cluster that will host the queues, shared state, and actionable @@ -2063,19 +1823,19 @@ On each node perform the following: gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_actioncable_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] # Set the network addresses that the exporters used for monitoring will listen on @@ -2147,7 +1907,7 @@ On each node perform the following: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. If you're [using NFS](#configure-nfs-optional): +1. If you're [using NFS](#configure-nfs): 1. If necessary, install the NFS client utility packages using the following commands: @@ -2289,9 +2049,9 @@ To configure the Monitoring node: ## Configure the object storage GitLab supports using an object storage service for holding numerous types of data. -It's recommended over [NFS](#configure-nfs-optional) and in general it's better -in larger setups as object storage is typically much more performant, reliable, -and scalable. +Object storage is also recommended over [NFS](#configure-nfs) and in general +it's better in larger setups as object storage is typically much more performant, +reliable, and scalable. GitLab has been tested on a number of object storage providers: @@ -2313,6 +2073,9 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. +GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. +In GitLab 13.6 and later, it's recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. + For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based on what features you intend to use: @@ -2359,7 +2122,7 @@ cluster alongside your instance, read how to </a> </div> -## Configure NFS (optional) +## Configure NFS [Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) are recommended over NFS wherever possible for improved performance. If you intend @@ -2373,7 +2136,7 @@ unavailable from GitLab 15.0. No further enhancements are planned for this featu Read: -- The [Gitaly and NFS deprecation notice](../gitaly/index.md#nfs-deprecation-notice). +- [Gitaly and NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation). - About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). ## Cloud Native Hybrid reference architecture with Helm Charts (alternative) @@ -2427,12 +2190,10 @@ services where applicable): | PostgreSQL<sup>1</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Internal load balancing node<sup>3</sup> | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | -| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | -| Redis Sentinel - Queues / Shared State<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | -| Gitaly | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | -| Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | +| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Gitaly<sup>5</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | +| Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Object storage<sup>4</sup> | n/a | n/a | n/a | @@ -2442,6 +2203,7 @@ services where applicable): 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. <!-- markdownlint-enable MD029 --> NOTE: @@ -2486,11 +2248,6 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 - collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347 - collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347 - - redis_persistent <.[#FF6347]- redis_persistent_sentinel - redis_cache <.[#FF6347]- redis_cache_sentinel } cloud "**Object Storage**" as object_storage #white diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index 99dd29c3a83..23b15b563f7 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -13,6 +13,7 @@ For a full list of reference architectures, see > - **Supported users (approximate):** 2,000 > - **High Availability:** No. For a highly-available environment, you can > follow a modified [3K reference architecture](3k_users.md#supported-modifications-for-lower-user-counts-ha). +> - **Cloud Native Hybrid:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) > - **Test requests per second (RPS) rates:** API: 40 RPS, Web: 4 RPS, Git (Pull): 4 RPS, Git (Push): 1 RPS > - **[Latest 2k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/2k)** @@ -302,8 +303,8 @@ further configuration steps. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Note the PostgreSQL node's IP address or hostname, port, and - plain text password. These will be necessary when configuring the [GitLab - application server](#configure-gitlab-rails) later. + plain text password. These will be necessary when configuring the + [GitLab application server](#configure-gitlab-rails) later. Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/database.html) are supported and can be added if needed. @@ -385,8 +386,8 @@ Omnibus: 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Note the Redis node's IP address or hostname, port, and - Redis password. These will be necessary when [configuring the GitLab - application servers](#configure-gitlab-rails) later. + Redis password. These will be necessary when + [configuring the GitLab application servers](#configure-gitlab-rails) later. Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. @@ -903,6 +904,9 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. +GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. +In GitLab 13.6 and later, it's recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. + For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based on what features you intend to use: @@ -964,7 +968,7 @@ unavailable from GitLab 15.0. No further enhancements are planned for this featu Read: -- The [Gitaly and NFS deprecation notice](../gitaly/index.md#nfs-deprecation-notice). +- [Gitaly and NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation). - About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). ## Cloud Native Hybrid reference architecture with Helm Charts (alternative) diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index da36968f053..575dd22b729 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -22,6 +22,7 @@ For a full list of reference architectures, see > - **Supported users (approximate):** 3,000 > - **High Availability:** Yes, although [Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution +> - **Cloud Native Hybrid:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) > - **Test requests per second (RPS) rates:** API: 60 RPS, Web: 6 RPS, Git (Pull): 6 RPS, Git (Push): 1 RPS > - **[Latest 3k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/3k)** @@ -33,8 +34,8 @@ For a full list of reference architectures, see | PostgreSQL<sup>1</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Gitaly | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Gitaly<sup>5</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Sidekiq | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | | GitLab Rails | 3 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | @@ -48,6 +49,7 @@ For a full list of reference architectures, see 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. <!-- markdownlint-enable MD029 --> NOTE: @@ -1065,6 +1067,10 @@ The following IPs will be used as an example: [Gitaly Cluster](../gitaly/praefect.md) is a GitLab provided and recommended fault tolerant solution for storing Git repositories. In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. +NOTE: +Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). +For implementations with Gitaly Sharded, the same Gitaly specs should be used. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. + The recommended cluster setup includes the following components: - 3 Gitaly nodes: Replicated storage of Git repositories. @@ -1230,7 +1236,7 @@ the details of each Gitaly node that makes up the cluster. Each storage is also and this name is used in several areas of the configuration. In this guide, the name of the storage will be `default`. Also, this guide is geared towards new installs, if upgrading an existing environment to use Gitaly Cluster, you may need to use a different name. -Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more info. +Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more information. The following IPs will be used as an example: @@ -1559,7 +1565,7 @@ To configure the Sidekiq nodes, one each one: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page. -1. Open `/etc/gitlab/gitlab.rb` with your editor: +1. Create or edit `/etc/gitlab/gitlab.rb` and use the following configuration: ```ruby # Avoid running unnecessary services on the Sidekiq server @@ -1574,6 +1580,10 @@ To configure the Sidekiq nodes, one each one: gitlab_exporter['enable'] = false nginx['enable'] = false + # External URL + ## This should match the URL of the external load balancer + external_url 'https://gitlab.example.com' + # Redis redis['master_name'] = 'gitlab-redis' @@ -2011,6 +2021,9 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. +GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. +In GitLab 13.6 and later, it's recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. + For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based on what features you intend to use: @@ -2071,7 +2084,7 @@ unavailable from GitLab 15.0. No further enhancements are planned for this featu Read: -- The [Gitaly and NFS deprecation notice](../gitaly/index.md#nfs-deprecation-notice). +- [Gitaly and NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation). - About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). ## Supported modifications for lower user counts (HA) @@ -2150,8 +2163,8 @@ services where applicable): | PostgreSQL<sup>1</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Gitaly | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Gitaly<sup>5</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Object storage<sup>4</sup> | n/a | n/a | n/a | @@ -2161,6 +2174,7 @@ services where applicable): 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. <!-- markdownlint-enable MD029 --> NOTE: diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index b071b48cbd9..be44f464e7e 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -12,6 +12,7 @@ full list of reference architectures, see > - **Supported users (approximate):** 50,000 > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) +> - **Cloud Native Hybrid:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) > - **Test requests per second (RPS) rates:** API: 1000 RPS, Web: 100 RPS, Git (Pull): 100 RPS, Git (Push): 20 RPS > - **[Latest 50k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/50k)** @@ -22,18 +23,16 @@ full list of reference architectures, see | PostgreSQL<sup>1</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Internal load balancing node<sup>3</sup> | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | -| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | -| Redis Sentinel - Queues / Shared State<sup>2</sup>| 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | -| Gitaly | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | `m5.16xlarge` | `D64s v3` | -| Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Gitaly<sup>5</sup> | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | `m5.16xlarge` | `D64s v3` | +| Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | GitLab Rails | 12 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | n/a | -| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> @@ -41,6 +40,7 @@ full list of reference architectures, see 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. <!-- markdownlint-enable MD029 --> NOTE: @@ -82,11 +82,6 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 - collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347 - collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347 - - redis_persistent <.[#FF6347]- redis_persistent_sentinel - redis_cache <.[#FF6347]- redis_cache_sentinel } cloud "**Object Storage**" as object_storage #white @@ -137,8 +132,7 @@ our [Sysbench](https://github.com/akopytov/sysbench)-based Due to better performance and availability, for data objects (such as LFS, uploads, or artifacts), using an [object storage service](#configure-the-object-storage) -is recommended instead of using NFS. Using an object storage service also -doesn't require you to provision and maintain a node. +is recommended. It's also worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and that to achieve full High Availability a third-party PostgreSQL database solution will be required. @@ -169,10 +163,8 @@ To set up GitLab and its components to accommodate up to 50,000 users: used for shared data objects. 1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, more advanced code search across your entire GitLab instance. -1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) - to have shared disk storage service as an alternative to Gitaly or object - storage. You can skip this step if you're not using GitLab Pages (which - requires NFS). +1. [Configure NFS](#configure-nfs) + to have shared disk storage service for certain GitLab operations (non Gitaly or Object Storage). The servers start on the same 10.6.0.0/24 private network range, and can connect to each other freely on these addresses. @@ -193,15 +185,9 @@ The following list includes descriptions of each server and its assigned IP: - `10.6.0.51`: Redis - Cache Primary - `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.53`: Redis - Cache Replica 2 -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 -- `10.6.0.61`: Redis - Queues Primary -- `10.6.0.62`: Redis - Queues Replica 1 -- `10.6.0.63`: Redis - Queues Replica 2 -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 +- `10.6.0.61`: Redis - Persistent Primary +- `10.6.0.62`: Redis - Persistent Replica 1 +- `10.6.0.63`: Redis - Persistent Replica 2 - `10.6.0.91`: Gitaly 1 - `10.6.0.92`: Gitaly 2 - `10.6.0.93`: Gitaly 3 @@ -802,15 +788,9 @@ to be used with GitLab. The following IPs will be used as an example: - `10.6.0.51`: Redis - Cache Primary - `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.53`: Redis - Cache Replica 2 -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 -- `10.6.0.61`: Redis - Queues Primary -- `10.6.0.62`: Redis - Queues Replica 1 -- `10.6.0.63`: Redis - Queues Replica 2 -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 +- `10.6.0.61`: Redis - Persistent Primary +- `10.6.0.62`: Redis - Persistent Replica 1 +- `10.6.0.63`: Redis - Persistent Replica 2 ### Providing your own Redis instance @@ -822,7 +802,7 @@ optional count argument to SPOP, which is required for [Merge Trains](../../ci/p Note the Redis node's IP address or hostname, port, and password (if required). These will be necessary later when configuring the [GitLab application servers](#configure-gitlab-rails). -### Configure the Redis and Sentinel Cache cluster +### Configure the Redis Cache cluster This is the section where we install and set up the new Redis Cache instances. @@ -840,8 +820,12 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' and enable Consul agent - roles(['redis_master_role', 'consul_role']) + # Specify server role as 'redis_master_role' with Sentinel and enable Consul agent + roles(['redis_sentinel_role', 'redis_master_role', 'consul_role']) + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -853,8 +837,19 @@ a node and change its status from primary to replica (and vice versa). # machines to connect to it. redis['port'] = 6379 - # Set up password authentication for Redis (use the same password in all nodes). + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + # Set up password authentication for Redis and replicas (use the same password in all nodes). redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-cache' + + ## The IP of this primary Redis node. + redis['master_ip'] = '10.6.0.51' # Set the Redis Cache instance as an LRU # 90% of available RAM in MB @@ -888,10 +883,6 @@ a node and change its status from primary to replica (and vice versa). 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - #### Configure the replica Redis Cache nodes 1. SSH in to the **replica** Redis server. @@ -899,11 +890,15 @@ You can specify multiple roles, like sentinel and Redis, as: package of your choice. Be sure to both follow _only_ installation steps 1 and 2 on the page, and to select the correct Omnibus GitLab package, with the same version and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: +1. Edit `/etc/gitlab/gitlab.rb` and add the same contents as the priimary node in the previous section by replacing `redis_master_node` with `redis_replica_node`: ```ruby - # Specify server role as 'redis_replica_role' and enable Consul agent - roles(['redis_replica_role', 'consul_role']) + # Specify server role as 'redis_replica_role' with Sentinel and enable Consul agent + roles(['roles_sentinel_role', 'redis_replica_role', 'consul_role']) + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -915,15 +910,19 @@ You can specify multiple roles, like sentinel and Redis, as: # machines to connect to it. redis['port'] = 6379 - # The same password for Redis authentication you set up for the primary node. + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + # Set up password authentication for Redis and replicas (use the same password in all nodes). redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' - # The IP of the primary Redis node. - redis['master_ip'] = '10.6.0.51' + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-cache' - # Port of primary Redis server, uncomment to change to non default. Defaults - # to `6379`. - #redis['master_port'] = 6379 + ## The IP of the primary Redis node. + redis['master_ip'] = '10.6.0.51' # Set the Redis Cache instance as an LRU # 90% of available RAM in MB @@ -952,25 +951,18 @@ You can specify multiple roles, like sentinel and Redis, as: gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace - the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus + node you configured and add or replace the file of the same name on this + server. If this is the first Omnibus node you are configuring then you + can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes + to take effect. -1. Go through the steps again for all the other replica nodes, and + 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - -These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after -a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-cache-nodes), and even after a -`gitlab-ctl reconfigure`, they will get their configuration restored by -the same Sentinels. - -Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) -are supported and can be added if needed. + Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -978,126 +970,7 @@ are supported and can be added if needed. </a> </div> -#### Configure the Sentinel Cache nodes - -Now that the Redis servers are all set up, let's configure the Sentinel -servers. The following IPs will be used as an example: - -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 - -NOTE: -If you're using an external Redis Sentinel instance, be sure to exclude the -`requirepass` parameter from the Sentinel configuration. This parameter causes -clients to report `NOAUTH Authentication required.`. -[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279). - -To configure the Sentinel Cache server: - -1. SSH in to the server that will host Consul/Sentinel. -1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab - package of your choice. Be sure to both follow _only_ installation steps 1 and 2 - on the page, and to select the correct Omnibus GitLab package, with the same version - and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: - - ```ruby - roles(['redis_sentinel_role', 'consul_role']) - - ## Must be the same in every sentinel node - redis['master_name'] = 'gitlab-redis-cache' - - ## The same password for Redis authentication you set up for the primary node. - redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' - - ## The IP of the primary Redis node. - redis['master_ip'] = '10.6.0.51' - - ## Define a port so Redis can listen for TCP requests which will allow other - ## machines to connect to it. - redis['port'] = 6379 - - ## Port of primary Redis server, uncomment to change to non default. Defaults - ## to `6379`. - #redis['master_port'] = 6379 - - ## Configure Sentinel's IP - sentinel['bind'] = '10.6.0.71' - - ## Port that Sentinel listens on, uncomment to change to non default. Defaults - ## to `26379`. - #sentinel['port'] = 26379 - - ## Quorum must reflect the amount of voting sentinels it take to start a failover. - ## Value must NOT be greater then the amount of sentinels. - ## - ## The quorum can be used to tune Sentinel in two ways: - ## 1. If a the quorum is set to a value smaller than the majority of Sentinels - ## we deploy, we are basically making Sentinel more sensible to primary failures, - ## triggering a failover as soon as even just a minority of Sentinels is no longer - ## able to talk with the primary. - ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are - ## making Sentinel able to failover only when there are a very large number (larger - ## than majority) of well connected Sentinels which agree about the primary being down.s - sentinel['quorum'] = 2 - - ## Consider unresponsive server down after x amount of ms. - #sentinel['down_after_milliseconds'] = 10000 - - ## Specifies the failover timeout in milliseconds. It is used in many ways: - ## - ## - The time needed to re-start a failover after a previous failover was - ## already tried against the same primary by a given Sentinel, is two - ## times the failover timeout. - ## - ## - The time needed for a replica replicating to a wrong primary according - ## to a Sentinel current configuration, to be forced to replicate - ## with the right primary, is exactly the failover timeout (counting since - ## the moment a Sentinel detected the misconfiguration). - ## - ## - The time needed to cancel a failover that is already in progress but - ## did not produced any configuration change (REPLICAOF NO ONE yet not - ## acknowledged by the promoted replica). - ## - ## - The maximum time a failover in progress waits for all the replica to be - ## reconfigured as replicas of the new primary. However even after this time - ## the replicas will be reconfigured by the Sentinels anyway, but not with - ## the exact parallel-syncs progression as specified. - #sentinel['failover_timeout'] = 60000 - - ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true - - ## The IPs of the Consul server nodes - ## You can also use FQDNs and intermix them with IPs - consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), - } - - # Set the network addresses that the exporters will listen on - node_exporter['listen_address'] = '0.0.0.0:9100' - redis_exporter['listen_address'] = '0.0.0.0:9121' - - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false - ``` - -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace - the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. - -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -1. Go through the steps again for all the other Consul/Sentinel nodes, and - make sure you set up the correct IPs. - -<div align="right"> - <a type="button" class="btn btn-default" href="#setup-components"> - Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> - -### Configure the Redis and Sentinel Queues cluster +### Configure the Redis Persistent cluster This is the section where we install and set up the new Redis Queues instances. @@ -1105,7 +978,7 @@ Both the primary and replica Redis nodes need the same password defined in `redis['password']`. At any time during a failover, the Sentinels can reconfigure a node and change its status from primary to replica (and vice versa). -#### Configure the primary Redis Queues node +#### Configure the primary Redis Persistent node 1. SSH in to the **Primary** Redis server. 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab @@ -1115,8 +988,12 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' and enable Consul agent - roles(['redis_master_role', 'consul_role']) + # Specify server roles as 'redis_master_role' with Sentinel and enable the Consul agent + roles ['redis_sentinel_role', 'redis_master_role', 'consul_role'] + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1128,8 +1005,19 @@ a node and change its status from primary to replica (and vice versa). # machines to connect to it. redis['port'] = 6379 - # Set up password authentication for Redis (use the same password in all nodes). - redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + # Set up password authentication for Redis and replicas (use the same password in all nodes). + redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-persistent' + + ## The IP of this primary Redis node. + redis['master_ip'] = '10.6.0.61' ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1153,13 +1041,9 @@ a node and change its status from primary to replica (and vice versa). 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). +#### Configure the replica Redis Persistent nodes -#### Configure the replica Redis Queues nodes - -1. SSH in to the **replica** Redis Queue server. +1. SSH in to the **replica** Redis Persistent server. 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to both follow _only_ installation steps 1 and 2 on the page, and to select the correct Omnibus GitLab package, with the same version @@ -1167,8 +1051,12 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_replica_role' and enable Consul agent - roles(['redis_replica_role', 'consul_role']) + # Specify server roles as 'redis_replica_role' with Sentinel and enable Consul agent + roles ['redis_sentinel_role', 'redis_replica_role', 'consul_role'] + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1180,16 +1068,20 @@ You can specify multiple roles, like sentinel and Redis, as: # machines to connect to it. redis['port'] = 6379 + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + # The same password for Redis authentication you set up for the primary node. redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-persistent' # The IP of the primary Redis node. redis['master_ip'] = '10.6.0.61' - # Port of primary Redis server, uncomment to change to non default. Defaults - # to `6379`. - #redis['master_port'] = 6379 - ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1215,144 +1107,7 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - -These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after -a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-queues-nodes), and even after a -`gitlab-ctl reconfigure`, they will get their configuration restored by -the same Sentinels. - -Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) -are supported and can be added if needed. - -<div align="right"> - <a type="button" class="btn btn-default" href="#setup-components"> - Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> - -#### Configure the Sentinel Queues nodes - -Now that the Redis servers are all set up, let's configure the Sentinel -servers. The following IPs will be used as an example: - -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 - -NOTE: -If you're using an external Redis Sentinel instance, be sure to exclude the -`requirepass` parameter from the Sentinel configuration. This parameter causes -clients to report `NOAUTH Authentication required.`. -[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279). - -To configure the Sentinel Queues server: - -1. SSH in to the server that will host Sentinel. -1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab - package of your choice. Be sure to both follow _only_ installation steps 1 and 2 - on the page, and to select the correct Omnibus GitLab package, with the same version - and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: - - ```ruby - roles(['redis_sentinel_role', 'consul_role']) - - ## Must be the same in every sentinel node - redis['master_name'] = 'gitlab-redis-persistent' - - ## The same password for Redis authentication you set up for the primary node. - redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' - - ## The IP of the primary Redis node. - redis['master_ip'] = '10.6.0.61' - - ## Define a port so Redis can listen for TCP requests which will allow other - ## machines to connect to it. - redis['port'] = 6379 - - ## Port of primary Redis server, uncomment to change to non default. Defaults - ## to `6379`. - #redis['master_port'] = 6379 - - ## Configure Sentinel's IP - sentinel['bind'] = '10.6.0.81' - - ## Port that Sentinel listens on, uncomment to change to non default. Defaults - ## to `26379`. - #sentinel['port'] = 26379 - - ## Quorum must reflect the amount of voting sentinels it take to start a failover. - ## Value must NOT be greater then the amount of sentinels. - ## - ## The quorum can be used to tune Sentinel in two ways: - ## 1. If a the quorum is set to a value smaller than the majority of Sentinels - ## we deploy, we are basically making Sentinel more sensible to primary failures, - ## triggering a failover as soon as even just a minority of Sentinels is no longer - ## able to talk with the primary. - ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are - ## making Sentinel able to failover only when there are a very large number (larger - ## than majority) of well connected Sentinels which agree about the primary being down.s - sentinel['quorum'] = 2 - - ## Consider unresponsive server down after x amount of ms. - #sentinel['down_after_milliseconds'] = 10000 - - ## Specifies the failover timeout in milliseconds. It is used in many ways: - ## - ## - The time needed to re-start a failover after a previous failover was - ## already tried against the same primary by a given Sentinel, is two - ## times the failover timeout. - ## - ## - The time needed for a replica replicating to a wrong primary according - ## to a Sentinel current configuration, to be forced to replicate - ## with the right primary, is exactly the failover timeout (counting since - ## the moment a Sentinel detected the misconfiguration). - ## - ## - The time needed to cancel a failover that is already in progress but - ## did not produced any configuration change (REPLICAOF NO ONE yet not - ## acknowledged by the promoted replica). - ## - ## - The maximum time a failover in progress waits for all the replica to be - ## reconfigured as replicas of the new primary. However even after this time - ## the replicas will be reconfigured by the Sentinels anyway, but not with - ## the exact parallel-syncs progression as specified. - #sentinel['failover_timeout'] = 60000 - - ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true - - ## The IPs of the Consul server nodes - ## You can also use FQDNs and intermix them with IPs - consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), - } - - # Set the network addresses that the exporters will listen on - node_exporter['listen_address'] = '0.0.0.0:9100' - redis_exporter['listen_address'] = '0.0.0.0:9121' - - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false - ``` - -1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: - - ```shell - sudo touch /etc/gitlab/skip-auto-reconfigure - ``` - - Only the primary GitLab application server should handle migrations. - -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace - the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. - -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -1. Go through the steps again for all the other Sentinel nodes, and - make sure you set up the correct IPs. +Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -1365,6 +1120,10 @@ To configure the Sentinel Queues server: [Gitaly Cluster](../gitaly/praefect.md) is a GitLab provided and recommended fault tolerant solution for storing Git repositories. In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. +NOTE: +Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). +For implementations with Gitaly Sharded, the same Gitaly specs should be used. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. + The recommended cluster setup includes the following components: - 3 Gitaly nodes: Replicated storage of Git repositories. @@ -1531,7 +1290,7 @@ the details of each Gitaly node that makes up the cluster. Each storage is also and this name is used in several areas of the configuration. In this guide, the name of the storage will be `default`. Also, this guide is geared towards new installs, if upgrading an existing environment to use Gitaly Cluster, you may need to use a different name. -Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more info. +Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more information. The following IPs will be used as an example: @@ -1858,7 +1617,7 @@ To configure the Sidekiq nodes, on each one: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page. -1. Open `/etc/gitlab/gitlab.rb` with your editor: +1. Create or edit `/etc/gitlab/gitlab.rb` and use the following configuration: ```ruby # Avoid running unnecessary services on the Sidekiq server @@ -1873,36 +1632,40 @@ To configure the Sidekiq nodes, on each one: gitlab_exporter['enable'] = false nginx['enable'] = false + # External URL + ## This should match the URL of the external load balancer + external_url 'https://gitlab.example.com' + # Redis ## Redis connection details ## First cluster that will host the cache gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ - {host: '10.6.0.71', port: 26379}, - {host: '10.6.0.72', port: 26379}, - {host: '10.6.0.73', port: 26379}, + {host: '10.6.0.51', port: 26379}, + {host: '10.6.0.52', port: 26379}, + {host: '10.6.0.53', port: 26379}, ] - ## Second cluster that will host the queues, shared state, and actioncable + ## Second cluster that will host the persistent queues, shared state, and actioncable gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_actioncable_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] # Gitaly @@ -2063,30 +1826,30 @@ On each node perform the following: gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ - {host: '10.6.0.71', port: 26379}, - {host: '10.6.0.72', port: 26379}, - {host: '10.6.0.73', port: 26379}, + {host: '10.6.0.51', port: 26379}, + {host: '10.6.0.52', port: 26379}, + {host: '10.6.0.53', port: 26379}, ] - ## Second cluster that will host the queues, shared state, and actionable + ## Second cluster that will host the persistent queues, shared state, and actionable gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_actioncable_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] # Set the network addresses that the exporters used for monitoring will listen on @@ -2158,7 +1921,7 @@ On each node perform the following: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. If you're [using NFS](#configure-nfs-optional): +1. If you're [using NFS](#configure-nfs): 1. If necessary, install the NFS client utility packages using the following commands: @@ -2300,7 +2063,7 @@ To configure the Monitoring node: ## Configure the object storage GitLab supports using an object storage service for holding numerous types of data. -It's recommended over [NFS](#configure-nfs-optional) and in general it's better +It's recommended over [NFS](#configure-nfs) and in general it's better in larger setups as object storage is typically much more performant, reliable, and scalable. @@ -2324,6 +2087,9 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. +GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. +In GitLab 13.6 and later, it's recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. + For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based on what features you intend to use: @@ -2370,7 +2136,7 @@ cluster alongside your instance, read how to </a> </div> -## Configure NFS (optional) +## Configure NFS [Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) are recommended over NFS wherever possible for improved performance. If you intend @@ -2384,7 +2150,7 @@ unavailable from GitLab 15.0. No further enhancements are planned for this featu Read: -- The [Gitaly and NFS deprecation notice](../gitaly/index.md#nfs-deprecation-notice). +- [Gitaly and NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation). - About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). ## Cloud Native Hybrid reference architecture with Helm Charts (alternative) @@ -2438,12 +2204,10 @@ services where applicable): | PostgreSQL<sup>1</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Internal load balancing node<sup>3</sup> | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | -| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | -| Redis Sentinel - Queues / Shared State<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | -| Gitaly | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | -| Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | +| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Gitaly<sup>5</sup> | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | +| Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Object storage<sup>4</sup> | n/a | n/a | n/a | @@ -2453,6 +2217,7 @@ services where applicable): 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. <!-- markdownlint-enable MD029 --> NOTE: @@ -2497,11 +2262,6 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 - collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347 - collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347 - - redis_persistent <.[#FF6347]- redis_persistent_sentinel - redis_cache <.[#FF6347]- redis_cache_sentinel } cloud "**Object Storage**" as object_storage #white diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 4dfe628039a..a5526986be1 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -19,6 +19,7 @@ costly-to-operate environment by using the > - **Supported users (approximate):** 5,000 > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) +> - **Cloud Native Hybrid:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) > - **Test requests per second (RPS) rates:** API: 100 RPS, Web: 10 RPS, Git (Pull): 10 RPS, Git (Push): 2 RPS > - **[Latest 5k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/5k)** @@ -30,8 +31,8 @@ costly-to-operate environment by using the | PostgreSQL<sup>1</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Gitaly | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | `D8s v3` | -| Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Gitaly<sup>5</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | `D8s v3` | +| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Sidekiq | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | | GitLab Rails | 3 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | `c5.4xlarge` | `F16s v2`| @@ -45,6 +46,7 @@ costly-to-operate environment by using the 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. <!-- markdownlint-enable MD029 --> NOTE: @@ -1056,6 +1058,10 @@ The following IPs will be used as an example: [Gitaly Cluster](../gitaly/praefect.md) is a GitLab provided and recommended fault tolerant solution for storing Git repositories. In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. +NOTE: +Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). +For implementations with Gitaly Sharded, the same Gitaly specs should be used. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. + The recommended cluster setup includes the following components: - 3 Gitaly nodes: Replicated storage of Git repositories. @@ -1222,7 +1228,7 @@ the details of each Gitaly node that makes up the cluster. Each storage is also and this name is used in several areas of the configuration. In this guide, the name of the storage will be `default`. Also, this guide is geared towards new installs, if upgrading an existing environment to use Gitaly Cluster, you may need to use a different name. -Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more info. +Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more information. The following IPs will be used as an example: @@ -1549,7 +1555,7 @@ To configure the Sidekiq nodes, one each one: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page. -1. Open `/etc/gitlab/gitlab.rb` with your editor: +1. Create or edit `/etc/gitlab/gitlab.rb` and use the following configuration: ```ruby # Avoid running unnecessary services on the Sidekiq server @@ -1564,6 +1570,10 @@ To configure the Sidekiq nodes, one each one: gitlab_exporter['enable'] = false nginx['enable'] = false + # External URL + ## This should match the URL of the external load balancer + external_url 'https://gitlab.example.com' + # Redis ## Must be the same in every sentinel node redis['master_name'] = 'gitlab-redis' @@ -2005,6 +2015,9 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. +GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. +In GitLab 13.6 and later, it's recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. + For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based on what features you intend to use: @@ -2065,7 +2078,7 @@ unavailable from GitLab 15.0. No further enhancements are planned for this featu Read: -- The [Gitaly and NFS deprecation notice](../gitaly/index.md#nfs-deprecation-notice). +- [Gitaly and NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation). - About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). ## Cloud Native Hybrid reference architecture with Helm Charts (alternative) @@ -2120,8 +2133,8 @@ services where applicable): | PostgreSQL<sup>1</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Gitaly | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | -| Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Gitaly<sup>5</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | +| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Object storage<sup>4</sup> | n/a | n/a | n/a | @@ -2131,6 +2144,7 @@ services where applicable): 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. <!-- markdownlint-enable MD029 --> NOTE: diff --git a/doc/administration/reply_by_email.md b/doc/administration/reply_by_email.md index c249f48b768..055f40ead64 100644 --- a/doc/administration/reply_by_email.md +++ b/doc/administration/reply_by_email.md @@ -45,6 +45,8 @@ following headers, in this order: 1. `To` header 1. `References` header +1. `Delivered-To` header +1. `Envelope-To` header If it finds a reply key, it leaves your reply as a comment on the entity the notification was about (issue, merge request, commit...). diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md index e7edfb9f338..ed50d0e7263 100644 --- a/doc/administration/repository_storage_paths.md +++ b/doc/administration/repository_storage_paths.md @@ -11,9 +11,10 @@ GitLab stores [repositories](../user/project/repository/index.md) on repository storage is either: - A `gitaly_address`, which points to a [Gitaly node](gitaly/index.md). -- A `path`, which points directly to the directory where the repositories are stored. This method is - deprecated and [scheduled to be removed](https://gitlab.com/gitlab-org/gitaly/-/issues/1690) in - GitLab 14.0. +- A `path`, which points directly to the directory where the repositories are stored. GitLab + directly accessing a directory containing repositories + [is deprecated](https://gitlab.com/gitlab-org/gitaly/-/issues/1690). + GitLab should be configured to access GitLab repositories though a `gitaly_address`. GitLab allows you to define multiple repository storages to distribute the storage load between several mount points. For example: @@ -173,4 +174,4 @@ information. ## Move repositories To move a repository to a different repository storage (for example, from `default` to `storage2`), use the -same process as [migrating to Gitaly Cluster](gitaly/index.md#migrate-to-gitaly-cluster). +same process as [migrating to Gitaly Cluster](gitaly/index.md#migrating-to-gitaly-cluster). diff --git a/doc/administration/sidekiq.md b/doc/administration/sidekiq.md index e753832f2c3..4aee88ed9cb 100644 --- a/doc/administration/sidekiq.md +++ b/doc/administration/sidekiq.md @@ -104,6 +104,16 @@ you want using steps 1 and 2 from the GitLab downloads page. You must also copy the `registry.key` file to each Sidekiq node. +1. Define the `external_url`. To maintain uniformity of links across nodes, the + `external_url` on the Sidekiq server should point to the external URL that users + will use to access GitLab. This will either be the `external_url` set on your + application server or the URL of a external load balancer which will route traffic + to the GitLab application server: + + ```ruby + external_url 'https://gitlab.example.com' + ``` + 1. Run `gitlab-ctl reconfigure`. You will need to restart the Sidekiq nodes after an update has occurred and database @@ -194,6 +204,9 @@ gitlab_rails['monitoring_whitelist'] = ['10.10.1.42', '127.0.0.1'] # Container Registry URL for cleanup jobs registry_external_url 'https://registry.example.com' gitlab_rails['registry_api_url'] = "https://registry.example.com" + +# External URL (this should match the URL used to access your GitLab instance) +external_url 'https://gitlab.example.com' ``` ## Further reading diff --git a/doc/administration/smime_signing_email.md b/doc/administration/smime_signing_email.md index ebc1723076b..8fdd87a5b2d 100644 --- a/doc/administration/smime_signing_email.md +++ b/doc/administration/smime_signing_email.md @@ -69,16 +69,16 @@ The key needs to be readable by the GitLab system user (`git` by default). The key needs to be readable by the GitLab system user (`git` by default). -### How to convert S/MIME PKCS#12 / PFX format to PEM encoding +### How to convert S/MIME PKCS #12 format to PEM encoding -Typically S/MIME certificates are handled in binary PKCS#12 format (`.pfx` or `.p12` -extensions), which contain the following in a single encrypted file: +Typically S/MIME certificates are handled in binary Public Key Cryptography Standards (PKCS) #12 format +(`.pfx` or `.p12` extensions), which contain the following in a single encrypted file: - Public certificate - Intermediate certificates (if any) - Private key -To export the required files in PEM encoding from the PKCS#12 file, the +To export the required files in PEM encoding from the PKCS #12 file, the `openssl` command can be used: ```shell diff --git a/doc/administration/timezone.md b/doc/administration/timezone.md index b6076c8ed26..f4339263d34 100644 --- a/doc/administration/timezone.md +++ b/doc/administration/timezone.md @@ -14,22 +14,22 @@ The global time zone configuration parameter can be changed in `config/gitlab.ym Uncomment and customize if you want to change the default time zone of the GitLab application. -## Viewing available timezones +## Viewing available time zones To see all available time zones, run `bundle exec rake time:zones:all`. For Omnibus installations, run `gitlab-rake time:zones:all`. NOTE: -This Rake task does not list timezones in TZInfo format required by Omnibus GitLab during a reconfigure: [#27209](https://gitlab.com/gitlab-org/gitlab/-/issues/27209). +This Rake task does not list time zones in TZInfo format required by Omnibus GitLab during a reconfigure: [#27209](https://gitlab.com/gitlab-org/gitlab/-/issues/27209). ## Changing time zone in Omnibus installations -GitLab defaults its time zone to UTC. It has a global timezone configuration parameter in `/etc/gitlab/gitlab.rb`. +GitLab defaults its time zone to UTC. It has a global time zone configuration parameter in `/etc/gitlab/gitlab.rb`. -To obtain a list of timezones, log in to your GitLab application server and run a command that generates a list of timezones in TZInfo format for the server. For example, install `timedatectl` and run `timedatectl list-timezones`. +To obtain a list of time zones, log in to your GitLab application server and run a command that generates a list of time zones in TZInfo format for the server. For example, install `timedatectl` and run `timedatectl list-timezones`. -To update, add the timezone that best applies to your location. For example: +To update, add the time zone that best applies to your location. For example: ```ruby gitlab_rails['time_zone'] = 'America/New_York' @@ -48,8 +48,12 @@ gitlab-ctl restart > - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/29669) in GitLab 13.9. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/29669) in GitLab 14.1. -A user can set their time zone in their profile. If a user has not set their time zone, it defaults -to the time zone [configured at the instance level](#changing-your-time-zone). On GitLab.com, the -default time zone is UTC. +Users can set their time zone in their profile. On GitLab.com, the default time zone is UTC. + +New users do not have a default time zone in [GitLab 14.4 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/340795). New users must +explicitly set their time zone before it displays on their profile. + +In GitLab 14.3 and earlier, users with no configured time zone default to the time zone +[configured at the instance level](#changing-your-time-zone). For more information, see [Set your time zone](../user/profile/index.md#set-your-time-zone). diff --git a/doc/administration/troubleshooting/debug.md b/doc/administration/troubleshooting/debug.md index aea891b8a77..e00243aca0a 100644 --- a/doc/administration/troubleshooting/debug.md +++ b/doc/administration/troubleshooting/debug.md @@ -225,7 +225,7 @@ gitlab_rails['env'] = { ``` For source installations, set the environment variable. -Refer to [Puma Worker timeout](https://docs.gitlab.com/omnibus/settings/puma.html#worker-timeout). +Refer to [Puma Worker timeout](../operations/puma.md#worker-timeout). [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 491db37d9da..109f451be5a 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -26,7 +26,7 @@ As GitLab changes, changes to the code are inevitable, and so some scripts may not work as they once used to. These are not kept up-to-date as these scripts/commands were added as they were found/needed. As mentioned above, we recommend running these scripts under the supervision of a -Support Engineer, who can also verify that they will continue to work as they +Support Engineer, who can also verify that they continue to work as they should and, if needed, update the script for the latest version of GitLab. ## Find specific methods for an object @@ -38,8 +38,6 @@ Array.methods.grep(/sing/) ## Find method source -Works for [non-instrumented methods](../../development/instrumentation.md#checking-instrumented-methods): - ```ruby instance_of_object.method(:foo).source_location @@ -187,7 +185,7 @@ Project.update_all(visibility_level: 0) ```ruby # -# This section will list all the projects which are pending deletion +# This section lists all the projects which are pending deletion # projects = Project.where(pending_delete: true) projects.each do |p| @@ -197,7 +195,7 @@ projects.each do |p| end # -# Assign a user (the root user will do) +# Assign a user (the root user does) # user = User.find_by_username('root') @@ -257,7 +255,7 @@ namespace = Namespace.find_by_full_path("<new_namespace>") ### For Removing webhooks that is getting timeout due to large webhook logs ```ruby -# ID will be the webhook_id +# ID is the webhook_id hook=WebHook.find(ID) WebHooks::DestroyService.new(current_user).execute(hook) @@ -399,7 +397,7 @@ projects = Project.find_by_sql("SELECT * FROM projects WHERE name LIKE '%ject'") ### Recreate WARNING: -This is a destructive operation, the Wiki will be empty. +This is a destructive operation, the Wiki becomes empty. A Projects Wiki can be recreated by this command: @@ -476,13 +474,13 @@ Projects::ImportExport::ExportService.new(project, user).execute If the project you wish to export is available at `https://gitlab.example.com/baltig/pipeline-templates`, the value to use for `PROJECT_PATH` would be `baltig/pipeline-templates`. -If this all runs successfully, you will see output like the following before being returned to the Rails console prompt: +If this all runs successfully, you see an output like the following before being returned to the Rails console prompt: ```ruby => nil ``` -The exported project will be located within a `.tar.gz` file in `/var/opt/gitlab/gitlab-rails/uploads/-/system/import_export_upload/export_file/`. +The exported project is located within a `.tar.gz` file in `/var/opt/gitlab/gitlab-rails/uploads/-/system/import_export_upload/export_file/`. ## Repository @@ -490,27 +488,29 @@ The exported project will be located within a `.tar.gz` file in `/var/opt/gitlab If it seems that a commit has gone "missing", search the sequence of pushes to a repository. [This StackOverflow article](https://stackoverflow.com/questions/13468027/the-mystery-of-the-missing-commit-across-merges) -describes how you can end up in this state without a force push. +describes how you can end up in this state without a force push. Another cause can be a misconfigured [server hook](../server_hooks.md) that changes a HEAD ref via a `git reset` operation. + +If you look at the output from the sample code below for the target branch, you +see a discontinuity in the from/to commits as you step through the output. The `commit_from` of each new push should equal the `commit_to` of the previous push. A break in that sequence indicates one or more commits have been "lost" from the repository history. -If you look at the output from the sample code below for the target branch, you will -see a discontinuity in the from/to commits as you step through the output. Each new -push should be "from" the "to" SHA of the previous push. When this discontinuity happens, -you will see two pushes with the same "from" SHA: +The following example checks the last 100 pushes and prints the `commit_from` and `commit_to` entries: ```ruby -p = Project.find_with_namespace('u/p') +p = Project.find_by_full_path('u/p') p.events.pushed_action.last(100).each do |e| - printf "%-20.20s %8s...%8s (%s)\n", e.data[:ref], e.data[:before], e.data[:after], e.author.try(:username) + printf "%-20.20s %8s...%8s (%s) +", e.push_event_payload[:ref], e.push_event_payload[:commit_from], e.push_event_payload[:commit_to], e.author.try(:username) end ``` -GitLab 9.5 and above: +Example output showing break in sequence at line 4: -```ruby -p = Project.find_by_full_path('u/p') -p.events.pushed_action.last(100).each do |e| - printf "%-20.20s %8s...%8s (%s)\n", e.push_event_payload[:ref], e.push_event_payload[:commit_from], e.push_event_payload[:commit_to], e.author.try(:username) -end +```plaintext +master f21b07713251e04575908149bdc8ac1f105aabc3...6bc56c1f46244792222f6c85b11606933af171de (root) +master 6bc56c1f46244792222f6c85b11606933af171de...132da6064f5d3453d445fd7cb452b148705bdc1b (root) +master 132da6064f5d3453d445fd7cb452b148705bdc1b...a62e1e693150a2e46ace0ce696cd4a52856dfa65 (root) +master 58b07b719a4b0039fec810efa52f479ba1b84756...f05321a5b5728bd8a89b7bf530aa44043c951dce (root) +master f05321a5b5728bd8a89b7bf530aa44043c951dce...7d02e575fd790e76a3284ee435368279a5eb3773 (root) ``` ## Mirrors @@ -558,7 +558,7 @@ end ```ruby u = User.new(username: 'test_user', email: 'test@example.com', name: 'Test User', password: 'password', password_confirmation: 'password') -u.skip_confirmation! # Use it only if you wish user to be automatically confirmed. If skipped, user will recieve confirmation e-mail +u.skip_confirmation! # Use it only if you wish user to be automatically confirmed. If skipped, user receives confirmation e-mail u.save! ``` @@ -612,7 +612,7 @@ identifier = Analytics::UsageTrends::Measurement.identifiers[:billable_users] ```ruby users = User.where('id NOT IN (select distinct(user_id) from project_authorizations)') -# How many users will be removed? +# How many users are removed? users.count # If that count looks sane: @@ -726,6 +726,18 @@ group.require_two_factor_authentication=false group.save ``` +## Authentication + +### Re-enable standard web sign-in form + +Re-enable the standard username and password-based sign-in form if it was disabled as a [Sign-in restriction](../../user/admin_area/settings/sign_in_restrictions.md#password-authentication-enabled). + +You can use this method when a configured external authentication provider (through SSO or an LDAP configuration) is facing an outage and direct sign-in access to GitLab is required. + +```ruby +Gitlab::CurrentSettings.update!(password_authentication_enabled_for_web: true) +``` + ## SCIM ### Fixing bad SCIM identities @@ -1061,7 +1073,7 @@ encrypted credentials to allow manual reentry: If `User OTP Secret Bad count:` is detected. For each user listed disable/enable two-factor authentication. -The following script will search in some of the tables for encrypted tokens that are +The following script searches in some of the tables for encrypted tokens that are causing decryption errors, and update or reset as needed: ```shell @@ -1123,7 +1135,7 @@ Geo::ProjectRegistry.sync_failed('repository') ### Resync repositories -#### Queue up all repositories for resync. Sidekiq will handle each sync +#### Queue up all repositories for resync. Sidekiq handles each sync ```ruby Geo::ProjectRegistry.update_all(resync_repository: true, resync_wiki: true) @@ -1170,10 +1182,10 @@ registry.replicator.send(:download) #### Verify package files on the secondary manually -This will iterate over all package files on the secondary, looking at the +This iterates over all package files on the secondary, looking at the `verification_checksum` stored in the database (which came from the primary) and then calculate this value on the secondary to check if they match. This -won't change anything in the UI: +does not change anything in the UI: ```ruby # Run on secondary @@ -1235,7 +1247,7 @@ Gitlab::UsageData.to_json ### Generate a fresh new Service Ping -This will also refresh the cached Service Ping displayed in the admin area +This also refreshes the cached Service Ping displayed in the Admin Area ```ruby Gitlab::UsageData.to_json(force_refresh: true) @@ -1269,7 +1281,7 @@ cluster = Clusters::Cluster.find_by(name: 'cluster_name') Delete cluster without associated resources: ```ruby -# Find an admin user +# Find users with the Administrator role user = User.find_by(username: 'admin_user') # Find the cluster with the ID @@ -1289,7 +1301,7 @@ Open the rails console (`gitlab rails c`) and run the following command to see a ApplicationSetting.last.attributes ``` -Among other attributes, in the output you will notice that all the settings available in the [Elasticsearch Integration page](../../integration/elasticsearch.md), like: `elasticsearch_indexing`, `elasticsearch_url`, `elasticsearch_replicas`, `elasticsearch_pause_indexing`, and so on. +Among other attributes, the output contains all the settings available in the [Elasticsearch Integration page](../../integration/elasticsearch.md), such as `elasticsearch_indexing`, `elasticsearch_url`, `elasticsearch_replicas`, and `elasticsearch_pause_indexing`. #### Setting attributes diff --git a/doc/administration/troubleshooting/img/database-query-dialog_v14_3.png b/doc/administration/troubleshooting/img/database-query-dialog_v14_3.png Binary files differnew file mode 100644 index 00000000000..197cfa17da2 --- /dev/null +++ b/doc/administration/troubleshooting/img/database-query-dialog_v14_3.png diff --git a/doc/administration/troubleshooting/img/obtaining-a-session-cookie-for-request_v14_3.png b/doc/administration/troubleshooting/img/obtaining-a-session-cookie-for-request_v14_3.png Binary files differnew file mode 100644 index 00000000000..a63ebf9ecf2 --- /dev/null +++ b/doc/administration/troubleshooting/img/obtaining-a-session-cookie-for-request_v14_3.png diff --git a/doc/administration/troubleshooting/img/paste-request-id-into-progress-bar_v14_3.png b/doc/administration/troubleshooting/img/paste-request-id-into-progress-bar_v14_3.png Binary files differnew file mode 100644 index 00000000000..a19585d7a89 --- /dev/null +++ b/doc/administration/troubleshooting/img/paste-request-id-into-progress-bar_v14_3.png diff --git a/doc/administration/troubleshooting/img/select-request-id-from-request-selector-drop-down-menu_v14_3.png b/doc/administration/troubleshooting/img/select-request-id-from-request-selector-drop-down-menu_v14_3.png Binary files differnew file mode 100644 index 00000000000..b8314056c9b --- /dev/null +++ b/doc/administration/troubleshooting/img/select-request-id-from-request-selector-drop-down-menu_v14_3.png diff --git a/doc/administration/troubleshooting/img/view-pg-details_v14_3.png b/doc/administration/troubleshooting/img/view-pg-details_v14_3.png Binary files differnew file mode 100644 index 00000000000..1fe12462e4e --- /dev/null +++ b/doc/administration/troubleshooting/img/view-pg-details_v14_3.png diff --git a/doc/administration/troubleshooting/linux_cheat_sheet.md b/doc/administration/troubleshooting/linux_cheat_sheet.md index 9eadbad171e..b66e88a8d60 100644 --- a/doc/administration/troubleshooting/linux_cheat_sheet.md +++ b/doc/administration/troubleshooting/linux_cheat_sheet.md @@ -14,7 +14,7 @@ having an issue with GitLab, you may want to check your [support options](https: first, before attempting to use this information. WARNING: -If you are administering GitLab you are expected to know these commands for your distribution +If you administer GitLab you are expected to know these commands for your distribution of choice. If you are a GitLab Support Engineer, consider this a cross-reference to translate `yum` -> `apt-get` and the like. diff --git a/doc/administration/troubleshooting/ssl.md b/doc/administration/troubleshooting/ssl.md index 80be30a6509..01a4c4af65b 100644 --- a/doc/administration/troubleshooting/ssl.md +++ b/doc/administration/troubleshooting/ssl.md @@ -45,7 +45,7 @@ following issues: ``` - The error `SSL certificate problem: unable to get local issuer certificate` - is displayed when setting up a [mirror](../../user/project/repository/repository_mirroring.md#repository-mirroring) + is displayed when setting up a [mirror](../../user/project/repository/mirror/index.md) from this GitLab instance. - `openssl` works when specifying the path to the certificate: @@ -115,7 +115,7 @@ and the restart the runner by running `gitlab-runner restart`. ## Mirroring a remote GitLab repository that uses a self-signed SSL certificate -When configuring a local GitLab instance to [mirror a repository](../../user/project/repository/repository_mirroring.md) +When configuring a local GitLab instance to [mirror a repository](../../user/project/repository/mirror/index.md) from a remote GitLab instance that uses a self-signed certificate, you may see the `SSL certificate problem: self signed certificate` error message in the user interface. diff --git a/doc/administration/troubleshooting/tracing_correlation_id.md b/doc/administration/troubleshooting/tracing_correlation_id.md index 3518f52e6f6..3bafbed4b3f 100644 --- a/doc/administration/troubleshooting/tracing_correlation_id.md +++ b/doc/administration/troubleshooting/tracing_correlation_id.md @@ -135,3 +135,69 @@ You can use the [performance bar](../monitoring/performance/performance_bar.md) To view the data, the correlation ID of the request must match the same session as the user viewing the performance bar. For API requests, this means that you must perform the request using the session cookie of the signed-in user. + +For example, if you want to view the database queries executed for the following API endpoint: + +```shell +https://gitlab.com/api/v4/groups/2564205/projects?with_security_reports=true&page=1&per_page=1 +``` + +First, enable the **Developer Tools** panel. See [Getting the correlation ID in your browser](#getting-the-correlation-id-in-your-browser) for details on how to do this. + +After developer tools have been enabled, obtain a session cookie as follows: + +1. Visit <https://gitlab.com> while logged in. +1. (Optional) Select **Fetch/XHR** request filter in the **Developer Tools** panel. This step is described for Google Chrome developer tools and is not strictly necessary, it just makes it easier to find the correct request. +1. Select the `results?request_id=<some-request-id>` request on the left hand side. +1. The session cookie is displayed under the `Request Headers` section of the `Headers` panel. Right-click on the cookie value and select `Copy value`. + + + +You have the value of the session cookie copied to your clipboard, for example: + +```shell +experimentation_subject_id=<subject-id>; _gitlab_session=<session-id>; event_filter=all; visitor_id=<visitor-id>; perf_bar_enabled=true; sidebar_collapsed=true; diff_view=inline; sast_entry_point_dismissed=true; auto_devops_settings_dismissed=true; cf_clearance=<cf-clearance>; collapsed_gutter=false; frequently_used_emojis=clap,thumbsup,rofl,tada,eyes,bow +``` + +Use the value of the session cookie to craft an API request by pasting it into a custom header of a `curl` request: + +```shell +$ curl --include "https://gitlab.com/api/v4/groups/2564205/projects?with_security_reports=true&page=1&per_page=1" \ +--header 'cookie: experimentation_subject_id=<subject-id>; _gitlab_session=<session-id>; event_filter=all; visitor_id=<visitor-id>; perf_bar_enabled=true; sidebar_collapsed=true; diff_view=inline; sast_entry_point_dismissed=true; auto_devops_settings_dismissed=true; cf_clearance=<cf-clearance>; collapsed_gutter=false; frequently_used_emojis=clap,thumbsup,rofl,tada,eyes,bow' + + date: Tue, 28 Sep 2021 03:55:33 GMT + content-type: application/json + ... + x-request-id: 01FGN8P881GF2E5J91JYA338Y3 + ... + [ + { + "id":27497069, + "description":"Analyzer for images used on live K8S containers based on Starboard" + }, + "container_registry_image_prefix":"registry.gitlab.com/gitlab-org/security-products/analyzers/cluster-image-scanning", + "..." + ] +``` + +The response contains the data from the API endpoint, and a `correlation_id` value, returned in the `x-request-id` header, as described in the [Identify the correlation ID for a request](#identify-the-correlation-id-for-a-request) section. + +You can then view the database details for this request: + +1. Paste the `x-request-id` value into the `request details` field of the [performance bar](../monitoring/performance/performance_bar.md) and press <kbd>Enter/Return</kbd>. This example uses the `x-request-id` value `01FGN8P881GF2E5J91JYA338Y3`, returned by the above response: + +  + +1. A new request is inserted into the `Request Selector` dropdown on the right-hand side of the Performance Bar. Select the new request to view the metrics of the API request: + +  + + <!-- vale gitlab.Substitutions = NO --> +1. Select the `pg` link in the Progress Bar to view the database queries executed by the API request: + +  + <!-- vale gitlab.Substitutions = YES --> + + The database query dialog is displayed: + +  diff --git a/doc/administration/user_settings.md b/doc/administration/user_settings.md index 681ce87edb6..891c11afaf4 100644 --- a/doc/administration/user_settings.md +++ b/doc/administration/user_settings.md @@ -4,40 +4,56 @@ group: Access info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Modifying global user settings +# Modify global user settings **(FREE SELF)** GitLab administrators can modify user settings for the entire GitLab instance. -## Disallow users creating top-level groups +## Prevent users from creating top-level groups -By default, new users can create top-level groups. To disable this, modify the appropriate configuration file, -and then [reconfigure and restart GitLab](restart_gitlab.md). +By default, new users can create top-level groups. To disable your users' +ability to create top-level groups: -For Omnibus installations, add the following to `/etc/gitlab/gitlab.rb`: +**Omnibus GitLab installations** -```ruby -gitlab_rails['gitlab_default_can_create_group'] = false -``` +1. Edit `/etc/gitlab/gitlab.rb` and add the following line: -For source installations, uncomment the following line in `config/gitlab.yml`: + ```ruby + gitlab_rails['gitlab_default_can_create_group'] = false + ``` -```yaml -# default_can_create_group: false # default: true -``` +1. [Reconfigure and restart GitLab](restart_gitlab.md#omnibus-installations). -## Disallow users changing usernames +**Source installations** -By default, new users can change their usernames. To disable this, modify the appropriate configuration file, -and then [reconfigure and restart GitLab](restart_gitlab.md). +1. Edit `config/gitlab.yml` and uncomment the following line: -For Omnibus installations, add the following to `/etc/gitlab/gitlab.rb`: + ```yaml + # default_can_create_group: false # default: true + ``` -```ruby -gitlab_rails['gitlab_username_changing_enabled'] = false -``` +1. [Restart GitLab](restart_gitlab.md#installations-from-source). -For source installations, uncomment the following line in `config/gitlab.yml`: +## Prevent users from changing their usernames -```yaml -# username_changing_enabled: false # default: true - User can change their username/namespace -``` +By default, new users can change their usernames. To disable your users' +ability to change their usernames: + +**Omnibus GitLab installations** + +1. Edit `/etc/gitlab/gitlab.rb` and add the following line: + + ```ruby + gitlab_rails['gitlab_username_changing_enabled'] = false + ``` + +1. [Reconfigure and restart GitLab](restart_gitlab.md#omnibus-installations). + +**Source installations** + +1. Edit `config/gitlab.yml` and uncomment the following line: + + ```yaml + # username_changing_enabled: false # default: true - User can change their username/namespace + ``` + +1. [Restart GitLab](restart_gitlab.md#installations-from-source). |
