diff options
Diffstat (limited to 'doc/administration/reference_architectures')
| -rw-r--r-- | doc/administration/reference_architectures/10k_users.md | 288 | ||||
| -rw-r--r-- | doc/administration/reference_architectures/1k_users.md | 4 | ||||
| -rw-r--r-- | doc/administration/reference_architectures/25k_users.md | 290 | ||||
| -rw-r--r-- | doc/administration/reference_architectures/2k_users.md | 117 | ||||
| -rw-r--r-- | doc/administration/reference_architectures/3k_users.md | 252 | ||||
| -rw-r--r-- | doc/administration/reference_architectures/50k_users.md | 290 | ||||
| -rw-r--r-- | doc/administration/reference_architectures/5k_users.md | 248 | ||||
| -rw-r--r-- | doc/administration/reference_architectures/index.md | 242 |
8 files changed, 989 insertions, 742 deletions
diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 529621813aa..ba4c5fd9046 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -31,8 +31,8 @@ full list of reference architectures, see | Gitaly<sup>5 6</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | | Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | -| GitLab Rails | 3 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | +| Sidekiq<sup>7</sup> | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| GitLab Rails<sup>7</sup> | 3 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | | Object storage<sup>4</sup> | - | - | - | - | @@ -53,6 +53,8 @@ full list of reference architectures, see 6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to [Large repositories](index.md#large-repositories) for more information. +7. Can be placed in Auto Scaling Groups (ASGs) as the component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes). + However, for GitLab Rails certain processes like [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md) should be run on only one node. <!-- markdownlint-enable MD029 --> NOTE: @@ -170,7 +172,7 @@ To set up GitLab and its components to accommodate up to 10,000 users: environment. 1. [Configure the object storage](#configure-the-object-storage) used for shared data objects. -1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, +1. [Configure advanced search](#configure-advanced-search) (optional) for faster, more advanced code search across your entire GitLab instance. The servers start on the same 10.6.0.0/24 private network range, and can @@ -309,7 +311,7 @@ Configure your load balancer to pass connections on port 443 as `TCP` rather than `HTTP(S)` protocol. This will pass the connection to the application node's NGINX service untouched. NGINX will have the SSL certificate and listen on port 443. -See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) +See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html) for details on managing SSL certificates and configuring NGINX. #### Load balancer terminates SSL without backend SSL @@ -320,7 +322,7 @@ terminating SSL. Since communication between the load balancer and GitLab will not be secure, there is some additional configuration needed. See the -[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination) +[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination) for details. #### Load balancer terminates SSL with backend SSL @@ -333,7 +335,7 @@ Traffic will also be secure between the load balancers and NGINX in this scenario. There is no need to add configuration for proxied SSL since the connection will be secure all the way. However, configuration will need to be added to GitLab to configure SSL certificates. See -the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) +the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html) for details on managing SSL certificates and configuring NGINX. <div align="right"> @@ -417,6 +419,11 @@ spread connections equally in practice. ## Configure Consul +Next, we set up the Consul servers. + +NOTE: +Consul must be deployed in an odd number of 3 nodes or more. This is to ensure the nodes can take votes as part of a quorum. + The following IPs will be used as an example: - `10.6.0.11`: Consul 1 @@ -801,6 +808,9 @@ Using [Redis](https://redis.io/) in scalable environment is possible using a **P topology with a [Redis Sentinel](https://redis.io/docs/manual/sentinel/) service to watch and automatically start the failover procedure. +NOTE: +Redis clusters must each be deployed in an odd number of 3 nodes or more. This is to ensure Redis Sentinel can take votes as part of a quorum. This does not apply when configuring Redis externally, such as a cloud provider service. + Redis requires authentication if used with Sentinel. See [Redis Security](https://redis.io/docs/manual/security/) documentation for more information. We recommend using a combination of a Redis password and tight @@ -834,9 +844,9 @@ to be used with GitLab. The following IPs will be used as an example: Managed Redis from cloud providers (such as AWS ElastiCache) will work. If these services support high availability, be sure it _isn't_ of the Redis Cluster type. -Redis version 5.0 or higher is required, which is included with Omnibus GitLab -packages starting with GitLab 13.0. Older Redis versions don't support an -optional count argument to SPOP, which is required for [Merge Trains](../../ci/pipelines/merge_trains.md). + +Because Omnibus GitLab packages ship with Redis 6.0 or later, Redis 6.0 or later is required. Older Redis versions have reached end-of-life. + Note the Redis node's IP address or hostname, port, and password (if required). These will be necessary later when configuring the [GitLab application servers](#configure-gitlab-rails). @@ -1189,7 +1199,7 @@ For more advanced setups refer to the [standalone Gitaly Cluster documentation]( Praefect, the routing and transaction manager for Gitaly Cluster, requires its own database server to store data on Gitaly Cluster status. If you want to have a highly available setup, Praefect requires a third-party PostgreSQL database. -A built-in solution is being [worked on](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919). +A built-in solution is being [worked on](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7292). #### Praefect non-HA PostgreSQL standalone using Omnibus GitLab @@ -1336,6 +1346,9 @@ This is how this would work with a Omnibus GitLab PostgreSQL setup: Praefect is the router and transaction manager for Gitaly Cluster and all connections to Gitaly go through it. This section details how to configure it. +NOTE: +Consul must be deployed in an odd number of 3 nodes or more. This is to ensure the nodes can take votes as part of a quorum. + Praefect requires several secret tokens to secure communications across the Cluster: - `<praefect_external_token>`: Used for repositories hosted on your Gitaly cluster and can only be accessed by Gitaly clients that carry this token. @@ -1389,7 +1402,6 @@ Updates to example must be made at: # Praefect Configuration praefect['enable'] = true - praefect['listen_addr'] = '0.0.0.0:2305' # Prevent database migrations from running on upgrade automatically praefect['auto_migrate'] = false @@ -1404,45 +1416,63 @@ Updates to example must be made at: # Please set the real values as explained in Required Information section # - # Praefect External Token - # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster - praefect['auth_token'] = '<praefect_external_token>' - - # Praefect Database Settings - praefect['database_host'] = '10.6.0.141' - praefect['database_port'] = 5432 - # `no_proxy` settings must always be a direct connection for caching - praefect['database_direct_host'] = '10.6.0.141' - praefect['database_direct_port'] = 5432 - praefect['database_dbname'] = 'praefect_production' - praefect['database_user'] = 'praefect' - praefect['database_password'] = '<praefect_postgresql_password>' - - # Praefect Virtual Storage config - # Name of storage hash must match storage name in git_data_dirs on GitLab - # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1') - praefect['virtual_storages'] = { - 'default' => { - 'nodes' => { - 'gitaly-1' => { - 'address' => 'tcp://10.6.0.91:8075', - 'token' => '<praefect_internal_token>' - }, - 'gitaly-2' => { - 'address' => 'tcp://10.6.0.92:8075', - 'token' => '<praefect_internal_token>' - }, - 'gitaly-3' => { - 'address' => 'tcp://10.6.0.93:8075', - 'token' => '<praefect_internal_token>' + praefect['configuration'] = { + # ... + listen_addr: '0.0.0.0:2305', + auth: { + # ... + # + # Praefect External Token + # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster + token: '<praefect_external_token>', + }, + # Praefect Database Settings + database: { + # ... + host: '10.6.0.141', + port: 5432, + # `no_proxy` settings must always be a direct connection for caching + session_pooled: { + # ... + host: '10.6.0.141', + port: 5432, + dbname: 'praefect_production', + user: 'praefect', + password: '<praefect_postgresql_password>', + }, + }, + # Praefect Virtual Storage config + # Name of storage hash must match storage name in git_data_dirs on GitLab + # server ('praefect') and in gitaly['configuration'][:storage] on Gitaly nodes ('gitaly-1') + virtual_storage: [ + { + # ... + name: 'default', + node: [ + { + storage: 'gitaly-1', + address: 'tcp://10.6.0.91:8075', + token: '<praefect_internal_token>' + }, + { + storage: 'gitaly-2', + address: 'tcp://10.6.0.92:8075', + token: '<praefect_internal_token>' + }, + { + storage: 'gitaly-3', + address: 'tcp://10.6.0.93:8075', + token: '<praefect_internal_token>' + }, + ], }, - } - } + ], + # Set the network address Praefect will listen on for monitoring + prometheus_listen_addr: '0.0.0.0:9652', } - # Set the network addresses that the exporters will listen on for monitoring + # Set the network address the node exporter will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' - praefect['prometheus_listen_addr'] = '0.0.0.0:9652' ## The IPs of the Consul server nodes ## You can also use FQDNs and intermix them with IPs @@ -1493,9 +1523,7 @@ Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs should have a throughput of at least 8,000 input/output operations per second (IOPS) for read operations and 2,000 IOPS for -write operations. These IOPS values are initial recommendations, and may be -adjusted to greater or lesser values depending on the scale of your -environment's workload. If you're running the environment on a Cloud provider, +write operations. If you're running the environment on a Cloud provider, refer to their documentation about how to configure IOPS correctly. Gitaly servers must not be exposed to the public internet, as Gitaly's network @@ -1505,7 +1533,7 @@ to restrict access to the Gitaly server. Another option is to For configuring Gitaly you should note the following: -- `git_data_dirs` should be configured to reflect the storage path for the specific Gitaly node +- `gitaly['configuration'][:storage]` should be configured to reflect the storage path for the specific Gitaly node - `auth_token` should be the same as `praefect_internal_token` The following IPs will be used as an example: @@ -1554,20 +1582,6 @@ Updates to example must be made at: # Gitaly gitaly['enable'] = true - # Make Gitaly accept connections on all network interfaces. You must use - # firewalls to restrict access to this address/port. - # Comment out following line if you only want to support TLS connections - gitaly['listen_addr'] = "0.0.0.0:8075" - - # Gitaly Auth Token - # Should be the same as praefect_internal_token - gitaly['auth_token'] = '<praefect_internal_token>' - - # Gitaly Pack-objects cache - # Recommended to be enabled for improved performance but can notably increase disk I/O - # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info - gitaly['pack_objects_cache_enabled'] = true - # Configure the Consul agent consul['enable'] = true ## Enable service discovery for Prometheus @@ -1582,9 +1596,29 @@ Updates to example must be made at: retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), } - # Set the network addresses that the exporters will listen on for monitoring + # Set the network address that the node exporter will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' - gitaly['prometheus_listen_addr'] = '0.0.0.0:9236' + + gitaly['configuration'] = { + # Make Gitaly accept connections on all network interfaces. You must use + # firewalls to restrict access to this address/port. + # Comment out following line if you only want to support TLS connections + listen_addr: '0.0.0.0:8075', + # Set the network address that Gitaly will listen on for monitoring + prometheus_listen_addr: '0.0.0.0:9236', + auth: { + # Gitaly Auth Token + # Should be the same as praefect_internal_token + token: '<praefect_internal_token>', + }, + pack_objects_cache: { + # Gitaly Pack-objects cache + # Recommended to be enabled for improved performance but can notably increase disk I/O + # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info + enabled: true, + }, + } + # # END user configuration ``` @@ -1593,31 +1627,43 @@ Updates to example must be made at: - On Gitaly node 1: ```ruby - git_data_dirs({ - "gitaly-1" => { - "path" => "/var/opt/gitlab/git-data" - } - }) + gitaly['configuration'] = { + # ... + storage: [ + { + name: 'gitaly-1', + path: '/var/opt/gitlab/git-data', + }, + ], + } ``` - On Gitaly node 2: ```ruby - git_data_dirs({ - "gitaly-2" => { - "path" => "/var/opt/gitlab/git-data" - } - }) + gitaly['configuration'] = { + # ... + storage: [ + { + name: 'gitaly-2', + path: '/var/opt/gitlab/git-data', + }, + ], + } ``` - On Gitaly node 3: ```ruby - git_data_dirs({ - "gitaly-3" => { - "path" => "/var/opt/gitlab/git-data" - } - }) + gitaly['configuration'] = { + # ... + storage: [ + { + name: 'gitaly-3', + path: '/var/opt/gitlab/git-data', + }, + ], + } ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace @@ -1637,7 +1683,7 @@ for secure connections, you must: Additionally the certificate, or its certificate authority, must be installed on all Gitaly servers and on all Praefect clients that communicate with it following the procedure described in -[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) (and repeated below). +[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl/index.html#install-custom-public-certificates) (and repeated below). Note the following: @@ -1646,7 +1692,7 @@ Note the following: - You can configure Praefect servers with both an unencrypted listening address `listen_addr` and an encrypted listening address `tls_listen_addr` at the same time. This allows you to do a gradual transition from unencrypted to encrypted traffic, if - necessary. To disable the unencrypted listener, set `praefect['listen_addr'] = nil`. + necessary. To disable the unencrypted listener, set `praefect['configuration'][:listen_addr] = nil`. - The Internal Load Balancer will also access to the certificates and need to be configured to allow for TLS pass-through. Refer to the load balancers documentation on how to configure this. @@ -1668,9 +1714,15 @@ To configure Praefect with TLS: 1. Edit `/etc/gitlab/gitlab.rb` and add: ```ruby - praefect['tls_listen_addr'] = "0.0.0.0:3305" - praefect['certificate_path'] = "/etc/gitlab/ssl/cert.pem" - praefect['key_path'] = "/etc/gitlab/ssl/key.pem" + praefect['configuration'] = { + # ... + tls_listen_addr: '0.0.0.0:3305', + tls: { + # ... + certificate_path: '/etc/gitlab/ssl/cert.pem', + key_path: '/etc/gitlab/ssl/key.pem', + }, + } ``` 1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -1751,7 +1803,7 @@ Updates to example must be made at: # Redis ## Redis connection details - ## First cluster that will host the cache + ## First cluster that will host the cache data gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ @@ -1760,22 +1812,11 @@ Updates to example must be made at: {host: '10.6.0.53', port: 26379}, ] - ## Second cluster that will host the persistent queues, shared state, and actioncable - gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' - gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' - gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' + ## Second cluster that hosts all other persistent data + redis['master_name'] = 'gitlab-redis-persistent' + redis['master_password'] = '<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>' - gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.61', port: 26379}, - {host: '10.6.0.62', port: 26379}, - {host: '10.6.0.63', port: 26379}, - ] - gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.61', port: 26379}, - {host: '10.6.0.62', port: 26379}, - {host: '10.6.0.63', port: 26379}, - ] - gitlab_rails['redis_actioncable_sentinels'] = [ + gitlab_rails['redis_sentinels'] = [ {host: '10.6.0.61', port: 26379}, {host: '10.6.0.62', port: 26379}, {host: '10.6.0.63', port: 26379}, @@ -1936,7 +1977,7 @@ On each node perform the following: gitlab_rails['auto_migrate'] = false ## Redis connection details - ## First cluster that will host the cache + ## First cluster that will host the cache data gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ @@ -1945,22 +1986,11 @@ On each node perform the following: {host: '10.6.0.53', port: 26379}, ] - ## Second cluster that will host the persistent queues, shared state, and actioncable - gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' - gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' - gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' + ## Second cluster that hosts all other persistent data + redis['master_name'] = 'gitlab-redis-persistent' + redis['master_password'] = '<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>' - gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.61', port: 26379}, - {host: '10.6.0.62', port: 26379}, - {host: '10.6.0.63', port: 26379}, - ] - gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.61', port: 26379}, - {host: '10.6.0.62', port: 26379}, - {host: '10.6.0.63', port: 26379}, - ] - gitlab_rails['redis_actioncable_sentinels'] = [ + gitlab_rails['redis_sentinels'] = [ {host: '10.6.0.61', port: 26379}, {host: '10.6.0.62', port: 26379}, {host: '10.6.0.63', port: 26379}, @@ -2058,7 +2088,7 @@ On each node perform the following: When you specify `https` in the `external_url`, as in the previous example, GitLab expects that the SSL certificates are in `/etc/gitlab/ssl/`. If the certificates aren't present, NGINX will fail to start. For more information, see -the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html). +the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html). ### GitLab Rails post-configuration @@ -2070,11 +2100,9 @@ the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html). sudo gitlab-rake gitlab:db:configure ``` - If you encounter a `rake aborted!` error message stating that PgBouncer is - failing to connect to PostgreSQL, it may be that your PgBouncer node's IP - address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` - on your database nodes. Before proceeding, see - [PgBouncer error `ERROR: pgbouncer cannot connect to server`](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). + Note that this requires the Rails node to be configured to connect to the primary database + directly, [bypassing PgBouncer](../postgresql/pgbouncer.md#procedure-for-bypassing-pgbouncer). + After migrations have completed, you must configure the node to pass through PgBouncer again. 1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). @@ -2170,15 +2198,15 @@ and scalable. There are two ways of specifying object storage configuration in GitLab: -- [Consolidated form](../object_storage.md#consolidated-object-storage-configuration): A single credential is +- [Consolidated form](../object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form): A single credential is shared by all supported object types. -- [Storage-specific form](../object_storage.md#storage-specific-configuration): Every object defines its - own object storage [connection and configuration](../object_storage.md#connection-settings). +- [Storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form): Every object defines its + own object storage [connection and configuration](../object_storage.md#configure-the-connection-settings). The consolidated form is used in the following examples when available. NOTE: -When using the [storage-specific form](../object_storage.md#storage-specific-configuration) +When using the [storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form) in GitLab 14.x and earlier, you should enable [direct upload mode](../../development/uploads/index.md#direct-upload). The previous [background upload](../../development/uploads/index.md#direct-upload) mode, which was deprecated in 14.9, requires shared storage such as NFS. @@ -2200,9 +2228,9 @@ GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs. -## Configure Advanced Search +## Configure advanced search -You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md) +You can leverage Elasticsearch and [enable advanced search](../../integration/advanced_search/elasticsearch.md) for faster, more advanced code search across your entire GitLab instance. Elasticsearch cluster design and requirements are dependent on your specific diff --git a/doc/administration/reference_architectures/1k_users.md b/doc/administration/reference_architectures/1k_users.md index d3aa6fef51e..adcf8eeb4c6 100644 --- a/doc/administration/reference_architectures/1k_users.md +++ b/doc/administration/reference_architectures/1k_users.md @@ -78,9 +78,9 @@ You can also optionally configure GitLab to use an [external PostgreSQL service] or an [external object storage service](../object_storage.md) for added performance and reliability at an increased complexity cost. -## Configure Advanced Search **(PREMIUM SELF)** +## Configure advanced search **(PREMIUM SELF)** -You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md) +You can leverage Elasticsearch and [enable advanced search](../../integration/advanced_search/elasticsearch.md) for faster, more advanced code search across your entire GitLab instance. Elasticsearch cluster design and requirements are dependent on your specific diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 71fe8b301e2..7984b9dd6c7 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -31,8 +31,8 @@ full list of reference architectures, see | Gitaly<sup>5 6</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | | Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | -| GitLab Rails | 5 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | +| Sidekiq<sup>7</sup> | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| GitLab Rails<sup>7</sup> | 5 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | | Object storage<sup>4</sup> | - | - | - | - | @@ -53,6 +53,8 @@ full list of reference architectures, see 6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to [Large repositories](index.md#large-repositories) for more information. +7. Can be placed in Auto Scaling Groups (ASGs) as the component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes). + However, for GitLab Rails certain processes like [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md) should be run on only one node. <!-- markdownlint-enable MD029 --> NOTE: @@ -170,7 +172,7 @@ To set up GitLab and its components to accommodate up to 25,000 users: environment. 1. [Configure the object storage](#configure-the-object-storage) used for shared data objects. -1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, +1. [Configure advanced search](#configure-advanced-search) (optional) for faster, more advanced code search across your entire GitLab instance. The servers start on the same 10.6.0.0/24 private network range, and can @@ -320,7 +322,7 @@ Configure your load balancer to pass connections on port 443 as `TCP` rather than `HTTP(S)` protocol. This will pass the connection to the application node's NGINX service untouched. NGINX will have the SSL certificate and listen on port 443. -See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) +See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html) for details on managing SSL certificates and configuring NGINX. #### Load balancer terminates SSL without backend SSL @@ -331,7 +333,7 @@ terminating SSL. Since communication between the load balancer and GitLab will not be secure, there is some additional configuration needed. See the -[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination) +[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination) for details. #### Load balancer terminates SSL with backend SSL @@ -344,7 +346,7 @@ Traffic will also be secure between the load balancers and NGINX in this scenario. There is no need to add configuration for proxied SSL since the connection will be secure all the way. However, configuration will need to be added to GitLab to configure SSL certificates. See -the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) +the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html) for details on managing SSL certificates and configuring NGINX. <div align="right"> @@ -434,6 +436,11 @@ spread connections equally in practice. ## Configure Consul +Next, we set up the Consul servers. + +NOTE: +Consul must be deployed in an odd number of 3 nodes or more. This is to ensure the nodes can take votes as part of a quorum. + The following IPs will be used as an example: - `10.6.0.11`: Consul 1 @@ -818,6 +825,9 @@ Using [Redis](https://redis.io/) in scalable environment is possible using a **P topology with a [Redis Sentinel](https://redis.io/docs/manual/sentinel/) service to watch and automatically start the failover procedure. +NOTE: +Redis clusters must each be deployed in an odd number of 3 nodes or more. This is to ensure Redis Sentinel can take votes as part of a quorum. This does not apply when configuring Redis externally, such as a cloud provider service. + Redis requires authentication if used with Sentinel. See [Redis Security](https://redis.io/docs/manual/security/) documentation for more information. We recommend using a combination of a Redis password and tight @@ -851,9 +861,9 @@ to be used with GitLab. The following IPs will be used as an example: Managed Redis from cloud providers (such as AWS ElastiCache) will work. If these services support high availability, be sure it _isn't_ of the Redis Cluster type. -Redis version 5.0 or higher is required, which is included with Omnibus GitLab -packages starting with GitLab 13.0. Older Redis versions don't support an -optional count argument to SPOP, which is required for [Merge Trains](../../ci/pipelines/merge_trains.md). + +Because Omnibus GitLab packages ship with Redis 6.0 or later, Redis 6.0 or later is required. Older Redis versions have reached end-of-life. + Note the Redis node's IP address or hostname, port, and password (if required). These will be necessary later when configuring the [GitLab application servers](#configure-gitlab-rails). @@ -1208,7 +1218,7 @@ For more advanced setups refer to the [standalone Gitaly Cluster documentation]( Praefect, the routing and transaction manager for Gitaly Cluster, requires its own database server to store data on Gitaly Cluster status. If you want to have a highly available setup, Praefect requires a third-party PostgreSQL database. -A built-in solution is being [worked on](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919). +A built-in solution is being [worked on](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7292). #### Praefect non-HA PostgreSQL standalone using Omnibus GitLab @@ -1353,6 +1363,9 @@ This is how this would work with a Omnibus GitLab PostgreSQL setup: Praefect is the router and transaction manager for Gitaly Cluster and all connections to Gitaly go through it. This section details how to configure it. +NOTE: +Praefect must be deployed in an odd number of 3 nodes or more. This is to ensure the nodes can take votes as part of a quorum. + Praefect requires several secret tokens to secure communications across the Cluster: - `<praefect_external_token>`: Used for repositories hosted on your Gitaly cluster and can only be accessed by Gitaly clients that carry this token. @@ -1406,7 +1419,6 @@ Updates to example must be made at: # Praefect Configuration praefect['enable'] = true - praefect['listen_addr'] = '0.0.0.0:2305' # Prevent database migrations from running on upgrade automatically praefect['auto_migrate'] = false @@ -1415,51 +1427,69 @@ Updates to example must be made at: # Configure the Consul agent consul['enable'] = true ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true + consul['monitoring_service_discovery'] = true # START user configuration # Please set the real values as explained in Required Information section # - # Praefect External Token - # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster - praefect['auth_token'] = '<praefect_external_token>' - - # Praefect Database Settings - praefect['database_host'] = '10.6.0.141' - praefect['database_port'] = 5432 - # `no_proxy` settings must always be a direct connection for caching - praefect['database_direct_host'] = '10.6.0.141' - praefect['database_direct_port'] = 5432 - praefect['database_dbname'] = 'praefect_production' - praefect['database_user'] = 'praefect' - praefect['database_password'] = '<praefect_postgresql_password>' - - # Praefect Virtual Storage config - # Name of storage hash must match storage name in git_data_dirs on GitLab - # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1') - praefect['virtual_storages'] = { - 'default' => { - 'nodes' => { - 'gitaly-1' => { - 'address' => 'tcp://10.6.0.91:8075', - 'token' => '<praefect_internal_token>' - }, - 'gitaly-2' => { - 'address' => 'tcp://10.6.0.92:8075', - 'token' => '<praefect_internal_token>' - }, - 'gitaly-3' => { - 'address' => 'tcp://10.6.0.93:8075', - 'token' => '<praefect_internal_token>' + praefect['configuration'] = { + # ... + listen_addr: '0.0.0.0:2305', + auth: { + # ... + # + # Praefect External Token + # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster + token: '<praefect_external_token>', + }, + # Praefect Database Settings + database: { + # ... + host: '10.6.0.141', + port: 5432, + # `no_proxy` settings must always be a direct connection for caching + session_pooled: { + # ... + host: '10.6.0.141', + port: 5432, + dbname: 'praefect_production', + user: 'praefect', + password: '<praefect_postgresql_password>', + }, + }, + # Praefect Virtual Storage config + # Name of storage hash must match storage name in git_data_dirs on GitLab + # server ('praefect') and in gitaly['configuration'][:storage] on Gitaly nodes ('gitaly-1') + virtual_storage: [ + { + # ... + name: 'default', + node: [ + { + storage: 'gitaly-1', + address: 'tcp://10.6.0.91:8075', + token: '<praefect_internal_token>' + }, + { + storage: 'gitaly-2', + address: 'tcp://10.6.0.92:8075', + token: '<praefect_internal_token>' + }, + { + storage: 'gitaly-3', + address: 'tcp://10.6.0.93:8075', + token: '<praefect_internal_token>' + }, + ], }, - } - } + ], + # Set the network address Praefect will listen on for monitoring + prometheus_listen_addr: '0.0.0.0:9652', } - # Set the network addresses that the exporters will listen on for monitoring + # Set the network address the node exporter will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' - praefect['prometheus_listen_addr'] = '0.0.0.0:9652' ## The IPs of the Consul server nodes ## You can also use FQDNs and intermix them with IPs @@ -1510,9 +1540,7 @@ Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs should have a throughput of at least 8,000 input/output operations per second (IOPS) for read operations and 2,000 IOPS for -write operations. These IOPS values are initial recommendations, and may be -adjusted to greater or lesser values depending on the scale of your -environment's workload. If you're running the environment on a Cloud provider, +write operations. If you're running the environment on a Cloud provider, refer to their documentation about how to configure IOPS correctly. Gitaly servers must not be exposed to the public internet, as Gitaly's network @@ -1522,7 +1550,7 @@ to restrict access to the Gitaly server. Another option is to For configuring Gitaly you should note the following: -- `git_data_dirs` should be configured to reflect the storage path for the specific Gitaly node +- `gitaly['configuration'][:storage]` should be configured to reflect the storage path for the specific Gitaly node - `auth_token` should be the same as `praefect_internal_token` The following IPs will be used as an example: @@ -1571,20 +1599,6 @@ Updates to example must be made at: # Gitaly gitaly['enable'] = true - # Make Gitaly accept connections on all network interfaces. You must use - # firewalls to restrict access to this address/port. - # Comment out following line if you only want to support TLS connections - gitaly['listen_addr'] = "0.0.0.0:8075" - - # Gitaly Auth Token - # Should be the same as praefect_internal_token - gitaly['auth_token'] = '<praefect_internal_token>' - - # Gitaly Pack-objects cache - # Recommended to be enabled for improved performance but can notably increase disk I/O - # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info - gitaly['pack_objects_cache_enabled'] = true - # Configure the Consul agent consul['enable'] = true ## Enable service discovery for Prometheus @@ -1599,9 +1613,29 @@ Updates to example must be made at: retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), } - # Set the network addresses that the exporters will listen on for monitoring + # Set the network address that the node exporter will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' - gitaly['prometheus_listen_addr'] = '0.0.0.0:9236' + + gitaly['configuration'] = { + # Make Gitaly accept connections on all network interfaces. You must use + # firewalls to restrict access to this address/port. + # Comment out following line if you only want to support TLS connections + listen_addr: '0.0.0.0:8075', + # Set the network address that Gitaly will listen on for monitoring + prometheus_listen_addr: '0.0.0.0:9236', + auth: { + # Gitaly Auth Token + # Should be the same as praefect_internal_token + token: '<praefect_internal_token>', + }, + pack_objects_cache: { + # Gitaly Pack-objects cache + # Recommended to be enabled for improved performance but can notably increase disk I/O + # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info + enabled: true, + }, + } + # # END user configuration ``` @@ -1610,31 +1644,43 @@ Updates to example must be made at: - On Gitaly node 1: ```ruby - git_data_dirs({ - "gitaly-1" => { - "path" => "/var/opt/gitlab/git-data" - } - }) + gitaly['configuration'] = { + # ... + storage: [ + { + name: 'gitaly-1', + path: '/var/opt/gitlab/git-data', + }, + ], + } ``` - On Gitaly node 2: ```ruby - git_data_dirs({ - "gitaly-2" => { - "path" => "/var/opt/gitlab/git-data" - } - }) + gitaly['configuration'] = { + # ... + storage: [ + { + name: 'gitaly-2', + path: '/var/opt/gitlab/git-data', + }, + ], + } ``` - On Gitaly node 3: ```ruby - git_data_dirs({ - "gitaly-3" => { - "path" => "/var/opt/gitlab/git-data" - } - }) + gitaly['configuration'] = { + # ... + storage: [ + { + name: 'gitaly-3', + path: '/var/opt/gitlab/git-data', + }, + ], + } ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace @@ -1654,7 +1700,7 @@ for secure connections, you must: Additionally the certificate, or its certificate authority, must be installed on all Gitaly servers and on all Praefect clients that communicate with it following the procedure described in -[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) (and repeated below). +[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl/index.html#install-custom-public-certificates) (and repeated below). Note the following: @@ -1663,7 +1709,7 @@ Note the following: - You can configure Praefect servers with both an unencrypted listening address `listen_addr` and an encrypted listening address `tls_listen_addr` at the same time. This allows you to do a gradual transition from unencrypted to encrypted traffic, if - necessary. To disable the unencrypted listener, set `praefect['listen_addr'] = nil`. + necessary. To disable the unencrypted listener, set `praefect['configuration'][:listen_addr] = nil`. - The Internal Load Balancer will also access to the certificates and need to be configured to allow for TLS passthrough. Refer to the load balancers documentation on how to configure this. @@ -1685,9 +1731,15 @@ To configure Praefect with TLS: 1. Edit `/etc/gitlab/gitlab.rb` and add: ```ruby - praefect['tls_listen_addr'] = "0.0.0.0:3305" - praefect['certificate_path'] = "/etc/gitlab/ssl/cert.pem" - praefect['key_path'] = "/etc/gitlab/ssl/key.pem" + praefect['configuration'] = { + # ... + tls_listen_addr: '0.0.0.0:3305', + tls: { + # ... + certificate_path: '/etc/gitlab/ssl/cert.pem', + key_path: '/etc/gitlab/ssl/key.pem', + }, + } ``` 1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -1768,7 +1820,7 @@ Updates to example must be made at: # Redis ## Redis connection details - ## First cluster that will host the cache + ## First cluster that will host the cache data gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ @@ -1777,22 +1829,11 @@ Updates to example must be made at: {host: '10.6.0.53', port: 26379}, ] - ## Second cluster that will host the persistent queues, shared state, and actioncable - gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' - gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' - gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' + ## Second cluster that hosts all other persistent data + redis['master_name'] = 'gitlab-redis-persistent' + redis['master_password'] = '<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>' - gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.61', port: 26379}, - {host: '10.6.0.62', port: 26379}, - {host: '10.6.0.63', port: 26379}, - ] - gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.61', port: 26379}, - {host: '10.6.0.62', port: 26379}, - {host: '10.6.0.63', port: 26379}, - ] - gitlab_rails['redis_actioncable_sentinels'] = [ + gitlab_rails['redis_sentinels'] = [ {host: '10.6.0.61', port: 26379}, {host: '10.6.0.62', port: 26379}, {host: '10.6.0.63', port: 26379}, @@ -1955,7 +1996,7 @@ On each node perform the following: gitlab_rails['auto_migrate'] = false ## Redis connection details - ## First cluster that will host the cache + ## First cluster that will host the cache data gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ @@ -1964,22 +2005,11 @@ On each node perform the following: {host: '10.6.0.53', port: 26379}, ] - ## Second cluster that will host the queues, shared state, and actionable - gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' - gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' - gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' + ## Second cluster that hosts all other persistent data + redis['master_name'] = 'gitlab-redis-persistent' + redis['master_password'] = '<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>' - gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.61', port: 26379}, - {host: '10.6.0.62', port: 26379}, - {host: '10.6.0.63', port: 26379}, - ] - gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.61', port: 26379}, - {host: '10.6.0.62', port: 26379}, - {host: '10.6.0.63', port: 26379}, - ] - gitlab_rails['redis_actioncable_sentinels'] = [ + gitlab_rails['redis_sentinels'] = [ {host: '10.6.0.61', port: 26379}, {host: '10.6.0.62', port: 26379}, {host: '10.6.0.63', port: 26379}, @@ -2077,7 +2107,7 @@ On each node perform the following: When you specify `https` in the `external_url`, as in the previous example, GitLab expects that the SSL certificates are in `/etc/gitlab/ssl/`. If the certificates aren't present, NGINX will fail to start. For more information, see -the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html). +the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html). ### GitLab Rails post-configuration @@ -2089,11 +2119,9 @@ the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html). sudo gitlab-rake gitlab:db:configure ``` - If you encounter a `rake aborted!` error message stating that PgBouncer is - failing to connect to PostgreSQL, it may be that your PgBouncer node's IP - address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` - on your database nodes. Before proceeding, see - [PgBouncer error `ERROR: pgbouncer cannot connect to server`](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). + Note that this requires the Rails node to be configured to connect to the primary database + directly, [bypassing PgBouncer](../postgresql/pgbouncer.md#procedure-for-bypassing-pgbouncer). + After migrations have completed, you must configure the node to pass through PgBouncer again. 1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). @@ -2188,15 +2216,15 @@ and scalable. There are two ways of specifying object storage configuration in GitLab: -- [Consolidated form](../object_storage.md#consolidated-object-storage-configuration): A single credential is +- [Consolidated form](../object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form): A single credential is shared by all supported object types. -- [Storage-specific form](../object_storage.md#storage-specific-configuration): Every object defines its - own object storage [connection and configuration](../object_storage.md#connection-settings). +- [Storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form): Every object defines its + own object storage [connection and configuration](../object_storage.md#configure-the-connection-settings). The consolidated form is used in the following examples when available. NOTE: -When using the [storage-specific form](../object_storage.md#storage-specific-configuration) +When using the [storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form) in GitLab 14.x and earlier, you should enable [direct upload mode](../../development/uploads/index.md#direct-upload). The previous [background upload](../../development/uploads/index.md#direct-upload) mode, which was deprecated in 14.9, requires shared storage such as NFS. @@ -2218,9 +2246,9 @@ GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs. -## Configure Advanced Search +## Configure advanced search -You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md) +You can leverage Elasticsearch and [enable advanced search](../../integration/advanced_search/elasticsearch.md) for faster, more advanced code search across your entire GitLab instance. Elasticsearch cluster design and requirements are dependent on your specific diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index ee26504902c..14dc9d26293 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -26,7 +26,7 @@ For a full list of reference architectures, see | PostgreSQL<sup>1</sup> | 1 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | | Redis<sup>2</sup> | 1 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `m5.large` | `D2s v3` | | Gitaly<sup>5</sup> | 1 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| GitLab Rails | 2 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | +| GitLab Rails<sup>6</sup> | 2 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | | Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Object storage<sup>4</sup> | - | - | - | - | - | @@ -45,6 +45,8 @@ For a full list of reference architectures, see 5. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to [Large repositories](index.md#large-repositories) for more information. +6. Can be placed in Auto Scaling Groups (ASGs) as the component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes). + However, for GitLab Rails certain processes like [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md) should be run on only one node. <!-- markdownlint-enable MD029 --> NOTE: @@ -101,7 +103,7 @@ To set up GitLab and its components to accommodate up to 2,000 users: environment. 1. [Configure the object storage](#configure-the-object-storage) used for shared data objects. -1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, +1. [Configure advanced search](#configure-advanced-search) (optional) for faster, more advanced code search across your entire GitLab instance. ## Configure the external load balancer @@ -211,7 +213,7 @@ Configure your load balancer to pass connections on port 443 as `TCP` rather than `HTTP(S)` protocol. This will pass the connection to the application node's NGINX service untouched. NGINX will have the SSL certificate and listen on port 443. -See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) +See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html) for details on managing SSL certificates and configuring NGINX. #### Load balancer terminates SSL without backend SSL @@ -222,7 +224,7 @@ terminating SSL. Since communication between the load balancer and GitLab will not be secure, there is some additional configuration needed. See the -[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination) +[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination) for details. #### Load balancer terminates SSL with backend SSL @@ -235,7 +237,7 @@ Traffic will also be secure between the load balancers and NGINX in this scenario. There is no need to add configuration for proxied SSL since the connection will be secure all the way. However, configuration will need to be added to GitLab to configure SSL certificates. See -the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) +the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html) for details on managing SSL certificates and configuring NGINX. <div align="right"> @@ -339,13 +341,7 @@ to be used with GitLab. ### Provide your own Redis instance -Redis version 5.0 or higher is required, as this is what ships with -Omnibus GitLab packages starting with GitLab 13.0. Older Redis versions -do not support an optional count argument to SPOP which is now required for -[Merge Trains](../../ci/pipelines/merge_trains.md). - -In addition, GitLab makes use of certain commands like `UNLINK` and `USAGE` which -were introduced only in Redis 4. +Because Omnibus GitLab packages ship with Redis 6.0 or later, Redis 6.0 or later is required. Older Redis versions have reached end-of-life. Managed Redis from cloud providers such as AWS ElastiCache will work. If these services support high availability, be sure it is not the Redis Cluster type. @@ -423,9 +419,7 @@ Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs should have a throughput of at least 8,000 input/output operations per second (IOPS) for read operations and 2,000 IOPS -for write operations. These IOPS values are initial recommendations, and may be -adjusted to greater or lesser values depending on the scale of your -environment's workload. If you're running the environment on a Cloud provider, +for write operations. If you're running the environment on a Cloud provider, refer to their documentation about how to configure IOPS correctly. Be sure to note the following items: @@ -459,7 +453,7 @@ To configure the Gitaly server, on the server node you want to use for Gitaly: storage paths, enable the network listener, and to configure the token: NOTE: - You can't remove the `default` entry from `git_data_dirs` because [GitLab requires it](../gitaly/configure_gitaly.md#gitlab-requires-a-default-repository-storage). + You can't remove the `default` entry from `gitaly['configuration'][:storage]` because [GitLab requires it](../gitaly/configure_gitaly.md#gitlab-requires-a-default-repository-storage). <!-- Updates to example must be made at: @@ -493,30 +487,48 @@ Updates to example must be made at: # Gitaly gitaly['enable'] = true - # Make Gitaly accept connections on all network interfaces. You must use - # firewalls to restrict access to this address/port. - # Comment out following line if you only want to support TLS connections - gitaly['listen_addr'] = "0.0.0.0:8075" - gitaly['prometheus_listen_addr'] = "0.0.0.0:9236" - - # Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests - # to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API. - # The following two values must be the same as their respective values - # of the GitLab Rails application setup - gitaly['auth_token'] = 'gitalysecret' + # The secret token is used for authentication callbacks from Gitaly to the GitLab internal API. + # This must match the respective value in GitLab Rails application setup. gitlab_shell['secret_token'] = 'shellsecret' # Set the network addresses that the exporters used for monitoring will listen on node_exporter['listen_address'] = '0.0.0.0:9100' - git_data_dirs({ - 'default' => { - 'path' => '/var/opt/gitlab/git-data' - }, - 'storage1' => { - 'path' => '/mnt/gitlab/git-data' - }, - }) + gitaly['configuration'] = { + # ... + # + # Make Gitaly accept connections on all network interfaces. You must use + # firewalls to restrict access to this address/port. + # Comment out following line if you only want to support TLS connections + listen_addr: '0.0.0.0:8075', + prometheus_listen_addr: '0.0.0.0:9236', + # Gitaly Auth Token + # Should be the same as praefect_internal_token + auth: { + # ... + # + # Gitaly's authentication token is used to authenticate gRPC requests to Gitaly. This must match + # the respective value in GitLab Rails application setup. + token: 'gitalysecret', + }, + # Gitaly Pack-objects cache + # Recommended to be enabled for improved performance but can notably increase disk I/O + # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info + pack_objects_cache: { + # ... + enabled: true, + }, + storage: [ + { + name: 'default', + path: '/var/opt/gitlab/git-data', + }, + { + name: 'storage1', + path: '/mnt/gitlab/git-data', + }, + ], + } ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace @@ -538,7 +550,7 @@ You will need to bring your own certificates as this isn't provided automaticall The certificate, or its certificate authority, must be installed on all Gitaly nodes (including the Gitaly node using the certificate) and on all client nodes that communicate with it following the procedure described in -[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). +[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl/index.html#install-custom-public-certificates). NOTE: The self-signed certificate must specify the address you use to access the @@ -574,9 +586,14 @@ To configure Gitaly with TLS: <!-- Updates to following example must also be made at https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab --> ```ruby - gitaly['tls_listen_addr'] = "0.0.0.0:9999" - gitaly['certificate_path'] = "/etc/gitlab/ssl/cert.pem" - gitaly['key_path'] = "/etc/gitlab/ssl/key.pem" + gitaly['configuration'] = { + # ... + tls_listen_addr: '0.0.0.0:9999', + tls: { + certificate_path: '/etc/gitlab/ssl/cert.pem', + key_path: '/etc/gitlab/ssl/key.pem', + }, + } ``` 1. Delete `gitaly['listen_addr']` to allow only encrypted connections. @@ -739,7 +756,7 @@ On each node perform the following: When you specify `https` in the `external_url`, as in the previous example, GitLab expects that the SSL certificates are in `/etc/gitlab/ssl/`. If the certificates aren't present, NGINX will fail to start. For more information, see -the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html). +the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html). ### GitLab Rails post-configuration @@ -751,11 +768,9 @@ the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html). sudo gitlab-rake gitlab:db:configure ``` - If you encounter a `rake aborted!` error message stating that PgBouncer is - failing to connect to PostgreSQL, it may be that your PgBouncer node's IP - address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` - on your database nodes. Before proceeding, see - [PgBouncer error `ERROR: pgbouncer cannot connect to server`](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). + Note that this requires the Rails node to be configured to connect to the primary database + directly, [bypassing PgBouncer](../postgresql/pgbouncer.md#procedure-for-bypassing-pgbouncer). + After migrations have completed, you must configure the node to pass through PgBouncer again. 1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). @@ -881,15 +896,15 @@ and scalable. There are two ways of specifying object storage configuration in GitLab: -- [Consolidated form](../object_storage.md#consolidated-object-storage-configuration): A single credential is +- [Consolidated form](../object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form): A single credential is shared by all supported object types. -- [Storage-specific form](../object_storage.md#storage-specific-configuration): Every object defines its - own object storage [connection and configuration](../object_storage.md#connection-settings). +- [Storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form): Every object defines its + own object storage [connection and configuration](../object_storage.md#configure-the-connection-settings). The consolidated form is used in the following examples when available. NOTE: -When using the [storage-specific form](../object_storage.md#storage-specific-configuration) +When using the [storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form) in GitLab 14.x and earlier, you should enable [direct upload mode](../../development/uploads/index.md#direct-upload). The previous [background upload](../../development/uploads/index.md#direct-upload) mode, which was deprecated in 14.9, requires shared storage such as NFS. @@ -911,9 +926,9 @@ GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs. -## Configure Advanced Search **(PREMIUM SELF)** +## Configure advanced search **(PREMIUM SELF)** -You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md) +You can leverage Elasticsearch and [enable advanced search](../../integration/advanced_search/elasticsearch.md) for faster, more advanced code search across your entire GitLab instance. Elasticsearch cluster design and requirements are dependent on your specific diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index 1d5dad02b18..191e5218c43 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -40,8 +40,8 @@ For a full list of reference architectures, see | Gitaly<sup>5 6</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | | Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | -| GitLab Rails | 3 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | +| Sidekiq<sup>7</sup> | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | +| GitLab Rails<sup>7</sup> | 3 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | | Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Object storage<sup>4</sup> | - | - | - | - | @@ -62,6 +62,8 @@ For a full list of reference architectures, see 6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to [Large repositories](index.md#large-repositories) for more information. +7. Can be placed in Auto Scaling Groups (ASGs) as the component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes). + However, for GitLab Rails certain processes like [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md) should be run on only one node. <!-- markdownlint-enable MD029 --> NOTE: @@ -176,7 +178,7 @@ To set up GitLab and its components to accommodate up to 3,000 users: environment. 1. [Configure the object storage](#configure-the-object-storage) used for shared data objects. -1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, +1. [Configure advanced search](#configure-advanced-search) (optional) for faster, more advanced code search across your entire GitLab instance. The servers start on the same 10.6.0.0/24 private network range, and can @@ -321,7 +323,7 @@ Configure your load balancer to pass connections on port 443 as `TCP` rather than `HTTP(S)` protocol. This will pass the connection to the application node's NGINX service untouched. NGINX will have the SSL certificate and listen on port 443. -See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) +See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html) for details on managing SSL certificates and configuring NGINX. #### Load balancer terminates SSL without backend SSL @@ -332,7 +334,7 @@ terminating SSL. Since communication between the load balancer and GitLab will not be secure, there is some additional configuration needed. See the -[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination) +[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination) for details. #### Load balancer terminates SSL with backend SSL @@ -345,7 +347,7 @@ Traffic will also be secure between the load balancers and NGINX in this scenario. There is no need to add configuration for proxied SSL since the connection will be secure all the way. However, configuration will need to be added to GitLab to configure SSL certificates. See -the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) +the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html) for details on managing SSL certificates and configuring NGINX. <div align="right"> @@ -453,10 +455,7 @@ to be used with GitLab. The following IPs will be used as an example: Managed Redis from cloud providers such as AWS ElastiCache will work. If these services support high availability, be sure it is **not** the Redis Cluster type. -Redis version 5.0 or higher is required, as this is what ships with -Omnibus GitLab packages starting with GitLab 13.0. Older Redis versions -do not support an optional count argument to SPOP which is now required for -[Merge Trains](../../ci/pipelines/merge_trains.md). +Because Omnibus GitLab packages ship with Redis 6.0 or later, Redis 6.0 or later is required. Older Redis versions have reached end-of-life. Note the Redis node's IP address or hostname, port, and password (if required). These will be necessary when configuring the @@ -640,8 +639,13 @@ are supported and can be added if needed. ## Configure Consul and Sentinel -Now that the Redis servers are all set up, let's configure the Sentinel -servers. The following IPs will be used as an example: +Now that the Redis servers are all set up, let's configure the Consul + Sentinel +servers. + +NOTE: +Consul and Redis Sentinel must be deployed in an odd number of 3 nodes or more. This is to ensure the nodes can take votes as part of a quorum. + +The following IPs will be used as an example: - `10.6.0.11`: Consul/Sentinel 1 - `10.6.0.12`: Consul/Sentinel 2 @@ -1144,7 +1148,7 @@ For more advanced setups refer to the [standalone Gitaly Cluster documentation]( Praefect, the routing and transaction manager for Gitaly Cluster, requires its own database server to store data on Gitaly Cluster status. If you want to have a highly available setup, Praefect requires a third-party PostgreSQL database. -A built-in solution is being [worked on](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919). +A built-in solution is being [worked on](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7292). #### Praefect non-HA PostgreSQL standalone using Omnibus GitLab @@ -1288,6 +1292,9 @@ This is how this would work with a Omnibus GitLab PostgreSQL setup: Praefect is the router and transaction manager for Gitaly Cluster and all connections to Gitaly go through it. This section details how to configure it. +NOTE: +Praefect must be deployed in an odd number of 3 nodes or more. This is to ensure the nodes can take votes as part of a quorum. + Praefect requires several secret tokens to secure communications across the Cluster: - `<praefect_external_token>`: Used for repositories hosted on your Gitaly cluster and can only be accessed by Gitaly clients that carry this token. @@ -1341,7 +1348,6 @@ Updates to example must be made at: # Praefect Configuration praefect['enable'] = true - praefect['listen_addr'] = '0.0.0.0:2305' # Prevent database migrations from running on upgrade automatically praefect['auto_migrate'] = false @@ -1350,51 +1356,69 @@ Updates to example must be made at: # Configure the Consul agent consul['enable'] = true ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true + consul['monitoring_service_discovery'] = true # START user configuration # Please set the real values as explained in Required Information section # - # Praefect External Token - # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster - praefect['auth_token'] = '<praefect_external_token>' - - # Praefect Database Settings - praefect['database_host'] = '10.6.0.141' - praefect['database_port'] = 5432 - # `no_proxy` settings must always be a direct connection for caching - praefect['database_direct_host'] = '10.6.0.141' - praefect['database_direct_port'] = 5432 - praefect['database_dbname'] = 'praefect_production' - praefect['database_user'] = 'praefect' - praefect['database_password'] = '<praefect_postgresql_password>' - - # Praefect Virtual Storage config - # Name of storage hash must match storage name in git_data_dirs on GitLab - # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1') - praefect['virtual_storages'] = { - 'default' => { - 'nodes' => { - 'gitaly-1' => { - 'address' => 'tcp://10.6.0.91:8075', - 'token' => '<praefect_internal_token>' - }, - 'gitaly-2' => { - 'address' => 'tcp://10.6.0.92:8075', - 'token' => '<praefect_internal_token>' - }, - 'gitaly-3' => { - 'address' => 'tcp://10.6.0.93:8075', - 'token' => '<praefect_internal_token>' + praefect['configuration'] = { + # ... + listen_addr: '0.0.0.0:2305', + auth: { + # ... + # + # Praefect External Token + # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster + token: '<praefect_external_token>', + }, + # Praefect Database Settings + database: { + # ... + host: '10.6.0.141', + port: 5432, + # `no_proxy` settings must always be a direct connection for caching + session_pooled: { + # ... + host: '10.6.0.141', + port: 5432, + dbname: 'praefect_production', + user: 'praefect', + password: '<praefect_postgresql_password>', + }, + }, + # Praefect Virtual Storage config + # Name of storage hash must match storage name in git_data_dirs on GitLab + # server ('praefect') and in gitaly['configuration'][:storage] on Gitaly nodes ('gitaly-1') + virtual_storage: [ + { + # ... + name: 'default', + node: [ + { + storage: 'gitaly-1', + address: 'tcp://10.6.0.91:8075', + token: '<praefect_internal_token>' + }, + { + storage: 'gitaly-2', + address: 'tcp://10.6.0.92:8075', + token: '<praefect_internal_token>' + }, + { + storage: 'gitaly-3', + address: 'tcp://10.6.0.93:8075', + token: '<praefect_internal_token>' + }, + ], }, - } - } + ], + # Set the network address Praefect will listen on for monitoring + prometheus_listen_addr: '0.0.0.0:9652', } - # Set the network addresses that the exporters will listen on for monitoring + # Set the network address the node exporter will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' - praefect['prometheus_listen_addr'] = '0.0.0.0:9652' ## The IPs of the Consul server nodes ## You can also use FQDNs and intermix them with IPs @@ -1445,9 +1469,7 @@ Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs should have a throughput of at least 8,000 input/output operations per second (IOPS) for read operations and 2,000 IOPS for -write operations. These IOPS values are initial recommendations, and may be -adjusted to greater or lesser values depending on the scale of your -environment's workload. If you're running the environment on a Cloud provider, +write operations. If you're running the environment on a Cloud provider, refer to their documentation about how to configure IOPS correctly. Gitaly servers must not be exposed to the public internet, as Gitaly's network @@ -1457,7 +1479,7 @@ to restrict access to the Gitaly server. Another option is to For configuring Gitaly you should note the following: -- `git_data_dirs` should be configured to reflect the storage path for the specific Gitaly node +- `gitaly['configuration'][:storage]` should be configured to reflect the storage path for the specific Gitaly node - `auth_token` should be the same as `praefect_internal_token` The following IPs will be used as an example: @@ -1506,20 +1528,6 @@ Updates to example must be made at: # balancer. gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' - # Make Gitaly accept connections on all network interfaces. You must use - # firewalls to restrict access to this address/port. - # Comment out following line if you only want to support TLS connections - gitaly['listen_addr'] = "0.0.0.0:8075" - - # Gitaly Auth Token - # Should be the same as praefect_internal_token - gitaly['auth_token'] = '<praefect_internal_token>' - - # Gitaly Pack-objects cache - # Recommended to be enabled for improved performance but can notably increase disk I/O - # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info - gitaly['pack_objects_cache_enabled'] = true - # Configure the Consul agent consul['enable'] = true ## Enable service discovery for Prometheus @@ -1534,9 +1542,33 @@ Updates to example must be made at: retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), } - # Set the network addresses that the exporters will listen on for monitoring + # Set the network address that the node exporter will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' - gitaly['prometheus_listen_addr'] = '0.0.0.0:9236' + + gitaly['configuration'] = { + # ... + # + # Make Gitaly accept connections on all network interfaces. You must use + # firewalls to restrict access to this address/port. + # Comment out following line if you only want to support TLS connections + listen_addr: '0.0.0.0:8075', + # Set the network address that Gitaly will listen on for monitoring + prometheus_listen_addr: '0.0.0.0:9236', + # Gitaly Auth Token + # Should be the same as praefect_internal_token + auth: { + # ... + token: '<praefect_internal_token>', + }, + # Gitaly Pack-objects cache + # Recommended to be enabled for improved performance but can notably increase disk I/O + # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info + pack_objects_cache: { + # ... + enabled: true, + }, + } + # # END user configuration ``` @@ -1545,31 +1577,43 @@ Updates to example must be made at: - On Gitaly node 1: ```ruby - git_data_dirs({ - "gitaly-1" => { - "path" => "/var/opt/gitlab/git-data" - } - }) + gitaly['configuration'] = { + # ... + storage: [ + { + name: 'gitaly-1', + path: '/var/opt/gitlab/git-data', + }, + ], + } ``` - On Gitaly node 2: ```ruby - git_data_dirs({ - "gitaly-2" => { - "path" => "/var/opt/gitlab/git-data" - } - }) + gitaly['configuration'] = { + # ... + storage: [ + { + name: 'gitaly-2', + path: '/var/opt/gitlab/git-data', + }, + ], + } ``` - On Gitaly node 3: ```ruby - git_data_dirs({ - "gitaly-3" => { - "path" => "/var/opt/gitlab/git-data" - } - }) + gitaly['configuration'] = { + # ... + storage: [ + { + name: 'gitaly-3', + path: '/var/opt/gitlab/git-data', + }, + ], + } ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace @@ -1589,7 +1633,7 @@ for secure connections, you must: Additionally the certificate, or its certificate authority, must be installed on all Gitaly servers and on all Praefect clients that communicate with it following the procedure described in -[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) (and repeated below). +[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl/index.html#install-custom-public-certificates) (and repeated below). Note the following: @@ -1598,7 +1642,7 @@ Note the following: - You can configure Praefect servers with both an unencrypted listening address `listen_addr` and an encrypted listening address `tls_listen_addr` at the same time. This allows you to do a gradual transition from unencrypted to encrypted traffic, if - necessary. To disable the unencrypted listener, set `praefect['listen_addr'] = nil`. + necessary. To disable the unencrypted listener, set `praefect['configuration'][:listen_addr] = nil`. - The Internal Load Balancer will also access to the certificates and need to be configured to allow for TLS passthrough. Refer to the load balancers documentation on how to configure this. @@ -1620,9 +1664,15 @@ To configure Praefect with TLS: 1. Edit `/etc/gitlab/gitlab.rb` and add: ```ruby - praefect['tls_listen_addr'] = "0.0.0.0:3305" - praefect['certificate_path'] = "/etc/gitlab/ssl/cert.pem" - praefect['key_path'] = "/etc/gitlab/ssl/key.pem" + praefect['configuration'] = { + # ... + tls_listen_addr: '0.0.0.0:3305', + tls: { + # ... + certificate_path: '/etc/gitlab/ssl/cert.pem', + key_path: '/etc/gitlab/ssl/key.pem', + }, + } ``` 1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -2018,7 +2068,7 @@ On each node perform the following: When you specify `https` in the `external_url`, as in the previous example, GitLab expects that the SSL certificates are in `/etc/gitlab/ssl/`. If the certificates aren't present, NGINX will fail to start. For more information, see -the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html). +the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html). ### GitLab Rails post-configuration @@ -2028,11 +2078,9 @@ the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html). gitlab-rake gitlab:db:configure ``` - If you encounter a `rake aborted!` error message stating that PgBouncer is - failing to connect to PostgreSQL, it may be that your PgBouncer node's IP - address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` - on your database nodes. Before proceeding, see - [PgBouncer error `ERROR: pgbouncer cannot connect to server`](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). + Note that this requires the Rails node to be configured to connect to the primary database + directly, [bypassing PgBouncer](../postgresql/pgbouncer.md#procedure-for-bypassing-pgbouncer). + After migrations have completed, you must configure the node to pass through PgBouncer again. 1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). @@ -2135,15 +2183,15 @@ and scalable. There are two ways of specifying object storage configuration in GitLab: -- [Consolidated form](../object_storage.md#consolidated-object-storage-configuration): A single credential is +- [Consolidated form](../object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form): A single credential is shared by all supported object types. -- [Storage-specific form](../object_storage.md#storage-specific-configuration): Every object defines its - own object storage [connection and configuration](../object_storage.md#connection-settings). +- [Storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form): Every object defines its + own object storage [connection and configuration](../object_storage.md#configure-the-connection-settings). The consolidated form is used in the following examples when available. NOTE: -When using the [storage-specific form](../object_storage.md#storage-specific-configuration) +When using the [storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form) in GitLab 14.x and earlier, you should enable [direct upload mode](../../development/uploads/index.md#direct-upload). The previous [background upload](../../development/uploads/index.md#direct-upload) mode, which was deprecated in 14.9, requires shared storage such as NFS. @@ -2165,9 +2213,9 @@ GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs. -## Configure Advanced Search +## Configure advanced search -You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md) +You can leverage Elasticsearch and [enable advanced search](../../integration/advanced_search/elasticsearch.md) for faster, more advanced code search across your entire GitLab instance. Elasticsearch cluster design and requirements are dependent on your specific diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 3bcffa8f606..7d3ddf7578c 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -31,8 +31,8 @@ full list of reference architectures, see | Gitaly<sup>5 6</sup> | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | `m5.16xlarge` | | Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | -| GitLab Rails | 12 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | +| Sidekiq<sup>7</sup> | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| GitLab Rails<sup>7</sup> | 12 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | | Object storage<sup>4</sup> | - | - | - | - | @@ -53,6 +53,8 @@ full list of reference architectures, see 6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to [Large repositories](index.md#large-repositories) for more information. +7. Can be placed in Auto Scaling Groups (ASGs) as the component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes). + However, for GitLab Rails certain processes like [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md) should be run on only one node. <!-- markdownlint-enable MD029 --> NOTE: @@ -170,7 +172,7 @@ To set up GitLab and its components to accommodate up to 50,000 users: environment. 1. [Configure the object storage](#configure-the-object-storage) used for shared data objects. -1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, +1. [Configure advanced search](#configure-advanced-search) (optional) for faster, more advanced code search across your entire GitLab instance. The servers start on the same 10.6.0.0/24 private network range, and can @@ -318,7 +320,7 @@ Configure your load balancer to pass connections on port 443 as `TCP` rather than `HTTP(S)` protocol. This will pass the connection to the application node's NGINX service untouched. NGINX will have the SSL certificate and listen on port 443. -See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) +See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html) for details on managing SSL certificates and configuring NGINX. #### Load balancer terminates SSL without backend SSL @@ -329,7 +331,7 @@ terminating SSL. Since communication between the load balancer and GitLab will not be secure, there is some additional configuration needed. See the -[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination) +[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination) for details. #### Load balancer terminates SSL with backend SSL @@ -342,7 +344,7 @@ Traffic will also be secure between the load balancers and NGINX in this scenario. There is no need to add configuration for proxied SSL since the connection will be secure all the way. However, configuration will need to be added to GitLab to configure SSL certificates. See -the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) +the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html) for details on managing SSL certificates and configuring NGINX. <div align="right"> @@ -426,6 +428,11 @@ spread connections equally in practice. ## Configure Consul +Next, we set up the Consul servers. + +NOTE: +Consul must be deployed in an odd number of 3 nodes or more. This is to ensure the nodes can take votes as part of a quorum. + The following IPs will be used as an example: - `10.6.0.11`: Consul 1 @@ -811,6 +818,9 @@ Using [Redis](https://redis.io/) in scalable environment is possible using a **P topology with a [Redis Sentinel](https://redis.io/docs/manual/sentinel/) service to watch and automatically start the failover procedure. +NOTE: +Redis clusters must each be deployed in an odd number of 3 nodes or more. This is to ensure Redis Sentinel can take votes as part of a quorum. This does not apply when configuring Redis externally, such as a cloud provider service. + Redis requires authentication if used with Sentinel. See [Redis Security](https://redis.io/docs/manual/security/) documentation for more information. We recommend using a combination of a Redis password and tight @@ -844,9 +854,9 @@ to be used with GitLab. The following IPs will be used as an example: Managed Redis from cloud providers (such as AWS ElastiCache) will work. If these services support high availability, be sure it _isn't_ of the Redis Cluster type. -Redis version 5.0 or higher is required, which is included with Omnibus GitLab -packages starting with GitLab 13.0. Older Redis versions don't support an -optional count argument to SPOP, which is required for [Merge Trains](../../ci/pipelines/merge_trains.md). + +Because Omnibus GitLab packages ship with Redis 6.0 or later, Redis 6.0 or later is required. Older Redis versions have reached end-of-life. + Note the Redis node's IP address or hostname, port, and password (if required). These will be necessary later when configuring the [GitLab application servers](#configure-gitlab-rails). @@ -1202,7 +1212,7 @@ For more advanced setups refer to the [standalone Gitaly Cluster documentation]( Praefect, the routing and transaction manager for Gitaly Cluster, requires its own database server to store data on Gitaly Cluster status. If you want to have a highly available setup, Praefect requires a third-party PostgreSQL database. -A built-in solution is being [worked on](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919). +A built-in solution is being [worked on](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7292). #### Praefect non-HA PostgreSQL standalone using Omnibus GitLab @@ -1349,6 +1359,9 @@ This is how this would work with a Omnibus GitLab PostgreSQL setup: Praefect is the router and transaction manager for Gitaly Cluster and all connections to Gitaly go through it. This section details how to configure it. +NOTE: +Praefect must be deployed in an odd number of 3 nodes or later. This is to ensure the nodes can take votes as part of a quorum. + Praefect requires several secret tokens to secure communications across the Cluster: - `<praefect_external_token>`: Used for repositories hosted on your Gitaly cluster and can only be accessed by Gitaly clients that carry this token. @@ -1402,7 +1415,6 @@ Updates to example must be made at: # Praefect Configuration praefect['enable'] = true - praefect['listen_addr'] = '0.0.0.0:2305' # Prevent database migrations from running on upgrade automatically praefect['auto_migrate'] = false @@ -1411,51 +1423,69 @@ Updates to example must be made at: # Configure the Consul agent consul['enable'] = true ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true + consul['monitoring_service_discovery'] = true # START user configuration # Please set the real values as explained in Required Information section # - # Praefect External Token - # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster - praefect['auth_token'] = '<praefect_external_token>' - - # Praefect Database Settings - praefect['database_host'] = '10.6.0.141' - praefect['database_port'] = 5432 - # `no_proxy` settings must always be a direct connection for caching - praefect['database_direct_host'] = '10.6.0.141' - praefect['database_direct_port'] = 5432 - praefect['database_dbname'] = 'praefect_production' - praefect['database_user'] = 'praefect' - praefect['database_password'] = '<praefect_postgresql_password>' - - # Praefect Virtual Storage config - # Name of storage hash must match storage name in git_data_dirs on GitLab - # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1') - praefect['virtual_storages'] = { - 'default' => { - 'nodes' => { - 'gitaly-1' => { - 'address' => 'tcp://10.6.0.91:8075', - 'token' => '<praefect_internal_token>' - }, - 'gitaly-2' => { - 'address' => 'tcp://10.6.0.92:8075', - 'token' => '<praefect_internal_token>' - }, - 'gitaly-3' => { - 'address' => 'tcp://10.6.0.93:8075', - 'token' => '<praefect_internal_token>' + praefect['configuration'] = { + # ... + listen_addr: '0.0.0.0:2305', + auth: { + # ... + # + # Praefect External Token + # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster + token: '<praefect_external_token>', + }, + # Praefect Database Settings + database: { + # ... + host: '10.6.0.141', + port: 5432, + # `no_proxy` settings must always be a direct connection for caching + session_pooled: { + # ... + host: '10.6.0.141', + port: 5432, + dbname: 'praefect_production', + user: 'praefect', + password: '<praefect_postgresql_password>', + }, + }, + # Praefect Virtual Storage config + # Name of storage hash must match storage name in git_data_dirs on GitLab + # server ('praefect') and in gitaly['configuration'][:storage] on Gitaly nodes ('gitaly-1') + virtual_storage: [ + { + # ... + name: 'default', + node: [ + { + storage: 'gitaly-1', + address: 'tcp://10.6.0.91:8075', + token: '<praefect_internal_token>' + }, + { + storage: 'gitaly-2', + address: 'tcp://10.6.0.92:8075', + token: '<praefect_internal_token>' + }, + { + storage: 'gitaly-3', + address: 'tcp://10.6.0.93:8075', + token: '<praefect_internal_token>' + }, + ], }, - } - } + ], + # Set the network address Praefect will listen on for monitoring + prometheus_listen_addr: '0.0.0.0:9652', } - # Set the network addresses that the exporters will listen on for monitoring + # Set the network address the node exporter will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' - praefect['prometheus_listen_addr'] = '0.0.0.0:9652' ## The IPs of the Consul server nodes ## You can also use FQDNs and intermix them with IPs @@ -1506,9 +1536,7 @@ Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs should have a throughput of at least 8,000 input/output operations per second (IOPS) for read operations and 2,000 IOPS for -write operations. These IOPS values are initial recommendations, and may be -adjusted to greater or lesser values depending on the scale of your -environment's workload. If you're running the environment on a Cloud provider, +write operations. If you're running the environment on a Cloud provider, refer to their documentation about how to configure IOPS correctly. Gitaly servers must not be exposed to the public internet, as Gitaly's network @@ -1518,7 +1546,7 @@ to restrict access to the Gitaly server. Another option is to For configuring Gitaly you should note the following: -- `git_data_dirs` should be configured to reflect the storage path for the specific Gitaly node +- `gitaly['configuration'][:storage]` should be configured to reflect the storage path for the specific Gitaly node - `auth_token` should be the same as `praefect_internal_token` The following IPs will be used as an example: @@ -1567,20 +1595,6 @@ Updates to example must be made at: # Gitaly gitaly['enable'] = true - # Make Gitaly accept connections on all network interfaces. You must use - # firewalls to restrict access to this address/port. - # Comment out following line if you only want to support TLS connections - gitaly['listen_addr'] = "0.0.0.0:8075" - - # Gitaly Auth Token - # Should be the same as praefect_internal_token - gitaly['auth_token'] = '<praefect_internal_token>' - - # Gitaly Pack-objects cache - # Recommended to be enabled for improved performance but can notably increase disk I/O - # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info - gitaly['pack_objects_cache_enabled'] = true - # Configure the Consul agent consul['enable'] = true ## Enable service discovery for Prometheus @@ -1595,9 +1609,29 @@ Updates to example must be made at: retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), } - # Set the network addresses that the exporters will listen on for monitoring + # Set the network address that the node exporter will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' - gitaly['prometheus_listen_addr'] = '0.0.0.0:9236' + + gitaly['configuration'] = { + # Make Gitaly accept connections on all network interfaces. You must use + # firewalls to restrict access to this address/port. + # Comment out following line if you only want to support TLS connections + listen_addr: '0.0.0.0:8075', + # Set the network address that Gitaly will listen on for monitoring + prometheus_listen_addr: '0.0.0.0:9236', + auth: { + # Gitaly Auth Token + # Should be the same as praefect_internal_token + token: '<praefect_internal_token>', + }, + pack_objects_cache: { + # Gitaly Pack-objects cache + # Recommended to be enabled for improved performance but can notably increase disk I/O + # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info + enabled: true, + }, + } + # # END user configuration ``` @@ -1606,31 +1640,43 @@ Updates to example must be made at: - On Gitaly node 1: ```ruby - git_data_dirs({ - "gitaly-1" => { - "path" => "/var/opt/gitlab/git-data" - } - }) + gitaly['configuration'] = { + # ... + storage: [ + { + name: 'gitaly-1', + path: '/var/opt/gitlab/git-data', + }, + ], + } ``` - On Gitaly node 2: ```ruby - git_data_dirs({ - "gitaly-2" => { - "path" => "/var/opt/gitlab/git-data" - } - }) + gitaly['configuration'] = { + # ... + storage: [ + { + name: 'gitaly-2', + path: '/var/opt/gitlab/git-data', + }, + ], + } ``` - On Gitaly node 3: ```ruby - git_data_dirs({ - "gitaly-3" => { - "path" => "/var/opt/gitlab/git-data" - } - }) + gitaly['configuration'] = { + # ... + storage: [ + { + name: 'gitaly-3', + path: '/var/opt/gitlab/git-data', + }, + ], + } ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace @@ -1650,7 +1696,7 @@ for secure connections, you must: Additionally the certificate, or its certificate authority, must be installed on all Gitaly servers and on all Praefect clients that communicate with it following the procedure described in -[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) (and repeated below). +[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl/index.html#install-custom-public-certificates) (and repeated below). Note the following: @@ -1659,7 +1705,7 @@ Note the following: - You can configure Praefect servers with both an unencrypted listening address `listen_addr` and an encrypted listening address `tls_listen_addr` at the same time. This allows you to do a gradual transition from unencrypted to encrypted traffic, if - necessary. To disable the unencrypted listener, set `praefect['listen_addr'] = nil`. + necessary. To disable the unencrypted listener, set `praefect['configuration'][:listen_addr] = nil`. - The Internal Load Balancer will also access to the certificates and need to be configured to allow for TLS passthrough. Refer to the load balancers documentation on how to configure this. @@ -1681,9 +1727,15 @@ To configure Praefect with TLS: 1. Edit `/etc/gitlab/gitlab.rb` and add: ```ruby - praefect['tls_listen_addr'] = "0.0.0.0:3305" - praefect['certificate_path'] = "/etc/gitlab/ssl/cert.pem" - praefect['key_path'] = "/etc/gitlab/ssl/key.pem" + praefect['configuration'] = { + # ... + tls_listen_addr: '0.0.0.0:3305', + tls: { + # ... + certificate_path: '/etc/gitlab/ssl/cert.pem', + key_path: '/etc/gitlab/ssl/key.pem', + }, + } ``` 1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -1764,7 +1816,7 @@ Updates to example must be made at: # Redis ## Redis connection details - ## First cluster that will host the cache + ## First cluster that will host the cache data gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ @@ -1773,22 +1825,11 @@ Updates to example must be made at: {host: '10.6.0.53', port: 26379}, ] - ## Second cluster that will host the persistent queues, shared state, and actioncable - gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' - gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' - gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' + ## Second cluster that hosts all other persistent data + redis['master_name'] = 'gitlab-redis-persistent' + redis['master_password'] = '<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>' - gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.61', port: 26379}, - {host: '10.6.0.62', port: 26379}, - {host: '10.6.0.63', port: 26379}, - ] - gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.61', port: 26379}, - {host: '10.6.0.62', port: 26379}, - {host: '10.6.0.63', port: 26379}, - ] - gitlab_rails['redis_actioncable_sentinels'] = [ + gitlab_rails['redis_sentinels'] = [ {host: '10.6.0.61', port: 26379}, {host: '10.6.0.62', port: 26379}, {host: '10.6.0.63', port: 26379}, @@ -1958,7 +1999,7 @@ On each node perform the following: gitlab_rails['auto_migrate'] = false ## Redis connection details - ## First cluster that will host the cache + ## First cluster that will host the cache data gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ @@ -1967,22 +2008,11 @@ On each node perform the following: {host: '10.6.0.53', port: 26379}, ] - ## Second cluster that will host the persistent queues, shared state, and actionable - gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' - gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' - gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' + ## Second cluster that hosts all other persistent data + redis['master_name'] = 'gitlab-redis-persistent' + redis['master_password'] = '<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>' - gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.61', port: 26379}, - {host: '10.6.0.62', port: 26379}, - {host: '10.6.0.63', port: 26379}, - ] - gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.61', port: 26379}, - {host: '10.6.0.62', port: 26379}, - {host: '10.6.0.63', port: 26379}, - ] - gitlab_rails['redis_actioncable_sentinels'] = [ + gitlab_rails['redis_sentinels'] = [ {host: '10.6.0.61', port: 26379}, {host: '10.6.0.62', port: 26379}, {host: '10.6.0.63', port: 26379}, @@ -2076,7 +2106,7 @@ On each node perform the following: When you specify `https` in the `external_url`, as in the previous example, GitLab expects that the SSL certificates are in `/etc/gitlab/ssl/`. If the certificates aren't present, NGINX will fail to start. For more information, see -the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html). +the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html). ### GitLab Rails post-configuration @@ -2088,11 +2118,9 @@ the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html). sudo gitlab-rake gitlab:db:configure ``` - If you encounter a `rake aborted!` error message stating that PgBouncer is - failing to connect to PostgreSQL, it may be that your PgBouncer node's IP - address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` - on your database nodes. Before proceeding, see - [PgBouncer error `ERROR: pgbouncer cannot connect to server`](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). + Note that this requires the Rails node to be configured to connect to the primary database + directly, [bypassing PgBouncer](../postgresql/pgbouncer.md#procedure-for-bypassing-pgbouncer). + After migrations have completed, you must configure the node to pass through PgBouncer again. 1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). @@ -2187,15 +2215,15 @@ and scalable. There are two ways of specifying object storage configuration in GitLab: -- [Consolidated form](../object_storage.md#consolidated-object-storage-configuration): A single credential is +- [Consolidated form](../object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form): A single credential is shared by all supported object types. -- [Storage-specific form](../object_storage.md#storage-specific-configuration): Every object defines its - own object storage [connection and configuration](../object_storage.md#connection-settings). +- [Storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form): Every object defines its + own object storage [connection and configuration](../object_storage.md#configure-the-connection-settings). The consolidated form is used in the following examples when available. NOTE: -When using the [storage-specific form](../object_storage.md#storage-specific-configuration) +When using the [storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form) in GitLab 14.x and earlier, you should enable [direct upload mode](../../development/uploads/index.md#direct-upload). The previous [background upload](../../development/uploads/index.md#direct-upload) mode, which was deprecated in 14.9, requires shared storage such as NFS. @@ -2217,9 +2245,9 @@ GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs. -## Configure Advanced Search +## Configure advanced search -You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md) +You can leverage Elasticsearch and [enable advanced search](../../integration/advanced_search/elasticsearch.md) for faster, more advanced code search across your entire GitLab instance. Elasticsearch cluster design and requirements are dependent on your specific diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 691f71289c3..82087c91910 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -37,8 +37,8 @@ costly-to-operate environment by using the | Gitaly<sup>5 6</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | | Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | -| GitLab Rails | 3 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | `c5.4xlarge` | +| Sidekiq<sup>7</sup> | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | +| GitLab Rails<sup>7</sup> | 3 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | `c5.4xlarge` | | Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Object storage<sup>4</sup> | - | - | - | - | @@ -59,6 +59,8 @@ costly-to-operate environment by using the 6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to [Large repositories](index.md#large-repositories) for more information. +7. Can be placed in Auto Scaling Groups (ASGs) as the component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes). + However, for GitLab Rails certain processes like [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md) should be run on only one node. <!-- markdownlint-enable MD029 --> NOTE: @@ -173,7 +175,7 @@ To set up GitLab and its components to accommodate up to 5,000 users: environment. 1. [Configure the object storage](#configure-the-object-storage) used for shared data objects. -1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, +1. [Configure advanced search](#configure-advanced-search) (optional) for faster, more advanced code search across your entire GitLab instance. The servers start on the same 10.6.0.0/24 private network range, and can @@ -318,7 +320,7 @@ Configure your load balancer to pass connections on port 443 as `TCP` rather than `HTTP(S)` protocol. This will pass the connection to the application node's NGINX service untouched. NGINX will have the SSL certificate and listen on port 443. -See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) +See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html) for details on managing SSL certificates and configuring NGINX. #### Load balancer terminates SSL without backend SSL @@ -329,7 +331,7 @@ terminating SSL. Since communication between the load balancer and GitLab will not be secure, there is some additional configuration needed. See the -[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination) +[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination) for details. #### Load balancer terminates SSL with backend SSL @@ -342,7 +344,7 @@ Traffic will also be secure between the load balancers and NGINX in this scenario. There is no need to add configuration for proxied SSL since the connection will be secure all the way. However, configuration will need to be added to GitLab to configure SSL certificates. See -the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) +the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html) for details on managing SSL certificates and configuring NGINX. <div align="right"> @@ -450,10 +452,7 @@ to be used with GitLab. The following IPs are used as an example: Managed Redis from cloud providers such as AWS ElastiCache works. If these services support high availability, be sure it is **not** the Redis Cluster type. -Redis version 5.0 or higher is required, as this is what ships with -Omnibus GitLab packages starting with GitLab 13.0. Older Redis versions -do not support an optional count argument to SPOP which is now required for -[Merge Trains](../../ci/pipelines/merge_trains.md). +Because Omnibus GitLab packages ship with Redis 6.0 or later, Redis 6.0 or later is required. Older Redis versions have reached end-of-life. Note the Redis node's IP address or hostname, port, and password (if required). These are necessary when configuring the @@ -637,8 +636,13 @@ are supported and can be added if needed. ## Configure Consul and Sentinel -Now that the Redis servers are all set up, let's configure the Sentinel -servers. The following IPs are used as an example: +Now that the Redis servers are all set up, let's configure the Consul + Sentinel +servers. + +NOTE: +Consul and Redis Sentinel must be deployed in an odd number of 3 nodes or later. This is to ensure the nodes can take votes as part of a quorum. + +The following IPs will be used as an example: - `10.6.0.11`: Consul/Sentinel 1 - `10.6.0.12`: Consul/Sentinel 2 @@ -1140,7 +1144,7 @@ For more advanced setups refer to the [standalone Gitaly Cluster documentation]( Praefect, the routing and transaction manager for Gitaly Cluster, requires its own database server to store data on Gitaly Cluster status. If you want to have a highly available setup, Praefect requires a third-party PostgreSQL database. -A built-in solution is being [worked on](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919). +A built-in solution is being [worked on](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7292). #### Praefect non-HA PostgreSQL standalone using Omnibus GitLab @@ -1285,6 +1289,9 @@ This is how this would work with a Omnibus GitLab PostgreSQL setup: Praefect is the router and transaction manager for Gitaly Cluster and all connections to Gitaly go through it. This section details how to configure it. +NOTE: +Praefect must be deployed in an odd number of 3 nodes or later. This is to ensure the nodes can take votes as part of a quorum. + Praefect requires several secret tokens to secure communications across the Cluster: - `<praefect_external_token>`: Used for repositories hosted on your Gitaly cluster and can only be accessed by Gitaly clients that carry this token. @@ -1338,7 +1345,6 @@ Updates to example must be made at: # Praefect Configuration praefect['enable'] = true - praefect['listen_addr'] = '0.0.0.0:2305' # Prevent database migrations from running on upgrade automatically praefect['auto_migrate'] = false @@ -1347,51 +1353,69 @@ Updates to example must be made at: # Configure the Consul agent consul['enable'] = true ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true + consul['monitoring_service_discovery'] = true # START user configuration # Please set the real values as explained in Required Information section # - # Praefect External Token - # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster - praefect['auth_token'] = '<praefect_external_token>' - - # Praefect Database Settings - praefect['database_host'] = '10.6.0.141' - praefect['database_port'] = 5432 - # `no_proxy` settings must always be a direct connection for caching - praefect['database_direct_host'] = '10.6.0.141' - praefect['database_direct_port'] = 5432 - praefect['database_dbname'] = 'praefect_production' - praefect['database_user'] = 'praefect' - praefect['database_password'] = '<praefect_postgresql_password>' - - # Praefect Virtual Storage config - # Name of storage hash must match storage name in git_data_dirs on GitLab - # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1') - praefect['virtual_storages'] = { - 'default' => { - 'nodes' => { - 'gitaly-1' => { - 'address' => 'tcp://10.6.0.91:8075', - 'token' => '<praefect_internal_token>' - }, - 'gitaly-2' => { - 'address' => 'tcp://10.6.0.92:8075', - 'token' => '<praefect_internal_token>' - }, - 'gitaly-3' => { - 'address' => 'tcp://10.6.0.93:8075', - 'token' => '<praefect_internal_token>' + praefect['configuration'] = { + # ... + listen_addr: '0.0.0.0:2305', + auth: { + # ... + # + # Praefect External Token + # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster + token: '<praefect_external_token>', + }, + # Praefect Database Settings + database: { + # ... + host: '10.6.0.141', + port: 5432, + # `no_proxy` settings must always be a direct connection for caching + session_pooled: { + # ... + host: '10.6.0.141', + port: 5432, + dbname: 'praefect_production', + user: 'praefect', + password: '<praefect_postgresql_password>', + }, + }, + # Praefect Virtual Storage config + # Name of storage hash must match storage name in git_data_dirs on GitLab + # server ('praefect') and in gitaly['configuration'][:storage] on Gitaly nodes ('gitaly-1') + virtual_storage: [ + { + # ... + name: 'default', + node: [ + { + storage: 'gitaly-1', + address: 'tcp://10.6.0.91:8075', + token: '<praefect_internal_token>' + }, + { + storage: 'gitaly-2', + address: 'tcp://10.6.0.92:8075', + token: '<praefect_internal_token>' + }, + { + storage: 'gitaly-3', + address: 'tcp://10.6.0.93:8075', + token: '<praefect_internal_token>' + }, + ], }, - } - } + ], + # Set the network address Praefect will listen on for monitoring + prometheus_listen_addr: '0.0.0.0:9652', } - # Set the network addresses that the exporters will listen on for monitoring + # Set the network address the node exporter will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' - praefect['prometheus_listen_addr'] = '0.0.0.0:9652' ## The IPs of the Consul server nodes ## You can also use FQDNs and intermix them with IPs @@ -1442,9 +1466,7 @@ Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs should have a throughput of at least 8,000 input/output operations per second (IOPS) for read operations and 2,000 IOPS for -write operations. These IOPS values are initial recommendations, and may be -adjusted to greater or lesser values depending on the scale of your -environment's workload. If you're running the environment on a Cloud provider, +write operations. If you're running the environment on a Cloud provider, refer to their documentation about how to configure IOPS correctly. Gitaly servers must not be exposed to the public internet, as Gitaly's network @@ -1454,7 +1476,7 @@ to restrict access to the Gitaly server. Another option is to For configuring Gitaly you should note the following: -- `git_data_dirs` should be configured to reflect the storage path for the specific Gitaly node +- `gitaly['configuration'][:storage]` should be configured to reflect the storage path for the specific Gitaly node - `auth_token` should be the same as `praefect_internal_token` The following IPs are used as an example: @@ -1503,20 +1525,6 @@ Updates to example must be made at: # Gitaly gitaly['enable'] = true - # Make Gitaly accept connections on all network interfaces. You must use - # firewalls to restrict access to this address/port. - # Comment out following line if you only want to support TLS connections - gitaly['listen_addr'] = "0.0.0.0:8075" - - # Gitaly Auth Token - # Should be the same as praefect_internal_token - gitaly['auth_token'] = '<praefect_internal_token>' - - # Gitaly Pack-objects cache - # Recommended to be enabled for improved performance but can notably increase disk I/O - # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info - gitaly['pack_objects_cache_enabled'] = true - # Configure the Consul agent consul['enable'] = true ## Enable service discovery for Prometheus @@ -1531,9 +1539,29 @@ Updates to example must be made at: retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), } - # Set the network addresses that the exporters will listen on for monitoring + # Set the network address that the node exporter will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' - gitaly['prometheus_listen_addr'] = '0.0.0.0:9236' + + gitaly['configuration'] = { + # Make Gitaly accept connections on all network interfaces. You must use + # firewalls to restrict access to this address/port. + # Comment out following line if you only want to support TLS connections + listen_addr: '0.0.0.0:8075', + # Set the network address that Gitaly will listen on for monitoring + prometheus_listen_addr: '0.0.0.0:9236', + auth: { + # Gitaly Auth Token + # Should be the same as praefect_internal_token + token: '<praefect_internal_token>', + }, + pack_objects_cache: { + # Gitaly Pack-objects cache + # Recommended to be enabled for improved performance but can notably increase disk I/O + # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info + enabled: true, + }, + } + # # END user configuration ``` @@ -1542,31 +1570,43 @@ Updates to example must be made at: - On Gitaly node 1: ```ruby - git_data_dirs({ - "gitaly-1" => { - "path" => "/var/opt/gitlab/git-data" - } - }) + gitaly['configuration'] = { + # ... + storage: [ + { + name: 'gitaly-1', + path: '/var/opt/gitlab/git-data', + }, + ], + } ``` - On Gitaly node 2: ```ruby - git_data_dirs({ - "gitaly-2" => { - "path" => "/var/opt/gitlab/git-data" - } - }) + gitaly['configuration'] = { + # ... + storage: [ + { + name: 'gitaly-2', + path: '/var/opt/gitlab/git-data', + }, + ], + } ``` - On Gitaly node 3: ```ruby - git_data_dirs({ - "gitaly-3" => { - "path" => "/var/opt/gitlab/git-data" - } - }) + gitaly['configuration'] = { + # ... + storage: [ + { + name: 'gitaly-3', + path: '/var/opt/gitlab/git-data', + }, + ], + } ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace @@ -1586,7 +1626,7 @@ for secure connections, you must: Additionally the certificate, or its certificate authority, must be installed on all Gitaly servers and on all Praefect clients that communicate with it following the procedure described in -[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) (and repeated below). +[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl/index.html#install-custom-public-certificates) (and repeated below). Note the following: @@ -1595,7 +1635,7 @@ Note the following: - You can configure Praefect servers with both an unencrypted listening address `listen_addr` and an encrypted listening address `tls_listen_addr` at the same time. This allows you to do a gradual transition from unencrypted to encrypted traffic, if - necessary. To disable the unencrypted listener, set `praefect['listen_addr'] = nil`. + necessary. To disable the unencrypted listener, set `praefect['configuration'][:listen_addr] = nil`. - The Internal Load Balancer will also access to the certificates and need to be configured to allow for TLS passthrough. Refer to the load balancers documentation on how to configure this. @@ -1617,9 +1657,15 @@ To configure Praefect with TLS: 1. Edit `/etc/gitlab/gitlab.rb` and add: ```ruby - praefect['tls_listen_addr'] = "0.0.0.0:3305" - praefect['certificate_path'] = "/etc/gitlab/ssl/cert.pem" - praefect['key_path'] = "/etc/gitlab/ssl/key.pem" + praefect['configuration'] = { + # ... + tls_listen_addr: '0.0.0.0:3305', + tls: { + # ... + certificate_path: '/etc/gitlab/ssl/cert.pem', + key_path: '/etc/gitlab/ssl/key.pem', + }, + } ``` 1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -2017,7 +2063,7 @@ On each node perform the following: When you specify `https` in the `external_url`, as in the previous example, GitLab expects that the SSL certificates are in `/etc/gitlab/ssl/`. If the certificates aren't present, NGINX fails to start. For more information, see -the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html). +the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html). ### GitLab Rails post-configuration @@ -2027,11 +2073,9 @@ the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html). gitlab-rake gitlab:db:configure ``` - If you encounter a `rake aborted!` error message stating that PgBouncer is - failing to connect to PostgreSQL, it may be that your PgBouncer node's IP - address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` - on your database nodes. Before proceeding, see - [PgBouncer error `ERROR: pgbouncer cannot connect to server`](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). + Note that this requires the Rails node to be configured to connect to the primary database + directly, [bypassing PgBouncer](../postgresql/pgbouncer.md#procedure-for-bypassing-pgbouncer). + After migrations have completed, you must configure the node to pass through PgBouncer again. 1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). @@ -2134,15 +2178,15 @@ and scalable. There are two ways of specifying object storage configuration in GitLab: -- [Consolidated form](../object_storage.md#consolidated-object-storage-configuration): A single credential is +- [Consolidated form](../object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form): A single credential is shared by all supported object types. -- [Storage-specific form](../object_storage.md#storage-specific-configuration): Every object defines its - own object storage [connection and configuration](../object_storage.md#connection-settings). +- [Storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form): Every object defines its + own object storage [connection and configuration](../object_storage.md#configure-the-connection-settings). The consolidated form is used in the following examples when available. NOTE: -When using the [storage-specific form](../object_storage.md#storage-specific-configuration) +When using the [storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form) in GitLab 14.x and earlier, you should enable [direct upload mode](../../development/uploads/index.md#direct-upload). The previous [background upload](../../development/uploads/index.md#direct-upload) mode, which was deprecated in 14.9, requires shared storage such as NFS. @@ -2164,9 +2208,9 @@ GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs. -## Configure Advanced Search +## Configure advanced search -You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md) +You can leverage Elasticsearch and [enable advanced search](../../integration/advanced_search/elasticsearch.md) for faster, more advanced code search across your entire GitLab instance. Elasticsearch cluster design and requirements are dependent on your specific diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index 7b01efa183b..4b9c26e8626 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -47,9 +47,9 @@ The following Cloud Native Hybrid reference architectures, where select recommen The first choice to consider is whether a Self Managed approach is correct for you and your requirements. -Running any application in production is complex, and the same applies for GitLab. While we aim to make this as smooth as possible, there are still the general complexities. This depends on the design chosen, but typically you'll need to manage all aspects such as hardware, operating systems, networking, storage, security, GitLab itself, and more. +Running any application in production is complex, and the same applies for GitLab. While we aim to make this as smooth as possible, there are still the general complexities. This depends on the design chosen, but typically you'll need to manage all aspects such as hardware, operating systems, networking, storage, security, GitLab itself, and more. This includes both the initial setup of the environment and the longer term maintenance. -As such, it's recommended that you have a working knowledge of running applications in production when deciding on going down this route. For users who want a more managed solution it's recommended to instead explore our other offerings such as [GitLab SaaS](../../subscriptions/gitlab_com/index.md) or [GitLab Dedicated](../../subscriptions/gitlab_dedicated/index.md). +As such, it's recommended that you have a working knowledge of running and maintaining applications in production when deciding on going down this route. If you aren't in this position, our [Professional Services](https://about.gitlab.com/services/#implementation-services) team offers implementation services, but for those who want a more managed solution long term, it's recommended to instead explore our other offerings such as [GitLab SaaS](../../subscriptions/gitlab_com/index.md) or [GitLab Dedicated](../../subscriptions/gitlab_dedicated/index.md). ## Deciding which architecture to use @@ -72,15 +72,13 @@ With standalone setups, especially single node environments, there are [various ### High Availability (HA) -High Availability ensures every component in the GitLab setup can handle failures through various mechanisms. To achieve this however is complex, and the environments required can be sizable. +High Availability ensures every component in the GitLab setup can handle failures through various mechanisms. However, to achieve this is complex and the environments required can be sizable. For environments serving 3,000 or more users we generally recommend that a HA strategy is used as at this level outages have a bigger impact against more users. All the architectures in this range have HA built in by design for this reason. -For users who still need to have HA for a lower number of users this can also be achieved with an adjusted [3K architecture](3k_users.md#supported-modifications-for-lower-user-counts-ha). - #### Do you need High Availability (HA)? -As mentioned above, achieving HA does come at a cost. The environment's required are sizable as each component needs to be multiplied, which comes with additional actual and maintenance costs. +As mentioned above, achieving HA does come at a cost. The environment requirements are sizable as each component needs to be multiplied, which comes with additional actual and maintenance costs. For a lot of our customers with fewer than 3,000 users, we've found a backup strategy is sufficient and even preferable. While this does have a slower recovery time, it also means you have a much smaller architecture and less maintenance costs as a result. @@ -89,13 +87,17 @@ In general then, we'd only recommend you employ HA in the following scenarios: - When you have 3,000 or more users. - When GitLab being down would critically impact your workflow. +#### Scaled-down High Availability (HA) approaches + +If you still need to have HA for a lower number of users, this can be achieved with an adjusted [3K architecture](3k_users.md#supported-modifications-for-lower-user-counts-ha). + #### Zero Downtime Upgrades [Zero Downtime Upgrades](../../update/zero_downtime.md) are available for standard Reference Architecture environments with HA (Cloud Native Hybrid is not supported at this time). This allows for an environment to stay up during an upgrade, but the process is more complex as a result and has some limitations as detailed in the documentation. -When going through this process it's worth noting that there may still be brief moments of downtime when the HA mechanisms tale effect. +When going through this process it's worth noting that there may still be brief moments of downtime when the HA mechanisms take effect. -In most cases the downtime required for doing an upgrade in general shouldn't be substantial, so this is only recommended if it's a key requirement for you. +In most cases the downtime required for doing an upgrade shouldn't be substantial, so this is only recommended if it's a key requirement for you. ### Cloud Native Hybrid (Kubernetes HA) @@ -163,7 +165,7 @@ Before implementing a reference architecture, refer to the following requirement These reference architectures were built and tested on Google Cloud Platform (GCP) using the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) -CPU platform as a baseline ([Sysbench benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks)). +CPU platform as a lowest common denominator baseline ([Sysbench benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks)). Newer, similarly-sized CPUs are supported and may have improved performance as a result. For Omnibus GitLab environments, ARM-based equivalents are also supported. @@ -237,10 +239,10 @@ for more information and guidance. [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and that to achieve full High Availability, a third-party PostgreSQL database solution is required. -We hope to offer a built in solutions for these restrictions in the future but, in the meantime, a non HA PostgreSQL server -can be set up using Omnibus GitLab, the specifications reflect. Refer to the following issues for more information: +We hope to offer a built-in solution for these restrictions in the future. In the meantime, a non-HA PostgreSQL server +can be set up using Omnibus GitLab as the specifications reflect. Refer to the following issues for more information: -- [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919). +- [`omnibus-gitlab#7292`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7292). - [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398). ## Recommended cloud providers and services @@ -324,10 +326,13 @@ When selecting a database service, it should run a standard, performant, and [su - Read Replicas for [Database Load Balancing](../postgresql/database_load_balancing.md). - Cross Region replication for [GitLab Geo](../geo/index.md). -Several cloud provider services are known not to support the above or have been found to have other issues and aren't recommended: +#### Unsupported database services + +Several database cloud provider services are known not to support the above or have been found to have other issues and aren't recommended: - [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is incompatible and not supported. See [14.4.0](../../update/index.md#1440) for more details. -- [Azure Database for PostgreSQL Single Server](https://azure.microsoft.com/en-gb/products/postgresql/#overview) (Single / Flexible) is **strongly not recommended** for use due to notable performance / stability issues or missing functionality. See [Recommendation Notes for Azure](#recommendation-notes-for-azure) for more details. +- [Azure Database for PostgreSQL Single Server](https://azure.microsoft.com/en-gb/products/postgresql/#overview) (Single / Flexible) is not supported for use due to notable performance / stability issues or missing functionality. See [Recommendation Notes for Azure](#recommendation-notes-for-azure) for more details. +- Azure Database for PostgreSQL Flexible Server uses Microsoft Azure Active Directory (Azure AD) as authentication mechanism, which is incompatible with GitLab database integration. - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo. - [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported. @@ -337,10 +342,55 @@ Due to performance issues that we found with several key Azure services, we only In addition to the above, you should be aware of the additional specific guidance for Azure: -- **We outright strongly do not recommend [Azure Database for PostgreSQL Single Server](https://learn.microsoft.com/en-us/azure/postgresql/single-server/overview-single-server)** specifically due to significant performance and stability issues found. **For GitLab 14.0 and higher the service is not supported** due to it only supporting up to PostgreSQL 11. - - A new service, [Azure Database for PostgreSQL Flexible Server](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/) has been released. [Internal testing](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/91) has shown that it does look to perform as expected, but this hasn't been validated in production, so generally isn't recommended at this time. Additionally, as it's a new service, you may find that it's missing some functionality depending on your requirements. +- [Azure Database for PostgreSQL Single Server](https://azure.microsoft.com/en-gb/products/postgresql/#overview) (Single / Flexible) is not supported for use due to notable performance / stability issues or missing functionality. +- A new service, [Azure Database for PostgreSQL Flexible Server](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/) has been released. [Internal testing](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/91) has shown that it does look to perform as expected, but this hasn't been validated in production, so generally isn't recommended at this time. Additionally, as it's a new service, you may find that it's missing some functionality depending on your requirements. - [Azure Blob Storage](https://azure.microsoft.com/en-gb/products/storage/blobs/) has been found to have [performance limits that can impact production use at certain times](https://gitlab.com/gitlab-org/gitlab/-/issues/344861). However, this has only been seen in our largest architectures (25k+) so far. +## Deviating from the suggested reference architectures + +As a general guideline, the further away you move from the reference architectures, +the harder it is to get support for it. With any deviation, you're introducing +a layer of complexity that adds challenges to finding out where potential +issues might lie. + +The reference architectures use the official GitLab Linux packages (Omnibus +GitLab) or [Helm Charts](https://docs.gitlab.com/charts/) to install and configure the various components. The components are +installed on separate machines (virtualized or bare metal), with machine hardware +requirements listed in the "Configuration" column and equivalent VM standard sizes listed +in GCP/AWS/Azure columns of each [available reference architecture](#available-reference-architectures). + +Running components on Docker (including Docker Compose) with the same specs should be fine, as Docker is well known in terms of support. +However, it is still an additional layer and may still add some support complexities, such as not being able to run `strace` easily in containers. + +### Unsupported designs + +While we endeavour to try and have a good range of support for GitLab environment designs, there are certain approaches we know definitively not to work, and as a result are not supported. Those approaches are detailed in the following sections. + +#### Stateful components in Kubernetes + +[Running stateful components in Kubernetes, such as Gitaly Cluster, is not supported](https://docs.gitlab.com/charts/installation/#configure-the-helm-chart-to-use-external-stateful-data). + +Gitaly Cluster is only supported to be run on VMs as Git itself doesn't match well with the Kubernetes design and attempting to run it can lead to significant and complex issues. +[Refer to epic 6127 for more information](https://gitlab.com/groups/gitlab-org/-/epics/6127). + +This also applies to other third-party stateful components such as Postgres and Redis, but you can explore other third-party solutions for those components if desired such as supported Cloud Provider services unless called out specifically as unsupported. + +#### Autoscaling of stateful nodes + +As a general guidance, only _stateless_ components of GitLab can be run in Autoscaling groups, namely GitLab Rails +and Sidekiq. + +Other components that have state, such as Gitaly, are not supported in this fashion (for more information, see [issue 2997](https://gitlab.com/gitlab-org/gitaly/-/issues/2997)). + +This also applies to other third-party stateful components such as Postgres and Redis, but you can explore other third-party solutions for those components if desired such as supported Cloud Provider services unless called out specifically as unsupported. + +#### Spreading one environment over multiple data centers + +Deploying one GitLab environment over multiple data centers is not supported due to potential split brain edge cases +if a data center were to go down. For example, several components of the GitLab setup, namely Consul, Redis Sentinel and Praefect require an odd number quorum to function correctly and splitting over multiple data centers can impact this notably. + +For deploying GitLab over multiple data centers or regions we offer [GitLab Geo](https://about.gitlab.com/solutions/geo/) as a comprehensive solution. + ## Validation and test results The [Quality Engineering team](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/) @@ -427,34 +477,34 @@ table.test-coverage th { <tr> <th scope="row">1k</th> <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/1k">Weekly</a></td> - <td></td> - <td></td> - <td></td> - <td></td> + <td style="background-color:lightgrey"></td> + <td style="background-color:lightgrey"></td> + <td style="background-color:lightgrey"></td> + <td style="background-color:lightgrey"></td> </tr> <tr> <th scope="row">2k</th> <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/2k">Weekly</a></td> - <td></td> - <td></td> - <td></td> - <td></td> + <td style="background-color:lightgrey"></td> + <td style="background-color:lightgrey"></td> + <td style="background-color:lightgrey"></td> + <td><i>Planned</i></td> </tr> <tr> <th scope="row">3k</th> <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/3k">Weekly</a></td> - <td></td> - <td></td> + <td style="background-color:lightgrey"></td> + <td style="background-color:lightgrey"></td> <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/3k_hybrid_aws_services">Weekly</a></td> - <td></td> + <td style="background-color:lightgrey"></td> </tr> <tr> <th scope="row">5k</th> <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/5k">Weekly</a></td> - <td></td> - <td></td> - <td></td> - <td></td> + <td style="background-color:lightgrey"></td> + <td style="background-color:lightgrey"></td> + <td style="background-color:lightgrey"></td> + <td style="background-color:lightgrey"></td> </tr> <tr> <th scope="row">10k</th> @@ -462,29 +512,33 @@ table.test-coverage th { <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k_hybrid">Weekly</a></td> <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k_aws">Weekly</a></td> <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k_hybrid_aws_services">Weekly</a></td> - <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/10k">Ad-Hoc</a></td> + <td style="background-color:lightgrey"></td> </tr> <tr> <th scope="row">25k</th> <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/25k">Weekly</a></td> - <td></td> - <td></td> - <td></td> - <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/25k">Ad-Hoc</a></td> + <td style="background-color:lightgrey"></td> + <td style="background-color:lightgrey"></td> + <td style="background-color:lightgrey"></td> + <td style="background-color:lightgrey"></td> </tr> <tr> <th scope="row">50k</th> <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/50k">Weekly</a></td> - <td></td> - <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/50k">Ad-Hoc (inc Cloud Services)</a></td> - <td></td> - <td></td> + <td style="background-color:lightgrey"></td> + <td style="background-color:lightgrey"></td> + <td style="background-color:lightgrey"></td> + <td style="background-color:lightgrey"></td> </tr> </table> ## Cost to run -The following table details the cost to run the different reference architectures across GCP, AWS, and Azure. Bare-metal costs are not included here as it varies widely depending on each customer configuration. +As a starting point, the following table details initial costs for the different reference architectures across GCP, AWS, and Azure via Omnibus. + +NOTE: +Due to the nature of Cloud Native Hybrid, it's not possible to give a static cost calculation. +Bare-metal costs are also not included here as it varies widely depending on each configuration. <table class="test-coverage"> <col> @@ -492,91 +546,93 @@ The following table details the cost to run the different reference architecture <colgroup span="2"></colgroup> <tr> <th rowspan="2">Reference<br/>Architecture</th> - <th style="text-align: center" colspan="2" scope="colgroup">GCP</th> - <th style="text-align: center" colspan="2" scope="colgroup">AWS</th> - <th style="text-align: center" colspan="2" scope="colgroup">Azure</th> + <th style="text-align: center" scope="colgroup">GCP</th> + <th style="text-align: center" scope="colgroup">AWS</th> + <th style="text-align: center" scope="colgroup">Azure</th> </tr> <tr> <th scope="col">Omnibus</th> - <th scope="col">Cloud Native Hybrid</th> <th scope="col">Omnibus</th> - <th scope="col">Cloud Native Hybrid</th> <th scope="col">Omnibus</th> </tr> <tr> <th scope="row">1k</th> <td><a href="https://cloud.google.com/products/calculator#id=a6d6a94a-c7dc-4c22-85c4-7c5747f272ed">Calculated cost</a></td> - <td></td> <td><a href="https://calculator.aws/#/estimate?id=b51f178f4403b69a63f6eb33ea425f82de3bf249">Calculated cost</a></td> - <td></td> <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=1adf30bef7e34ceba9efa97c4470417b">Calculated cost</a></td> </tr> <tr> <th scope="row">2k</th> - <td><a href="https://cloud.google.com/products/calculator#id=84d11491-d72a-493c-a16e-650931faa658">Calculated cost</a></td> - <td></td> - <td><a href="https://calculator.aws/#/estimate?id=dce36b5cb6ab25211f74e47233d77f58fefb54e2">Calculated cost</a></td> - <td></td> - <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=72764902f3854f798407fb03c3de4b6f">Calculated cost</a></td> + <td><a href="https://cloud.google.com/products/calculator#id=0d3aff1f-ea3d-43f9-aa59-df49d27c35ca">Calculated cost</a></td> + <td><a href="https://calculator.aws/#/estimate?id=3b3e3b81953737132789591d3a5179521943f1c0">Calculated cost</a></td> + <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=25f66c35ba454bb98fb4034a8a50bb8c">Calculated cost</a></td> </tr> <tr> <th scope="row">3k</th> - <td><a href="https://cloud.google.com/products/calculator/#id=ac4838e6-9c40-4a36-ac43-6d1bc1843e08">Calculated cost</a></td> - <td></td> - <td><a href="https://calculator.aws/#/estimate?id=b1c5b4e32e990eaeb035a148255132bd28988760">Calculated cost</a></td> - <td></td> - <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=0dbfc575051943b9970e5d8ace03680d">Calculated cost</a></td> + <td><a href="https://cloud.google.com/products/calculator/#id=15fc2bd9-5b1c-479d-bc46-d5ce096b8107">Calculated cost</a></td> + <td><a href="https://calculator.aws/#/estimate?id=7e94eb8712f6845fdeb05e61f459598a91dac3cb">Calculated cost</a></td> + <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=24ac11fd947a4985ae9c9a5142649ad3">Calculated cost</a></td> </tr> <tr> <th scope="row">5k</th> - <td><a href="https://cloud.google.com/products/calculator/#id=8742e8ea-c08f-4e0a-b058-02f3a1c38a2f">Calculated cost</a></td> - <td></td> - <td><a href="https://calculator.aws/#/estimate?id=2bf1af883096e6f4c6efddb4f3c35febead7fec2">Calculated cost</a></td> - <td></td> - <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=8f618711ffec4b039f1581871ca6a7c9">Calculated cost</a></td> + <td><a href="https://cloud.google.com/products/calculator/#id=9a798136-53f2-4c35-be43-8e1e975a6663">Calculated cost</a></td> + <td><a href="https://calculator.aws/#/estimate?id=ad4c9db623a214c92d780cd9dff33f444d62cf02">Calculated cost</a></td> + <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=bcf23017ddfd40649fdc885cacd08d0c">Calculated cost</a></td> </tr> <tr> <th scope="row">10k</th> - <td><a href="https://cloud.google.com/products/calculator#id=e77713f6-dc0b-4bb3-bcef-cea904ac8efd">Calculated cost</a></td> - <td></td> - <td><a href="https://calculator.aws/#/estimate?id=1d374df13c0f2088d332ab0134f5b1d0f717259e">Calculated cost</a></td> - <td></td> - <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=de3da8286dda4d4db1362932bc75410b">Calculated cost</a></td> + <td><a href="https://cloud.google.com/products/calculator#id=cbe61840-31a1-487f-88fa-631251c2fde5">Calculated cost</a></td> + <td><a href="https://calculator.aws/#/estimate?id=3e2970f919915a6337acea76a9f07655a1ecda4a">Calculated cost</a></td> + <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=5748068be4864af6a34efb1cde685fa1">Calculated cost</a></td> </tr> <tr> <th scope="row">25k</th> - <td><a href="https://cloud.google.com/products/calculator#id=925386e1-c01c-4c0a-8d7d-ebde1824b7b0">Calculated cost</a></td> - <td></td> - <td><a href="https://calculator.aws/#/estimate?id=46fe6a6e9256d9b7779fae59fbbfa7e836942b7d">Calculated cost</a></td> - <td></td> - <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=69724ebd82914a60857da6a3ace05a64">Calculate cost</a></td> + <td><a href="https://cloud.google.com/products/calculator#id=b4b8b587-508a-4433-adc8-dc506bbe924f">Calculated cost</a></td> + <td><a href="https://calculator.aws/#/estimate?id=32acaeaa93366110cd5fbf98a66a8a141db7adcb">Calculated cost</a></td> + <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=24f878f20ee64b5cb64de459d34c8128">Calculate cost</a></td> </tr> <tr> <th scope="row">50k</th> - <td><a href="https://cloud.google.com/products/calculator/#id=8006396b-88ee-40cd-a1c8-77cdefa4d3c8">Calculated cost</a></td> - <td></td> - <td><a href="https://calculator.aws/#/estimate?id=e15926b1a3c7139e4faf390a3875ff807d2ab91c">Calculated cost</a></td> - <td></td> - <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=3f973040ebc14023933d35f576c89846">Calculated cost</a></td> + <td><a href="https://cloud.google.com/products/calculator/#id=48b4d817-d6cd-44b8-b069-0ba9a5d123ea">Calculated cost</a></td> + <td><a href="https://calculator.aws/#/estimate?id=5a0bba1338e3577d627ec97833dbc80ac9615562">Calculated cost</a></td> + <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=4dd065eea2194d70b44d6d897e81f460">Calculated cost</a></td> </tr> </table> -## Deviating from the suggested reference architectures +## Maintaining a Reference Architecture environment -As a general guideline, the further away you move from the Reference Architectures, -the harder it is to get support for it. With any deviation, you're introducing -a layer of complexity that adds challenges to finding out where potential -issues might lie. +Maintaining a Reference Architecture environment is generally the same as any other GitLab environment is generally covered in other sections of this documentation. -The reference architectures use the official GitLab Linux packages (Omnibus -GitLab) or [Helm Charts](https://docs.gitlab.com/charts/) to install and configure the various components. The components are -installed on separate machines (virtualized or bare metal), with machine hardware -requirements listed in the "Configuration" column and equivalent VM standard sizes listed -in GCP/AWS/Azure columns of each [available reference architecture](#available-reference-architectures). +In this section you'll find links to documentation for relevant areas as well as any specific Reference Architecture notes. -Running components on Docker (including Compose) with the same specs should be fine, as Docker is well known in terms of support. -However, it is still an additional layer and may still add some support complexities, such as not being able to run `strace` easily in containers. +### Upgrades + +Upgrades for a Reference Architecture environment is the same as any other GitLab environment. +The main [Upgrade GitLab](../../update/index.md) section has detailed steps on how to approach this. + +[Zero-downtime upgrades](#zero-downtime-upgrades) are also available. + +NOTE: +You should upgrade a Reference Architecture in the same order as you created it. + +### Scaling an environment + +Scaling a GitLab environment is designed to be as seamless as possible. + +In terms of the Reference Architectures, you would look to the next size and adjust accordingly. +Most setups would only need vertical scaling, but there are some specific areas that can be adjusted depending on the setup: + +- If you're scaling from a non-HA environment to an HA environment, various components are recommended to be deployed in their HA forms: + - Redis to multi-node Redis w/ Redis Sentinel + - Postgres to multi-node Postgres w/ Consul + PgBouncer + - Gitaly to Gitaly Cluster w/ Praefect +- From 10k users and higher, Redis is recommended to be split into multiple HA servers as it's single threaded. + +Conversely, if you have robust metrics in place that show the environment is over-provisioned you can apply the same process for +scaling downloads. It's recommended to take an iterative approach when scaling downwards however to ensure there are no issues. + +### How to monitor your environment -Other technologies, like [Docker swarm](https://docs.docker.com/engine/swarm/) -are not officially supported, but can be implemented at your own risk. In that -case, GitLab Support is not able to help you. +To monitor your GitLab environment, you can use the tools +[bundled with GitLab](../monitoring/index.md), but it's also possible to use third-party +options if desired. |