diff options
Diffstat (limited to 'doc/administration/reference_architectures/50k_users.md')
| -rw-r--r-- | doc/administration/reference_architectures/50k_users.md | 100 |
1 files changed, 42 insertions, 58 deletions
diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 7d3ddf7578c..9a2c354f27c 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -38,17 +38,11 @@ full list of reference architectures, see <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. - - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo. - - Note that [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported. - - [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440). - - Consul is primarily used for Omnibus PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. -2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Memorystore](https://cloud.google.com/memorystore) and [Amazon ElastiCache](https://aws.amazon.com/elasticache/) are known to work. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance) for more information. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Provide your own Redis instance](#provide-your-own-redis-instance) for more information. + - Redis is primarily single threaded. It's strongly recommended separating out the instances as specified into Cache and Persistent data to achieve optimum performance at this scale. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. -4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. +4. Should be run on reputable Cloud Provider or Self Managed solutions. See [Configure the object storage](#configure-the-object-storage) for more information. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. 6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to @@ -396,9 +390,9 @@ backend pgbouncer mode tcp option tcp-check - server pgbouncer1 10.6.0.21:6432 check - server pgbouncer2 10.6.0.22:6432 check - server pgbouncer3 10.6.0.23:6432 check + server pgbouncer1 10.6.0.31:6432 check + server pgbouncer2 10.6.0.32:6432 check + server pgbouncer3 10.6.0.33:6432 check backend praefect mode tcp @@ -471,7 +465,7 @@ To configure Consul: 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Go through the steps again for all the other Consul nodes, and make sure you set up the correct IPs. @@ -513,24 +507,21 @@ cluster to be used with GitLab. ### Provide your own PostgreSQL instance -If you're hosting GitLab on a cloud provider, you can optionally use a -managed service for PostgreSQL. +You can optionally use a [third party external service for PostgreSQL](../../administration/postgresql/external.md). -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default from [14.4.0](../../update/index.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. -If you use a cloud-managed service, or provide your own PostgreSQL: +If you use a third party external service: +1. Note that the HA Omnibus PostgreSQL setup encompasses PostgreSQL, PgBouncer and Consul. All of these components would no longer be required when using a third party external service. 1. Set up PostgreSQL according to the [database requirements document](../../install/requirements.md#database). 1. Set up a `gitlab` username with a password of your choice. The `gitlab` user needs privileges to create the `gitlabhq_production` database. 1. Configure the GitLab application servers with the appropriate details. This step is covered in [Configuring the GitLab Rails application](#configure-gitlab-rails). -1. For improved performance, configuring [Database Load Balancing](../postgresql/database_load_balancing.md) - with multiple read replicas is recommended. - -See [Configure GitLab using an external PostgreSQL service](../postgresql/external.md) for -further configuration steps. +1. The number of nodes required to achieve HA may differ depending on the service compared to Omnibus and doesn't need to match accordingly. +1. However, if [Database Load Balancing](../postgresql/database_load_balancing.md) via Read Replicas is desired for further improved performance it's recommended to follow the node count for the Reference Architecture. ### Standalone PostgreSQL using Omnibus GitLab @@ -675,7 +666,7 @@ For more information, see the various [Patroni replication methods](../postgresq 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/database.html) are supported and can be added if needed. @@ -764,7 +755,7 @@ The following IPs will be used as an example: 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. If an error `execute[generate databases.ini]` occurs, this is due to an existing [known issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4713). @@ -777,7 +768,7 @@ The following IPs will be used as an example: gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul ``` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) once again +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) once again to resolve any potential errors from the previous steps. 1. Ensure each node is talking to the current primary: @@ -821,6 +812,9 @@ start the failover procedure. NOTE: Redis clusters must each be deployed in an odd number of 3 nodes or more. This is to ensure Redis Sentinel can take votes as part of a quorum. This does not apply when configuring Redis externally, such as a cloud provider service. +NOTE: +Redis is primarily single threaded. It's strongly recommended separating out the instances as specified into Cache and Persistent data to achieve optimum performance at this scale. + Redis requires authentication if used with Sentinel. See [Redis Security](https://redis.io/docs/manual/security/) documentation for more information. We recommend using a combination of a Redis password and tight @@ -850,15 +844,11 @@ to be used with GitLab. The following IPs will be used as an example: - `10.6.0.62`: Redis - Persistent Replica 1 - `10.6.0.63`: Redis - Persistent Replica 2 -### Providing your own Redis instance +### Provide your own Redis instance -Managed Redis from cloud providers (such as AWS ElastiCache) will work. If these -services support high availability, be sure it _isn't_ of the Redis Cluster type. +You can optionally use a third party external service for Redis as long as it meets the [requirements](../../install/requirements.md#redis). -Because Omnibus GitLab packages ship with Redis 6.0 or later, Redis 6.0 or later is required. Older Redis versions have reached end-of-life. - -Note the Redis node's IP address or hostname, port, and password (if required). -These will be necessary later when configuring the [GitLab application servers](#configure-gitlab-rails). +A reputable provider or solution should be used for this. [Google Memorystore](https://cloud.google.com/memorystore/docs/redis/redis-overview) and [AWS Elasticache](https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/WhatIs.html) are known to work. However, note that the Redis Cluster mode specifically isn't supported by GitLab. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. ### Configure the Redis Cache cluster @@ -939,7 +929,7 @@ a node and change its status from primary to replica (and vice versa). 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. #### Configure the replica Redis Cache nodes @@ -1014,7 +1004,7 @@ a node and change its status from primary to replica (and vice versa). server. If this is the first Omnibus node you are configuring then you can skip this step. - 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes + 1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Go through the steps again for all the other replica nodes, and @@ -1068,7 +1058,7 @@ a node and change its status from primary to replica (and vice versa). #redis['master_port'] = 6379 # Set up password authentication for Redis and replicas (use the same password in all nodes). - redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' ## Must be the same in every Redis node @@ -1097,7 +1087,7 @@ a node and change its status from primary to replica (and vice versa). 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. #### Configure the replica Redis Persistent nodes @@ -1160,7 +1150,7 @@ a node and change its status from primary to replica (and vice versa). 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. @@ -1279,7 +1269,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Follow the [post configuration](#praefect-postgresql-post-configuration). @@ -1511,10 +1501,10 @@ the file of the same name on this server. If this is the first Omnibus node you sudo touch /etc/gitlab/skip-auto-reconfigure ``` - 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect and + 1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect and to run the Praefect database migrations. -1. On all other Praefect nodes, [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. On all other Praefect nodes, [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. ### Configure Gitaly @@ -1682,7 +1672,7 @@ Updates to example must be made at: 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). ### Gitaly Cluster TLS support @@ -1738,7 +1728,7 @@ To configure Praefect with TLS: } ``` -1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file and [reconfigure](../restart_gitlab.md#reconfigure-a-linux-package-installation). 1. On the Praefect clients (including each Gitaly server), copy the certificates, or their certificate authority, into `/etc/gitlab/trusted-certs`: @@ -1759,7 +1749,7 @@ To configure Praefect with TLS: }) ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -1916,7 +1906,7 @@ Updates to example must be made at: Only a single designated node should handle migrations as detailed in the [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. NOTE: If you find that the environment's Sidekiq job processing is slow with long queues, @@ -2085,7 +2075,7 @@ On each node perform the following: Only a single designated node should handle migrations as detailed in the [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. [Enable incremental logging](#enable-incremental-logging). 1. Confirm the node can connect to Gitaly: @@ -2196,7 +2186,7 @@ To configure the Monitoring node: nginx['enable'] = true ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). 1. In the GitLab UI, set `admin/application_settings/metrics_and_profiling` > Metrics - Grafana to `/-/grafana` to `http[s]://<MONITOR NODE>/-/grafana` @@ -2211,7 +2201,7 @@ To configure the Monitoring node: GitLab supports using an [object storage](../object_storage.md) service for holding numerous types of data. It's recommended over [NFS](../nfs.md) for data objects and in general it's better in larger setups as object storage is typically much more performant, reliable, -and scalable. +and scalable. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. There are two ways of specifying object storage configuration in GitLab: @@ -2326,17 +2316,11 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. - - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo. - - Note that [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported. - - [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440). - - Consul is primarily used for Omnibus PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. -2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Memorystore](https://cloud.google.com/memorystore) and [Amazon ElastiCache](https://aws.amazon.com/elasticache/) are known to work. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance) for more information. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Provide your own Redis instance](#provide-your-own-redis-instance) for more information. + - Redis is primarily single threaded. It's strongly recommended separating out the instances as specified into Cache and Persistent data to achieve optimum performance at this scale. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. -4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. +4. Should be run on reputable Cloud Provider or Self Managed solutions. See [Configure the object storage](#configure-the-object-storage) for more information. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. 6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to |