diff options
Diffstat (limited to 'doc/administration/reference_architectures/3k_users.md')
| -rw-r--r-- | doc/administration/reference_architectures/3k_users.md | 366 |
1 files changed, 189 insertions, 177 deletions
diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index d81f98a35d4..bca5e4c3dab 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -27,26 +27,31 @@ For a full list of reference architectures, see | Service | Nodes | Configuration | GCP | AWS | Azure | |--------------------------------------------|-------------|-----------------------|-----------------|--------------|----------| -| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Redis** | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | -| Consul* + Sentinel** | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| PostgreSQL* | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | -| PgBouncer* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| External load balancing node(3) | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Redis(2) | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | +| Consul(1) + Sentinel(2) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| PostgreSQL(1) | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | +| PgBouncer(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Internal load balancing node(3) | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Gitaly | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Praefect PostgreSQL* | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Praefect PostgreSQL(1) | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Sidekiq | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | | GitLab Rails | 3 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | | Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Object storage | n/a | n/a | n/a | n/a | n/a | +| Object storage(4) | n/a | n/a | n/a | n/a | n/a | | NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> +<!-- markdownlint-disable MD029 --> +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. +4. Should be run on reputable third party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +<!-- markdownlint-enable MD029 --> + NOTE: -Components marked with * can be optionally run on reputable -third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work. -Components marked with ** can be optionally run on reputable -third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +For all PaaS solutions that involve configuring instances, it is strongly recommended to implement a minimum of three nodes in three different availability zones to align with resilient cloud architecture practices. ```plantuml @startuml 3k @@ -169,7 +174,7 @@ To set up GitLab and its components to accommodate up to 3,000 users: provides access to the Git repositories. 1. [Configure Sidekiq](#configure-sidekiq). 1. [Configure the main GitLab Rails application](#configure-gitlab-rails) - to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend + to run Puma, Workhorse, GitLab Shell, and to serve all frontend requests (which include UI, API, and Git over HTTP/SSH). 1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab environment. @@ -470,8 +475,8 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' - roles ['redis_master_role'] + # Specify server role as 'redis_master_role' and enable Consul agent + roles(['redis_master_role', 'consul_role']) # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -487,7 +492,6 @@ a node and change its status from primary to replica (and vice versa). redis['password'] = 'redis-password-goes-here' ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -508,6 +512,9 @@ a node and change its status from primary to replica (and vice versa). gitlab_rails['auto_migrate'] = false ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. You can specify multiple roles, like sentinel and Redis, as: @@ -546,8 +553,8 @@ run: redis-exporter: (pid 30075) 76861s; run: log: (pid 29674) 76896s 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_replica_role' - roles ['redis_replica_role'] + # Specify server role as 'redis_replica_role' and enable Consul agent + roles(['redis_replica_role', 'consul_role']) # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -570,7 +577,6 @@ run: redis-exporter: (pid 30075) 76861s; run: log: (pid 29674) 76896s #redis['master_port'] = 6379 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -591,12 +597,15 @@ run: redis-exporter: (pid 30075) 76861s; run: log: (pid 29674) 76896s gitlab_rails['auto_migrate'] = false ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after @@ -638,7 +647,7 @@ To configure the Sentinel: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - roles ['redis_sentinel_role', 'consul_role'] + roles(['redis_sentinel_role', 'consul_role']) # Must be the same in every sentinel node redis['master_name'] = 'gitlab-redis' @@ -702,7 +711,6 @@ To configure the Sentinel: # sentinel['failover_timeout'] = 60000 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -720,7 +728,11 @@ To configure the Sentinel: gitlab_rails['auto_migrate'] = false ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Go through the steps again for all the other Consul/Sentinel nodes, and make sure you set up the correct IPs. @@ -813,6 +825,15 @@ in the second step, do not supply the `EXTERNAL_URL` value. sudo gitlab-ctl pg-password-md5 pgbouncer ``` +1. Generate a password hash for the PostgreSQL replication username/password pair. This assumes you will use the default + username of `gitlab_replicator` (recommended). The command will request a password + and a confirmation. Use the value that is output by this command in the next step + as the value of `<postgresql_replication_password_hash>`: + + ```shell + sudo gitlab-ctl pg-password-md5 gitlab_replicator + ``` + 1. Generate a password hash for the Consul database username/password pair. This assumes you will use the default username of `gitlab-consul` (recommended). The command will request a password and confirmation. Use the value that is output by this command in the next @@ -825,27 +846,28 @@ in the second step, do not supply the `EXTERNAL_URL` value. 1. On every database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: ```ruby - # Disable all components except PostgreSQL, Patroni, and Consul - roles ['postgres_role'] - + # Disable all components except Patroni and Consul + roles(['patroni_role']) + # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' - # Enable Patroni - patroni['enable'] = true - # Set `max_wal_senders` to one more than the number of database nodes in the cluster. + # Sets `max_replication_slots` to double the number of database nodes. + # Patroni uses one extra slot per node when initiating the replication. + patroni['postgresql']['max_replication_slots'] = 6 + + # Set `max_wal_senders` to one more than the number of replication slots in the cluster. # This is used to prevent replication from using up all of the # available database connections. - patroni['postgresql']['max_wal_senders'] = 4 - patroni['postgresql']['max_replication_slots'] = 4 + patroni['postgresql']['max_wal_senders'] = 7 + # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. patroni['postgresql']['max_connections'] = 500 # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false - + # Configure the Consul agent - consul['enable'] = true consul['services'] = %w(postgresql) ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -855,6 +877,8 @@ in the second step, do not supply the `EXTERNAL_URL` value. # # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value postgresql['pgbouncer_user_password'] = '<pgbouncer_password_hash>' + # Replace POSTGRESQL_REPLICATION_PASSWORD_HASH with a generated md5 value + postgresql['sql_replication_password'] = '<postgresql_replication_password_hash>' # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value postgresql['sql_user_password'] = '<postgresql_password_hash>' @@ -878,9 +902,8 @@ PostgreSQL, with Patroni managing its failover, will default to use `pg_rewind` Like most failover handling methods, this has a small chance of leading to data loss. Learn more about the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method). -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -895,22 +918,7 @@ are supported and can be added if needed. #### PostgreSQL post-configuration -SSH in to the **primary node**: - -1. Open a database prompt: - - ```shell - gitlab-psql -d gitlabhq_production - ``` - -1. Enable the `pg_trgm` and `btree_gist` extensions: - - ```shell - CREATE EXTENSION pg_trgm; - CREATE EXTENSION btree_gist; - ``` - -1. Exit the database prompt by typing `\q` and Enter. +SSH in to any of the Patroni nodes on the **primary site**: 1. Check the status of the leader and cluster: @@ -952,7 +960,7 @@ The following IPs will be used as an example: ```ruby # Disable all components except Pgbouncer and Consul agent - roles ['pgbouncer_role'] + roles(['pgbouncer_role']) # Configure PgBouncer pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) @@ -969,7 +977,6 @@ The following IPs will be used as an example: # Configure Consul agent consul['watchers'] = %w(postgresql) - consul['enable'] = true consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } @@ -982,6 +989,9 @@ The following IPs will be used as an example: pgbouncer_exporter['listen_address'] = '0.0.0.0:9188' ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Create a `.pgpass` file so Consul is able to @@ -1097,9 +1107,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. ```ruby # Disable all components except PostgreSQL and Consul - roles ['postgres_role'] - repmgr['enable'] = false - patroni['enable'] = false + roles(['postgres_role', 'consul_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' @@ -1109,7 +1117,6 @@ in the second step, do not supply the `EXTERNAL_URL` value. gitlab_rails['auto_migrate'] = false # Configure the Consul agent - consul['enable'] = true ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1135,6 +1142,9 @@ in the second step, do not supply the `EXTERNAL_URL` value. # END user configuration ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Follow the [post configuration](#praefect-postgresql-post-configuration). @@ -1213,7 +1223,7 @@ Praefect requires several secret tokens to secure communications across the Clus Gitaly Cluster nodes are configured in Praefect via a `virtual storage`. Each storage contains the details of each Gitaly node that makes up the cluster. Each storage is also given a name -and this name is used in several areas of the config. In this guide, the name of the storage will be +and this name is used in several areas of the configuration. In this guide, the name of the storage will be `default`. Also, this guide is geared towards new installs, if upgrading an existing environment to use Gitaly Cluster, you may need to use a different name. Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more info. @@ -1233,19 +1243,18 @@ To configure the Praefect nodes, on each one: 1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect: ```ruby - # Avoid running unnecessary services on the Gitaly server + # Avoid running unnecessary services on the Praefect server + gitaly['enable'] = false postgresql['enable'] = false redis['enable'] = false - nginx['enable'] = false puma['enable'] = false - unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false - grafana['enable'] = false - - # If you run a separate monitoring node you can disable these services - alertmanager['enable'] = false prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false + gitlab_exporter['enable'] = false + nginx['enable'] = false # Praefect Configuration praefect['enable'] = true @@ -1283,19 +1292,20 @@ To configure the Praefect nodes, on each one: # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1') praefect['virtual_storages'] = { 'default' => { - 'gitaly-1' => { - 'address' => 'tcp://10.6.0.91:8075', - 'token' => '<praefect_internal_token>', - 'primary' => true - }, - 'gitaly-2' => { - 'address' => 'tcp://10.6.0.92:8075', - 'token' => '<praefect_internal_token>' - }, - 'gitaly-3' => { - 'address' => 'tcp://10.6.0.93:8075', - 'token' => '<praefect_internal_token>' - }, + 'nodes' => { + 'gitaly-1' => { + 'address' => 'tcp://10.6.0.91:8075', + 'token' => '<praefect_internal_token>' + }, + 'gitaly-2' => { + 'address' => 'tcp://10.6.0.92:8075', + 'token' => '<praefect_internal_token>' + }, + 'gitaly-3' => { + 'address' => 'tcp://10.6.0.93:8075', + 'token' => '<praefect_internal_token>' + }, + } } } @@ -1312,11 +1322,25 @@ To configure the Praefect nodes, on each one: # END user configuration ``` - 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and - then replace the file of the same name on this server. If that file isn't on - this server, add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace +the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. - 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Praefect requires to run some database migrations, much like the main GitLab application. For this + you should select **one Praefect node only to run the migrations**, AKA the _Deploy Node_. This node + must be configured first before the others as follows: + + 1. In the `/etc/gitlab/gitlab.rb` file, change the `praefect['auto_migrate']` setting value from `false` to `true` + + 1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect and + to run the Praefect database migrations. + +1. On all other Praefect nodes, [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. ### Configure Gitaly @@ -1360,29 +1384,27 @@ On each node: storage paths, enable the network listener, and to configure the token: ```ruby - # /etc/gitlab/gitlab.rb - # Avoid running unnecessary services on the Gitaly server postgresql['enable'] = false redis['enable'] = false - nginx['enable'] = false puma['enable'] = false - unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false - grafana['enable'] = false - - # If you run a separate monitoring node you can disable these services - alertmanager['enable'] = false prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false + gitlab_exporter['enable'] = false + nginx['enable'] = false # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false + # Gitaly + gitaly['enable'] = true + # Configure the gitlab-shell API callback URL. Without this, `git push` will # fail. This can be your 'front door' GitLab URL or an internal load # balancer. - # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server. gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' # Make Gitaly accept connections on all network interfaces. You must use @@ -1426,9 +1448,8 @@ On each node: }) ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and - then replace the file of the same name on this server. If that file isn't on - this server, add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -1537,29 +1558,19 @@ To configure the Sidekiq nodes, one each one: 1. Open `/etc/gitlab/gitlab.rb` with your editor: ```ruby - ######################################## - ##### Services Disabled ### - ######################################## - - nginx['enable'] = false - grafana['enable'] = false - prometheus['enable'] = false - alertmanager['enable'] = false + # Avoid running unnecessary services on the Sidekiq server gitaly['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = false - puma['enable'] = false - postgres_exporter['enable'] = false postgresql['enable'] = false redis['enable'] = false - redis_exporter['enable'] = false + puma['enable'] = false + gitlab_workhorse['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false gitlab_exporter['enable'] = false + nginx['enable'] = false - ######################################## - #### Redis ### - ######################################## - - ## Must be the same in every sentinel node + # Redis redis['master_name'] = 'gitlab-redis' ## The same password for Redis authentication you set up for the master node. @@ -1572,13 +1583,10 @@ To configure the Sidekiq nodes, one each one: {'host' => '10.6.0.13', 'port' => 26379}, ] - ####################################### - ### Gitaly ### - ####################################### - - # git_data_dirs get configured for the Praefect virtual storage - # Address is Internal Load Balancer for Praefect - # Token is praefect_external_token + # Gitaly Cluster + ## git_data_dirs get configured for the Praefect virtual storage + ## Address is Internal Load Balancer for Praefect + ## Token is praefect_external_token git_data_dirs({ "default" => { "gitaly_address" => "tcp://10.6.0.40:2305", # internal load balancer IP @@ -1586,31 +1594,26 @@ To configure the Sidekiq nodes, one each one: } }) - ####################################### - ### Postgres ### - ####################################### + # PostgreSQL gitlab_rails['db_host'] = '10.6.0.40' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' gitlab_rails['db_adapter'] = 'postgresql' gitlab_rails['db_encoding'] = 'unicode' - # Prevent database migrations from running on upgrade automatically + ## Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false - ####################################### - ### Sidekiq configuration ### - ####################################### + # Sidekiq + sidekiq['enable'] = true sidekiq['listen_address'] = "0.0.0.0" - # Set number of Sidekiq queue processes to the same number as available CPUs + ## Set number of Sidekiq queue processes to the same number as available CPUs sidekiq['queue_groups'] = ['*'] * 2 - # Set number of Sidekiq threads per queue process to the recommend number of 10 + ## Set number of Sidekiq threads per queue process to the recommend number of 10 sidekiq['max_concurrency'] = 10 - ####################################### - ### Monitoring configuration ### - ####################################### + # Monitoring consul['enable'] = true consul['monitoring_service_discovery'] = true @@ -1618,19 +1621,16 @@ To configure the Sidekiq nodes, one each one: retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } - # Set the network addresses that the exporters will listen on + ## Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' - # Rails Status for prometheus + ## Add the monitoring node's IP address to the monitoring whitelist gitlab_rails['monitoring_whitelist'] = ['10.6.0.81/32', '127.0.0.0/8'] gitlab_rails['prometheus_address'] = '10.6.0.81:9090' - ############################# - ### Object storage ### - ############################# - - # This is an example for configuring Object Storage on GCP - # Replace this config with your chosen Object Storage provider as desired + # Object Storage + ## This is an example for configuring Object Storage on GCP + ## Replace this config with your chosen Object Storage provider as desired gitlab_rails['object_store']['connection'] = { 'provider' => 'Google', 'google_project' => '<gcp-project-name>', @@ -1644,9 +1644,28 @@ To configure the Sidekiq nodes, one each one: gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } + gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + Only a single designated node should handle migrations as detailed in the + [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. + 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + 1. Verify the GitLab services are running: ```shell @@ -1728,7 +1747,7 @@ On each node perform the following: }) ## Disable components that will not be on the GitLab application server - roles ['application_role'] + roles(['application_role']) gitaly['enable'] = false nginx['enable'] = true sidekiq['enable'] = false @@ -1793,10 +1812,7 @@ On each node perform the following: #registry['uid'] = 9002 #registry['gid'] = 9002 - ############################# - ### Object storage ### - ############################# - + # Object storage # This is an example for configuring Object Storage on GCP # Replace this config with your chosen Object Storage provider as desired gitlab_rails['object_store']['connection'] = { @@ -1811,6 +1827,13 @@ On each node perform the following: gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>" gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" + + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } + gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" ``` 1. If you're using [Gitaly with TLS support](#gitaly-cluster-tls-support), make sure the @@ -1831,7 +1854,20 @@ On each node perform the following: sudo cp cert.pem /etc/gitlab/trusted-certs/ ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + Only a single designated node should handle migrations as detailed in the + [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Run `sudo gitlab-rake gitlab:gitaly:check` to confirm the node can connect to Gitaly. 1. Tail the logs to see the requests: @@ -1839,11 +1875,6 @@ On each node perform the following: sudo gitlab-ctl tail gitaly ``` -1. Save the `/etc/gitlab/gitlab-secrets.json` file from one of the two - application nodes and install it on the other application node, the - [Gitaly node](#configure-gitaly) and the [Sidekiq node](#configure-sidekiq) and - [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - 1. Verify the GitLab services are running: ```shell @@ -1902,45 +1933,26 @@ running [Prometheus](../monitoring/prometheus/index.md) and 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - external_url 'http://gitlab.example.com' + roles(['monitoring_role', 'consul_role']) - # Disable all other services - alertmanager['enable'] = false - gitaly['enable'] = false - gitlab_exporter['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = true - postgres_exporter['enable'] = false - postgresql['enable'] = false - redis['enable'] = false - redis_exporter['enable'] = false - sidekiq['enable'] = false - puma['enable'] = false - unicorn['enable'] = false - node_exporter['enable'] = false - gitlab_exporter['enable'] = false + external_url 'http://gitlab.example.com' - # Enable Prometheus - prometheus['enable'] = true + # Prometheus prometheus['listen_address'] = '0.0.0.0:9090' prometheus['monitor_kubernetes'] = false - # Enable Login form - grafana['disable_login_form'] = false - - # Enable Grafana - grafana['enable'] = true + # Grafana grafana['admin_password'] = '<grafana_password>' + grafana['disable_login_form'] = false # Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false + # Nginx - For Grafana access + nginx['enable'] = true ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -2074,7 +2086,7 @@ but with smaller performance requirements, several modifications can be consider - PostgreSQL: Can be run on reputable Cloud PaaS solutions such as Google Cloud SQL or AWS RDS. In this setup, the PgBouncer and Consul nodes are no longer required: - Consul may still be desired if [Prometheus](../monitoring/prometheus/index.md) auto discovery is a requirement, otherwise you would need to [manually add scrape configurations](../monitoring/prometheus/index.md#adding-custom-scrape-configurations) for all nodes. - As Redis Sentinel runs on the same box as Consul in this architecture, it may need to be run on a separate box if Redis is still being run via Omnibus. - - Redis: Can be run on reputable Cloud PaaS solutions such as Google Memorystore and AWS Elasticache. In this setup, the Redis Sentinel is no longer required. + - Redis: Can be run on reputable Cloud PaaS solutions such as Google Memorystore and AWS ElastiCache. In this setup, the Redis Sentinel is no longer required. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> |