diff options
Diffstat (limited to 'doc/administration/reference_architectures/25k_users.md')
| -rw-r--r-- | doc/administration/reference_architectures/25k_users.md | 198 |
1 files changed, 43 insertions, 155 deletions
diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index c5a8c3927c0..d96e93d4ab4 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -422,9 +422,9 @@ install the necessary dependencies from step 1, and add the GitLab package repository from step 2. When installing GitLab in the second step, do not supply the `EXTERNAL_URL` value. -#### PostgreSQL primary node +#### PostgreSQL nodes -1. SSH in to the PostgreSQL primary node. +1. SSH in to one of the PostgreSQL nodes. 1. Generate a password hash for the PostgreSQL username/password pair. This assumes you will use the default username of `gitlab` (recommended). The command will request a password and confirmation. Use the value that is output by this command in the next @@ -452,23 +452,33 @@ in the second step, do not supply the `EXTERNAL_URL` value. sudo gitlab-ctl pg-password-md5 gitlab-consul ``` -1. On the primary database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: +1. On every database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: ```ruby - # Disable all components except PostgreSQL and Repmgr and Consul + # Disable all components except PostgreSQL, Patroni, and Consul roles ['postgres_role'] # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' - postgresql['hot_standby'] = 'on' - postgresql['wal_level'] = 'replica' - postgresql['shared_preload_libraries'] = 'repmgr_funcs' + + # Enable Patroni + patroni['enable'] = true + # Set `max_wal_senders` to one more than the number of database nodes in the cluster. + # This is used to prevent replication from using up all of the + # available database connections. + patroni['postgresql']['max_wal_senders'] = 4 + patroni['postgresql']['max_replication_slots'] = 4 + # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. + patroni['postgresql']['max_connections'] = 500 # Disable automatic database migrations gitlab_rails['auto_migrate'] = false # Configure the Consul agent + consul['enable'] = true consul['services'] = %w(postgresql) + ## Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true # START user configuration # Please set the real values as explained in Required Information section @@ -477,18 +487,9 @@ in the second step, do not supply the `EXTERNAL_URL` value. postgresql['pgbouncer_user_password'] = '<pgbouncer_password_hash>' # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value postgresql['sql_user_password'] = '<postgresql_password_hash>' - # Set `max_wal_senders` to one more than the number of database nodes in the cluster. - # This is used to prevent replication from using up all of the - # available database connections. - postgresql['max_wal_senders'] = 4 - postgresql['max_replication_slots'] = 4 # Replace XXX.XXX.XXX.XXX/YY with Network Address postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24) - repmgr['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.6.0.0/24) - - ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true # Set the network addresses that the exporters will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' @@ -503,70 +504,9 @@ in the second step, do not supply the `EXTERNAL_URL` value. # END user configuration ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. - -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -<div align="right"> - <a type="button" class="btn btn-default" href="#setup-components"> - Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> - -#### PostgreSQL secondary nodes - -1. On both the secondary nodes, add the same configuration specified above for the primary node - with an additional setting (`repmgr['master_on_initialization'] = false`) that will inform `gitlab-ctl` that they are standby nodes initially - and there's no need to attempt to register them as a primary node: - - ```ruby - # Disable all components except PostgreSQL and Repmgr and Consul - roles ['postgres_role'] - - # PostgreSQL configuration - postgresql['listen_address'] = '0.0.0.0' - postgresql['hot_standby'] = 'on' - postgresql['wal_level'] = 'replica' - postgresql['shared_preload_libraries'] = 'repmgr_funcs' - - # Disable automatic database migrations - gitlab_rails['auto_migrate'] = false - - # Configure the Consul agent - consul['services'] = %w(postgresql) - - # Specify if a node should attempt to be primary on initialization. - repmgr['master_on_initialization'] = false - - # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value - postgresql['pgbouncer_user_password'] = '<pgbouncer_password_hash>' - # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value - postgresql['sql_user_password'] = '<postgresql_password_hash>' - # Set `max_wal_senders` to one more than the number of database nodes in the cluster. - # This is used to prevent replication from using up all of the - # available database connections. - postgresql['max_wal_senders'] = 4 - postgresql['max_replication_slots'] = 4 - - # Replace with your network addresses - postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24) - repmgr['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.6.0.0/24) - - ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true - - # Set the network addresses that the exporters will listen on for monitoring - node_exporter['listen_address'] = '0.0.0.0:9100' - postgres_exporter['listen_address'] = '0.0.0.0:9187' - - ## The IPs of the Consul server nodes - ## You can also use FQDNs and intermix them with IPs - consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), - } - ``` +PostgreSQL, with Patroni managing its failover, will default to use `pg_rewind` by default to handle conflicts. +Like most failover handling methods, this has a small chance of leading to data loss. +Learn more about the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method). 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace the file of the same name on this server. If that file is not on this server, @@ -601,84 +541,25 @@ SSH in to the **primary node**: 1. Exit the database prompt by typing `\q` and Enter. -1. Verify the cluster is initialized with one node: +1. Check the status of the leader and cluster: ```shell - gitlab-ctl repmgr cluster show + gitlab-ctl patroni members ``` The output should be similar to the following: ```plaintext - Role | Name | Upstream | Connection String - ----------+----------|----------|---------------------------------------- - * master | HOSTNAME | | host=HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr - ``` - -1. Note down the hostname or IP address in the connection string: `host=HOSTNAME`. We will - refer to the hostname in the next section as `<primary_node_name>`. If the value - is not an IP address, it will need to be a resolvable name (via DNS or - `/etc/hosts`) - -SSH in to the **secondary node**: - -1. Set up the repmgr standby: - - ```shell - gitlab-ctl repmgr standby setup <primary_node_name> + | Cluster | Member | Host | Role | State | TL | Lag in MB | Pending restart | + |---------------|-----------------------------------|-----------|--------|---------|-----|-----------|-----------------| + | postgresql-ha | <PostgreSQL primary hostname> | 10.6.0.21 | Leader | running | 175 | | * | + | postgresql-ha | <PostgreSQL secondary 1 hostname> | 10.6.0.22 | | running | 175 | 0 | * | + | postgresql-ha | <PostgreSQL secondary 2 hostname> | 10.6.0.23 | | running | 175 | 0 | * | ``` - Do note that this will remove the existing data on the node. The command - has a wait time. - - The output should be similar to the following: - - ```console - Doing this will delete the entire contents of /var/opt/gitlab/postgresql/data - If this is not what you want, hit Ctrl-C now to exit - To skip waiting, rerun with the -w option - Sleeping for 30 seconds - Stopping the database - Removing the data - Cloning the data - Starting the database - Registering the node with the cluster - ok: run: repmgrd: (pid 19068) 0s - ``` - -Before moving on, make sure the databases are configured correctly. Run the -following command on the **primary** node to verify that replication is working -properly and the secondary nodes appear in the cluster: - -```shell -gitlab-ctl repmgr cluster show -``` - -The output should be similar to the following: - -```plaintext -Role | Name | Upstream | Connection String -----------+---------|-----------|------------------------------------------------ -* master | MASTER | | host=<primary_node_name> user=gitlab_repmgr dbname=gitlab_repmgr - standby | STANDBY | MASTER | host=<secondary_node_name> user=gitlab_repmgr dbname=gitlab_repmgr - standby | STANDBY | MASTER | host=<secondary_node_name> user=gitlab_repmgr dbname=gitlab_repmgr -``` - -If the 'Role' column for any node says "FAILED", check the +If the 'State' column for any node doesn't say "running", check the [Troubleshooting section](troubleshooting.md) before proceeding. -Also, check that the `repmgr-check-master` command works successfully on each node: - -```shell -su - gitlab-consul -gitlab-ctl repmgr-check-master || echo 'This node is a standby repmgr node' -``` - -This command relies on exit codes to tell Consul whether a particular node is a master -or secondary. The most important thing here is that this command does not produce errors. -If there are errors it's most likely due to incorrect `gitlab-consul` database user permissions. -Check the [Troubleshooting section](troubleshooting.md) before proceeding. - <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> @@ -696,7 +577,7 @@ The following IPs will be used as an example: 1. On each PgBouncer node, edit `/etc/gitlab/gitlab.rb`, and replace `<consul_password_hash>` and `<pgbouncer_password_hash>` with the - password hashes you [set up previously](#postgresql-primary-node): + password hashes you [set up previously](#postgresql-nodes): ```ruby # Disable all components except Pgbouncer and Consul agent @@ -704,15 +585,16 @@ The following IPs will be used as an example: # Configure PgBouncer pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) - pgbouncer['users'] = { - 'gitlab-consul': { - password: '<consul_password_hash>' - }, - 'pgbouncer': { - password: '<pgbouncer_password_hash>' - } + 'gitlab-consul': { + password: '<consul_password_hash>' + }, + 'pgbouncer': { + password: '<pgbouncer_password_hash>' + } } + # Incoming recommended value for max db connections is 150. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. + pgbouncer['max_db_connections'] = 150 # Configure Consul agent consul['watchers'] = %w(postgresql) @@ -2125,6 +2007,12 @@ to use GitLab Pages, this currently [requires NFS](troubleshooting.md#gitlab-pag See how to [configure NFS](../nfs.md). +WARNING: +From GitLab 13.0, using NFS for Git repositories is deprecated. +From GitLab 14.0, technical support for NFS for Git repositories +will no longer be provided. Upgrade to [Gitaly Cluster](../gitaly/praefect.md) +as soon as possible. + <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> |