diff options
Diffstat (limited to 'doc/administration/reference_architectures')
9 files changed, 1301 insertions, 187 deletions
diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 7f31f336251..a15fcf722a5 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -8,8 +8,8 @@ For a full list of reference architectures, see > - **High Availability:** True > - **Test RPS rates:** API: 200 RPS, Web: 20 RPS, Git: 20 RPS -| Service | Nodes | Configuration ([8](#footnotes)) | GCP | AWS ([9](#footnotes)) | Azure([9](#footnotes)) | -|--------------------------------------------------------------|-------|---------------------------------|----------------|-----------------------|------------------------| +| Service | Nodes | Configuration ([8](#footnotes)) | GCP | AWS | Azure | +|--------------------------------------------------------------|-------|---------------------------------|----------------|-----------------------|----------------| | GitLab Rails ([1](#footnotes)) | 3 | 32 vCPU, 28.8GB Memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | | PostgreSQL | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | | PgBouncer | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | @@ -56,7 +56,7 @@ For a full list of reference architectures, see 1. NFS can be used as an alternative for both repository data (replacing Gitaly) and object storage but this isn't typically recommended for performance reasons. Note however it is required for - [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/issues/196). + [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/196). 1. Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) as the load balancer. Although other load balancers with similar feature sets @@ -74,6 +74,3 @@ For a full list of reference architectures, see or higher, are required for your CPU or Node counts accordingly. For more information, a [Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found [here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). - -1. AWS-equivalent and Azure-equivalent configurations are rough suggestions - and may change in the future. They have not yet been tested and validated. diff --git a/doc/administration/reference_architectures/1k_users.md b/doc/administration/reference_architectures/1k_users.md index b6aaffa9488..34805a8ac68 100644 --- a/doc/administration/reference_architectures/1k_users.md +++ b/doc/administration/reference_architectures/1k_users.md @@ -7,11 +7,18 @@ For a full list of reference architectures, see > - **Supported users (approximate):** 1,000 > - **High Availability:** False -| Users | Configuration([8](#footnotes)) | GCP | AWS([9](#footnotes)) | Azure([9](#footnotes)) | -|-------|--------------------------------|---------------|----------------------|------------------------| -| 100 | 2 vCPU, 7.2GB Memory | n1-standard-2 | m5.large | D2s v3 | -| 500 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | -| 1000 | 8 vCPU, 30GB Memory | n1-standard-8 | m5.2xlarge | D8s v3 | +| Users | Configuration([8](#footnotes)) | GCP | AWS | Azure | +|-------|------------------------------------|----------------|---------------------|------------------------| +| 500 | 4 vCPU, 3.6GB Memory | `n1-highcpu-4` | `c5.xlarge` | F4s v2 | +| 1000 | 8 vCPU, 7.2GB Memory | `n1-highcpu-8` | `c5.2xlarge` | F8s v2 | + +In addition to the above, we recommend having at least +2GB of swap on your server, even if you currently have +enough available RAM. Having swap will help reduce the chance of errors occurring +if your available memory changes. We also recommend +configuring the kernel's swappiness setting +to a low value like `10` to make the most of your RAM while still having the swap +available when needed. For situations where you need to serve up to 1,000 users, a single-node solution with [frequent backups](index.md#automated-backups-core-only) is appropriate @@ -25,7 +32,7 @@ requirements, this is the ideal solution. NOTE: **Note:** You can also optionally configure GitLab to use an -[external PostgreSQL service](../external_database.md) or an +[external PostgreSQL service](../postgresql/external.md) or an [external object storage service](../high_availability/object_storage.md) for added performance and reliability at a reduced complexity cost. @@ -59,7 +66,7 @@ added performance and reliability at a reduced complexity cost. 1. NFS can be used as an alternative for both repository data (replacing Gitaly) and object storage but this isn't typically recommended for performance reasons. Note however it is required for - [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/issues/196). + [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/196). 1. Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) as the load balancer. Although other load balancers with similar feature sets @@ -77,6 +84,3 @@ added performance and reliability at a reduced complexity cost. or higher, are required for your CPU or Node counts accordingly. For more information, a [Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found [here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). - -1. AWS-equivalent and Azure-equivalent configurations are rough suggestions - and may change in the future. They have not yet been tested and validated. diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 2ee692d635c..d851fa124c6 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -8,23 +8,23 @@ For a full list of reference architectures, see > - **High Availability:** True > - **Test RPS rates:** API: 500 RPS, Web: 50 RPS, Git: 50 RPS -| Service | Nodes | Configuration ([8](#footnotes)) | GCP | AWS ([9](#footnotes)) | Azure([9](#footnotes)) | -|--------------------------------------------------------------|-------|---------------------------------|----------------|-----------------------|------------------------| -| GitLab Rails ([1](#footnotes)) | 5 | 32 vCPU, 28.8GB Memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | -| PostgreSQL | 3 | 8 vCPU, 30GB Memory | n1-standard-8 | m5.2xlarge | D8s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | -| Gitaly ([2](#footnotes)) ([5](#footnotes)) ([7](#footnotes)) | X | 32 vCPU, 120GB Memory | n1-standard-32 | m5.8xlarge | D32s v3 | -| Redis ([3](#footnotes)) - Cache | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis ([3](#footnotes)) - Queues / Shared State | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis Sentinel ([3](#footnotes)) - Cache | 3 | 1 vCPU, 1.7GB Memory | g1-small | t2.small | B1MS | -| Redis Sentinel ([3](#footnotes)) - Queues / Shared State | 3 | 1 vCPU, 1.7GB Memory | g1-small | t2.small | B1MS | -| Consul | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | -| Sidekiq | 4 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Service | Nodes | Configuration ([8](#footnotes)) | GCP | AWS | Azure | +|--------------------------------------------------------------|-------|---------------------------------|------------------|-----------------------|----------------| +| GitLab Rails ([1](#footnotes)) | 5 | 32 vCPU, 28.8GB Memory | `n1-highcpu-32` | `c5.9xlarge` | F32s v2 | +| PostgreSQL | 3 | 8 vCPU, 30GB Memory | `n1-standard-8` | `m5.2xlarge` | D8s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | F2s v2 | +| Gitaly ([2](#footnotes)) ([5](#footnotes)) ([7](#footnotes)) | X | 32 vCPU, 120GB Memory | `n1-standard-32` | `m5.8xlarge` | D32s v3 | +| Redis ([3](#footnotes)) - Cache | 3 | 4 vCPU, 15GB Memory | `n1-standard-4` | `m5.xlarge` | D4s v3 | +| Redis ([3](#footnotes)) - Queues / Shared State | 3 | 4 vCPU, 15GB Memory | `n1-standard-4` | `m5.xlarge` | D4s v3 | +| Redis Sentinel ([3](#footnotes)) - Cache | 3 | 1 vCPU, 1.7GB Memory | `g1-small` | `t2.small` | B1MS | +| Redis Sentinel ([3](#footnotes)) - Queues / Shared State | 3 | 1 vCPU, 1.7GB Memory | `g1-small` | `t2.small` | B1MS | +| Consul | 3 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | F2s v2 | +| Sidekiq | 4 | 4 vCPU, 15GB Memory | `n1-standard-4` | `m5.xlarge` | D4s v3 | | Object Storage ([4](#footnotes)) | - | - | - | - | - | -| NFS Server ([5](#footnotes)) ([7](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | F4s v2 | -| Monitoring node | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | F4s v2 | -| External load balancing node ([6](#footnotes)) | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | -| Internal load balancing node ([6](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| NFS Server ([5](#footnotes)) ([7](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | `n1-highcpu-4` | `c5.xlarge` | F4s v2 | +| Monitoring node | 1 | 4 vCPU, 3.6GB Memory | `n1-highcpu-4` | `c5.xlarge` | F4s v2 | +| External load balancing node ([6](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | `n1-highcpu-4` | `c5.xlarge` | F4s v2 | +| Internal load balancing node ([6](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | `n1-highcpu-4` | `c5.xlarge` | F4s v2 | ## Footnotes @@ -56,7 +56,7 @@ For a full list of reference architectures, see 1. NFS can be used as an alternative for both repository data (replacing Gitaly) and object storage but this isn't typically recommended for performance reasons. Note however it is required for - [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/issues/196). + [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/196). 1. Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) as the load balancer. Although other load balancers with similar feature sets @@ -74,6 +74,3 @@ For a full list of reference architectures, see or higher, are required for your CPU or Node counts accordingly. For more information, a [Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found [here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). - -1. AWS-equivalent and Azure-equivalent configurations are rough suggestions - and may change in the future. They have not yet been tested and validated. diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index 874e00e6722..0a3ade1acf1 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -1,3 +1,7 @@ +--- +reading_time: true +--- + # Reference architecture: up to 2,000 users This page describes GitLab reference architecture for up to 2,000 users. @@ -6,85 +10,864 @@ For a full list of reference architectures, see > - **Supported users (approximate):** 2,000 > - **High Availability:** False -> - **Test RPS rates:** API: 40 RPS, Web: 4 RPS, Git: 4 RPS - -| Service | Nodes | Configuration ([8](#footnotes)) | GCP | AWS ([9](#footnotes)) | Azure([9](#footnotes)) | -|--------------------------------------------------------------|-------|---------------------------------|---------------|-----------------------|----------------| -| External load balancing node ([6](#footnotes)) | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | -| Object Storage ([4](#footnotes)) | - | - | - | - | - | -| NFS Server ([5](#footnotes)) ([7](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | F4s v2 | -| PostgreSQL | 1 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | D2s v3 | -| Redis ([3](#footnotes)) | 1 | 1 vCPU, 3.75GB Memory | n1-standard-1 | m5.large | D2s v3 | -| Gitaly ([5](#footnotes)) ([7](#footnotes)) | X ([2](#footnotes)) | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | -| GitLab Rails ([1](#footnotes)) | 2 | 8 vCPU, 7.2GB Memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | -| Monitoring node | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | - -## Setup instructions - -1. [Configure the external load balancing node](../high_availability/load_balancer.md) - that will handle the load balancing of the two GitLab application services nodes. -1. [Configure the Object Storage](../object_storage.md) ([4](#footnotes)) used for shared data objects. -1. (Optional) [Configure NFS](../high_availability/nfs.md) to have - shared disk storage service as an alternative to Gitaly and/or - [Object Storage](../object_storage.md) (although not recommended). - NFS is required for GitLab Pages, you can skip this step if you're not using that feature. -1. [Configure PostgreSQL](../high_availability/load_balancer.md), the database for GitLab. -1. [Configure Redis](../high_availability/redis.md). -1. [Configure Gitaly](../gitaly/index.md#running-gitaly-on-its-own-server), - which is used to provide access to the Git repositories. -1. [Configure the main GitLab Rails application](../high_availability/gitlab.md) - to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all - frontend requests (UI, API, Git over HTTP/SSH). -1. [Configure Prometheus](../high_availability/monitoring_node.md) to monitor your GitLab environment. - -## Footnotes - -1. In our architectures we run each GitLab Rails node using the Puma webserver - and have its number of workers set to 90% of available CPUs along with four threads. For - nodes that are running Rails with other components the worker value should be reduced - accordingly where we've found 50% achieves a good balance but this is dependent - on workload. - -1. Gitaly node requirements are dependent on customer data, specifically the number of - projects and their sizes. We recommend two nodes as an absolute minimum for HA environments - and at least four nodes should be used when supporting 50,000 or more users. - We also recommend that each Gitaly node should store no more than 5TB of data - and have the number of [`gitaly-ruby` workers](../gitaly/index.md#gitaly-ruby) - set to 20% of available CPUs. Additional nodes should be considered in conjunction - with a review of expected data size and spread based on the recommendations above. - -1. Recommended Redis setup differs depending on the size of the architecture. - For smaller architectures (less than 3,000 users) a single instance should suffice. - For medium sized installs (3,000 - 5,000) we suggest one Redis cluster for all - classes and that Redis Sentinel is hosted alongside Consul. - For larger architectures (10,000 users or more) we suggest running a separate - [Redis Cluster](../high_availability/redis.md#running-multiple-redis-clusters) for the Cache class - and another for the Queues and Shared State classes respectively. We also recommend - that you run the Redis Sentinel clusters separately for each Redis Cluster. - -1. For data objects such as LFS, Uploads, Artifacts, etc. We recommend an [Object Storage service](../object_storage.md) - over NFS where possible, due to better performance and availability. - -1. NFS can be used as an alternative for both repository data (replacing Gitaly) and - object storage but this isn't typically recommended for performance reasons. Note however it is required for - [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/issues/196). - -1. Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) - as the load balancer. Although other load balancers with similar feature sets - could also be used, those load balancers have not been validated. - -1. We strongly recommend that any Gitaly or NFS nodes be set up with SSD disks over - HDD with a throughput of at least 8,000 IOPS for read operations and 2,000 IOPS for write - as these components have heavy I/O. These IOPS values are recommended only as a starter - as with time they may be adjusted higher or lower depending on the scale of your - environment's workload. If you're running the environment on a Cloud provider - you may need to refer to their documentation on how configure IOPS correctly. - -1. The architectures were built and tested with the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) - CPU platform on GCP. On different hardware you may find that adjustments, either lower - or higher, are required for your CPU or Node counts accordingly. For more information, a - [Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found - [here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). - -1. AWS-equivalent and Azure-equivalent configurations are rough suggestions - and may change in the future. They have not yet been tested and validated. +> - **Test requests per second (RPS) rates:** API: 40 RPS, Web: 4 RPS, Git: 4 RPS + +| Service | Nodes | Configuration | GCP | AWS | Azure | +|--------------------------------------------------------------|-----------|---------------------------------|---------------|-----------------------|----------------| +| Load balancer | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Object storage | n/a | n/a | n/a | n/a | n/a | +| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| PostgreSQL | 1 | 2 vCPU, 7.5GB memory | n1-standard-2 | m5.large | D2s v3 | +| Redis | 1 | 1 vCPU, 3.75GB memory | n1-standard-1 | m5.large | D2s v3 | +| Gitaly | 1 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| GitLab Rails | 2 | 8 vCPU, 7.2GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | +| Monitoring node | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | + +The Google Cloud Platform (GCP) architectures were built and tested using the +[Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) +CPU platform. On different hardware you may find that adjustments, either lower +or higher, are required for your CPU or node counts. For more information, see +our [Sysbench](https://github.com/akopytov/sysbench)-based +[CPU benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). + +AWS-equivalent and Azure-equivalent configurations are rough suggestions that +may change in the future, and haven't been tested or validated. + +Due to better performance and availability, for data objects (such as LFS, +uploads, or artifacts), using an [object storage service](#configure-the-object-storage) +is recommended instead of using NFS. Using an object storage service also +doesn't require you to provision and maintain a node. + +## Setup components + +To set up GitLab and its components to accommodate up to 2,000 users: + +1. [Configure the external load balancing node](#configure-the-load-balancer) + to handle the load balancing of the two GitLab application services nodes. +1. [Configure the object storage](#configure-the-object-storage) used for + shared data objects. +1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) + to have shared disk storage service as an alternative to Gitaly or object + storage. You can skip this step if you're not using GitLab Pages (which + requires NFS). +1. [Configure PostgreSQL](#configure-postgresql), the database for GitLab. +1. [Configure Redis](#configure-redis). +1. [Configure Gitaly](#configure-gitaly), which provides access to the Git + repositories. +1. [Configure the main GitLab Rails application](#configure-gitlab-rails) + to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend + requests (which include UI, API, and Git over HTTP/SSH). +1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab + environment. + +## Configure the load balancer + +NOTE: **Note:** +This architecture has been tested and validated with [HAProxy](https://www.haproxy.org/). +Although you can use a load balancer with a similar set of features, GitLab +hasn't validated other load balancers. + +In an active/active GitLab configuration, you'll need a load balancer to route +traffic to the application servers. The specifics for which load balancer to +use or its exact configuration is out of scope for the GitLab documentation. +If you're managing multi-node systems (including GitLab) you'll probably +already have a load balancer of choice. Some examples including HAProxy +(open-source), F5 Big-IP LTM, and Citrix Net Scaler. This documentation +includes the ports and protocols for use with GitLab. + +The next question is how you will handle SSL in your environment. There are +several different options: + +- [The application node terminates SSL](#application-node-terminates-ssl). +- [The load balancer terminates SSL without backend SSL](#load-balancer-terminates-ssl-without-backend-ssl) + and communication is not secure between the load balancer and the application node. +- [The load balancer terminates SSL with backend SSL](#load-balancer-terminates-ssl-with-backend-ssl) + and communication is *secure* between the load balancer and the application node. + +### Application node terminates SSL + +Configure your load balancer to pass connections on port 443 as `TCP` instead +of `HTTP(S)`. This will pass the connection unaltered to the application node's +NGINX service, which has the SSL certificate and listens to port 443. + +For details about managing SSL certificates and configuring NGINX, see the +[NGINX HTTPS documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https). + +### Load balancer terminates SSL without backend SSL + +Configure your load balancer to use the `HTTP(S)` protocol instead of `TCP`. +The load balancer will be responsible for both managing SSL certificates and +terminating SSL. + +Due to communication between the load balancer and GitLab not being secure, +you'll need to complete some additional configuration. For details, see the +[NGINX proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#supporting-proxied-ssl). + +### Load balancer terminates SSL with backend SSL + +Configure your load balancers (or single balancer, if you have only one) to use +the `HTTP(S)` protocol rather than `TCP`. The load balancers will be +responsible for the managing SSL certificates for end users. + +Traffic will be secure between the load balancers and NGINX in this scenario, +and there's no need to add a configuration for proxied SSL. However, you'll +need to add a configuration to GitLab to configure SSL certificates. For +details about managing SSL certificates and configuring NGINX, see the +[NGINX HTTPS documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https). + +### Ports + +The basic load balancer ports you should use are described in the following +table: + +| Port | Backend Port | Protocol | +| ------- | ------------ | ------------------------ | +| 80 | 80 | HTTP (*1*) | +| 443 | 443 | TCP or HTTPS (*1*) (*2*) | +| 22 | 22 | TCP | + +- (*1*): [Web terminal](../../ci/environments/index.md#web-terminals) support + requires your load balancer to correctly handle WebSocket connections. + When using HTTP or HTTPS proxying, your load balancer must be configured + to pass through the `Connection` and `Upgrade` hop-by-hop headers. For + details, see the [web terminal](../integration/terminal.md) integration guide. +- (*2*): When using the HTTPS protocol for port 443, you'll need to add an SSL + certificate to the load balancers. If you need to terminate SSL at the + GitLab application server, use the TCP protocol. + +If you're using GitLab Pages with custom domain support you will need some +additional port configurations. GitLab Pages requires a separate virtual IP +address. Configure DNS to point the `pages_external_url` from +`/etc/gitlab/gitlab.rb` to the new virtual IP address. For more information, +see the [GitLab Pages documentation](../pages/index.md). + +| Port | Backend Port | Protocol | +| ------- | ------------- | --------- | +| 80 | Varies (*1*) | HTTP | +| 443 | Varies (*1*) | TCP (*2*) | + +- (*1*): The backend port for GitLab Pages depends on the + `gitlab_pages['external_http']` and `gitlab_pages['external_https']` + settings. For details, see the [GitLab Pages documentation](../pages/index.md). +- (*2*): Port 443 for GitLab Pages must use the TCP protocol. Users can + configure custom domains with custom SSL, which wouldn't be possible if SSL + was terminated at the load balancer. + +#### Alternate SSH Port + +Some organizations have policies against opening SSH port 22. In this case, +it may be helpful to configure an alternate SSH hostname that instead allows +users to use SSH over port 443. An alternate SSH hostname requires a new +virtual IP address compared to the previously described GitLab HTTP +configuration. + +Configure DNS for an alternate SSH hostname, such as `altssh.gitlab.example.com`: + +| LB Port | Backend Port | Protocol | +| ------- | ------------ | -------- | +| 443 | 22 | TCP | + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + +## Configure the object storage + +GitLab supports using an object storage service for holding several types of +data, and is recommended over [NFS](#configure-nfs-optional). In general, +object storage services are better for larger environments, as object storage +is typically much more performant, reliable, and scalable. + +Object storage options that GitLab has either tested or is aware of customers +using, includes: + +- SaaS/Cloud solutions (such as [Amazon S3](https://aws.amazon.com/s3/) or + [Google Cloud Storage](https://cloud.google.com/storage)). +- On-premises hardware and appliances, from various storage vendors. +- MinIO ([Deployment guide](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html)). + +To configure GitLab to use object storage, refer to the following guides based +on the features you intend to use: + +1. [Object storage for backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage). +1. [Object storage for job artifacts](../job_artifacts.md#using-object-storage) + including [incremental logging](../job_logs.md#new-incremental-logging-architecture). +1. [Object storage for LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage). +1. [Object storage for uploads](../uploads.md#using-object-storage-core-only). +1. [Object storage for merge request diffs](../merge_request_diffs.md#using-object-storage). +1. [Object storage for Container Registry](../packages/container_registry.md#container-registry-storage-driver) (optional feature). +1. [Object storage for Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage) (optional feature). +1. [Object storage for packages](../packages/index.md#using-object-storage) (optional feature). **(PREMIUM ONLY)** +1. [Object storage for Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature). **(PREMIUM ONLY)** +1. [Object storage for Pseudonymizer](../pseudonymizer.md#configuration) (optional feature). **(ULTIMATE ONLY)** +1. [Object storage for autoscale Runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional, for improved performance). +1. [Object storage for Terraform state files](../terraform_state.md#using-object-storage-core-only). + +Using separate buckets for each data type is the recommended approach for GitLab. + +A limitation of our configuration is that each use of object storage is +separately configured. We have an [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/23345) +for improving this, which would allow for one bucket with separate folders. + +Using a single bucket when GitLab is deployed with the Helm chart causes +restoring from a backup to +[not function properly](https://docs.gitlab.com/charts/advanced/external-object-storage/#lfs-artifacts-uploads-packages-external-diffs-pseudonymizer). +Although you may not be using a Helm deployment right now, if you migrate +GitLab to a Helm deployment later, GitLab would still work, but you may not +realize backups aren't working correctly until a critical requirement for +functioning backups is encountered. + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + +## Configure NFS (optional) + +For improved performance, [object storage](#configure-the-object-storage), +along with [Gitaly](#configure-gitaly), are recommended over using NFS whenever +possible. However, if you intend to use GitLab Pages, +[you must use NFS](troubleshooting.md#gitlab-pages-requires-nfs). + +For information about configuring NFS, see the [NFS documentation page](../high_availability/nfs.md). + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + +## Configure PostgreSQL + +In this section, you'll be guided through configuring an external PostgreSQL database +to be used with GitLab. + +### Provide your own PostgreSQL instance + +If you're hosting GitLab on a cloud provider, you can optionally use a +managed service for PostgreSQL. For example, AWS offers a managed relational +database service (RDS) that runs PostgreSQL. + +If you use a cloud-managed service, or provide your own PostgreSQL: + +1. Set up PostgreSQL according to the + [database requirements document](../../install/requirements.md#database). +1. Create a `gitlab` username with a password of your choice. The `gitlab` user + needs privileges to create the `gitlabhq_production` database. +1. Configure the GitLab application servers with the appropriate details. + This step is covered in [Configuring the GitLab Rails application](#configure-gitlab-rails). + +### Standalone PostgreSQL using Omnibus GitLab + +1. SSH into the PostgreSQL server. +1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + - Do not complete any other steps on the download page. +1. Generate a password hash for PostgreSQL. This assumes you will use the default + username of `gitlab` (recommended). The command will request a password + and confirmation. Use the value that is output by this command in the next + step as the value of `POSTGRESQL_PASSWORD_HASH`. + + ```shell + sudo gitlab-ctl pg-password-md5 gitlab + ``` + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents below, updating placeholder + values appropriately. + + - `POSTGRESQL_PASSWORD_HASH` - The value output from the previous step + - `APPLICATION_SERVER_IP_BLOCKS` - A space delimited list of IP subnets or IP + addresses of the GitLab application servers that will connect to the + database. Example: `%w(123.123.123.123/32 123.123.123.234/32)` + + ```ruby + # Disable all components except PostgreSQL + roles ['postgres_role'] + repmgr['enable'] = false + consul['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + pgbouncer_exporter['enable'] = false + redis_exporter['enable'] = false + gitlab_exporter['enable'] = false + + # Set the network addresses that the exporters used for monitoring will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + postgres_exporter['listen_address'] = '0.0.0.0:9187' + postgres_exporter['dbname'] = 'gitlabhq_production' + postgres_exporter['password'] = 'POSTGRESQL_PASSWORD_HASH' + + # Set the PostgreSQL address and port + postgresql['listen_address'] = '0.0.0.0' + postgresql['port'] = 5432 + + # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value + postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' + + # Replace APPLICATION_SERVER_IP_BLOCK with the CIDR address of the application node + postgresql['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 APPLICATION_SERVER_IP_BLOCK) + + # Disable automatic database migrations + gitlab_rails['auto_migrate'] = false + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Note the PostgreSQL node's IP address or hostname, port, and + plain text password. These will be necessary when configuring the [GitLab + application server](#configure-gitlab-rails) later. + +Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/database.html) +are supported and can be added if needed. + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + +## Configure Redis + +In this section, you'll be guided through configuring an external Redis instance +to be used with GitLab. + +### Provide your own Redis instance + +Redis version 5.0 or higher is required, as this is what ships with +Omnibus GitLab packages starting with GitLab 13.0. Older Redis versions +do not support an optional count argument to SPOP which is now required for +[Merge Trains](../../ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md). + +In addition, GitLab makes use of certain commands like `UNLINK` and `USAGE` which +were introduced only in Redis 4. + +Managed Redis from cloud providers such as AWS ElastiCache will work. If these +services support high availability, be sure it is not the Redis Cluster type. + +Note the Redis node's IP address or hostname, port, and password (if required). +These will be necessary when configuring the +[GitLab application servers](#configure-gitlab-rails) later. + +### Standalone Redis using Omnibus GitLab + +The Omnibus GitLab package can be used to configure a standalone Redis server. +The steps below are the minimum necessary to configure a Redis server with +Omnibus: + +1. SSH into the Redis server. +1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + - Do not complete any other steps on the download page. + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + ## Enable Redis + redis['enable'] = true + + ## Disable all other services + sidekiq['enable'] = false + gitlab_workhorse['enable'] = false + puma['enable'] = false + unicorn['enable'] = false + postgresql['enable'] = false + nginx['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + pgbouncer_exporter['enable'] = false + gitlab_exporter['enable'] = false + gitaly['enable'] = false + grafana['enable'] = false + + redis['bind'] = '0.0.0.0' + redis['port'] = 6379 + redis['password'] = 'SECRET_PASSWORD_HERE' + + gitlab_rails['enable'] = false + + # Set the network addresses that the exporters used for monitoring will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + redis_exporter['listen_address'] = '0.0.0.0:9121' + redis_exporter['flags'] = { + 'redis.addr' => 'redis://0.0.0.0:6379', + 'redis.password' => 'SECRET_PASSWORD_HERE', + } + ``` + +1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Note the Redis node's IP address or hostname, port, and + Redis password. These will be necessary when [configuring the GitLab + application servers](#configure-gitlab-rails) later. + +Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) +are supported and can be added if needed. + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + +## Configure Gitaly + +Deploying Gitaly in its own server can benefit GitLab installations that are +larger than a single machine. Gitaly node requirements are dependent on data, +specifically the number of projects and their sizes. It's recommended that each +Gitaly node store no more than 5TB of data. Your 2K setup may require one or more +nodes depending on your repository storage requirements. + +We strongly recommend that all Gitaly nodes should be set up with SSD disks with a throughput of at least +8,000 IOPS for read operations and 2,000 IOPS for write, as Gitaly has heavy I/O. +These IOPS values are recommended only as a starter as with time they may be +adjusted higher or lower depending on the scale of your environment's workload. +If you're running the environment on a Cloud provider +you may need to refer to their documentation on how configure IOPS correctly. + +Some things to note: + +- The GitLab Rails application shards repositories into [repository storages](../repository_storage_paths.md). +- A Gitaly server can host one or more storages. +- A GitLab server can use one or more Gitaly servers. +- Gitaly addresses must be specified in such a way that they resolve + correctly for ALL Gitaly clients. +- Gitaly servers must not be exposed to the public internet, as Gitaly's network + traffic is unencrypted by default. The use of a firewall is highly recommended + to restrict access to the Gitaly server. Another option is to + [use TLS](#gitaly-tls-support). + +TIP: **Tip:** +For more information about Gitaly's history and network architecture see the +[standalone Gitaly documentation](../gitaly/index.md). + +Note: **Note:** The token referred to throughout the Gitaly documentation is +just an arbitrary password selected by the administrator. It is unrelated to +tokens created for the GitLab API or other similar web API tokens. + +Below we describe how to configure one Gitaly server `gitaly1.internal` with +secret token `gitalysecret`. We assume your GitLab installation has two +repository storages: `default` and `storage1`. + +To configure the Gitaly server: + +1. [Download/Install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page but + **without** providing the `EXTERNAL_URL` value. +1. Edit `/etc/gitlab/gitlab.rb` to configure storage paths, enable + the network listener and configure the token: + + <!-- + updates to following example must also be made at + https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab + --> + + ```ruby + # /etc/gitlab/gitlab.rb + + # Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests + # to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API. + # The following two values must be the same as their respective values + # of the GitLab Rails application setup + gitaly['auth_token'] = 'gitlaysecret' + gitlab_shell['secret_token'] = 'shellsecret' + + # Avoid running unnecessary services on the Gitaly server + postgresql['enable'] = false + redis['enable'] = false + nginx['enable'] = false + puma['enable'] = false + unicorn['enable'] = false + sidekiq['enable'] = false + gitlab_workhorse['enable'] = false + grafana['enable'] = false + + # If you run a seperate monitoring node you can disable these services + alertmanager['enable'] = false + prometheus['enable'] = false + + # Prevent database connections during 'gitlab-ctl reconfigure' + gitlab_rails['rake_cache_clear'] = false + gitlab_rails['auto_migrate'] = false + + # Configure the gitlab-shell API callback URL. Without this, `git push` will + # fail. This can be your 'front door' GitLab URL or an internal load + # balancer. + # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server. + gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' + + # Make Gitaly accept connections on all network interfaces. You must use + # firewalls to restrict access to this address/port. + # Comment out following line if you only want to support TLS connections + gitaly['listen_addr'] = "0.0.0.0:8075" + gitaly['prometheus_listen_addr'] = "0.0.0.0:9236" + + # Set the network addresses that the exporters used for monitoring will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + ``` + +1. Append the following to `/etc/gitlab/gitlab.rb` on `gitaly1.internal`: + + ```ruby + git_data_dirs({ + 'default' => { + 'path' => '/var/opt/gitlab/git-data' + }, + 'storage1' => { + 'path' => '/mnt/gitlab/git-data' + }, + }) + ``` + + <!-- + updates to following example must also be made at + https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab + --> + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Confirm that Gitaly can perform callbacks to the internal API: + + ```shell + sudo /opt/gitlab/embedded/service/gitlab-shell/bin/check -config /opt/gitlab/embedded/service/gitlab-shell/config.yml + ``` + +### Gitaly TLS support + +Gitaly supports TLS encryption. To be able to communicate +with a Gitaly instance that listens for secure connections you will need to use `tls://` URL +scheme in the `gitaly_address` of the corresponding storage entry in the GitLab configuration. + +You will need to bring your own certificates as this isn't provided automatically. +The certificate, or its certificate authority, must be installed on all Gitaly +nodes (including the Gitaly node using the certificate) and on all client nodes +that communicate with it following the procedure described in +[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). + +NOTE: **Note** +The self-signed certificate must specify the address you use to access the +Gitaly server. If you are addressing the Gitaly server by a hostname, you can +either use the Common Name field for this, or add it as a Subject Alternative +Name. If you are addressing the Gitaly server by its IP address, you must add it +as a Subject Alternative Name to the certificate. +[gRPC does not support using an IP address as Common Name in a certificate](https://github.com/grpc/grpc/issues/2691). + +NOTE: **Note:** +It is possible to configure Gitaly servers with both an +unencrypted listening address `listen_addr` and an encrypted listening +address `tls_listen_addr` at the same time. This allows you to do a +gradual transition from unencrypted to encrypted traffic, if necessary. + +To configure Gitaly with TLS: + +1. Create the `/etc/gitlab/ssl` directory and copy your key and certificate there: + + ```shell + sudo mkdir -p /etc/gitlab/ssl + sudo chmod 755 /etc/gitlab/ssl + sudo cp key.pem cert.pem /etc/gitlab/ssl/ + sudo chmod 644 key.pem cert.pem + ``` + +1. Copy the cert to `/etc/gitlab/trusted-certs` so Gitaly will trust the cert when + calling into itself: + + ```shell + sudo cp /etc/gitlab/ssl/cert.pem /etc/gitlab/trusted-certs/ + ``` + +1. Edit `/etc/gitlab/gitlab.rb` and add: + + <!-- + updates to following example must also be made at + https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab + --> + + ```ruby + gitaly['tls_listen_addr'] = "0.0.0.0:9999" + gitaly['certificate_path'] = "/etc/gitlab/ssl/cert.pem" + gitaly['key_path'] = "/etc/gitlab/ssl/key.pem" + ``` + +1. Delete `gitaly['listen_addr']` to allow only encrypted connections. +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + +## Configure GitLab Rails + +NOTE: **Note:** +In our architectures we run each GitLab Rails node using the Puma webserver +and have its number of workers set to 90% of available CPUs along with four threads. For +nodes that are running Rails with other components the worker value should be reduced +accordingly where we've found 50% achieves a good balance but this is dependent +on workload. + +This section describes how to configure the GitLab application (Rails) component. +On each node perform the following: + +1. If you're [using NFS](#configure-nfs-optional): + + 1. If necessary, install the NFS client utility packages using the following + commands: + + ```shell + # Ubuntu/Debian + apt-get install nfs-common + + # CentOS/Red Hat + yum install nfs-utils nfs-utils-lib + ``` + + 1. Specify the necessary NFS mounts in `/etc/fstab`. + The exact contents of `/etc/fstab` will depend on how you chose + to configure your NFS server. See the [NFS documentation](../high_availability/nfs.md) + for examples and the various options. + + 1. Create the shared directories. These may be different depending on your NFS + mount locations. + + ```shell + mkdir -p /var/opt/gitlab/.ssh /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/git-data + ``` + +1. Download/install Omnibus GitLab using **steps 1 and 2** from + [GitLab downloads](https://about.gitlab.com/install/). Do not complete other + steps on the download page. +1. Create/edit `/etc/gitlab/gitlab.rb` and use the following configuration. + To maintain uniformity of links across nodes, the `external_url` + on the application server should point to the external URL that users will use + to access GitLab. This would be the URL of the [load balancer](#configure-the-load-balancer) + which will route traffic to the GitLab application server: + + ```ruby + external_url 'https://gitlab.example.com' + + # Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests + # to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API. + # The following two values must be the same as their respective values + # of the Gitaly setup + gitlab_rails['gitaly_token'] = 'gitalyecret' + gitlab_shell['secret_token'] = 'shellsecret' + + git_data_dirs({ + 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, + 'storage1' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, + 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, + }) + + ## Disable components that will not be on the GitLab application server + roles ['application_role'] + gitaly['enable'] = false + nginx['enable'] = true + + ## PostgreSQL connection details + gitlab_rails['db_adapter'] = 'postgresql' + gitlab_rails['db_encoding'] = 'unicode' + gitlab_rails['db_host'] = '10.1.0.5' # IP/hostname of database server + gitlab_rails['db_password'] = 'DB password' + + ## Redis connection details + gitlab_rails['redis_port'] = '6379' + gitlab_rails['redis_host'] = '10.1.0.6' # IP/hostname of Redis server + gitlab_rails['redis_password'] = 'Redis Password' + + # Set the network addresses that the exporters used for monitoring will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + gitlab_workhorse['prometheus_listen_addr'] = '0.0.0.0:9229' + sidekiq['listen_address'] = "0.0.0.0" + puma['listen'] = '0.0.0.0' + + # Add the monitoring node's IP address to the monitoring whitelist and allow it to + # scrape the NGINX metrics. Replace placeholder `monitoring.gitlab.example.com` with + # the address and/or subnets gathered from the monitoring node + gitlab_rails['monitoring_whitelist'] = ['<MONITOR NODE IP>/32', '127.0.0.0/8'] + nginx['status']['options']['allow'] = ['<MONITOR NODE IP>/32', '127.0.0.0/8'] + + ## Uncomment and edit the following options if you have set up NFS + ## + ## Prevent GitLab from starting if NFS data mounts are not available + ## + #high_availability['mountpoint'] = '/var/opt/gitlab/git-data' + ## + ## Ensure UIDs and GIDs match between servers for permissions via NFS + ## + #user['uid'] = 9000 + #user['gid'] = 9000 + #web_server['uid'] = 9001 + #web_server['gid'] = 9001 + #registry['uid'] = 9002 + #registry['gid'] = 9002 + ``` + +1. If you're using [Gitaly with TLS support](#gitaly-tls-support), make sure the + `git_data_dirs` entry is configured with `tls` instead of `tcp`: + + ```ruby + git_data_dirs({ + 'default' => { 'gitaly_address' => 'tls://gitaly1.internal:9999' }, + 'storage1' => { 'gitaly_address' => 'tls://gitaly1.internal:9999' }, + 'storage2' => { 'gitaly_address' => 'tls://gitaly2.internal:9999' }, + }) + ``` + + 1. Copy the cert into `/etc/gitlab/trusted-certs`: + + ```shell + sudo cp cert.pem /etc/gitlab/trusted-certs/ + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Run `sudo gitlab-rake gitlab:gitaly:check` to confirm the node can connect to Gitaly. +1. Tail the logs to see the requests: + + ```shell + sudo gitlab-ctl tail gitaly + ``` + +NOTE: **Note:** When you specify `https` in the `external_url`, as in the example +above, GitLab assumes you have SSL certificates in `/etc/gitlab/ssl/`. If +certificates are not present, NGINX will fail to start. See the +[NGINX documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) +for more information. + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + +## Configure Prometheus + +The Omnibus GitLab package can be used to configure a standalone Monitoring node +running [Prometheus](../monitoring/prometheus/index.md) and +[Grafana](../monitoring/performance/grafana_configuration.md): + +1. SSH into the Monitoring node. +1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + Do not complete any other steps on the download page. +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + external_url 'http://gitlab.example.com' + + # Enable Prometheus + prometheus['enable'] = true + prometheus['listen_address'] = '0.0.0.0:9090' + prometheus['monitor_kubernetes'] = false + + # Enable Login form + grafana['disable_login_form'] = false + + # Enable Grafana + grafana['enable'] = true + grafana['admin_password'] = 'toomanysecrets' + + # Disable all other services + gitlab_rails['auto_migrate'] = false + alertmanager['enable'] = false + gitaly['enable'] = false + gitlab_exporter['enable'] = false + gitlab_workhorse['enable'] = false + nginx['enable'] = true + postgres_exporter['enable'] = false + postgresql['enable'] = false + redis['enable'] = false + redis_exporter['enable'] = false + sidekiq['enable'] = false + puma['enable'] = false + unicorn['enable'] = false + node_exporter['enable'] = false + gitlab_exporter['enable'] = false + ``` + +1. Prometheus also needs some scrape configs to pull all the data from the various + nodes where we configured exporters. Assuming that your nodes' IPs are: + + ```plaintext + 1.1.1.1: postgres + 1.1.1.2: redis + 1.1.1.3: gitaly1 + 1.1.1.4: rails1 + 1.1.1.5: rails2 + ``` + + Add the following to `/etc/gitlab/gitlab.rb`: + + ```ruby + prometheus['scrape_configs'] = [ + { + 'job_name': 'postgres', + 'static_configs' => [ + 'targets' => ['1.1.1.1:9187'], + ], + }, + { + 'job_name': 'redis', + 'static_configs' => [ + 'targets' => ['1.1.1.2:9121'], + ], + }, + { + 'job_name': 'gitaly', + 'static_configs' => [ + 'targets' => ['1.1.1.3:9236'], + ], + }, + { + 'job_name': 'gitlab-nginx', + 'static_configs' => [ + 'targets' => ['1.1.1.4:8060', '1.1.1.5:8060'], + ], + }, + { + 'job_name': 'gitlab-workhorse', + 'static_configs' => [ + 'targets' => ['1.1.1.4:9229', '1.1.1.5:9229'], + ], + }, + { + 'job_name': 'gitlab-rails', + 'metrics_path': '/-/metrics', + 'static_configs' => [ + 'targets' => ['1.1.1.4:8080', '1.1.1.5:8080'], + ], + }, + { + 'job_name': 'gitlab-sidekiq', + 'static_configs' => [ + 'targets' => ['1.1.1.4:8082', '1.1.1.5:8082'], + ], + }, + { + 'job_name': 'node', + 'static_configs' => [ + 'targets' => ['1.1.1.1:9100', '1.1.1.2:9100', '1.1.1.3:9100', '1.1.1.4:9100', '1.1.1.5:9100'], + ], + }, + ] + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. In the GitLab UI, set `admin/application_settings/metrics_and_profiling` > Metrics - Grafana to `/-/grafana` to +`http[s]://<MONITOR NODE>/-/grafana` + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + +## Troubleshooting + +See the [troubleshooting documentation](troubleshooting.md). + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index bd429fbc4b4..efeed3e9ffd 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -14,20 +14,20 @@ following the [2,000-user reference architecture](2k_users.md). > - **High Availability:** True > - **Test RPS rates:** API: 60 RPS, Web: 6 RPS, Git: 6 RPS -| Service | Nodes | Configuration ([8](#footnotes)) | GCP | AWS ([9](#footnotes)) | Azure([9](#footnotes)) | -|--------------------------------------------------------------|-------|---------------------------------|---------------|-----------------------|------------------------| -| GitLab Rails ([1](#footnotes)) | 3 | 8 vCPU, 7.2GB Memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | -| PostgreSQL | 3 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | D2s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | -| Gitaly ([2](#footnotes)) ([5](#footnotes)) ([7](#footnotes)) | X | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis ([3](#footnotes)) | 3 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | D2s v3 | -| Consul + Sentinel ([3](#footnotes)) | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | -| Sidekiq | 4 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | D2s v3 | +| Service | Nodes | Configuration ([8](#footnotes)) | GCP | AWS | Azure | +|--------------------------------------------------------------|-------|---------------------------------|---------------|-----------------------|----------------| +| GitLab Rails ([1](#footnotes)) | 3 | 8 vCPU, 7.2GB Memory | `n1-highcpu-8` | `c5.2xlarge` | F8s v2 | +| PostgreSQL | 3 | 2 vCPU, 7.5GB Memory | `n1-standard-2` | `m5.large` | D2s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | F2s v2 | +| Gitaly ([2](#footnotes)) ([5](#footnotes)) ([7](#footnotes)) | X | 4 vCPU, 15GB Memory | `n1-standard-4` | `m5.xlarge` | D4s v3 | +| Redis ([3](#footnotes)) | 3 | 2 vCPU, 7.5GB Memory | `n1-standard-2` | `m5.large` | D2s v3 | +| Consul + Sentinel ([3](#footnotes)) | 3 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | F2s v2 | +| Sidekiq | 4 | 2 vCPU, 7.5GB Memory | `n1-standard-2` | `m5.large` | D2s v3 | | Object Storage ([4](#footnotes)) | - | - | - | - | - | -| NFS Server ([5](#footnotes)) ([7](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | F4s v2 | -| Monitoring node | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | -| External load balancing node ([6](#footnotes)) | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | -| Internal load balancing node ([6](#footnotes)) | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | +| NFS Server ([5](#footnotes)) ([7](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | `n1-highcpu-4` | `c5.xlarge` | F4s v2 | +| Monitoring node | 1 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | F2s v2 | +| External load balancing node ([6](#footnotes)) | 1 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | F2s v2 | +| Internal load balancing node ([6](#footnotes)) | 1 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | F2s v2 | ## Footnotes @@ -59,7 +59,7 @@ following the [2,000-user reference architecture](2k_users.md). 1. NFS can be used as an alternative for both repository data (replacing Gitaly) and object storage but this isn't typically recommended for performance reasons. Note however it is required for - [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/issues/196). + [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/196). 1. Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) as the load balancer. Although other load balancers with similar feature sets @@ -77,6 +77,3 @@ following the [2,000-user reference architecture](2k_users.md). or higher, are required for your CPU or Node counts accordingly. For more information, a [Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found [here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). - -1. AWS-equivalent and Azure-equivalent configurations are rough suggestions - and may change in the future. They have not yet been tested and validated. diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 67f773a021f..dd94f5470b4 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -8,23 +8,23 @@ For a full list of reference architectures, see > - **High Availability:** True > - **Test RPS rates:** API: 1000 RPS, Web: 100 RPS, Git: 100 RPS -| Service | Nodes | Configuration ([8](#footnotes)) | GCP | AWS ([9](#footnotes)) | Azure([9](#footnotes)) | -|--------------------------------------------------------------|-------|---------------------------------|----------------|-----------------------|------------------------| -| GitLab Rails ([1](#footnotes)) | 12 | 32 vCPU, 28.8GB Memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | -| PostgreSQL | 3 | 16 vCPU, 60GB Memory | n1-standard-16 | m5.4xlarge | D16s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | -| Gitaly ([2](#footnotes)) ([5](#footnotes)) ([7](#footnotes)) | X | 64 vCPU, 240GB Memory | n1-standard-64 | m5.16xlarge | D64s v3 | -| Redis ([3](#footnotes)) - Cache | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis ([3](#footnotes)) - Queues / Shared State | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis Sentinel ([3](#footnotes)) - Cache | 3 | 1 vCPU, 1.7GB Memory | g1-small | t2.small | B1MS | -| Redis Sentinel ([3](#footnotes)) - Queues / Shared State | 3 | 1 vCPU, 1.7GB Memory | g1-small | t2.small | B1MS | -| Consul | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | -| Sidekiq | 4 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | -| NFS Server ([5](#footnotes)) ([7](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| Service | Nodes | Configuration ([8](#footnotes)) | GCP | AWS | Azure | +|--------------------------------------------------------------|-------|---------------------------------|----------------|-----------------------|----------------| +| GitLab Rails ([1](#footnotes)) | 12 | 32 vCPU, 28.8GB Memory | `n1-highcpu-32` | `c5.9xlarge` | F32s v2 | +| PostgreSQL | 3 | 16 vCPU, 60GB Memory | `n1-standard-16` | `m5.4xlarge` | D16s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | F2s v2 | +| Gitaly ([2](#footnotes)) ([5](#footnotes)) ([7](#footnotes)) | X | 64 vCPU, 240GB Memory | `n1-standard-64` | `m5.16xlarge` | D64s v3 | +| Redis ([3](#footnotes)) - Cache | 3 | 4 vCPU, 15GB Memory | `n1-standard-4` | `m5.xlarge` | D4s v3 | +| Redis ([3](#footnotes)) - Queues / Shared State | 3 | 4 vCPU, 15GB Memory | `n1-standard-4` | `m5.xlarge` | D4s v3 | +| Redis Sentinel ([3](#footnotes)) - Cache | 3 | 1 vCPU, 1.7GB Memory | `g1-small` | `t2.small` | B1MS | +| Redis Sentinel ([3](#footnotes)) - Queues / Shared State | 3 | 1 vCPU, 1.7GB Memory | `g1-small` | `t2.small` | B1MS | +| Consul | 3 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | F2s v2 | +| Sidekiq | 4 | 4 vCPU, 15GB Memory | `n1-standard-4` | `m5.xlarge` | D4s v3 | +| NFS Server ([5](#footnotes)) ([7](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | `n1-highcpu-4` | `c5.xlarge` | F4s v2 | | Object Storage ([4](#footnotes)) | - | - | - | - | - | -| Monitoring node | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | F4s v2 | -| External load balancing node ([6](#footnotes)) | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | -| Internal load balancing node ([6](#footnotes)) | 1 | 8 vCPU, 7.2GB Memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | +| Monitoring node | 1 | 4 vCPU, 3.6GB Memory | `n1-highcpu-4` | `c5.xlarge` | F4s v2 | +| External load balancing node ([6](#footnotes)) | 1 | 8 vCPU, 7.2GB Memory | `n1-highcpu-8` | `c5.2xlarge` | F8s v2 | +| Internal load balancing node ([6](#footnotes)) | 1 | 8 vCPU, 7.2GB Memory | `n1-highcpu-8` | `c5.2xlarge` | F8s v2 | ## Footnotes @@ -56,7 +56,7 @@ For a full list of reference architectures, see 1. NFS can be used as an alternative for both repository data (replacing Gitaly) and object storage but this isn't typically recommended for performance reasons. Note however it is required for - [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/issues/196). + [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/196). 1. Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) as the load balancer. Although other load balancers with similar feature sets @@ -74,6 +74,3 @@ For a full list of reference architectures, see or higher, are required for your CPU or Node counts accordingly. For more information, a [Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found [here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). - -1. AWS-equivalent and Azure-equivalent configurations are rough suggestions - and may change in the future. They have not yet been tested and validated. diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 41ef6f369c2..604572b083e 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -8,20 +8,20 @@ For a full list of reference architectures, see > - **High Availability:** True > - **Test RPS rates:** API: 100 RPS, Web: 10 RPS, Git: 10 RPS -| Service | Nodes | Configuration ([8](#footnotes)) | GCP | AWS ([9](#footnotes)) | Azure([9](#footnotes)) | -|--------------------------------------------------------------|-------|---------------------------------|---------------|-----------------------|------------------------| -| GitLab Rails ([1](#footnotes)) | 3 | 16 vCPU, 14.4GB Memory | n1-highcpu-16 | c5.4xlarge | F16s v2 | -| PostgreSQL | 3 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | D2s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | -| Gitaly ([2](#footnotes)) ([5](#footnotes)) ([7](#footnotes)) | X | 8 vCPU, 30GB Memory | n1-standard-8 | m5.2xlarge | D8s v3 | -| Redis ([3](#footnotes)) | 3 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | D2s v3 | -| Consul + Sentinel ([3](#footnotes)) | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | -| Sidekiq | 4 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | D2s v3 | +| Service | Nodes | Configuration ([8](#footnotes)) | GCP | AWS | Azure | +|--------------------------------------------------------------|-------|---------------------------------|---------------|-----------------------|----------------| +| GitLab Rails ([1](#footnotes)) | 3 | 16 vCPU, 14.4GB Memory | `n1-highcpu-16` | `c5.4xlarge` | F16s v2 | +| PostgreSQL | 3 | 2 vCPU, 7.5GB Memory | `n1-standard-2` | `m5.large` | D2s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | F2s v2 | +| Gitaly ([2](#footnotes)) ([5](#footnotes)) ([7](#footnotes)) | X | 8 vCPU, 30GB Memory | `n1-standard-8` | `m5.2xlarge` | D8s v3 | +| Redis ([3](#footnotes)) | 3 | 2 vCPU, 7.5GB Memory | `n1-standard-2` | `m5.large` | D2s v3 | +| Consul + Sentinel ([3](#footnotes)) | 3 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | F2s v2 | +| Sidekiq | 4 | 2 vCPU, 7.5GB Memory | `n1-standard-2` | `m5.large` | D2s v3 | | Object Storage ([4](#footnotes)) | - | - | - | - | - | -| NFS Server ([5](#footnotes)) ([7](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | F4s v2 | -| Monitoring node | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | -| External load balancing node ([6](#footnotes)) | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | -| Internal load balancing node ([6](#footnotes)) | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | +| NFS Server ([5](#footnotes)) ([7](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | `n1-highcpu-4` | `c5.xlarge` | F4s v2 | +| Monitoring node | 1 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | F2s v2 | +| External load balancing node ([6](#footnotes)) | 1 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | F2s v2 | +| Internal load balancing node ([6](#footnotes)) | 1 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | F2s v2 | ## Footnotes @@ -53,7 +53,7 @@ For a full list of reference architectures, see 1. NFS can be used as an alternative for both repository data (replacing Gitaly) and object storage but this isn't typically recommended for performance reasons. Note however it is required for - [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/issues/196). + [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/196). 1. Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) as the load balancer. Although other load balancers with similar feature sets @@ -71,6 +71,3 @@ For a full list of reference architectures, see or higher, are required for your CPU or Node counts accordingly. For more information, a [Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found [here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). - -1. AWS-equivalent and Azure-equivalent configurations are rough suggestions - and may change in the future. They have not yet been tested and validated. diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index 26244368234..623d7f3f776 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -126,8 +126,8 @@ As long as at least one of each component is online and capable of handling the By adding automatic failover for database systems, you can enable higher uptime with additional database nodes. This extends the default database with cluster management and failover policies. -[PgBouncer](../../development/architecture.md#pgbouncer) in conjunction with -[Repmgr](../high_availability/database.md) is recommended. +[PgBouncer in conjunction with Repmgr](../postgresql/replication_and_failover.md) +is recommended. ### Instance level replication with GitLab Geo **(PREMIUM ONLY)** @@ -141,6 +141,9 @@ that can also be promoted in case of disaster. ## Configure GitLab to scale +NOTE: **Note:** +From GitLab 13.0, using NFS for Git repositories is deprecated. In GitLab 14.0, support for NFS for Git repositories is scheduled to be removed. Upgrade to [Gitaly Cluster](../gitaly/praefect.md) as soon as possible. + The following components are the ones you need to configure in order to scale GitLab. They are listed in the order you'll typically configure them if they are required by your [reference architecture](#reference-architectures) of choice. @@ -160,14 +163,33 @@ column. | [Consul](../../development/architecture.md#consul) ([3](#footnotes)) | Service discovery and health checks/failover | [Consul configuration](../high_availability/consul.md) **(PREMIUM ONLY)** | Yes | | [PostgreSQL](../../development/architecture.md#postgresql) | Database | [PostgreSQL configuration](https://docs.gitlab.com/omnibus/settings/database.html) | Yes | | [PgBouncer](../../development/architecture.md#pgbouncer) | Database connection pooler | [PgBouncer configuration](../high_availability/pgbouncer.md#running-pgbouncer-as-part-of-a-non-ha-gitlab-installation) **(PREMIUM ONLY)** | Yes | -| Repmgr | PostgreSQL cluster management and failover | [PostgreSQL and Repmgr configuration](../high_availability/database.md) | Yes | +| Repmgr | PostgreSQL cluster management and failover | [PostgreSQL and Repmgr configuration](../postgresql/replication_and_failover.md) | Yes | | [Redis](../../development/architecture.md#redis) ([3](#footnotes)) | Key/value store for fast data lookup and caching | [Redis configuration](../high_availability/redis.md) | Yes | | Redis Sentinel | Redis | [Redis Sentinel configuration](../high_availability/redis.md) | Yes | -| [Gitaly](../../development/architecture.md#gitaly) ([2](#footnotes)) ([7](#footnotes)) ([10](#footnotes)) | Provides access to Git repositories | [Gitaly configuration](../gitaly/index.md#running-gitaly-on-its-own-server) | Yes | +| [Gitaly](../../development/architecture.md#gitaly) ([2](#footnotes)) ([7](#footnotes)) | Provides access to Git repositories | [Gitaly configuration](../gitaly/index.md#run-gitaly-on-its-own-server) | Yes | | [Sidekiq](../../development/architecture.md#sidekiq) | Asynchronous/background jobs | [Sidekiq configuration](../high_availability/sidekiq.md) | Yes | | [GitLab application services](../../development/architecture.md#unicorn)([1](#footnotes)) | Puma/Unicorn, Workhorse, GitLab Shell - serves front-end requests (UI, API, Git over HTTP/SSH) | [GitLab app scaling configuration](../high_availability/gitlab.md) | Yes | | [Prometheus](../../development/architecture.md#prometheus) and [Grafana](../../development/architecture.md#grafana) | GitLab environment monitoring | [Monitoring node for scaling](../high_availability/monitoring_node.md) | Yes | +### Configuring select components with Cloud Native Helm + +We also provide [Helm charts](https://docs.gitlab.com/charts/) as a Cloud Native installation +method for GitLab. For the reference architectures, select components can be set up in this +way as an alternative if so desired. + +For these kind of setups we support using the charts in an [advanced configuration](https://docs.gitlab.com/charts/#advanced-configuration) +where stateful backend components, such as the database or Gitaly, are run externally - either +via Omnibus or reputable third party services. Note that we don't currently support running the +stateful components via Helm _at large scales_. + +When designing these environments you should refer to the respective [Reference Architecture](#available-reference-architectures) +above for guidance on sizing. Components run via Helm would be similarly scaled to their Omnibus +specs, only translated into Kubernetes resources. + +For example, if you were to set up a 50k installation with the Rails nodes being run in Helm, +then the same amount of resources as given for Omnibus should be given to the Kubernetes +cluster with the Rails nodes broken down into a number of smaller Pods across that cluster. + ## Footnotes 1. In our architectures we run each GitLab Rails node using the Puma webserver @@ -177,9 +199,7 @@ column. on workload. 1. Gitaly node requirements are dependent on customer data, specifically the number of - projects and their sizes. We recommend two nodes as an absolute minimum, - and at least four nodes should be used when supporting 50,000 or more users. - We also recommend that each Gitaly node should store no more than 5TB of data + projects and their sizes. We recommend that each Gitaly node should store no more than 5TB of data and have the number of [`gitaly-ruby` workers](../gitaly/index.md#gitaly-ruby) set to 20% of available CPUs. Additional nodes should be considered in conjunction with a review of expected data size and spread based on the recommendations above. @@ -198,7 +218,7 @@ column. 1. NFS can be used as an alternative for object storage but this isn't typically recommended for performance reasons. Note however it is required for [GitLab - Pages](https://gitlab.com/gitlab-org/gitlab-pages/issues/196). + Pages](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/196). 1. Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) as the load balancer. Although other load balancers with similar feature sets @@ -216,10 +236,3 @@ column. or higher, are required for your CPU or Node counts accordingly. For more information, a [Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found [here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). - -1. AWS-equivalent and Azure-equivalent configurations are rough suggestions - and may change in the future. They have not yet been tested and validated. - -1. From GitLab 13.0, using NFS for Git repositories is deprecated. In GitLab - 14.0, support for NFS for Git repositories is scheduled to be removed. - Upgrade to [Gitaly Cluster](../gitaly/praefect.md) as soon as possible. diff --git a/doc/administration/reference_architectures/troubleshooting.md b/doc/administration/reference_architectures/troubleshooting.md new file mode 100644 index 00000000000..15e377fe183 --- /dev/null +++ b/doc/administration/reference_architectures/troubleshooting.md @@ -0,0 +1,329 @@ +# Troubleshooting a reference architecture set up + +This page serves as the troubleshooting documentation if you followed one of +the [reference architectures](index.md#reference-architectures). + +## Troubleshooting object storage + +### S3 API compatibility issues + +Not all S3 providers [are fully compatible](../../raketasks/backup_restore.md#other-s3-providers) +with the Fog library that GitLab uses. Symptoms include: + +```plaintext +411 Length Required +``` + +### GitLab Pages requires NFS + +If you intend to use [GitLab Pages](../../user/project/pages/index.md), this currently requires +[NFS](../high_availability/nfs.md). There is [work in progress](https://gitlab.com/gitlab-org/gitlab-pages/issues/196) +to remove this dependency. In the future, GitLab Pages may use +[object storage](https://gitlab.com/gitlab-org/gitlab/-/issues/208135). + +The dependency on disk storage also prevents Pages being deployed using the +[GitLab Helm chart](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/37). + +### Incremental logging is required for CI to use object storage + +If you configure GitLab to use object storage for CI logs and artifacts, +[you must also enable incremental logging](../job_logs.md#new-incremental-logging-architecture). + +### Proxy Download + +A number of the use cases for object storage allow client traffic to be redirected to the +object storage back end, like when Git clients request large files via LFS or when +downloading CI artifacts and logs. + +When the files are stored on local block storage or NFS, GitLab has to act as a proxy. +With object storage, the default behavior is for GitLab to redirect to the object +storage device rather than proxy the request. + +The `proxy_download` setting controls this behavior: the default is generally `false`. +Verify this in the documentation for each use case. Set it to `true` to make +GitLab proxy the files rather than redirect. + +When not proxying files, GitLab returns an +[HTTP 302 redirect with a pre-signed, time-limited object storage URL](https://gitlab.com/gitlab-org/gitlab/-/issues/32117#note_218532298). +This can result in some of the following problems: + +- If GitLab is using non-secure HTTP to access the object storage, clients may generate +`https->http` downgrade errors and refuse to process the redirect. The solution to this +is for GitLab to use HTTPS. LFS, for example, will generate this error: + + ```plaintext + LFS: lfsapi/client: refusing insecure redirect, https->http + ``` + +- Clients will need to trust the certificate authority that issued the object storage +certificate, or may return common TLS errors such as: + + ```plaintext + x509: certificate signed by unknown authority + ``` + +- Clients will need network access to the object storage. Errors that might result +if this access is not in place include: + + ```plaintext + Received status code 403 from server: Forbidden + ``` + +### ETag mismatch + +Using the default GitLab settings, some object storage back-ends such as +[MinIO](https://gitlab.com/gitlab-org/gitlab/-/issues/23188) +and [Alibaba](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/1564) +might generate `ETag mismatch` errors. + +When using GitLab direct upload, the +[workaround for MinIO](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/1564#note_244497658) +is to use the `--compat` parameter on the server. + +We are working on a fix to GitLab component Workhorse, and also +a workaround, in the mean time, to +[allow ETag verification to be disabled](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18175). + +## Troubleshooting Redis + +If the application node cannot connect to the Redis node, check your firewall rules and +make sure Redis can accept TCP connections under port `6379`. + +## Troubleshooting Gitaly + +### Checking versions when using standalone Gitaly nodes + +When using standalone Gitaly nodes, you must make sure they are the same version +as GitLab to ensure full compatibility. Check **Admin Area > Gitaly Servers** on +your GitLab instance and confirm all Gitaly Servers are `Up to date`. + + + +### `gitaly-debug` + +The `gitaly-debug` command provides "production debugging" tools for Gitaly and Git +performance. It is intended to help production engineers and support +engineers investigate Gitaly performance problems. + +If you're using GitLab 11.6 or newer, this tool should be installed on +your GitLab / Gitaly server already at `/opt/gitlab/embedded/bin/gitaly-debug`. +If you're investigating an older GitLab version you can compile this +tool offline and copy the executable to your server: + +```shell +git clone https://gitlab.com/gitlab-org/gitaly.git +cd cmd/gitaly-debug +GOOS=linux GOARCH=amd64 go build -o gitaly-debug +``` + +To see the help page of `gitaly-debug` for a list of supported sub-commands, run: + +```shell +gitaly-debug -h +``` + +### Commits, pushes, and clones return a 401 + +```plaintext +remote: GitLab: 401 Unauthorized +``` + +You will need to sync your `gitlab-secrets.json` file with your GitLab +app nodes. + +### Client side gRPC logs + +Gitaly uses the [gRPC](https://grpc.io/) RPC framework. The Ruby gRPC +client has its own log file which may contain useful information when +you are seeing Gitaly errors. You can control the log level of the +gRPC client with the `GRPC_LOG_LEVEL` environment variable. The +default level is `WARN`. + +You can run a gRPC trace with: + +```shell +sudo GRPC_TRACE=all GRPC_VERBOSITY=DEBUG gitlab-rake gitlab:gitaly:check +``` + +### Observing `gitaly-ruby` traffic + +[`gitaly-ruby`](../gitaly/index.md#gitaly-ruby) is an internal implementation detail of Gitaly, +so, there's not that much visibility into what goes on inside +`gitaly-ruby` processes. + +If you have Prometheus set up to scrape your Gitaly process, you can see +request rates and error codes for individual RPCs in `gitaly-ruby` by +querying `grpc_client_handled_total`. Strictly speaking, this metric does +not differentiate between `gitaly-ruby` and other RPCs, but in practice +(as of GitLab 11.9), all gRPC calls made by Gitaly itself are internal +calls from the main Gitaly process to one of its `gitaly-ruby` sidecars. + +Assuming your `grpc_client_handled_total` counter only observes Gitaly, +the following query shows you RPCs are (most likely) internally +implemented as calls to `gitaly-ruby`: + +```prometheus +sum(rate(grpc_client_handled_total[5m])) by (grpc_method) > 0 +``` + +### Repository changes fail with a `401 Unauthorized` error + +If you're running Gitaly on its own server and notice that users can +successfully clone and fetch repositories (via both SSH and HTTPS), but can't +push to them or make changes to the repository in the web UI without getting a +`401 Unauthorized` message, then it's possible Gitaly is failing to authenticate +with the other nodes due to having the wrong secrets file. + +Confirm the following are all true: + +- When any user performs a `git push` to any repository on this Gitaly node, it + fails with the following error (note the `401 Unauthorized`): + + ```shell + remote: GitLab: 401 Unauthorized + To <REMOTE_URL> + ! [remote rejected] branch-name -> branch-name (pre-receive hook declined) + error: failed to push some refs to '<REMOTE_URL>' + ``` + +- When any user adds or modifies a file from the repository using the GitLab + UI, it immediately fails with a red `401 Unauthorized` banner. +- Creating a new project and [initializing it with a README](../../gitlab-basics/create-project.md#blank-projects) + successfully creates the project but doesn't create the README. +- When [tailing the logs](https://docs.gitlab.com/omnibus/settings/logs.html#tail-logs-in-a-console-on-the-server) on an app node and reproducing the error, you get `401` errors + when reaching the `/api/v4/internal/allowed` endpoint: + + ```shell + # api_json.log + { + "time": "2019-07-18T00:30:14.967Z", + "severity": "INFO", + "duration": 0.57, + "db": 0, + "view": 0.57, + "status": 401, + "method": "POST", + "path": "\/api\/v4\/internal\/allowed", + "params": [ + { + "key": "action", + "value": "git-receive-pack" + }, + { + "key": "changes", + "value": "REDACTED" + }, + { + "key": "gl_repository", + "value": "REDACTED" + }, + { + "key": "project", + "value": "\/path\/to\/project.git" + }, + { + "key": "protocol", + "value": "web" + }, + { + "key": "env", + "value": "{\"GIT_ALTERNATE_OBJECT_DIRECTORIES\":[],\"GIT_ALTERNATE_OBJECT_DIRECTORIES_RELATIVE\":[],\"GIT_OBJECT_DIRECTORY\":null,\"GIT_OBJECT_DIRECTORY_RELATIVE\":null}" + }, + { + "key": "user_id", + "value": "2" + }, + { + "key": "secret_token", + "value": "[FILTERED]" + } + ], + "host": "gitlab.example.com", + "ip": "REDACTED", + "ua": "Ruby", + "route": "\/api\/:version\/internal\/allowed", + "queue_duration": 4.24, + "gitaly_calls": 0, + "gitaly_duration": 0, + "correlation_id": "XPUZqTukaP3" + } + + # nginx_access.log + [IP] - - [18/Jul/2019:00:30:14 +0000] "POST /api/v4/internal/allowed HTTP/1.1" 401 30 "" "Ruby" + ``` + +To fix this problem, confirm that your `gitlab-secrets.json` file +on the Gitaly node matches the one on all other nodes. If it doesn't match, +update the secrets file on the Gitaly node to match the others, then +[reconfigure the node](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +### Command line tools cannot connect to Gitaly + +If you are having trouble connecting to a Gitaly node with command line (CLI) tools, and certain actions result in a `14: Connect Failed` error message, it means that gRPC cannot reach your Gitaly node. + +Verify that you can reach Gitaly via TCP: + +```shell +sudo gitlab-rake gitlab:tcp_check[GITALY_SERVER_IP,GITALY_LISTEN_PORT] +``` + +If the TCP connection fails, check your network settings and your firewall rules. If the TCP connection succeeds, your networking and firewall rules are correct. + +If you use proxy servers in your command line environment, such as Bash, these can interfere with your gRPC traffic. + +If you use Bash or a compatible command line environment, run the following commands to determine whether you have proxy servers configured: + +```shell +echo $http_proxy +echo $https_proxy +``` + +If either of these variables have a value, your Gitaly CLI connections may be getting routed through a proxy which cannot connect to Gitaly. + +To remove the proxy setting, run the following commands (depending on which variables had values): + +```shell +unset http_proxy +unset https_proxy +``` + +### Gitaly not listening on new address after reconfiguring + +When updating the `gitaly['listen_addr']` or `gitaly['prometheus_listen_addr']` values, Gitaly may continue to listen on the old address after a `sudo gitlab-ctl reconfigure`. + +When this occurs, performing a `sudo gitlab-ctl restart` will resolve the issue. This will no longer be necessary after [this issue](https://gitlab.com/gitlab-org/gitaly/issues/2521) is resolved. + +### Permission denied errors appearing in Gitaly logs when accessing repositories from a standalone Gitaly node + +If this error occurs even though file permissions are correct, it's likely that +the Gitaly node is experiencing +[clock drift](https://en.wikipedia.org/wiki/Clock_drift). + +Please ensure that the GitLab and Gitaly nodes are synchronized and use an NTP time +server to keep them synchronized if possible. + +## Troubleshooting the GitLab Rails application + +- `mount: wrong fs type, bad option, bad superblock on` + +You have not installed the necessary NFS client utilities. See step 1 above. + +- `mount: mount point /var/opt/gitlab/... does not exist` + +This particular directory does not exist on the NFS server. Ensure +the share is exported and exists on the NFS server and try to remount. + +## Troubleshooting Monitoring + +If the monitoring node is not receiving any data, check that the exporters are +capturing data. + +```shell +curl http[s]://localhost:<EXPORTER LISTENING PORT>/metric +``` + +or + +```shell +curl http[s]://localhost:<EXPORTER LISTENING PORT>/-/metric +``` |