diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2020-04-09 00:09:50 +0300 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2020-04-09 00:09:50 +0300 |
commit | 76358aee81a471a5e71eaf3e8c2d91b7c9a0a5a9 (patch) | |
tree | df9ba3dcc09eb404de31e0d79cb8f0b77812e655 /doc | |
parent | 80e9fdc9682cfbcfb9202a2733605a6a6bd23f05 (diff) |
Add latest changes from gitlab-org/gitlab@master
Diffstat (limited to 'doc')
-rw-r--r-- | doc/administration/high_availability/README.md | 143 | ||||
-rw-r--r-- | doc/administration/scaling/index.md | 187 | ||||
-rw-r--r-- | doc/ci/variables/README.md | 4 | ||||
-rw-r--r-- | doc/subscriptions/index.md | 8 | ||||
-rw-r--r-- | doc/user/compliance/license_compliance/index.md | 33 | ||||
-rw-r--r-- | doc/user/group/saml_sso/index.md | 12 | ||||
-rw-r--r-- | doc/user/group/saml_sso/scim_setup.md | 2 | ||||
-rw-r--r-- | doc/user/project/issues/csv_export.md | 2 | ||||
-rw-r--r-- | doc/user/project/issues/index.md | 2 |
9 files changed, 236 insertions, 157 deletions
diff --git a/doc/administration/high_availability/README.md b/doc/administration/high_availability/README.md index 71d483128b9..585c4903692 100644 --- a/doc/administration/high_availability/README.md +++ b/doc/administration/high_availability/README.md @@ -17,7 +17,7 @@ how GitLab can be scaled out and made highly available. These examples progress from simple to complex as scaling or highly-available components are added. For larger setups serving 2,000 or more users, we provide -[reference architectures](#reference-architectures) based on GitLab's +[reference architectures](../scaling/index.md#reference-architectures) based on GitLab's experience with GitLab.com and internal scale testing that aim to achieve the right balance of scalability and availability. @@ -93,147 +93,6 @@ them. In some cases, components can be combined on the same nodes to reduce complexity as well. -## Recommended setups based on number of users - -- 1 - 1000 Users: A single-node [Omnibus](https://docs.gitlab.com/omnibus/) setup with frequent backups. Refer to the [requirements page](../../install/requirements.md) for further details of the specs you will require. -- 1000 - 10000 Users: A scaled environment based on one of our [Reference Architectures](#reference-architectures), without the HA components applied. This can be a reasonable step towards a fully HA environment. -- 2000 - 50000+ Users: A scaled HA environment based on one of our [Reference Architectures](#reference-architectures) below. - -## Reference architectures - -In this section we'll detail the Reference Architectures that can support large numbers -of users. These were built, tested and verified by our Quality and Support teams. - -Testing was done with our GitLab Performance Tool at specific coded workloads, and the -throughputs used for testing were calculated based on sample customer data. We -test each endpoint type with the following number of requests per second (RPS) -per 1000 users: - -- API: 20 RPS -- Web: 2 RPS -- Git: 2 RPS - -NOTE: **Note:** Note that depending on your workflow the below recommended -reference architectures may need to be adapted accordingly. Your workload -is influenced by factors such as - but not limited to - how active your users are, -how much automation you use, mirroring, and repo/change size. Additionally the -shown memory values are given directly by [GCP machine types](https://cloud.google.com/compute/docs/machine-types). -On different cloud vendors a best effort like for like can be used. - -### 2,000 user configuration - -- **Supported users (approximate):** 2,000 -- **Test RPS rates:** API: 40 RPS, Web: 4 RPS, Git: 4 RPS -- **Known issues:** [List of known performance issues](https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=Quality%3Aperformance-issues) - -| Service | Nodes | Configuration[^8] | GCP type | AWS type[^9] | -| ----------------------------|-------|-----------------------|---------------|--------------| -| GitLab Rails[^1] | 3 | 8 vCPU, 7.2GB Memory | n1-highcpu-8 | c5.2xlarge | -| PostgreSQL | 3 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | -| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -| Gitaly[^2] [^5] [^7] | X | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | -| Redis[^3] | 3 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | -| Consul + Sentinel[^3] | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -| Sidekiq | 4 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | -| Cloud Object Storage[^4] | - | - | - | - | -| NFS Server[^5] [^7] | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | -| Monitoring node | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -| External load balancing node[^6] | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -| Internal load balancing node[^6] | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | - -### 5,000 user configuration - -- **Supported users (approximate):** 5,000 -- **Test RPS rates:** API: 100 RPS, Web: 10 RPS, Git: 10 RPS -- **Known issues:** [List of known performance issues](https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=Quality%3Aperformance-issues) - -| Service | Nodes | Configuration[^8] | GCP type | AWS type[^9] | -| ----------------------------|-------|------------------------|---------------|--------------| -| GitLab Rails[^1] | 3 | 16 vCPU, 14.4GB Memory | n1-highcpu-16 | c5.4xlarge | -| PostgreSQL | 3 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | -| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -| Gitaly[^2] [^5] [^7] | X | 8 vCPU, 30GB Memory | n1-standard-8 | m5.2xlarge | -| Redis[^3] | 3 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | -| Consul + Sentinel[^3] | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -| Sidekiq | 4 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | -| Cloud Object Storage[^4] | - | - | - | - | -| NFS Server[^5] [^7] | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | -| Monitoring node | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -| External load balancing node[^6] | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -| Internal load balancing node[^6] | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | - -### 10,000 user configuration - -- **Supported users (approximate):** 10,000 -- **Test RPS rates:** API: 200 RPS, Web: 20 RPS, Git: 20 RPS -- **Known issues:** [List of known performance issues](https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=Quality%3Aperformance-issues) - -| Service | Nodes | GCP Configuration[^8] | GCP type | AWS type[^9] | -| ----------------------------|-------|------------------------|----------------|--------------| -| GitLab Rails[^1] | 3 | 32 vCPU, 28.8GB Memory | n1-highcpu-32 | c5.9xlarge | -| PostgreSQL | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | -| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -| Gitaly[^2] [^5] [^7] | X | 16 vCPU, 60GB Memory | n1-standard-16 | m5.4xlarge | -| Redis[^3] - Cache | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | -| Redis[^3] - Queues / Shared State | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | -| Redis Sentinel[^3] - Cache | 3 | 1 vCPU, 1.7GB Memory | g1-small | t2.small | -| Redis Sentinel[^3] - Queues / Shared State | 3 | 1 vCPU, 1.7GB Memory | g1-small | t2.small | -| Consul | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -| Sidekiq | 4 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | -| Cloud Object Storage[^4] | - | - | - | - | -| NFS Server[^5] [^7] | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | -| Monitoring node | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | -| External load balancing node[^6] | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -| Internal load balancing node[^6] | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | - -### 25,000 user configuration - -- **Supported users (approximate):** 25,000 -- **Test RPS rates:** API: 500 RPS, Web: 50 RPS, Git: 50 RPS -- **Known issues:** [List of known performance issues](https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=Quality%3Aperformance-issues) - -| Service | Nodes | Configuration[^8] | GCP type | AWS type[^9] | -| ----------------------------|-------|------------------------|----------------|--------------| -| GitLab Rails[^1] | 5 | 32 vCPU, 28.8GB Memory | n1-highcpu-32 | c5.9xlarge | -| PostgreSQL | 3 | 8 vCPU, 30GB Memory | n1-standard-8 | m5.2xlarge | -| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -| Gitaly[^2] [^5] [^7] | X | 32 vCPU, 120GB Memory | n1-standard-32 | m5.8xlarge | -| Redis[^3] - Cache | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | -| Redis[^3] - Queues / Shared State | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | -| Redis Sentinel[^3] - Cache | 3 | 1 vCPU, 1.7GB Memory | g1-small | t2.small | -| Redis Sentinel[^3] - Queues / Shared State | 3 | 1 vCPU, 1.7GB Memory | g1-small | t2.small | -| Consul | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -| Sidekiq | 4 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | -| Cloud Object Storage[^4] | - | - | - | - | -| NFS Server[^5] [^7] | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | -| Monitoring node | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | -| External load balancing node[^6] | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -| Internal load balancing node[^6] | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | - -### 50,000 user configuration - -- **Supported users (approximate):** 50,000 -- **Test RPS rates:** API: 1000 RPS, Web: 100 RPS, Git: 100 RPS -- **Known issues:** [List of known performance issues](https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=Quality%3Aperformance-issues) - -| Service | Nodes | Configuration[^8] | GCP type | AWS type[^9] | -| ----------------------------|-------|------------------------|----------------|--------------| -| GitLab Rails[^1] | 12 | 32 vCPU, 28.8GB Memory | n1-highcpu-32 | c5.9xlarge | -| PostgreSQL | 3 | 16 vCPU, 60GB Memory | n1-standard-16 | m5.4xlarge | -| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -| Gitaly[^2] [^5] [^7] | X | 64 vCPU, 240GB Memory | n1-standard-64 | m5.16xlarge | -| Redis[^3] - Cache | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | -| Redis[^3] - Queues / Shared State | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | -| Redis Sentinel[^3] - Cache | 3 | 1 vCPU, 1.7GB Memory | g1-small | t2.small | -| Redis Sentinel[^3] - Queues / Shared State | 3 | 1 vCPU, 1.7GB Memory | g1-small | t2.small | -| Consul | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -| Sidekiq | 4 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | -| NFS Server[^5] [^7] | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | -| Cloud Object Storage[^4] | - | - | - | - | -| Monitoring node | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | -| External load balancing node[^6] | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -| Internal load balancing node[^6] | 1 | 8 vCPU, 7.2GB Memory | n1-highcpu-8 | c5.2xlarge | - [^1]: In our architectures we run each GitLab Rails node using the Puma webserver and have its number of workers set to 90% of available CPUs along with 4 threads. diff --git a/doc/administration/scaling/index.md b/doc/administration/scaling/index.md index 1ef9eca2497..23a96e11805 100644 --- a/doc/administration/scaling/index.md +++ b/doc/administration/scaling/index.md @@ -71,3 +71,190 @@ References: - [Configure your NFS server to work with GitLab](../high_availability/nfs.md) - [Configure packaged PostgreSQL server to listen on TCP/IP](https://docs.gitlab.com/omnibus/settings/database.html#configure-packaged-postgresql-server-to-listen-on-tcpip) - [Setting up a Redis-only server](https://docs.gitlab.com/omnibus/settings/redis.html#setting-up-a-redis-only-server) + +## Recommended setups based on number of users + +- 1 - 1000 Users: A single-node [Omnibus](https://docs.gitlab.com/omnibus/) setup with frequent backups. Refer to the [requirements page](../../install/requirements.md) for further details of the specs you will require. +- 1000 - 10000 Users: A scaled environment based on one of our [Reference Architectures](#reference-architectures), without the HA components applied. This can be a reasonable step towards a fully HA environment. +- 2000 - 50000+ Users: A scaled HA environment based on one of our [Reference Architectures](#reference-architectures) below. + +## Reference architectures + +In this section we'll detail the Reference Architectures that can support large numbers +of users. These were built, tested and verified by our Quality and Support teams. + +Testing was done with our GitLab Performance Tool at specific coded workloads, and the +throughputs used for testing were calculated based on sample customer data. We +test each endpoint type with the following number of requests per second (RPS) +per 1000 users: + +- API: 20 RPS +- Web: 2 RPS +- Git: 2 RPS + +NOTE: **Note:** Note that depending on your workflow the below recommended +reference architectures may need to be adapted accordingly. Your workload +is influenced by factors such as - but not limited to - how active your users are, +how much automation you use, mirroring, and repo/change size. Additionally the +shown memory values are given directly by [GCP machine types](https://cloud.google.com/compute/docs/machine-types). +On different cloud vendors a best effort like for like can be used. + +### 2,000 user configuration + +- **Supported users (approximate):** 2,000 +- **Test RPS rates:** API: 40 RPS, Web: 4 RPS, Git: 4 RPS +- **Known issues:** [List of known performance issues](https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=Quality%3Aperformance-issues) + +| Service | Nodes | Configuration[^8] | GCP type | AWS type[^9] | +| ----------------------------|-------|-----------------------|---------------|--------------| +| GitLab Rails[^1] | 3 | 8 vCPU, 7.2GB Memory | n1-highcpu-8 | c5.2xlarge | +| PostgreSQL | 3 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | +| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | +| Gitaly[^2] [^5] [^7] | X | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | +| Redis[^3] | 3 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | +| Consul + Sentinel[^3] | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | +| Sidekiq | 4 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | +| Cloud Object Storage[^4] | - | - | - | - | +| NFS Server[^5] [^7] | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | +| Monitoring node | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | +| External load balancing node[^6] | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | +| Internal load balancing node[^6] | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | + +### 5,000 user configuration + +- **Supported users (approximate):** 5,000 +- **Test RPS rates:** API: 100 RPS, Web: 10 RPS, Git: 10 RPS +- **Known issues:** [List of known performance issues](https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=Quality%3Aperformance-issues) + +| Service | Nodes | Configuration[^8] | GCP type | AWS type[^9] | +| ----------------------------|-------|------------------------|---------------|--------------| +| GitLab Rails[^1] | 3 | 16 vCPU, 14.4GB Memory | n1-highcpu-16 | c5.4xlarge | +| PostgreSQL | 3 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | +| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | +| Gitaly[^2] [^5] [^7] | X | 8 vCPU, 30GB Memory | n1-standard-8 | m5.2xlarge | +| Redis[^3] | 3 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | +| Consul + Sentinel[^3] | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | +| Sidekiq | 4 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | +| Cloud Object Storage[^4] | - | - | - | - | +| NFS Server[^5] [^7] | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | +| Monitoring node | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | +| External load balancing node[^6] | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | +| Internal load balancing node[^6] | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | + +### 10,000 user configuration + +- **Supported users (approximate):** 10,000 +- **Test RPS rates:** API: 200 RPS, Web: 20 RPS, Git: 20 RPS +- **Known issues:** [List of known performance issues](https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=Quality%3Aperformance-issues) + +| Service | Nodes | GCP Configuration[^8] | GCP type | AWS type[^9] | +| ----------------------------|-------|------------------------|----------------|--------------| +| GitLab Rails[^1] | 3 | 32 vCPU, 28.8GB Memory | n1-highcpu-32 | c5.9xlarge | +| PostgreSQL | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | +| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | +| Gitaly[^2] [^5] [^7] | X | 16 vCPU, 60GB Memory | n1-standard-16 | m5.4xlarge | +| Redis[^3] - Cache | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | +| Redis[^3] - Queues / Shared State | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | +| Redis Sentinel[^3] - Cache | 3 | 1 vCPU, 1.7GB Memory | g1-small | t2.small | +| Redis Sentinel[^3] - Queues / Shared State | 3 | 1 vCPU, 1.7GB Memory | g1-small | t2.small | +| Consul | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | +| Sidekiq | 4 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | +| Cloud Object Storage[^4] | - | - | - | - | +| NFS Server[^5] [^7] | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | +| Monitoring node | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | +| External load balancing node[^6] | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | +| Internal load balancing node[^6] | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | + +### 25,000 user configuration + +- **Supported users (approximate):** 25,000 +- **Test RPS rates:** API: 500 RPS, Web: 50 RPS, Git: 50 RPS +- **Known issues:** [List of known performance issues](https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=Quality%3Aperformance-issues) + +| Service | Nodes | Configuration[^8] | GCP type | AWS type[^9] | +| ----------------------------|-------|------------------------|----------------|--------------| +| GitLab Rails[^1] | 5 | 32 vCPU, 28.8GB Memory | n1-highcpu-32 | c5.9xlarge | +| PostgreSQL | 3 | 8 vCPU, 30GB Memory | n1-standard-8 | m5.2xlarge | +| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | +| Gitaly[^2] [^5] [^7] | X | 32 vCPU, 120GB Memory | n1-standard-32 | m5.8xlarge | +| Redis[^3] - Cache | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | +| Redis[^3] - Queues / Shared State | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | +| Redis Sentinel[^3] - Cache | 3 | 1 vCPU, 1.7GB Memory | g1-small | t2.small | +| Redis Sentinel[^3] - Queues / Shared State | 3 | 1 vCPU, 1.7GB Memory | g1-small | t2.small | +| Consul | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | +| Sidekiq | 4 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | +| Cloud Object Storage[^4] | - | - | - | - | +| NFS Server[^5] [^7] | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | +| Monitoring node | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | +| External load balancing node[^6] | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | +| Internal load balancing node[^6] | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | + +### 50,000 user configuration + +- **Supported users (approximate):** 50,000 +- **Test RPS rates:** API: 1000 RPS, Web: 100 RPS, Git: 100 RPS +- **Known issues:** [List of known performance issues](https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=Quality%3Aperformance-issues) + +| Service | Nodes | Configuration[^8] | GCP type | AWS type[^9] | +| ----------------------------|-------|------------------------|----------------|--------------| +| GitLab Rails[^1] | 12 | 32 vCPU, 28.8GB Memory | n1-highcpu-32 | c5.9xlarge | +| PostgreSQL | 3 | 16 vCPU, 60GB Memory | n1-standard-16 | m5.4xlarge | +| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | +| Gitaly[^2] [^5] [^7] | X | 64 vCPU, 240GB Memory | n1-standard-64 | m5.16xlarge | +| Redis[^3] - Cache | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | +| Redis[^3] - Queues / Shared State | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | +| Redis Sentinel[^3] - Cache | 3 | 1 vCPU, 1.7GB Memory | g1-small | t2.small | +| Redis Sentinel[^3] - Queues / Shared State | 3 | 1 vCPU, 1.7GB Memory | g1-small | t2.small | +| Consul | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | +| Sidekiq | 4 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | +| NFS Server[^5] [^7] | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | +| Cloud Object Storage[^4] | - | - | - | - | +| Monitoring node | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | +| External load balancing node[^6] | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | +| Internal load balancing node[^6] | 1 | 8 vCPU, 7.2GB Memory | n1-highcpu-8 | c5.2xlarge | + +[^1]: In our architectures we run each GitLab Rails node using the Puma webserver + and have its number of workers set to 90% of available CPUs along with 4 threads. + +[^2]: Gitaly node requirements are dependent on customer data, specifically the number of + projects and their sizes. We recommend 2 nodes as an absolute minimum for HA environments + and at least 4 nodes should be used when supporting 50,000 or more users. + We also recommend that each Gitaly node should store no more than 5TB of data + and have the number of [`gitaly-ruby` workers](../gitaly/index.md#gitaly-ruby) + set to 20% of available CPUs. Additional nodes should be considered in conjunction + with a review of expected data size and spread based on the recommendations above. + +[^3]: Recommended Redis setup differs depending on the size of the architecture. + For smaller architectures (up to 5,000 users) we suggest one Redis cluster for all + classes and that Redis Sentinel is hosted alongside Consul. + For larger architectures (10,000 users or more) we suggest running a separate + [Redis Cluster](../high_availability/redis.md#running-multiple-redis-clusters) for the Cache class + and another for the Queues and Shared State classes respectively. We also recommend + that you run the Redis Sentinel clusters separately as well for each Redis Cluster. + +[^4]: For data objects such as LFS, Uploads, Artifacts, etc. We recommend a [Cloud Object Storage service](../object_storage.md) + over NFS where possible, due to better performance and availability. + +[^5]: NFS can be used as an alternative for both repository data (replacing Gitaly) and + object storage but this isn't typically recommended for performance reasons. Note however it is required for + [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/issues/196). + +[^6]: Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) + as the load balancer. However other reputable load balancers with similar feature sets + should also work instead but be aware these aren't validated. + +[^7]: We strongly recommend that any Gitaly and / or NFS nodes are set up with SSD disks over + HDD with a throughput of at least 8,000 IOPS for read operations and 2,000 IOPS for write + as these components have heavy I/O. These IOPS values are recommended only as a starter + as with time they may be adjusted higher or lower depending on the scale of your + environment's workload. If you're running the environment on a Cloud provider + you may need to refer to their documentation on how configure IOPS correctly. + +[^8]: The architectures were built and tested with the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) + CPU platform on GCP. On different hardware you may find that adjustments, either lower + or higher, are required for your CPU or Node counts accordingly. For more information, a + [Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found + [here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). + +[^9]: AWS-equivalent configurations are rough suggestions and may change in the + future. They have not yet been tested and validated. diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index d363a4387d9..2f4408aaa6c 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -715,8 +715,8 @@ if [[ -d "/builds/gitlab-examples/ci-debug-trace/.git" ]]; then ++ CI_SERVER_VERSION_PATCH=0 ++ export CI_SERVER_REVISION=f4cc00ae823 ++ CI_SERVER_REVISION=f4cc00ae823 -++ export GITLAB_FEATURES=audit_events,burndown_charts,code_owners,contribution_analytics,description_diffs,elastic_search,export_issues,group_bulk_edit,group_burndown_charts,group_webhooks,issuable_default_templates,issue_weights,jenkins_integration,ldap_group_sync,member_lock,merge_request_approvers,multiple_issue_assignees,multiple_ldap_servers,multiple_merge_request_assignees,protected_refs_for_users,push_rules,related_issues,repository_mirrors,repository_size_limit,scoped_issue_board,usage_quotas,visual_review_app,wip_limits,adjourned_deletion_for_projects_and_groups,admin_audit_log,auditor_user,batch_comments,blocking_merge_requests,board_assignee_lists,board_milestone_lists,ci_cd_projects,cluster_deployments,code_analytics,code_owner_approval_required,commit_committer_check,cross_project_pipelines,custom_file_templates,custom_file_templates_for_namespace,custom_project_templates,custom_prometheus_metrics,cycle_analytics_for_groups,db_load_balancing,default_project_deletion_protection,dependency_proxy,deploy_board,design_management,email_additional_text,extended_audit_events,external_authorization_service_api_management,feature_flags,file_locks,geo,github_project_service_integration,group_allowed_email_domains,group_project_templates,group_saml,issues_analytics,jira_dev_panel_integration,ldap_group_sync_filter,merge_pipelines,merge_request_performance_metrics,merge_trains,metrics_reports,multiple_approval_rules,multiple_clusters,multiple_group_issue_boards,object_storage,operations_dashboard,packages,productivity_analytics,project_aliases,protected_environments,reject_unsigned_commits,required_ci_templates,scoped_labels,service_desk,smartcard_auth,group_timelogs,type_of_work_analytics,unprotection_restrictions,ci_project_subscriptions,cluster_health,container_scanning,dast,dependency_scanning,epics,group_ip_restriction,incident_management,insights,license_management,personal_access_token_expiration_policy,pod_logs,prometheus_alerts,pseudonymizer,report_approver_rules,sast,security_dashboard,tracing,web_ide_terminal -++ GITLAB_FEATURES=audit_events,burndown_charts,code_owners,contribution_analytics,description_diffs,elastic_search,export_issues,group_bulk_edit,group_burndown_charts,group_webhooks,issuable_default_templates,issue_weights,jenkins_integration,ldap_group_sync,member_lock,merge_request_approvers,multiple_issue_assignees,multiple_ldap_servers,multiple_merge_request_assignees,protected_refs_for_users,push_rules,related_issues,repository_mirrors,repository_size_limit,scoped_issue_board,usage_quotas,visual_review_app,wip_limits,adjourned_deletion_for_projects_and_groups,admin_audit_log,auditor_user,batch_comments,blocking_merge_requests,board_assignee_lists,board_milestone_lists,ci_cd_projects,cluster_deployments,code_analytics,code_owner_approval_required,commit_committer_check,cross_project_pipelines,custom_file_templates,custom_file_templates_for_namespace,custom_project_templates,custom_prometheus_metrics,cycle_analytics_for_groups,db_load_balancing,default_project_deletion_protection,dependency_proxy,deploy_board,design_management,email_additional_text,extended_audit_events,external_authorization_service_api_management,feature_flags,file_locks,geo,github_project_service_integration,group_allowed_email_domains,group_project_templates,group_saml,issues_analytics,jira_dev_panel_integration,ldap_group_sync_filter,merge_pipelines,merge_request_performance_metrics,merge_trains,metrics_reports,multiple_approval_rules,multiple_clusters,multiple_group_issue_boards,object_storage,operations_dashboard,packages,productivity_analytics,project_aliases,protected_environments,reject_unsigned_commits,required_ci_templates,scoped_labels,service_desk,smartcard_auth,group_timelogs,type_of_work_analytics,unprotection_restrictions,ci_project_subscriptions,cluster_health,container_scanning,dast,dependency_scanning,epics,group_ip_restriction,incident_management,insights,license_management,personal_access_token_expiration_policy,pod_logs,prometheus_alerts,pseudonymizer,report_approver_rules,sast,security_dashboard,tracing,web_ide_terminal +++ export GITLAB_FEATURES=audit_events,burndown_charts,code_owners,contribution_analytics,description_diffs,elastic_search,group_bulk_edit,group_burndown_charts,group_webhooks,issuable_default_templates,issue_weights,jenkins_integration,ldap_group_sync,member_lock,merge_request_approvers,multiple_issue_assignees,multiple_ldap_servers,multiple_merge_request_assignees,protected_refs_for_users,push_rules,related_issues,repository_mirrors,repository_size_limit,scoped_issue_board,usage_quotas,visual_review_app,wip_limits,adjourned_deletion_for_projects_and_groups,admin_audit_log,auditor_user,batch_comments,blocking_merge_requests,board_assignee_lists,board_milestone_lists,ci_cd_projects,cluster_deployments,code_analytics,code_owner_approval_required,commit_committer_check,cross_project_pipelines,custom_file_templates,custom_file_templates_for_namespace,custom_project_templates,custom_prometheus_metrics,cycle_analytics_for_groups,db_load_balancing,default_project_deletion_protection,dependency_proxy,deploy_board,design_management,email_additional_text,extended_audit_events,external_authorization_service_api_management,feature_flags,file_locks,geo,github_project_service_integration,group_allowed_email_domains,group_project_templates,group_saml,issues_analytics,jira_dev_panel_integration,ldap_group_sync_filter,merge_pipelines,merge_request_performance_metrics,merge_trains,metrics_reports,multiple_approval_rules,multiple_clusters,multiple_group_issue_boards,object_storage,operations_dashboard,packages,productivity_analytics,project_aliases,protected_environments,reject_unsigned_commits,required_ci_templates,scoped_labels,service_desk,smartcard_auth,group_timelogs,type_of_work_analytics,unprotection_restrictions,ci_project_subscriptions,cluster_health,container_scanning,dast,dependency_scanning,epics,group_ip_restriction,incident_management,insights,license_management,personal_access_token_expiration_policy,pod_logs,prometheus_alerts,pseudonymizer,report_approver_rules,sast,security_dashboard,tracing,web_ide_terminal +++ GITLAB_FEATURES=audit_events,burndown_charts,code_owners,contribution_analytics,description_diffs,elastic_search,group_bulk_edit,group_burndown_charts,group_webhooks,issuable_default_templates,issue_weights,jenkins_integration,ldap_group_sync,member_lock,merge_request_approvers,multiple_issue_assignees,multiple_ldap_servers,multiple_merge_request_assignees,protected_refs_for_users,push_rules,related_issues,repository_mirrors,repository_size_limit,scoped_issue_board,usage_quotas,visual_review_app,wip_limits,adjourned_deletion_for_projects_and_groups,admin_audit_log,auditor_user,batch_comments,blocking_merge_requests,board_assignee_lists,board_milestone_lists,ci_cd_projects,cluster_deployments,code_analytics,code_owner_approval_required,commit_committer_check,cross_project_pipelines,custom_file_templates,custom_file_templates_for_namespace,custom_project_templates,custom_prometheus_metrics,cycle_analytics_for_groups,db_load_balancing,default_project_deletion_protection,dependency_proxy,deploy_board,design_management,email_additional_text,extended_audit_events,external_authorization_service_api_management,feature_flags,file_locks,geo,github_project_service_integration,group_allowed_email_domains,group_project_templates,group_saml,issues_analytics,jira_dev_panel_integration,ldap_group_sync_filter,merge_pipelines,merge_request_performance_metrics,merge_trains,metrics_reports,multiple_approval_rules,multiple_clusters,multiple_group_issue_boards,object_storage,operations_dashboard,packages,productivity_analytics,project_aliases,protected_environments,reject_unsigned_commits,required_ci_templates,scoped_labels,service_desk,smartcard_auth,group_timelogs,type_of_work_analytics,unprotection_restrictions,ci_project_subscriptions,cluster_health,container_scanning,dast,dependency_scanning,epics,group_ip_restriction,incident_management,insights,license_management,personal_access_token_expiration_policy,pod_logs,prometheus_alerts,pseudonymizer,report_approver_rules,sast,security_dashboard,tracing,web_ide_terminal ++ export CI_PROJECT_ID=17893 ++ CI_PROJECT_ID=17893 ++ export CI_PROJECT_NAME=ci-debug-trace diff --git a/doc/subscriptions/index.md b/doc/subscriptions/index.md index 70726f51d32..e2868d648de 100644 --- a/doc/subscriptions/index.md +++ b/doc/subscriptions/index.md @@ -173,7 +173,7 @@ To see the status of your GitLab.com subscription, log into GitLab.com and go to 1. Go to **User Avatar > Settings**. 1. Click **Billing**. - For groups: - 1. From the group page (*not* from a project within the group), go to **Administration > Billing**. + 1. From the group page (*not* from a project within the group), go to **Settings > Billing**. The following table describes details of your subscription for groups: @@ -430,7 +430,7 @@ CI pipeline minutes are the execution time for your [pipelines](../ci/pipelines/ Quotas apply to: -- Groups, where the minutes are shared across all members of the group, its subgroups, and nested projects. To view the group's usage, navigate to the group, then **{settings}** **Administration > Usage Quotas**. +- Groups, where the minutes are shared across all members of the group, its subgroups, and nested projects. To view the group's usage, navigate to the group, then **{settings}** **Settings > Usage Quotas**. - Your personal account, where the minutes are available for your personal projects. To view and buy personal minutes, click your avatar, then **{settings}** **Settings > Pipeline quota**. Only pipeline minutes for GitLab shared runners are restricted. If you have a specific runner set up for your projects, there is no limit to your build time on GitLab.com. @@ -451,10 +451,10 @@ main quota. Additional minutes: To purchase additional minutes for your group on GitLab.com: -1. From your group, go to **{settings}** **Administration > Usage Quotas**. +1. From your group, go to **{settings}** **Settings > Usage Quotas**. 1. Locate the subscription card that's linked to your group on GitLab.com, click **Buy more CI minutes**, and complete the details about the transaction. 1. Once we have processed your payment, the extra CI minutes will be synced to your group. -1. To confirm the available CI minutes, go to your group, then **{settings}** **Administration > Usage Quotas**. +1. To confirm the available CI minutes, go to your group, then **{settings}** **Settings > Usage Quotas**. The **Additional minutes** displayed now includes the purchased additional CI minutes, plus any minutes rolled over from last month. To purchase additional minutes for your personal namespace: diff --git a/doc/user/compliance/license_compliance/index.md b/doc/user/compliance/license_compliance/index.md index 96b051c1673..485e9d8213d 100644 --- a/doc/user/compliance/license_compliance/index.md +++ b/doc/user/compliance/license_compliance/index.md @@ -121,6 +121,8 @@ License Compliance can be configured using environment variables. | `LM_JAVA_VERSION` | no | Version of Java. If set to `11`, Maven and Gradle use Java 11 instead of Java 8. | | `LM_PYTHON_VERSION` | no | Version of Python. If set to `3`, dependencies are installed using Python 3 instead of Python 2.7. | | `SETUP_CMD` | no | Custom setup for the dependency installation. (experimental) | +| `PIP_INDEX_URL` | no | Base URL of Python Package Index (default: `https://pypi.org/simple/`). | +| `ADDITIONAL_CA_CERT_BUNDLE` | no | Bundle of trusted CA certificates (currently supported in Python projects). | ### Installing custom dependencies @@ -215,6 +217,37 @@ license_scanning: LM_PYTHON_VERSION: 2 ``` +### Custom root certificates for Python + +You can supply a custom root certificate to complete TLS verification by using the +`ADDITIONAL_CA_CERT_BUNDLE` [environment variable](#available-variables). + +To bypass TLS verification, you can use a custom [`pip.conf`](https://pip.pypa.io/en/stable/user_guide/#config-file) +file to configure trusted hosts. + +The following `gitlab-ci.yml` file uses a [`before_script`](../../../ci/yaml/README.md#before_script-and-after_script) +to inject a custom [`pip.conf`](https://pip.pypa.io/en/stable/user_guide/#config-file): + +```yaml +include: + - template: License-Scanning.gitlab-ci.yml + +license_scanning: + variables: + PIP_INDEX_URL: 'https://pypi.example.com/simple/' + before_script: + - mkdir -p ~/.config/pip/ + - cp pip.conf ~/.config/pip/pip.conf +``` + +The [`pip.conf`](https://pip.pypa.io/en/stable/reference/pip/) allows you to specify a list of +[trusted hosts](https://pip.pypa.io/en/stable/reference/pip/#cmdoption-trusted-host): + +```text +[global] +trusted-host = pypi.example.com +``` + ### Migration from `license_management` to `license_scanning` In GitLab 12.8 a new name for `license_management` job was introduced. This change was made to improve clarity around the purpose of the scan, which is to scan and collect the types of licenses present in a projects dependencies. diff --git a/doc/user/group/saml_sso/index.md b/doc/user/group/saml_sso/index.md index 308a7b2cb4a..4fcb5064c8c 100644 --- a/doc/user/group/saml_sso/index.md +++ b/doc/user/group/saml_sso/index.md @@ -24,7 +24,7 @@ Note the following: ## Configuring your Identity Provider -1. Navigate to the group and click **Administration > SAML SSO**. +1. Navigate to the group and click **Settings > SAML SSO**. 1. Configure your SAML server using the **Assertion consumer service URL** and **Identifier**. Alternatively GitLab provides [metadata XML configuration](#metadata-configuration). See [your identity provider's documentation](#providers) for more details. 1. Configure the SAML response to include a NameID that uniquely identifies each user. 1. Configure required assertions using the [table below](#assertions). @@ -116,7 +116,7 @@ This feature is similar to the [Credentials inventory for self-managed instances > [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/34648) in GitLab 12.9. Groups with group-managed accounts can disallow forking of projects to destinations outside the group. -To do so, enable the "Prohibit outer forks" option in **Administration > SAML SSO**. +To do so, enable the "Prohibit outer forks" option in **Settings > SAML SSO**. When enabled, projects within the group can only be forked to other destinations within the group (including its subgroups). ##### Other restrictions for Group-managed accounts @@ -146,7 +146,7 @@ assertions to be able to create a user. GitLab provides metadata XML that can be used to configure your Identity Provider. -1. Navigate to the group and click **Administration > SAML SSO**. +1. Navigate to the group and click **Settings > SAML SSO**. 1. Copy the provided **GitLab metadata URL**. 1. Follow your Identity Provider's documentation and paste the metadata URL when it is requested. @@ -154,7 +154,7 @@ GitLab provides metadata XML that can be used to configure your Identity Provide Once you've set up your identity provider to work with GitLab, you'll need to configure GitLab to use it for authentication: -1. Navigate to the group's **Administration > SAML SSO**. +1. Navigate to the group's **Settings > SAML SSO**. 1. Find the SSO URL from your Identity Provider and enter it the **Identity provider single sign on URL** field. 1. Find and enter the fingerprint for the SAML token signing certificate in the **Certificate** field. 1. Click the **Enable SAML authentication for this group** toggle switch. @@ -288,14 +288,14 @@ If the information you need isn't listed above you may wish to check our [troubl To link SAML to your existing GitLab.com account: 1. Sign in to your GitLab.com account. -1. Locate the SSO URL for the group you are signing in to. A group Admin can find this on the group's **Administration > SAML SSO** page. +1. Locate the SSO URL for the group you are signing in to. A group Admin can find this on the group's **Settings > SAML SSO** page. 1. Visit the SSO URL and click **Authorize**. 1. Enter your credentials on the Identity Provider if prompted. 1. You will be redirected back to GitLab.com and should now have access to the group. In the future, you can use SAML to sign in to GitLab.com. ## Signing in to GitLab.com with SAML -1. Locate the SSO URL for the group you are signing in to. A group Admin can find this on a group's **Administration > SAML SSO** page. If configured, it might also be possible to sign in to GitLab starting from your Identity Provider. +1. Locate the SSO URL for the group you are signing in to. A group Admin can find this on a group's **Settings > SAML SSO** page. If configured, it might also be possible to sign in to GitLab starting from your Identity Provider. 1. Visit the SSO URL and click the **Sign in with Single Sign-On** button. 1. Enter your credentials on the Identity Provider if prompted. 1. You will be signed in to GitLab.com and redirected to the group. diff --git a/doc/user/group/saml_sso/scim_setup.md b/doc/user/group/saml_sso/scim_setup.md index e4dff12b004..42bc52a9201 100644 --- a/doc/user/group/saml_sso/scim_setup.md +++ b/doc/user/group/saml_sso/scim_setup.md @@ -30,7 +30,7 @@ The following identity providers are supported: Once [Single sign-on](index.md) has been configured, we can: -1. Navigate to the group and click **Administration > SAML SSO**. +1. Navigate to the group and click **Settings > SAML SSO**. 1. Click on the **Generate a SCIM token** button. 1. Save the token and URL so they can be used in the next step. diff --git a/doc/user/project/issues/csv_export.md b/doc/user/project/issues/csv_export.md index c5b048c1d69..53af878cbd6 100644 --- a/doc/user/project/issues/csv_export.md +++ b/doc/user/project/issues/csv_export.md @@ -1,4 +1,4 @@ -# Export Issues to CSV **(STARTER)** +# Export Issues to CSV > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/1126) in [GitLab Starter 9.0](https://about.gitlab.com/releases/2017/03/22/gitlab-9-0-released/#export-issues-ees-eep). diff --git a/doc/user/project/issues/index.md b/doc/user/project/issues/index.md index 4da65793dbb..3c480490e02 100644 --- a/doc/user/project/issues/index.md +++ b/doc/user/project/issues/index.md @@ -197,7 +197,7 @@ Feature.disable(:save_issuable_health_status) - [Bulk edit issues](../bulk_editing.md) - From the Issues List, select multiple issues in order to change their status, assignee, milestone, or labels in bulk. - [Import issues](csv_import.md) -- [Export issues](csv_export.md) **(STARTER)** +- [Export issues](csv_export.md) - [Issues API](../../../api/issues.md) - Configure an [external issue tracker](../../../integration/external-issue-tracker.md) such as Jira, Redmine, or Bugzilla. |