diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2023-11-09 06:12:10 +0300 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2023-11-09 06:12:10 +0300 |
| commit | 1bee0901f0b23d885cdeed263a930cd11e23a4ee (patch) | |
| tree | cb2dfda241c51cbcc6cadd2389f5a1212bb5fa6d /doc | |
| parent | ea3cfa07a4dcd772c26a2b7dd11419eed2489257 (diff) | |
Add latest changes from gitlab-org/gitlab@master
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/administration/pages/index.md | 4 | ||||
| -rw-r--r-- | doc/api/import.md | 12 | ||||
| -rw-r--r-- | doc/ci/cloud_services/azure/index.md | 21 | ||||
| -rw-r--r-- | doc/ci/secrets/azure_key_vault.md | 66 | ||||
| -rw-r--r-- | doc/development/github_importer.md | 16 | ||||
| -rw-r--r-- | doc/install/aws/eks_clusters_aws.md | 49 | ||||
| -rw-r--r-- | doc/install/aws/gitlab_hybrid_on_aws.md | 380 | ||||
| -rw-r--r-- | doc/install/aws/gitlab_sre_for_aws.md | 98 | ||||
| -rw-r--r-- | doc/install/aws/index.md | 880 | ||||
| -rw-r--r-- | doc/install/aws/manual_install_aws.md | 859 | ||||
| -rw-r--r-- | doc/solutions/cloud/aws/gitaly_sre_for_aws.md | 91 | ||||
| -rw-r--r-- | doc/solutions/cloud/aws/gitlab_aws_integration.md | 15 | ||||
| -rw-r--r-- | doc/solutions/cloud/aws/gitlab_instance_on_aws.md | 55 | ||||
| -rw-r--r-- | doc/solutions/cloud/aws/gitlab_single_box_on_aws.md | 51 | ||||
| -rw-r--r-- | doc/solutions/cloud/aws/index.md | 74 | ||||
| -rw-r--r-- | doc/solutions/index.md | 4 | ||||
| -rw-r--r-- | doc/update/deprecations.md | 20 | ||||
| -rw-r--r-- | doc/user/compliance/compliance_center/index.md | 6 | ||||
| -rw-r--r-- | doc/user/packages/container_registry/troubleshoot_container_registry.md | 21 |
19 files changed, 1217 insertions, 1505 deletions
diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index f64c53e28a2..97acbf717fe 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -200,13 +200,13 @@ then run `gitlab-ctl reconfigure`. For more information, read **Requirements:** - [Wildcard DNS setup](#dns-configuration) -- [TLS-terminating load balancer](../../install/aws/manual_install_aws.md#load-balancer) +- [TLS-terminating load balancer](../../install/aws/index.md#load-balancer) --- URL scheme: `https://<namespace>.example.io/<project_slug>` -This setup is primarily intended to be used when [installing a GitLab POC on Amazon Web Services](../../install/aws/manual_install_aws.md). This includes a TLS-terminating [classic load balancer](../../install/aws/manual_install_aws.md#load-balancer) that listens for HTTPS connections, manages TLS certificates, and forwards HTTP traffic to the instance. +This setup is primarily intended to be used when [installing a GitLab POC on Amazon Web Services](../../install/aws/index.md). This includes a TLS-terminating [classic load balancer](../../install/aws/index.md#load-balancer) that listens for HTTPS connections, manages TLS certificates, and forwards HTTP traffic to the instance. 1. In `/etc/gitlab/gitlab.rb` specify the following configuration: diff --git a/doc/api/import.md b/doc/api/import.md index 677848a0ed3..4b7abfdfec1 100644 --- a/doc/api/import.md +++ b/doc/api/import.md @@ -34,7 +34,6 @@ POST /import/github | `target_namespace` | string | yes | Namespace to import repository into. Supports subgroups like `/namespace/subgroup`. In GitLab 15.8 and later, must not be blank | | `github_hostname` | string | no | Custom GitHub Enterprise hostname. Do not set for GitHub.com. | | `optional_stages` | object | no | [Additional items to import](../user/project/import/github.md#select-additional-items-to-import). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/373705) in GitLab 15.5 | -| `additional_access_tokens` | string | no | Comma-separated list of [additional](#use-multiple-github-personal-access-tokens) GitHub personal access tokens. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/337232) in GitLab 16.2 | | `timeout_strategy` | string | no | Strategy for handling import timeouts. Valid values are `optimistic` (continue to next stage of import) or `pessimistic` (fail immediately). Defaults to `pessimistic`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/422979) in GitLab 16.5. | ```shell @@ -80,17 +79,6 @@ Example response: } ``` -### Use multiple GitHub personal access tokens - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/337232) in GitLab 16.2. - -The GitHub import API can accept more than one GitHub personal access token using the `additional_access_tokens` -property so the API can make more calls to GitHub before hitting the rate limit. The additional GitHub personal access -tokens: - -- Cannot be from the same account because they would all share one rate limit. -- Must have the same permissions and sufficient privileges to the repositories to import. - ### Import a public project through the API using a group access token > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/362683) in GitLab 15.7, projects are not imported into a [bot user's](../user/group/settings/group_access_tokens.md#bot-users-for-groups) namespace in any circumstances. Projects imported into a bot user's namespace could not be deleted by users with valid tokens, which represented a security risk. diff --git a/doc/ci/cloud_services/azure/index.md b/doc/ci/cloud_services/azure/index.md index 3a882cf6820..b26533562f4 100644 --- a/doc/ci/cloud_services/azure/index.md +++ b/doc/ci/cloud_services/azure/index.md @@ -25,6 +25,7 @@ Prerequisites: - Access to the corresponding Azure Active Directory Tenant with at least the `Application Developer` access level. - A local installation of the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli). Alternatively, you can follow all the steps below with the [Azure Cloud Shell](https://portal.azure.com/#cloudshell/). +- Your GitLab instance must be publicly accessible over the internet as Azure must to connect to the GitLab OIDC endpoint. - A GitLab project. To complete this tutorial: @@ -167,3 +168,23 @@ CI/CD variables, from the Azure Portal: Azure AD federated identity credentials. Review [Connect to cloud services](../index.md) for further details. + +### `Request to External OIDC endpoint failed` message + +If you receive the error `ERROR: AADSTS501661: Request to External OIDC endpoint failed.` +you should verify that your GitLab instance is publicly accessible from the internet. + +Azure must be able to access the following GitLab endpoints to authenticate with OIDC: + +- `GET /.well-known/openid-configuration` +- `GET /oauth/discovery/keys` + +If you update your firewall and still receive this error, [clear the Redis cache](../../../administration/raketasks/maintenance.md#clear-redis-cache) +and try again. + +### `No matching federated identity record found for presented assertion audience` message + +If you receive the error `ERROR: AADSTS700212: No matching federated identity record found for presented assertion audience 'https://gitlab.com'` +you should verify that your CI/CD job uses the correct `aud` value. + +The `aud` value should match the audience used to [create the federated identity credentials](#create-azure-ad-federated-identity-credentials). diff --git a/doc/ci/secrets/azure_key_vault.md b/doc/ci/secrets/azure_key_vault.md index 645ab5db0d1..d8a511e8bdf 100644 --- a/doc/ci/secrets/azure_key_vault.md +++ b/doc/ci/secrets/azure_key_vault.md @@ -9,14 +9,19 @@ type: concepts, howto > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/271271) in GitLab and GitLab Runner 16.3. +NOTE: +A [bug was discovered](https://gitlab.com/gitlab-org/gitlab/-/issues/424746) and this feature might not work as expected or at all. A fix is scheduled for a future release. + You can use secrets stored in the [Azure Key Vault](https://azure.microsoft.com/en-us/products/key-vault/) in your GitLab CI/CD pipelines. Prerequisites: -- Have a key vault on Azure. -- Have an application with key vault permissions. -- [Configure OpenID Connect in Azure to retrieve temporary credentials](../../ci/cloud_services/azure/index.md). +- Have a [Key Vault](https://learn.microsoft.com/en-us/azure/key-vault/general/quick-create-portal) on Azure. + - Your IAM user must be granted [granted the **Key Vault Administrator** role assignment](https://learn.microsoft.com/en-us/azure/role-based-access-control/quickstart-assign-role-user-portal#grant-access) + for the **resource group** assigned to the Key Vault. Otherwise, you can't create secrets inside the Key Vault. +- [Configure OpenID Connect in Azure to retrieve temporary credentials](../../ci/cloud_services/azure/index.md). These + steps include instructions on how to create an Azure AD application for Key Vault access. - Add [CI/CD variables to your project](../variables/index.md#for-a-project) to provide details about your Vault server: - `AZURE_KEY_VAULT_SERVER_URL`: The URL of your Azure Key Vault server, such as `https://vault.example.com`. - `AZURE_CLIENT_ID`: The client ID of the Azure application. @@ -31,19 +36,64 @@ You can use a secret stored in your Azure Key Vault in a job by defining it with job: id_tokens: AZURE_JWT: - aud: 'azure' + aud: 'https://gitlab.com' secrets: DATABASE_PASSWORD: - token: AZURE_JWT + token: $AZURE_JWT azure_key_vault: name: 'test' - version: 'test' + version: '00000000000000000000000000000000' ``` In this example: -- `name` is the name of the secret. -- `version` is the version of the secret. +- `aud` is the audience, which must match the audience used when [creating the federated identity credentials](../../ci/cloud_services/azure/index.md#create-azure-ad-federated-identity-credentials) +- `name` is the name of the secret in Azure Key Vault. +- `version` is the version of the secret in Azure Key Vault. The version is a generated + GUID without dashes, which can be found on the Azure Key Vault secrets page. - GitLab fetches the secret from Azure Key Vault and stores the value in a temporary file. The path to this file is stored in a `DATABASE_PASSWORD` CI/CD variable, similar to [file type CI/CD variables](../variables/index.md#use-file-type-cicd-variables). + +## Troubleshooting + +Refer to [OIDC for Azure troubleshooting](../../ci/cloud_services/azure/index.md#troubleshooting) for general +problems when setting up OIDC with Azure. + +### `JWT token is invalid or malformed` message + +You might receive this error when fetching secrets from Azure Key Vault: + +```plaintext +RESPONSE 400 Bad Request +AADSTS50027: JWT token is invalid or malformed. +``` + +This occurs due to a known issue in GitLab Runner where the JWT token isn't parsed correctly. +A fix is [scheduled for a future GitLab Runner release](https://gitlab.com/gitlab-org/gitlab/-/issues/424746). + +### `Caller is not authorized to perform action on resource` message + +You might receive this error when fetching secrets from Azure Key Vault: + +```plaintext +RESPONSE 403: 403 Forbidden +ERROR CODE: Forbidden +Caller is not authorized to perform action on resource.\r\nIf role assignments, deny assignments or role definitions were changed recently, please observe propagation time. +ForbiddenByRbac +``` + +If your Azure Key Vault is using RBAC, you must add the **Key Vault Secrets User** to your Azure AD +application. + +For example: + +```shell +appId=$(az ad app list --display-name gitlab-oidc --query '[0].appId' -otsv) +az role assignment create --assignee $appId --role "Key Vault Secrets User" --scope /subscriptions/<subscription-id> +``` + +You can find your subscription ID in: + +- The [Azure Portal](https://learn.microsoft.com/en-us/azure/azure-portal/get-subscription-tenant-id#find-your-azure-subscription). +- The [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/manage-azure-subscriptions-azure-cli#get-the-active-subscription). diff --git a/doc/development/github_importer.md b/doc/development/github_importer.md index 45554ae465d..d4c51ad54dc 100644 --- a/doc/development/github_importer.md +++ b/doc/development/github_importer.md @@ -213,22 +213,6 @@ process. This is done by calling `ProjectImportState#refresh_jid_expiration`. By refreshing this TTL we can ensure our import does not get marked as failed so long we're still performing work. -## GitHub rate limit - -GitHub has a rate limit of 5,000 API calls per hour. The number of requests -necessary to import a project is largely dominated by the number of unique users -involved in a project (for example, issue authors), because we need the email address of users to map -them to GitLab users. Other data such as issue pages and comments typically only requires a few dozen requests to import. - -We handle the rate limit by doing the following: - -1. After we hit the rate limit, we either: - - Automatically reschedule jobs in such a way that they are not executed until the rate limit has been reset. - - Move onto another GitHub access token if multiple GitHub access tokens were passed to the API. -1. We cache the mapping of GitHub users to GitLab users in Redis. - -More information on user caching can be found below. - ## Caching user lookups When mapping GitHub users to GitLab users we need to (in the worst case) diff --git a/doc/install/aws/eks_clusters_aws.md b/doc/install/aws/eks_clusters_aws.md index 45ba46fce1e..b05749bdde3 100644 --- a/doc/install/aws/eks_clusters_aws.md +++ b/doc/install/aws/eks_clusters_aws.md @@ -1,46 +1,11 @@ --- -stage: Systems -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: '../../solutions/cloud/aws/index.md' +remove_date: '2024-03-31' --- -# EKS cluster provisioning best practices **(FREE SELF)** +This document was moved to [Solutions](../../solutions/cloud/aws/index.md). -GitLab can be used to provision an EKS cluster into AWS, however, it necessarily focuses on a basic EKS configuration. Using the AWS tools can help with advanced cluster configuration, automation, and maintenance. - -This documentation is not for clusters for deployment of GitLab itself, but instead clusters purpose built for: - -- EKS Clusters for GitLab Runners -- Application Deployment Clusters for GitLab review apps -- Application Deployment Cluster for production applications - -Information on deploying GitLab onto EKS can be found in [Provisioning GitLab Cloud Native Hybrid on AWS EKS](gitlab_hybrid_on_aws.md). - -## Use `eksctl` - -Using `eksctl` enables the following when building an EKS Cluster: - -- You have various cluster configuration options: - - Selection of operating system: Amazon Linux 2, Windows, Bottlerocket - - Selection of Hardware Architecture: x86, ARM, GPU - - Selection of Kubernetes version (the GitLab-managed clusters for your project's applications have [specific Kubernetes version requirements](../../user/clusters/agent/index.md#supported-kubernetes-versions-for-gitlab-features)) -- It can deploy high value-add items to the cluster, including: - - A bastion host to keep the cluster endpoint private and possible perform performance testing. - - Prometheus and Grafana for monitoring. -- EKS Autoscaler for automatic K8s Node scaling. -- 2 or 3 Availability Zones (AZ) spread for balance between High Availability (HA) and cost control. -- Ability to specify spot compute. - -Read more about configuring Amazon EKS in the [`eksctl` guide](https://eksctl.io/getting-started/) and the [Amazon EKS User Guide](https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html). - -## Inject GitLab configuration for integrating clusters - -Read more how to [configure an App Deployment cluster](../../user/project/clusters/add_existing_cluster.md) and extract information from it to integrate it into GitLab. - -## Provision GitLab Runners using Helm charts - -Read how to [use the GitLab Runner Helm Chart](https://docs.gitlab.com/runner/install/kubernetes.html) to deploy a runner into a cluster. - -## Runner Cache - -Because the EKS Quick Start provides for EFS provisioning, the best approach is to use EFS for runner caching. Eventually we will publish information on using an S3 bucket for runner caching here. +<!-- This redirect file can be deleted after <YYYY-MM-DD>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
\ No newline at end of file diff --git a/doc/install/aws/gitlab_hybrid_on_aws.md b/doc/install/aws/gitlab_hybrid_on_aws.md index b39f39f293e..84474e6615c 100644 --- a/doc/install/aws/gitlab_hybrid_on_aws.md +++ b/doc/install/aws/gitlab_hybrid_on_aws.md @@ -1,377 +1,11 @@ --- -stage: Systems -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: '../../solutions/cloud/aws/gitlab_instance_on_aws.md' +remove_date: '2024-03-31' --- -{::options parse_block_html="true" /} +This document was moved to [Solutions](../../solutions/cloud/aws/gitlab_instance_on_aws.md). -# Provision GitLab Cloud Native Hybrid on AWS EKS **(FREE SELF)** - -GitLab "Cloud Native Hybrid" is a hybrid of the cloud native technology Kubernetes (EKS) and EC2. While as much of the GitLab application as possible runs in Kubernetes or on AWS services (PaaS), the GitLab service Gitaly must still be run on EC2. Gitaly is a layer designed to overcome limitations of the Git binaries in a horizontally scaled architecture. You can read more here about why Gitaly was built and why the limitations of Git mean that it must currently run on instance compute in [Git Characteristics That Make Horizontal Scaling Difficult](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/DESIGN.md#git-characteristics-that-make-horizontal-scaling-difficult). - -Amazon provides a managed Kubernetes service offering known as [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). - -## Tested AWS Bill of Materials by reference architecture size - -| GitLab Cloud Native Hybrid Ref Arch | GitLab Baseline Performance Test Results (using the Linux package on instances) | AWS Bill of Materials (BOM) for CNH | AWS Build Performance Testing Results for [CNH](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128_results.txt) | CNH Cost Estimate 3 AZs* | -| ------------------------------------------------------------ | ------------------------------------------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | -| [2K Linux package installation](../../administration/reference_architectures/2k_users.md) | [2K Baseline](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/2k) | [2K Cloud Native Hybrid on EKS](#2k-cloud-native-hybrid-on-eks) | GPT Test Results | [1 YR Ec2 Compute Savings + 1 YR RDS & ElastiCache RIs](https://calculator.aws/#/estimate?id=544bcf1162beae6b8130ad257d081cdf9d4504e3)<br />(2 AZ Cost Estimate is in BOM Below) | -| [3K](../../administration/reference_architectures/3k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) | [3k Baseline](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/3k) | [3K Cloud Native Hybrid on EKS](#3k-cloud-native-hybrid-on-eks) | [3K Full Fixed Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/3K/3k-QuickStart-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_124216/3k-QuickStart-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_124216_results.txt)<br /><br />[3K Elastic Auto Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/3K/3k-QuickStart-AutoScale-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_194200/3k-QuickStart-AutoScale-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_194200_results.txt) | [1 YR Ec2 Compute Savings + 1 YR RDS & ElastiCache RIs](https://calculator.aws/#/estimate?id=f1294fec554e21be999711cddcdab9c5e7f83f14)<br />(2 AZ Cost Estimate is in BOM Below) | -| [5K](../../administration/reference_architectures/5k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) | [5k Baseline](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/5k) | [5K Cloud Native Hybrid on EKS](#5k-cloud-native-hybrid-on-eks) | [5K Full Fixed Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128_results.txt)<br /><br />[5K AutoScale from 25% GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-AutoScale-From-25Percent-ARM-RDS-Redis_v13-12-3-ee_2021-07-24_102717/5k-QuickStart-AutoScale-From-25Percent-ARM-RDS-Redis_v13-12-3-ee_2021-07-24_102717_results.txt) | [1 YR Ec2 Compute Savings + 1 YR RDS & ElastiCache RIs](https://calculator.aws/#/estimate?id=330ee43c5b14662db5df6e52b34898d181a09e16) | -| [10K](../../administration/reference_architectures/10k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) | [10k Baseline](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k) | [10K Cloud Native Hybrid on EKS](#10k-cloud-native-hybrid-on-eks) | [10K Full Fixed Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/10K/GL-CloudNative-10k-RDS-Graviton_v13-12-3-ee_2021-07-08_194647/GL-CloudNative-10k-RDS-Graviton_v13-12-3-ee_2021-07-08_194647_results.txt)<br /><br />[10K Elastic Auto Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/10K/GL-CloudNative-10k-AutoScaling-Test_v13-12-3-ee_2021-07-09_115139/GL-CloudNative-10k-AutoScaling-Test_v13-12-3-ee_2021-07-09_115139_results.txt) | [10K 1 YR Ec2 Compute Savings + 1 YR RDS & ElastiCache RIs](https://calculator.aws/#/estimate?id=5ac2e07a22e01c36ee76b5477c5a046cd1bea792) | -| [50K](../../administration/reference_architectures/50k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) | [50k Baseline](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/50k) | [50K Cloud Native Hybrid on EKS](#50k-cloud-native-hybrid-on-eks) | [50K Full Fixed Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/50K/50k-Fixed-Scale-Test_v13-12-3-ee_2021-08-13_172819/50k-Fixed-Scale-Test_v13-12-3-ee_2021-08-13_172819_results.txt)<br /><br />[10K Elastic Auto Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/50K/50k-AutoScale-Test_v13-12-3-ee_2021-08-13_192633/50k-AutoScale-Test_v13-12-3-ee_2021-08-13_192633.txt) | [50K 1 YR Ec2 Compute Savings + 1 YR RDS & ElastiCache RIs](https://calculator.aws/#/estimate?id=b9c9d6ac1d4a7848011d2050cef3120931fb7c22) | - -\*Cost calculations for actual implementations are a rough guideline with the following considerations: - -- Actual choices about instance types should be based on GPT testing of your configuration. -- The first year of actual usage will reveal potential savings due to lower than expected usage, especially for ramping migrations where the full loading takes months, so be careful not to commit to savings plans too early or for too long. -- The cost estimates assume full scale of the Kubernetes cluster nodes 24 x 7 x 365. Savings due to 'idling scale-in' are not considered because they are highly dependent on the usage patterns of the specific implementation. -- Costs such as GitLab Runners, data egress and storage costs are not included as they are very dependent on the configuration of a specific implementation and on development behaviors (for example, frequency of committing or frequency of builds). -- These estimates will change over time as GitLab tests and optimizes compute choices. - -## Available Infrastructure as Code for GitLab Cloud Native Hybrid - -The [GitLab Environment Toolkit (GET)](https://gitlab.com/gitlab-org/gitlab-environment-toolkit/-/blob/main/README.md) is a set of opinionated Terraform -and Ansible scripts. These scripts help with the deployment of Linux package or Cloud Native Hybrid environments on selected cloud providers and are used -by GitLab developers for [GitLab Dedicated](../../subscriptions/gitlab_dedicated/index.md) (for example). - -You can use the GitLab Environment Toolkit to deploy a Cloud Native Hybrid environment on AWS. However, it's not required and may not support every valid -permutation. That said, the scripts are presented as-is and you can adapt them accordingly. - -### Two and Three Zone High Availability - -While GitLab Reference Architectures generally encourage three zone redundancy, AWS Quick Starts and AWS Well Architected consider two zone redundancy as AWS Well Architected. Individual implementations should weigh the costs of two and three zone configurations against their own high availability requirements for a final configuration. - -Gitaly Cluster uses a consistency voting system to implement strong consistency between synchronized nodes. Regardless of the number of availability zones implemented, there will always need to be a minimum of three Gitaly and three Praefect nodes in the cluster to avoid voting stalemates cause by an even number of nodes. - -### Streamlined Performance Testing of AWS Quick Start Prepared GitLab Instances - -A set of performance testing instructions have been abbreviated for testing a GitLab instance prepared using the AWS Quick Start for GitLab Cloud Native Hybrid on EKS. They assume zero familiarity with GitLab Performance Tool. They can be accessed here: [Performance Testing an Instance Prepared using AWS Quick Start for GitLab Cloud Native Hybrid on EKS](https://gitlab.com/guided-explorations/aws/implementation-patterns/getting-started-gitlab-aws-quick-start/-/wikis/Easy-Performance-Testing-for-AWS-Quick-Start-for-GitLab-CNH). - -### AWS GovCloud Support for AWS Quick Start for GitLab CNH on EKS - -The AWS Quick Start for GitLab Cloud Native Hybrid on EKS has been tested with GovCloud and works with the following restrictions and understandings. - -- GovCloud does not have public Route53 hosted zones, so you must set the following parameters: - - | CloudFormation Quick Start form field | CloudFormation Parameter | Setting | - | --------------------------------------------------- | ------------------------ | ------- | - | **Create Route 53 hosted zone** | CreatedHostedZone | No | - | **Request AWS Certificate Manager SSL certificate** | CreateSslCertificate | No | - -- The Quick Start creates public load balancer IPs, so that you can easily configure your local hosts file to get to the GUI for GitLab when deploying tests. However, you may need to manually alter this if public load balancers are not part of your provisioning plan. We are planning to make non-public load balancers a configuration option issue link: [Short Term: Documentation and/or Automation for private GitLab instance with no internet Ingress](https://github.com/aws-quickstart/quickstart-eks-gitlab/issues/55) -- As of 2021-08-19, AWS GovCloud has Graviton instances for Amazon RDS PostgreSQL available, but does not for ElastiCache Redis. -- It is challenging to get the Quick Start template to load in GovCloud from the Standard Quick Start URL, so the generic ones are provided here: - - [Launch for New VPC in us-gov-east-1](https://us-gov-east-1.console.amazonaws-us-gov.com/cloudformation/home?region=us-gov-east-1#/stacks/quickcreate?templateUrl=https://aws-quickstart.s3.us-east-1.amazonaws.com/quickstart-eks-gitlab/templates/gitlab-entry-new-vpc.template.yaml&stackName=Gitlab-for-EKS-New-VPC) - - [Launch for New VPC in us-gov-west-1](https://us-gov-west-1.console.amazonaws-us-gov.com/cloudformation/home?region=us-gov-west-1#/stacks/quickcreate?templateUrl=https://aws-quickstart.s3.us-east-1.amazonaws.com/quickstart-eks-gitlab/templates/gitlab-entry-new-vpc.template.yaml&stackName=Gitlab-for-EKS-New-VPC) - -## AWS PaaS qualified for all GitLab implementations - -For both implementations that used the Linux package or Cloud Native Hybrid implementations, the following GitLab Service roles can be performed by AWS Services (PaaS). Any PaaS solutions that require preconfigured sizing based on the scale of your instance will also be listed in the per-instance size Bill of Materials lists. Those PaaS that do not require specific sizing, are not repeated in the BOM lists (for example, AWS Certification Manager). - -These services have been tested with GitLab. - -Some services, such as log aggregation, outbound email are not specified by GitLab, but where provided are noted. - -| GitLab Services | AWS PaaS (Tested) | Provided by AWS Cloud <br />Native Hybrid Quick Start | -| ------------------------------------------------------------ | ------------------------------ | ------------------------------------------------------------ | -| <u>Tested PaaS Mentioned in Reference Architectures</u> | | | -| **PostgreSQL Database** | Amazon RDS PostgreSQL | Yes. | -| **Redis Caching** | Redis ElastiCache | Yes. | -| **Gitaly Cluster (Git Repository Storage)**<br />(Including Praefect and PostgreSQL) | ASG and Instances | Yes - ASG and Instances<br />**Note: Gitaly cannot be put into a Kubernetes Cluster.** | -| **All GitLab storages besides Git Repository Storage**<br />(Includes Git-LFS which is S3 Compatible) | AWS S3 | Yes | -| | | | -| <u>Tested PaaS for Supplemental Services</u> | | | -| **Front End Load Balancing** | AWS ELB | Yes | -| **Internal Load Balancing** | AWS ELB | Yes | -| **Outbound Email Services** | AWS Simple Email Service (SES) | Yes | -| **Certificate Authority and Management** | AWS Certificate Manager (ACM) | Yes | -| **DNS** | AWS Route53 (tested) | Yes | -| **GitLab and Infrastructure Log Aggregation** | AWS CloudWatch Logs | Yes (ContainerInsights Agent for EKS) | -| **Infrastructure Performance Metrics** | AWS CloudWatch Metrics | Yes | -| | | | -| <u>Supplemental Services and Configurations (Tested)</u> | | | -| **Prometheus for GitLab** | AWS EKS (Cloud Native Only) | Yes | -| **Grafana for GitLab** | AWS EKS (Cloud Native Only) | Yes | -| **Administrative Access to GitLab Backend** | Bastion Host in VPC | Yes - HA - Preconfigured for Cluster Management. | -| **Encryption (In Transit / At Rest)** | AWS KMS | Yes | -| **Secrets Storage for Provisioning** | AWS Secrets Manager | Yes | -| **Configuration Data for Provisioning** | AWS Parameter Store | Yes | -| **AutoScaling Kubernetes** | EKS AutoScaling Agent | Yes | - -## GitLab Cloud Native Hybrid on AWS - -### 2K Cloud Native Hybrid on EKS - -**2K Cloud Native Hybrid on EKS Bill of Materials (BOM)** - -**GPT Test Results** - -- TBD - - **Deploy Now** - Deploy Now links leverage the AWS Quick Start automation and only pre-populate the number of instances and instance types for the Quick Start based on the Bill of Materials below. You must provide appropriate input for all other parameters by following the guidance in the [Quick Start documentation's Deployment steps](https://aws-quickstart.github.io/quickstart-eks-gitlab/#_deployment_steps) section. - -- **Deploy Now: AWS Quick Start for 2 AZs** -- **Deploy Now: AWS Quick Start for 3 AZs** - -NOTE: -On Demand pricing is used in this table for comparisons, but should not be used for budgeting nor purchasing AWS resources for a GitLab production instance. Do not use these tables to calculate actual monthly or yearly price estimates, instead use the AWS Calculator links in the "GitLab on AWS Compute" table above and customize it with your desired savings plan. - -**BOM Total:** = Bill of Materials Total - this is what you use when building this configuration - -**Ref Arch Raw Total:** = The totals if the configuration was built on regular VMs with no PaaS services. Configuring on pure VMs generally requires additional VMs for cluster management activities. - -**Idle Configuration (Scaled-In)** = can be used to scale-in during time of low demand and/or for warm standby Geo instances. Requires configuration, testing and management of EKS autoscaling to meet your internal requirements. - -| Service | Ref Arch Raw (Full Scaled) | AWS BOM | Example Full Scaled Cost<br />(On Demand, US East) | -| ------------------------------------------------------------ | -------------------------- | ------------------------------------------------------------ | -------------------------------------------------- | -| Webservice | 12 vCPU,16 GB | | | -| Sidekiq | 2 vCPU, 8 GB | | | -| Supporting services such as NGINX, Prometheus, etc | 2 vCPU, 8 GB | | | -| **GitLab Ref Arch Raw Total K8s Node Capacity** | 16 vCPU, 32 GB | | | -| One Node for Overhead and Miscellaneous (EKS Cluster AutoScaler, Grafana, Prometheus, etc) | + 8 vCPU, 16 GB | | | -| **Grand Total w/ Overheads**<br />Minimum hosts = 3 | 24 vCPU, 48 GB | **c5.2xlarge** <br />(8vCPU/16 GB) x 3 nodes<br />24 vCPU, 48 GB | $1.02/hr | -| **Idle Configuration (Scaled-In)** | 16 vCPU, 32 GB | **c5.2xlarge** x 2 | $0.68/hr | - -NOTE: -If EKS node autoscaling is employed, it is likely that your average loading will run lower than this, especially during non-working hours and weekends. - -| Non-Kubernetes Compute | Ref Arch Raw Total | AWS BOM<br />(Directly Usable in AWS Quick Start) | Example Cost<br />US East, 3 AZ | Example Cost<br />US East, 2 AZ | -| ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------- | ------------------------------- | ------------------------------- | -| **Bastion Host (Quick Start)** | 1 HA instance in ASG | **t2.micro** for prod, **m4.2xlarge** for performance testing | | | -| **PostgreSQL**<br />AWS Amazon RDS PostgreSQL Nodes Configuration (GPT tested) | 2vCPU, 7.5 GB<br />Tested with Graviton ARM | **db.r6g.large** x 3 nodes <br />(6vCPU, 48 GB) | 3 nodes x $0.26 = $0.78/hr | 3 nodes x $0.26 = $0.78/hr | -| **Redis** | 1vCPU, 3.75GB<br />(across 12 nodes for Redis Cache, Redis Queues/Shared State, Sentinel Cache, Sentinel Queues/Shared State) | **cache.m6g.large** x 3 nodes<br />(6vCPU, 19 GB) | 3 nodes x $0.15 = $0.45/hr | 2 nodes x $0.15 = $0.30/hr | -| **<u>Gitaly Cluster</u>** [Details](gitlab_sre_for_aws.md#gitaly-sre-considerations) | [Gitaly & Praefect Must Have an Uneven Node Count for HA](gitlab_sre_for_aws.md#gitaly-and-praefect-elections) | | | | -| Gitaly Instances (in ASG) | 12 vCPU, 45GB<br />(across 3 nodes) | **m5.xlarge** x 3 nodes<br />(48 vCPU, 180 GB) | $0.192 x 3 = $0.58/hr | $0.192 x 3 = $0.58/hr | -| | The GitLab Reference architecture for 2K is not Highly Available and therefore has a single Gitaly no Praefect. AWS Quick Starts MUST be HA, so it implements Praefect from the 3K Ref Architecture to meet that requirement | | | | -| Praefect (Instances in ASG with load balancer) | 6 vCPU, 10 GB<br />([across 3 nodes](gitlab_sre_for_aws.md#gitaly-and-praefect-elections)) | **c5.large** x 3 nodes<br />(6 vCPU, 12 GB) | $0.09 x 3 = $0.21/hr | $0.09 x 3 = $0.21/hr | -| Praefect PostgreSQL(1) (AWS RDS) | 6 vCPU, 5.4 GB<br />([across 3 nodes](gitlab_sre_for_aws.md#gitaly-and-praefect-elections)) | Not applicable; reuses GitLab PostgreSQL | $0 | $0 | -| Internal Load Balancing Node | 2 vCPU, 1.8 GB | AWS ELB | $0.10/hr | $0.10/hr | - -### 3K Cloud Native Hybrid on EKS - -**3K Cloud Native Hybrid on EKS Bill of Materials (BOM)** - -**GPT Test Results** - -- [3K Full Fixed Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/3K/3k-QuickStart-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_124216/3k-QuickStart-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_124216_results.txt) - -- [3K AutoScale from 25% GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/3K/3k-QuickStart-AutoScale-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_194200/3k-QuickStart-AutoScale-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_194200_results.txt) - - Elastic Auto Scale GPT Test Results start with an idle scaled cluster and then start the standard GPT test to determine if the EKS Auto Scaler performs well enough to keep up with performance test demands. In general this is substantially harder ramping than the scaling required when the ramping is driven by standard production workloads. - -**Deploy Now** - -Deploy Now links leverage the AWS Quick Start automation and only pre-populate the number of instances and instance types for the Quick Start based on the Bill of Materials below. You must provide appropriate input for all other parameters by following the guidance in the [Quick Start documentation's Deployment steps](https://aws-quickstart.github.io/quickstart-eks-gitlab/#_deployment_steps) section. - -- **[Deploy Now: AWS Quick Start for 2 AZs](https://us-east-2.console.aws.amazon.com/cloudformation/home?region=us-east-2#/stacks/quickcreate?templateUrl=https://aws-quickstart.s3.us-east-1.amazonaws.com/quickstart-eks-gitlab/templates/gitlab-entry-new-vpc.template.yaml&stackName=Gitlab-EKS-3K-Users-2AZs¶m_NumberOfAZs=2¶m_NodeInstanceType=c5.2xlarge¶m_NumberOfNodes=3¶m_MaxNumberOfNodes=3¶m_DBInstanceClass=db.r6g.xlarge¶m_CacheNodes=2¶m_CacheNodeType=cache.m6g.large¶m_GitalyInstanceType=m5.large¶m_NumberOfGitalyReplicas=3¶m_PraefectInstanceType=c5.large¶m_NumberOfPraefectReplicas=3)** -- **[Deploy Now: AWS Quick Start for 3 AZs](https://us-east-2.console.aws.amazon.com/cloudformation/home?region=us-east-2#/stacks/quickcreate?templateUrl=https://aws-quickstart.s3.us-east-1.amazonaws.com/quickstart-eks-gitlab/templates/gitlab-entry-new-vpc.template.yaml&stackName=Gitlab-EKS-3K-Users-3AZs¶m_NumberOfAZs=3¶m_NodeInstanceType=c5.2xlarge¶m_NumberOfNodes=3¶m_MaxNumberOfNodes=3¶m_DBInstanceClass=db.r6g.xlarge¶m_CacheNodes=3¶m_CacheNodeType=cache.m6g.large¶m_GitalyInstanceType=m5.large¶m_NumberOfGitalyReplicas=3¶m_PraefectInstanceType=c5.large¶m_NumberOfPraefectReplicas=3)** - -NOTE: -On Demand pricing is used in this table for comparisons, but should not be used for budgeting nor purchasing AWS resources for a GitLab production instance. Do not use these tables to calculate actual monthly or yearly price estimates, instead use the AWS Calculator links in the "GitLab on AWS Compute" table above and customize it with your desired savings plan. - -**BOM Total:** = Bill of Materials Total - this is what you use when building this configuration - -**Ref Arch Raw Total:** = The totals if the configuration was built on regular VMs with no PaaS services. Configuring on pure VMs generally requires additional VMs for cluster management activities. - - **Idle Configuration (Scaled-In)** = can be used to scale-in during time of low demand and/or for warm standby Geo instances. Requires configuration, testing and management of EKS autoscaling to meet your internal requirements. - -| Service | Ref Arch Raw (Full Scaled) | AWS BOM | Example Full Scaled Cost<br />(On Demand, US East) | -| ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | -------------------------------------------------- | -| Webservice | [4 pods](https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/examples/ref/3k.yaml#L7) x ([5 vCPU & 6.25 GB](../../administration/reference_architectures/3k_users.md#webservice)) = <br />20 vCPU, 25 GB | | | -| Sidekiq | [8 pods](https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/examples/ref/3k.yaml#L24) x ([1 vCPU & 2 GB](../../administration/reference_architectures/3k_users.md#sidekiq)) = <br />8 vCPU, 16 GB | | | -| Supporting services such as NGINX, Prometheus, etc | [2 allocations](../../administration/reference_architectures/3k_users.md#cluster-topology) x ([2 vCPU and 7.5 GB](../../administration/reference_architectures/3k_users.md#cluster-topology)) = <br />4 vCPU, 15 GB | | | -| **GitLab Ref Arch Raw Total K8s Node Capacity** | 32 vCPU, 56 GB | | | -| One Node for Overhead and Miscellaneous (EKS Cluster AutoScaler, Grafana, Prometheus, etc) | + 16 vCPU, 32GB | | | -| **Grand Total w/ Overheads Full Scale**<br />Minimum hosts = 3 | 48 vCPU, 88 GB | **c5.2xlarge** (8vCPU/16 GB) x 5 nodes<br />40 vCPU, 80 GB<br />[Full Fixed Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/3K/3k-QuickStart-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_124216/3k-QuickStart-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_124216_results.txt) | $1.70/hr | -| **Possible Idle Configuration (Scaled-In 75% - round up)**<br />Pod autoscaling must be also adjusted to enable lower idling configuration. | 24 vCPU, 48 GB | c5.2xlarge x 4 | $1.36/hr | - -Other combinations of node type and quantity can be used to meet the Grand Total. Due to the properties of pods, hosts that are overly small may have significant unused capacity. - -NOTE: -If EKS node autoscaling is employed, it is likely that your average loading will run lower than this, especially during non-working hours and weekends. - -| Non-Kubernetes Compute | Ref Arch Raw Total | AWS BOM<br />(Directly Usable in AWS Quick Start) | Example Cost<br />US East, 3 AZ | Example Cost<br />US East, 2 AZ | -| ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------- | ------------------------------- | ------------------------------------------------------------ | -| **Bastion Host (Quick Start)** | 1 HA instance in ASG | **t2.micro** for prod, **m4.2xlarge** for performance testing | | | -| **PostgreSQL**<br />Amazon RDS PostgreSQL Nodes Configuration (GPT tested) | 18vCPU, 36 GB <br />(across 9 nodes for PostgreSQL, PgBouncer, Consul)<br />Tested with Graviton ARM | **db.r6g.xlarge** x 3 nodes <br />(12vCPU, 96 GB) | 3 nodes x $0.52 = $1.56/hr | 3 nodes x $0.52 = $1.56/hr | -| **Redis** | 6vCPU, 18 GB<br />(across 6 nodes for Redis Cache, Sentinel) | **cache.m6g.large** x 3 nodes<br />(6vCPU, 19 GB) | 3 nodes x $0.15 = $0.45/hr | 2 nodes x $0.15 = $0.30/hr | -| **<u>Gitaly Cluster</u>** [Details](gitlab_sre_for_aws.md#gitaly-sre-considerations) | | | | | -| Gitaly Instances (in ASG) | 12 vCPU, 45GB<br />([across 3 nodes](gitlab_sre_for_aws.md#gitaly-and-praefect-elections)) | **m5.large** x 3 nodes<br />(12 vCPU, 48 GB) | $0.192 x 3 = $0.58/hr | [Gitaly & Praefect Must Have an Uneven Node Count for HA](gitlab_sre_for_aws.md#gitaly-and-praefect-elections) | -| Praefect (Instances in ASG with load balancer) | 6 vCPU, 5.4 GB<br />([across 3 nodes](gitlab_sre_for_aws.md#gitaly-and-praefect-elections)) | **c5.large** x 3 nodes<br />(6 vCPU, 12 GB) | $0.09 x 3 = $0.21/hr | [Gitaly & Praefect Must Have an Uneven Node Count for HA](gitlab_sre_for_aws.md#gitaly-and-praefect-elections) | -| Praefect PostgreSQL(1) (Amazon RDS) | 6 vCPU, 5.4 GB<br />([across 3 nodes](gitlab_sre_for_aws.md#gitaly-and-praefect-elections)) | Not applicable; reuses GitLab PostgreSQL | $0 | | -| Internal Load Balancing Node | 2 vCPU, 1.8 GB | AWS ELB | $0.10/hr | $0.10/hr | - -### 5K Cloud Native Hybrid on EKS - -**5K Cloud Native Hybrid on EKS Bill of Materials (BOM)** - -**GPT Test Results** - -- [5K Full Fixed Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128_results.txt) - -- [5K AutoScale from 25% GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-AutoScale-From-25Percent-ARM-RDS-Redis_v13-12-3-ee_2021-07-24_102717/5k-QuickStart-AutoScale-From-25Percent-ARM-RDS-Redis_v13-12-3-ee_2021-07-24_102717_results.txt) - - Elastic Auto Scale GPT Test Results start with an idle scaled cluster and then start the standard GPT test to determine if the EKS Auto Scaler performs well enough to keep up with performance test demands. In general this is substantially harder ramping than the scaling required when the ramping is driven by standard production workloads. - -**Deploy Now** - -Deploy Now links leverage the AWS Quick Start automation and only prepopulate the number of instances and instance types for the Quick Start based on the Bill of Materials below. You must provide appropriate input for all other parameters by following the guidance in the [Quick Start documentation's Deployment steps](https://aws-quickstart.github.io/quickstart-eks-gitlab/#_deployment_steps) section. - -- **[Deploy Now: AWS Quick Start for 2 AZs](https://us-east-2.console.aws.amazon.com/cloudformation/home?region=us-east-2#/stacks/quickcreate?templateUrl=https://aws-quickstart.s3.us-east-1.amazonaws.com/quickstart-eks-gitlab/templates/gitlab-entry-new-vpc.template.yaml&stackName=Gitlab-EKS-5K-Users-2AZs¶m_NumberOfAZs=2¶m_NodeInstanceType=c5.2xlarge¶m_NumberOfNodes=5¶m_MaxNumberOfNodes=5¶m_DBInstanceClass=db.r6g.2xlarge¶m_CacheNodes=2¶m_CacheNodeType=cache.m6g.xlarge¶m_GitalyInstanceType=m5.2xlarge¶m_NumberOfGitalyReplicas=2¶m_PraefectInstanceType=c5.large¶m_NumberOfPraefectReplicas=2)** -- **[Deploy Now: AWS Quick Start for 3 AZs](https://us-east-2.console.aws.amazon.com/cloudformation/home?region=us-east-2#/stacks/quickcreate?templateUrl=https://aws-quickstart.s3.us-east-1.amazonaws.com/quickstart-eks-gitlab/templates/gitlab-entry-new-vpc.template.yaml&stackName=Gitlab-EKS-5K-Users-3AZs¶m_NumberOfAZs=3¶m_NodeInstanceType=c5.2xlarge¶m_NumberOfNodes=5¶m_MaxNumberOfNodes=5¶m_DBInstanceClass=db.r6g.2xlarge¶m_CacheNodes=3¶m_CacheNodeType=cache.m6g.xlarge¶m_GitalyInstanceType=m5.2xlarge¶m_NumberOfGitalyReplicas=3¶m_PraefectInstanceType=c5.large¶m_NumberOfPraefectReplicas=3)** - -NOTE: -On Demand pricing is used in this table for comparisons, but should not be used for budgeting nor purchasing AWS resources for a GitLab production instance. Do not use these tables to calculate actual monthly or yearly price estimates, instead use the AWS Calculator links in the "GitLab on AWS Compute" table above and customize it with your desired savings plan. - -**BOM Total:** = Bill of Materials Total - this is what you use when building this configuration - -**Ref Arch Raw Total:** = The totals if the configuration was built on regular VMs with no PaaS services. Configuring on pure VMs generally requires additional VMs for cluster management activities. - -**Idle Configuration (Scaled-In)** = can be used to scale-in during time of low demand and/or for warm standby Geo instances. Requires configuration, testing and management of EKS autoscaling to meet your internal requirements. - -| Service | Ref Arch Raw (Full Scaled) | AWS BOM | Example Full Scaled Cost<br />(On Demand, US East) | -| ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | -------------------------------------------------- | -| Webservice | [10 pods](https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/examples/ref/5k.yaml#L7) x ([5 vCPU & 6.25GB](../../administration/reference_architectures/5k_users.md#webservice)) = <br />50 vCPU, 62.5 GB | | | -| Sidekiq | [8 pods](https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/examples/ref/5k.yaml#L24) x ([1 vCPU & 2 GB](../../administration/reference_architectures/5k_users.md#sidekiq)) = <br />8 vCPU, 16 GB | | | -| Supporting services such as NGINX, Prometheus, etc | [2 allocations](../../administration/reference_architectures/5k_users.md#cluster-topology) x ([2 vCPU and 7.5 GB](../../administration/reference_architectures/5k_users.md#cluster-topology)) = <br />4 vCPU, 15 GB | | | -| **GitLab Ref Arch Raw Total K8s Node Capacity** | 62 vCPU, 96.5 GB | | | -| One Node for Quick Start Overhead and Miscellaneous (EKS Cluster AutoScaler, Grafana, Prometheus, etc) | + 8 vCPU, 16 GB | | | -| **Grand Total w/ Overheads Full Scale**<br />Minimum hosts = 3 | 70 vCPU, 112.5 GB | **c5.2xlarge** (8vCPU/16 GB) x 9 nodes<br />72 vCPU, 144 GB<br />[Full Fixed Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128_results.txt) | $2.38/hr | -| **Possible Idle Configuration (Scaled-In 75% - round up)**<br />Pod autoscaling must be also adjusted to enable lower idling configuration. | 24 vCPU, 48 GB | c5.2xlarge x 7 | $1.85/hr | - -Other combinations of node type and quantity can be used to meet the Grand Total. Due to the CPU and memory requirements of pods, hosts that are overly small may have significant unused capacity. - -NOTE: -If EKS node autoscaling is employed, it is likely that your average loading will run lower than this, especially during non-working hours and weekends. - -| Non-Kubernetes Compute | Ref Arch Raw Total | AWS BOM<br />(Directly Usable in AWS Quick Start) | Example Cost<br />US East, 3 AZ | Example Cost<br />US East, 2 AZ | -| ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------- | ------------------------------- | ------------------------------------------------------------ | -| **Bastion Host (Quick Start)** | 1 HA instance in ASG | **t2.micro** for prod, **m4.2xlarge** for performance testing | | | -| **PostgreSQL**<br />Amazon RDS PostgreSQL Nodes Configuration (GPT tested) | 21vCPU, 51 GB <br />(across 9 nodes for PostgreSQL, PgBouncer, Consul)<br />Tested with Graviton ARM | **db.r6g.2xlarge** x 3 nodes <br />(24vCPU, 192 GB) | 3 nodes x $1.04 = $3.12/hr | 3 nodes x $1.04 = $3.12/hr | -| **Redis** | 9vCPU, 27GB<br />(across 6 nodes for Redis, Sentinel) | **cache.m6g.xlarge** x 3 nodes<br />(12vCPU, 39GB) | 3 nodes x $0.30 = $0.90/hr | 2 nodes x $0.30 = $0.60/hr | -| **<u>Gitaly Cluster</u>** [Details](gitlab_sre_for_aws.md#gitaly-sre-considerations) | | | | | -| Gitaly Instances (in ASG) | 24 vCPU, 90GB<br />([across 3 nodes](gitlab_sre_for_aws.md#gitaly-and-praefect-elections)) | **m5.2xlarge** x 3 nodes<br />(24 vCPU, 96GB) | $0.384 x 3 = $1.15/hr | [Gitaly & Praefect Must Have an Uneven Node Count for HA](gitlab_sre_for_aws.md#gitaly-and-praefect-elections) | -| Praefect (Instances in ASG with load balancer) | 6 vCPU, 5.4 GB<br />([across 3 nodes](gitlab_sre_for_aws.md#gitaly-and-praefect-elections)) | **c5.large** x 3 nodes<br />(6 vCPU, 12 GB) | $0.09 x 3 = $0.21/hr | [Gitaly & Praefect Must Have an Uneven Node Count for HA](gitlab_sre_for_aws.md#gitaly-and-praefect-elections) | -| Praefect PostgreSQL(1) (Amazon RDS) | 6 vCPU, 5.4 GB<br />([across 3 nodes](gitlab_sre_for_aws.md#gitaly-and-praefect-elections)) | Not applicable; reuses GitLab PostgreSQL | $0 | | -| Internal Load Balancing Node | 2 vCPU, 1.8 GB | AWS ELB | $0.10/hr | $0.10/hr | - -### 10K Cloud Native Hybrid on EKS - -**10K Cloud Native Hybrid on EKS Bill of Materials (BOM)** - -**GPT Test Results** - -- [10K Full Fixed Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/10K/GL-CloudNative-10k-RDS-Graviton_v13-12-3-ee_2021-07-08_194647/GL-CloudNative-10k-RDS-Graviton_v13-12-3-ee_2021-07-08_194647_results.txt) - -- [10K Elastic Auto Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/10K/GL-CloudNative-10k-AutoScaling-Test_v13-12-3-ee_2021-07-09_115139/GL-CloudNative-10k-AutoScaling-Test_v13-12-3-ee_2021-07-09_115139_results.txt) - - Elastic Auto Scale GPT Test Results start with an idle scaled cluster and then start the standard GPT test to determine if the EKS Auto Scaler performs well enough to keep up with performance test demands. In general this is substantially harder ramping than the scaling required when the ramping is driven by standard production workloads. - -**Deploy Now** - -Deploy Now links leverage the AWS Quick Start automation and only prepopulate the number of instances and instance types for the Quick Start based on the Bill of Materials below. You must provide appropriate input for all other parameters by following the guidance in the [Quick Start documentation's Deployment steps](https://aws-quickstart.github.io/quickstart-eks-gitlab/#_deployment_steps) section. - -- **[Deploy Now: AWS Quick Start for 3 AZs](https://us-east-2.console.aws.amazon.com/cloudformation/home?region=us-east-2#/stacks/quickcreate?templateUrl=https://aws-quickstart.s3.us-east-1.amazonaws.com/quickstart-eks-gitlab/templates/gitlab-entry-new-vpc.template.yaml&stackName=Gitlab-EKS-10K-Users-3AZs¶m_NumberOfAZs=3¶m_NodeInstanceType=c5.4xlarge¶m_NumberOfNodes=9¶m_MaxNumberOfNodes=9¶m_DBInstanceClass=db.r6g.2xlarge¶m_CacheNodes=3¶m_CacheNodeType=cache.m6g.2xlarge¶m_GitalyInstanceType=m5.4xlarge¶m_NumberOfGitalyReplicas=3¶m_PraefectInstanceType=c5.large¶m_NumberOfPraefectReplicas=3)** - -NOTE: -On Demand pricing is used in this table for comparisons, but should not be used for budgeting nor purchasing AWS resources for a GitLab production instance. Do not use these tables to calculate actual monthly or yearly price estimates, instead use the AWS Calculator links in the "GitLab on AWS Compute" table above and customize it with your desired savings plan. - -**BOM Total:** = Bill of Materials Total - this is what you use when building this configuration - -**Ref Arch Raw Total:** = The totals if the configuration was built on regular VMs with no PaaS services. Configuring on pure VMs generally requires additional VMs for cluster management activities. - - **Idle Configuration (Scaled-In)** = can be used to scale-in during time of low demand and/or for warm standby Geo instances. Requires configuration, testing and management of EKS autoscaling to meet your internal requirements. - -| Service | Ref Arch Raw (Full Scaled) | AWS BOM<br />(Directly Usable in AWS Quick Start) | Example Full Scaled Cost<br />(On Demand, US East) | -| ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | -------------------------------------------------- | -| Webservice | [20 pods](https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/examples/ref/10k.yaml#L7) x ([5 vCPU & 6.25 GB](../../administration/reference_architectures/10k_users.md#webservice)) = <br />100 vCPU, 125 GB | | | -| Sidekiq | [14 pods](https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/examples/ref/10k.yaml#L24) x ([1 vCPU & 2 GB](../../administration/reference_architectures/10k_users.md#sidekiq))<br />14 vCPU, 28 GB | | | -| Supporting services such as NGINX, Prometheus, etc | [2 allocations](../../administration/reference_architectures/10k_users.md#cluster-topology) x ([2 vCPU and 7.5 GB](../../administration/reference_architectures/10k_users.md#cluster-topology))<br />4 vCPU, 15 GB | | | -| **GitLab Ref Arch Raw Total K8s Node Capacity** | 128 vCPU, 158 GB | | | -| One Node for Overhead and Miscellaneous (EKS Cluster AutoScaler, Grafana, Prometheus, etc) | + 16 vCPU, 32GB | | | -| **Grand Total w/ Overheads Fully Scaled**<br />Minimum hosts = 3 | 142 vCPU, 190 GB | **c5.4xlarge** (16vCPU/32GB) x 9 nodes<br />144 vCPU, 288GB<br /><br />[Full Fixed Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/10K/GL-CloudNative-10k-RDS-Graviton_v13-12-3-ee_2021-07-08_194647/GL-CloudNative-10k-RDS-Graviton_v13-12-3-ee_2021-07-08_194647_results.txt) | $6.12/hr | -| **Possible Idle Configuration (Scaled-In 75% - round up)**<br />Pod autoscaling must be also adjusted to enable lower idling configuration. | 40 vCPU, 80 GB | c5.4xlarge x 7<br /><br />[Elastic Auto Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/10K/GL-CloudNative-10k-AutoScaling-Test_v13-12-3-ee_2021-07-09_115139/GL-CloudNative-10k-AutoScaling-Test_v13-12-3-ee_2021-07-09_115139_results.txt) | $4.76/hr | - -Other combinations of node type and quantity can be used to meet the Grand Total. Due to the CPU and memory requirements of pods, hosts that are overly small may have significant unused capacity. - -NOTE: -If EKS node autoscaling is employed, it is likely that your average loading will run lower than this, especially during non-working hours and weekends. - -| Non-Kubernetes Compute | Ref Arch Raw Total | AWS BOM | Example Cost<br />US East, 3 AZ | Example Cost<br />US East, 2 AZ | -| ------------------------------------------------------------ | ------------------------------ | ------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | -| **Bastion Host (Quick Start)** | 1 HA instance in ASG | **t2.micro** for prod, **m4.2xlarge** for performance testing | | | -| **PostgreSQL**<br />Amazon RDS PostgreSQL Nodes Configuration (GPT tested) | 36vCPU, 102 GB <br />(across 9 nodes for PostgreSQL, PgBouncer, Consul) | **db.r6g.2xlarge** x 3 nodes <br />(24vCPU, 192 GB) | 3 nodes x $1.04 = $3.12/hr | 3 nodes x $1.04 = $3.12/hr | -| **Redis** | 30vCPU, 114 GB<br />(across 12 nodes for Redis Cache, Redis Queues/Shared State, Sentinel Cache, Sentinel Queues/Shared State) | **cache.m5.2xlarge** x 3 nodes<br />(24vCPU, 78GB) | 3 nodes x $0.62 = $1.86/hr | 2 nodes x $0.62 = $1.24/hr | -| **<u>Gitaly Cluster</u>** [Details](gitlab_sre_for_aws.md#gitaly-sre-considerations) | | | | | -| Gitaly Instances (in ASG) | 48 vCPU, 180 GB<br />([across 3 nodes](gitlab_sre_for_aws.md#gitaly-and-praefect-elections)) | **m5.4xlarge** x 3 nodes<br />(48 vCPU, 180 GB) | $0.77 x 3 = $2.31/hr | [Gitaly & Praefect Must Have an Uneven Node Count for HA](gitlab_sre_for_aws.md#gitaly-and-praefect-elections) | -| Praefect (Instances in ASG with load balancer) | 6 vCPU, 5.4 GB<br />([across 3 nodes](gitlab_sre_for_aws.md#gitaly-and-praefect-elections)) | **c5.large** x 3 nodes<br />(6 vCPU, 12 GB) | $0.09 x 3 = $0.21/hr | [Gitaly & Praefect Must Have an Uneven Node Count for HA](gitlab_sre_for_aws.md#gitaly-and-praefect-elections) | -| Praefect PostgreSQL(1) (Amazon RDS) | 6 vCPU, 5.4 GB<br />([across 3 nodes](gitlab_sre_for_aws.md#gitaly-and-praefect-elections)) | Not applicable; reuses GitLab PostgreSQL | $0 | | -| Internal Load Balancing Node | 2 vCPU, 1.8 GB | AWS ELB | $0.10/hr | $0.10/hr | - -### 50K Cloud Native Hybrid on EKS - -**50K Cloud Native Hybrid on EKS Bill of Materials (BOM)** - -**GPT Test Results** - -- [50K Full Fixed Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/50K/50k-Fixed-Scale-Test_v13-12-3-ee_2021-08-13_172819/50k-Fixed-Scale-Test_v13-12-3-ee_2021-08-13_172819_results.txt) - -- [50K Elastic Auto Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/50K/50k-AutoScale-Test_v13-12-3-ee_2021-08-13_192633/50k-AutoScale-Test_v13-12-3-ee_2021-08-13_192633.txt) - - Elastic Auto Scale GPT Test Results start with an idle scaled cluster and then start the standard GPT test to determine if the EKS Auto Scaler performs well enough to keep up with performance test demands. In general this is substantially harder ramping than the scaling required when the ramping is driven by standard production workloads. - -**Deploy Now** - -Deploy Now links leverage the AWS Quick Start automation and only prepopulate the number of instances and instance types for the Quick Start based on the Bill of Materials below. You must provide appropriate input for all other parameters by following the guidance in the [Quick Start documentation's Deployment steps](https://aws-quickstart.github.io/quickstart-eks-gitlab/#_deployment_steps) section. - -- **[Deploy Now: AWS Quick Start for 3 AZs - 1/4 Scale EKS](https://us-east-2.console.aws.amazon.com/cloudformation/home?region=us-east-2#/stacks/quickcreate?templateUrl=https://aws-quickstart.s3.us-east-1.amazonaws.com/quickstart-eks-gitlab/templates/gitlab-entry-new-vpc.template.yaml&stackName=Gitlab-EKS-50K-Users-3AZs¶m_NumberOfAZs=3¶m_NodeInstanceType=c5.4xlarge¶m_NumberOfNodes=7¶m_MaxNumberOfNodes=9¶m_DBInstanceClass=db.r6g.8xlarge¶m_CacheNodes=3¶m_CacheNodeType=cache.m6g.2xlarge¶m_GitalyInstanceType=m5.16xlarge¶m_NumberOfGitalyReplicas=3¶m_PraefectInstanceType=c5.xlarge¶m_NumberOfPraefectReplicas=3)** - -NOTE: -On Demand pricing is used in this table for comparisons, but should not be used for budgeting nor purchasing AWS resources for a GitLab production instance. Do not use these tables to calculate actual monthly or yearly price estimates, instead use the AWS Calculator links in the "GitLab on AWS Compute" table above and customize it with your desired savings plan. - -**BOM Total:** = Bill of Materials Total - this is what you use when building this configuration - -**Ref Arch Raw Total:** = The totals if the configuration was built on regular VMs with no PaaS services. Configuring on pure VMs generally requires additional VMs for cluster management activities. - - **Idle Configuration (Scaled-In)** = can be used to scale-in during time of low demand and/or for warm standby Geo instances. Requires configuration, testing and management of EKS autoscaling to meet your internal requirements. - -| Service | Ref Arch Raw (Full Scaled) | AWS BOM<br />(Directly Usable in AWS Quick Start) | Example Full Scaled Cost<br />(On Demand, US East) | -| ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | -------------------------------------------------- | -| Webservice | [80 pods](https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/examples/ref/10k.yaml#L7) x ([5 vCPU & 6.25 GB](../../administration/reference_architectures/10k_users.md#webservice)) = <br />400 vCPU, 500 GB | | | -| Sidekiq | [14 pods](https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/examples/ref/10k.yaml#L24) x ([1 vCPU & 2 GB](../../administration/reference_architectures/10k_users.md#sidekiq))<br />14 vCPU, 28 GB | | | -| Supporting services such as NGINX, Prometheus, etc | [2 allocations](../../administration/reference_architectures/10k_users.md#cluster-topology) x ([2 vCPU and 7.5 GB](../../administration/reference_architectures/10k_users.md#cluster-topology))<br />4 vCPU, 15 GB | | | -| **GitLab Ref Arch Raw Total K8s Node Capacity** | 428 vCPU, 533 GB | | | -| One Node for Overhead and Miscellaneous (EKS Cluster AutoScaler, Grafana, Prometheus, etc) | + 16 vCPU, 32GB | | | -| **Grand Total w/ Overheads Fully Scaled**<br />Minimum hosts = 3 | 444 vCPU, 565 GB | **c5.4xlarge** (16vCPU/32GB) x 28 nodes<br />448 vCPU, 896GB<br /><br />[Full Fixed Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/50K/50k-Fixed-Scale-Test_v13-12-3-ee_2021-08-13_172819/50k-Fixed-Scale-Test_v13-12-3-ee_2021-08-13_172819_results.txt) | $19.04/hr | -| **Possible Idle Configuration (Scaled-In 75% - round up)**<br />Pod autoscaling must be also adjusted to enable lower idling configuration. | 40 vCPU, 80 GB | c5.2xlarge x 10<br /><br />[Elastic Auto Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/50K/50k-AutoScale-Test_v13-12-3-ee_2021-08-13_192633/50k-AutoScale-Test_v13-12-3-ee_2021-08-13_192633.txt) | $6.80/hr | - -Other combinations of node type and quantity can be used to meet the Grand Total. Due to the CPU and memory requirements of pods, hosts that are overly small may have significant unused capacity. - -NOTE: -If EKS node autoscaling is employed, it is likely that your average loading will run lower than this, especially during non-working hours and weekends. - -| Non-Kubernetes Compute | Ref Arch Raw Total | AWS BOM | Example Cost<br />US East, 3 AZ | Example Cost<br />US East, 2 AZ | -| ------------------------------------------------------------ | ------------------------------------------------------------ | --------------------------------------------------------- | ------------------------------- | ------------------------------------------------------------ | -| **Bastion Host (Quick Start)** | 1 HA instance in ASG | **t2.micro** for prod, **m4.2xlarge** for performance testing | | | -| **PostgreSQL**<br />Amazon RDS PostgreSQL Nodes Configuration (GPT tested) | 96vCPU, 360 GB <br />(across 3 nodes) | **db.r6g.8xlarge** x 3 nodes <br />(96vCPU, 768 GB total) | 3 nodes x $4.15 = $12.45/hr | 3 nodes x $4.15 = $12.45/hr | -| **Redis** | 30vCPU, 114 GB<br />(across 12 nodes for Redis Cache, Redis Queues/Shared State, Sentinel Cache, Sentinel Queues/Shared State) | **cache.m6g.2xlarge** x 3 nodes<br />(24vCPU, 78GB total) | 3 nodes x $0.60 = $1.80/hr | 2 nodes x $0.60 = $1.20/hr | -| **<u>Gitaly Cluster</u>** [Details](gitlab_sre_for_aws.md#gitaly-sre-considerations) | | | | | -| Gitaly Instances (in ASG) | 64 vCPU, 240GB x [3 nodes](gitlab_sre_for_aws.md#gitaly-and-praefect-elections) | **m5.16xlarge** x 3 nodes<br />(64 vCPU, 256 GB each) | $3.07 x 3 = $9.21/hr | [Gitaly & Praefect Must Have an Uneven Node Count for HA](gitlab_sre_for_aws.md#gitaly-and-praefect-elections) | -| Praefect (Instances in ASG with load balancer) | 4 vCPU, 3.6 GB x [3 nodes](gitlab_sre_for_aws.md#gitaly-and-praefect-elections) | **c5.xlarge** x 3 nodes<br />(4 vCPU, 8 GB each) | $0.17 x 3 = $0.51/hr | [Gitaly & Praefect Must Have an Uneven Node Count for HA](gitlab_sre_for_aws.md#gitaly-and-praefect-elections) | -| Praefect PostgreSQL(1) (AWS RDS) | 2 vCPU, 1.8 GB x [3 nodes](gitlab_sre_for_aws.md#gitaly-and-praefect-elections) | Not applicable; reuses GitLab PostgreSQL | $0 | | -| Internal Load Balancing Node | 2 vCPU, 1.8 GB | AWS ELB | $0.10/hr | $0.10/hr | - -## Helpful Resources - -- [Architecting Kubernetes clusters — choosing a worker node size](https://learnk8s.io/kubernetes-node-size) - -DISCLAIMER: -This page contains information related to upcoming products, features, and functionality. -It is important to note that the information presented is for informational purposes only. -Please do not rely on this information for purchasing or planning purposes. -As with all projects, the items mentioned on this page are subject to change or delay. -The development, release, and timing of any products, features, or functionality remain at the -sole discretion of GitLab Inc. +<!-- This redirect file can be deleted after <YYYY-MM-DD>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
\ No newline at end of file diff --git a/doc/install/aws/gitlab_sre_for_aws.md b/doc/install/aws/gitlab_sre_for_aws.md index 5f3fe9fefac..222bcbc1ed8 100644 --- a/doc/install/aws/gitlab_sre_for_aws.md +++ b/doc/install/aws/gitlab_sre_for_aws.md @@ -1,95 +1,11 @@ --- -stage: Systems -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -description: Doing SRE for GitLab instances and runners on AWS. +redirect_to: '../../solutions/cloud/aws/gitaly_sre_for_aws.md' +remove_date: '2024-03-31' --- -# GitLab Site Reliability Engineering for AWS **(FREE SELF)** +This document was moved to [Solutions](../../solutions/cloud/aws/gitaly_sre_for_aws.md). -## Gitaly SRE considerations - -Gitaly is an embedded service for Git Repository Storage. Gitaly and Gitaly Cluster have been engineered by GitLab to overcome fundamental challenges with horizontal scaling of the open source Git binaries that must be used on the service side of GitLab. Here is in-depth technical reading on the topic: - -### Why Gitaly was built - -If you would like to understand the underlying rationale on why GitLab had to invest in creating Gitaly, read the following minimal list of topics: - -- [Git characteristics that make horizontal scaling difficult](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/DESIGN.md#git-characteristics-that-make-horizontal-scaling-difficult) -- [Git architectural characteristics and assumptions](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/DESIGN.md#git-architectural-characteristics-and-assumptions) -- [Affects on horizontal compute architecture](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/DESIGN.md#affects-on-horizontal-compute-architecture) -- [Evidence to back building a new horizontal layer to scale Git](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/DESIGN.md#evidence-to-back-building-a-new-horizontal-layer-to-scale-git) - -### Gitaly and Praefect elections - -As part of Gitaly cluster consistency, Praefect nodes must occasionally vote on what data copy is the most accurate. This requires an uneven number of Praefect nodes to avoid stalemates. This means that for HA, Gitaly and Praefect require a minimum of three nodes. - -### Gitaly performance monitoring - -Complete performance metrics should be collected for Gitaly instances for identification of bottlenecks, as they could have to do with disk IO, network IO, or memory. - -### Gitaly performance guidelines - -Gitaly functions as the primary Git Repository Storage in GitLab. However, it's not a streaming file server. It also does a lot of demanding computing work, such as preparing and caching Git packfiles which informs some of the performance recommendations below. - -NOTE: -All recommendations are for production configurations, including performance testing. For test configurations, like training or functional testing, you can use less expensive options. However, you should adjust or rebuild if performance is an issue. - -#### Overall recommendations - -- Production-grade Gitaly must be implemented on instance compute due to all of the above and below characteristics. -- Never use [burstable instance types](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-performance-instances.html) (such as `t2`, `t3`, `t4g`) for Gitaly. -- Always use at least the [AWS Nitro generation of instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-types.html#ec2-nitro-instances) to ensure many of the below concerns are automatically handled. -- Use Amazon Linux 2 to ensure that all [AWS oriented hardware and OS optimizations](https://aws.amazon.com/amazon-linux-2/faqs/) are maximized without additional configuration or SRE management. - -#### CPU and memory recommendations - -- The general GitLab Gitaly node recommendations for CPU and Memory assume relatively even loading across repositories. GitLab Performance Tool (GPT) testing of any non-characteristic repositories and/or SRE monitoring of Gitaly metrics may inform when to choose memory and/or CPU higher than general recommendations. - -**To accommodate:** - -- Git packfile operations are memory and CPU intensive. -- If repository commit traffic is dense, large, or very frequent, then more CPU and Memory are required to handle the load. Patterns such as storing binaries and/or busy or large monorepos are examples that can cause high loading. - -#### Disk I/O recommendations - -- Use only SSD storage and the [class of Elastic Block Store (EBS) storage](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html) that suites your durability and speed requirements. -- When not using provisioned EBS IO, EBS volume size determines the I/O level, so provisioning volumes that are much larger than needed can be the least expensive way to improve EBS IO. -- If Gitaly performance monitoring shows signs of disk stress then one of the provisioned IOPS levels can be chosen. EBS IOPS levels also have enhanced durability which may be appealing for some implementations aside from performance considerations. - -**To accommodate:** - -- Gitaly storage is expected to be local (not NFS of any type including EFS). -- Gitaly servers also need disk space for building and caching Git packfiles. This is above and beyond the permanent storage of your Git Repositories. -- Git packfiles are cached in Gitaly. Creation of packfiles in temporary disk benefits from fast disk, and disk caching of packfiles benefits from ample disk space. - -#### Network I/O recommendations - -- Use only instance types [from the list of ones that support Elastic Network Adapter (ENA) advanced networking](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-types.html#instance-type-summary-table) to ensure that cluster replication latency is not due to instance level network I/O bottlenecks. -- Choose instances with sizes with more than 10 Gbps - but only if needed and only when having proven a node level network bottleneck with monitoring and/or stress testing. - -**To accommodate:** - -- Gitaly nodes do the main work of streaming repositories for push and pull operations (to add development endpoints, and to CI/CD). -- Gitaly servers need reasonable low latency between cluster nodes and with Praefect services in order for the cluster to maintain operational and data integrity. -- Gitaly nodes should be selected with network bottleneck avoidance as a primary consideration. -- Gitaly nodes should be monitored for network saturation. -- Not all networking issues can be solved through optimizing the node level networking: - - Gitaly cluster node replication depends on all networking between nodes. - - Gitaly networking performance to pull and push endpoints depends on all networking in between. - -### AWS Gitaly backup - -Due to the nature of how Praefect tracks the replication metadata of Gitaly disk information, the best backup method is [the official backup and restore Rake tasks](../../administration/backup_restore/index.md). - -### AWS Gitaly recovery - -Gitaly Cluster does not support snapshot backups as these can cause issues where the Praefect database becomes out of syn with the disk storage. Due to the nature of how Praefect rebuilds the replication metadata of Gitaly disk information during a restore, the best recovery method is [the official backup and restore Rake tasks](../../administration/backup_restore/index.md). - -### Gitaly HA in EKS quick start - -The [AWS GitLab Cloud Native Hybrid on EKS Quick Start](gitlab_hybrid_on_aws.md#available-infrastructure-as-code-for-gitlab-cloud-native-hybrid) for GitLab Cloud Native implements Gitaly as a multi-zone, self-healing infrastructure. It has specific code for reestablishing a Gitaly node when one fails, including AZ failure. - -### Gitaly long term management - -Gitaly node disk sizes must be monitored and increased to accommodate Git repository growth and Gitaly temporary and caching storage needs. The storage configuration on all nodes should be kept identical. +<!-- This redirect file can be deleted after <YYYY-MM-DD>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
\ No newline at end of file diff --git a/doc/install/aws/index.md b/doc/install/aws/index.md index febe54a8bb6..2c1f2529426 100644 --- a/doc/install/aws/index.md +++ b/doc/install/aws/index.md @@ -3,167 +3,857 @@ stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments description: Read through the GitLab installation methods. -type: index --- -# AWS implementation patterns **(FREE SELF)** +{::options parse_block_html="true" /} -GitLab [Reference Architectures](../../administration/reference_architectures/index.md) give qualified and tested guidance on the recommended ways GitLab can be configured to meet the performance requirements of various workloads. Reference Architectures are purpose-designed to be non-implementation specific so they can be extrapolated to as many environments as possible. This generally means they have a highly-granular "machine" to "server role" specification and focus on system elements that impact performance. This is what enables Reference Architectures to be adaptable to the broadest number of supported implementations. +# Installing a GitLab POC on Amazon Web Services (AWS) **(FREE SELF)** -Implementation patterns are built on the foundational information and testing done for Reference Architectures and allow architects and implementers at GitLab, GitLab Customers, and GitLab Partners to build out deployments with less experimentation and a higher degree of confidence that the results perform as expected. A more thorough discussion of implementation patterns is below in [Additional details on implementation patterns](#additional-details-on-implementation-patterns). +This page offers a walkthrough of a common configuration for GitLab on AWS using the official Linux package. You should customize it to accommodate your needs. -## AWS Implementation patterns information +NOTE: +For organizations with 1,000 users or less, the recommended AWS installation method is to launch an EC2 single box [Linux package installation](https://about.gitlab.com/install/) and implement a snapshot strategy for backing up the data. See the [1,000 user reference architecture](../../administration/reference_architectures/1k_users.md) for more information. + +## Getting started for production-grade GitLab + +NOTE: +This document is an installation guide for a proof of concept instance. It is not a reference architecture and it does not result in a highly available configuration. + +Following this guide exactly results in a proof of concept instance that roughly equates to a **scaled down** version of a **two availability zone implementation** of the **Non-HA** [2000 User Reference Architecture](../../administration/reference_architectures/2k_users.md). The 2K reference architecture is not HA because it is primarily intended to provide some scaling while keeping costs and complexity low. The [3000 User Reference Architecture](../../administration/reference_architectures/3k_users.md) is the smallest size that is GitLab HA. It has additional service roles to achieve HA, most notably it uses Gitaly Cluster to achieve HA for Git repository storage and specifies triple redundancy. + +GitLab maintains and tests two main types of Reference Architectures. The **Linux package architectures** are implemented on instance compute while **Cloud Native Hybrid architectures** maximize the use of a Kubernetes cluster. Cloud Native Hybrid reference architecture specifications are addendum sections to the Reference Architecture size pages that start by describing the Linux package architecture. For example, the 3000 User Cloud Native Reference Architecture is in the subsection titled [Cloud Native Hybrid reference architecture with Helm Charts (alternative)](../../administration/reference_architectures/3k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) in the 3000 User Reference Architecture page. + +### Getting started for production-grade Linux package installations + +The Infrastructure as Code tooling [GitLab Environment Tool (GET)](https://gitlab.com/gitlab-org/gitlab-environment-toolkit/-/tree/main) is the best place to start for building using the Linux package on AWS and most especially if you are targeting an HA setup. While it does not automate everything, it does complete complex setups like Gitaly Cluster for you. GET is open source so anyone can build on top of it and contribute improvements to it. + +### Getting started for production-grade Cloud Native Hybrid GitLab + +The [GitLab Environment Toolkit (GET)](https://gitlab.com/gitlab-org/gitlab-environment-toolkit/-/blob/main/README.md) is a set of opinionated Terraform and Ansible scripts. These scripts help with the deployment of Linux package or Cloud Native Hybrid environments on selected cloud providers and are used by GitLab developers for [GitLab Dedicated](../../subscriptions/gitlab_dedicated/index.md) (for example). + +You can use the GitLab Environment Toolkit to deploy a Cloud Native Hybrid environment on AWS. However, it's not required and may not support every valid permutation. That said, the scripts are presented as-is and you can adapt them accordingly. + +## Introduction + +For the most part, we make use of the Linux package in our setup, but we also leverage native AWS services. Instead of using the Linux package-bundled PostgreSQL and Redis, we use Amazon RDS and ElastiCache. + +In this guide, we go through a multi-node setup where we start by +configuring our Virtual Private Cloud and subnets to later integrate +services such as RDS for our database server and ElastiCache as a Redis +cluster to finally manage them in an auto scaling group with custom +scaling policies. + +## Requirements + +In addition to having a basic familiarity with [AWS](https://docs.aws.amazon.com/) and [Amazon EC2](https://docs.aws.amazon.com/ec2/), you need: + +- [An AWS account](https://console.aws.amazon.com/console/home) +- [To create or upload an SSH key](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html) + to connect to the instance via SSH +- A domain name for the GitLab instance +- An SSL/TLS certificate to secure your domain. If you do not already own one, you can provision a free public SSL/TLS certificate through [AWS Certificate Manager](https://aws.amazon.com/certificate-manager/)(ACM) for use with the [Elastic Load Balancer](#load-balancer) we create. -The following are the currently available implementation patterns for GitLab when it is implemented on AWS. +NOTE: +It can take a few hours to validate a certificate provisioned through ACM. To avoid delays later, request your certificate as soon as possible. + +## Architecture + +Below is a diagram of the recommended architecture. + + + +## AWS costs + +GitLab uses the following AWS services, with links to pricing information: + +- **EC2**: GitLab is deployed on shared hardware, for which + [on-demand pricing](https://aws.amazon.com/ec2/pricing/on-demand/) applies. + If you want to run GitLab on a dedicated or reserved instance, see the + [EC2 pricing page](https://aws.amazon.com/ec2/pricing/) for information about + its cost. +- **S3**: GitLab uses S3 ([pricing page](https://aws.amazon.com/s3/pricing/)) to + store backups, artifacts, and LFS objects. +- **ELB**: A Classic Load Balancer ([pricing page](https://aws.amazon.com/elasticloadbalancing/pricing/)), + used to route requests to the GitLab instances. +- **RDS**: An Amazon Relational Database Service using PostgreSQL + ([pricing page](https://aws.amazon.com/rds/postgresql/pricing/)). +- **ElastiCache**: An in-memory cache environment ([pricing page](https://aws.amazon.com/elasticache/pricing/)), + used to provide a Redis configuration. + +## Create an IAM EC2 instance role and profile + +As we are using [Amazon S3 object storage](#amazon-s3-object-storage), our EC2 instances must have read, write, and list permissions for our S3 buckets. To avoid embedding AWS keys in our GitLab configuration, we make use of an [IAM Role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) to allow our GitLab instance with this access. We must create an IAM policy to attach to our IAM role: + +### Create an IAM Policy + +1. Go to the IAM dashboard and select **Policies** in the left menu. +1. Select **Create policy**, select the `JSON` tab, and add a policy. We want to [follow security best practices and grant _least privilege_](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege), giving our role only the permissions needed to perform the required actions. + 1. Assuming you prefix the S3 bucket names with `gl-` as shown in the diagram, add the following policy: + + ```json + { "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "s3:PutObject", + "s3:GetObject", + "s3:DeleteObject", + "s3:PutObjectAcl" + ], + "Resource": "arn:aws:s3:::gl-*/*" + }, + { + "Effect": "Allow", + "Action": [ + "s3:ListBucket", + "s3:AbortMultipartUpload", + "s3:ListMultipartUploadParts", + "s3:ListBucketMultipartUploads" + ], + "Resource": "arn:aws:s3:::gl-*" + } + ] + } + ``` + +1. Select **Review policy**, give your policy a name (we use `gl-s3-policy`), and select **Create policy**. + +### Create an IAM Role + +1. Still on the IAM dashboard, select **Roles** in the left menu, and + select **Create role**. +1. Create a new role by selecting **AWS service > EC2**, then select + **Next: Permissions**. +1. In the policy filter, search for the `gl-s3-policy` we created above, select it, and select **Tags**. +1. Add tags if needed and select **Review**. +1. Give the role a name (we use `GitLabS3Access`) and select **Create Role**. + +We use this role when we [create a launch configuration](#create-a-launch-configuration) later on. + +## Configuring the network + +We start by creating a VPC for our GitLab cloud infrastructure, then +we can create subnets to have public and private instances in at least +two [Availability Zones (AZs)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html). Public subnets require a Route Table keep and an associated +Internet Gateway. -### GitLab Site Reliability Engineering (SRE) for AWS +### Creating the Virtual Private Cloud (VPC) -[GitLab Site Reliability Engineering (SRE) for AWS](gitlab_sre_for_aws.md) - information for planning, implementing, upgrading, and long term management of GitLab instances and runners on AWS. +We now create a VPC, a virtual networking environment that you control: -### Patterns to Install GitLab Cloud Native Hybrid on AWS EKS (HA) +1. Sign in to [Amazon Web Services](https://console.aws.amazon.com/vpc/home). +1. Select **Your VPCs** from the left menu and then select **Create VPC**. + At the "Name tag" enter `gitlab-vpc` and at the "IPv4 CIDR block" enter + `10.0.0.0/16`. If you don't require dedicated hardware, you can leave + "Tenancy" as default. Select **Yes, Create** when ready. -[Provision GitLab Cloud Native Hybrid on AWS EKS (HA)](gitlab_hybrid_on_aws.md). This document includes instructions, patterns, and automation for installing GitLab Cloud Native Hybrid on AWS EKS. It also includes [Bill of Materials](https://en.wikipedia.org/wiki/Bill_of_materials) listings and links to Infrastructure as Code. GitLab Cloud Native Hybrid is the supported way to put as much of GitLab as possible into Kubernetes. +  -### Patterns to Install GitLab by using the Linux package on AWS EC2 (HA) +1. Select the VPC, select **Actions**, select **Edit DNS resolution**, and enable DNS resolution. Select **Save** when done. -[Installing a GitLab POC on Amazon Web Services (AWS)](manual_install_aws.md) - instructions for installing GitLab on EC2 instances. Manual instructions to build a GitLab instance or create your own Infrastructure as Code (IaC). +### Subnets -### Patterns for EKS cluster provisioning +Now, let's create some subnets in different Availability Zones. Make sure +that each subnet is associated to the VPC we just created and +that CIDR blocks don't overlap. This also +allows us to enable multi AZ for redundancy. -[EKS Cluster Provisioning Patterns](eks_clusters_aws.md) - considerations for setting up EKS cluster for runners and for integrating. +We create private and public subnets to match load balancers and +RDS instances as well: -### Patterns for Scaling HA GitLab Runner on AWS EC2 Auto Scaling group (ASG) +1. Select **Subnets** from the left menu. +1. Select **Create subnet**. Give it a descriptive name tag based on the IP, + for example `gitlab-public-10.0.0.0`, select the VPC we created previously, select an availability zone (we use `us-west-2a`), + and at the IPv4 CIDR block let's give it a 24 subnet `10.0.0.0/24`: + +  + +1. Follow the same steps to create all subnets: + + | Name tag | Type | Availability Zone | CIDR block | + | ------------------------- | ------- | ----------------- | ------------- | + | `gitlab-public-10.0.0.0` | public | `us-west-2a` | `10.0.0.0/24` | + | `gitlab-private-10.0.1.0` | private | `us-west-2a` | `10.0.1.0/24` | + | `gitlab-public-10.0.2.0` | public | `us-west-2b` | `10.0.2.0/24` | + | `gitlab-private-10.0.3.0` | private | `us-west-2b` | `10.0.3.0/24` | -The following repository is self-contained in regard to enabling this pattern: [GitLab HA Scaling Runner Vending Machine for AWS EC2 ASG](https://gitlab.com/guided-explorations/aws/gitlab-runner-autoscaling-aws-asg/). The [feature list for this implementation pattern](https://gitlab.com/guided-explorations/aws/gitlab-runner-autoscaling-aws-asg/-/blob/main/FEATURES.md) is good to review to understand the complete value it can deliver. +1. Once all the subnets are created, enable **Auto-assign IPv4** for the two public subnets: + 1. Select each public subnet in turn, select **Actions**, and select **Modify auto-assign IP settings**. Enable the option and save. -### Patterns for Using GitLab with AWS +### Internet Gateway -[The Guided Explorations' subgroup for AWS](https://gitlab.com/guided-explorations/aws) contains a variety of working example projects for: +Now, still on the same dashboard, go to Internet Gateways and +create a new one: -- Using GitLab and AWS together. -- Running GitLab infrastructure on AWS. -- Retrieving temporary credentials for access to AWS services. +1. Select **Internet Gateways** from the left menu. +1. Select **Create internet gateway**, give it the name `gitlab-gateway` and + select **Create**. +1. Select it from the table, and then under the **Actions** dropdown list choose + "Attach to VPC". + +  + +1. Choose `gitlab-vpc` from the list and hit **Attach**. + +### Create NAT Gateways + +Instances deployed in our private subnets must connect to the internet for updates, but should not be reachable from the public internet. To achieve this, we make use of [NAT Gateways](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html) deployed in each of our public subnets: -## AWS known issues list +1. Go to the VPC dashboard and select **NAT Gateways** in the left menu bar. +1. Select **Create NAT Gateway** and complete the following: + 1. **Subnet**: Select `gitlab-public-10.0.0.0` from the dropdown list. + 1. **Elastic IP Allocation ID**: Enter an existing Elastic IP or select **Allocate Elastic IP address** to allocate a new IP to your NAT gateway. + 1. Add tags if needed. + 1. Select **Create NAT Gateway**. -Known issues are gathered from within GitLab and from customer reported issues. Customers successfully implement GitLab with a variety of "as a Service" components that GitLab has not specifically been designed for, nor has ongoing testing for. While GitLab does take partner technologies very seriously, the highlighting of known issues here is a convenience for implementers and it does not imply that GitLab has targeted compatibility with, nor carries any type of guarantee of running on the partner technology where the issues occur. Consult individual issues to understand the GitLab stance and plans on any given known issue. +Create a second NAT gateway but this time place it in the second public subnet, `gitlab-public-10.0.2.0`. -See the [GitLab AWS known issues list](https://gitlab.com/gitlab-com/alliances/aws/public-tracker/-/issues?label_name%5B%5D=AWS+Known+Issue) for a complete list. +### Route Tables -## Provision a single GitLab instance on AWS +#### Public Route Table -If you want to provision a single GitLab instance on AWS, you have two options: +We must create a route table for our public subnets to reach the internet via the internet gateway we created in the previous step. -- The marketplace subscription -- The official GitLab AMIs +On the VPC dashboard: -### Marketplace subscription +1. Select **Route Tables** from the left menu. +1. Select **Create Route Table**. +1. At the "Name tag" enter `gitlab-public` and choose `gitlab-vpc` under "VPC". +1. Select **Create**. -GitLab provides a 5 user subscription as an AWS Marketplace subscription to help teams of all sizes to get started with an Ultimate licensed instance in record time. The Marketplace subscription can be easily upgraded to any GitLab licensing via an AWS Marketplace Private Offer, with the convenience of continued AWS billing. No migration is necessary to obtain a larger, non-time based license from GitLab. Per-minute licensing is automatically removed when you accept the private offer. +We now must add our internet gateway as a new target and have +it receive traffic from any destination. -For a tutorial on provisioning a GitLab Instance via a Marketplace Subscription, [use this tutorial](https://gitlab.awsworkshop.io/040_partner_setup.html). The tutorial links to the [GitLab Ultimate Marketplace Listing](https://aws.amazon.com/marketplace/pp/prodview-g6ktjmpuc33zk), but you can also use the [GitLab Premium Marketplace Listing](https://aws.amazon.com/marketplace/pp/prodview-amk6tacbois2k) to provision an instance. +1. Select **Route Tables** from the left menu and select the `gitlab-public` + route to show the options at the bottom. +1. Select the **Routes** tab, select **Edit routes > Add route** and set `0.0.0.0/0` + as the destination. In the target column, select the `gitlab-gateway` we created previously. + Select **Save routes** when done. -### Official GitLab releases as AMIs +Next, we must associate the **public** subnets to the route table: -GitLab produces Amazon Machine Images (AMI) during the regular release process. The AMIs can be used for single instance GitLab installation or, by configuring `/etc/gitlab/gitlab.rb`, can be specialized for specific GitLab service roles (for example a Gitaly server). Older releases remain available and can be used to migrate an older GitLab server to AWS. +1. Select the **Subnet Associations** tab and select **Edit subnet associations**. +1. Check only the public subnets and select **Save**. -Initial licensing can either be the Free Enterprise License (EE) or the open source Community Edition (CE). The Enterprise Edition provides the easiest path forward to a licensed version if the need arises. +#### Private Route Tables -Currently the Amazon AMI uses the Amazon prepared Ubuntu AMI (x86 and ARM are available) as its starting point. +We also must create two private route tables so that instances in each private subnet can reach the internet via the NAT gateway in the corresponding public subnet in the same availability zone. + +1. Follow the same steps as above to create two private route tables. Name them `gitlab-private-a` and `gitlab-private-b`. +1. Next, add a new route to each of the private route tables where the destination is `0.0.0.0/0` and the target is one of the NAT gateways we created earlier. + 1. Add the NAT gateway we created in `gitlab-public-10.0.0.0` as the target for the new route in the `gitlab-private-a` route table. + 1. Similarly, add the NAT gateway in `gitlab-public-10.0.2.0` as the target for the new route in the `gitlab-private-b`. +1. Lastly, associate each private subnet with a private route table. + 1. Associate `gitlab-private-10.0.1.0` with `gitlab-private-a`. + 1. Associate `gitlab-private-10.0.3.0` with `gitlab-private-b`. + +## Load Balancer + +We create a load balancer to evenly distribute inbound traffic on ports `80` and `443` across our GitLab application servers. Based on the [scaling policies](#create-an-auto-scaling-group) we create later, instances are added to or removed from our load balancer as needed. Additionally, the load balancer performs health checks on our instances. + +On the EC2 dashboard, look for Load Balancer in the left navigation bar: + +1. Select **Create Load Balancer**. + 1. Choose the **Classic Load Balancer**. + 1. Give it a name (we use `gitlab-loadbalancer`) and for the **Create LB Inside** option, select `gitlab-vpc` from the dropdown list. + 1. In the **Listeners** section, set the following listeners: + - HTTP port 80 for both load balancer and instance protocol and ports + - TCP port 22 for both load balancer and instance protocols and ports + - HTTPS port 443 for load balancer protocol and ports, forwarding to HTTP port 80 on the instance (we configure GitLab to listen on port 80 [later in the guide](#add-support-for-proxied-ssl)) + 1. In the **Select Subnets** section, select both public subnets from the list so that the load balancer can route traffic to both availability zones. +1. We add a security group for our load balancer to act as a firewall to control what traffic is allowed through. Select **Assign Security Groups** and select **Create a new security group**, give it a name + (we use `gitlab-loadbalancer-sec-group`) and description, and allow both HTTP and HTTPS traffic + from anywhere (`0.0.0.0/0, ::/0`). Also allow SSH traffic, select a custom source, and add a single trusted IP address or an IP address range in CIDR notation. This allows users to perform Git actions over SSH. +1. Select **Configure Security Settings** and set the following: + 1. Select an SSL/TLS certificate from ACM or upload a certificate to IAM. + 1. Under **Select a Cipher**, pick a predefined security policy from the dropdown list. You can see a breakdown of [Predefined SSL Security Policies for Classic Load Balancers](https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-security-policy-table.html) in the AWS documentation. Check the GitLab codebase for a list of [supported SSL ciphers and protocols](https://gitlab.com/gitlab-org/gitlab/-/blob/9ee7ad433269b37251e0dd5b5e00a0f00d8126b4/lib/support/nginx/gitlab-ssl#L97-99). +1. Select **Configure Health Check** and set up a health check for your EC2 instances. + 1. For **Ping Protocol**, select HTTP. + 1. For **Ping Port**, enter 80. + 1. For **Ping Path** - we recommend that you [use the Readiness check endpoint](../../administration/load_balancer.md#readiness-check). You must add [the VPC IP Address Range (CIDR)](https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-security-groups.html#elb-vpc-nacl) to the [IP allowlist](../../administration/monitoring/ip_allowlist.md) for the [Health Check endpoints](../../administration/monitoring/health_check.md) + 1. Keep the default **Advanced Details** or adjust them according to your needs. +1. Select **Add EC2 Instances** - don't add anything as we create an Auto Scaling Group later to manage instances for us. +1. Select **Add Tags** and add any tags you need. +1. Select **Review and Create**, review all your settings, and select **Create** if you're happy. + +After the Load Balancer is up and running, you can revisit your Security +Groups to refine the access only through the ELB and any other requirements +you might have. + +### Configure DNS for Load Balancer + +On the Route 53 dashboard, select **Hosted zones** in the left navigation bar: + +1. Select an existing hosted zone or, if you do not already have one for your domain, select **Create Hosted Zone**, enter your domain name, and select **Create**. +1. Select **Create Record Set** and provide the following values: + 1. **Name:** Use the domain name (the default value) or enter a subdomain. + 1. **Type:** Select **A - IPv4 address**. + 1. **Alias:** Defaults to **No**. Select **Yes**. + 1. **Alias Target:** Find the **ELB Classic Load Balancers** section and select the classic load balancer we created earlier. + 1. **Routing Policy:** We use **Simple** but you can choose a different policy based on your use case. + 1. **Evaluate Target Health:** We set this to **No** but you can choose to have the load balancer route traffic based on target health. + 1. Select **Create**. +1. If you registered your domain through Route 53, you're done. If you used a different domain registrar, you must update your DNS records with your domain registrar. You must: + 1. Select **Hosted zones** and select the domain you added above. + 1. You see a list of `NS` records. From your domain registrar's administrator panel, add each of these as `NS` records to your domain's DNS records. These steps may vary between domain registrars. If you're stuck, Google **"name of your registrar" add DNS records** and you should find a help article specific to your domain registrar. + +The steps for doing this vary depending on which registrar you use and is beyond the scope of this guide. + +## PostgreSQL with RDS + +For our database server we use Amazon RDS for PostgreSQL which offers Multi AZ +for redundancy (Aurora is **not** supported). First we create a security group and subnet group, then we +create the actual RDS instance. + +### RDS Security Group + +We need a security group for our database that allows inbound traffic from the instances we deploy in our `gitlab-loadbalancer-sec-group` later on: + +1. From the EC2 dashboard, select **Security Groups** from the left menu bar. +1. Select **Create security group**. +1. Give it a name (we use `gitlab-rds-sec-group`), a description, and select the `gitlab-vpc` from the **VPC** dropdown list. +1. In the **Inbound rules** section, select **Add rule** and set the following: + 1. **Type:** search for and select the **PostgreSQL** rule. + 1. **Source type:** set as "Custom". + 1. **Source:** select the `gitlab-loadbalancer-sec-group` we created earlier. +1. When done, select **Create security group**. + +### RDS Subnet Group + +1. Go to the RDS dashboard and select **Subnet Groups** from the left menu. +1. Select **Create DB Subnet Group**. +1. Under **Subnet group details**, enter a name (we use `gitlab-rds-group`), a description, and choose the `gitlab-vpc` from the VPC dropdown list. +1. From the **Availability Zones** dropdown list, select the Availability Zones that include the subnets you've configured. In our case, we add `eu-west-2a` and `eu-west-2b`. +1. From the **Subnets** dropdown list, select the two private subnets (`10.0.1.0/24` and `10.0.3.0/24`) as we defined them in the [subnets section](#subnets). +1. Select **Create** when ready. + +### Create the database + +WARNING: +Avoid using burstable instances (t class instances) for the database as this could lead to performance issues due to CPU credits running out during sustained periods of high load. + +Now, it's time to create the database: + +1. Go to the RDS dashboard, select **Databases** from the left menu, and select **Create database**. +1. Select **Standard Create** for the database creation method. +1. Select **PostgreSQL** as the database engine and select the minimum PostgreSQL version as defined for your GitLab version in our [database requirements](../../install/requirements.md#postgresql-requirements). +1. Because this is a production server, let's choose **Production** from the **Templates** section. +1. Under **Settings**, use: + - `gitlab-db-ha` for the DB instance identifier. + - `gitlab` for a master username. + - A very secure password for the master password. + + Make a note of these as we need them later. + +1. For the DB instance size, select **Standard classes** and select an instance size that meets your requirements from the dropdown list. We use a `db.m4.large` instance. +1. Under **Storage**, configure the following: + 1. Select **Provisioned IOPS (SSD)** from the storage type dropdown list. Provisioned IOPS (SSD) storage is best suited for this use (though you can choose General Purpose (SSD) to reduce the costs). Read more about it at [Storage for Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Storage.html). + 1. Allocate storage and set provisioned IOPS. We use the minimum values, `100` and `1000`, respectively. + 1. Enable storage autoscaling (optional) and set a maximum storage threshold. +1. Under **Availability & durability**, select **Create a standby instance** to have a standby RDS instance provisioned in a different [Availability Zone](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZ.html). +1. Under **Connectivity**, configure the following: + 1. Select the VPC we created earlier (`gitlab-vpc`) from the **Virtual Private Cloud (VPC)** dropdown list. + 1. Expand the **Additional connectivity configuration** section and select the subnet group (`gitlab-rds-group`) we created earlier. + 1. Set public accessibility to **No**. + 1. Under **VPC security group**, select **Choose existing** and select the `gitlab-rds-sec-group` we create above from the dropdown list. + 1. Leave the database port as the default `5432`. +1. For **Database authentication**, select **Password authentication**. +1. Expand the **Additional configuration** section and complete the following: + 1. The initial database name. We use `gitlabhq_production`. + 1. Configure your preferred backup settings. + 1. The only other change we make here is to disable auto minor version updates under **Maintenance**. + 1. Leave all the other settings as is or tweak according to your needs. + 1. If you're happy, select **Create database**. + +Now that the database is created, let's move on to setting up Redis with ElastiCache. + +## Redis with ElastiCache + +ElastiCache is an in-memory hosted caching solution. Redis maintains its own +persistence and is used to store session data, temporary cache information, and background job queues for the GitLab application. + +### Create a Redis Security Group + +1. Go to the EC2 dashboard. +1. Select **Security Groups** from the left menu. +1. Select **Create security group** and fill in the details. Give it a name (we use `gitlab-redis-sec-group`), + add a description, and choose the VPC we created previously +1. In the **Inbound rules** section, select **Add rule** and add a **Custom TCP** rule, set port `6379`, and set the "Custom" source as the `gitlab-loadbalancer-sec-group` we created earlier. +1. When done, select **Create security group**. + +### Redis Subnet Group + +1. Go to the ElastiCache dashboard from your AWS console. +1. Go to **Subnet Groups** in the left menu, and create a new subnet group (we name ours `gitlab-redis-group`). + Make sure to select our VPC and its [private subnets](#subnets). +1. Select **Create** when ready. + +  + +### Create the Redis Cluster + +1. Go back to the ElastiCache dashboard. +1. Select **Redis** on the left menu and select **Create** to create a new + Redis cluster. Do not enable **Cluster Mode** as it is [not supported](../../administration/redis/replication_and_failover_external.md#requirements). Even without cluster mode on, you still get the + chance to deploy Redis in multiple availability zones. +1. In the settings section: + 1. Give the cluster a name (`gitlab-redis`) and a description. + 1. For the version, select the latest. + 1. Leave the port as `6379` because this is what we used in our Redis security group above. + 1. Select the node type (at least `cache.t3.medium`, but adjust to your needs) and the number of replicas. +1. In the advanced settings section: + 1. Select the multi-AZ auto-failover option. + 1. Select the subnet group we created previously. + 1. Manually select the preferred availability zones, and under "Replica 2" + choose a different zone than the other two. + +  + +1. In the security settings, edit the security groups and choose the + `gitlab-redis-sec-group` we had previously created. +1. Leave the rest of the settings to their default values or edit to your liking. +1. When done, select **Create**. + +## Setting up Bastion Hosts + +Because our GitLab instances are in private subnets, we need a way to connect +to these instances with SSH for actions that include making configuration changes +and performing upgrades. One way of doing this is by using a [bastion host](https://en.wikipedia.org/wiki/Bastion_host), +sometimes also referred to as a jump box. NOTE: -When deploying a GitLab instance using the official AMI, the root password to the instance is the EC2 **Instance** ID (not the AMI ID). This way of setting the root account password is specific to official GitLab published AMIs ONLY. +If you do not want to maintain bastion hosts, you can set up [AWS Systems Manager Session Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html) for access to instances. This is beyond the scope of this document. + +### Create Bastion Host A + +1. Go to the EC2 Dashboard and select **Launch instance**. +1. Select the **Ubuntu Server 18.04 LTS (HVM)** AMI. +1. Choose an instance type. We use a `t2.micro` as we only use the bastion host to SSH into our other instances. +1. Select **Configure Instance Details**. + 1. Under **Network**, select the `gitlab-vpc` from the dropdown list. + 1. Under **Subnet**, select the public subnet we created earlier (`gitlab-public-10.0.0.0`). + 1. Double check that under **Auto-assign Public IP** you have **Use subnet setting (Enable)** selected. + 1. Leave everything else as default and select **Add Storage**. +1. For storage, we leave everything as default and only add an 8GB root volume. We do not store anything on this instance. +1. Select **Add Tags** and on the next screen select **Add Tag**. + 1. We only set `Key: Name` and `Value: Bastion Host A`. +1. Select **Configure Security Group**. + 1. Select **Create a new security group**, enter a **Security group name** (we use `bastion-sec-group`), and add a description. + 1. We enable SSH access from anywhere (`0.0.0.0/0`). If you want stricter security, specify a single IP address or an IP address range in CIDR notation. + 1. Select **Review and Launch** +1. Review all your settings and, if you're happy, select **Launch**. +1. Acknowledge that you have access to an existing key pair or create a new one. Select **Launch Instance**. + +Confirm that you can SSH into the instance: + +1. On the EC2 Dashboard, select **Instances** in the left menu. +1. Select **Bastion Host A** from your list of instances. +1. Select **Connect** and follow the connection instructions. +1. If you are able to connect successfully, let's move on to setting up our second bastion host for redundancy. + +### Create Bastion Host B + +1. Create an EC2 instance following the same steps as above with the following changes: + 1. For the **Subnet**, select the second public subnet we created earlier (`gitlab-public-10.0.2.0`). + 1. Under the **Add Tags** section, we set `Key: Name` and `Value: Bastion Host B` so that we can easily identify our two instances. + 1. For the security group, select the existing `bastion-sec-group` we created above. + +### Use SSH Agent Forwarding + +EC2 instances running Linux use private key files for SSH authentication. You connect to your bastion host using an SSH client and the private key file stored on your client. Because the private key file is not present on the bastion host, you are not able to connect to your instances in private subnets. + +Storing private key files on your bastion host is a bad idea. To get around this, use SSH agent forwarding on your client. See [Securely Connect to Linux Instances Running in a Private Amazon VPC](https://aws.amazon.com/blogs/security/securely-connect-to-linux-instances-running-in-a-private-amazon-vpc/) for a step-by-step guide on how to use SSH agent forwarding. + +## Install GitLab and create custom AMI + +We need a preconfigured, custom GitLab AMI to use in our launch configuration later. As a starting point, we use the official GitLab AMI to create a GitLab instance. Then, we add our custom configuration for PostgreSQL, Redis, and Gitaly. If you prefer, instead of using the official GitLab AMI, you can also spin up an EC2 instance of your choosing and [manually install GitLab](https://about.gitlab.com/install/). + +### Install GitLab + +From the EC2 dashboard: + +1. Use the section below titled "[Find official GitLab-created AMI IDs on AWS](#find-official-gitlab-created-ami-ids-on-aws)" to find the correct AMI to launch. +1. After selecting **Launch** on the desired AMI, select an instance type based on your workload. Consult the [hardware requirements](../../install/requirements.md#hardware-requirements) to choose one that fits your needs (at least `c5.xlarge`, which is sufficient to accommodate 100 users). +1. Select **Configure Instance Details**: + 1. In the **Network** dropdown list, select `gitlab-vpc`, the VPC we created earlier. + 1. In the **Subnet** dropdown list, select `gitlab-private-10.0.1.0` from the list of subnets we created earlier. + 1. Double check that **Auto-assign Public IP** is set to `Use subnet setting (Disable)`. + 1. Select **Add Storage**. + 1. The root volume is 8GiB by default and should be enough given that we do not store any data there. +1. Select **Add Tags** and add any tags you may need. In our case, we only set `Key: Name` and `Value: GitLab`. +1. Select **Configure Security Group**. Check **Select an existing security group** and select the `gitlab-loadbalancer-sec-group` we created earlier. +1. Select **Review and launch** followed by **Launch** if you're happy with your settings. +1. Finally, acknowledge that you have access to the selected private key file or create a new one. Select **Launch Instances**. + +### Add custom configuration + +Connect to your GitLab instance via **Bastion Host A** using [SSH Agent Forwarding](#use-ssh-agent-forwarding). Once connected, add the following custom configuration: + +#### Disable Let's Encrypt + +Because we're adding our SSL certificate at the load balancer, we do not need the GitLab built-in support for Let's Encrypt. Let's Encrypt [is enabled by default](https://docs.gitlab.com/omnibus/settings/ssl/index.html#enable-the-lets-encrypt-integration) when using an `https` domain in GitLab 10.7 and later, so we must explicitly disable it: + +1. Open `/etc/gitlab/gitlab.rb` and disable it: + + ```ruby + letsencrypt['enable'] = false + ``` + +1. Save the file and reconfigure for the changes to take effect: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +#### Install the required extensions for PostgreSQL + +From your GitLab instance, connect to the RDS instance to verify access and to install the required `pg_trgm` and `btree_gist` extensions. + +To find the host or endpoint, go to **Amazon RDS > Databases** and select the database you created earlier. Look for the endpoint under the **Connectivity & security** tab. + +Do not to include the colon and port number: + +```shell +sudo /opt/gitlab/embedded/bin/psql -U gitlab -h <rds-endpoint> -d gitlabhq_production +``` + +At the `psql` prompt create the extension and then quit the session: + +```shell +psql (10.9) +Type "help" for help. + +gitlab=# CREATE EXTENSION pg_trgm; +gitlab=# CREATE EXTENSION btree_gist; +gitlab=# \q +``` + +#### Configure GitLab to connect to PostgreSQL and Redis + +1. Edit `/etc/gitlab/gitlab.rb`, find the `external_url 'http://<domain>'` option + and change it to the `https` domain you are using. + +1. Look for the GitLab database settings and uncomment as necessary. In + our current case we specify the database adapter, encoding, host, name, + username, and password: + + ```ruby + # Disable the built-in Postgres + postgresql['enable'] = false + + # Fill in the connection details + gitlab_rails['db_adapter'] = "postgresql" + gitlab_rails['db_encoding'] = "unicode" + gitlab_rails['db_database'] = "gitlabhq_production" + gitlab_rails['db_username'] = "gitlab" + gitlab_rails['db_password'] = "mypassword" + gitlab_rails['db_host'] = "<rds-endpoint>" + ``` + +1. Next, we must configure the Redis section by adding the host and + uncommenting the port: -Instances running on Community Edition (CE) require a migration to Enterprise Edition (EE) to subscribe to the GitLab Premium or Ultimate plan. If you want to pursue a subscription, using the Free-forever plan of Enterprise Edition is the least disruptive method. + ```ruby + # Disable the built-in Redis + redis['enable'] = false + + # Fill in the connection details + gitlab_rails['redis_host'] = "<redis-endpoint>" + gitlab_rails['redis_port'] = 6379 + ``` + +1. Finally, reconfigure GitLab for the changes to take effect: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. You can also run a check and a service status to make sure + everything has been setup correctly: + + ```shell + sudo gitlab-rake gitlab:check + sudo gitlab-ctl status + ``` + +#### Set up Gitaly + +WARNING: +In this architecture, having a single Gitaly server creates a single point of failure. Use +[Gitaly Cluster](../../administration/gitaly/praefect.md) to remove this limitation. + +Gitaly is a service that provides high-level RPC access to Git repositories. +It should be enabled and configured on a separate EC2 instance in one of the +[private subnets](#subnets) we configured previously. + +Let's create an EC2 instance where we install Gitaly: + +1. From the EC2 dashboard, select **Launch instance**. +1. Choose an AMI. In this example, we select the **Ubuntu Server 18.04 LTS (HVM), SSD Volume Type**. +1. Choose an instance type. We pick a `c5.xlarge`. +1. Select **Configure Instance Details**. + 1. In the **Network** dropdown list, select `gitlab-vpc`, the VPC we created earlier. + 1. In the **Subnet** dropdown list, select `gitlab-private-10.0.1.0` from the list of subnets we created earlier. + 1. Double check that **Auto-assign Public IP** is set to `Use subnet setting (Disable)`. + 1. Select **Add Storage**. +1. Increase the Root volume size to `20 GiB` and change the **Volume Type** to `Provisioned IOPS SSD (io1)`. (This is an arbitrary size. Create a volume big enough for your repository storage requirements.) + 1. For **IOPS** set `1000` (20 GiB x 50 IOPS). You can provision up to 50 IOPS per GiB. If you select a larger volume, increase the IOPS accordingly. Workloads where many small files are written in a serialized manner, like `git`, requires performant storage, hence the choice of `Provisioned IOPS SSD (io1)`. +1. Select **Add Tags** and add your tags. In our case, we only set `Key: Name` and `Value: Gitaly`. +1. Select **Configure Security Group** and let's **Create a new security group**. + 1. Give your security group a name and description. We use `gitlab-gitaly-sec-group` for both. + 1. Create a **Custom TCP** rule and add port `8075` to the **Port Range**. For the **Source**, select the `gitlab-loadbalancer-sec-group`. + 1. Also add an inbound rule for SSH from the `bastion-sec-group` so that we can connect using [SSH Agent Forwarding](#use-ssh-agent-forwarding) from the Bastion hosts. +1. Select **Review and launch** followed by **Launch** if you're happy with your settings. +1. Finally, acknowledge that you have access to the selected private key file or create a new one. Select **Launch Instances**. NOTE: -Because any given GitLab upgrade might involve data disk updates or database schema upgrades, swapping out the AMI is not sufficient for taking upgrades. +Instead of storing configuration _and_ repository data on the root volume, you can also choose to add an additional EBS volume for repository storage. Follow the same guidance as above. See the [Amazon EBS pricing](https://aws.amazon.com/ebs/pricing/). We do not recommend using EFS as it may negatively impact the performance of GitLab. You can review the [relevant documentation](../../administration/nfs.md#avoid-using-cloud-based-file-systems) for more details. + +Now that we have our EC2 instance ready, follow the [documentation to install GitLab and set up Gitaly on its own server](../../administration/gitaly/configure_gitaly.md#run-gitaly-on-its-own-server). Perform the client setup steps from that document on the [GitLab instance we created](#install-gitlab) above. + +#### Add Support for Proxied SSL + +As we are terminating SSL at our [load balancer](#load-balancer), follow the steps at [Supporting proxied SSL](https://docs.gitlab.com/omnibus/settings/ssl/index.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination) to configure this in `/etc/gitlab/gitlab.rb`. + +Remember to run `sudo gitlab-ctl reconfigure` after saving the changes to the `gitlab.rb` file. + +#### Fast lookup of authorized SSH keys + +The public SSH keys for users allowed to access GitLab are stored in `/var/opt/gitlab/.ssh/authorized_keys`. Typically we'd use shared storage so that all the instances are able to access this file when a user performs a Git action over SSH. Because we do not have shared storage in our setup, we update our configuration to authorize SSH users via indexed lookup in the GitLab database. + +Follow the instructions at [Set up fast SSH key lookup](../../administration/operations/fast_ssh_key_lookup.md#set-up-fast-lookup) to switch from using the `authorized_keys` file to the database. -1. Log in to the AWS Web Console, so that selecting the links in the following step take you directly to the AMI list. -1. Pick the edition you want: +If you do not configure fast lookup, Git actions over SSH results in the following error: - - [GitLab Enterprise Edition](https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Images:visibility=public-images;ownerAlias=782774275127;search=GitLab%20EE;sort=desc:name): If you want to unlock the enterprise features, a license is needed. - - [GitLab Community Edition](https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Images:visibility=public-images;ownerAlias=782774275127;search=GitLab%20CE;sort=desc:name): The open source version of GitLab. - - [GitLab Premium or Ultimate Marketplace (pre-licensed)](https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Images:visibility=public-images;source=Marketplace;search=GitLab%20EE;sort=desc:name): 5 user license built into per-minute billing. +```shell +Permission denied (publickey). +fatal: Could not read from remote repository. -1. AMI IDs are unique per region. After you've loaded any of these editions, in the upper-right corner, select the desired target region of the console to see the appropriate AMIs. -1. After the console is loaded, you can add additional search criteria to narrow further. For instance, type `13.` to find only 13.x versions. -1. To launch an EC2 Machine with one of the listed AMIs, check the box at the start of the relevant row, and select **Launch** near the top of left of the page. +Please make sure you have the correct access rights +and the repository exists. +``` + +#### Configure host keys + +Ordinarily we would manually copy the contents (primary and public keys) of `/etc/ssh/` on the primary application server to `/etc/ssh` on all secondary servers. This prevents false man-in-the-middle-attack alerts when accessing servers in your cluster behind a load balancer. + +We automate this by creating static host keys as part of our custom AMI. As these host keys are also rotated every time an EC2 instance boots up, "hard coding" them into our custom AMI serves as a workaround. + +On your GitLab instance run the following: + +```shell +sudo mkdir /etc/ssh_static +sudo cp -R /etc/ssh/* /etc/ssh_static +``` + +In `/etc/ssh/sshd_config` update the following: + +```shell +# HostKeys for protocol version 2 +HostKey /etc/ssh_static/ssh_host_rsa_key +HostKey /etc/ssh_static/ssh_host_dsa_key +HostKey /etc/ssh_static/ssh_host_ecdsa_key +HostKey /etc/ssh_static/ssh_host_ed25519_key +``` + +#### Amazon S3 object storage + +Because we're not using NFS for shared storage, we use [Amazon S3](https://aws.amazon.com/s3/) buckets to store backups, artifacts, LFS objects, uploads, merge request diffs, container registry images, and more. Our documentation includes [instructions on how to configure object storage](../../administration/object_storage.md) for each of these data types, and other information about using object storage with GitLab. NOTE: -If you are trying to restore from an older version of GitLab while moving to AWS, find the -[Enterprise and Community Editions before GitLab 11.10.3](https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Images:visibility=public-images;ownerAlias=855262394183;sort=desc:name). +Because we are using the [AWS IAM profile](#create-an-iam-role) we created earlier, be sure to omit the AWS access key and secret access key/value pairs when configuring object storage. Instead, use `'use_iam_profile' => true` in your configuration as shown in the object storage documentation linked above. + +Remember to run `sudo gitlab-ctl reconfigure` after saving the changes to the `gitlab.rb` file. + +--- + +That concludes the configuration changes for our GitLab instance. Next, we create a custom AMI based on this instance to use for our launch configuration and auto scaling group. + +### Log in for the first time + +Using the domain name you used when setting up [DNS for the load balancer](#configure-dns-for-load-balancer), you should now be able to visit GitLab in your browser. + +Depending on how you installed GitLab and if you did not change the password by any other means, the default password is either: -## Additional details on implementation patterns +- Your instance ID if you used the official GitLab AMI. +- A randomly generated password stored for 24 hours in `/etc/gitlab/initial_root_password`. -GitLab implementation patterns build upon [GitLab Reference Architectures](../../administration/reference_architectures/index.md) in the following ways. +To change the default password, log in as the `root` user with the default password and [change it in the user profile](../../user/profile/user_passwords.md#change-your-password). -### Cloud platform well architected compliance +When our [auto scaling group](#create-an-auto-scaling-group) spins up new instances, we are able to sign in with username `root` and the newly created password. -Testing-backed architectural qualification is a fundamental concept behind implementation patterns: +### Create custom AMI -- Implementation patterns maintain GitLab Reference Architecture compliance and provide [GitLab Performance Tool](https://gitlab.com/gitlab-org/quality/performance) (GPT) reports to demonstrate adherence to them. -- Implementation patterns may be qualified by and/or contributed to by the technology vendor. For instance, an implementation pattern for AWS may be officially reviewed by AWS. -- Implementation patterns may specify and test Cloud Platform PaaS services for suitability for GitLab. This testing can be coordinated and help qualify these technologies for Reference Architectures. For instance, qualifying compatibility with and availability of runtime versions of top level PaaS such as those for PostgreSQL and Redis. -- Implementation patterns can provided qualified testing for platform limitations, for example, ensuring Gitaly Cluster can work correctly on specific Cloud Platform availability zone latency and throughput characteristics or qualifying what levels of available platform partner local disk performance is workable for Gitaly server to operate with integrity. +On the EC2 dashboard: -### Platform partner specificity +1. Select the `GitLab` instance we [created earlier](#install-gitlab). +1. Select **Actions**, scroll down to **Image** and select **Create Image**. +1. Give your image a name and description (we use `GitLab-Source` for both). +1. Leave everything else as default and select **Create Image** -Implementation patterns enable platform-specific terminology, best practice architecture, and platform-specific build manifests: +Now we have a custom AMI that we use to create our launch configuration the next step. -- Implementation patterns are more vendor specific. For instance, advising specific compute instances / VMs / nodes instead of vCPUs or other generalized measures. -- Implementation patterns are oriented to implementing good architecture for the vendor in view. -- Implementation patterns are written to an audience who is familiar with building on the infrastructure that the implementation pattern targets. For example, if the implementation pattern is for GCP, the specific terminology of GCP is used - including using the specific names for PaaS services. -- Implementation patterns can test and qualify if the versions of PaaS available are compatible with GitLab (for example, PostgreSQL, Redis, etc.). +## Deploy GitLab inside an auto scaling group + +### Create a launch configuration + +From the EC2 dashboard: + +1. Select **Launch Configurations** from the left menu and select **Create launch configuration**. +1. Select **My AMIs** from the left menu and select the `GitLab` custom AMI we created above. +1. Select an instance type best suited for your needs (at least a `c5.xlarge`) and select **Configure details**. +1. Enter a name for your launch configuration (we use `gitlab-ha-launch-config`). +1. **Do not** check **Request Spot Instance**. +1. From the **IAM Role** dropdown list, pick the `GitLabAdmin` instance role we [created earlier](#create-an-iam-ec2-instance-role-and-profile). +1. Leave the rest as defaults and select **Add Storage**. +1. The root volume is 8GiB by default and should be enough given that we do not store any data there. Select **Configure Security Group**. +1. Check **Select and existing security group** and select the `gitlab-loadbalancer-sec-group` we created earlier. +1. Select **Review**, review your changes, and select **Create launch configuration**. +1. Acknowledge that you have access to the private key or create a new one. Select **Create launch configuration**. + +### Create an auto scaling group + +1. After the launch configuration is created, select **Create an Auto Scaling group using this launch configuration** to start creating the auto scaling group. +1. Enter a **Group name** (we use `gitlab-auto-scaling-group`). +1. For **Group size**, enter the number of instances you want to start with (we enter `2`). +1. Select the `gitlab-vpc` from the **Network** dropdown list. +1. Add both the private [subnets we created earlier](#subnets). +1. Expand the **Advanced Details** section and check the **Receive traffic from one or more load balancers** option. +1. From the **Classic Load Balancers** dropdown list, select the load balancer we created earlier. +1. For **Health Check Type**, select **ELB**. +1. We leave our **Health Check Grace Period** as the default `300` seconds. Select **Configure scaling policies**. +1. Check **Use scaling policies to adjust the capacity of this group**. +1. For this group we scale between 2 and 4 instances where one instance is added if CPU +utilization is greater than 60% and one instance is removed if it falls +to less than 45%. + + + +1. Finally, configure notifications and tags as you see fit, review your changes, and create the +auto scaling group. + +As the auto scaling group is created, you see your new instances spinning up in your EC2 dashboard. You also see the new instances added to your load balancer. After the instances pass the heath check, they are ready to start receiving traffic from the load balancer. + +Because our instances are created by the auto scaling group, go back to your instances and terminate the [instance we created manually above](#install-gitlab). We only needed this instance to create our custom AMI. + +## Health check and monitoring with Prometheus + +Apart from Amazon's Cloudwatch which you can enable on various services, +GitLab provides its own integrated monitoring solution based on Prometheus. +For more information about how to set it up, see +[GitLab Prometheus](../../administration/monitoring/prometheus/index.md). + +GitLab also has various [health check endpoints](../../administration/monitoring/health_check.md) +that you can ping and get reports. + +## GitLab Runner + +If you want to take advantage of [GitLab CI/CD](../../ci/index.md), you have to +set up at least one [runner](https://docs.gitlab.com/runner/). + +Read more on configuring an +[autoscaling GitLab Runner on AWS](https://docs.gitlab.com/runner/configuration/runner_autoscale_aws/). + +## Backup and restore + +GitLab provides [a tool to back up](../../administration/backup_restore/index.md) +and restore its Git data, database, attachments, LFS objects, and so on. + +Some important things to know: + +- The backup/restore tool **does not** store some configuration files, like secrets; you + must [configure this yourself](../../administration/backup_restore/backup_gitlab.md#storing-configuration-files). +- By default, the backup files are stored locally, but you can + [backup GitLab using S3](../../administration/backup_restore/backup_gitlab.md#using-amazon-s3). +- You can [exclude specific directories form the backup](../../administration/backup_restore/backup_gitlab.md#excluding-specific-directories-from-the-backup). + +### Backing up GitLab + +To back up GitLab: + +1. SSH into your instance. +1. Take a backup: + + ```shell + sudo gitlab-backup create + ``` + +NOTE: +For GitLab 12.1 and earlier, use `gitlab-rake gitlab:backup:create`. + +### Restoring GitLab from a backup + +To restore GitLab, first review the [restore documentation](../../administration/backup_restore/index.md#restore-gitlab), +and primarily the restore prerequisites. Then, follow the steps under the +[Linux package installations section](../../administration/backup_restore/restore_gitlab.md#restore-for-linux-package-installations). + +## Updating GitLab + +GitLab releases a new version every month on the [release date](https://about.gitlab.com/releases/). Whenever a new version is +released, you can update your GitLab instance: + +1. SSH into your instance +1. Take a backup: + + ```shell + sudo gitlab-backup create + ``` + +NOTE: +For GitLab 12.1 and earlier, use `gitlab-rake gitlab:backup:create`. -### Platform as a Service (PaaS) specification and usage +1. Update the repositories and install GitLab: -Platform as a Service options are a huge portion of the value provided by Cloud Platforms as they simplify operational complexity and reduce the SRE and security skilling required to operate advanced, highly available technology services. Implementation patterns can be pre-qualified against the partner PaaS options. + ```shell + sudo apt update + sudo apt install gitlab-ee + ``` -- Implementation patterns help implementers understand what PaaS options are known to work and how to choose between PaaS solutions when a single platform has more than one PaaS option for the same GitLab role. -- For instance, where reference architectures do not have a specific recommendation on what technology is leveraged for GitLab outbound email services or what the sizing should be - a Reference Implementation may advise using a cloud providers Email as a Service (PaaS) and possibly even with specific settings. +After a few minutes, the new version should be up and running. -### Cost optimizing engineering +## Find official GitLab-created AMI IDs on AWS -Cost engineering is a fundamental aspect of Cloud Architecture and frequently the savings capabilities available on a platform exert strong influence on how to build out scaled computing. +Read more on how to use [GitLab releases as AMIs](../../solutions/cloud/aws/gitlab_single_box_on_aws.md#official-gitlab-releases-as-amis). -- Implementation patterns may define GPT tested autoscaling for various aspects of GitLab infrastructure, including minimum idling configurations and scaling speeds. -- Implementation patterns may provide GPT testing for advised configurations that go beyond the scope of reference architectures, for instance GPT tested elastic scaling configurations for Cloud Native Hybrid that enable lower resourcing during periods of lower usage (for example on the weekend). -- Implementation patterns may engineer specifically for the savings models available on a platform provider. An AWS example would be maximizing the occurrence of a specific instance type for taking advantage of reserved instances. -- Implementation patterns may leverage ephemeral compute where appropriate and with appropriate customer guidelines. For instance, a Kubernetes node group dedicated to runners on ephemeral compute (with appropriate GitLab Runner tagging to indicate the compute type). -- Implementation patterns may include vendor specific cost calculators. +## Conclusion -### Actionability and automatability orientation +In this guide, we went mostly through scaling and some redundancy options, +your mileage may vary. -Implementation patterns are one step closer to specifics that can be used as a source for build instructions and automation code: +Keep in mind that all solutions come with a trade-off between +cost/complexity and uptime. The more uptime you want, the more complex the solution. +And the more complex the solution, the more work is involved in setting up and +maintaining it. -- Implementation patterns enable builders to generate a list of vendor specific resources required to implement GitLab for a given Reference Architecture. -- Implementation patterns enable builders to use manual instructions or to create automation to build out the reference implementation. +Have a read through these other resources and feel free to +[open an issue](https://gitlab.com/gitlab-org/gitlab/-/issues/new) +to request additional material: -## Supplementary implementation patterns +- [Scaling GitLab](../../administration/reference_architectures/index.md): + GitLab supports several different types of clustering. +- [Geo replication](../../administration/geo/index.md): + Geo is the solution for widely distributed development teams. +- [Linux package](https://docs.gitlab.com/omnibus/) - Everything you must know + about administering your GitLab instance. +- [Add a license](../../administration/license.md): + Activate all GitLab Enterprise Edition functionality with a license. +- [Pricing](https://about.gitlab.com/pricing/): Pricing for the different tiers. -Implementation patterns may also provide specialized implementations beyond the scope of reference architecture compliance, especially where the cost of enablement can be more appropriately managed. +## Troubleshooting -For example: +### Instances are failing health checks -- Small, self-contained GitLab instances for per-person administration training, perhaps on Kubernetes so that a deployment cluster is self-contained as well. -- GitLab Runner implementation patterns, including using platform-specific PaaS. +If your instances are failing the load balancer's health checks, verify that they are returning a status `200` from the health check endpoint we configured earlier. Any other status, including redirects like status `302`, causes the health check to fail. -## Intended audiences and contributors +You may have to set a password on the `root` user to prevent automatic redirects on the sign-in endpoint before health checks pass. -The primary audiences for and contributors to this information is the GitLab **Implementation Eco System** which consists of at least: +### "The change you requested was rejected (422)" -GitLab Implementation Community: +If you see this page when trying to set a password via the web interface, make sure `external_url` in `gitlab.rb` matches the domain you are making a request from, and run `sudo gitlab-ctl reconfigure` after making any changes to it. -- Customers -- GitLab Channel Partners (Integrators) -- Platform Partners +### Some job logs are not uploaded to object storage -GitLab Internal Implementation Teams: +When the GitLab deployment is scaled up to more than one node, some job logs may not be uploaded to [object storage](../../administration/object_storage.md) properly. [Incremental logging is required](../../administration/object_storage.md#alternatives-to-file-system-storage) for CI to use object storage. -- Quality / Distribution / Self-Managed -- Alliances -- Training -- Support -- Professional Services -- Public Sector +Enable [incremental logging](../../administration/job_logs.md#enable-or-disable-incremental-logging) if it has not already been enabled. diff --git a/doc/install/aws/manual_install_aws.md b/doc/install/aws/manual_install_aws.md index cd42e689c41..0019c8c3472 100644 --- a/doc/install/aws/manual_install_aws.md +++ b/doc/install/aws/manual_install_aws.md @@ -1,856 +1,11 @@ --- -stage: Systems -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: 'index.md' +remove_date: '2024-03-31' --- -{::options parse_block_html="true" /} +This document was moved to [AWS](index.md). -# Installing a GitLab POC on Amazon Web Services (AWS) **(FREE SELF)** - -This page offers a walkthrough of a common configuration for GitLab on AWS using the official Linux package. You should customize it to accommodate your needs. - -NOTE: -For organizations with 1,000 users or less, the recommended AWS installation method is to launch an EC2 single box [Linux package installation](https://about.gitlab.com/install/) and implement a snapshot strategy for backing up the data. See the [1,000 user reference architecture](../../administration/reference_architectures/1k_users.md) for more information. - -## Getting started for production-grade GitLab - -NOTE: -This document is an installation guide for a proof of concept instance. It is not a reference architecture and it does not result in a highly available configuration. - -Following this guide exactly results in a proof of concept instance that roughly equates to a **scaled down** version of a **two availability zone implementation** of the **Non-HA** [2000 User Reference Architecture](../../administration/reference_architectures/2k_users.md). The 2K reference architecture is not HA because it is primarily intended to provide some scaling while keeping costs and complexity low. The [3000 User Reference Architecture](../../administration/reference_architectures/3k_users.md) is the smallest size that is GitLab HA. It has additional service roles to achieve HA, most notably it uses Gitaly Cluster to achieve HA for Git repository storage and specifies triple redundancy. - -GitLab maintains and tests two main types of Reference Architectures. The **Linux package architectures** are implemented on instance compute while **Cloud Native Hybrid architectures** maximize the use of a Kubernetes cluster. Cloud Native Hybrid reference architecture specifications are addendum sections to the Reference Architecture size pages that start by describing the Linux package architecture. For example, the 3000 User Cloud Native Reference Architecture is in the subsection titled [Cloud Native Hybrid reference architecture with Helm Charts (alternative)](../../administration/reference_architectures/3k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) in the 3000 User Reference Architecture page. - -### Getting started for production-grade Linux package installations - -The Infrastructure as Code tooling [GitLab Environment Tool (GET)](https://gitlab.com/gitlab-org/gitlab-environment-toolkit/-/tree/main) is the best place to start for building using the Linux package on AWS and most especially if you are targeting an HA setup. While it does not automate everything, it does complete complex setups like Gitaly Cluster for you. GET is open source so anyone can build on top of it and contribute improvements to it. - -### Getting started for production-grade Cloud Native Hybrid GitLab - -For the Cloud Native Hybrid architectures there are two Infrastructure as Code options which are compared in GitLab Cloud Native Hybrid on AWS EKS implementation pattern in the section [Available Infrastructure as Code for GitLab Cloud Native Hybrid](gitlab_hybrid_on_aws.md#available-infrastructure-as-code-for-gitlab-cloud-native-hybrid). It compares the [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/gitlab-environment-toolkit/-/tree/main) to the AWS Quick Start for GitLab Cloud Native Hybrid on EKS which was co-developed by GitLab and AWS. GET and the AWS Quick Start are both open source so anyone can build on top of them and contribute improvements to them. - -## Introduction - -For the most part, we make use of the Linux package in our setup, but we also leverage native AWS services. Instead of using the Linux package-bundled PostgreSQL and Redis, we use Amazon RDS and ElastiCache. - -In this guide, we go through a multi-node setup where we start by -configuring our Virtual Private Cloud and subnets to later integrate -services such as RDS for our database server and ElastiCache as a Redis -cluster to finally manage them in an auto scaling group with custom -scaling policies. - -## Requirements - -In addition to having a basic familiarity with [AWS](https://docs.aws.amazon.com/) and [Amazon EC2](https://docs.aws.amazon.com/ec2/), you need: - -- [An AWS account](https://console.aws.amazon.com/console/home) -- [To create or upload an SSH key](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html) - to connect to the instance via SSH -- A domain name for the GitLab instance -- An SSL/TLS certificate to secure your domain. If you do not already own one, you can provision a free public SSL/TLS certificate through [AWS Certificate Manager](https://aws.amazon.com/certificate-manager/)(ACM) for use with the [Elastic Load Balancer](#load-balancer) we create. - -NOTE: -It can take a few hours to validate a certificate provisioned through ACM. To avoid delays later, request your certificate as soon as possible. - -## Architecture - -Below is a diagram of the recommended architecture. - - - -## AWS costs - -GitLab uses the following AWS services, with links to pricing information: - -- **EC2**: GitLab is deployed on shared hardware, for which - [on-demand pricing](https://aws.amazon.com/ec2/pricing/on-demand/) applies. - If you want to run GitLab on a dedicated or reserved instance, see the - [EC2 pricing page](https://aws.amazon.com/ec2/pricing/) for information about - its cost. -- **S3**: GitLab uses S3 ([pricing page](https://aws.amazon.com/s3/pricing/)) to - store backups, artifacts, and LFS objects. -- **ELB**: A Classic Load Balancer ([pricing page](https://aws.amazon.com/elasticloadbalancing/pricing/)), - used to route requests to the GitLab instances. -- **RDS**: An Amazon Relational Database Service using PostgreSQL - ([pricing page](https://aws.amazon.com/rds/postgresql/pricing/)). -- **ElastiCache**: An in-memory cache environment ([pricing page](https://aws.amazon.com/elasticache/pricing/)), - used to provide a Redis configuration. - -## Create an IAM EC2 instance role and profile - -As we are using [Amazon S3 object storage](#amazon-s3-object-storage), our EC2 instances must have read, write, and list permissions for our S3 buckets. To avoid embedding AWS keys in our GitLab configuration, we make use of an [IAM Role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) to allow our GitLab instance with this access. We must create an IAM policy to attach to our IAM role: - -### Create an IAM Policy - -1. Go to the IAM dashboard and select **Policies** in the left menu. -1. Select **Create policy**, select the `JSON` tab, and add a policy. We want to [follow security best practices and grant _least privilege_](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege), giving our role only the permissions needed to perform the required actions. - 1. Assuming you prefix the S3 bucket names with `gl-` as shown in the diagram, add the following policy: - - ```json - { "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Action": [ - "s3:PutObject", - "s3:GetObject", - "s3:DeleteObject", - "s3:PutObjectAcl" - ], - "Resource": "arn:aws:s3:::gl-*/*" - }, - { - "Effect": "Allow", - "Action": [ - "s3:ListBucket", - "s3:AbortMultipartUpload", - "s3:ListMultipartUploadParts", - "s3:ListBucketMultipartUploads" - ], - "Resource": "arn:aws:s3:::gl-*" - } - ] - } - ``` - -1. Select **Review policy**, give your policy a name (we use `gl-s3-policy`), and select **Create policy**. - -### Create an IAM Role - -1. Still on the IAM dashboard, select **Roles** in the left menu, and - select **Create role**. -1. Create a new role by selecting **AWS service > EC2**, then select - **Next: Permissions**. -1. In the policy filter, search for the `gl-s3-policy` we created above, select it, and select **Tags**. -1. Add tags if needed and select **Review**. -1. Give the role a name (we use `GitLabS3Access`) and select **Create Role**. - -We use this role when we [create a launch configuration](#create-a-launch-configuration) later on. - -## Configuring the network - -We start by creating a VPC for our GitLab cloud infrastructure, then -we can create subnets to have public and private instances in at least -two [Availability Zones (AZs)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html). Public subnets require a Route Table keep and an associated -Internet Gateway. - -### Creating the Virtual Private Cloud (VPC) - -We now create a VPC, a virtual networking environment that you control: - -1. Sign in to [Amazon Web Services](https://console.aws.amazon.com/vpc/home). -1. Select **Your VPCs** from the left menu and then select **Create VPC**. - At the "Name tag" enter `gitlab-vpc` and at the "IPv4 CIDR block" enter - `10.0.0.0/16`. If you don't require dedicated hardware, you can leave - "Tenancy" as default. Select **Yes, Create** when ready. - -  - -1. Select the VPC, select **Actions**, select **Edit DNS resolution**, and enable DNS resolution. Select **Save** when done. - -### Subnets - -Now, let's create some subnets in different Availability Zones. Make sure -that each subnet is associated to the VPC we just created and -that CIDR blocks don't overlap. This also -allows us to enable multi AZ for redundancy. - -We create private and public subnets to match load balancers and -RDS instances as well: - -1. Select **Subnets** from the left menu. -1. Select **Create subnet**. Give it a descriptive name tag based on the IP, - for example `gitlab-public-10.0.0.0`, select the VPC we created previously, select an availability zone (we use `us-west-2a`), - and at the IPv4 CIDR block let's give it a 24 subnet `10.0.0.0/24`: - -  - -1. Follow the same steps to create all subnets: - - | Name tag | Type | Availability Zone | CIDR block | - | ------------------------- | ------- | ----------------- | ------------- | - | `gitlab-public-10.0.0.0` | public | `us-west-2a` | `10.0.0.0/24` | - | `gitlab-private-10.0.1.0` | private | `us-west-2a` | `10.0.1.0/24` | - | `gitlab-public-10.0.2.0` | public | `us-west-2b` | `10.0.2.0/24` | - | `gitlab-private-10.0.3.0` | private | `us-west-2b` | `10.0.3.0/24` | - -1. Once all the subnets are created, enable **Auto-assign IPv4** for the two public subnets: - 1. Select each public subnet in turn, select **Actions**, and select **Modify auto-assign IP settings**. Enable the option and save. - -### Internet Gateway - -Now, still on the same dashboard, go to Internet Gateways and -create a new one: - -1. Select **Internet Gateways** from the left menu. -1. Select **Create internet gateway**, give it the name `gitlab-gateway` and - select **Create**. -1. Select it from the table, and then under the **Actions** dropdown list choose - "Attach to VPC". - -  - -1. Choose `gitlab-vpc` from the list and hit **Attach**. - -### Create NAT Gateways - -Instances deployed in our private subnets must connect to the internet for updates, but should not be reachable from the public internet. To achieve this, we make use of [NAT Gateways](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html) deployed in each of our public subnets: - -1. Go to the VPC dashboard and select **NAT Gateways** in the left menu bar. -1. Select **Create NAT Gateway** and complete the following: - 1. **Subnet**: Select `gitlab-public-10.0.0.0` from the dropdown list. - 1. **Elastic IP Allocation ID**: Enter an existing Elastic IP or select **Allocate Elastic IP address** to allocate a new IP to your NAT gateway. - 1. Add tags if needed. - 1. Select **Create NAT Gateway**. - -Create a second NAT gateway but this time place it in the second public subnet, `gitlab-public-10.0.2.0`. - -### Route Tables - -#### Public Route Table - -We must create a route table for our public subnets to reach the internet via the internet gateway we created in the previous step. - -On the VPC dashboard: - -1. Select **Route Tables** from the left menu. -1. Select **Create Route Table**. -1. At the "Name tag" enter `gitlab-public` and choose `gitlab-vpc` under "VPC". -1. Select **Create**. - -We now must add our internet gateway as a new target and have -it receive traffic from any destination. - -1. Select **Route Tables** from the left menu and select the `gitlab-public` - route to show the options at the bottom. -1. Select the **Routes** tab, select **Edit routes > Add route** and set `0.0.0.0/0` - as the destination. In the target column, select the `gitlab-gateway` we created previously. - Select **Save routes** when done. - -Next, we must associate the **public** subnets to the route table: - -1. Select the **Subnet Associations** tab and select **Edit subnet associations**. -1. Check only the public subnets and select **Save**. - -#### Private Route Tables - -We also must create two private route tables so that instances in each private subnet can reach the internet via the NAT gateway in the corresponding public subnet in the same availability zone. - -1. Follow the same steps as above to create two private route tables. Name them `gitlab-private-a` and `gitlab-private-b`. -1. Next, add a new route to each of the private route tables where the destination is `0.0.0.0/0` and the target is one of the NAT gateways we created earlier. - 1. Add the NAT gateway we created in `gitlab-public-10.0.0.0` as the target for the new route in the `gitlab-private-a` route table. - 1. Similarly, add the NAT gateway in `gitlab-public-10.0.2.0` as the target for the new route in the `gitlab-private-b`. -1. Lastly, associate each private subnet with a private route table. - 1. Associate `gitlab-private-10.0.1.0` with `gitlab-private-a`. - 1. Associate `gitlab-private-10.0.3.0` with `gitlab-private-b`. - -## Load Balancer - -We create a load balancer to evenly distribute inbound traffic on ports `80` and `443` across our GitLab application servers. Based on the [scaling policies](#create-an-auto-scaling-group) we create later, instances are added to or removed from our load balancer as needed. Additionally, the load balancer performs health checks on our instances. - -On the EC2 dashboard, look for Load Balancer in the left navigation bar: - -1. Select **Create Load Balancer**. - 1. Choose the **Classic Load Balancer**. - 1. Give it a name (we use `gitlab-loadbalancer`) and for the **Create LB Inside** option, select `gitlab-vpc` from the dropdown list. - 1. In the **Listeners** section, set the following listeners: - - HTTP port 80 for both load balancer and instance protocol and ports - - TCP port 22 for both load balancer and instance protocols and ports - - HTTPS port 443 for load balancer protocol and ports, forwarding to HTTP port 80 on the instance (we configure GitLab to listen on port 80 [later in the guide](#add-support-for-proxied-ssl)) - 1. In the **Select Subnets** section, select both public subnets from the list so that the load balancer can route traffic to both availability zones. -1. We add a security group for our load balancer to act as a firewall to control what traffic is allowed through. Select **Assign Security Groups** and select **Create a new security group**, give it a name - (we use `gitlab-loadbalancer-sec-group`) and description, and allow both HTTP and HTTPS traffic - from anywhere (`0.0.0.0/0, ::/0`). Also allow SSH traffic, select a custom source, and add a single trusted IP address or an IP address range in CIDR notation. This allows users to perform Git actions over SSH. -1. Select **Configure Security Settings** and set the following: - 1. Select an SSL/TLS certificate from ACM or upload a certificate to IAM. - 1. Under **Select a Cipher**, pick a predefined security policy from the dropdown list. You can see a breakdown of [Predefined SSL Security Policies for Classic Load Balancers](https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-security-policy-table.html) in the AWS documentation. Check the GitLab codebase for a list of [supported SSL ciphers and protocols](https://gitlab.com/gitlab-org/gitlab/-/blob/9ee7ad433269b37251e0dd5b5e00a0f00d8126b4/lib/support/nginx/gitlab-ssl#L97-99). -1. Select **Configure Health Check** and set up a health check for your EC2 instances. - 1. For **Ping Protocol**, select HTTP. - 1. For **Ping Port**, enter 80. - 1. For **Ping Path** - we recommend that you [use the Readiness check endpoint](../../administration/load_balancer.md#readiness-check). You must add [the VPC IP Address Range (CIDR)](https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-security-groups.html#elb-vpc-nacl) to the [IP allowlist](../../administration/monitoring/ip_allowlist.md) for the [Health Check endpoints](../../administration/monitoring/health_check.md) - 1. Keep the default **Advanced Details** or adjust them according to your needs. -1. Select **Add EC2 Instances** - don't add anything as we create an Auto Scaling Group later to manage instances for us. -1. Select **Add Tags** and add any tags you need. -1. Select **Review and Create**, review all your settings, and select **Create** if you're happy. - -After the Load Balancer is up and running, you can revisit your Security -Groups to refine the access only through the ELB and any other requirements -you might have. - -### Configure DNS for Load Balancer - -On the Route 53 dashboard, select **Hosted zones** in the left navigation bar: - -1. Select an existing hosted zone or, if you do not already have one for your domain, select **Create Hosted Zone**, enter your domain name, and select **Create**. -1. Select **Create Record Set** and provide the following values: - 1. **Name:** Use the domain name (the default value) or enter a subdomain. - 1. **Type:** Select **A - IPv4 address**. - 1. **Alias:** Defaults to **No**. Select **Yes**. - 1. **Alias Target:** Find the **ELB Classic Load Balancers** section and select the classic load balancer we created earlier. - 1. **Routing Policy:** We use **Simple** but you can choose a different policy based on your use case. - 1. **Evaluate Target Health:** We set this to **No** but you can choose to have the load balancer route traffic based on target health. - 1. Select **Create**. -1. If you registered your domain through Route 53, you're done. If you used a different domain registrar, you must update your DNS records with your domain registrar. You must: - 1. Select **Hosted zones** and select the domain you added above. - 1. You see a list of `NS` records. From your domain registrar's administrator panel, add each of these as `NS` records to your domain's DNS records. These steps may vary between domain registrars. If you're stuck, Google **"name of your registrar" add DNS records** and you should find a help article specific to your domain registrar. - -The steps for doing this vary depending on which registrar you use and is beyond the scope of this guide. - -## PostgreSQL with RDS - -For our database server we use Amazon RDS for PostgreSQL which offers Multi AZ -for redundancy (Aurora is **not** supported). First we create a security group and subnet group, then we -create the actual RDS instance. - -### RDS Security Group - -We need a security group for our database that allows inbound traffic from the instances we deploy in our `gitlab-loadbalancer-sec-group` later on: - -1. From the EC2 dashboard, select **Security Groups** from the left menu bar. -1. Select **Create security group**. -1. Give it a name (we use `gitlab-rds-sec-group`), a description, and select the `gitlab-vpc` from the **VPC** dropdown list. -1. In the **Inbound rules** section, select **Add rule** and set the following: - 1. **Type:** search for and select the **PostgreSQL** rule. - 1. **Source type:** set as "Custom". - 1. **Source:** select the `gitlab-loadbalancer-sec-group` we created earlier. -1. When done, select **Create security group**. - -### RDS Subnet Group - -1. Go to the RDS dashboard and select **Subnet Groups** from the left menu. -1. Select **Create DB Subnet Group**. -1. Under **Subnet group details**, enter a name (we use `gitlab-rds-group`), a description, and choose the `gitlab-vpc` from the VPC dropdown list. -1. From the **Availability Zones** dropdown list, select the Availability Zones that include the subnets you've configured. In our case, we add `eu-west-2a` and `eu-west-2b`. -1. From the **Subnets** dropdown list, select the two private subnets (`10.0.1.0/24` and `10.0.3.0/24`) as we defined them in the [subnets section](#subnets). -1. Select **Create** when ready. - -### Create the database - -WARNING: -Avoid using burstable instances (t class instances) for the database as this could lead to performance issues due to CPU credits running out during sustained periods of high load. - -Now, it's time to create the database: - -1. Go to the RDS dashboard, select **Databases** from the left menu, and select **Create database**. -1. Select **Standard Create** for the database creation method. -1. Select **PostgreSQL** as the database engine and select the minimum PostgreSQL version as defined for your GitLab version in our [database requirements](../../install/requirements.md#postgresql-requirements). -1. Because this is a production server, let's choose **Production** from the **Templates** section. -1. Under **Settings**, use: - - `gitlab-db-ha` for the DB instance identifier. - - `gitlab` for a master username. - - A very secure password for the master password. - - Make a note of these as we need them later. - -1. For the DB instance size, select **Standard classes** and select an instance size that meets your requirements from the dropdown list. We use a `db.m4.large` instance. -1. Under **Storage**, configure the following: - 1. Select **Provisioned IOPS (SSD)** from the storage type dropdown list. Provisioned IOPS (SSD) storage is best suited for this use (though you can choose General Purpose (SSD) to reduce the costs). Read more about it at [Storage for Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Storage.html). - 1. Allocate storage and set provisioned IOPS. We use the minimum values, `100` and `1000`, respectively. - 1. Enable storage autoscaling (optional) and set a maximum storage threshold. -1. Under **Availability & durability**, select **Create a standby instance** to have a standby RDS instance provisioned in a different [Availability Zone](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZ.html). -1. Under **Connectivity**, configure the following: - 1. Select the VPC we created earlier (`gitlab-vpc`) from the **Virtual Private Cloud (VPC)** dropdown list. - 1. Expand the **Additional connectivity configuration** section and select the subnet group (`gitlab-rds-group`) we created earlier. - 1. Set public accessibility to **No**. - 1. Under **VPC security group**, select **Choose existing** and select the `gitlab-rds-sec-group` we create above from the dropdown list. - 1. Leave the database port as the default `5432`. -1. For **Database authentication**, select **Password authentication**. -1. Expand the **Additional configuration** section and complete the following: - 1. The initial database name. We use `gitlabhq_production`. - 1. Configure your preferred backup settings. - 1. The only other change we make here is to disable auto minor version updates under **Maintenance**. - 1. Leave all the other settings as is or tweak according to your needs. - 1. If you're happy, select **Create database**. - -Now that the database is created, let's move on to setting up Redis with ElastiCache. - -## Redis with ElastiCache - -ElastiCache is an in-memory hosted caching solution. Redis maintains its own -persistence and is used to store session data, temporary cache information, and background job queues for the GitLab application. - -### Create a Redis Security Group - -1. Go to the EC2 dashboard. -1. Select **Security Groups** from the left menu. -1. Select **Create security group** and fill in the details. Give it a name (we use `gitlab-redis-sec-group`), - add a description, and choose the VPC we created previously -1. In the **Inbound rules** section, select **Add rule** and add a **Custom TCP** rule, set port `6379`, and set the "Custom" source as the `gitlab-loadbalancer-sec-group` we created earlier. -1. When done, select **Create security group**. - -### Redis Subnet Group - -1. Go to the ElastiCache dashboard from your AWS console. -1. Go to **Subnet Groups** in the left menu, and create a new subnet group (we name ours `gitlab-redis-group`). - Make sure to select our VPC and its [private subnets](#subnets). -1. Select **Create** when ready. - -  - -### Create the Redis Cluster - -1. Go back to the ElastiCache dashboard. -1. Select **Redis** on the left menu and select **Create** to create a new - Redis cluster. Do not enable **Cluster Mode** as it is [not supported](../../administration/redis/replication_and_failover_external.md#requirements). Even without cluster mode on, you still get the - chance to deploy Redis in multiple availability zones. -1. In the settings section: - 1. Give the cluster a name (`gitlab-redis`) and a description. - 1. For the version, select the latest. - 1. Leave the port as `6379` because this is what we used in our Redis security group above. - 1. Select the node type (at least `cache.t3.medium`, but adjust to your needs) and the number of replicas. -1. In the advanced settings section: - 1. Select the multi-AZ auto-failover option. - 1. Select the subnet group we created previously. - 1. Manually select the preferred availability zones, and under "Replica 2" - choose a different zone than the other two. - -  - -1. In the security settings, edit the security groups and choose the - `gitlab-redis-sec-group` we had previously created. -1. Leave the rest of the settings to their default values or edit to your liking. -1. When done, select **Create**. - -## Setting up Bastion Hosts - -Because our GitLab instances are in private subnets, we need a way to connect -to these instances with SSH for actions that include making configuration changes -and performing upgrades. One way of doing this is by using a [bastion host](https://en.wikipedia.org/wiki/Bastion_host), -sometimes also referred to as a jump box. - -NOTE: -If you do not want to maintain bastion hosts, you can set up [AWS Systems Manager Session Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html) for access to instances. This is beyond the scope of this document. - -### Create Bastion Host A - -1. Go to the EC2 Dashboard and select **Launch instance**. -1. Select the **Ubuntu Server 18.04 LTS (HVM)** AMI. -1. Choose an instance type. We use a `t2.micro` as we only use the bastion host to SSH into our other instances. -1. Select **Configure Instance Details**. - 1. Under **Network**, select the `gitlab-vpc` from the dropdown list. - 1. Under **Subnet**, select the public subnet we created earlier (`gitlab-public-10.0.0.0`). - 1. Double check that under **Auto-assign Public IP** you have **Use subnet setting (Enable)** selected. - 1. Leave everything else as default and select **Add Storage**. -1. For storage, we leave everything as default and only add an 8GB root volume. We do not store anything on this instance. -1. Select **Add Tags** and on the next screen select **Add Tag**. - 1. We only set `Key: Name` and `Value: Bastion Host A`. -1. Select **Configure Security Group**. - 1. Select **Create a new security group**, enter a **Security group name** (we use `bastion-sec-group`), and add a description. - 1. We enable SSH access from anywhere (`0.0.0.0/0`). If you want stricter security, specify a single IP address or an IP address range in CIDR notation. - 1. Select **Review and Launch** -1. Review all your settings and, if you're happy, select **Launch**. -1. Acknowledge that you have access to an existing key pair or create a new one. Select **Launch Instance**. - -Confirm that you can SSH into the instance: - -1. On the EC2 Dashboard, select **Instances** in the left menu. -1. Select **Bastion Host A** from your list of instances. -1. Select **Connect** and follow the connection instructions. -1. If you are able to connect successfully, let's move on to setting up our second bastion host for redundancy. - -### Create Bastion Host B - -1. Create an EC2 instance following the same steps as above with the following changes: - 1. For the **Subnet**, select the second public subnet we created earlier (`gitlab-public-10.0.2.0`). - 1. Under the **Add Tags** section, we set `Key: Name` and `Value: Bastion Host B` so that we can easily identify our two instances. - 1. For the security group, select the existing `bastion-sec-group` we created above. - -### Use SSH Agent Forwarding - -EC2 instances running Linux use private key files for SSH authentication. You connect to your bastion host using an SSH client and the private key file stored on your client. Because the private key file is not present on the bastion host, you are not able to connect to your instances in private subnets. - -Storing private key files on your bastion host is a bad idea. To get around this, use SSH agent forwarding on your client. See [Securely Connect to Linux Instances Running in a Private Amazon VPC](https://aws.amazon.com/blogs/security/securely-connect-to-linux-instances-running-in-a-private-amazon-vpc/) for a step-by-step guide on how to use SSH agent forwarding. - -## Install GitLab and create custom AMI - -We need a preconfigured, custom GitLab AMI to use in our launch configuration later. As a starting point, we use the official GitLab AMI to create a GitLab instance. Then, we add our custom configuration for PostgreSQL, Redis, and Gitaly. If you prefer, instead of using the official GitLab AMI, you can also spin up an EC2 instance of your choosing and [manually install GitLab](https://about.gitlab.com/install/). - -### Install GitLab - -From the EC2 dashboard: - -1. Use the section below titled "[Find official GitLab-created AMI IDs on AWS](#find-official-gitlab-created-ami-ids-on-aws)" to find the correct AMI to launch. -1. After selecting **Launch** on the desired AMI, select an instance type based on your workload. Consult the [hardware requirements](../../install/requirements.md#hardware-requirements) to choose one that fits your needs (at least `c5.xlarge`, which is sufficient to accommodate 100 users). -1. Select **Configure Instance Details**: - 1. In the **Network** dropdown list, select `gitlab-vpc`, the VPC we created earlier. - 1. In the **Subnet** dropdown list, select `gitlab-private-10.0.1.0` from the list of subnets we created earlier. - 1. Double check that **Auto-assign Public IP** is set to `Use subnet setting (Disable)`. - 1. Select **Add Storage**. - 1. The root volume is 8GiB by default and should be enough given that we do not store any data there. -1. Select **Add Tags** and add any tags you may need. In our case, we only set `Key: Name` and `Value: GitLab`. -1. Select **Configure Security Group**. Check **Select an existing security group** and select the `gitlab-loadbalancer-sec-group` we created earlier. -1. Select **Review and launch** followed by **Launch** if you're happy with your settings. -1. Finally, acknowledge that you have access to the selected private key file or create a new one. Select **Launch Instances**. - -### Add custom configuration - -Connect to your GitLab instance via **Bastion Host A** using [SSH Agent Forwarding](#use-ssh-agent-forwarding). Once connected, add the following custom configuration: - -#### Disable Let's Encrypt - -Because we're adding our SSL certificate at the load balancer, we do not need the GitLab built-in support for Let's Encrypt. Let's Encrypt [is enabled by default](https://docs.gitlab.com/omnibus/settings/ssl/index.html#enable-the-lets-encrypt-integration) when using an `https` domain in GitLab 10.7 and later, so we must explicitly disable it: - -1. Open `/etc/gitlab/gitlab.rb` and disable it: - - ```ruby - letsencrypt['enable'] = false - ``` - -1. Save the file and reconfigure for the changes to take effect: - - ```shell - sudo gitlab-ctl reconfigure - ``` - -#### Install the required extensions for PostgreSQL - -From your GitLab instance, connect to the RDS instance to verify access and to install the required `pg_trgm` and `btree_gist` extensions. - -To find the host or endpoint, go to **Amazon RDS > Databases** and select the database you created earlier. Look for the endpoint under the **Connectivity & security** tab. - -Do not to include the colon and port number: - -```shell -sudo /opt/gitlab/embedded/bin/psql -U gitlab -h <rds-endpoint> -d gitlabhq_production -``` - -At the `psql` prompt create the extension and then quit the session: - -```shell -psql (10.9) -Type "help" for help. - -gitlab=# CREATE EXTENSION pg_trgm; -gitlab=# CREATE EXTENSION btree_gist; -gitlab=# \q -``` - -#### Configure GitLab to connect to PostgreSQL and Redis - -1. Edit `/etc/gitlab/gitlab.rb`, find the `external_url 'http://<domain>'` option - and change it to the `https` domain you are using. - -1. Look for the GitLab database settings and uncomment as necessary. In - our current case we specify the database adapter, encoding, host, name, - username, and password: - - ```ruby - # Disable the built-in Postgres - postgresql['enable'] = false - - # Fill in the connection details - gitlab_rails['db_adapter'] = "postgresql" - gitlab_rails['db_encoding'] = "unicode" - gitlab_rails['db_database'] = "gitlabhq_production" - gitlab_rails['db_username'] = "gitlab" - gitlab_rails['db_password'] = "mypassword" - gitlab_rails['db_host'] = "<rds-endpoint>" - ``` - -1. Next, we must configure the Redis section by adding the host and - uncommenting the port: - - ```ruby - # Disable the built-in Redis - redis['enable'] = false - - # Fill in the connection details - gitlab_rails['redis_host'] = "<redis-endpoint>" - gitlab_rails['redis_port'] = 6379 - ``` - -1. Finally, reconfigure GitLab for the changes to take effect: - - ```shell - sudo gitlab-ctl reconfigure - ``` - -1. You can also run a check and a service status to make sure - everything has been setup correctly: - - ```shell - sudo gitlab-rake gitlab:check - sudo gitlab-ctl status - ``` - -#### Set up Gitaly - -WARNING: -In this architecture, having a single Gitaly server creates a single point of failure. Use -[Gitaly Cluster](../../administration/gitaly/praefect.md) to remove this limitation. - -Gitaly is a service that provides high-level RPC access to Git repositories. -It should be enabled and configured on a separate EC2 instance in one of the -[private subnets](#subnets) we configured previously. - -Let's create an EC2 instance where we install Gitaly: - -1. From the EC2 dashboard, select **Launch instance**. -1. Choose an AMI. In this example, we select the **Ubuntu Server 18.04 LTS (HVM), SSD Volume Type**. -1. Choose an instance type. We pick a `c5.xlarge`. -1. Select **Configure Instance Details**. - 1. In the **Network** dropdown list, select `gitlab-vpc`, the VPC we created earlier. - 1. In the **Subnet** dropdown list, select `gitlab-private-10.0.1.0` from the list of subnets we created earlier. - 1. Double check that **Auto-assign Public IP** is set to `Use subnet setting (Disable)`. - 1. Select **Add Storage**. -1. Increase the Root volume size to `20 GiB` and change the **Volume Type** to `Provisioned IOPS SSD (io1)`. (This is an arbitrary size. Create a volume big enough for your repository storage requirements.) - 1. For **IOPS** set `1000` (20 GiB x 50 IOPS). You can provision up to 50 IOPS per GiB. If you select a larger volume, increase the IOPS accordingly. Workloads where many small files are written in a serialized manner, like `git`, requires performant storage, hence the choice of `Provisioned IOPS SSD (io1)`. -1. Select **Add Tags** and add your tags. In our case, we only set `Key: Name` and `Value: Gitaly`. -1. Select **Configure Security Group** and let's **Create a new security group**. - 1. Give your security group a name and description. We use `gitlab-gitaly-sec-group` for both. - 1. Create a **Custom TCP** rule and add port `8075` to the **Port Range**. For the **Source**, select the `gitlab-loadbalancer-sec-group`. - 1. Also add an inbound rule for SSH from the `bastion-sec-group` so that we can connect using [SSH Agent Forwarding](#use-ssh-agent-forwarding) from the Bastion hosts. -1. Select **Review and launch** followed by **Launch** if you're happy with your settings. -1. Finally, acknowledge that you have access to the selected private key file or create a new one. Select **Launch Instances**. - -NOTE: -Instead of storing configuration _and_ repository data on the root volume, you can also choose to add an additional EBS volume for repository storage. Follow the same guidance as above. See the [Amazon EBS pricing](https://aws.amazon.com/ebs/pricing/). We do not recommend using EFS as it may negatively impact the performance of GitLab. You can review the [relevant documentation](../../administration/nfs.md#avoid-using-cloud-based-file-systems) for more details. - -Now that we have our EC2 instance ready, follow the [documentation to install GitLab and set up Gitaly on its own server](../../administration/gitaly/configure_gitaly.md#run-gitaly-on-its-own-server). Perform the client setup steps from that document on the [GitLab instance we created](#install-gitlab) above. - -#### Add Support for Proxied SSL - -As we are terminating SSL at our [load balancer](#load-balancer), follow the steps at [Supporting proxied SSL](https://docs.gitlab.com/omnibus/settings/ssl/index.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination) to configure this in `/etc/gitlab/gitlab.rb`. - -Remember to run `sudo gitlab-ctl reconfigure` after saving the changes to the `gitlab.rb` file. - -#### Fast lookup of authorized SSH keys - -The public SSH keys for users allowed to access GitLab are stored in `/var/opt/gitlab/.ssh/authorized_keys`. Typically we'd use shared storage so that all the instances are able to access this file when a user performs a Git action over SSH. Because we do not have shared storage in our setup, we update our configuration to authorize SSH users via indexed lookup in the GitLab database. - -Follow the instructions at [Set up fast SSH key lookup](../../administration/operations/fast_ssh_key_lookup.md#set-up-fast-lookup) to switch from using the `authorized_keys` file to the database. - -If you do not configure fast lookup, Git actions over SSH results in the following error: - -```shell -Permission denied (publickey). -fatal: Could not read from remote repository. - -Please make sure you have the correct access rights -and the repository exists. -``` - -#### Configure host keys - -Ordinarily we would manually copy the contents (primary and public keys) of `/etc/ssh/` on the primary application server to `/etc/ssh` on all secondary servers. This prevents false man-in-the-middle-attack alerts when accessing servers in your cluster behind a load balancer. - -We automate this by creating static host keys as part of our custom AMI. As these host keys are also rotated every time an EC2 instance boots up, "hard coding" them into our custom AMI serves as a workaround. - -On your GitLab instance run the following: - -```shell -sudo mkdir /etc/ssh_static -sudo cp -R /etc/ssh/* /etc/ssh_static -``` - -In `/etc/ssh/sshd_config` update the following: - -```shell -# HostKeys for protocol version 2 -HostKey /etc/ssh_static/ssh_host_rsa_key -HostKey /etc/ssh_static/ssh_host_dsa_key -HostKey /etc/ssh_static/ssh_host_ecdsa_key -HostKey /etc/ssh_static/ssh_host_ed25519_key -``` - -#### Amazon S3 object storage - -Because we're not using NFS for shared storage, we use [Amazon S3](https://aws.amazon.com/s3/) buckets to store backups, artifacts, LFS objects, uploads, merge request diffs, container registry images, and more. Our documentation includes [instructions on how to configure object storage](../../administration/object_storage.md) for each of these data types, and other information about using object storage with GitLab. - -NOTE: -Because we are using the [AWS IAM profile](#create-an-iam-role) we created earlier, be sure to omit the AWS access key and secret access key/value pairs when configuring object storage. Instead, use `'use_iam_profile' => true` in your configuration as shown in the object storage documentation linked above. - -Remember to run `sudo gitlab-ctl reconfigure` after saving the changes to the `gitlab.rb` file. - ---- - -That concludes the configuration changes for our GitLab instance. Next, we create a custom AMI based on this instance to use for our launch configuration and auto scaling group. - -### Log in for the first time - -Using the domain name you used when setting up [DNS for the load balancer](#configure-dns-for-load-balancer), you should now be able to visit GitLab in your browser. - -Depending on how you installed GitLab and if you did not change the password by any other means, the default password is either: - -- Your instance ID if you used the official GitLab AMI. -- A randomly generated password stored for 24 hours in `/etc/gitlab/initial_root_password`. - -To change the default password, log in as the `root` user with the default password and [change it in the user profile](../../user/profile/user_passwords.md#change-your-password). - -When our [auto scaling group](#create-an-auto-scaling-group) spins up new instances, we are able to sign in with username `root` and the newly created password. - -### Create custom AMI - -On the EC2 dashboard: - -1. Select the `GitLab` instance we [created earlier](#install-gitlab). -1. Select **Actions**, scroll down to **Image** and select **Create Image**. -1. Give your image a name and description (we use `GitLab-Source` for both). -1. Leave everything else as default and select **Create Image** - -Now we have a custom AMI that we use to create our launch configuration the next step. - -## Deploy GitLab inside an auto scaling group - -### Create a launch configuration - -From the EC2 dashboard: - -1. Select **Launch Configurations** from the left menu and select **Create launch configuration**. -1. Select **My AMIs** from the left menu and select the `GitLab` custom AMI we created above. -1. Select an instance type best suited for your needs (at least a `c5.xlarge`) and select **Configure details**. -1. Enter a name for your launch configuration (we use `gitlab-ha-launch-config`). -1. **Do not** check **Request Spot Instance**. -1. From the **IAM Role** dropdown list, pick the `GitLabAdmin` instance role we [created earlier](#create-an-iam-ec2-instance-role-and-profile). -1. Leave the rest as defaults and select **Add Storage**. -1. The root volume is 8GiB by default and should be enough given that we do not store any data there. Select **Configure Security Group**. -1. Check **Select and existing security group** and select the `gitlab-loadbalancer-sec-group` we created earlier. -1. Select **Review**, review your changes, and select **Create launch configuration**. -1. Acknowledge that you have access to the private key or create a new one. Select **Create launch configuration**. - -### Create an auto scaling group - -1. After the launch configuration is created, select **Create an Auto Scaling group using this launch configuration** to start creating the auto scaling group. -1. Enter a **Group name** (we use `gitlab-auto-scaling-group`). -1. For **Group size**, enter the number of instances you want to start with (we enter `2`). -1. Select the `gitlab-vpc` from the **Network** dropdown list. -1. Add both the private [subnets we created earlier](#subnets). -1. Expand the **Advanced Details** section and check the **Receive traffic from one or more load balancers** option. -1. From the **Classic Load Balancers** dropdown list, select the load balancer we created earlier. -1. For **Health Check Type**, select **ELB**. -1. We leave our **Health Check Grace Period** as the default `300` seconds. Select **Configure scaling policies**. -1. Check **Use scaling policies to adjust the capacity of this group**. -1. For this group we scale between 2 and 4 instances where one instance is added if CPU -utilization is greater than 60% and one instance is removed if it falls -to less than 45%. - - - -1. Finally, configure notifications and tags as you see fit, review your changes, and create the -auto scaling group. - -As the auto scaling group is created, you see your new instances spinning up in your EC2 dashboard. You also see the new instances added to your load balancer. After the instances pass the heath check, they are ready to start receiving traffic from the load balancer. - -Because our instances are created by the auto scaling group, go back to your instances and terminate the [instance we created manually above](#install-gitlab). We only needed this instance to create our custom AMI. - -## Health check and monitoring with Prometheus - -Apart from Amazon's Cloudwatch which you can enable on various services, -GitLab provides its own integrated monitoring solution based on Prometheus. -For more information about how to set it up, see -[GitLab Prometheus](../../administration/monitoring/prometheus/index.md). - -GitLab also has various [health check endpoints](../../administration/monitoring/health_check.md) -that you can ping and get reports. - -## GitLab Runner - -If you want to take advantage of [GitLab CI/CD](../../ci/index.md), you have to -set up at least one [runner](https://docs.gitlab.com/runner/). - -Read more on configuring an -[autoscaling GitLab Runner on AWS](https://docs.gitlab.com/runner/configuration/runner_autoscale_aws/). - -## Backup and restore - -GitLab provides [a tool to back up](../../administration/backup_restore/index.md) -and restore its Git data, database, attachments, LFS objects, and so on. - -Some important things to know: - -- The backup/restore tool **does not** store some configuration files, like secrets; you - must [configure this yourself](../../administration/backup_restore/backup_gitlab.md#storing-configuration-files). -- By default, the backup files are stored locally, but you can - [backup GitLab using S3](../../administration/backup_restore/backup_gitlab.md#using-amazon-s3). -- You can [exclude specific directories form the backup](../../administration/backup_restore/backup_gitlab.md#excluding-specific-directories-from-the-backup). - -### Backing up GitLab - -To back up GitLab: - -1. SSH into your instance. -1. Take a backup: - - ```shell - sudo gitlab-backup create - ``` - -NOTE: -For GitLab 12.1 and earlier, use `gitlab-rake gitlab:backup:create`. - -### Restoring GitLab from a backup - -To restore GitLab, first review the [restore documentation](../../administration/backup_restore/index.md#restore-gitlab), -and primarily the restore prerequisites. Then, follow the steps under the -[Linux package installations section](../../administration/backup_restore/restore_gitlab.md#restore-for-linux-package-installations). - -## Updating GitLab - -GitLab releases a new version every month on the [release date](https://about.gitlab.com/releases/). Whenever a new version is -released, you can update your GitLab instance: - -1. SSH into your instance -1. Take a backup: - - ```shell - sudo gitlab-backup create - ``` - -NOTE: -For GitLab 12.1 and earlier, use `gitlab-rake gitlab:backup:create`. - -1. Update the repositories and install GitLab: - - ```shell - sudo apt update - sudo apt install gitlab-ee - ``` - -After a few minutes, the new version should be up and running. - -## Find official GitLab-created AMI IDs on AWS - -Read more on how to use [GitLab releases as AMIs](index.md#official-gitlab-releases-as-amis). - -## Conclusion - -In this guide, we went mostly through scaling and some redundancy options, -your mileage may vary. - -Keep in mind that all solutions come with a trade-off between -cost/complexity and uptime. The more uptime you want, the more complex the solution. -And the more complex the solution, the more work is involved in setting up and -maintaining it. - -Have a read through these other resources and feel free to -[open an issue](https://gitlab.com/gitlab-org/gitlab/-/issues/new) -to request additional material: - -- [Scaling GitLab](../../administration/reference_architectures/index.md): - GitLab supports several different types of clustering. -- [Geo replication](../../administration/geo/index.md): - Geo is the solution for widely distributed development teams. -- [Linux package](https://docs.gitlab.com/omnibus/) - Everything you must know - about administering your GitLab instance. -- [Add a license](../../administration/license.md): - Activate all GitLab Enterprise Edition functionality with a license. -- [Pricing](https://about.gitlab.com/pricing/): Pricing for the different tiers. - -## Troubleshooting - -### Instances are failing health checks - -If your instances are failing the load balancer's health checks, verify that they are returning a status `200` from the health check endpoint we configured earlier. Any other status, including redirects like status `302`, causes the health check to fail. - -You may have to set a password on the `root` user to prevent automatic redirects on the sign-in endpoint before health checks pass. - -### "The change you requested was rejected (422)" - -If you see this page when trying to set a password via the web interface, make sure `external_url` in `gitlab.rb` matches the domain you are making a request from, and run `sudo gitlab-ctl reconfigure` after making any changes to it. - -### Some job logs are not uploaded to object storage - -When the GitLab deployment is scaled up to more than one node, some job logs may not be uploaded to [object storage](../../administration/object_storage.md) properly. [Incremental logging is required](../../administration/object_storage.md#alternatives-to-file-system-storage) for CI to use object storage. - -Enable [incremental logging](../../administration/job_logs.md#enable-or-disable-incremental-logging) if it has not already been enabled. +<!-- This redirect file can be deleted after <YYYY-MM-DD>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
\ No newline at end of file diff --git a/doc/solutions/cloud/aws/gitaly_sre_for_aws.md b/doc/solutions/cloud/aws/gitaly_sre_for_aws.md new file mode 100644 index 00000000000..318316b95b8 --- /dev/null +++ b/doc/solutions/cloud/aws/gitaly_sre_for_aws.md @@ -0,0 +1,91 @@ +--- +stage: Solutions Architecture +group: Solutions Architecture +info: This page is owned by the Solutions Architecture team. +description: Doing SRE for Gitaly instances on AWS. +--- + +# SRE Considerations for Gitaly on AWS **(FREE SELF)** + +## Gitaly SRE considerations + +Gitaly is an embedded service for Git Repository Storage. Gitaly and Gitaly Cluster have been engineered by GitLab to overcome fundamental challenges with horizontal scaling of the open source Git binaries that must be used on the service side of GitLab. Here is in-depth technical reading on the topic: + +### Why Gitaly was built + +If you would like to understand the underlying rationale on why GitLab had to invest in creating Gitaly, read the following minimal list of topics: + +- [Git characteristics that make horizontal scaling difficult](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/DESIGN.md#git-characteristics-that-make-horizontal-scaling-difficult) +- [Git architectural characteristics and assumptions](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/DESIGN.md#git-architectural-characteristics-and-assumptions) +- [Affects on horizontal compute architecture](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/DESIGN.md#affects-on-horizontal-compute-architecture) +- [Evidence to back building a new horizontal layer to scale Git](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/DESIGN.md#evidence-to-back-building-a-new-horizontal-layer-to-scale-git) + +### Gitaly and Praefect elections + +As part of Gitaly cluster consistency, Praefect nodes must occasionally vote on what data copy is the most accurate. This requires an uneven number of Praefect nodes to avoid stalemates. This means that for HA, Gitaly and Praefect require a minimum of three nodes. + +### Gitaly performance monitoring + +Complete performance metrics should be collected for Gitaly instances for identification of bottlenecks, as they could have to do with disk IO, network IO, or memory. + +### Gitaly performance guidelines + +Gitaly functions as the primary Git Repository Storage in GitLab. However, it's not a streaming file server. It also does a lot of demanding computing work, such as preparing and caching Git packfiles which informs some of the performance recommendations below. + +NOTE: +All recommendations are for production configurations, including performance testing. For test configurations, like training or functional testing, you can use less expensive options. However, you should adjust or rebuild if performance is an issue. + +#### Overall recommendations + +- Production-grade Gitaly must be implemented on instance compute due to all of the above and below characteristics. +- Never use [burstable instance types](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-performance-instances.html) (such as `t2`, `t3`, `t4g`) for Gitaly. +- Always use at least the [AWS Nitro generation of instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-types.html#ec2-nitro-instances) to ensure many of the below concerns are automatically handled. +- Use Amazon Linux 2 to ensure that all [AWS oriented hardware and OS optimizations](https://aws.amazon.com/amazon-linux-2/faqs/) are maximized without additional configuration or SRE management. + +#### CPU and memory recommendations + +- The general GitLab Gitaly node recommendations for CPU and Memory assume relatively even loading across repositories. GitLab Performance Tool (GPT) testing of any non-characteristic repositories and/or SRE monitoring of Gitaly metrics may inform when to choose memory and/or CPU higher than general recommendations. + +**To accommodate:** + +- Git packfile operations are memory and CPU intensive. +- If repository commit traffic is dense, large, or very frequent, then more CPU and Memory are required to handle the load. Patterns such as storing binaries and/or busy or large monorepos are examples that can cause high loading. + +#### Disk I/O recommendations + +- Use only SSD storage and the [class of Elastic Block Store (EBS) storage](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html) that suites your durability and speed requirements. +- When not using provisioned EBS IO, EBS volume size determines the I/O level, so provisioning volumes that are much larger than needed can be the least expensive way to improve EBS IO. +- If Gitaly performance monitoring shows signs of disk stress then one of the provisioned IOPS levels can be chosen. EBS IOPS levels also have enhanced durability which may be appealing for some implementations aside from performance considerations. + +**To accommodate:** + +- Gitaly storage is expected to be local (not NFS of any type including EFS). +- Gitaly servers also need disk space for building and caching Git packfiles. This is above and beyond the permanent storage of your Git Repositories. +- Git packfiles are cached in Gitaly. Creation of packfiles in temporary disk benefits from fast disk, and disk caching of packfiles benefits from ample disk space. + +#### Network I/O recommendations + +- Use only instance types [from the list of ones that support Elastic Network Adapter (ENA) advanced networking](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-types.html#instance-type-summary-table) to ensure that cluster replication latency is not due to instance level network I/O bottlenecks. +- Choose instances with sizes with more than 10 Gbps - but only if needed and only when having proven a node level network bottleneck with monitoring and/or stress testing. + +**To accommodate:** + +- Gitaly nodes do the main work of streaming repositories for push and pull operations (to add development endpoints, and to CI/CD). +- Gitaly servers need reasonable low latency between cluster nodes and with Praefect services in order for the cluster to maintain operational and data integrity. +- Gitaly nodes should be selected with network bottleneck avoidance as a primary consideration. +- Gitaly nodes should be monitored for network saturation. +- Not all networking issues can be solved through optimizing the node level networking: + - Gitaly cluster node replication depends on all networking between nodes. + - Gitaly networking performance to pull and push endpoints depends on all networking in between. + +### AWS Gitaly backup + +Due to the nature of how Praefect tracks the replication metadata of Gitaly disk information, the best backup method is [the official backup and restore Rake tasks](../../../administration/backup_restore/index.md). + +### AWS Gitaly recovery + +Gitaly Cluster does not support snapshot backups as these can cause issues where the Praefect database becomes out of syn with the disk storage. Due to the nature of how Praefect rebuilds the replication metadata of Gitaly disk information during a restore, the best recovery method is [the official backup and restore Rake tasks](../../../administration/backup_restore/index.md). + +### Gitaly long term management + +Gitaly node disk sizes must be monitored and increased to accommodate Git repository growth and Gitaly temporary and caching storage needs. The storage configuration on all nodes should be kept identical. diff --git a/doc/solutions/cloud/aws/gitlab_aws_integration.md b/doc/solutions/cloud/aws/gitlab_aws_integration.md index 0e15b3dc9b2..ba0b9717562 100644 --- a/doc/solutions/cloud/aws/gitlab_aws_integration.md +++ b/doc/solutions/cloud/aws/gitlab_aws_integration.md @@ -23,14 +23,6 @@ This page attempts to index the ways in which GitLab can integrate with AWS. It | `[GitLab Solution]` | Built as Solution Example by GitLab or GitLab Partners | Community/Example | | `[CI Solution]` | Built, at least in part, using GitLab CI and therefore <br />more customer customizable. | Items tagged `[CI Solution will]` <br />also carry one of the other tags <br />that indicates the maintenance status. | -## Partnership Information - -### AWS known issues list - -Known issues are gathered from within GitLab and from customer reported issues. Customers successfully implement GitLab with a variety of “as a Service” components that GitLab has not specifically been designed for, nor has ongoing testing for. While GitLab does take partner technologies very seriously, the highlighting of known issues here is a convenience for implementers and it does not imply that GitLab has targeted compatibility with, nor carries any type of guarantee of running on the partner technology where the issues occur. Consult individual issues to understand the GitLab stance and plans on any given known issue. - -See the [GitLab AWS known issues list](https://gitlab.com/gitlab-com/alliances/aws/public-tracker/-/issues?label_name[]=AWS+Known+Issue) for a complete list. - ## Integrations For Development Activities ### SCM Integrations @@ -91,8 +83,8 @@ Generally solutions demonstrate end-to-end capabilities for the development fram - Installing GitLab Self-Managed on AWS - GitLab Single EC2 Instance. `[GitLab Built]` - - [Using 5 Seat AWS marketplace subscription](../../../install/aws/index.md#marketplace-subscription) - - [Using Prepared AMIs](../../../install/aws/index.md#official-gitlab-releases-as-amis) - Bring Your Own License for Enterprise Edition. + - [Using 5 Seat AWS marketplace subscription](gitlab_single_box_on_aws.md#marketplace-subscription) + - [Using Prepared AMIs](gitlab_single_box_on_aws.md#official-gitlab-releases-as-amis) - Bring Your Own License for Enterprise Edition. - GitLab Cloud Native Hybrid Scaled on AWS EKS and Paas. `[GitLab Built]` - Using GitLab Environment Toolkit (GET) - `[GitLab Solution]` @@ -101,6 +93,9 @@ Generally solutions demonstrate end-to-end capabilities for the development fram - Using GitLab Environment Toolkit (GET) - `[GitLab Solution]` - [Amazon Managed Grafana](https://docs.aws.amazon.com/grafana/latest/userguide/gitlab-AMG-datasource.html) for GitLab self-managed Prometheus metrics. `[AWS Built]` + +## GitLab Runner on AWS Compute + - [Autoscaling GitLab Runner on AWS EC2](https://docs.gitlab.com/runner/configuration/runner_autoscale_aws/). `[GitLab Built]` - [GitLab HA Scaling Runner Vending Machine for AWS EC2 ASG](https://gitlab.com/guided-explorations/aws/gitlab-runner-autoscaling-aws-asg/). `[GitLab Solution]` - Runner vending machine training resources. diff --git a/doc/solutions/cloud/aws/gitlab_instance_on_aws.md b/doc/solutions/cloud/aws/gitlab_instance_on_aws.md new file mode 100644 index 00000000000..320c317d446 --- /dev/null +++ b/doc/solutions/cloud/aws/gitlab_instance_on_aws.md @@ -0,0 +1,55 @@ +--- +stage: Solutions Architecture +group: Solutions Architecture +info: This page is owned by the Solutions Architecture team. +--- + +{::options parse_block_html="true" /} + +# Provision GitLab Instances on AWS EKS **(FREE SELF)** + +## Available Infrastructure as Code for GitLab Instance Installation on AWS + +The [GitLab Environment Toolkit (GET)](https://gitlab.com/gitlab-org/gitlab-environment-toolkit/-/blob/main/README.md) is a set of opinionated Terraform and Ansible scripts. These scripts help with the deployment of Linux package or Cloud Native Hybrid environments on selected cloud providers and are used by GitLab developers for [GitLab Dedicated](../../../subscriptions/gitlab_dedicated/index.md) (for example). + +You can use the GitLab Environment Toolkit to deploy a Cloud Native Hybrid environment on AWS. However, it's not required and may not support every valid permutation. That said, the scripts are presented as-is and you can adapt them accordingly. + +### Two and Three Zone High Availability + +While GitLab Reference Architectures generally encourage three zone redundancy, AWS Quick Starts and AWS Well Architected consider two zone redundancy as AWS Well Architected. Individual implementations should weigh the costs of two and three zone configurations against their own high availability requirements for a final configuration. + +Gitaly Cluster uses a consistency voting system to implement strong consistency between synchronized nodes. Regardless of the number of availability zones implemented, there will always need to be a minimum of three Gitaly and three Praefect nodes in the cluster to avoid voting stalemates cause by an even number of nodes. + +## AWS PaaS qualified for all GitLab implementations + +For both implementations that used the Linux package or Cloud Native Hybrid implementations, the following GitLab Service roles can be performed by AWS Services (PaaS). Any PaaS solutions that require preconfigured sizing based on the scale of your instance will also be listed in the per-instance size Bill of Materials lists. Those PaaS that do not require specific sizing, are not repeated in the BOM lists (for example, AWS Certification Manager). + +These services have been tested with GitLab. + +Some services, such as log aggregation, outbound email are not specified by GitLab, but where provided are noted. + +| GitLab Services | AWS PaaS (Tested) | Provided by AWS Cloud <br />Native Hybrid Quick Start | +| ------------------------------------------------------------ | ------------------------------ | ------------------------------------------------------------ | +| <u>Tested PaaS Mentioned in Reference Architectures</u> | | | +| **PostgreSQL Database** | Amazon RDS PostgreSQL | Yes. | +| **Redis Caching** | Redis ElastiCache | Yes. | +| **Gitaly Cluster (Git Repository Storage)**<br />(Including Praefect and PostgreSQL) | ASG and Instances | Yes - ASG and Instances<br />**Note: Gitaly cannot be put into a Kubernetes Cluster.** | +| **All GitLab storages besides Git Repository Storage**<br />(Includes Git-LFS which is S3 Compatible) | AWS S3 | Yes | +| | | | +| <u>Tested PaaS for Supplemental Services</u> | | | +| **Front End Load Balancing** | AWS ELB | Yes | +| **Internal Load Balancing** | AWS ELB | Yes | +| **Outbound Email Services** | AWS Simple Email Service (SES) | Yes | +| **Certificate Authority and Management** | AWS Certificate Manager (ACM) | Yes | +| **DNS** | AWS Route53 (tested) | Yes | +| **GitLab and Infrastructure Log Aggregation** | AWS CloudWatch Logs | Yes (ContainerInsights Agent for EKS) | +| **Infrastructure Performance Metrics** | AWS CloudWatch Metrics | Yes | +| | | | +| <u>Supplemental Services and Configurations (Tested)</u> | | | +| **Prometheus for GitLab** | AWS EKS (Cloud Native Only) | Yes | +| **Grafana for GitLab** | AWS EKS (Cloud Native Only) | Yes | +| **Administrative Access to GitLab Backend** | Bastion Host in VPC | Yes - HA - Preconfigured for Cluster Management. | +| **Encryption (In Transit / At Rest)** | AWS KMS | Yes | +| **Secrets Storage for Provisioning** | AWS Secrets Manager | Yes | +| **Configuration Data for Provisioning** | AWS Parameter Store | Yes | +| **AutoScaling Kubernetes** | EKS AutoScaling Agent | Yes | diff --git a/doc/solutions/cloud/aws/gitlab_single_box_on_aws.md b/doc/solutions/cloud/aws/gitlab_single_box_on_aws.md new file mode 100644 index 00000000000..7a647f1d8d7 --- /dev/null +++ b/doc/solutions/cloud/aws/gitlab_single_box_on_aws.md @@ -0,0 +1,51 @@ +--- +stage: Solutions Architecture +group: Solutions Architecture +info: This page is owned by the Solutions Architecture team. +--- + +{::options parse_block_html="true" /} + +# Provision GitLab on a single EC2 instance in AWS **(FREE SELF)** + +If you want to provision a single GitLab instance on AWS, you have two options: + +- The marketplace subscription +- The official GitLab AMIs + +## Marketplace subscription + +GitLab provides a 5 user subscription as an AWS Marketplace subscription to help teams of all sizes to get started with an Ultimate licensed instance in record time. The Marketplace subscription can be easily upgraded to any GitLab licensing via an AWS Marketplace Private Offer, with the convenience of continued AWS billing. No migration is necessary to obtain a larger, non-time based license from GitLab. Per-minute licensing is automatically removed when you accept the private offer. + +For a tutorial on provisioning a GitLab Instance via a Marketplace Subscription, [use this tutorial](https://gitlab.awsworkshop.io/040_partner_setup.html). The tutorial links to the [GitLab Ultimate Marketplace Listing](https://aws.amazon.com/marketplace/pp/prodview-g6ktjmpuc33zk), but you can also use the [GitLab Premium Marketplace Listing](https://aws.amazon.com/marketplace/pp/prodview-amk6tacbois2k) to provision an instance. + +## Official GitLab releases as AMIs + +GitLab produces Amazon Machine Images (AMI) during the regular release process. The AMIs can be used for single instance GitLab installation or, by configuring `/etc/gitlab/gitlab.rb`, can be specialized for specific GitLab service roles (for example a Gitaly server). Older releases remain available and can be used to migrate an older GitLab server to AWS. + +Initial licensing can either be the Free Enterprise License (EE) or the open source Community Edition (CE). The Enterprise Edition provides the easiest path forward to a licensed version if the need arises. + +Currently the Amazon AMI uses the Amazon prepared Ubuntu AMI (x86 and ARM are available) as its starting point. + +NOTE: +When deploying a GitLab instance using the official AMI, the root password to the instance is the EC2 **Instance** ID (not the AMI ID). This way of setting the root account password is specific to official GitLab published AMIs ONLY. + +Instances running on Community Edition (CE) require a migration to Enterprise Edition (EE) to subscribe to the GitLab Premium or Ultimate plan. If you want to pursue a subscription, using the Free-forever plan of Enterprise Edition is the least disruptive method. + +NOTE: +Because any given GitLab upgrade might involve data disk updates or database schema upgrades, swapping out the AMI is not sufficient for taking upgrades. + +1. Log in to the AWS Web Console, so that selecting the links in the following step take you directly to the AMI list. +1. Pick the edition you want: + + - [GitLab Enterprise Edition](https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Images:visibility=public-images;ownerAlias=782774275127;search=GitLab%20EE;sort=desc:name): If you want to unlock the enterprise features, a license is needed. + - [GitLab Community Edition](https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Images:visibility=public-images;ownerAlias=782774275127;search=GitLab%20CE;sort=desc:name): The open source version of GitLab. + - [GitLab Premium or Ultimate Marketplace (pre-licensed)](https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Images:visibility=public-images;source=Marketplace;search=GitLab%20EE;sort=desc:name): 5 user license built into per-minute billing. + +1. AMI IDs are unique per region. After you've loaded any of these editions, in the upper-right corner, select the desired target region of the console to see the appropriate AMIs. +1. After the console is loaded, you can add additional search criteria to narrow further. For instance, type `13.` to find only 13.x versions. +1. To launch an EC2 Machine with one of the listed AMIs, check the box at the start of the relevant row, and select **Launch** near the top of left of the page. + +NOTE: +If you are trying to restore from an older version of GitLab while moving to AWS, find the +[Enterprise and Community Editions before GitLab 11.10.3](https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Images:visibility=public-images;ownerAlias=855262394183;sort=desc:name). diff --git a/doc/solutions/cloud/aws/index.md b/doc/solutions/cloud/aws/index.md index 3d5c777de30..81141b9907a 100644 --- a/doc/solutions/cloud/aws/index.md +++ b/doc/solutions/cloud/aws/index.md @@ -6,6 +6,78 @@ info: This page is owned by the Solutions Architecture team. # AWS Solutions -This documentation covers solutions relating to Amazon Web Services (AWS). +This documentation covers solutions relating to GitLab and Amazon Web Services (AWS). [GitLab AWS Integration Index](gitlab_aws_integration.md) +[GitLab Instances on AWS EKS](gitlab_instance_on_aws.md) +[SRE Considerations Gitaly on AWS](gitaly_sre_for_aws.md) +[Provision GitLab on a single EC2 instance in AWS](gitlab_single_box_on_aws.md) + +## Cloud platform well architected compliance + +Testing-backed architectural qualification is a fundamental concept behind implementation patterns: + +- Implementation patterns maintain GitLab Reference Architecture compliance and provide [GitLab Performance Tool](https://gitlab.com/gitlab-org/quality/performance) (GPT) reports to demonstrate adherence to them. +- Implementation patterns may be qualified by and/or contributed to by the technology vendor. For instance, an implementation pattern for AWS may be officially reviewed by AWS. +- Implementation patterns may specify and test Cloud Platform PaaS services for suitability for GitLab. This testing can be coordinated and help qualify these technologies for Reference Architectures. For instance, qualifying compatibility with and availability of runtime versions of top level PaaS such as those for PostgreSQL and Redis. +- Implementation patterns can provided qualified testing for platform limitations, for example, ensuring Gitaly Cluster can work correctly on specific Cloud Platform availability zone latency and throughput characteristics or qualifying what levels of available platform partner local disk performance is workable for Gitaly server to operate with integrity. + +## AWS known issues list + +Known issues are gathered from within GitLab and from customer reported issues. Customers successfully implement GitLab with a variety of “as a Service” components that GitLab has not specifically been designed for, nor has ongoing testing for. While GitLab does take partner technologies very seriously, the highlighting of known issues here is a convenience for implementers and it does not imply that GitLab has targeted compatibility with, nor carries any type of guarantee of running on the partner technology where the issues occur. Consult individual issues to understand the GitLab stance and plans on any given known issue. + +See the [GitLab AWS known issues list](https://gitlab.com/gitlab-com/alliances/aws/public-tracker/-/issues?label_name[]=AWS+Known+Issue) for a complete list. + +## Patterns with working code examples for using GitLab with AWS + +[The Guided Explorations' subgroup for AWS](https://gitlab.com/guided-explorations/aws) contains a variety of working example projects. + +## Platform partner specificity + +Implementation patterns enable platform-specific terminology, best practice architecture, and platform-specific build manifests: + +- Implementation patterns are more vendor specific. For instance, advising specific compute instances / VMs / nodes instead of vCPUs or other generalized measures. +- Implementation patterns are oriented to implementing good architecture for the vendor in view. +- Implementation patterns are written to an audience who is familiar with building on the infrastructure that the implementation pattern targets. For example, if the implementation pattern is for GCP, the specific terminology of GCP is used - including using the specific names for PaaS services. +- Implementation patterns can test and qualify if the versions of PaaS available are compatible with GitLab (for example, PostgreSQL, Redis, etc.). + +## Platform as a Service (PaaS) specification and usage + +Platform as a Service options are a huge portion of the value provided by Cloud Platforms as they simplify operational complexity and reduce the SRE and security skilling required to operate advanced, highly available technology services. Implementation patterns can be pre-qualified against the partner PaaS options. + +- Implementation patterns help implementers understand what PaaS options are known to work and how to choose between PaaS solutions when a single platform has more than one PaaS option for the same GitLab role. +- For instance, where reference architectures do not have a specific recommendation on what technology is leveraged for GitLab outbound email services or what the sizing should be - a Reference Implementation may advise using a cloud providers Email as a Service (PaaS) and possibly even with specific settings. + +## Cost optimizing engineering + +Cost engineering is a fundamental aspect of Cloud Architecture and frequently the savings capabilities available on a platform exert strong influence on how to build out scaled computing. + +- Implementation patterns may engineer specifically for the savings models available on a platform provider. An AWS example would be maximizing the occurrence of a specific instance type for taking advantage of reserved instances. +- Implementation patterns may leverage ephemeral compute where appropriate and with appropriate customer guidelines. For instance, a Kubernetes node group dedicated to runners on ephemeral compute (with appropriate GitLab Runner tagging to indicate the compute type). +- Implementation patterns may include vendor specific cost calculators. + +## Actionability and automatability orientation + +Implementation patterns are one step closer to specifics that can be used as a source for build instructions and automation code: + +- Implementation patterns enable builders to generate a list of vendor specific resources required to implement GitLab for a given Reference Architecture. +- Implementation patterns enable builders to use manual instructions or to create automation to build out the reference implementation. + +## Intended audiences and contributors + +The primary audiences for and contributors to this information is the GitLab **Implementation Eco System** which consists of at least: + +GitLab Implementation Community: + +- Customers +- GitLab Channel Partners (Integrators) +- Platform Partners + +GitLab Internal Implementation Teams: + +- Quality / Distribution / Self-Managed +- Alliances +- Training +- Support +- Professional Services +- Public Sector diff --git a/doc/solutions/index.md b/doc/solutions/index.md index c1aa356fe64..9d7fec1b549 100644 --- a/doc/solutions/index.md +++ b/doc/solutions/index.md @@ -13,3 +13,7 @@ This documentation is the home for solutions GitLab wishes to share with custome ## Relationship to documentation While information in this section gives valuable and qualified guidance on ways to solve problems by using the GitLab platform, the product documentation is the authoritative reference for product features and functions. + +## Solutions categories + +[Cloud Solutions](cloud/index.md) diff --git a/doc/update/deprecations.md b/doc/update/deprecations.md index 614136bedc6..b6ccfea020e 100644 --- a/doc/update/deprecations.md +++ b/doc/update/deprecations.md @@ -209,6 +209,26 @@ Because Cloud Native Buildpacks do not support automatic testing, the Auto Test <div class="deprecation breaking-change" data-milestone="17.0"> +### Breaking change to the Maven repository group permissions + +<div class="deprecation-notes"> +- Announced in GitLab <span class="milestone">16.6</span> +- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change)) +- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/393933). +</div> + +The Maven repository exposes an API endpoint at the group level that allows Maven clients to download files from a specific package. The package finder first locates the package within the group, and then finds the file within the package. +However, there is a limitation that affects duplicate package names hosted in different projects. The Maven package finder always returns the most recent package, but the "most recent" filter depends on user permissions. It is possible for a user with different permissions in different projects to download the wrong Maven package. + +In GitLab 17.0, the package finder logic will be fixed so that the "most recent" package is the last updated name and version of a package in a group. User permissions will be checked after the most recent package is found. +After the change, download requests for users without correct permissions will be rejected. If your workflow depends on the current bugged behavior, this fix will introduce a breaking change. + +The change will be introduced in GitLab 16.6 behind a feature flag. If you are interested in enabling the feature flag for your group, leave a comment in [issue 393933](https://gitlab.com/gitlab-org/gitlab/-/issues/393933). + +</div> + +<div class="deprecation breaking-change" data-milestone="17.0"> + ### CiRunner.projects default sort is changing to `id_desc` <div class="deprecation-notes"> diff --git a/doc/user/compliance/compliance_center/index.md b/doc/user/compliance/compliance_center/index.md index 6bc6be8ddb6..4a42a70a7e7 100644 --- a/doc/user/compliance/compliance_center/index.md +++ b/doc/user/compliance/compliance_center/index.md @@ -111,9 +111,9 @@ You can sort the compliance report on: You can filter the compliance violations report on: -- Project. -- Date range of merge. -- Target branch. +- The project that the violation was found on. +- The date range of violation. +- The target branch of the violation. Select a row to see details of the compliance violation. diff --git a/doc/user/packages/container_registry/troubleshoot_container_registry.md b/doc/user/packages/container_registry/troubleshoot_container_registry.md index acef00640ef..cb0bcf3f35b 100644 --- a/doc/user/packages/container_registry/troubleshoot_container_registry.md +++ b/doc/user/packages/container_registry/troubleshoot_container_registry.md @@ -145,3 +145,24 @@ This is typically a result of [a performance issue with `kaniko` and HTTP/2](htt The current workaround is to use HTTP/1.1 when pushing with `kaniko`. To use HTTP/1.1, set the `GODEBUG` environment variable to `"http2client=0"`. + +## `docker login` command fails with `access forbidden` + +The container registry [returns the GitLab API URL to the Docker client](../../../administration/packages/container_registry.md#architecture-of-gitlab-container-registry) +to validate credentials. The Docker client uses basic auth, so the request contains +the `Authorization` header. If the `Authorization` header is missing in the request to the +`/jwt/auth` endpoint configured in the `token_realm` for the registry configuration, +you receive an `access forbidden` error message. + +For example: + +```plaintext +> docker login gitlab.example.com:4567 + +Username: user +Password: +Error response from daemon: Get "https://gitlab.company.com:4567/v2/": denied: access forbidden +``` + +To avoid this error, ensure the `Authorization` header is not stripped from the request. +For example, a proxy in front of GitLab might be redirecting to the `/jwt/auth` endpoint. |