Welcome to mirror list, hosted at ThFree Co, Russian Federation.

gitlab.com/gitlab-org/gitlab-foss.git - Unnamed repository; edit this file 'description' to name the repository.
summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorGitLab Bot <gitlab-bot@gitlab.com>2023-02-20 16:49:51 +0300
committerGitLab Bot <gitlab-bot@gitlab.com>2023-02-20 16:49:51 +0300
commit71786ddc8e28fbd3cb3fcc4b3ff15e5962a1c82e (patch)
tree6a2d93ef3fb2d353bb7739e4b57e6541f51cdd71 /doc/administration
parenta7253423e3403b8c08f8a161e5937e1488f5f407 (diff)
Add latest changes from gitlab-org/gitlab@15-9-stable-eev15.9.0-rc42
Diffstat (limited to 'doc/administration')
-rw-r--r--doc/administration/audit_events.md7
-rw-r--r--doc/administration/auth/atlassian.md5
-rw-r--r--doc/administration/auth/authentiq.md98
-rw-r--r--doc/administration/auth/cognito.md10
-rw-r--r--doc/administration/auth/crowd.md5
-rw-r--r--doc/administration/auth/index.md6
-rw-r--r--doc/administration/auth/jwt.md4
-rw-r--r--doc/administration/auth/ldap/index.md5
-rw-r--r--doc/administration/auth/ldap/ldap-troubleshooting.md13
-rw-r--r--doc/administration/auth/oidc.md20
-rw-r--r--doc/administration/clusters/kas.md2
-rw-r--r--doc/administration/compliance.md2
-rw-r--r--doc/administration/dedicated/index.md140
-rw-r--r--doc/administration/docs_self_host.md2
-rw-r--r--doc/administration/encrypted_configuration.md2
-rw-r--r--doc/administration/geo/disaster_recovery/index.md23
-rw-r--r--doc/administration/geo/index.md3
-rw-r--r--doc/administration/geo/replication/configuration.md2
-rw-r--r--doc/administration/geo/replication/container_registry.md7
-rw-r--r--doc/administration/geo/replication/datatypes.md4
-rw-r--r--doc/administration/geo/replication/multiple_servers.md4
-rw-r--r--doc/administration/geo/replication/security_review.md6
-rw-r--r--doc/administration/geo/replication/troubleshooting.md187
-rw-r--r--doc/administration/geo/replication/version_specific_upgrades.md8
-rw-r--r--doc/administration/geo/secondary_proxy/index.md9
-rw-r--r--doc/administration/geo/secondary_proxy/location_aware_external_url.md2
-rw-r--r--doc/administration/get_started.md16
-rw-r--r--doc/administration/gitaly/configure_gitaly.md8
-rw-r--r--doc/administration/gitaly/index.md12
-rw-r--r--doc/administration/gitaly/monitoring.md15
-rw-r--r--doc/administration/gitaly/praefect.md4
-rw-r--r--doc/administration/gitaly/recovery.md51
-rw-r--r--doc/administration/gitaly/reference.md6
-rw-r--r--doc/administration/gitaly/troubleshooting.md18
-rw-r--r--doc/administration/housekeeping.md31
-rw-r--r--doc/administration/inactive_project_deletion.md79
-rw-r--r--doc/administration/incoming_email.md141
-rw-r--r--doc/administration/index.md2
-rw-r--r--doc/administration/instance_limits.md24
-rw-r--r--doc/administration/instance_review.md2
-rw-r--r--doc/administration/integration/kroki.md2
-rw-r--r--doc/administration/integration/mailgun.md4
-rw-r--r--doc/administration/integration/plantuml.md2
-rw-r--r--doc/administration/issue_closing_pattern.md11
-rw-r--r--doc/administration/job_artifacts.md19
-rw-r--r--doc/administration/lfs/index.md4
-rw-r--r--doc/administration/load_balancer.md3
-rw-r--r--doc/administration/logs/index.md13
-rw-r--r--doc/administration/logs/log_parsing.md8
-rw-r--r--doc/administration/monitoring/performance/performance_bar.md4
-rw-r--r--doc/administration/monitoring/prometheus/gitlab_metrics.md5
-rw-r--r--doc/administration/monitoring/prometheus/web_exporter.md4
-rw-r--r--doc/administration/nfs.md4
-rw-r--r--doc/administration/object_storage.md332
-rw-r--r--doc/administration/operations/fast_ssh_key_lookup.md75
-rw-r--r--doc/administration/operations/gitlab_sshd.md139
-rw-r--r--doc/administration/operations/index.md1
-rw-r--r--doc/administration/operations/moving_repositories.md2
-rw-r--r--doc/administration/operations/puma.md4
-rw-r--r--doc/administration/operations/rails_console.md8
-rw-r--r--doc/administration/operations/ssh_certificates.md10
-rw-r--r--doc/administration/package_information/defaults.md12
-rw-r--r--doc/administration/package_information/licensing.md4
-rw-r--r--doc/administration/package_information/omnibus_packages.md4
-rw-r--r--doc/administration/package_information/signed_packages.md2
-rw-r--r--doc/administration/packages/container_registry.md8
-rw-r--r--doc/administration/pages/index.md7
-rw-r--r--doc/administration/postgresql/index.md25
-rw-r--r--doc/administration/postgresql/moving.md4
-rw-r--r--doc/administration/postgresql/multiple_databases.md4
-rw-r--r--doc/administration/postgresql/pgbouncer.md6
-rw-r--r--doc/administration/postgresql/replication_and_failover.md12
-rw-r--r--doc/administration/raketasks/check.md34
-rw-r--r--doc/administration/raketasks/github_import.md4
-rw-r--r--doc/administration/raketasks/incoming_email.md149
-rw-r--r--doc/administration/raketasks/maintenance.md9
-rw-r--r--doc/administration/raketasks/project_import_export.md2
-rw-r--r--doc/administration/raketasks/service_desk_email.md149
-rw-r--r--doc/administration/reference_architectures/10k_users.md129
-rw-r--r--doc/administration/reference_architectures/1k_users.md48
-rw-r--r--doc/administration/reference_architectures/25k_users.md130
-rw-r--r--doc/administration/reference_architectures/2k_users.md79
-rw-r--r--doc/administration/reference_architectures/3k_users.md129
-rw-r--r--doc/administration/reference_architectures/50k_users.md125
-rw-r--r--doc/administration/reference_architectures/5k_users.md127
-rw-r--r--doc/administration/reference_architectures/index.md169
-rw-r--r--doc/administration/repository_storage_types.md2
-rw-r--r--doc/administration/sidekiq/extra_sidekiq_processes.md12
-rw-r--r--doc/administration/sidekiq/index.md4
-rw-r--r--doc/administration/sidekiq/processing_specific_job_classes.md6
-rw-r--r--doc/administration/sidekiq/sidekiq_job_migration.md4
-rw-r--r--doc/administration/sidekiq/sidekiq_memory_killer.md131
-rw-r--r--doc/administration/sidekiq/sidekiq_troubleshooting.md6
-rw-r--r--doc/administration/system_hooks.md2
-rw-r--r--doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md2
-rw-r--r--doc/administration/troubleshooting/test_environments.md2
96 files changed, 2108 insertions, 1073 deletions
diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md
index 30e6ca44972..c51b5fbb431 100644
--- a/doc/administration/audit_events.md
+++ b/doc/administration/audit_events.md
@@ -214,6 +214,8 @@ The following actions on groups generate group audit events:
[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/366350) in GitLab 15.3.
- Group had a security policy project linked, changed, or unlinked.
[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/377877) in GitLab 15.6.
+- An environment is protected or unprotected.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216164) in GitLab 15.8.
### Project events
@@ -309,6 +311,8 @@ The following actions on projects generate project audit events:
[Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/85689) in GitLab 15.0.
- Project had a security policy project linked, changed, or unlinked.
[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/377877) in GitLab 15.6.
+- An environment is protected or unprotected.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216164) in GitLab 15.8.
### Instance events **(PREMIUM SELF)**
@@ -354,11 +358,12 @@ Issue [343933](https://gitlab.com/gitlab-org/gitlab/-/issues/343933) proposes to
## Unsupported events
-Some events are not tracked in audit events. The following epics propose support for more events:
+Some events are not tracked in audit events. The following epics and issues propose support for more events:
- [Project settings and activity](https://gitlab.com/groups/gitlab-org/-/epics/474).
- [Group settings and activity](https://gitlab.com/groups/gitlab-org/-/epics/475).
- [Instance-level settings and activity](https://gitlab.com/groups/gitlab-org/-/epics/476).
+- [Deployment Approval activity](https://gitlab.com/gitlab-org/gitlab/-/issues/354782).
If you don't see the event you want in any of the epics, you can either:
diff --git a/doc/administration/auth/atlassian.md b/doc/administration/auth/atlassian.md
index b02ac06b67f..58412078802 100644
--- a/doc/administration/auth/atlassian.md
+++ b/doc/administration/auth/atlassian.md
@@ -41,7 +41,10 @@ To enable the Atlassian OmniAuth provider for passwordless authentication you mu
sudo -u git -H editor /home/git/gitlab/config/gitlab.yml
```
-1. See [Configure initial settings](../../integration/omniauth.md#configure-initial-settings) for initial settings to enable single sign-on and add `atlassian_oauth2` as an OAuth provider.
+1. Edit the [common configuration file settings](../../integration/omniauth.md#configure-common-settings)
+ to add `atlassian_oauth2` as a single sign-on provider. This enables
+ Just-In-Time account provisioning for users who do not have an existing
+ GitLab account.
1. Add the provider configuration for Atlassian:
For Omnibus GitLab installations:
diff --git a/doc/administration/auth/authentiq.md b/doc/administration/auth/authentiq.md
index d51601439f9..a32d2a2cf94 100644
--- a/doc/administration/auth/authentiq.md
+++ b/doc/administration/auth/authentiq.md
@@ -1,100 +1,12 @@
---
-type: reference
stage: Manage
group: Authentication and Authorization
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+remove_date: '2023-02-22'
+redirect_to: '../../integration/omniauth.md'
---
-# Authentiq OmniAuth Provider **(FREE SELF)**
+# Authentiq OmniAuth Provider (removed) **(FREE SELF)**
-To enable the Authentiq OmniAuth provider for passwordless authentication, you must register an application with Authentiq.
-
-Authentiq generates a Client ID and the accompanying Client Secret for you to use.
-
-1. Get your Client credentials (Client ID and Client Secret) at [Authentiq](https://www.authentiq.com/developers).
-
-1. On your GitLab server, open the configuration file:
-
- For omnibus installation
-
- ```shell
- sudo editor /etc/gitlab/gitlab.rb
- ```
-
- For installations from source:
-
- ```shell
- sudo -u git -H editor /home/git/gitlab/config/gitlab.yml
- ```
-
-1. See [Configure initial settings](../../integration/omniauth.md#configure-initial-settings) for initial settings to enable single sign-on and add Authentiq as an OAuth provider.
-
-1. Add the provider configuration for Authentiq:
-
- For Omnibus packages:
-
- ```ruby
- gitlab_rails['omniauth_providers'] = [
- {
- name: "authentiq",
- # label: "Provider name", # optional label for login button, defaults to "Authentiq"
- app_id: "<your_client_id>",
- app_secret: "<your_client_secret>",
- args: {
- "scope": 'aq:name email~rs address aq:push'
- }
- }
- ]
- ```
-
- For installations from source:
-
- ```yaml
- - { name: 'authentiq',
- # label: 'Provider name', # optional label for login button, defaults to "Authentiq"
- app_id: '<your_client_id>',
- app_secret: '<your_client_secret>',
- args: {
- scope: 'aq:name email~rs address aq:push'
- }
- }
- ```
-
-1. The `scope` is set to request the:
- - User's name.
- - Required and signed email.
- - Permission to send push notifications to sign in on subsequent visits.
-
- See [OmniAuth Authentiq strategy](https://github.com/AuthentiqID/omniauth-authentiq/wiki/Scopes,-callback-url-configuration-and-responses) for more information on scopes and modifiers.
-
-1. Change `<your_client_id>` and `<your_client_secret>` to the Client credentials you received from Authentiq.
-
-1. Save the configuration file.
-
-1. For the changes to take effect:
- - [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) if you installed GitLab using Omnibus.
- - [Restart GitLab](../restart_gitlab.md#installations-from-source) if you installed GitLab from source.
-
-On the sign in page there should now be an Authentiq icon below the regular sign in form. Select the
-icon to begin the authentication process. If the user:
-
-- Has the Authentiq ID app installed in their iOS or Android device, they can:
- 1. Scan the QR code.
- 1. Decide what personal details to share.
- 1. Sign in to your GitLab installation.
-- Does not have the app installed, they are prompted to download the app and then follow the
- previous procedure.
-
-If everything works, the user is returned to GitLab and is signed in.
-
-<!-- ## Troubleshooting
-
-Include any troubleshooting steps that you can foresee. If you know beforehand what issues
-one might have when setting this up, or when something is changed, or on upgrading, it's
-important to describe those, too. Think of things that may go wrong and include them here.
-This is important to minimize requests for support, and to avoid doc comments with
-questions that you know someone might ask.
-
-Each scenario can be a third-level heading, for example `### Getting error message X`.
-If you have none to add when creating a doc, leave this section in place
-but commented out to help encourage others to add to it in the future. -->
+This feature was [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/389452) in 15.9.
+Use another [OmniAuth provider](../../integration/omniauth.md) instead.
diff --git a/doc/administration/auth/cognito.md b/doc/administration/auth/cognito.md
index bb06d5d1a58..a73595c731b 100644
--- a/doc/administration/auth/cognito.md
+++ b/doc/administration/auth/cognito.md
@@ -17,7 +17,7 @@ To enable AWS Cognito as an authentication provider, complete the following step
1. Sign in to the [AWS console](https://console.aws.amazon.com/console/home).
1. From the **Services** menu, select **Cognito**.
-1. Select **Manage User Pools** and then select **Create a user pool** in the top right corner.
+1. Select **Manage User Pools** and then in the upper-right corner, select **Create a user pool**.
1. Enter the user pool name and then select **Step through settings**.
1. Under **How do you want your end users to sign in?**, select **Email address or phone number** and **Allow email addresses**.
1. Under **Which standard attributes do you want to require?**, select **email**.
@@ -33,7 +33,7 @@ To enable AWS Cognito as an authentication provider, complete the following step
- **Enabled Identity Providers** - select all
- **Callback URL** - `https://<your_gitlab_instance_url>/users/auth/cognito/callback`
- **Allowed OAuth Flows** - Authorization code grant
- - **Allowed OAuth2 Scopes** - `email`, `openid`, and `profile`
+ - **Allowed OAuth 2.0 Scopes** - `email`, `openid`, and `profile`
1. Save changes for the app client settings.
1. Under **Domain name**, include the AWS domain name for your AWS Cognito application.
@@ -41,7 +41,9 @@ To enable AWS Cognito as an authentication provider, complete the following step
## Configure GitLab
-1. See [Configure initial settings](../../integration/omniauth.md#configure-initial-settings) for initial settings.
+1. Edit the [common configuration file settings](../../integration/omniauth.md#configure-common-settings)
+ to add `cognito` as a single sign-on provider. This enables Just-In-Time
+ account provisioning for users who do not have an existing GitLab account.
1. On your GitLab server, open the configuration file.
**For Omnibus installations**
@@ -95,4 +97,4 @@ Select this option to begin the authentication process.
AWS Cognito then asks you to sign in and authorize the GitLab application.
If the authorization is successful, you're redirected and signed in to your GitLab instance.
-For more information, see [Configure initial settings](../../integration/omniauth.md#configure-initial-settings).
+For more information, see [Configure common settings](../../integration/omniauth.md#configure-common-settings).
diff --git a/doc/administration/auth/crowd.md b/doc/administration/auth/crowd.md
index 4395309e91e..277e59b1a8a 100644
--- a/doc/administration/auth/crowd.md
+++ b/doc/administration/auth/crowd.md
@@ -40,8 +40,9 @@ this provider also allows Crowd authentication for Git-over-https requests.
sudo -u git -H editor config/gitlab.yml
```
-1. See [Configure initial settings](../../integration/omniauth.md#configure-initial-settings)
- for initial settings.
+1. Edit the [common configuration file settings](../../integration/omniauth.md#configure-common-settings)
+ to add `crowd` as a single sign-on provider. This enables Just-In-Time
+ account provisioning for users who do not have an existing GitLab account.
1. Add the provider configuration:
diff --git a/doc/administration/auth/index.md b/doc/administration/auth/index.md
index 307d9a4518d..2b978dd1f2c 100644
--- a/doc/administration/auth/index.md
+++ b/doc/administration/auth/index.md
@@ -27,10 +27,10 @@ For more information, see the links shown on this page for each external provide
| Capability | SaaS | Self-managed |
|-------------------------------------------------|-----------------------------------------|------------------------------------|
-| **User Provisioning** | SCIM<br>SAML <sup>1</sup> | LDAP <sup>1</sup><br>SAML <sup>1</sup><br>[OmniAuth Providers](../../integration/omniauth.md#supported-providers) <sup>1</sup> |
+| **User Provisioning** | SCIM<br>SAML <sup>1</sup> | LDAP <sup>1</sup><br>SAML <sup>1</sup><br>[OmniAuth Providers](../../integration/omniauth.md#supported-providers) <sup>1</sup><br>SCIM |
| **User Detail Updating** (not group management) | Not Available | LDAP Sync |
-| **Authentication** | SAML at top-level group (1 provider) | LDAP (multiple providers)<br>Generic OAuth2<br>SAML (only 1 permitted per unique provider)<br>Kerberos<br>JWT<br>Smartcard<br>[OmniAuth Providers](../../integration/omniauth.md#supported-providers) (only 1 permitted per unique provider) |
+| **Authentication** | SAML at top-level group (1 provider) | LDAP (multiple providers)<br>Generic OAuth 2.0<br>SAML (only 1 permitted per unique provider)<br>Kerberos<br>JWT<br>Smartcard<br>[OmniAuth Providers](../../integration/omniauth.md#supported-providers) (only 1 permitted per unique provider) |
| **Provider-to-GitLab Role Sync** | SAML Group Sync | LDAP Group Sync<br>SAML Group Sync ([GitLab 15.1](https://gitlab.com/gitlab-org/gitlab/-/issues/285150) and later) |
-| **User Removal** | SCIM (remove user from top-level group) | LDAP (remove user from groups and block from the instance) |
+| **User Removal** | SCIM (remove user from top-level group) | LDAP (remove user from groups and block from the instance)<br>SCIM |
1. Using Just-In-Time (JIT) provisioning, user accounts are created when the user first signs in.
diff --git a/doc/administration/auth/jwt.md b/doc/administration/auth/jwt.md
index c1e76d1c2ed..8c9800aa792 100644
--- a/doc/administration/auth/jwt.md
+++ b/doc/administration/auth/jwt.md
@@ -25,7 +25,9 @@ JWT provides you with a secret key for you to use.
sudo -u git -H editor config/gitlab.yml
```
-1. See [Configure initial settings](../../integration/omniauth.md#configure-initial-settings) for initial settings.
+1. Edit the [common configuration file settings](../../integration/omniauth.md#configure-common-settings)
+ to add `jwt` as a single sign-on provider. This enables Just-In-Time
+ account provisioning for users who do not have an existing GitLab account.
1. Add the provider configuration.
For Omnibus GitLab:
diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md
index 43d13b8ea32..0c7bd33c2c1 100644
--- a/doc/administration/auth/ldap/index.md
+++ b/doc/administration/auth/ldap/index.md
@@ -13,12 +13,13 @@ to support user authentication.
This integration works with most LDAP-compliant directory servers, including:
- Microsoft Active Directory.
- [Microsoft Active Directory Trusts](https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2008-R2-and-2008/cc771568(v=ws.10))
- are not supported.
- Apple Open Directory.
- Open LDAP.
- 389 Server.
+NOTE:
+GitLab does not support [Microsoft Active Directory Trusts](https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2008-R2-and-2008/cc771568(v=ws.10)).
+
Users added through LDAP:
- Usually use a [licensed seat](../../../subscriptions/self_managed/index.md#billable-users).
diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md
index 95064b296af..7e21d3c6655 100644
--- a/doc/administration/auth/ldap/ldap-troubleshooting.md
+++ b/doc/administration/auth/ldap/ldap-troubleshooting.md
@@ -170,13 +170,12 @@ We have a workaround, based on toggling the access level of affected users:
1. As an administrator, on the top bar, select **Main menu > Admin**.
1. On the left sidebar, select **Overview > Users**.
1. Select the name of the affected user.
-1. In the user's administrative page, press **Edit** on the top right of the page.
-1. Change the user's access level from `Regular` to `Admin` (or vice versa),
- and press **Save changes** at the bottom of the page.
-1. Press **Edit** on the top right of the user's profile page
- again.
-1. Restore the user's original access level (`Regular` or `Admin`)
- and press **Save changes** again.
+1. In the upper right, select **Edit**.
+1. Change the user's access level from `Regular` to `Administrator` (or vice versa).
+1. At the bottom of the page, select **Save changes**.
+1. In the upper right, select **Edit** again.
+1. Restore the user's original access level (`Regular` or `Administrator`)
+ and select **Save changes** again.
The user should now be able to sign in.
diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md
index 1f73d8bff38..ea6922629d8 100644
--- a/doc/administration/auth/oidc.md
+++ b/doc/administration/auth/oidc.md
@@ -29,7 +29,9 @@ The OpenID Connect provides you with a client's details and secret for you to us
sudo -u git -H editor config/gitlab.yml
```
-1. [Configure initial settings](../../integration/omniauth.md#configure-initial-settings).
+1. Edit the [common configuration file settings](../../integration/omniauth.md#configure-common-settings)
+ to add `openid_connect` as a single sign-on provider. This enables Just-In-Time
+ account provisioning for users who do not have an existing GitLab account.
1. Add the provider configuration.
@@ -50,6 +52,7 @@ The OpenID Connect provides you with a client's details and secret for you to us
client_auth_method: "query",
uid_field: "<uid_field>",
send_scope_to_token_endpoint: "false",
+ pkce: true,
client_options: {
identifier: "<your_oidc_client_id>",
secret: "<your_oidc_client_secret>",
@@ -75,6 +78,7 @@ The OpenID Connect provides you with a client's details and secret for you to us
client_auth_method: 'query',
uid_field: '<uid_field>',
send_scope_to_token_endpoint: false,
+ pkce: true,
client_options: {
identifier: '<your_oidc_client_id>',
secret: '<your_oidc_client_secret>',
@@ -118,9 +122,10 @@ The OpenID Connect provides you with a client's details and secret for you to us
If you do not provide this value, or the field with the configured value is missing
from the `user_info.raw_attributes` details, `uid` uses the `sub` field.
- `send_scope_to_token_endpoint` is `true` by default, so the `scope` parameter
- is normally included in requests to the token endpoint.
+ is usually included in requests to the token endpoint.
However, if your OpenID Connect provider does not accept the `scope` parameter
in such requests, set this to `false`.
+ - `pkce` (optional): Enable [Proof Key for Code Exchange](https://www.rfc-editor.org/rfc/rfc766). Available in [GitLab 15.9](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/109557).
- `client_options` are the OpenID Connect client-specific options. Specifically:
- `identifier` is the client identifier as configured in the OpenID Connect service provider.
- `secret` is the client secret as configured in the OpenID Connect service provider. For example,
@@ -170,6 +175,7 @@ gitlab_rails['omniauth_providers'] = [
client_auth_method: "query",
discovery: true,
uid_field: "preferred_username",
+ pkce: true,
client_options: {
identifier: "<YOUR PROJECT CLIENT ID>",
secret: "<YOUR PROJECT CLIENT SECRET>",
@@ -207,6 +213,7 @@ gitlab_rails['omniauth_providers'] = [
client_auth_method: "query",
discovery: true,
uid_field: "preferred_username",
+ pkce: true,
client_options: {
identifier: "<YOUR APP CLIENT ID>",
secret: "<YOUR APP CLIENT SECRET>",
@@ -254,7 +261,7 @@ Azure B2C [offers two ways of defining the business logic for logging in a user]
Custom policies are required because standard Azure B2C user flows
[do not send the OpenID `email` claim](https://github.com/MicrosoftDocs/azure-docs/issues/16566).
Therefore, the standard user flows do not work with the
-[`allow_single_sign_on` or `auto_link_user` parameters](../../integration/omniauth.md#configure-initial-settings).
+[`allow_single_sign_on` or `auto_link_user` parameters](../../integration/omniauth.md#configure-common-settings).
With a standard Azure B2C policy, GitLab cannot create a new account or
link to an existing account with an email address.
@@ -339,6 +346,7 @@ but `LocalAccounts` authenticates against local Active Directory accounts. Befor
client_auth_method: "query",
discovery: true,
send_scope_to_token_endpoint: true,
+ pkce: true,
client_options: {
identifier: "<YOUR APP CLIENT ID>",
secret: "<YOUR APP CLIENT SECRET>",
@@ -356,7 +364,7 @@ but `LocalAccounts` authenticates against local Active Directory accounts. Befor
Ensure the payload includes `email` that matches the user's email access.
- After you enable the custom policy, users might see `Invalid username or password`
after they try to sign in. This might be a configuration issue with the `IdentityExperienceFramework`
- app. See [this Microsoft comment](https://learn.microsoft.com/en-us/answers/questions/50355/unable-to-sign-on-using-custom-policy.html?childToView=122370#comment-122370) that suggests you check that the app manifest
+ app. See [this Microsoft comment](https://learn.microsoft.com/en-us/answers/questions/50355/unable-to-sign-on-using-custom-policy?childtoview=122370#comment-122370) that suggests you check that the app manifest
contains these settings:
- `"accessTokenAcceptedVersion": null`
@@ -393,10 +401,11 @@ gitlab_rails['omniauth_providers'] = [
name: "openid_connect",
scope: ["openid", "profile", "email"],
response_type: "code",
- issuer: "https://keycloak.example.com/auth/realms/myrealm",
+ issuer: "https://keycloak.example.com/realms/myrealm",
client_auth_method: "query",
discovery: true,
uid_field: "preferred_username",
+ pkce: true,
client_options: {
identifier: "<YOUR CLIENT ID>",
secret: "<YOUR CLIENT SECRET>",
@@ -477,6 +486,7 @@ To use symmetric key encryption:
discovery: true,
uid_field: "preferred_username",
jwt_secret_base64: "<YOUR BASE64-ENCODED SECRET>",
+ pkce: true,
client_options: {
identifier: "<YOUR CLIENT ID>",
secret: "<YOUR CLIENT SECRET>",
diff --git a/doc/administration/clusters/kas.md b/doc/administration/clusters/kas.md
index 9f0f7e836f7..a7f8f8e712b 100644
--- a/doc/administration/clusters/kas.md
+++ b/doc/administration/clusters/kas.md
@@ -62,7 +62,7 @@ To enable the agent server on multiple nodes:
gitlab_rails['gitlab_kas_enabled'] = true
gitlab_rails['gitlab_kas_external_url'] = 'wss://gitlab.example.com/-/kubernetes-agent/'
gitlab_rails['gitlab_kas_internal_url'] = 'grpc://kas.internal.gitlab.example.com'
- gitlab_rails['gitlab_kas_external_k8s_proxy_url'] = 'https://gitlab.example.com/-/kubernetes-agent/'
+ gitlab_rails['gitlab_kas_external_k8s_proxy_url'] = 'https://gitlab.example.com/-/kubernetes-agent/k8s-proxy/'
```
In this configuration:
diff --git a/doc/administration/compliance.md b/doc/administration/compliance.md
index ee7476ed6d4..592c110b446 100644
--- a/doc/administration/compliance.md
+++ b/doc/administration/compliance.md
@@ -50,7 +50,7 @@ compliance:
- [**Compliance frameworks**](../user/group/compliance_frameworks.md) (for groups): Create a custom
compliance framework at the group level to describe the type of compliance requirements any child project needs to follow.
-- [**Compliance pipelines**](../user/group/compliance_frameworks.md#configure-a-compliance-pipeline) (for groups): Define a
+- [**Compliance pipelines**](../user/group/compliance_frameworks.md#compliance-pipelines) (for groups): Define a
pipeline configuration to run for any projects with a given compliance framework.
## Audit management
diff --git a/doc/administration/dedicated/index.md b/doc/administration/dedicated/index.md
new file mode 100644
index 00000000000..22b8cc13637
--- /dev/null
+++ b/doc/administration/dedicated/index.md
@@ -0,0 +1,140 @@
+---
+stage: SaaS Platforms
+group: GitLab Dedicated
+info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments"
+description: 'Learn how to configure your GitLab Dedicated instance.'
+---
+
+# Configure GitLab Dedicated **(ULTIMATE)**
+
+GitLab Dedicated is a single-tenant SaaS solution, fully managed and hosted by GitLab. For more information about this offering, see the [subscription page](../../subscriptions/gitlab_dedicated/index.md).
+
+The instructions on this page guide you through:
+
+1. Onboarding and initial setup of your GitLab Dedicated instance.
+1. Configuring your GitLab Dedicated instance including enabling and updating the settings for [available functionality](../../subscriptions/gitlab_dedicated/index.md#available-features).
+
+Any functionality in the GitLab application that is not controlled by the SaaS environment can be configured by using the [Admin Panel](../../user/admin_area/index.md).
+
+## Onboarding
+
+To request the creation of a new GitLab Dedicated environment for your organization, you must provide the following information to your account team:
+
+- Expected number of users.
+- Desired primary region: Primary AWS region in which your data is stored (do note the [list of unsupported regions](../../subscriptions/gitlab_dedicated/index.md#aws-regions-not-supported)).
+- Desired secondary region: Secondary AWS region in which your data is stored. This region is used to recover your GitLab Dedicated instance in case of a disaster.
+- Desired backup region: An AWS region where the primary backups of your data are replicated. This can be the same as the primary or secondary region or different.
+- Desired instance subdomain: The main domain for GitLab Dedicated instances is `gitlab-dedicated.com`. You get to choose the subdomain name where your instance is accessible from (for example, `customer_name.gitlab-dedicated.com`).
+- Initial storage: Initial storage size for your repositories in GB.
+- Availability Zone IDs for PrivateLink: If you plan to later add a PrivateLink connection (either [inbound](#inbound-private-link) or [outbound](#outbound-private-link)) to your environment, and you require the connections to be available in specific Availability Zones, you must provide up to two [Availability Zone IDs](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#az-ids) during onboarding. If not specified, GitLab will select two random Availability Zone IDs in which the connections will be available.
+
+### Maintenance window
+
+When onboarding, you must also specify your preference for the weekly four-hour timeslot that GitLab uses to perform maintenance and upgrade operations on the tenant instance.
+
+- APAC (outside working hours): Wednesday 1pm-5pm UTC
+- EU (outside working hours): Tuesday 1am-5am UTC
+- AMER (outside working hours): Tuesday 7am-11am UTC
+
+NOTE:
+Some downtime may be incurred during this window. This downtime is not counting towards [the system SLA](https://about.gitlab.com/handbook/engineering/infrastructure/team/gitlab-dedicated/slas/).
+
+## Configuration changes
+
+To change or update the configuration for your GitLab Dedicated instance, open a [support ticket](https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=4414917877650) with your request. You can request configuration changes for the options originally specified during onboarding, or for any of the optional features below.
+
+The turnaround time for processing configuration change requests is [documented in the GitLab handbook](https://about.gitlab.com/handbook/engineering/infrastructure/team/gitlab-dedicated/#handling-configuration-changes-for-tenant-environments).
+
+### Inbound Private Link
+
+[AWS Private Link](https://docs.aws.amazon.com/vpc/latest/privatelink/what-is-privatelink.html) allows users and applications in your VPC on AWS to securely connect to the GitLab Dedicated endpoint without network traffic going over the public internet.
+
+To enable the Inbound Private Link:
+
+1. In the body of your [support ticket](#configuration-changes), include the IAM Principal for the AWS user or role in your own AWS Organization that will be establishing the VPC endpoint within your AWS account. GitLab Dedicated uses this IAM Principal for access-control. This IAM principal will be the only one able to set up an endpoint to the service.
+1. After your IAM Principal has been allowlisted, GitLab [creates the Endpoint Service](https://docs.aws.amazon.com/vpc/latest/privatelink/create-endpoint-service.html) and communicates the `Service Endpoint Name` on the support ticket. The service name is generated by AWS upon creation of the service endpoint.
+ - GitLab handles the domain verification for the Private DNS name, so that DNS resolution of the tenant instance domain name in your VPC resolves to the PrivateLink endpoint.
+ - GitLab makes the Endpoint Service available in the Availability Zones you specified during the initial onboarding. If you did not specify any Availability Zones, GitLab randomly selects the Availability Zones IDs.
+1. In your own AWS account, create an [Endpoint Interface](https://docs.aws.amazon.com/vpc/latest/privatelink/create-interface-endpoint.html) in your VPC, with the following settings:
+ - Service Endpoint Name: use the name provided by GitLab on the support ticket.
+ - Private DNS names enabled: yes.
+ - Subnets: choose all matching subnets.
+
+1. After you create the endpoint, use the instance URL provided to you during onboarding to securely connect to your GitLab Dedicated instance from your VPC, without the traffic going over the public internet.
+
+### Outbound Private Link
+
+Outbound Private Links allow the GitLab Dedicated instance to securely communicate with services running in your VPC on AWS. This type of connection can execute [webhooks](../../user/project/integrations/webhooks.md) where the targeted services are running in your VPC, import or mirror projects and repositories, or use any other GitLab functionality to access private services.
+
+To enable an Outbound Private Link:
+
+1. In your [support ticket](#configuration-changes), GitLab provides you with an IAM role ARN that connects to your endpoint service. You can then add this ARN to the allowlist on your side to restrict connections to your endpoint service.
+1. [Create the Endpoint service](https://docs.aws.amazon.com/vpc/latest/privatelink/create-endpoint-service.html) through which your internal service will be made available to GitLab Dedicated. Provide the associated `Service Endpoint Name` on the [support ticket](#configuration-changes).
+1. When creating the Endpoint service, you must provide GitLab with a [verified Private DNS Name](https://docs.aws.amazon.com/vpc/latest/privatelink/manage-dns-names.html#verify-domain-ownership) for your service. Optionally, if you would like GitLab Dedicated to reach your service via other aliases, you have the ability to specify a list of Private Hosted Zone (PHZ) entries. With this option, GitLab sets up a Private Hosted Zone with DNS names aliased to the verified Private DNS name. To enable this functionality, you must provide GitLab a list of PHZ entries on your support ticket. After the PHZ has been created in the tenant environment, DNS resolution of any of the provided records will correctly resolve to the PrivateLink endpoint.
+1. GitLab then configures the tenant instance to create the necessary Endpoint Interfaces based on the service names you provided. Any outbound calls made from the tenant GitLab instance are directed through the PrivateLink into your VPC.
+
+#### Custom certificates
+
+In some cases, the GitLab Dedicated instance can't reach an internal service you own because it exposes a certificate that can't be validated using a public Certification Authority (CA). In these cases, custom certificates are required.
+
+To request that GitLab add custom certificates when communicating with your services over PrivateLink, attach the custom public certificate files to your [support ticket](#configuration-changes).
+
+#### Maximum number of reverse PrivateLink connections
+
+GitLab Dedicated limits the number of reverse PrivateLink connections to 10.
+
+### IP allowlist
+
+GitLab Dedicated allows you to control which IP addresses can access your instance through an IP allowlist.
+
+Specify a comma separated list of IP addresses that can access your GitLab Dedicated instance in your [support ticket](#configuration-changes). After the configuration has been applied, when an IP not on the allowlist tries to access your instance, the connection is refused.
+
+### SAML
+
+Prerequisites:
+
+- You must configure the identity provider before sending the required data to GitLab.
+
+To activate SAML for your GitLab Dedicated instance:
+
+1. Read the [GitLab documentation about SAML](../../integration/saml.md#https://docs.gitlab.com/ee/integration/saml.html#configure-saml-on-your-idp) to gather all data your identity provider requires for configuration. You can also find some providers and their requirements in the [group SAML documentation](../../user/group/saml_sso/index.md#providers).
+1. To make the necessary changes, include in your [support ticket](#configuration-changes) the desired [SAML configuration block](../../integration/saml.md#configure-saml-support-in-gitlab) that will be set on the GitLab application. At a minimum, GitLab needs the following information to enable SAML for your instance:
+ - Assertion consumer service URL
+ - Certificate fingerprint or certificate
+ - NameID format
+ - SSO login button description
+
+1. After GitLab deploys the SAML configuration to your instance, you will be notified on your support ticket.
+1. To verify the SAML configuration is successful:
+ - Check that the SSO login button description is displayed on your instance's login page.
+ - Go to the metadata URL of your instance (`https://INSTANCE-URL/users/auth/saml/metadata`). This page can be used to simplify much of the configuration of the identity provider, as well as manually validate the settings.
+
+#### Request signing
+
+If [SAML request signing](../../integration/saml.md#sign-saml-authentication-requests-optional) is desired, a certificate must be obtained. This certificate can be self-signed which has the advantage of not having to prove ownership of an arbitrary Common Name (CN) to a public Certificate Authority (CA)).
+
+To enable SAML request signing, indicate on your SAML [support ticket](#configuration-changes) that you would like request signing enabled. GitLab will then work with you on sending the Certificate Signing Request (CSR) for you to sign. Alternatively, the CSR can be signed with a public CA. After the certificate is signed, GitLab will add the certificate and its associated private key to the `security` section of the SAML configuration. Authentication requests from GitLab to your identity provider will then be signed.
+
+#### SAML groups
+
+With SAML groups you can configure GitLab users based on SAML group membership.
+
+To enable SAML groups, add the [required elements](../../integration/saml.md#configure-users-based-on-saml-group-membership) to the SAML configuration block you provide in your [support ticket](#configuration-changes).
+
+#### Group sync
+
+With [group sync](../../user/group/saml_sso/group_sync.md), you can sync users across identity provider groups to mapped groups in GitLab.
+
+To enable group sync:
+
+1. Add the [required elements](../../user/group/saml_sso/group_sync.md#configure-saml-group-sync) to the SAML configuration block you provide in your [support ticket](#configuration-changes).
+1. Configure the [Group Links](../../user/group/saml_sso/group_sync.md#configure-saml-group-links).
+
+### Access to application logs
+
+GitLab [application logs](../../administration/logs/index.md) are delivered to an S3 bucket in the GitLab tenant account, which can be shared with you.
+
+To gain read only access to this bucket:
+
+1. Open a [support ticket](#configuration-changes) with the title "Customer Log Access". In the body of the ticket, include a list of IAM Principal ARNs (users or roles) that will be fetching the logs from S3.
+1. GitLab will then inform you of the name of the S3 bucket. Your nominated users/roles will then be able to list and get all objects in the S3 bucket.
diff --git a/doc/administration/docs_self_host.md b/doc/administration/docs_self_host.md
index f049dafea76..97eff35da91 100644
--- a/doc/administration/docs_self_host.md
+++ b/doc/administration/docs_self_host.md
@@ -177,7 +177,7 @@ documentation URL requests as needed. For example, if your GitLab version is
- When you select the link, you are redirected to
`http://0.0.0.0:4000/14.5/ee/user/admin_area/settings/help_page/#destination-requirements`.
-To test the setting, select a **Learn more** link within the GitLab application.
+To test the setting, select a **Learn more** link in the GitLab application.
## Upgrade the product documentation to a later version
diff --git a/doc/administration/encrypted_configuration.md b/doc/administration/encrypted_configuration.md
index 648f6d7018e..1ddf2951f70 100644
--- a/doc/administration/encrypted_configuration.md
+++ b/doc/administration/encrypted_configuration.md
@@ -11,7 +11,9 @@ type: reference
GitLab can read settings for certain features from encrypted settings files. The supported features are:
+- [Incoming email `user` and `password`](incoming_email.md#use-encrypted-credentials).
- [LDAP `bind_dn` and `password`](auth/ldap/index.md#use-encrypted-credentials).
+- [Service Desk email `user` and `password`](../user/project/service_desk.md#use-encrypted-credentials).
- [SMTP `user_name` and `password`](raketasks/smtp.md#secrets).
To enable the encrypted configuration settings, a new base key must be generated for
diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md
index b6703e8a5fb..24a91a7a9c5 100644
--- a/doc/administration/geo/disaster_recovery/index.md
+++ b/doc/administration/geo/disaster_recovery/index.md
@@ -479,25 +479,24 @@ changing Git remotes and API URLs.
external_url 'https://<new_external_url>'
```
- If you provide GitLab with its certificate
- [manually](https://docs.gitlab.com/omnibus/settings/ssl/index.html#configure-https-manually),
- ensure:
+ NOTE:
+ Changing `external_url` does not prevent access via the old secondary URL, as
+ long as the secondary DNS records are still intact.
- - The new URL is one of the subject alternative names:
+1. Update the **secondary**'s SSL certificate:
+
+ - If you use the [Let's Encrypt integration](https://docs.gitlab.com/omnibus/settings/ssl/index.html#enable-the-lets-encrypt-integration),
+ the certificate updates automatically.
+ - If you had [manually set up](https://docs.gitlab.com/omnibus/settings/ssl/index.html#configure-https-manually),
+ the **secondary**'s certificate, copy the certificate from the **primary** to the **secondary**.
+ If you don't have access to the **primary**, issue a new certificate and make sure it contains
+ both the **primary** and **secondary** URLs in the subject alternative names. You can check with:
```shell
/opt/gitlab/embedded/bin/openssl x509 -noout -dates -subject -issuer \
-nameopt multiline -ext subjectAltName -in /etc/gitlab/ssl/new-gitlab.new-example.com.crt
```
- - The certificate and key filenames match the new `external_url`,
- or those filenames are
- [specified in `/etc/gitlab/gitlab.rb`](https://docs.gitlab.com/omnibus/settings/ssl/index.html#change-the-default-ssl-certificate-location).
-
- NOTE:
- Changing `external_url` does not prevent access via the old secondary URL, as
- long as the secondary DNS records are still intact.
-
1. Reconfigure the **secondary** site for the change to take effect:
```shell
diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md
index 8728ab57bba..3f98f1e12fe 100644
--- a/doc/administration/geo/index.md
+++ b/doc/administration/geo/index.md
@@ -200,6 +200,7 @@ This list of limitations only reflects the latest version of GitLab. If you are
- [Pages access control](../../user/project/pages/pages_access_control.md) doesn't work on secondaries. See [GitLab issue #9336](https://gitlab.com/gitlab-org/gitlab/-/issues/9336) for details.
- [GitLab chart with Geo](https://docs.gitlab.com/charts/advanced/geo/) does not support [Unified URLs](secondary_proxy/index.md#set-up-a-unified-url-for-geo-sites). See [GitLab issue #3522](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/3522) for more details.
- [Disaster recovery](disaster_recovery/index.md) for multi-secondary sites causes downtime due to the complete re-synchronization and re-configuration of all non-promoted secondaries.
+- For Git over SSH, secondary sites must use the same port as the primary. [GitLab issue #339262](https://gitlab.com/gitlab-org/gitlab/-/issues/339262) proposes to remove this limitation.
### Limitations on replication/verification
@@ -219,6 +220,8 @@ An [epic exists](https://gitlab.com/groups/gitlab-org/-/epics/4624) to fix this
Keep in mind that mentioned URLs don't work when [Admin Mode](../../user/admin_area/settings/sign_in_restrictions.md#admin-mode) is enabled.
+When using Unified URL, visiting the secondary site directly means you must route your requests to the secondary site. Exactly how this might be done depends on your networking configuration. If using DNS to route requests to the appropriate site, then you can, for example, edit your local machine's `/etc/hosts` file to route your requests to the desired secondary site. If the Geo sites are all behind a load balancer, then depending on the load balancer, you might be able to configure all requests from your IP to go to a particular secondary site.
+
## Setup instructions
For setup instructions, see [Setting up Geo](setup/index.md).
diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md
index 79a3f65377b..912de360e43 100644
--- a/doc/administration/geo/replication/configuration.md
+++ b/doc/administration/geo/replication/configuration.md
@@ -366,7 +366,7 @@ former is ideal for replicating data belonging to a subset of users, while the
latter is more suited to progressively rolling out Geo to a large GitLab
instance.
-It is important to note that selective synchronization:
+Selective synchronization:
1. Does not restrict permissions from **secondary** sites.
1. Does not hide project metadata from **secondary** sites.
diff --git a/doc/administration/geo/replication/container_registry.md b/doc/administration/geo/replication/container_registry.md
index abf34efa56e..88ca8781dc3 100644
--- a/doc/administration/geo/replication/container_registry.md
+++ b/doc/administration/geo/replication/container_registry.md
@@ -7,7 +7,12 @@ type: howto
# Container Registry for a secondary site **(PREMIUM SELF)**
-You can set up a Container Registry on your **secondary** Geo site that mirrors the one on the **primary** Geo site.
+You can set up a Container Registry on your **secondary** Geo site that mirrors the one on the **primary** Geo site.
+
+NOTE:
+The Container Registry replication is used only for disaster recovery purposes. We do not recommend
+pulling the Container Registry data from the secondary. For a feature proposal to implement it in the
+future, see [Geo: Accelerate container images by serving read request from secondary site](https://gitlab.com/gitlab-org/gitlab/-/issues/365864) for details.
## Supported container registries
diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md
index 26acab510cf..27e1b990b2b 100644
--- a/doc/administration/geo/replication/datatypes.md
+++ b/doc/administration/geo/replication/datatypes.md
@@ -87,7 +87,7 @@ GitLab does not require a special file system and can work with:
Geo triggers garbage collection in Gitaly to [deduplicate forked repositories](../../../development/git_object_deduplication.md#git-object-deduplication-and-gitlab-geo) on Geo secondary sites.
-Communication is done via Gitaly's own gRPC API, with three possible ways of synchronization:
+The Gitaly gRPC API does the communication, with three possible ways of synchronization:
- Using regular Git clone/fetch from one Geo site to another (with special authentication).
- Using repository snapshots (for when the first method fails or repository is corrupt).
@@ -151,10 +151,8 @@ and verification status on a **secondary** site.
You can keep track of the progress to implement the missing items in
these epics/issues:
-- [Geo: Build a scalable, self-service Geo replication and verification framework](https://gitlab.com/groups/gitlab-org/-/epics/2161)
- [Geo: Improve the self-service Geo replication framework](https://gitlab.com/groups/gitlab-org/-/epics/3761)
- [Geo: Move existing blobs to framework](https://gitlab.com/groups/gitlab-org/-/epics/3588)
-- [Geo: Add unreplicated data types](https://gitlab.com/groups/gitlab-org/-/epics/893)
### Replicated data types behind a feature flag
diff --git a/doc/administration/geo/replication/multiple_servers.md b/doc/administration/geo/replication/multiple_servers.md
index 5be7d40890c..155fefb2dbf 100644
--- a/doc/administration/geo/replication/multiple_servers.md
+++ b/doc/administration/geo/replication/multiple_servers.md
@@ -81,7 +81,7 @@ After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus
NOTE:
PostgreSQL and Redis should have already been disabled on the
-application nodes during normal GitLab multi-node setup. Connections
+application nodes during typical GitLab multi-node setup. Connections
from the application nodes to services on the backend nodes should
have also been configured. See multi-node configuration documentation for
[PostgreSQL](../../postgresql/replication_and_failover.md#configuring-the-application-nodes)
@@ -100,7 +100,7 @@ major differences:
- There is an additional GitLab service [`geo-logcursor`](../index.md#geo-log-cursor)
Therefore, we set up the multi-node components one by one and include deviations
-from the normal multi-node setup. However, we highly recommend configuring a
+from the typical multi-node setup. However, we highly recommend configuring a
brand-new GitLab site first, as if it were not part of a Geo setup. This allows
verifying that it is a working GitLab site. And only then should it be modified
for use as a Geo **secondary** site. This helps to separate Geo setup problems from
diff --git a/doc/administration/geo/replication/security_review.md b/doc/administration/geo/replication/security_review.md
index afe831dcb9c..92eff2faf60 100644
--- a/doc/administration/geo/replication/security_review.md
+++ b/doc/administration/geo/replication/security_review.md
@@ -114,7 +114,7 @@ from [owasp.org](https://owasp.org/).
### What private and public network links support the application?
- Customers choose their own networks. As sites are intended to be
- geographically separated, it is envisioned that replication traffic will pass
+ geographically separated, it is envisioned that replication traffic passes
over the public Internet in a typical deployment, but this is not a requirement.
## Systems
@@ -168,7 +168,7 @@ from [owasp.org](https://owasp.org/).
- GitLab is "cloud native" and this applies to Geo as much as to the rest of the
product. Deployment in clouds is a common and supported scenario.
-## If applicable, what approach(es) to cloud computing will be taken (Managed Hosting versus "Pure" Cloud, a "full machine" approach such as AWS-EC2 versus a "hosted database" approach such as AWS-RDS and Azure, etc)?
+## If applicable, what approach(es) to cloud computing is taken (Managed Hosting versus "Pure" Cloud, a "full machine" approach such as AWS-EC2 versus a "hosted database" approach such as AWS-RDS and Azure, etc)?
- To be decided by our customers, according to their operational needs.
@@ -186,7 +186,7 @@ from [owasp.org](https://owasp.org/).
- PostgreSQL >= 12, Redis, Sidekiq, Puma.
-### How will database connection strings, encryption keys, and other sensitive components be stored, accessed, and protected from unauthorized detection?
+### How can database connection strings, encryption keys, and other sensitive components be stored, accessed, and protected from unauthorized detection?
- There are some Geo-specific values. Some are shared secrets which must be
securely transmitted from the **primary** site to the **secondary** site at setup time. Our
diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md
index 2f2759d7339..403f8525d39 100644
--- a/doc/administration/geo/replication/troubleshooting.md
+++ b/doc/administration/geo/replication/troubleshooting.md
@@ -38,9 +38,28 @@ to help identify if something is wrong:
- Is the secondary site's tracking database configured?
- Is the secondary site's tracking database connected?
- Is the secondary site's tracking database up-to-date?
+- Is the secondary site's status less than 10 minutes old?
![Geo health check](img/geo_site_health_v14_0.png)
+A site shows as "Unhealthy" if the site's status is more than 10 minutes old. In that case, try running the following in the [Rails console](../../operations/rails_console.md) on the affected site:
+
+```ruby
+Geo::MetricsUpdateWorker.new.perform
+```
+
+If it raises an error, then the error is probably also preventing the jobs from completing. If it takes longer than 10 minutes, then there may be a performance issue, and the UI may always show "Unhealthy" even if the status eventually does get updated.
+
+If it successfully updates the status, then something may be wrong with Sidekiq. Is it running? Do the logs show errors? This job is supposed to be enqueued every minute. It takes an exclusive lease in Redis to ensure that only one of these jobs can run at a time. The primary site updates its status directly in the PostgreSQL database. Secondary sites send an HTTP Post request to the primary site with their status data.
+
+A site also shows as "Unhealthy" if certain health checks fail. You can reveal the failure by running the following in the [Rails console](../../operations/rails_console.md) on the affected site:
+
+```ruby
+Gitlab::Geo::HealthCheck.new.perform_checks
+```
+
+If it returns `""` (an empty string) or `"Healthy"`, then the checks succeeded. If it returns anything else, then the message should explain what failed, or show the exception message.
+
For information about how to resolve common error messages reported from the user interface,
see [Fixing Common Errors](#fixing-common-errors).
@@ -66,7 +85,7 @@ Checking Geo ...
GitLab Geo is available ... yes
GitLab Geo is enabled ... yes
This machine's Geo node name matches a database record ... yes, found a secondary node named "Shanghai"
-GitLab Geo secondary database is correctly configured ... yes
+GitLab Geo tracking database is correctly configured ... yes
Database replication enabled? ... yes
Database replication working? ... yes
GitLab Geo HTTP(S) connectivity ...
@@ -147,11 +166,27 @@ http://secondary.example.com/
Last status report was: 1 minute ago
```
+There are up to three statuses for each item. For example, for `Repositories`, you see the following lines:
+
+```plaintext
+ Repositories: succeeded 12345 / total 12345 (100%)
+ Verified Repositories: succeeded 12345 / total 12345 (100%)
+ Repositories Checked: failed 5 / succeeded 0 / total 5 (0%)
+```
+
+The 3 status items are defined as follows:
+
+- The `Repositories` output shows how many repositories are synced from the primary to the secondary.
+- The `Verified Repositories` output shows how many repositories on this secondary have a matching repository checksum with the Primary.
+- The `Repositories Checked` output shows how many repositories have passed a local Git repository check (`git fsck`) on the secondary.
+
To find more details about failed items, check
[the `gitlab-rails/geo.log` file](../../logs/log_parsing.md#find-most-common-geo-sync-errors)
If you notice replication or verification failures, you can try to [resolve them](#fixing-non-postgresql-replication-failures).
+If there are Repository check failures, you can try to [resolve them](#find-repository-check-failures-in-a-geo-secondary-site).
+
### Check if PostgreSQL replication is working
To check if PostgreSQL replication is working, check if:
@@ -203,7 +238,7 @@ This machine's Geo node name matches a database record ... no
doc/administration/geo/replication/troubleshooting.md#can-geo-detect-the-current-node-correctly
```
-Learn more about recommended site names in the description of the Name field in
+For more information about recommended site names in the description of the Name field, see
[Geo Admin Area Common Settings](../../../user/admin_area/geo_sites.md#common-settings).
### Reverify all uploads (or any SSF data type which is verified)
@@ -221,7 +256,7 @@ Commands that change data can cause damage if not run correctly or under the rig
end
```
-1. This will cause the primary to start checksumming all Uploads.
+1. This causes the primary to start checksumming all Uploads.
1. When a primary successfully checksums a record, then all secondaries recalculate the checksum as well, and they compare the values.
A similar thing can be done for all Models handled by the [Geo Self-Service Framework](../../../development/geo/framework.md) which have implemented verification:
@@ -325,7 +360,7 @@ sudo gitlab-rake gitlab:geo:check
GitLab Geo is available ... yes
GitLab Geo is enabled ... yes
- GitLab Geo secondary database is correctly configured ... not a secondary node
+ GitLab Geo tracking database is correctly configured ... not a secondary node
Database replication enabled? ... not a secondary node
...
Checking Geo ... Finished
@@ -364,15 +399,35 @@ sudo gitlab-rake gitlab:geo:check
Checking Geo ... Finished
```
- When performing a PostgreSQL major version (9 > 10) update this is expected. Follow
+ When performing a PostgreSQL major version (9 > 10), update this is expected. Follow
the [initiate-the-replication-process](../setup/database.md#step-3-initiate-the-replication-process).
+- Rails does not appear to have the configuration necessary to connect to the Geo tracking database.
+
+ ```plaintext
+ Checking Geo ...
+
+ GitLab Geo is available ... yes
+ GitLab Geo is enabled ... yes
+ GitLab Geo tracking database is correctly configured ... no
+ Try fixing it:
+ Rails does not appear to have the configuration necessary to connect to the Geo tracking database. If the tracking database is running on a node other than this one, then you may need to add configuration.
+ ...
+ Checking Geo ... Finished
+ ```
+
+ - If you are running the secondary site on a single node for all services, then follow [Geo database replication - Configure the secondary server](../setup/database.md#step-2-configure-the-secondary-server).
+ - If you are running the secondary site's tracking database on its own node, then follow [Geo for multiple servers - Configure the Geo tracking database on the Geo secondary site](multiple_servers.md#step-3-configure-the-geo-tracking-database-on-the-geo-secondary-site)
+ - If you are running the secondary site's tracking database in a Patroni cluster, then follow [Geo database replication - Configure the tracking database on the secondary sites](../setup/database.md#step-3-configure-the-tracking-database-on-the-secondary-sites)
+ - If you are running the secondary site's tracking database in an external database, then follow [Geo with external PostgreSQL instances](../setup/external_database.md#configure-the-tracking-database)
+ - If the Geo check task was run on a node which is not running a service which runs the GitLab Rails app (Puma, Sidekiq, or Geo Log Cursor), then this error can be ignored. The node does not need Rails to be configured.
+
### Message: Machine clock is synchronized ... Exception
The Rake task attempts to verify that the server clock is synchronized with NTP. Synchronized clocks
are required for Geo to function correctly. As an example, for security, when the server time on the
primary site and secondary site differ by about a minute or more, requests between Geo sites
-will fail. If this check task fails to complete due to a reason other than mismatching times, it
+fail. If this check task fails to complete due to a reason other than mismatching times, it
does not necessarily mean that Geo will not work.
The Ruby gem which performs the check is hard coded with `pool.ntp.org` as its reference time source.
@@ -385,13 +440,13 @@ The Ruby gem which performs the check is hard coded with `pool.ntp.org` as its r
This issue occurs when the hostname `pool.ntp.org` resolves to a server which does not provide a time service.
-There is [an issue open](https://gitlab.com/gitlab-org/gitlab/-/issues/381422) for this dependency on `pool.ntp.org`.
+In this case, in GitLab 15.7 and newer, [specify a custom NTP server using environment variables](#health-check-rake-task).
-To workaround this, do one of the following:
+In GitLab 15.6 and older, use one of the following workarounds:
- Add entries in `/etc/hosts` for `pool.ntp.org` to direct the request to valid local time servers.
This fixes the long timeout and the timeout error.
-- Direct the check to any valid IP address. This resolves the timeout issue, but the check will fail
+- Direct the check to any valid IP address. This resolves the timeout issue, but the check fails
with the `No route to host` error, as noted above.
[Cloud native GitLab deployments](https://docs.gitlab.com/charts/advanced/geo/#set-the-geo-primary-site)
@@ -401,6 +456,39 @@ generate an error because containers in Kubernetes do not have access to the hos
Machine clock is synchronized ... Exception: getaddrinfo: Servname not supported for ai_socktype
```
+### Message: `ActiveRecord::StatementInvalid: PG::ReadOnlySqlTransaction: ERROR: cannot execute INSERT in a read-only transaction`
+
+When this error is encountered on a secondary site, it likely affects all usages of GitLab Rails such as `gitlab-rails` or `gitlab-rake` commands, as well the Puma, Sidekiq, and Geo Log Cursor services.
+
+```plaintext
+ActiveRecord::StatementInvalid: PG::ReadOnlySqlTransaction: ERROR: cannot execute INSERT in a read-only transaction
+/opt/gitlab/embedded/service/gitlab-rails/app/models/application_record.rb:86:in `block in safe_find_or_create_by'
+/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cross_database_modification.rb:92:in `block in transaction'
+/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:332:in `block in transaction'
+/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:331:in `transaction'
+/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cross_database_modification.rb:83:in `transaction'
+/opt/gitlab/embedded/service/gitlab-rails/app/models/application_record.rb:86:in `safe_find_or_create_by'
+/opt/gitlab/embedded/service/gitlab-rails/app/models/shard.rb:21:in `by_name'
+/opt/gitlab/embedded/service/gitlab-rails/app/models/shard.rb:17:in `block in populate!'
+/opt/gitlab/embedded/service/gitlab-rails/app/models/shard.rb:17:in `map'
+/opt/gitlab/embedded/service/gitlab-rails/app/models/shard.rb:17:in `populate!'
+/opt/gitlab/embedded/service/gitlab-rails/config/initializers/fill_shards.rb:9:in `<top (required)>'
+/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
+/opt/gitlab/embedded/bin/bundle:23:in `load'
+/opt/gitlab/embedded/bin/bundle:23:in `<main>'
+```
+
+The PostgreSQL read-replica database would be producing these errors:
+
+```plaintext
+2023-01-17_17:44:54.64268 ERROR: cannot execute INSERT in a read-only transaction
+2023-01-17_17:44:54.64271 STATEMENT: /*application:web,db_config_name:main*/ INSERT INTO "shards" ("name") VALUES ('storage1') RETURNING "id"
+```
+
+This situation can occur during initial configuration when a secondary site is not yet aware that it is a secondary site.
+
+To resolve the error, follow [Step 3. Add the secondary site](configuration.md#step-3-add-the-secondary-site).
+
## Fixing PostgreSQL database replication errors
The following sections outline troubleshooting steps for fixing replication
@@ -477,7 +565,7 @@ Slots where `active` is `f` are not active.
### Message: "ERROR: canceling statement due to conflict with recovery"
-This error message occurs infrequently under normal usage, and the system is resilient
+This error message occurs infrequently under typical usage, and the system is resilient
enough to recover.
However, under certain conditions, some database queries on secondaries may run
@@ -1256,11 +1344,39 @@ To fix this issue, set the primary site's internal URL to a URL that is:
GeoNode.where(primary: true).first.update!(internal_url: "https://unique.url.for.primary.site")
```
+### Secondary site returns `Received HTTP code 403 from proxy after CONNECT`
+
+If you have installed GitLab using the Linux package (Omnibus) and have configured the `no_proxy` [custom environment variable](https://docs.gitlab.com/omnibus/settings/environment-variables.html) for Gitaly, you may experience this issue. Affected versions:
+
+- `15.4.6`
+- `15.5.0`-`15.5.6`
+- `15.6.0`-`15.6.3`
+- `15.7.0`-`15.7.1`
+
+This is due to [a bug introduced in the included version of cURL](https://github.com/curl/curl/issues/10122) shipped with Omnibus GitLab 15.4.6 and later. You are encouraged to upgrade to a later version where this has been [fixed](https://about.gitlab.com/releases/2023/01/09/security-release-gitlab-15-7-2-released/).
+
+The bug causes all wildcard domains (`.example.com`) to be ignored except for the last on in the `no_proxy` environment variable list. Therefore, if for any reason you cannot upgrade to a newer version, you can work around the issue by moving your wildcard domain to the end of the list:
+
+1. Edit `/etc/gitlab/gitlab.rb`:
+
+ ```ruby
+ gitaly['env'] = {
+ "no_proxy" => "sever.yourdomain.org, .yourdomain.com",
+ }
+
+1. Reconfigure GitLab:
+
+ ```shell
+ sudo gitlab-ctl reconfigure
+ ```
+
+You can have only one wildcard domain in the `no_proxy` list.
+
## Fixing non-PostgreSQL replication failures
If you notice replication failures in `Admin > Geo > Sites` or the [Sync status Rake task](#sync-status-rake-task), you can try to resolve the failures with the following general steps:
-1. Geo will automatically retry failures. If the failures are new and few in number, or if you suspect the root cause is already resolved, then you can wait to see if the failures go away.
+1. Geo automatically retries failures. If the failures are new and few in number, or if you suspect the root cause is already resolved, then you can wait to see if the failures go away.
1. If failures were present for a long time, then many retries have already occurred, and the interval between automatic retries has increased to up to 4 hours depending on the type of failure. If you suspect the root cause is already resolved, you can [manually retry replication or verification](#manually-retry-replication-or-verification).
1. If the failures persist, use the following sections to try to resolve them.
@@ -1368,7 +1484,7 @@ status
end
```
-1. This will cause the primary to start checksumming all Uploads.
+1. This causes the primary to start checksumming all Uploads.
1. When a primary successfully checksums a record, then all secondaries recalculate the checksum as well, and they compare the values.
For other SSF data types replace `Upload` in the command above with the desired model class.
@@ -1464,11 +1580,54 @@ project = Project.find_by_full_path('<group/project>')
Geo::RepositorySyncService.new(project).execute
```
+#### Find repository check failures in a Geo secondary site
+
+When [enabled for all projects](../../repository_checks.md#enable-repository-checks-for-all-projects), [Repository checks](../../repository_checks.md) are also performed on Geo secondary sites. The metadata is stored in the Geo tracking database.
+
+Repository check failures on a Geo secondary site do not necessarily imply a replication problem. Here is a general approach to resolve these failures.
+
+1. Find affected repositories as mentioned below, as well as their [logged errors](../../repository_checks.md#what-to-do-if-a-check-failed).
+1. Try to diagnose specific `git fsck` errors. The range of possible errors is wide, try putting them into search engines.
+1. Test normal functions of the affected repositories. Pull from the secondary, view the files.
+1. Check if the primary site's copy of the repository has an identical `git fsck` error. If you are planning a failover, then consider prioritizing that the secondary site has the same information that the primary site has. Ensure you have a backup of the primary, and follow [planned failover guidelines](../disaster_recovery/planned_failover.md).
+1. Push to the primary and check if the change gets replicated to the secondary site.
+1. If replication is not automatically working, try to manually sync the repository.
+
+[Start a Rails console session](../../operations/rails_console.md#starting-a-rails-console-session)
+to enact the following, basic troubleshooting steps.
+
+WARNING:
+Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore.
+
+##### Get the number of repositories that failed the repository check
+
+```ruby
+Geo::ProjectRegistry.where(last_repository_check_failed: true).count
+```
+
+##### Find the repositories that failed the repository check
+
+```ruby
+Geo::ProjectRegistry.where(last_repository_check_failed: true)
+```
+
+##### Recheck repositories that failed the repository check
+
+When you run this, `fsck` is executed against each failed repository.
+
+The [`fsck` Rake command](../../raketasks/check.md#check-project-code-repositories) can be used on the secondary site to understand why the repository check might be failing.
+
+```ruby
+Geo::ProjectRegistry.where(last_repository_check_failed: true).each do |pr|
+ RepositoryCheck::SingleRepositoryWorker.new.perform(pr.project_id)
+end
+```
+
## Fixing client errors
### Authorization errors from LFS HTTP(S) client requests
-You may have problems if you're running a version of [Git LFS](https://git-lfs.github.com/) before 2.4.2.
+You may have problems if you're running a version of [Git LFS](https://git-lfs.com/) before 2.4.2.
As noted in [this authentication issue](https://github.com/git-lfs/git-lfs/issues/3025),
requests redirected from the secondary to the primary site do not properly send the
Authorization header. This may result in either an infinite `Authorization <-> Redirect`
@@ -1549,7 +1708,7 @@ On all hosts running PostgreSQL, across all Geo sites, run the following shell c
( echo "1-1"; echo "11" ) | LC_COLLATE=en_US.UTF-8 sort
```
-The output will either look like:
+The output looks like either:
```plaintext
1-1
diff --git a/doc/administration/geo/replication/version_specific_upgrades.md b/doc/administration/geo/replication/version_specific_upgrades.md
index aa8e5d77f67..f8e013a8776 100644
--- a/doc/administration/geo/replication/version_specific_upgrades.md
+++ b/doc/administration/geo/replication/version_specific_upgrades.md
@@ -8,10 +8,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w
NOTE:
We're in the process of merging all the version-specific upgrade information
-into a single page. See [epic 9581](https://gitlab.com/groups/gitlab-org/-/epics/9581)
-for more information. For the time being, visit the
-[general upgrade page](../../../update/index.md)
-for the newest Geo version-specific upgrade instructions.
+into a single page. For more information,
+see [epic 9581](https://gitlab.com/groups/gitlab-org/-/epics/9581).
+For the latest Geo version-specific upgrade instructions,
+see the [general upgrade page](../../../update/index.md).
Review this page for upgrade instructions for your version. These steps
accompany the [general steps](upgrading_the_geo_sites.md#general-upgrade-steps)
diff --git a/doc/administration/geo/secondary_proxy/index.md b/doc/administration/geo/secondary_proxy/index.md
index ac8b88a91d5..addf894f0a5 100644
--- a/doc/administration/geo/secondary_proxy/index.md
+++ b/doc/administration/geo/secondary_proxy/index.md
@@ -32,6 +32,9 @@ Use secondary proxying for use-cases including:
- Having all Geo sites behind a single URL.
- Geographically load-balancing traffic without worrying about write access.
+NOTE:
+Geo proxying for secondary sites is separate from Geo proxying/redirecting for Git push operations.
+
<i class="fa fa-youtube-play youtube" aria-hidden="true"></i>
For an overview, see: [Secondary proxying using geographic load-balancer and AWS Route53](https://www.youtube.com/watch?v=TALLy7__Na8).
@@ -122,9 +125,9 @@ for details.
- [Viewing projects and designs data from a primary site is not possible when using a unified URL](../index.md#view-replication-data-on-the-primary-site).
- When secondary proxying is used together with separate URLs, registering [GitLab runners](https://docs.gitlab.com/runner/) to clone from
-secondary sites is not supported. The runner registration will succeed, but the clone URL will default to the primary site. The runner
+secondary sites is not supported. The runner registration succeeds, but the clone URL defaults to the primary site. The runner
[clone URL](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) is configured per GitLab deployment
-and cannot be configured per Geo site. Therefore, all runners will clone from the primary site (or configured clone URL) irrespective of
+and cannot be configured per Geo site. Therefore, all runners clone from the primary site (or configured clone URL) irrespective of
which Geo site they register on. For information about GitLab CI using a specific Geo secondary to clone from, see issue
[3294](https://gitlab.com/gitlab-org/gitlab/-/issues/3294#note_1009488466).
@@ -147,7 +150,7 @@ secondary Geo sites are able to support write requests. Certain **read** request
sites for improved latency and bandwidth nearby. All write requests are proxied to the primary site.
The following table details the components currently tested through the Geo secondary site Workhorse proxy.
-It does not cover all data types, more will be added in the future as they are tested.
+It does not cover all data types.
| Feature / component | Accelerated reads? |
|:----------------------------------------------------|:-----------------------|
diff --git a/doc/administration/geo/secondary_proxy/location_aware_external_url.md b/doc/administration/geo/secondary_proxy/location_aware_external_url.md
index b71b813ee9f..ae701bb8482 100644
--- a/doc/administration/geo/secondary_proxy/location_aware_external_url.md
+++ b/doc/administration/geo/secondary_proxy/location_aware_external_url.md
@@ -86,7 +86,7 @@ When creating Geo-Based record sets, GCP applies a nearest match for the source
1. Select **Network Services** > **Cloud DNS**.
1. Select the Zone configured for your domain.
1. Select **Add Record Set**.
-1. Enter the DNS Name for your Location-aware public URL e.g. `gitlab.example.com`.
+1. Enter the DNS Name for your Location-aware public URL, for example, `gitlab.example.com`.
1. Select the **Routing Policy**: **Geo-Based**.
1. Select **Add Managed RRData**.
1. Select **Source Region**: **us-central1**.
diff --git a/doc/administration/get_started.md b/doc/administration/get_started.md
index 2b9b5291c54..d9191440a24 100644
--- a/doc/administration/get_started.md
+++ b/doc/administration/get_started.md
@@ -1,5 +1,5 @@
---
-info: For assistance with this TAM Onboarding page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-other-projects-and-subjects.
+info: For assistance with this CSM Onboarding page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-other-projects-and-subjects.
stage: none
group: unassigned
---
@@ -45,11 +45,11 @@ Get started:
**More resources**
-- Learn more about [running multiple Agile teams](https://www.youtube.com/watch?v=VR2r1TJCDew).
-- Sync group memberships [by using LDAP](../administration/auth/ldap/ldap_synchronization.md#group-sync).
+- [Run multiple Agile teams](https://www.youtube.com/watch?v=VR2r1TJCDew).
+- [Sync group memberships by using LDAP](../administration/auth/ldap/ldap_synchronization.md#group-sync).
- Manage user access with inherited permissions. Use up to 20 levels of subgroups to organize both teams and projects.
- - Learn more about [inherited permissions](../user/project/members/index.md#inherited-membership).
- - View an [example](../user/group/subgroups/index.md).
+ - [Inherited membership](../user/project/members/index.md#inherited-membership).
+ - [Example](../user/group/subgroups/index.md).
## Import projects
@@ -241,7 +241,7 @@ You can make changes to your default rate limits from the Admin Area. For more i
- Review the [rate limit on raw endpoints](../user/admin_area/settings/rate_limits_on_raw_endpoints.md). The default setting is 300 requests per minute for raw file access.
- Review the [import/export rate limits](../user/admin_area/settings/import_export_rate_limits.md) of the six active defaults.
-For more information about API and rate limits, see our [API page](../api/index.md).
+For more information about API and rate limits, see our [API page](../api/rest/index.md).
## API and rate limits for GitLab SaaS
@@ -255,7 +255,7 @@ Rate limits also improve the security of your application.
You can make changes to your default rate limits from the Admin Area. For more information about configuration, see the [Admin Area page](../security/rate_limits.md#configurable-limits).
- Review the rate limit page.
-- Read our [API page](../api/index.md) for more information about API and rate limiting.
+- Read our [API page](../api/rest/index.md) for more information about API and rate limiting.
### GitLab SaaS-specific block and error responses
@@ -290,5 +290,5 @@ You can learn more about how to administer GitLab.
- Udemy: For a more affordable, guided training option, consider
[GitLab CI: Pipelines, CI/CD, and DevOps for Beginners](https://www.udemy.com/course/gitlab-ci-pipelines-ci-cd-and-devops-for-beginners/) on Udemy.
-- LinkedIn Learning: Check out [Continuous Delivery with GitLab](https://www.linkedin.com/learning/continuous-delivery-with-gitlab) on LinkedIn Learning
+- LinkedIn Learning: Check out [Continuous Delivery with GitLab](https://www.linkedin.com/learning/continuous-integration-and-continuous-delivery-with-gitlab?replacementOf=continuous-delivery-with-gitlab) on LinkedIn Learning
for another low-cost, guided training option.
diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md
index d2f5282a69a..143f7dca7d3 100644
--- a/doc/administration/gitaly/configure_gitaly.md
+++ b/doc/administration/gitaly/configure_gitaly.md
@@ -91,7 +91,7 @@ the default ports for HTTP and HTTPs communication.
![Gitaly network architecture diagram](img/gitaly_network_13_9.png)
WARNING:
-Gitaly servers must not be exposed to the public internet as Gitaly's network traffic is unencrypted
+Gitaly servers must not be exposed to the public internet as Gitaly network traffic is unencrypted
by default. The use of firewall is highly recommended to restrict access to the Gitaly server.
Another option is to [use TLS](#enable-tls-support).
@@ -128,7 +128,7 @@ To configure Gitaly servers, you must:
The `git` user must be able to read, write, and set permissions on the configured storage path.
-To avoid downtime while rotating Gitaly's token, you can temporarily disable authentication using the `gitaly['auth_transitioning']` setting. For more information, see the documentation on
+To avoid downtime while rotating the Gitaly token, you can temporarily disable authentication using the `gitaly['auth_transitioning']` setting. For more information, see the documentation on
[enabling "auth transitioning mode"](#enable-auth-transitioning-mode).
#### Configure authentication
@@ -770,7 +770,7 @@ These RPCs can consume a large amount of resources, which can have a significant
- Unexpectedly high traffic.
- Running against [large repositories](../../user/project/repository/managing_large_repositories.md) that don't follow best practices.
-You can limit these processes from overwhelming your Gitaly server in these scenarios using the concurrency limits in Gitaly's configuration file. For
+You can limit these processes from overwhelming your Gitaly server in these scenarios using the concurrency limits in the Gitaly configuration file. For
example:
```ruby
@@ -778,7 +778,7 @@ example:
gitaly['concurrency'] = [
{
- 'rpc' => "/gitaly.SmartHTTPService/PostUploadPackWithSidechanel",
+ 'rpc' => "/gitaly.SmartHTTPService/PostUploadPackWithSidechannel",
'max_per_repo' => 20,
'max_queue_time' => "1s",
'max_queue_size' => 10
diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md
index b2b6962a222..405c56284cf 100644
--- a/doc/administration/gitaly/index.md
+++ b/doc/administration/gitaly/index.md
@@ -60,7 +60,7 @@ If you have:
- A sharded Gitaly instance.
- Gitaly Cluster.
-Contact your Technical Account Manager or customer support if you have any questions.
+Contact your [Customer Success Manager](https://about.gitlab.com/job-families/sales/customer-success-management/) or customer support if you have any questions.
### Known issues
@@ -70,7 +70,7 @@ the current status of these issues, refer to the referenced issues and epics.
| Issue | Summary | How to avoid |
|:--------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------|
| Gitaly Cluster + Geo - Issues retrying failed syncs | If Gitaly Cluster is used on a Geo secondary site, repositories that have failed to sync could continue to fail when Geo tries to resync them. Recovering from this state requires assistance from support to run manual steps. | No known solution prior to GitLab 15.0. In GitLab 15.0 to 15.2, enable the [`gitaly_praefect_generated_replica_paths` feature flag](#praefect-generated-replica-paths-gitlab-150-and-later). In GitLab 15.3, the feature flag is enabled by default. |
-| Praefect unable to insert data into the database due to migrations not being applied after an upgrade | If the database is not kept up to date with completed migrations, then the Praefect node is unable to perform normal operation. | Make sure the Praefect database is up and running with all migrations completed (For example: `/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migrate-status` should show a list of all applied migrations). Consider [requesting upgrade assistance](https://about.gitlab.com/support/scheduling-upgrade-assistance/) so your upgrade plan can be reviewed by support. |
+| Praefect unable to insert data into the database due to migrations not being applied after an upgrade | If the database is not kept up to date with completed migrations, then the Praefect node is unable to perform standard operation. | Make sure the Praefect database is up and running with all migrations completed (For example: `/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migrate-status` should show a list of all applied migrations). Consider [requesting upgrade assistance](https://about.gitlab.com/support/scheduling-upgrade-assistance/) so your upgrade plan can be reviewed by support. |
| Restoring a Gitaly Cluster node from a snapshot in a running cluster | Because the Gitaly Cluster runs with consistent state, introducing a single node that is behind results in the cluster not being able to reconcile the nodes data and other nodes data | Don't restore a single Gitaly Cluster node from a backup snapshot. If you must restore from backup:<br/><br/>1. [Shut down GitLab](../read_only_gitlab.md#shut-down-the-gitlab-ui).<br/>2. Snapshot all Gitaly Cluster nodes at the same time.<br/>3. Take a database dump of the Praefect database. |
| Rebuilding or replacing an existing Gitaly Cluster node | There is no way to replace existing nodes in place because the Praefect database is relied on to determine the current state of each Gitaly node. Changing how Gitaly Cluster stores repositories is proposed in issue [4218](https://gitlab.com/gitlab-org/gitaly/-/issues/4218). Turning on [repository verification](praefect.md#repository-verification) is proposed in issue [4429](https://gitlab.com/gitlab-org/gitaly/-/issues/4429) so Praefect can detect that data is missing and needs to replicated to a new Gitaly node. | No known solution at this time. |
@@ -289,7 +289,7 @@ be uneconomical to have the same replication factor for all repositories.
To provide greater flexibility for extremely large GitLab instances,
variable replication factor is tracked in [this issue](https://gitlab.com/groups/gitlab-org/-/epics/3372).
-As with normal Gitaly storages, virtual storages can be sharded.
+As with standard Gitaly storages, virtual storages can be sharded.
### Storage layout
@@ -388,7 +388,7 @@ Gitaly Cluster uses the PostgreSQL metadata store with the storage layout to ens
deletion, and move operations. The disk operations can't be atomically applied across multiple storages. However, PostgreSQL guarantees
the atomicity of the metadata operations. Gitaly Cluster models the operations in a manner that the failing operations always leave
the metadata consistent. The disks may contain stale state even after successful operations. This is expected and the leftover state
-won't interfere with future operations but may use up disk space unnecessarily until a clean up is performed.
+does not interfere with future operations but may use up disk space unnecessarily until a clean up is performed.
There is on-going work on a [background crawler](https://gitlab.com/gitlab-org/gitaly/-/issues/3719) that cleans up the leftover
repositories from the storages.
@@ -617,8 +617,8 @@ a commit), you still have:
- The cost of a network roundtrip to Gitaly.
- Inside Gitaly, a write/read roundtrip on the Unix pipes that connect Gitaly to the `git` process.
-Using GitLab.com to measure, we reduced the number of Gitaly calls per request until the loss of
-Rugged's efficiency was no longer felt. It also helped that we run Gitaly itself directly on the Git
+Using GitLab.com to measure, we reduced the number of Gitaly calls per request until we no longer felt
+the efficiency loss of losing Rugged. It also helped that we run Gitaly itself directly on the Git
file servers, rather than by using NFS mounts. This gave us a speed boost that counteracted the
negative effect of not using Rugged anymore.
diff --git a/doc/administration/gitaly/monitoring.md b/doc/administration/gitaly/monitoring.md
index 9024da269ca..0fd34d5c89f 100644
--- a/doc/administration/gitaly/monitoring.md
+++ b/doc/administration/gitaly/monitoring.md
@@ -167,14 +167,8 @@ To monitor [strong consistency](index.md#strong-consistency), you can use the fo
- `gitaly_hook_transaction_voting_delay_seconds`, the client-side delay introduced by waiting for
the transaction to be committed.
-To monitor the number of repositories that have no healthy, up-to-date replicas:
-
-- `gitaly_praefect_unavailable_repositories`
-
To monitor [repository verification](praefect.md#repository-verification), use the following Prometheus metrics:
-- `gitaly_praefect_verification_queue_depth`, the total number of replicas pending verification. This
- metric is scraped from the database and is only available when Prometheus is scraping the database metrics.
- `gitaly_praefect_verification_jobs_dequeued_total`, the number of verification jobs picked up by the
worker.
- `gitaly_praefect_verification_jobs_completed_total`, the number of verification jobs completed by the
@@ -185,6 +179,9 @@ To monitor [repository verification](praefect.md#repository-verification), use t
- `gitaly_praefect_stale_verification_leases_released_total`, the number of stale verification leases
released.
+The `/metrics` endpoint also provides all the metrics available under the `/db_metrics` endpoint. Using `/metrics` for `/db_metrics` metrics
+is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/390266) in GitLab 15.9 and will be removed in GitLab 16.0.
+
You can also monitor the [Praefect logs](../logs/index.md#praefect-logs).
### Database metrics `/db_metrics` endpoint
@@ -194,7 +191,7 @@ You can also monitor the [Praefect logs](../logs/index.md#praefect-logs).
The following metrics are available from the `/db_metrics` endpoint:
- `gitaly_praefect_unavailable_repositories`, the number of repositories that have no healthy, up to date replicas.
-- `gitaly_praefect_read_only_repositories`, the number of repositories in read-only mode in a virtual storage.
- This metric is available for backwards compatibility reasons. `gitaly_praefect_unavailable_repositories` is more
- accurate.
- `gitaly_praefect_replication_queue_depth`, the number of jobs in the replication queue.
+- `gitaly_praefect_verification_queue_depth`, the total number of replicas pending verification.
+- `gitaly_praefect_read_only_repositories`, the number of repositories in read-only mode in a virtual storage.
+ - This metric was [removed](https://gitlab.com/gitlab-org/gitaly/-/issues/4229) in GitLab 15.4.
diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md
index 8b4750b299d..440bd7427ae 100644
--- a/doc/administration/gitaly/praefect.md
+++ b/doc/administration/gitaly/praefect.md
@@ -802,7 +802,7 @@ To complete this section you need:
These should be dedicated nodes, do not run other services on these nodes.
Every Gitaly server assigned to the Praefect cluster needs to be configured. The
-configuration is the same as a normal [standalone Gitaly server](index.md),
+configuration is the same as a standard [standalone Gitaly server](index.md),
except:
- The storage names are exposed to Praefect, not GitLab
@@ -1025,7 +1025,7 @@ Particular attention should be shown to:
WARNING:
If you have existing data stored on the default Gitaly storage,
- you should [migrate the data your Gitaly Cluster storage](index.md#migrate-to-gitaly-cluster)
+ you should [migrate the data to your Gitaly Cluster storage](index.md#migrate-to-gitaly-cluster)
first.
```ruby
diff --git a/doc/administration/gitaly/recovery.md b/doc/administration/gitaly/recovery.md
index 56894f3e963..1207d7af3e7 100644
--- a/doc/administration/gitaly/recovery.md
+++ b/doc/administration/gitaly/recovery.md
@@ -9,6 +9,57 @@ info: To determine the technical writer assigned to the Stage/Group associated w
Gitaly Cluster can recover from primary-node failure and unavailable repositories. Gitaly Cluster can perform data
recovery and has Praefect tracking database tools.
+## Manage Gitaly nodes on a Gitaly Cluster
+
+You can add and replace Gitaly nodes on a Gitaly Cluster.
+
+### Add new Gitaly nodes
+
+To add a new Gitaly node to a Gitaly Cluster that has [replication factor](praefect.md#configure-replication-factor):
+
+- Set, set the [replication factor](praefect.md#configure-replication-factor) for each repository using `set-replication-factor` Praefect command. New repositories are
+ replicated based on [replication factor](praefect.md#configure-replication-factor). Praefect doesn't automatically replicate existing repositories to the new Gitaly node.
+- Not set, add the new node in your [Praefect configuration](praefect.md#praefect) under `praefect['virtual_storages']`. Praefect automatically replicates all data to any
+ new Gitaly node added to the configuration.
+
+### Replace an existing Gitaly node
+
+You can replace an existing Gitaly node with a new node with either the same name or a different name.
+
+#### With a node with the same name
+
+To use the same name for the replacement node, use [repository verifier](praefect.md#enable-deletions) to scan the storage and remove dangling metadata records.
+[Manually prioritize verification](praefect.md#prioritize-verification-manually) of the replaced storage to speed up the process.
+
+#### With a node with a different name
+
+To use a different name for the replacement node for a Gitaly Cluster that has [replication factor](praefect.md#configure-replication-factor):
+
+- Set, use [`praefect set-replication-factor`](praefect.md#configure-replication-factor) to set the replication factor per repository again to get new storage assigned.
+ For example:
+
+ ```shell
+ $ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml set-replication-factor -virtual-storage default -repository @hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.git -replication-factor 2
+
+ current assignments: gitaly-1, gitaly-2
+ ```
+
+ To reassign all repositories from the old storage to the new one, after configuring the new Gitaly node:
+
+ 1. Connect to Praefect database:
+
+ ```shell
+ /opt/gitlab/embedded/bin/psql -h <psql host> -U <user> -d <database name>
+ ```
+
+ 1. Update `repository_assignments` table to replace the old Gitaly node name (for example, `old-gitaly`) with the new Gitaly node name (for example, `new-gitaly`):
+
+ ```sql
+ UPDATE repository_assignments SET storage='new-gitaly' WHERE storage='old-gitaly';
+ ```
+
+- Not set, replace the node in the configuration. The old node's state remains in the Praefect database but it is ignored.
+
## Primary node failure
> - Introduced in GitLab 13.0, Gitaly Cluster, elects the secondary with the least unreplicated writes from the primary to be the new primary. There can still be some unreplicated writes, so [data loss can occur](#check-for-data-loss).
diff --git a/doc/administration/gitaly/reference.md b/doc/administration/gitaly/reference.md
index cec18960dfd..1516b82a906 100644
--- a/doc/administration/gitaly/reference.md
+++ b/doc/administration/gitaly/reference.md
@@ -26,7 +26,7 @@ At the top level, `config.toml` defines the items described on the table below.
| `socket_path` | string | yes (if `listen_addr` is not set) | A path which Gitaly should open a Unix socket. |
| `listen_addr` | string | yes (if `socket_path` is not set) | TCP address for Gitaly to listen on. |
| `tls_listen_addr` | string | no | TCP over TLS address for Gitaly to listen on. |
-| `bin_dir` | string | yes | Directory containing Gitaly's executables. |
+| `bin_dir` | string | yes | Directory containing Gitaly executables. |
| `prometheus_listen_addr` | string | no | TCP listen address for Prometheus metrics. If not set, no Prometheus listener is started. |
For example:
@@ -100,7 +100,7 @@ by GitLab with names, such as `default`.
These names and paths are also defined in the `gitlab.yml` configuration file of
GitLab. When you run Gitaly on the same machine as GitLab (the default
-and recommended configuration) storage paths defined in Gitaly's `config.toml`
+and recommended configuration) storage paths defined in the Gitaly `config.toml`
must match those in `gitlab.yml`.
| Name | Type | Required | Description |
@@ -146,7 +146,7 @@ The default limit is 100 `cat-file`s, which constitute a pair of
you are seeing errors complaining about "too many open files", or an
inability to create new processes, you may want to lower this limit.
-Ideally, the number should be large enough to handle normal
+Ideally, the number should be large enough to handle standard
traffic. If you raise the limit, you should measure the cache hit ratio
before and after. If the hit ratio does not improve, the higher limit is
probably not making a meaningful difference. Here is an example
diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md
index 4b8ed74ec62..5f05b6322b0 100644
--- a/doc/administration/gitaly/troubleshooting.md
+++ b/doc/administration/gitaly/troubleshooting.md
@@ -401,6 +401,20 @@ push, which causes a significant delay.
If Git pushes are too slow when Dynatrace is enabled, disable Dynatrace.
+## Gitaly fails to fork processes stored on `noexec` file systems
+
+Because of changes [introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5999) in GitLab 14.10, applying the `noexec` option to a mount
+point (for example, `/var`) causes Gitaly to throw `permission denied` errors related to forking processes. For example:
+
+```shell
+fork/exec /var/opt/gitlab/gitaly/run/gitaly-2057/gitaly-git2go: permission denied
+```
+
+To resolve this, remove the `noexec` option from the file system mount. An alternative is to change the Gitaly runtime directory:
+
+1. Add `gitaly['runtime_dir'] = '<PATH_WITH_EXEC_PERM>'` to `/etc/gitlab/gitlab.rb` and specify a location without `noexec` set.
+1. Run `sudo gitlab-ctl reconfigure`.
+
## Troubleshoot Praefect (Gitaly Cluster)
The following sections provide possible solutions to Gitaly Cluster errors.
@@ -703,7 +717,7 @@ Possible solutions:
## Profiling Gitaly
-Gitaly exposes several of Golang's built-in performance profiling tools on the Prometheus listen port. For example, if Prometheus is listening
+Gitaly exposes several of the Golang built-in performance profiling tools on the Prometheus listen port. For example, if Prometheus is listening
on port `9236` of the GitLab server:
- Get a list of running `goroutines` and their backtraces:
@@ -724,7 +738,7 @@ on port `9236` of the GitLab server:
curl --output heap.bin "http://<gitaly_server>:9236/debug/pprof/heap"
```
-- Record a 5 second execution trace. This will impact Gitaly's performance while running:
+- Record a 5 second execution trace. This impacts the Gitaly performance while running:
```shell
curl --output trace.bin "http://<gitaly_server>:9236/debug/pprof/trace?seconds=5"
diff --git a/doc/administration/housekeeping.md b/doc/administration/housekeeping.md
index e1b26595bc4..d58989db70c 100644
--- a/doc/administration/housekeeping.md
+++ b/doc/administration/housekeeping.md
@@ -119,6 +119,29 @@ background worker asks Gitaly to perform a number of optimizations.
Housekeeping also [removes unreferenced LFS files](../raketasks/cleanup.md#remove-unreferenced-lfs-files)
from your project every `200` push, freeing up storage space for your project.
+### Prune unreachable objects
+
+Unreachable objects are pruned as part of scheduled housekeeping. However,
+you can trigger manual pruning as well. An example: removing commits that contain sensitive
+information. Triggering housekeeping prunes unreachable objects with a grace period of
+two weeks. When you manually trigger the pruning of unreachable objects, the grace period
+is reduced to 30 minutes.
+
+WARNING:
+If a concurrent process (like `git push`) has created an object but hasn't created
+a reference to the object yet, your repository can become corrupted if a reference
+to the object is added after the object is deleted. The grace period exists to
+reduce the likelihood of such race conditions.
+
+To trigger a manual prune of unreachable objects:
+
+1. On the top bar, select **Main menu > Projects** and find your project.
+1. On the left sidebar, select **Settings > General**.
+1. Expand **Advanced**.
+1. Select **Run housekeeping**.
+1. Wait 30 minutes for the operation to complete.
+1. Return to the page where you selected **Run housekeeping**, and select **Prune unreachable objects**.
+
### Scheduled housekeeping
While GitLab automatically performs housekeeping tasks based on the number of
@@ -130,7 +153,7 @@ strategy.
Administrators can enable a background job that performs housekeeping in all
repositories at a customizable interval to remedy this situation. This
background job processes all repositories hosted by a Gitaly node in a random
-order and eagerly performs housekeeping tasks on them. The Gitaly node will stop
+order and eagerly performs housekeeping tasks on them. The Gitaly node stops
processing repositories if it takes longer than the configured interval.
#### Configure scheduled housekeeping
@@ -166,7 +189,7 @@ of a repository. When creating the first fork, we:
1. Create an object pool repository that contains all objects of the repository
that is about to be forked.
-1. Link the repository to this new object pool via Git's alternates mechanism.
+1. Link the repository to this new object pool via the alternates mechanism of Git.
1. Repack the repository so that it uses objects from the object pool. It thus
can drop its own copy of the objects.
@@ -181,12 +204,12 @@ GitLab needs to perform special housekeeping operations in object pools:
thus maintain references to unreachable "dangling" objects so that they don't
ever get deleted.
- GitLab must update object pools regularly to pull in new objects that have
- been added in the primary repository. Otherwise, an object pool will become
+ been added in the primary repository. Otherwise, an object pool becomes
increasingly inefficient at deduplicating objects.
These housekeeping operations are performed by the specialized
`FetchIntoObjectPool` RPC that handles all of these special tasks while also
-executing the regular housekeeping tasks we execute for normal Git
+executing the regular housekeeping tasks we execute for standard Git
repositories.
Object pools are getting optimized automatically whenever the primary member is
diff --git a/doc/administration/inactive_project_deletion.md b/doc/administration/inactive_project_deletion.md
index ea5658bef84..aad6a420246 100644
--- a/doc/administration/inactive_project_deletion.md
+++ b/doc/administration/inactive_project_deletion.md
@@ -8,70 +8,55 @@ info: To determine the technical writer assigned to the Stage/Group associated w
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/85689) in GitLab 15.0 [with a flag](../administration/feature_flags.md) named `inactive_projects_deletion`. Disabled by default.
> - [Feature flag `inactive_projects_deletion`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/96803) removed in GitLab 15.4.
+> - Configuration through GitLab UI [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/85575) in GitLab 15.1.
Administrators of large GitLab instances can find that over time, projects become inactive and are no longer used.
-These projects take up unnecessary disk space. With inactive project deletion, you can identify these projects, warn
-the maintainers ahead of time, and then delete the projects if they remain inactive. When an inactive project is
-deleted, the action generates an audit event that it was performed by the @GitLab-Admin-Bot.
+These projects take up unnecessary disk space.
+
+With inactive project deletion, you can identify these projects, warn the maintainers ahead of time, and then delete the
+projects if they remain inactive. When an inactive project is deleted, the action generates an audit event that it was
+performed by the @GitLab-Admin-Bot.
For the default setting on GitLab.com, see the [GitLab.com settings page](../user/gitlab_com/index.md#inactive-project-deletion).
## Configure inactive project deletion
-You can configure inactive projects deletion or turn it off using either:
-
-- [The GitLab API](#using-the-api) (GitLab 15.0 and later).
-- [The GitLab UI](#using-the-gitlab-ui) (GitLab 15.1 and later).
-
-The following options are available:
-
-- **Delete inactive projects** (`delete_inactive_projects`): Enable or disable inactive project deletion.
-- **Delete inactive projects that exceed** (`inactive_projects_min_size_mb`): Minimum size (MB) of inactive projects to
- be considered for deletion. Projects smaller in size than this threshold aren't considered inactive.
-- **Delete project after** (`inactive_projects_delete_after_months`): Minimum duration (months) after which a project is
- scheduled for deletion if it continues be inactive.
-- **Send warning email** (`inactive_projects_send_warning_email_after_months`): Minimum duration (months) after which a
- deletion warning email is sent if a project continues to be inactive. The warning email is sent to users with the
- Owner and Maintainer roles of the inactive project. This duration must be less than the
- **Delete project after** (`inactive_projects_delete_after_months`) duration.
+To configure deletion of inactive projects:
-For example (using the API):
-
-- `delete_inactive_projects` enabled.
-- `inactive_projects_min_size_mb` set to `50`.
-- `inactive_projects_delete_after_months` set to `12`.
-- `inactive_projects_send_warning_email_after_months` set to `6`.
-
-In this scenario, when a project's size is:
+1. On the top bar, select **Main menu > Admin**.
+1. On the left sidebar, select **Settings > Repository**.
+1. Expand **Repository maintenance**.
+1. In the **Inactive project deletion** section, select **Delete inactive projects**.
+1. Configure the settings.
+ - The warning email is sent to users who have the Owner and Maintainer role for the inactive project.
+ - The email duration must be less than the **Delete project after** duration.
+1. Select **Save changes**.
-- Less than 50 MB, the project is not considered inactive.
-- Greater than 50 MB and it is inactive for:
- - More than 6 months, a deletion warning is email is sent to users with the Owner and Maintainer role on the project
- with the scheduled date of deletion.
- - More than 12 months, the project is scheduled for deletion.
+Inactive projects that meet the criteria are scheduled for deletion and a warning email is sent. If the
+projects remain inactive, they are deleted after the specified duration.
-### Using the API
+### Configuration example
-You can use the [Application settings API](../api/settings.md#change-application-settings) to configure inactive projects.
+If you use these settings:
-### Using the GitLab UI
+- **Delete inactive projects** enabled.
+- **Delete inactive projects that exceed** set to `50`.
+- **Delete project after** set to `12`.
+- **Send warning email** set to `6`.
-> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/85575) in GitLab 15.1.
+If a project is less than 50 MB, the project is not considered inactive.
-To configure inactive projects with the GitLab UI:
+If a project is more than 50 MB and it is inactive for:
-1. On the top bar, select **Main menu > Admin**.
-1. On the left sidebar, select **Settings > Repository**.
-1. Expand **Repository maintenance**.
-1. In the **Inactive project deletion** section, configure the necessary options.
-1. Select **Save changes**.
+- More than 6 months: A deletion warning email is sent. This mail includes the date that the project will be deleted.
+- More than 12 months: The project is scheduled for deletion.
## Determine when a project was last active
You can view a project's activities and determine when the project was last active in the following ways:
-1. Go to the [activity page](../user/project/working_with_projects.md#view-project-activity) for the project and view
- the date of the latest event.
-1. View the `last_activity_at` attribute for the project using the [Projects API](../api/projects.md).
-1. List the visible events for the project using the [Events API](../api/events.md#list-a-projects-visible-events).
- View the `created_at` attribute of the latest event.
+- Go to the [activity page](../user/project/working_with_projects.md#view-project-activity) for the project and view
+ the date of the latest event.
+- View the `last_activity_at` attribute for the project using the [Projects API](../api/projects.md).
+- List the visible events for the project using the [Events API](../api/events.md#list-a-projects-visible-events).
+ View the `created_at` attribute of the latest event.
diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md
index 2093d55d8c0..ea051e2067d 100644
--- a/doc/administration/incoming_email.md
+++ b/doc/administration/incoming_email.md
@@ -798,20 +798,19 @@ incoming_email:
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214900) in GitLab 13.11.
GitLab can read incoming email using the Microsoft Graph API instead of
-IMAP. Because [Microsoft is deprecating IMAP usage with Basic Authentication](https://techcommunity.microsoft.com/t5/exchange-team-blog/announcing-oauth-2-0-support-for-imap-and-smtp-auth-protocols-in/ba-p/1330432), the Microsoft Graph API will soon be required for new Microsoft Exchange Online
-mailboxes.
+IMAP. Because [Microsoft is deprecating IMAP usage with Basic Authentication](https://techcommunity.microsoft.com/t5/exchange-team-blog/announcing-oauth-2-0-support-for-imap-and-smtp-auth-protocols-in/ba-p/1330432), the Microsoft Graph API is be required for new Microsoft Exchange Online mailboxes.
-To configure GitLab for Microsoft Graph, you will need to register an
-OAuth2 application in your Azure Active Directory that has the
+To configure GitLab for Microsoft Graph, you need to register an
+OAuth 2.0 application in your Azure Active Directory that has the
`Mail.ReadWrite` permission for all mailboxes. See the [MailRoom step-by-step guide](https://github.com/tpitale/mail_room/#microsoft-graph-configuration)
and [Microsoft instructions](https://learn.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app)
for more details.
-Record the following when you configure your OAuth2 application:
+Record the following when you configure your OAuth 2.0 application:
- Tenant ID for your Azure Active Directory
-- Client ID for your OAuth2 application
-- Client secret your OAuth2 application
+- Client ID for your OAuth 2.0 application
+- Client secret your OAuth 2.0 application
##### Restrict mailbox access
@@ -868,3 +867,131 @@ gitlab_rails['incoming_email_inbox_options'] = {
```
The Microsoft Graph API is not yet supported in source installations. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/326169) for more details.
+
+### Use encrypted credentials
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/108279) in GitLab 15.9.
+
+Instead of having the incoming email credentials stored in plaintext in the configuration files, you can optionally
+use an encrypted file for the incoming email credentials.
+
+Prerequisites:
+
+- To use encrypted credentials, you must first enable the
+ [encrypted configuration](encrypted_configuration.md).
+
+The supported configuration items for the encrypted file are:
+
+- `user`
+- `password`
+
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
+
+1. If initially your incoming email configuration in `/etc/gitlab/gitlab.rb` looked like:
+
+ ```ruby
+ gitlab_rails['incoming_email_email'] = "incoming-email@mail.example.com"
+ gitlab_rails['incoming_email_password'] = "examplepassword"
+ ```
+
+1. Edit the encrypted secret:
+
+ ```shell
+ sudo gitlab-rake gitlab:incoming_email:secret:edit EDITOR=vim
+ ```
+
+1. Enter the unencrypted contents of the incoming email secret:
+
+ ```yaml
+ user: 'incoming-email@mail.example.com'
+ password: 'examplepassword'
+ ```
+
+1. Edit `/etc/gitlab/gitlab.rb` and remove the `incoming_email` settings for `email` and `password`.
+1. Save the file and reconfigure GitLab:
+
+ ```shell
+ sudo gitlab-ctl reconfigure
+ ```
+
+:::TabTitle Helm chart (Kubernetes)
+
+Use a Kubernetes secret to store the incoming email password. For more information,
+read about [Helm IMAP secrets](https://docs.gitlab.com/charts/installation/secrets.html#imap-password-for-incoming-emails).
+
+:::TabTitle Docker
+
+1. If initially your incoming email configuration in `docker-compose.yml` looked like:
+
+ ```yaml
+ version: "3.6"
+ services:
+ gitlab:
+ image: 'gitlab/gitlab-ee:latest'
+ restart: always
+ hostname: 'gitlab.example.com'
+ environment:
+ GITLAB_OMNIBUS_CONFIG: |
+ gitlab_rails['incoming_email_email'] = "incoming-email@mail.example.com"
+ gitlab_rails['incoming_email_password'] = "examplepassword"
+ ```
+
+1. Get inside the container, and edit the encrypted secret:
+
+ ```shell
+ sudo docker exec -t <container_name> bash
+ gitlab-rake gitlab:incoming_email:secret:edit EDITOR=editor
+ ```
+
+1. Enter the unencrypted contents of the incoming email secret:
+
+ ```yaml
+ user: 'incoming-email@mail.example.com'
+ password: 'examplepassword'
+ ```
+
+1. Edit `docker-compose.yml` and remove the `incoming_email` settings for `email` and `password`.
+1. Save the file and restart GitLab:
+
+ ```shell
+ docker compose up -d
+ ```
+
+:::TabTitle Self-compiled (source)
+
+1. If initially your incoming email configuration in `/home/git/gitlab/config/gitlab.yml` looked like:
+
+ ```yaml
+ production:
+ incoming_email:
+ user: 'incoming-email@mail.example.com'
+ password: 'examplepassword'
+ ```
+
+1. Edit the encrypted secret:
+
+ ```shell
+ bundle exec rake gitlab:incoming_email:secret:edit EDITOR=vim RAILS_ENVIRONMENT=production
+ ```
+
+1. Enter the unencrypted contents of the incoming email secret:
+
+ ```yaml
+ user: 'incoming-email@mail.example.com'
+ password: 'examplepassword'
+ ```
+
+1. Edit `/home/git/gitlab/config/gitlab.yml` and remove the `incoming_email:` settings for `user` and `password`.
+1. Save the file and restart GitLab and Mailroom
+
+ ```shell
+ # For systems running systemd
+ sudo systemctl restart gitlab.target
+
+ # For systems running SysV init
+ sudo service gitlab restart
+ ```
+
+::EndTabs
diff --git a/doc/administration/index.md b/doc/administration/index.md
index 1059424da27..9f4477bcbf7 100644
--- a/doc/administration/index.md
+++ b/doc/administration/index.md
@@ -5,7 +5,7 @@ info: "To determine the technical writer assigned to the Stage/Group associated
description: 'Learn how to install, configure, update, and maintain your GitLab instance.'
---
-# Administrator documentation **(FREE SELF)**
+# Administer GitLab **(FREE SELF)**
If you use GitLab.com, only GitLab team members have access to administration tools and settings.
If you use a self-managed GitLab instance, learn how to administer it.
diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md
index 5215f0eaed2..b1d97fd16a9 100644
--- a/doc/administration/instance_limits.md
+++ b/doc/administration/instance_limits.md
@@ -156,16 +156,17 @@ Set the limit to `0` to disable it.
### Search rate limit
-> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/80631) in GitLab 14.9.
+> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/80631) in GitLab 14.9.
+> - [Changed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/104208) to include issue, merge request, and epic searches to the rate limit in GitLab 15.9.
-This setting limits global search requests as follows:
+This setting limits search requests as follows:
| Limit | Default (requests per minute) |
|-------------------------|-------------------------------|
| Authenticated user | 30 |
| Unauthenticated user | 10 |
-Depending on the number of enabled [scopes](../user/search/index.md#global-search-scopes), a global search request can consume two to seven requests per minute. You may want to disable one or more scopes to use fewer requests. Global search requests that exceed the search rate limit per minute return the following error:
+Depending on the number of enabled [scopes](../user/search/index.md#global-search-scopes), a global search request can consume two to seven requests per minute. You may want to disable one or more scopes to use fewer requests. Search requests that exceed the search rate limit per minute return the following error:
```plaintext
This endpoint has been requested too many times. Try again later.
@@ -181,7 +182,7 @@ Read more about [pipeline creation rate limits](../user/admin_area/settings/rate
## Gitaly concurrency limit
-Clone traffic can put a large strain on your Gitaly service. To prevent such workloads from overwhelming your Gitaly server, you can set concurrency limits in Gitaly's configuration file.
+Clone traffic can put a large strain on your Gitaly service. To prevent such workloads from overwhelming your Gitaly server, you can set concurrency limits in the Gitaly configuration file.
Read more about [Gitaly concurrency limits](gitaly/configure_gitaly.md#limit-rpc-concurrency).
@@ -379,7 +380,7 @@ and to limit memory consumption.
When using offset-based pagination in the REST API, there is a limit to the maximum
requested offset into the set of results. This limit is only applied to endpoints that
also support keyset-based pagination. More information about pagination options can be
-found in the [API documentation section on pagination](../api/index.md#pagination).
+found in the [API documentation section on pagination](../api/rest/index.md#pagination).
To set this limit for a self-managed installation, run the following in the
[GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session):
@@ -654,7 +655,6 @@ setting is used:
| `ci_max_artifact_size_archive` | 0 |
| `ci_max_artifact_size_browser_performance` | 0 |
| `ci_max_artifact_size_cluster_applications` | 0 |
-| `ci_max_artifact_size_cluster_image_scanning` | 0 |
| `ci_max_artifact_size_cobertura` | 0 |
| `ci_max_artifact_size_codequality` | 0 |
| `ci_max_artifact_size_container_scanning` | 0 |
@@ -1073,9 +1073,19 @@ varies by file type:
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/320902) in GitLab 13.9.
If a branch is merged while open merge requests still point to it, GitLab can
-retarget merge requests pointing to the now-merged branch. To learn more, read
+retarget merge requests pointing to the now-merged branch. For more information, see
[Update merge requests when target branch merges](../user/project/merge_requests/index.md#update-merge-requests-when-target-branch-merges).
+## Maximum number of assignees and reviewers
+
+> - Maximum assignees [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/368936) in GitLab 15.6.
+> - Maximum reviewers [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/366485) in GitLab 15.9.
+
+Issues and merge requests enforce these maximums:
+
+- Maximum assignees: 200
+- Maximum reviewers: 200
+
## CDN-based limits on GitLab.com
In addition to application-based limits, GitLab.com is configured to use Cloudflare's standard DDoS protection and Spectrum to protect Git over SSH. Cloudflare terminates client TLS connections but is not application aware and cannot be used for limits tied to users or groups. Cloudflare page rules and rate limits are configured with Terraform. These configurations are [not public](https://about.gitlab.com/handbook/communication/#not-public) because they include security and abuse implementations that detect malicious activities and making them public would undermine those operations.
diff --git a/doc/administration/instance_review.md b/doc/administration/instance_review.md
index 5516a47637d..277595190b3 100644
--- a/doc/administration/instance_review.md
+++ b/doc/administration/instance_review.md
@@ -20,7 +20,7 @@ details and contact you with suggestions to improve your use of GitLab.
To request an instance review:
1. Sign in as an administrator.
-1. On the top bar, in the top right corner, select your avatar.
+1. On the top bar, in the upper-right corner, select your avatar.
1. Select **Get a free instance review**.
![Instance review](img/instance_review_v14_7.png)
diff --git a/doc/administration/integration/kroki.md b/doc/administration/integration/kroki.md
index 491eeb002e4..081c4d8a08c 100644
--- a/doc/administration/integration/kroki.md
+++ b/doc/administration/integration/kroki.md
@@ -4,7 +4,7 @@ group: Project Management
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
-# Kroki diagrams **(FREE SELF)**
+# Kroki **(FREE SELF)**
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/241744) in GitLab 13.7.
> - Support for reStructuredText and Textile documents [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/324766) in GitLab 13.12.
diff --git a/doc/administration/integration/mailgun.md b/doc/administration/integration/mailgun.md
index d78cd9e1796..218b2888033 100644
--- a/doc/administration/integration/mailgun.md
+++ b/doc/administration/integration/mailgun.md
@@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w
type: reference, howto
---
-# Mailgun and GitLab **(FREE SELF)**
+# Mailgun **(FREE SELF)**
When you use Mailgun to send emails for your GitLab instance and [Mailgun](https://www.mailgun.com/)
integration is enabled and configured in GitLab, you can receive their webhook for
@@ -43,7 +43,7 @@ After configuring your Mailgun domain for the webhook endpoints,
you're ready to enable the Mailgun integration:
1. Sign in to GitLab as an [Administrator](../../user/permissions.md) user.
-1. On the top bar, select **Main menu >** **{admin}** **Admin**.
+1. On the top bar, select **Main menu > Admin**.
1. On the left sidebar, go to **Settings > General** and expand the **Mailgun** section.
1. Select the **Enable Mailgun** checkbox.
1. Enter the Mailgun HTTP webhook signing key as described in
diff --git a/doc/administration/integration/plantuml.md b/doc/administration/integration/plantuml.md
index f0b3e949b35..33434e15c4e 100644
--- a/doc/administration/integration/plantuml.md
+++ b/doc/administration/integration/plantuml.md
@@ -5,7 +5,7 @@ info: "To determine the technical writer assigned to the Stage/Group associated
type: reference, howto
---
-# PlantUML and GitLab **(FREE)**
+# PlantUML **(FREE)**
When the [PlantUML](https://plantuml.com) integration is enabled and configured in
GitLab, you can create diagrams in snippets, wikis, and repositories. This integration
diff --git a/doc/administration/issue_closing_pattern.md b/doc/administration/issue_closing_pattern.md
index bdbcdd0093c..1ea6b3bb49c 100644
--- a/doc/administration/issue_closing_pattern.md
+++ b/doc/administration/issue_closing_pattern.md
@@ -8,8 +8,9 @@ type: reference
# Issue closing pattern **(FREE SELF)**
NOTE:
-This is the administration documentation. There is a separate [user documentation](../user/project/issues/managing_issues.md#closing-issues-automatically)
-on issue closing pattern.
+This page explains how an administrator can configure issue closing patterns.
+For user documentation about the feature, see
+[Closing issues automatically](../user/project/issues/managing_issues.md#closing-issues-automatically).
When a commit or merge request resolves one or more issues, it is possible to
automatically close these issues when the commit or merge request lands
@@ -21,9 +22,9 @@ The [default issue closing pattern](../user/project/issues/managing_issues.md#de
covers a wide range of words. You can change the pattern to suit your needs.
NOTE:
-You are advised to use <https://rubular.com> to test the issue closing pattern.
-However, since Rubular doesn't understand `%{issue_ref}`, you can replace this by
-`#\d+` when testing your patterns, which matches only local issue references like `#123`.
+To test the issue closing pattern, use <https://rubular.com>.
+However, Rubular doesn't understand `%{issue_ref}`. When testing your patterns,
+replace this string with `#\d+`, which matches only local issue references like `#123`.
To change the default issue closing pattern:
diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md
index 816cfcb508d..4f2eb20877e 100644
--- a/doc/administration/job_artifacts.md
+++ b/doc/administration/job_artifacts.md
@@ -788,3 +788,22 @@ FATAL: invalid argument
```
If a job artifact fails to upload with the above error when using consolidated object storage, make sure you are [using separate buckets](object_storage.md#use-separate-buckets) for each data type.
+
+### Job artifacts fail to upload with `FATAL: invalid argument` when using Windows mount
+
+If you are using a Windows mount with CIFS for job artifacts, you may see an
+`invalid argument` error when the runner attempts to upload artifacts:
+
+```plaintext
+WARNING: Uploading artifacts as "dotenv" to coordinator... POST https://<your-gitlab-instance>/api/v4/jobs/<JOB_ID>/artifacts: 500 Internal Server Error id=1296 responseStatus=500 Internal Server Error status=500 token=*****
+FATAL: invalid argument
+```
+
+To work around this issue, you can try:
+
+- Switching to an ext4 mount instead of CIFS.
+- Upgrading to at least Linux kernel 5.15 which contains a number of important bug fixes
+ relating to CIFS file leases.
+- For older kernels, using the `nolease` mount option to disable file leasing.
+
+For more information, [see the investigation details](https://gitlab.com/gitlab-org/gitlab/-/issues/389995).
diff --git a/doc/administration/lfs/index.md b/doc/administration/lfs/index.md
index 302dc38cf28..3638729a6b6 100644
--- a/doc/administration/lfs/index.md
+++ b/doc/administration/lfs/index.md
@@ -12,7 +12,7 @@ For user documentation about Git LFS, see [Git Large File Storage](../../topics/
Prerequisites:
-- Users need to install [Git LFS client](https://git-lfs.github.com) version 1.0.1 or later.
+- Users need to install [Git LFS client](https://git-lfs.com/) version 1.0.1 or later.
## Enable or disable LFS
@@ -439,7 +439,7 @@ To fix this issue, configure your object storage provider's CORS
settings to allow the GitLab domain. See the following documentation
for more details:
-1. [AWS S3](https://aws.amazon.com/premiumsupport/knowledge-center/s3-configure-cors/)
+1. [AWS S3](https://repost.aws/knowledge-center/s3-configure-cors)
1. [Google Cloud Storage](https://cloud.google.com/storage/docs/configuring-cors)
1. [Azure Storage](https://learn.microsoft.com/en-us/rest/api/storageservices/cross-origin-resource-sharing--cors--support-for-the-azure-storage-services).
diff --git a/doc/administration/load_balancer.md b/doc/administration/load_balancer.md
index 83b42295035..298d22f1da5 100644
--- a/doc/administration/load_balancer.md
+++ b/doc/administration/load_balancer.md
@@ -115,6 +115,9 @@ Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`.
It is strongly recommend that multi-node deployments configure load balancers to use the [readiness check](../user/admin_area/monitoring/health_check.md#readiness) to ensure a node is ready to accept traffic, before routing traffic to it. This is especially important when utilizing Puma, as there is a brief period during a restart where Puma doesn't accept requests.
+WARNING:
+Using the `all=1` parameter with the readiness check in GitLab versions 15.4 to 15.8 may cause [increased Praefect memory usage](https://gitlab.com/gitlab-org/gitaly/-/issues/4751) and lead to memory errors.
+
## Troubleshooting
### The health check is returning a `408` HTTP code via the load balancer
diff --git a/doc/administration/logs/index.md b/doc/administration/logs/index.md
index 9893c6935a3..eab4c9b7d83 100644
--- a/doc/administration/logs/index.md
+++ b/doc/administration/logs/index.md
@@ -866,6 +866,19 @@ Depending on your installation method, this file is located at:
- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/database_load_balancing.log`
- Installations from source: `/home/git/gitlab/log/database_load_balancing.log`
+## `zoekt.log` **(PREMIUM SELF)**
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110980) in GitLab 15.9.
+
+This file logs information related to the
+[Exact code search](../../user/search/exact_code_search.md) feature which is
+powered by Zoekt.
+
+Depending on your installation method, this file is located at:
+
+- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/zoekt.log`
+- Installations from source: `/home/git/gitlab/log/zoekt.log`
+
## `elasticsearch.log` **(PREMIUM SELF)**
> Introduced in GitLab 12.6.
diff --git a/doc/administration/logs/log_parsing.md b/doc/administration/logs/log_parsing.md
index dfe434ed1f2..84c517e6120 100644
--- a/doc/administration/logs/log_parsing.md
+++ b/doc/administration/logs/log_parsing.md
@@ -120,6 +120,12 @@ jq 'select(.queue_duration_s > 10000)' <FILE>
jq -s 'map(select(.gitaly_calls != null)) | sort_by(-.gitaly_calls) | limit(10; .[])' <FILE>
```
+#### Output a specific time range
+
+```shell
+jq 'select(.time >= "2023-01-10T00:00:00Z" and .time <= "2023-01-10T12:00:00Z")' <FILE>
+```
+
### Parsing `gitlab-rails/production_json.log`
#### Print the top three controller methods by request volume and their three longest durations
@@ -166,7 +172,7 @@ jq --raw-output '[.route, .ua] | @tsv' api_json.log | sort | uniq -c | sort -n
1234 /api/:version/internal/allowed GitLab-Shell
```
-This sample response seems normal. A custom tool or script might be causing a high load
+This sample response seems typical. A custom tool or script might be causing a high load
if the output contains many:
- Third party libraries like `python-requests` or `curl`.
diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md
index 171a441eaec..fc6318922f1 100644
--- a/doc/administration/monitoring/performance/performance_bar.md
+++ b/doc/administration/monitoring/performance/performance_bar.md
@@ -51,7 +51,7 @@ From left to right, the performance bar displays:
values in milliseconds, separated by slashes.
Select to display a modal window with more details. The values, from left to right:
- **Backend**: time needed for the base page to load.
- - [**First Contentful Paint**](https://web.dev/first-contentful-paint/):
+ - [**First Contentful Paint**](https://developer.chrome.com/docs/lighthouse/performance/first-contentful-paint/):
Time until something was visible to the user. Displays `NaN` if your browser does not
support this feature.
- [**DomContentLoaded**](https://web.dev/critical-rendering-path-measure-crp/) Event.
@@ -81,7 +81,7 @@ From left to right, the performance bar displays:
NOTE:
Not all indicators are available in all environments. For instance, the memory view
requires running Ruby with [specific patches](https://gitlab.com/gitlab-org/gitlab-build-images/-/blob/master/patches/ruby/2.7.4/thread-memory-allocations-2.7.patch)
-applied. When running GitLab locally using [GDK](../../../development/contributing/index.md#gitlab-development-kit),
+applied. When running GitLab locally using the [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit),
this is typically not the case and the memory view cannot be used.
## Keyboard shortcut
diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md
index 29182077d66..ed55e7d7ff3 100644
--- a/doc/administration/monitoring/prometheus/gitlab_metrics.md
+++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md
@@ -161,6 +161,8 @@ The following metrics are available:
| `gitlab_diffs_write_cache_real_duration_seconds` | Histogram | 15.8 | Duration in seconds spent on caching highlighted lines and stats on diffs batch request | `controller`, `action` |
| `gitlab_diffs_highlight_cache_decorate_real_duration_seconds` | Histogram | 15.8 | Duration in seconds spent on setting highlighted lines from cache on diffs batch request | `controller`, `action` |
| `gitlab_diffs_render_real_duration_seconds` | Histogram | 15.8 | Duration in seconds spent on serializing and rendering diffs on diffs batch request | `controller`, `action` |
+| `gitlab_memwd_violations_total` | Counter | 15.9 | Total number of times a Ruby process violated a memory threshold | |
+| `gitlab_memwd_violations_handled_total` | Counter | 15.9 | Total number of times Ruby process memory violations were handled | |
## Metrics controlled by a feature flag
@@ -350,6 +352,9 @@ configuration option in `gitlab.yml`. These metrics are served from the
| `geo_dependency_proxy_manifests_verification_total` | Gauge | 15.6 | Number of dependency proxy manifests verifications tried on secondary | `url` |
| `geo_dependency_proxy_manifests_verified` | Gauge | 15.6 | Number of dependency proxy manifests verified on secondary | `url` |
| `geo_dependency_proxy_manifests_verification_failed` | Gauge | 15.6 | Number of dependency proxy manifests verifications failed on secondary | `url` |
+| `gitlab_memwd_violations_total` | Counter | 15.9 | Total number of times a Sidekiq process violated a memory threshold | |
+| `gitlab_memwd_violations_handled_total` | Counter | 15.9 | Total number of times Sidekiq process memory violations were handled | |
+| `sidekiq_watchdog_running_jobs_total` | Counter | 15.9 | Current running jobs when RSS limit was reached | `worker_class` |
## Database load balancing metrics **(PREMIUM SELF)**
diff --git a/doc/administration/monitoring/prometheus/web_exporter.md b/doc/administration/monitoring/prometheus/web_exporter.md
index 5539526c501..0212091f14c 100644
--- a/doc/administration/monitoring/prometheus/web_exporter.md
+++ b/doc/administration/monitoring/prometheus/web_exporter.md
@@ -22,7 +22,7 @@ We provide two mechanisms by which web application metrics can be exported:
makes metric data available via its own `/-/metrics` endpoint. This is the default,
and is described in [GitLab Metrics](index.md#gitlab-metrics). We recommend this
default for small GitLab installations where the amount of metrics collected is small.
-- Through a dedicated metrics server. Enabling this server will cause Puma to launch an
+- Through a dedicated metrics server. Enabling this server causes Puma to launch an
additional process whose sole responsibility is to serve metrics. This approach leads
to better fault isolation and performance for very large GitLab installations, but
comes with additional memory use. We recommend this approach for medium to large
@@ -69,5 +69,5 @@ To serve metrics via HTTPS instead of HTTP, enable TLS in the exporter settings:
1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure)
for the changes to take effect.
-When TLS is enabled, the same `port` and `address` will be used as described above.
+When TLS is enabled, the same `port` and `address` is used as described above.
The metrics server cannot serve both HTTP and HTTPS at the same time.
diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md
index 972db315268..7349e8ad9ba 100644
--- a/doc/administration/nfs.md
+++ b/doc/administration/nfs.md
@@ -101,7 +101,7 @@ specifically test NFSv3.
When you define your NFS exports, we recommend you also add the following
options:
-- `no_root_squash` - NFS normally changes the `root` user to `nobody`. This is
+- `no_root_squash` - NFS usually changes the `root` user to `nobody`. This is
a good security measure when NFS shares are accessed by many different
users. However, in this case only GitLab uses the NFS share so it
is safe. GitLab recommends the `no_root_squash` setting because we need to
@@ -279,7 +279,7 @@ to store the data on an NFS mount.
Bind mounts provide a way to specify just one NFS mount and then
bind the default GitLab data locations to the NFS mount. Start by defining your
-single NFS mount point as you normally would in `/etc/fstab`. Let's assume your
+single NFS mount point as you typically would in `/etc/fstab`. Let's assume your
NFS mount point is `/gitlab-nfs`. Then, add the following bind mounts in
`/etc/fstab`:
diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md
index 6064ae7525e..4b3e251d407 100644
--- a/doc/administration/object_storage.md
+++ b/doc/administration/object_storage.md
@@ -105,7 +105,13 @@ When the consolidated form is:
See the section on [ETag mismatch errors](#etag-mismatch) for more details.
-**In Omnibus installations:**
+#### Use AWS S3
+
+The following example uses AWS S3 to enable object storage for all supported services:
+
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
1. Edit `/etc/gitlab/gitlab.rb` and add the following lines, substituting
the values you want:
@@ -116,7 +122,7 @@ See the section on [ETag mismatch errors](#etag-mismatch) for more details.
gitlab_rails['object_store']['proxy_download'] = true
gitlab_rails['object_store']['connection'] = {
'provider' => 'AWS',
- 'region' => '<eu-central-1>',
+ 'region' => 'eu-central-1',
'aws_access_key_id' => '<AWS_ACCESS_KEY_ID>',
'aws_secret_access_key' => '<AWS_SECRET_ACCESS_KEY>'
}
@@ -125,62 +131,231 @@ See the section on [ETag mismatch errors](#etag-mismatch) for more details.
'server_side_encryption' => '<AES256 or aws:kms>',
'server_side_encryption_kms_key_id' => '<arn:aws:kms:xxx>'
}
- gitlab_rails['object_store']['objects']['artifacts']['bucket'] = '<artifacts>'
- gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = '<external-diffs>'
- gitlab_rails['object_store']['objects']['lfs']['bucket'] = '<lfs-objects>'
- gitlab_rails['object_store']['objects']['uploads']['bucket'] = '<uploads>'
- gitlab_rails['object_store']['objects']['packages']['bucket'] = '<packages>'
- gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = '<dependency-proxy>'
- gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = '<terraform-state>'
- gitlab_rails['object_store']['objects']['pages']['bucket'] = '<pages>'
+ gitlab_rails['object_store']['objects']['artifacts']['bucket'] = 'gitlab-artifacts'
+ gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = 'gitlab-mr-diffs'
+ gitlab_rails['object_store']['objects']['lfs']['bucket'] = 'gitlab-lfs'
+ gitlab_rails['object_store']['objects']['uploads']['bucket'] = 'gitlab-uploads'
+ gitlab_rails['object_store']['objects']['packages']['bucket'] = 'gitlab-packages'
+ gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = 'gitlab-dependency-proxy'
+ gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = 'gitlab-terraform-state'
+ gitlab_rails['object_store']['objects']['ci_secure_files']['bucket'] = 'gitlab-ci-secure-files'
+ gitlab_rails['object_store']['objects']['pages']['bucket'] = 'gitlab-pages'
```
- If you're using AWS IAM profiles, omit the AWS access key and secret access
- key/value pairs. For example:
+ If you’re using [AWS IAM profiles](#using-amazon-instance-profiles), omit
+ the AWS access key and secret access key/value pairs. For example:
+
+ ```ruby
+ gitlab_rails['object_store']['connection'] = {
+ 'provider' => 'AWS',
+ 'region' => 'eu-central-1',
+ 'use_iam_profile' => true
+ }
+ ```
+
+1. Save the file and reconfigure GitLab:
+
+ ```shell
+ sudo gitlab-ctl reconfigure
+ ```
+
+:::TabTitle Helm chart (Kubernetes)
+
+1. Put the following content in a file named `object_storage.yaml` to be used as a
+ [Kubernetes Secret](https://docs.gitlab.com/charts/charts/globals.html#connection):
+
+ ```yaml
+ provider: AWS
+ region: us-east-1
+ aws_access_key_id: <AWS_ACCESS_KEY_ID>
+ aws_secret_access_key: <AWS_SECRET_ACCESS_KEY>
+ ```
+
+ If you’re using [AWS IAM profiles](#using-amazon-instance-profiles), omit
+ the AWS access key and secret access key/value pairs. For example:
+
+ ```yaml
+ provider: AWS
+ region: us-east-1
+ use_iam_profile: true
+ ```
+
+1. Create the Kubernetes Secret:
+
+ ```shell
+ kubectl create secret generic -n <namespace> gitlab-object-storage --from-file=connection=object_storage.yaml
+ ```
+
+1. Export the Helm values:
+
+ ```shell
+ helm get values gitlab > gitlab_values.yaml
+ ```
+
+1. Edit `gitlab_values.yaml`:
+
+ ```yaml
+ global:
+ appConfig:
+ object_store:
+ enabled: false
+ proxy_download: true
+ storage_options: {}
+ # server_side_encryption:
+ # server_side_encryption_kms_key_id
+ connection:
+ secret: gitlab-object-storage
+ lfs:
+ enabled: true
+ proxy_download: true
+ bucket: gitlab-lfs
+ connection: {}
+ # secret:
+ # key:
+ artifacts:
+ enabled: true
+ proxy_download: true
+ bucket: gitlab-artifacts
+ connection: {}
+ # secret:
+ # key:
+ uploads:
+ enabled: true
+ proxy_download: true
+ bucket: gitlab-uploads
+ connection: {}
+ # secret:
+ # key:
+ packages:
+ enabled: true
+ proxy_download: true
+ bucket: gitlab-packages
+ connection: {}
+ externalDiffs:
+ enabled: true
+ when:
+ proxy_download: true
+ bucket: gitlab-mr-diffs
+ connection: {}
+ terraformState:
+ enabled: true
+ bucket: gitlab-terraform-state
+ connection: {}
+ ciSecureFiles:
+ enabled: true
+ bucket: gitlab-ci-secure-files
+ connection: {}
+ dependencyProxy:
+ enabled: true
+ proxy_download: true
+ bucket: gitlab-dependency-proxy
+ connection: {}
+ ```
+
+1. Save the file and apply the new values:
+
+ ```shell
+ helm upgrade -f gitlab_values.yaml gitlab gitlab/gitlab
+ ```
+
+:::TabTitle Docker
+
+1. Edit `docker-compose.yml`:
+
+ ```yaml
+ version: "3.6"
+ services:
+ gitlab:
+ environment:
+ GITLAB_OMNIBUS_CONFIG: |
+ # Consolidated object storage configuration
+ gitlab_rails['object_store']['enabled'] = true
+ gitlab_rails['object_store']['proxy_download'] = true
+ gitlab_rails['object_store']['connection'] = {
+ 'provider' => 'AWS',
+ 'region' => 'eu-central-1',
+ 'aws_access_key_id' => '<AWS_ACCESS_KEY_ID>',
+ 'aws_secret_access_key' => '<AWS_SECRET_ACCESS_KEY>'
+ }
+ # OPTIONAL: The following lines are only needed if server side encryption is required
+ gitlab_rails['object_store']['storage_options'] = {
+ 'server_side_encryption' => '<AES256 or aws:kms>',
+ 'server_side_encryption_kms_key_id' => '<arn:aws:kms:xxx>'
+ }
+ gitlab_rails['object_store']['objects']['artifacts']['bucket'] = 'gitlab-artifacts'
+ gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = 'gitlab-mr-diffs'
+ gitlab_rails['object_store']['objects']['lfs']['bucket'] = 'gitlab-lfs'
+ gitlab_rails['object_store']['objects']['uploads']['bucket'] = 'gitlab-uploads'
+ gitlab_rails['object_store']['objects']['packages']['bucket'] = 'gitlab-packages'
+ gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = 'gitlab-dependency-proxy'
+ gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = 'gitlab-terraform-state'
+ gitlab_rails['object_store']['objects']['ci_secure_files']['bucket'] = 'gitlab-ci-secure-files'
+ gitlab_rails['object_store']['objects']['pages']['bucket'] = 'gitlab-pages'
+ ```
+
+ If you’re using [AWS IAM profiles](#using-amazon-instance-profiles), omit
+ the AWS access key and secret access key/value pairs. For example:
```ruby
gitlab_rails['object_store']['connection'] = {
'provider' => 'AWS',
- 'region' => '<eu-central-1>',
+ 'region' => 'eu-central-1',
'use_iam_profile' => true
}
```
-1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+1. Save the file and restart GitLab:
+
+ ```shell
+ docker compose up -d
+ ```
-**In installations from source:**
+:::TabTitle Self-compiled (source)
1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines:
```yaml
- object_store:
- enabled: true
- proxy_download: true
- connection:
- provider: AWS
- aws_access_key_id: <AWS_ACCESS_KEY_ID>
- aws_secret_access_key: <AWS_SECRET_ACCESS_KEY>
- region: <eu-central-1>
- storage_options:
- server_side_encryption: <AES256 or aws:kms>
- server_side_encryption_key_kms_id: <arn:aws:kms:xxx>
- objects:
- artifacts:
- bucket: <artifacts>
- external_diffs:
- bucket: <external-diffs>
- lfs:
- bucket: <lfs-objects>
- uploads:
- bucket: <uploads>
- packages:
- bucket: <packages>
- dependency_proxy:
- bucket: <dependency_proxy>
- terraform_state:
- bucket: <terraform>
- pages:
- bucket: <pages>
+ production: &base
+ object_store:
+ enabled: true
+ proxy_download: true
+ connection:
+ provider: AWS
+ aws_access_key_id: <AWS_ACCESS_KEY_ID>
+ aws_secret_access_key: <AWS_SECRET_ACCESS_KEY>
+ region: eu-central-1
+ storage_options:
+ server_side_encryption: <AES256 or aws:kms>
+ server_side_encryption_key_kms_id: <arn:aws:kms:xxx>
+ objects:
+ artifacts:
+ bucket: gitlab-artifacts
+ external_diffs:
+ bucket: gitlab-mr-diffs
+ lfs:
+ bucket: gitlab-lfs
+ uploads:
+ bucket: gitlab-uploads
+ packages:
+ bucket: gitlab-packages
+ dependency_proxy:
+ bucket: gitlab-dependency-proxy
+ terraform_state:
+ bucket: gitlab-terraform-state
+ ci_secure_files:
+ bucket: gitlab-ci-secure-files
+ pages:
+ bucket: gitlab-pages
+ ```
+
+ If you’re using [AWS IAM profiles](#using-amazon-instance-profiles), omit
+ the AWS access key and secret access key/value pairs. For example:
+
+ ```yaml
+ connection:
+ provider: AWS
+ region: eu-central-1
+ use_iam_profile: true
```
1. Edit `/home/git/gitlab-workhorse/config.toml` and add or amend the following lines:
@@ -194,7 +369,25 @@ See the section on [ETag mismatch errors](#etag-mismatch) for more details.
aws_secret_access_key = "<AWS_SECRET_ACCESS_KEY>"
```
-1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect.
+ If you’re using [AWS IAM profiles](#using-amazon-instance-profiles), omit
+ the AWS access key and secret access key/value pairs. For example:
+
+ ```yaml
+ [object_storage.s3]
+ use_iam_profile = true
+ ```
+
+1. Save the file and restart GitLab:
+
+ ```shell
+ # For systems running systemd
+ sudo systemctl restart gitlab.target
+
+ # For systems running SysV init
+ sudo service gitlab restart
+ ```
+
+::EndTabs
#### Common parameters
@@ -288,12 +481,11 @@ Here are the valid connection parameters for GCS:
GitLab reads the value of `google_json_key_location`, then `google_json_key_string`, and finally, `google_application_default`.
It uses the first of these settings that has a value.
-The service account must have permission to access the bucket. Learn more
-in Google's
-[Cloud Storage authentication documentation](https://cloud.google.com/storage/docs/authentication).
+The service account must have permission to access the bucket. For more information,
+see the [Cloud Storage authentication documentation](https://cloud.google.com/storage/docs/authentication).
NOTE:
-Bucket encryption with the [Cloud Key Management Service (KMS)](https://cloud.google.com/kms/docs) is not supported and will result in [ETag mismatch errors](#etag-mismatch).
+Bucket encryption with the [Cloud Key Management Service (KMS)](https://cloud.google.com/kms/docs) is not supported and results in [ETag mismatch errors](#etag-mismatch).
##### Google example (consolidated form)
@@ -347,9 +539,8 @@ because a single set of credentials are used to access multiple
containers. The [storage-specific form](#storage-specific-configuration)
is not supported. For more details, see [how to transition to consolidated form](#transition-to-consolidated-form).
-The following are the valid connection parameters for Azure. Read the
-[Azure Blob storage documentation](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction)
-to learn more.
+The following are the valid connection parameters for Azure. For more information, see the
+[Azure Blob Storage documentation](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction).
| Setting | Description | Example |
|------------------------------|----------------|-----------|
@@ -393,6 +584,30 @@ If you are using a custom Azure storage domain,
configuration. This information is exchanged in an API call between
GitLab Rails and Workhorse.
+#### Storj Gateway Configuration (SJ)
+
+NOTE:
+The Storj Gateway [does not support](https://github.com/storj/gateway-st/blob/4b74c3b92c63b5de7409378b0d1ebd029db9337d/docs/s3-compatibility.md) multi-threaded copying (see `UploadPartCopy` in the table).
+While an implementation [is planned](https://github.com/storj/roadmap/issues/40), you must [disable multi-threaded copying](#multi-threaded-copying) until completion.
+
+The [Storj Network](https://www.storj.io/) provides an S3-compatible API gateway. Use the following configuration example:
+
+```ruby
+gitlab_rails['object_store']['connection'] = {
+ 'provider' => 'AWS',
+ 'endpoint' => 'https://gateway.storjshare.io',
+ 'path_style' => true,
+ 'region' => 'eu1',
+ 'aws_access_key_id' => 'ACCESS_KEY',
+ 'aws_secret_access_key' => 'SECRET_KEY',
+ 'aws_signature_version' => 2,
+ 'enable_signature_v4_streaming' => false
+}
+```
+
+The signature version must be `2`. Using v4 results in a HTTP 411 Length Required error.
+For more information, see [issue #4419](https://gitlab.com/gitlab-org/gitlab/-/issues/4419).
+
### Object-specific configuration
The following YAML shows how the `object_store` section defines
@@ -583,8 +798,8 @@ With Omnibus and source installations it is possible to split a single
real bucket into multiple virtual buckets. If your object storage
bucket is called `my-gitlab-objects` you can configure uploads to go
into `my-gitlab-objects/uploads`, artifacts into
-`my-gitlab-objects/artifacts`, etc. The application will act as if
-these are separate buckets. Note that use of bucket prefixes
+`my-gitlab-objects/artifacts`, etc. The application acts as if
+these are separate buckets. Use of bucket prefixes
[may not work correctly with Helm backups](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/3376).
Helm-based installs require separate buckets to
@@ -686,7 +901,7 @@ With the consolidated object configuration and instance profile, Workhorse has
S3 credentials so that it can compute the `Content-MD5` header. This
eliminates the need to compare ETag headers returned from the S3 server.
-Encrypting buckets with GCS' [Cloud Key Management Service (KMS)](https://cloud.google.com/kms/docs) is not supported and will result in ETag mismatch errors.
+Encrypting buckets with the GCS [Cloud Key Management Service (KMS)](https://cloud.google.com/kms/docs) is not supported and results in ETag mismatch errors.
### Using Amazon instance profiles
@@ -697,6 +912,11 @@ When this is used, GitLab fetches temporary credentials each time an
S3 bucket is accessed, so no hard-coded values are needed in the
configuration.
+To use an Amazon instance profile, GitLab must be able to connect to the
+[instance metadata endpoint](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html).
+If GitLab is [configured to use an Internet proxy](https://docs.gitlab.com/omnibus/settings/environment-variables.html), the endpoint IP
+address must be added to the `no_proxy` list.
+
#### Encrypted S3 buckets
> - [Introduced](https://gitlab.com/gitlab-org/gitlab-workhorse/-/merge_requests/466) in GitLab 13.1 for instance profiles only and [S3 default encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/bucket-encryption.html).
@@ -714,7 +934,7 @@ Customer master keys (CMKs) and SSE-C encryption are
Setting a default encryption on an S3 bucket is the easiest way to
enable encryption, but you may want to
-[set a bucket policy to ensure only encrypted objects are uploaded](https://aws.amazon.com/premiumsupport/knowledge-center/s3-bucket-store-kms-encrypted-objects/).
+[set a bucket policy to ensure only encrypted objects are uploaded](https://repost.aws/knowledge-center/s3-bucket-store-kms-encrypted-objects).
To do this, you must configure GitLab to send the proper encryption headers
in the `storage_options` configuration section:
@@ -817,5 +1037,5 @@ Prerequisites:
After you have done at least one successful Rclone copy from the old location to the new location, schedule maintenance and take your GitLab server offline. During your maintenance window you must do two things:
-1. Perform a final `rclone sync` run, knowing that your users cannot add new objects so you will not leave any behind in the old bucket.
+1. Perform a final `rclone sync` run, knowing that your users cannot add new objects so you do not leave any behind in the old bucket.
1. Update the object storage configuration of your GitLab server to use the new provider for `uploads`.
diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md
index a34b21e676a..1e887d8bd67 100644
--- a/doc/administration/operations/fast_ssh_key_lookup.md
+++ b/doc/administration/operations/fast_ssh_key_lookup.md
@@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w
NOTE:
This document describes a drop-in replacement for the
-`authorized_keys` file. For normal (non-deploy key) users, consider using
+`authorized_keys` file. For standard (non-deploy key) users, consider using
[SSH certificates](ssh_certificates.md). They are even faster, but are not a
drop-in replacement.
@@ -25,10 +25,6 @@ GitLab Shell solves this by providing a way to authorize SSH users via a fast,
indexed lookup in the GitLab database. This page describes how to enable the fast
lookup of authorized SSH keys.
-WARNING:
-OpenSSH version 6.9+ is required because `AuthorizedKeysCommand` must be
-able to accept a fingerprint. Check the version of OpenSSH on your server with `sshd -V`.
-
## Fast lookup is required for Geo **(PREMIUM)**
Unlike [Cloud Native GitLab](https://docs.gitlab.com/charts/), Omnibus GitLab by default
@@ -51,12 +47,31 @@ secondary nodes, but **Write to "authorized keys" file**
must be unchecked only on the primary node, because it is reflected
automatically on the secondary if database replication is working.
-## Setting up fast lookup via GitLab Shell
+## Set up fast lookup
GitLab Shell provides a way to authorize SSH users via a fast, indexed lookup
to the GitLab database. GitLab Shell uses the fingerprint of the SSH key to
check whether the user is authorized to access GitLab.
+Fast lookup can be enabled with the following SSH servers:
+
+- [`gitlab-sshd`](gitlab_sshd.md)
+- OpenSSH
+
+You can run both services simultaneously by using separate ports for each service.
+
+### With `gitlab-sshd`
+
+To set up `gitlab-sshd`, see [the `gitlab-sshd` documentation](gitlab_sshd.md).
+After `gitlab-sshd` is enabled, GitLab Shell and `gitlab-sshd` are configured
+to use fast lookup automatically.
+
+### With OpenSSH
+
+WARNING:
+OpenSSH version 6.9+ is required because `AuthorizedKeysCommand` must be
+able to accept a fingerprint. Check the version of OpenSSH on your server with `sshd -V`.
+
Add the following to your `sshd_config` file. This file is usually located at
`/etc/ssh/sshd_config`, but it is at `/assets/sshd_config` if you're using
Omnibus Docker:
@@ -119,7 +134,7 @@ Then you can backup and delete your `authorized_keys` file for best performance.
The current users' keys are already present in the database, so there is no need for migration
or for users to re-add their keys.
-## How to go back to using the `authorized_keys` file
+### How to go back to using the `authorized_keys` file
This overview is brief. Refer to the above instructions for more context.
@@ -132,52 +147,6 @@ This overview is brief. Refer to the above instructions for more context.
1. Remove the `AuthorizedKeysCommand` lines from `/etc/ssh/sshd_config` or from `/assets/sshd_config` if you are using Omnibus Docker.
1. Reload `sshd`: `sudo service sshd reload`.
-## Use `gitlab-sshd` instead of OpenSSH
-
-> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299109) in GitLab 14.5 as an **Alpha** release for self-managed customers.
-
-WARNING:
-`gitlab-sshd` is in [**Alpha**](../../policy/alpha-beta-support.md#alpha-features).
-It is not ready for production use.
-
-`gitlab-sshd` is [a standalone SSH server](https://gitlab.com/gitlab-org/gitlab-shell/-/tree/main/internal/sshd)
-written in Go. It is provided as a part of the `gitlab-shell` package. It has a lower memory
-use as a OpenSSH alternative, and supports
-[group access restriction by IP address](../../user/group/index.md) for applications
-running behind the proxy.
-
-`gitlab-sshd` is a lightweight alternative to OpenSSH for providing
-[SSH operations](https://gitlab.com/gitlab-org/gitlab-shell/-/blob/71a7f34a476f778e62f8fe7a453d632d395eaf8f/doc/features.md).
-While OpenSSH uses a restricted shell approach, `gitlab-sshd` behaves more like a
-modern multi-threaded server application, responding to incoming requests. The major
-difference is that OpenSSH uses SSH as a transport protocol while `gitlab-sshd` uses Remote Procedure Calls (RPCs).
-
-The capabilities of GitLab Shell are not limited to Git operations.
-
-If you are considering switching from OpenSSH to `gitlab-sshd`, consider these concerns:
-
-- The `gitlab-sshd` component is only available for
- [GitLab Helm chart](https://docs.gitlab.com/charts/) deployments.
-- `gitlab-sshd` supports the PROXY protocol. It can run behind proxy servers that rely
- on it, such as HAProxy. The PROXY protocol not enabled by default, but can be enabled with a Helm chart setting.
-- By default, `gitlab-sshd` binds to port 22, but you can configure a different port in the Helm chart.
-- `gitlab-sshd` **does not** support SSH certificates. For more details, read
- [issue #495](https://gitlab.com/gitlab-org/gitlab-shell/-/issues/495).
-
-To switch from OpenSSH to `gitlab-sshd`:
-
-1. Set the `gitlab-shell` charts `sshDaemon` option to
- [`gitlab-sshd`](https://docs.gitlab.com/charts/charts/gitlab/gitlab-shell/index.html#installation-command-line-options).
- For example:
-
- ```yaml
- gitlab:
- gitlab-shell:
- sshDaemon: gitlab-sshd
- ```
-
-1. Perform a Helm upgrade.
-
## SELinux support and limitations
GitLab supports `authorized_keys` database lookups with [SELinux](https://en.wikipedia.org/wiki/Security-Enhanced_Linux).
diff --git a/doc/administration/operations/gitlab_sshd.md b/doc/administration/operations/gitlab_sshd.md
new file mode 100644
index 00000000000..7b61631fe3a
--- /dev/null
+++ b/doc/administration/operations/gitlab_sshd.md
@@ -0,0 +1,139 @@
+---
+stage: Create
+group: Source Code
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# `gitlab-sshd` **(FREE SELF)**
+
+> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299109) in GitLab 14.5 as an **Alpha** release for self-managed customers.
+> - Ready for production use with [Cloud Native GitLab in GitLab 15.1](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/2540) and [Omnibus GitLab in GitLab 15.9](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5937).
+
+`gitlab-sshd` is [a standalone SSH server](https://gitlab.com/gitlab-org/gitlab-shell/-/tree/main/internal/sshd)
+written in Go. It is provided as a part of the `gitlab-shell` package. It has a lower memory
+use as a OpenSSH alternative, and supports
+[group access restriction by IP address](../../user/group/index.md) for applications
+running behind the proxy.
+
+`gitlab-sshd` is a lightweight alternative to OpenSSH for providing
+[SSH operations](https://gitlab.com/gitlab-org/gitlab-shell/-/blob/71a7f34a476f778e62f8fe7a453d632d395eaf8f/doc/features.md).
+While OpenSSH uses a restricted shell approach, `gitlab-sshd` behaves more like a
+modern multi-threaded server application, responding to incoming requests. The major
+difference is that OpenSSH uses SSH as a transport protocol while `gitlab-sshd` uses Remote Procedure Calls (RPCs). See [the blog post](https://about.gitlab.com/blog/2022/08/17/why-we-have-implemented-our-own-sshd-solution-on-gitlab-sass/) for more details.
+
+The capabilities of GitLab Shell are not limited to Git operations.
+
+If you are considering switching from OpenSSH to `gitlab-sshd`, consider these concerns:
+
+- `gitlab-sshd` supports the PROXY protocol. It can run behind proxy servers that rely
+ on it, such as HAProxy. The PROXY protocol is not enabled by default, but [it can be enabled](#proxy-protocol-support).
+- `gitlab-sshd` **does not** support SSH certificates. For more details, read
+ [issue #495](https://gitlab.com/gitlab-org/gitlab-shell/-/issues/495).
+
+## Enable `gitlab-sshd`
+
+To use `gitlab-sshd`:
+
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
+
+The following instructions enable `gitlab-sshd` on a different port than OpenSSH:
+
+1. Edit `/etc/gitlab/gitlab.rb`:
+
+ ```ruby
+ gitlab_sshd['enable'] = true
+ gitlab_sshd['listen_address'] = '[::]:2222' # Adjust the port accordingly
+ ```
+
+1. Optional. By default, Omnibus GitLab generates SSH host keys for `gitlab-sshd` if
+they do not exist in `/var/opt/gitlab/gitlab-sshd`. If you wish to disable this automatic generation, add this line:
+
+ ```ruby
+ gitlab_sshd['generate_host_keys'] = false
+ ```
+
+1. Save the file and reconfigure GitLab:
+
+ ```shell
+ sudo gitlab-ctl reconfigure
+ ```
+
+By default, `gitlab-sshd` runs as the `git` user. As a result, `gitlab-sshd` cannot
+run on privileged port numbers lower than 1024. This means users must
+access Git with the `gitlab-sshd` port, or use a load balancer that
+directs SSH traffic to the `gitlab-sshd` port to hide this.
+
+Users may see host key warnings because the newly-generated host keys
+differ from the OpenSSH host keys. Consider disabling host key
+generation and copy the existing OpenSSH host keys into
+`/var/opt/gitlab/gitlab-sshd` if this is an issue.
+
+:::TabTitle Helm chart (Kubernetes)
+
+The following instructions switch OpenSSH in favor of `gitlab-sshd`:
+
+1. Set the `gitlab-shell` charts `sshDaemon` option to
+ [`gitlab-sshd`](https://docs.gitlab.com/charts/charts/gitlab/gitlab-shell/index.html#installation-command-line-options).
+ For example:
+
+ ```yaml
+ gitlab:
+ gitlab-shell:
+ sshDaemon: gitlab-sshd
+ ```
+
+1. Perform a Helm upgrade.
+
+By default, `gitlab-sshd` listens for:
+
+- External requests on port 22 (`global.shell.port`).
+- Internal requests on port 2222 (`gitlab.gitlab-shell.service.internalPort`).
+
+You can [configure different ports in the Helm chart](https://docs.gitlab.com/charts/charts/gitlab/gitlab-shell/#configuration).
+
+::EndTabs
+
+## PROXY protocol support
+
+When a load balancer is used in front of `gitlab-sshd`, GitLab reports the IP
+address of the proxy instead of the actual IP address of the client. `gitlab-sshd`
+supports the [PROXY protocol](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt) to
+obtain the real IP address.
+
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
+
+To enable the PROXY protocol:
+
+1. Edit `/etc/gitlab/gitlab.rb`:
+
+ ```ruby
+ gitlab_sshd['proxy_protocol'] = true
+ # # Proxy protocol policy ("use", "require", "reject", "ignore"), "use" is the default value
+ gitlab_sshd['proxy_policy'] = "use"
+ ```
+
+1. Save the file and reconfigure GitLab:
+
+ ```shell
+ sudo gitlab-ctl reconfigure
+ ```
+
+:::TabTitle Helm chart (Kubernetes)
+
+1. Set the [`gitlab.gitlab-shell.config` options](https://docs.gitlab.com/charts/charts/gitlab/gitlab-shell/index.html#installation-command-line-options). For example:
+
+ ```yaml
+ gitlab:
+ gitlab-shell:
+ config:
+ proxyProtocol: true
+ proxyPolicy: "use"
+ ```
+
+1. Perform a Helm upgrade.
+
+::EndTabs
diff --git a/doc/administration/operations/index.md b/doc/administration/operations/index.md
index 527a72c5933..f6ab46b9fbf 100644
--- a/doc/administration/operations/index.md
+++ b/doc/administration/operations/index.md
@@ -17,6 +17,7 @@ Keep your GitLab instance up and running smoothly.
to restart Sidekiq.
- [Multiple Sidekiq processes](../sidekiq/extra_sidekiq_processes.md): Configure multiple Sidekiq processes to ensure certain queues always have dedicated workers, no matter the number of jobs that must be processed. **(FREE SELF)**
- [Puma](puma.md): Understand Puma and puma-worker-killer.
+- [`gitlab-sshd`](gitlab_sshd.md): Use GitLab SSH daemon instead of OpenSSH.
- Speed up SSH operations by
[Authorizing SSH users via a fast, indexed lookup to the GitLab database](fast_ssh_key_lookup.md), and/or
by [doing away with user SSH keys stored on GitLab entirely in favor of SSH certificates](ssh_certificates.md).
diff --git a/doc/administration/operations/moving_repositories.md b/doc/administration/operations/moving_repositories.md
index 5066f6d99d8..aa0477be788 100644
--- a/doc/administration/operations/moving_repositories.md
+++ b/doc/administration/operations/moving_repositories.md
@@ -65,6 +65,8 @@ To move repositories:
[individual snippets](../../api/snippet_repository_storage_moves.md#schedule-a-repository-storage-move-for-a-snippet).
- [All groups](#move-all-groups) or
[individual groups](../../api/group_repository_storage_moves.md#schedule-a-repository-storage-move-for-a-group). **(PREMIUM SELF)**
+1. If [Geo](../geo/index.md) is enabled,
+ [resync all repositories](../geo/replication/troubleshooting.md#queue-up-all-repositories-for-resync).
#### Move all projects
diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md
index 51d6e9ae1fd..f2f9f1cdcda 100644
--- a/doc/administration/operations/puma.md
+++ b/doc/administration/operations/puma.md
@@ -138,7 +138,7 @@ When running Puma in single mode, some features are not supported:
- [Phased restart](https://gitlab.com/gitlab-org/gitlab/-/issues/300665)
- [Memory killers](#reducing-memory-use)
-To learn more, visit [epic 5303](https://gitlab.com/groups/gitlab-org/-/epics/5303).
+For more information, see [epic 5303](https://gitlab.com/groups/gitlab-org/-/epics/5303).
## Performance caveat when using Puma with Rugged
@@ -223,7 +223,7 @@ from the Linux package and is no longer supported.
Puma has a multi-thread architecture that uses less memory than a multi-process
application server like Unicorn. On GitLab.com, we saw a 40% reduction in memory
-consumption. Most Rails application requests normally include a proportion of I/O wait time.
+consumption. Most Rails application requests usually include a proportion of I/O wait time.
During I/O wait time, MRI Ruby releases the GVL to other threads.
Multi-threaded Puma can therefore still serve more requests than a single process.
diff --git a/doc/administration/operations/rails_console.md b/doc/administration/operations/rails_console.md
index f2143435755..652a4fa5497 100644
--- a/doc/administration/operations/rails_console.md
+++ b/doc/administration/operations/rails_console.md
@@ -158,7 +158,7 @@ sudo -u git -H bundle exec rails runner -e production /path/to/script.rb
Rails Runner does not produce the same output as the console.
-If you set a variable on the console, the console will generate useful debug output
+If you set a variable on the console, the console generates useful debug output
such as the variable contents or properties of referenced entity:
```ruby
@@ -176,7 +176,7 @@ root
1
```
-Some basic knowledge of Ruby will be very useful. Try
+Some basic knowledge of Ruby is very useful. Try
[this 30-minute tutorial](https://try.ruby-lang.org/) for a quick introduction.
Rails experience is helpful but not essential.
@@ -425,7 +425,7 @@ D, [2020-03-05T17:18:30.406047 #910] DEBUG -- : User Load (2.6ms) SELECT "use
```
For more on different ways to retrieve data from the database using Active
-Record, please see the [Active Record Query Interface documentation](https://guides.rubyonrails.org/active_record_querying.html).
+Record, see the [Active Record Query Interface documentation](https://guides.rubyonrails.org/active_record_querying.html).
## Query the database using an Active Record model
@@ -547,7 +547,7 @@ be the fastest way to get to the root of the problem.
### Interacting with Active Record objects
-At the end of the day, Active Record objects are just normal Ruby objects. As
+At the end of the day, Active Record objects are just standard Ruby objects. As
such, we can define methods on them which perform arbitrary actions.
For example, GitLab developers have added some methods which help with
diff --git a/doc/administration/operations/ssh_certificates.md b/doc/administration/operations/ssh_certificates.md
index 401451d58b4..bdd1045f7a7 100644
--- a/doc/administration/operations/ssh_certificates.md
+++ b/doc/administration/operations/ssh_certificates.md
@@ -74,7 +74,7 @@ $ ssh-add -L | grep cert | ssh-keygen -L -f -
```
Technically that's not strictly true, for example, it could be
-`prod-aearnfjord` if it's a SSH certificate you'd normally sign in to
+`prod-aearnfjord` if it's a SSH certificate you'd usually sign in to
servers as the `prod-aearnfjord` user, but then you must specify your
own `AuthorizedPrincipalsCommand` to do that mapping instead of using
our provided default.
@@ -122,7 +122,7 @@ You can supply as many principals as you want, these are turned
into multiple lines of `authorized_keys` output, as described in the
`AuthorizedPrincipalsFile` documentation in `sshd_config(5)`.
-Normally when using the `AuthorizedKeysCommand` with OpenSSH the
+Usually when using the `AuthorizedKeysCommand` with OpenSSH the
principal is some "group" that's allowed to sign in to that
server. However with GitLab it's only used to appease OpenSSH's
requirement for it, we effectively only care about the "key ID" being
@@ -145,13 +145,13 @@ authenticate the user, OpenSSH falls back on
Therefore there may still be a reason to use the [Fast lookup of authorized SSH keys in the database](fast_ssh_key_lookup.md) method
in conjunction with this. Since you are using SSH certificates for
-all your normal users, and relying on the `~/.ssh/authorized_keys`
+all your typical users, and relying on the `~/.ssh/authorized_keys`
fallback for deploy keys, if you make use of those.
But you may find that there's no reason to do that, since all your
-normal users use the fast `AuthorizedPrincipalsCommand` path, and
+typical users use the fast `AuthorizedPrincipalsCommand` path, and
only automated deployment key access falls back on
-`~/.ssh/authorized_keys`, or that you have a lot more keys for normal
+`~/.ssh/authorized_keys`, or that you have a lot more keys for typical
users (especially if they're renewed) than you have deploy keys.
## Other security caveats
diff --git a/doc/administration/package_information/defaults.md b/doc/administration/package_information/defaults.md
index 717e5ca31c9..3b52b6bca82 100644
--- a/doc/administration/package_information/defaults.md
+++ b/doc/administration/package_information/defaults.md
@@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# Package defaults **(FREE SELF)**
Unless configuration is specified in the `/etc/gitlab/gitlab.rb` file,
-the package will assume the defaults as noted below.
+the package assumes the defaults as noted below.
## Ports
@@ -49,7 +49,7 @@ by default:
| PgBouncer | No | Port | X | 6432 |
| Consul | No | Port | X | 8300, 8301(UDP), 8500, 8600[^Consul-notes] |
| Patroni | No | Port | X | 8008 |
-| GitLab KAS | No | Port | X | 8150 |
+| GitLab KAS | Yes | Port | X | 8150 |
| Gitaly | No | Port | X | 8075 |
Legend:
@@ -63,13 +63,13 @@ Legend:
GitLab also expects a file system to be ready for the storage of Git repositories
and various other files.
-Note that if you are using NFS (Network File System), files will be carried
-over a network which will require, based on implementation, ports `111` and
+If you are using NFS (Network File System), files are carried
+over a network which requires, based on implementation, ports `111` and
`2049` to be open.
NOTE:
-In some cases, the GitLab Registry will be automatically enabled by default. See [our documentation](../packages/container_registry.md) for more details.
+In some cases, the GitLab Registry is automatically enabled by default. See [our documentation](../packages/container_registry.md) for more details.
[^Consul-notes]: If using additional Consul functionality, more ports may need to be opened. See the [official documentation](https://developer.hashicorp.com/consul/docs/install/ports#ports-table) for the list.
- [^Sidekiq-health]: If Sidekiq health check settings are not set, they will default to the Sidekiq metrics exporter settings. This default is deprecated and is set to be removed in [GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/347509).
+ [^Sidekiq-health]: If Sidekiq health check settings are not set, they default to the Sidekiq metrics exporter settings. This default is deprecated and is set to be removed in [GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/347509).
diff --git a/doc/administration/package_information/licensing.md b/doc/administration/package_information/licensing.md
index ece1597c521..bbc0e864e95 100644
--- a/doc/administration/package_information/licensing.md
+++ b/doc/administration/package_information/licensing.md
@@ -41,7 +41,7 @@ Starting with version 8.13, GitLab has placed an additional step into
Omnibus GitLab. The `license_check` step calls
`lib/gitlab/tasks/license_check.rake`, which checks the compiled `LICENSE` file
against the current list of approved and questionable licenses as denoted in the
-arrays at the top of the script. This script will output one of `Good`,
+arrays at the top of the script. This script outputs one of `Good`,
`Unknown` or `Check` for each piece of software that is a part of the
Omnibus GitLab package.
@@ -70,7 +70,7 @@ All trademarks, materials, documentation, and other intellectual property remain
### Trademark Requirements
-Use of GitLab Trademarks must be in compliance with the standards set forth in [our guidelines](https://about.gitlab.com/handbook/marketing/corporate-marketing/brand-activation/trademark-guidelines/) (as updated from time to time).
+Use of GitLab Trademarks must be in compliance with the standards set forth in [our guidelines](https://about.gitlab.com/handbook/marketing/brand-and-product-marketing/brand/brand-activation/trademark-guidelines/) (as updated from time to time).
CHEF® and all Chef marks are owned by Progress Software Corporation and must be used in accordance with the [Progress Software Trademark Usage Policy](https://www.progress.com/legal/trademarks).
When using a GitLab or 3rd party trademark in documentation, include the (R) symbol in the first instance, for example, "Chef(R) is used for configuring...." You may omit the symbol in subsequent instances.
diff --git a/doc/administration/package_information/omnibus_packages.md b/doc/administration/package_information/omnibus_packages.md
index 7b3892ace70..222341c7308 100644
--- a/doc/administration/package_information/omnibus_packages.md
+++ b/doc/administration/package_information/omnibus_packages.md
@@ -23,7 +23,7 @@ We have a few core goals with these packages:
GitLab in its core is a Ruby on Rails project. However, GitLab as a whole
application is more complex and has multiple components. If these components are
-not present or are incorrectly configured, GitLab will not work or it will work
+not present or are incorrectly configured, GitLab does not work or it works
unpredictably.
The [GitLab Architecture Overview](../../development/architecture.md#gitlab-architecture-overview) shows some of these components and how they
@@ -112,4 +112,4 @@ what was noted above:
1. Running separate services in multiple containers and keeping them running
can be more complex and might not be required for a given install.
-This method is useful for organizations just getting started with containers and schedulers, and may not be ready for a more complex installation. This method is a great introduction, and will work well for smaller organizations.
+This method is useful for organizations just getting started with containers and schedulers, and may not be ready for a more complex installation. This method is a great introduction, and works well for smaller organizations.
diff --git a/doc/administration/package_information/signed_packages.md b/doc/administration/package_information/signed_packages.md
index e4566be7534..638dd7820b8 100644
--- a/doc/administration/package_information/signed_packages.md
+++ b/doc/administration/package_information/signed_packages.md
@@ -6,8 +6,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# Package Signatures **(FREE SELF)**
-As of the release of GitLab 9.5 on August 22, 2017, GitLab provides signed Omnibus GitLab packages for RPM and DEB based distributions. This means that all packages provided on <https://packages.gitlab.com> are signed, starting with `9.5.0`, and all future versions of supported branches (for example `9.3.x` and `9.4.x` after August 22, 2017). Any package version prior to August 22, 2017, will not be signed. Pass the appropriate argument to your package manager. (Example: `yum --nogpgcheck`)
-
Omnibus GitLab packages produced by GitLab are created via the [Omnibus](https://github.com/chef/omnibus) tool, for which GitLab has added DEB signing via `debsigs` in [our own fork](https://gitlab.com/gitlab-org/omnibus). This addition, combined with the existing functionality of RPM signing, allows GitLab to provide signed packages for all supported distributions using DEB or RPM.
These packages are produced by the GitLab CI process, as found in the [Omnibus GitLab project](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/.gitlab-ci.yml), prior to their delivery to <https://packages.gitlab.com> to ensure provide assurance that the packages are not altered prior to delivery to our community.
diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md
index 6446252e143..34acf57ce70 100644
--- a/doc/administration/packages/container_registry.md
+++ b/doc/administration/packages/container_registry.md
@@ -475,7 +475,7 @@ To configure the `s3` storage driver in Omnibus:
`bucket_name.host/object`. [Set to false for AWS S3](https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/).
You can set a rate limit on connections to S3 to avoid 503 errors from the S3 API. To do this,
- set `maxrequestspersecond` to a number within the [S3 request rate threshold](https://aws.amazon.com/premiumsupport/knowledge-center/s3-503-within-request-rate-prefix/):
+ set `maxrequestspersecond` to a number within the [S3 request rate threshold](https://repost.aws/knowledge-center/s3-503-within-request-rate-prefix):
```ruby
registry['storage'] = {
@@ -552,7 +552,7 @@ you can pull from the Container Registry, but you cannot push.
NOTE:
If you have a lot of data, you may be able to improve performance by
- [running parallel sync operations](https://aws.amazon.com/premiumsupport/knowledge-center/s3-improve-transfer-sync-command/).
+ [running parallel sync operations](https://repost.aws/knowledge-center/s3-improve-transfer-sync-command).
1. To perform the final data sync,
[put the Container Registry in `read-only` mode](#performing-garbage-collection-without-downtime) and
@@ -1136,7 +1136,7 @@ To enable the read-only mode:
1. Next, trigger one of the garbage collect commands:
WARNING:
- You must use `/opt/gitlab/embedded/bin/registry` to recycle unused tags. If you use `gitlab-ctl registry-garbage-collect`, you **will bring the container registry down**.
+ You must use `/opt/gitlab/embedded/bin/registry` to recycle unused tags. If you use `gitlab-ctl registry-garbage-collect`, **the container registry goes down**.
```shell
# Recycling unused tags
@@ -1706,7 +1706,7 @@ case, since we know that since the login succeeded, we probably need to look
at the communication between the client and the Registry.
The REST API between the Docker client and Registry is described
-[in the Docker documentation](https://docs.docker.com/registry/spec/api/). Normally, one would just
+[in the Docker documentation](https://docs.docker.com/registry/spec/api/). Usually, one would just
use Wireshark or tcpdump to capture the traffic and see where things went
wrong. However, since all communications between Docker clients and servers
are done over HTTPS, it's a bit difficult to decrypt the traffic quickly even
diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md
index 58003758c8a..ed08b10fe97 100644
--- a/doc/administration/pages/index.md
+++ b/doc/administration/pages/index.md
@@ -272,7 +272,7 @@ control over how the Pages daemon runs and serves content in your environment.
| `log_verbose` | Verbose logging, true/false. |
| `propagate_correlation_id` | Set to true (false by default) to re-use existing Correlation ID from the incoming request header `X-Request-ID` if present. If a reverse proxy sets this header, the value is propagated in the request chain. |
| `max_connections` | Limit on the number of concurrent connections to the HTTP, HTTPS or proxy listeners. |
-| `max_uri_length` | The maximum length of URIs accepted by GitLab Pages. Set to 0 for unlimited length. [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/659) in GitLab 14.5.
+| `max_uri_length` | The maximum length of URIs accepted by GitLab Pages. Set to 0 for unlimited length. [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5729) in GitLab 14.5.
| `metrics_address` | The address to listen on for metrics requests. |
| `redirect_http` | Redirect pages from HTTP to HTTPS, true/false. |
| `redirects_max_config_size` | The maximum size of the `_redirects` file, in bytes (default: 65536). |
@@ -719,7 +719,10 @@ To set the maximum size of GitLab Pages site in a project, overriding the inheri
1. On the top bar, select **Main menu > Projects** and find your project.
1. On the left sidebar, select **Settings > Pages**.
-1. Enter a value under **Maximum size of pages** in MB.
+
+ If this path is not visible, select **Deployments > Pages**.
+ [This location is part of an experiment](../../user/project/pages/index.md#menu-position-test).
+1. In **Maximum size of pages**, enter the size in MB.
1. Select **Save changes**.
## Set maximum number of GitLab Pages custom domains for a project
diff --git a/doc/administration/postgresql/index.md b/doc/administration/postgresql/index.md
index f48e537064a..e829397abc1 100644
--- a/doc/administration/postgresql/index.md
+++ b/doc/administration/postgresql/index.md
@@ -8,26 +8,29 @@ info: To determine the technical writer assigned to the Stage/Group associated w
In this section, you are guided through configuring a PostgreSQL database to
be used with GitLab in one of our [reference architectures](../reference_architectures/index.md).
-There are essentially three setups to choose from.
-## Standalone PostgreSQL using Omnibus GitLab
+## Configuration options
+
+Choose one of the following PostgreSQL configuration options:
+
+### Standalone PostgreSQL using Omnibus GitLab
This setup is for when you have installed the
[Omnibus GitLab packages](https://about.gitlab.com/install/) (CE or EE),
to use the bundled PostgreSQL having only its service enabled.
-[> Read how to set up a standalone PostgreSQL instance using Omnibus GitLab](standalone.md)
+Read how to [set up a standalone PostgreSQL instance](standalone.md) using Omnibus GitLab.
-## Provide your own PostgreSQL instance
+### Provide your own PostgreSQL instance
This setup is for when you have installed GitLab using the
[Omnibus GitLab packages](https://about.gitlab.com/install/) (CE or EE),
or installed it [from source](../../install/installation.md), but you want to use
your own external PostgreSQL server.
-[> Read how to set up an external PostgreSQL instance](external.md)
+Read how to [set up an external PostgreSQL instance](external.md).
-## PostgreSQL replication and failover with Omnibus GitLab **(PREMIUM SELF)**
+### PostgreSQL replication and failover with Omnibus GitLab **(PREMIUM SELF)**
This setup is for when you have installed GitLab using the
[Omnibus GitLab **Enterprise Edition** (EE) package](https://about.gitlab.com/install/?version=ee).
@@ -35,4 +38,12 @@ This setup is for when you have installed GitLab using the
All the tools that are needed like PostgreSQL, PgBouncer, and Patroni are bundled in
the package, so you can use it to set up the whole PostgreSQL infrastructure (primary, replica).
-[> Read how to set up PostgreSQL replication and failover using Omnibus GitLab](replication_and_failover.md)
+Read how to [set up PostgreSQL replication and failover](replication_and_failover.md) using Omnibus GitLab.
+
+## Related topics
+
+- [Working with the bundled PgBouncer service](pgbouncer.md)
+- [Database load balancing](database_load_balancing.md)
+- [Moving GitLab databases to a different PostgreSQL instance](moving.md)
+- [Multiple databases](multiple_databases.md)
+- [Database guides for GitLab development](../../development/database/index.md)
diff --git a/doc/administration/postgresql/moving.md b/doc/administration/postgresql/moving.md
index 96076205281..7c18297a175 100644
--- a/doc/administration/postgresql/moving.md
+++ b/doc/administration/postgresql/moving.md
@@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w
Sometimes it is necessary to move your databases from one PostgreSQL instance to
another. For example, if you are using AWS Aurora and are preparing to
-enable Database Load Balancing, you will need to move your databases to
+enable Database Load Balancing, you need to move your databases to
RDS for PostgreSQL.
To move databases from one instance to another:
@@ -36,7 +36,7 @@ To move databases from one instance to another:
/opt/gitlab/embedded/bin/pg_dump -h $SRC_PGHOST -U $SRC_PGUSER -c -C -f praefect_production.sql praefect_production
```
-1. Restore the databases to the destination (this will overwrite any existing databases with the same names):
+1. Restore the databases to the destination (this overwrites any existing databases with the same names):
```shell
/opt/gitlab/embedded/bin/psql -h $DST_PGHOST -U $DST_PGUSER -f praefect_production.sql postgres
diff --git a/doc/administration/postgresql/multiple_databases.md b/doc/administration/postgresql/multiple_databases.md
index 25f148dbe84..836736fb73f 100644
--- a/doc/administration/postgresql/multiple_databases.md
+++ b/doc/administration/postgresql/multiple_databases.md
@@ -67,6 +67,9 @@ the other way around.
1. Save the `config/database.yml` file.
+1. Update the service files to set the `GITLAB_ALLOW_SEPARATE_CI_DATABASE`
+ environment variable to `true`.
+
1. Create the `gitlabhq_production_ci` database:
```shell
@@ -100,6 +103,7 @@ the other way around.
1. Edit `/etc/gitlab/gitlab.rb` and add the following lines:
```ruby
+ gitlab_rails['env'] = { 'GITLAB_ALLOW_SEPARATE_CI_DATABASE' => 'true' }
gitlab_rails['databases']['ci']['enable'] = true
gitlab_rails['databases']['ci']['db_database'] = 'gitlabhq_production_ci'
```
diff --git a/doc/administration/postgresql/pgbouncer.md b/doc/administration/postgresql/pgbouncer.md
index 25c4c940b97..5dd0aad7162 100644
--- a/doc/administration/postgresql/pgbouncer.md
+++ b/doc/administration/postgresql/pgbouncer.md
@@ -5,7 +5,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w
type: reference
---
-# Working with the bundled PgBouncer service **(PREMIUM SELF)**
+# Working with the bundled PgBouncer service **(FREE SELF)**
+
+NOTE:
+PgBouncer is bundled in the `gitlab-ee` package, but is free to use.
+For support, you need a [Premium subscription](https://about.gitlab.com/pricing/).
[PgBouncer](https://www.pgbouncer.org/) is used to seamlessly migrate database
connections between servers in a failover scenario. Additionally, it can be used
diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md
index ececa12f3ad..46890b0b2ca 100644
--- a/doc/administration/postgresql/replication_and_failover.md
+++ b/doc/administration/postgresql/replication_and_failover.md
@@ -26,7 +26,7 @@ replication failover requires:
- A minimum of three PgBouncer nodes that track and handle primary database reads and writes.
- An internal load balancer (TCP) to balance requests between the PgBouncer nodes.
- [Database Load Balancing](database_load_balancing.md) enabled.
- - A local PgBouncer service configured on each PostgreSQL node. Note that this is separate from the main PgBouncer cluster that tracks the primary.
+ - A local PgBouncer service configured on each PostgreSQL node. This is separate from the main PgBouncer cluster that tracks the primary.
```plantuml
@startuml
@@ -356,7 +356,7 @@ If you enable Monitoring, it must be enabled on **all** database servers.
#### Enable TLS support for the Patroni API
-By default, Patroni's [REST API](https://patroni.readthedocs.io/en/latest/rest_api.html#rest-api) is served over HTTP.
+By default, the Patroni [REST API](https://patroni.readthedocs.io/en/latest/rest_api.html#rest-api) is served over HTTP.
You have the option to enable TLS and use HTTPS over the same [port](../package_information/defaults.md).
To enable TLS, you need PEM-formatted certificate and private key files. Both files must be readable by the PostgreSQL user (`gitlab-psql` by default, or the one set by `postgresql['username']`):
@@ -1009,7 +1009,7 @@ Here are a few key facts that you must consider before upgrading PostgreSQL:
GitLab deployment is down for the duration of database upgrade or, at least, as long as your leader
node is upgraded. This can be **a significant downtime depending on the size of your database**.
-- Upgrading PostgreSQL creates a new data directory with a new control data. From Patroni's perspective this is a new cluster that needs to be bootstrapped again. Therefore, as part of the upgrade procedure, the cluster state (stored in Consul) is wiped out. After the upgrade is complete, Patroni bootstraps a new cluster. **This changes your _cluster ID_**.
+- Upgrading PostgreSQL creates a new data directory with a new control data. From the perspective of Petroni, this is a new cluster that needs to be bootstrapped again. Therefore, as part of the upgrade procedure, the cluster state (stored in Consul) is wiped out. After the upgrade is complete, Patroni bootstraps a new cluster. **This changes your _cluster ID_**.
- The procedures for upgrading leader and replicas are not the same. That is why it is important to use the right procedure on each node.
@@ -1017,7 +1017,7 @@ Here are a few key facts that you must consider before upgrading PostgreSQL:
configured replication method (`pg_basebackup` is the only available option). It might take some
time for replica to catch up with the leader, depending on the size of your database.
-- An overview of the upgrade procedure is outlined in [Patroni's documentation](https://patroni.readthedocs.io/en/latest/existing_data.html#major-upgrade-of-postgresql-version).
+- An overview of the upgrade procedure is outlined in [the Patroni documentation](https://patroni.readthedocs.io/en/latest/existing_data.html#major-upgrade-of-postgresql-version).
You can still use `gitlab-ctl pg-upgrade` which implements this procedure with a few adjustments.
Considering these, you should carefully plan your PostgreSQL upgrade:
@@ -1322,7 +1322,7 @@ If a replica cannot start or rejoin the cluster, or when it lags behind and cann
+-------------------------------------+--------------+---------+--------------+----+-----------+
```
-1. Sign in to the broken server and reinitialize the database and replication. Patroni will shut
+1. Sign in to the broken server and reinitialize the database and replication. Patroni shuts
down PostgreSQL on that server, remove the data directory, and reinitialize it from scratch:
```shell
@@ -1330,7 +1330,7 @@ If a replica cannot start or rejoin the cluster, or when it lags behind and cann
```
This can be run on any Patroni node, but be aware that `sudo gitlab-ctl patroni
- reinitialize-replica` without `--member` will reinitialize the server it is run on.
+ reinitialize-replica` without `--member` restarts the server it is run on.
It is recommended to run it locally on the broken server to reduce the risk of
unintended data loss.
1. Monitor the logs:
diff --git a/doc/administration/raketasks/check.md b/doc/administration/raketasks/check.md
index 2660caa80b3..2cb664b0859 100644
--- a/doc/administration/raketasks/check.md
+++ b/doc/administration/raketasks/check.md
@@ -381,3 +381,37 @@ To delete these references to missing local and/or remote artifacts (`job.log` f
If `gitlab-rake gitlab:lfs:check VERBOSE=1` detects LFS objects that exist in the database
but not on disk, [follow the procedure in the LFS documentation](../lfs/index.md#missing-lfs-objects)
to remove the database entries.
+
+### Update dangling object storage references
+
+If you have [migrated from object storage to local storage](../job_artifacts.md#migrating-from-object-storage-to-local-storage) and files were missing, then dangling database references remain.
+
+This is visible in the migration logs with errors like the following:
+
+```shell
+W, [2022-11-28T13:14:09.283833 #10025] WARN -- : Failed to transfer Ci::JobArtifact ID 11 with error: undefined method `body' for nil:NilClass
+W, [2022-11-28T13:14:09.296911 #10025] WARN -- : Failed to transfer Ci::JobArtifact ID 12 with error: undefined method `body' for nil:NilClass
+```
+
+Attempting to [delete references to missing artifacts](check.md#delete-references-to-missing-artifacts) after you have disabled object storage, results in the following error:
+
+```shell
+RuntimeError (Object Storage is not enabled for JobArtifactUploader)
+```
+
+To update these references to point to local storage:
+
+1. Open the [GitLab Rails Console](../operations/rails_console.md#starting-a-rails-console-session).
+1. Run the following Ruby code:
+
+ ```ruby
+ artifacts_updated = 0
+ ::Ci::JobArtifact.find_each do |artifact| ### Iterate artifacts
+ next if artifact.file_store != 2 ### Skip if file_store already points to local storage
+ artifacts_updated += 1
+ # artifact.update(file_store: 1) ### Uncomment to actually update
+ end
+ puts "Updated file_store count: #{artifacts_updated}"
+ ```
+
+The script to [delete references to missing artifacts](check.md#delete-references-to-missing-artifacts) now functions correctly and cleans up the database.
diff --git a/doc/administration/raketasks/github_import.md b/doc/administration/raketasks/github_import.md
index 2e6a88a77e2..d089682f78e 100644
--- a/doc/administration/raketasks/github_import.md
+++ b/doc/administration/raketasks/github_import.md
@@ -6,6 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# GitHub import Rake task **(FREE SELF)**
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/390690) in GitLab 15.9, Rake task no longer automatically creates namespaces or groups that don't exist.
+
To retrieve and import GitHub repositories, you need a [GitHub personal access token](https://github.com/settings/tokens).
A username should be passed as the second argument to the Rake task,
which becomes the owner of the project. You can resume an import
@@ -39,7 +41,7 @@ bundle exec rake "import:github[access_token,root,foo/bar]" RAILS_ENV=production
In this case, `access_token` is your GitHub personal access token, `root`
is your GitLab username, and `foo/bar` is the new GitLab namespace/project
-created from your GitHub project. Subgroups are also possible: `foo/foo/bar`. The importer creates any missing intermediate namespaces (groups) if they do not exist.
+created from your GitHub project. Subgroups are also possible: `foo/foo/bar`.
## Importing a single project
diff --git a/doc/administration/raketasks/incoming_email.md b/doc/administration/raketasks/incoming_email.md
new file mode 100644
index 00000000000..6b9c27ed144
--- /dev/null
+++ b/doc/administration/raketasks/incoming_email.md
@@ -0,0 +1,149 @@
+---
+stage: Systems
+group: Distribution
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Incoming email Rake tasks **(FREE SELF)**
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/108279) in GitLab 15.9.
+
+The following are Incoming email-related Rake tasks.
+
+## Secrets
+
+GitLab can use [Incoming email](../incoming_email.md) secrets read from an encrypted file instead of storing them in plaintext in the file system. The following Rake tasks are provided for updating the contents of the encrypted file.
+
+### Show secret
+
+Show the contents of the current Incoming email secrets.
+
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
+
+```shell
+sudo gitlab-rake gitlab:incoming_email:secret:show
+```
+
+:::TabTitle Helm chart (Kubernetes)
+
+Use a Kubernetes secret to store the incoming email password. For more information,
+read about [Helm IMAP secrets](https://docs.gitlab.com/charts/installation/secrets.html#imap-password-for-incoming-emails).
+
+:::TabTitle Docker
+
+```shell
+sudo docker exec -t <container name> gitlab:incoming_email:secret:show
+```
+
+:::TabTitle Self-compiled (source)
+
+```shell
+bundle exec rake gitlab:incoming_email:secret:show RAILS_ENV=production
+```
+
+::EndTabs
+
+#### Example output
+
+```plaintext
+password: 'examplepassword'
+user: 'incoming-email@mail.example.com'
+```
+
+### Edit secret
+
+Opens the secret contents in your editor, and writes the resulting content to the encrypted secret file when you exit.
+
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
+
+```shell
+sudo gitlab-rake gitlab:incoming_email:secret:edit EDITOR=vim
+```
+
+:::TabTitle Helm chart (Kubernetes)
+
+Use a Kubernetes secret to store the incoming email password. For more information,
+read about [Helm IMAP secrets](https://docs.gitlab.com/charts/installation/secrets.html#imap-password-for-incoming-emails).
+
+:::TabTitle Docker
+
+```shell
+sudo docker exec -t <container name> gitlab:incoming_email:secret:edit EDITOR=editor
+```
+
+:::TabTitle Self-compiled (source)
+
+```shell
+bundle exec rake gitlab:incoming_email:secret:edit RAILS_ENV=production EDITOR=vim
+```
+
+::EndTabs
+
+### Write raw secret
+
+Write new secret content by providing it on `STDIN`.
+
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
+
+```shell
+echo -e "password: 'examplepassword'" | sudo gitlab-rake gitlab:incoming_email:secret:write
+```
+
+:::TabTitle Helm chart (Kubernetes)
+
+Use a Kubernetes secret to store the incoming email password. For more information,
+read about [Helm IMAP secrets](https://docs.gitlab.com/charts/installation/secrets.html#imap-password-for-incoming-emails).
+
+:::TabTitle Docker
+
+```shell
+sudo docker exec -t <container name> /bin/bash
+echo -e "password: 'examplepassword'" | gitlab-rake gitlab:incoming_email:secret:write
+```
+
+:::TabTitle Self-compiled (source)
+
+```shell
+echo -e "password: 'examplepassword'" | bundle exec rake gitlab:incoming_email:secret:write RAILS_ENV=production
+```
+
+::EndTabs
+
+### Secrets examples
+
+**Editor example**
+
+The write task can be used in cases where the edit command does not work with your editor:
+
+```shell
+# Write the existing secret to a plaintext file
+sudo gitlab-rake gitlab:incoming_email:secret:show > incoming_email.yaml
+# Edit the incoming_email file in your editor
+...
+# Re-encrypt the file
+cat incoming_email.yaml | sudo gitlab-rake gitlab:incoming_email:secret:write
+# Remove the plaintext file
+rm incoming_email.yaml
+```
+
+**KMS integration example**
+
+It can also be used as a receiving application for content encrypted with a KMS:
+
+```shell
+gcloud kms decrypt --key my-key --keyring my-test-kms --plaintext-file=- --ciphertext-file=my-file --location=us-west1 | sudo gitlab-rake gitlab:incoming_email:secret:write
+```
+
+**Google Cloud secret integration example**
+
+It can also be used as a receiving application for secrets out of Google Cloud:
+
+```shell
+gcloud secrets versions access latest --secret="my-test-secret" > $1 | sudo gitlab-rake gitlab:incoming_email:secret:write
+```
diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md
index ba095b33bf5..5c258d73fdb 100644
--- a/doc/administration/raketasks/maintenance.md
+++ b/doc/administration/raketasks/maintenance.md
@@ -234,8 +234,11 @@ sudo -u git -H bundle exec rake cache:clear RAILS_ENV=production
Sometimes during version upgrades you might end up with some wrong CSS or
missing some icons. In that case, try to precompile the assets again.
-This only applies to source installations and does not apply to
-Omnibus packages.
+This Rake task only applies to source installations. [Read more](../../update/package/index.md#missing-asset-files)
+about troubleshooting this problem when running the Omnibus GitLab package.
+The guidance for Omnibus GitLab might be applicable for Kubernetes and Docker Omnibus
+deployments of GitLab, though in general, container-based installations
+don't have issues with missing assets.
**Source Installation**
@@ -374,7 +377,7 @@ The following index types are not supported:
Optionally, this Rake task sends annotations to a Grafana (4.6 or later) endpoint. Use the following custom environment variables to enable annotations:
-1. `GRAFANA_API_URL` - Grafana's base URL, for example `http://some-host:3000`.
+1. `GRAFANA_API_URL` - The base URL for Grafana, for example `http://some-host:3000`.
1. `GRAFANA_API_KEY` - Grafana API key with at least `Editor role`.
You can also [enable reindexing as a regular cron job](https://docs.gitlab.com/omnibus/settings/database.html#automatic-database-reindexing).
diff --git a/doc/administration/raketasks/project_import_export.md b/doc/administration/raketasks/project_import_export.md
index e43fbac25e9..4694af18af2 100644
--- a/doc/administration/raketasks/project_import_export.md
+++ b/doc/administration/raketasks/project_import_export.md
@@ -42,7 +42,7 @@ bundle exec rake gitlab:import_export:data RAILS_ENV=production
Note the following:
- Importing is only possible if the version of the import and export GitLab instances are
- compatible as described in the [Version history](../../user/project/settings/import_export.md#version-history).
+ [compatible](../../user/project/settings/import_export.md#compatibility).
- The project import option must be enabled:
1. On the top bar, select **Main menu > Admin**.
diff --git a/doc/administration/raketasks/service_desk_email.md b/doc/administration/raketasks/service_desk_email.md
new file mode 100644
index 00000000000..10de379b1cd
--- /dev/null
+++ b/doc/administration/raketasks/service_desk_email.md
@@ -0,0 +1,149 @@
+---
+stage: Systems
+group: Distribution
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Service Desk email Rake tasks **(FREE SELF)**
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/108279) in GitLab 15.9.
+
+The following are Service Desk email-related Rake tasks.
+
+## Secrets
+
+GitLab can use [Service Desk email](../../user/project/service_desk.md#configuring-a-custom-mailbox) secrets read from an encrypted file instead of storing them in plaintext in the file system. The following Rake tasks are provided for updating the contents of the encrypted file.
+
+### Show secret
+
+Show the contents of the current Service Desk email secrets.
+
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
+
+```shell
+sudo gitlab-rake gitlab:service_desk_email:secret:show
+```
+
+:::TabTitle Helm chart (Kubernetes)
+
+Use a Kubernetes secret to store the Service Desk email password. For more information,
+read about [Helm IMAP secrets](https://docs.gitlab.com/charts/installation/secrets.html#imap-password-for-service-desk-emails).
+
+:::TabTitle Docker
+
+```shell
+sudo docker exec -t <container name> gitlab:service_desk_email:secret:show
+```
+
+:::TabTitle Self-compiled (source)
+
+```shell
+bundle exec rake gitlab:service_desk_email:secret:show RAILS_ENV=production
+```
+
+::EndTabs
+
+#### Example output
+
+```plaintext
+password: 'examplepassword'
+user: 'service-desk-email@mail.example.com'
+```
+
+### Edit secret
+
+Opens the secret contents in your editor, and writes the resulting content to the encrypted secret file when you exit.
+
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
+
+```shell
+sudo gitlab-rake gitlab:service_desk_email:secret:edit EDITOR=vim
+```
+
+:::TabTitle Helm chart (Kubernetes)
+
+Use a Kubernetes secret to store the Service Desk email password. For more information,
+read about [Helm IMAP secrets](https://docs.gitlab.com/charts/installation/secrets.html#imap-password-for-service-desk-emails).
+
+:::TabTitle Docker
+
+```shell
+sudo docker exec -t <container name> gitlab:service_desk_email:secret:edit EDITOR=editor
+```
+
+:::TabTitle Self-compiled (source)
+
+```shell
+bundle exec rake gitlab:service_desk_email:secret:edit RAILS_ENV=production EDITOR=vim
+```
+
+::EndTabs
+
+### Write raw secret
+
+Write new secret content by providing it on `STDIN`.
+
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
+
+```shell
+echo -e "password: 'examplepassword'" | sudo gitlab-rake gitlab:service_desk_email:secret:write
+```
+
+:::TabTitle Helm chart (Kubernetes)
+
+Use a Kubernetes secret to store the Service Desk email password. For more information,
+read about [Helm IMAP secrets](https://docs.gitlab.com/charts/installation/secrets.html#imap-password-for-service-desk-emails).
+
+:::TabTitle Docker
+
+```shell
+sudo docker exec -t <container name> /bin/bash
+echo -e "password: 'examplepassword'" | gitlab-rake gitlab:service_desk_email:secret:write
+```
+
+:::TabTitle Self-compiled (source)
+
+```shell
+echo -e "password: 'examplepassword'" | bundle exec rake gitlab:service_desk_email:secret:write RAILS_ENV=production
+```
+
+::EndTabs
+
+### Secrets examples
+
+**Editor example**
+
+The write task can be used in cases where the edit command does not work with your editor:
+
+```shell
+# Write the existing secret to a plaintext file
+sudo gitlab-rake gitlab:service_desk_email:secret:show > service_desk_email.yaml
+# Edit the service_desk_email file in your editor
+...
+# Re-encrypt the file
+cat service_desk_email.yaml | sudo gitlab-rake gitlab:service_desk_email:secret:write
+# Remove the plaintext file
+rm service_desk_email.yaml
+```
+
+**KMS integration example**
+
+It can also be used as a receiving application for content encrypted with a KMS:
+
+```shell
+gcloud kms decrypt --key my-key --keyring my-test-kms --plaintext-file=- --ciphertext-file=my-file --location=us-west1 | sudo gitlab-rake gitlab:service_desk_email:secret:write
+```
+
+**Google Cloud secret integration example**
+
+It can also be used as a receiving application for secrets out of Google Cloud:
+
+```shell
+gcloud secrets versions access latest --secret="my-test-secret" > $1 | sudo gitlab-rake gitlab:service_desk_email:secret:write
+```
diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md
index bb50f66aff0..529621813aa 100644
--- a/doc/administration/reference_architectures/10k_users.md
+++ b/doc/administration/reference_architectures/10k_users.md
@@ -50,7 +50,9 @@ full list of reference architectures, see
- [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work.
4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section.
5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`.
-6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to the [Large Repositories](#large-repositories) for more info.
+6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large
+ repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to
+ [Large repositories](index.md#large-repositories) for more information.
<!-- markdownlint-enable MD029 -->
NOTE:
@@ -144,66 +146,7 @@ monitor .[#7FFFD4,norank]u--> elb
## Requirements
-Before starting, you should take note of the following requirements / guidance for this reference architecture.
-
-### Supported CPUs
-
-This reference architecture was built and tested on Google Cloud Platform (GCP) using the
-[Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms)
-CPU platform as a baseline ([Sysbench benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks)).
-
-Newer, similarly sized CPUs are supported and may have improved performance as a result. For Omnibus environments, ARM-based equivalents are also supported.
-
-NOTE:
-Any "burstable" instance types are not recommended due to inconsistent performance.
-
-### Supported infrastructure
-
-As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP) and their services,
-or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section.
-However, this does not constitute a guarantee for every potential permutation.
-
-See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
-
-### Additional workloads
-
-The Reference Architectures have been [designed and tested](index.md#validation-and-test-results) for standard GitLab setups with
-good headroom in mind to cover most scenarios. However, if any additional workloads are being added on the nodes,
-such as security software, you may still need to adjust the specs accordingly to compensate.
-
-This also applies for some GitLab features where it's possible to run custom scripts, for example [server hooks](../server_hooks.md).
-
-As a general rule, it's recommended to have robust monitoring in place to measure the impact of
-any additional workloads to inform any changes needed to be made.
-
-### Large repositories
-
-The Reference Architectures were tested with repositories of varying sizes that follow best practices.
-
-However, large repositories or monorepos (several gigabytes or more) can **significantly** impact the performance
-of Git and in turn the environment itself if best practices aren't being followed such as not storing
-binary or blob files in LFS. Repositories are at the core of any environment the consequences can be wide-ranging
-when they are not optimized. Some examples of this impact include [Git packing operations](https://git-scm.com/book/en/v2/Git-Internals-Packfiles)
-taking longer and consuming high CPU / Memory resources or Git checkouts taking longer that affect both users and
-CI pipelines alike.
-
-As such, large repositories come with notable cost and typically will require more resources to handle,
-significantly so in some cases. It's therefore **strongly** recommended then to review large repositories
-to ensure they maintain good health and reduce their size wherever possible.
-
-NOTE:
-If best practices aren't followed and large repositories are present on the environment,
-increased Gitaly specs may be required to ensure stable performance.
-
-Refer to the [Managing large repositories documentation](../../user/project/repository/managing_large_repositories.md)
-for more information and guidance.
-
-### Praefect PostgreSQL
-
-It's worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and
-that to achieve full High Availability a third-party PostgreSQL database solution will be required.
-We hope to offer a built in solutions for these restrictions in the future but in the meantime a non HA PostgreSQL server
-can be set up via Omnibus GitLab, which the above specs reflect. Refer to the following issues for more information: [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398).
+Before starting, see the [requirements](index.md#requirements) for reference architectures.
## Setup components
@@ -710,7 +653,7 @@ in the second step, do not supply the `EXTERNAL_URL` value.
PostgreSQL, with Patroni managing its failover, will default to use `pg_rewind` by default to handle conflicts.
Like most failover handling methods, this has a small chance of leading to data loss.
-Learn more about the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method).
+For more information, see the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method).
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
@@ -1209,18 +1152,25 @@ are supported and can be added if needed.
## Configure Gitaly Cluster
-[Gitaly Cluster](../gitaly/praefect.md) is a GitLab provided and recommended fault tolerant solution for storing Git repositories.
-In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down.
+[Gitaly Cluster](../gitaly/praefect.md) is a GitLab-provided and recommended fault tolerant solution for storing Git
+repositories. In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being
+designated the primary, and failover occurs automatically if the primary node goes down.
-NOTE:
-Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster).
-For implementations with sharded Gitaly, use the same Gitaly specs. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section.
+Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management.
+Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster).
+
+For guidance on:
+
+- Implementing sharded Gitaly instead, follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md)
+ instead of this section. Use the same Gitaly specs.
+- Migrating existing repositories that aren't managed by Gitaly Cluster, see
+ [migrate to Gitaly Cluster](../gitaly/index.md#migrate-to-gitaly-cluster).
NOTE:
Gitaly has been designed and tested with repositories of varying sizes that follow best practices.
However, large repositories or monorepos not following these practices can significantly
impact Gitaly performance and requirements.
-Refer to the [Large Repositories](#large-repositories) for more info.
+Refer to [Large repositories](index.md#large-repositories) for more information.
The recommended cluster setup includes the following components:
@@ -1229,9 +1179,9 @@ The recommended cluster setup includes the following components:
- 1 Praefect PostgreSQL node: Database server for Praefect. A third-party solution
is required for Praefect database connections to be made highly available.
- 1 load balancer: A load balancer is required for Praefect. The
- [internal load balancer](#configure-the-internal-load-balancer) will be used.
+ [internal load balancer](#configure-the-internal-load-balancer) is used.
-This section will detail how to configure the recommended standard setup in order.
+This section details how to configure the recommended standard setup in order.
For more advanced setups refer to the [standalone Gitaly Cluster documentation](../gitaly/praefect.md).
### Configure Praefect PostgreSQL
@@ -1530,14 +1480,14 @@ requirements that are dependent on data and load.
NOTE:
Increased specs for Gitaly nodes may be required in some circumstances such as
-significantly large repositories or if any [additional workloads](#additional-workloads),
+significantly large repositories or if any [additional workloads](index.md#additional-workloads),
such as [server hooks](../server_hooks.md), have been added.
NOTE:
Gitaly has been designed and tested with repositories of varying sizes that follow best practices.
However, large repositories or monorepos not following these practices can significantly
impact Gitaly performance and requirements.
-Refer to the [Large Repositories](#large-repositories) for more info.
+Refer to [large repositories](index.md#large-repositories) for more information.
Due to Gitaly having notable input and output requirements, we strongly
recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs
@@ -1756,7 +1706,10 @@ To configure Praefect with TLS:
Sidekiq requires connection to the [Redis](#configure-redis),
[PostgreSQL](#configure-postgresql) and [Gitaly](#configure-gitaly) instances.
-Because you must use [Object storage](#configure-the-object-storage) instead of NFS for data objects, the following
+It also requires a connection to [Object Storage](#configure-the-object-storage) as recommended.
+
+NOTE:
+[Because it's recommended to use Object storage](../object_storage.md) instead of NFS for data objects, the following
examples include the Object storage configuration.
- `10.6.0.101`: Sidekiq 1
@@ -1925,7 +1878,13 @@ run [multiple Sidekiq processes](../sidekiq/extra_sidekiq_processes.md).
## Configure GitLab Rails
This section describes how to configure the GitLab application (Rails) component.
-Because you must use [Object storage](#configure-the-object-storage) instead of NFS for data objects, the following
+
+Rails requires connections to the [Redis](#configure-redis),
+[PostgreSQL](#configure-postgresql) and [Gitaly](#configure-gitaly) instances.
+It also requires a connection to [Object Storage](#configure-the-object-storage) as recommended.
+
+NOTE:
+[Because it's recommended to use Object storage](../object_storage.md) instead of NFS for data objects, the following
examples include the Object storage configuration.
The following IPs will be used as an example:
@@ -2065,6 +2024,10 @@ On each node perform the following:
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+1. Copy the SSH host keys (all in the name format `/etc/ssh/ssh_host_*_key*`) from the first Omnibus node you configured and
+ add or replace the files of the same name on this server. This ensures host mismatch errors aren't thrown
+ for your users as they hit the load balanced Rails nodes. If this is the first Omnibus node you are configuring,
+ then you can skip this step.
1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run:
```shell
@@ -2200,16 +2163,10 @@ To configure the Monitoring node:
## Configure the object storage
-GitLab supports using an object storage service for holding numerous types of data.
-
-GitLab has been tested on a number of object storage providers:
-
-- [Amazon S3](https://aws.amazon.com/s3/)
-- [Google Cloud Storage](https://cloud.google.com/storage)
-- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces)
-- [Oracle Cloud Infrastructure](https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm)
-- [OpenStack Swift (S3 compatibility mode)](https://docs.openstack.org/swift/latest/s3_compat.html)
-- MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation.
+GitLab supports using an [object storage](../object_storage.md) service for holding numerous types of data.
+It's recommended over [NFS](../nfs.md) for data objects and in general it's better
+in larger setups as object storage is typically much more performant, reliable,
+and scalable.
There are two ways of specifying object storage configuration in GitLab:
@@ -2336,7 +2293,9 @@ services where applicable):
- [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work.
4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section.
5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`.
-6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to the [Large Repositories](#large-repositories) for more info.
+6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large
+ repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to
+ [Large repositories](index.md#large-repositories) for more information.
<!-- markdownlint-enable MD029 -->
NOTE:
diff --git a/doc/administration/reference_architectures/1k_users.md b/doc/administration/reference_architectures/1k_users.md
index 2a9636b6e05..d3aa6fef51e 100644
--- a/doc/administration/reference_architectures/1k_users.md
+++ b/doc/administration/reference_architectures/1k_users.md
@@ -10,9 +10,9 @@ This page describes GitLab reference architecture for up to 1,000 users. For a
full list of reference architectures, see
[Available reference architectures](index.md#available-reference-architectures).
-If you are serving up to 1,000 users and you don't have strict availability
-requirements, a single-node solution with
-[frequent backups](index.md#backups) is appropriate for
+If you are serving up to 1,000 users, and you don't have strict availability
+requirements, a [standalone](index.md#standalone-non-ha) single-node solution with
+frequent backups is appropriate for
many organizations.
> - **Supported users (approximate):** 1,000
@@ -67,47 +67,7 @@ The diagram above shows that while GitLab can be installed on a single server, i
## Requirements
-Before starting, you should take note of the following requirements / guidance for this reference architecture.
-
-### Supported CPUs
-
-This reference architecture was built and tested on Google Cloud Platform (GCP) using the
-[Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms)
-CPU platform as a baseline ([Sysbench benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks)).
-
-Newer, similarly sized CPUs are supported and may have improved performance as a result. For Omnibus environments, ARM-based equivalents are also supported.
-
-NOTE:
-Any "burstable" instance types are not recommended due to inconsistent performance.
-
-### Supported infrastructure
-
-As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP) and their services,
-or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section.
-However, this does not constitute a guarantee for every potential permutation.
-
-See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
-
-### Additional workloads
-
-The Reference Architectures have been [designed and tested](index.md#validation-and-test-results) for standard GitLab setups with
-good headroom in mind to cover most scenarios. However, if any additional workloads are being added on the nodes,
-such as security software, you may still need to adjust the specs accordingly to compensate.
-
-This also applies for some GitLab features where it's possible to run custom scripts, for example [server hooks](../server_hooks.md).
-
-As a general rule, it's recommended to have robust monitoring in place to measure the impact of
-any additional workloads to inform any changes needed to be made.
-
-### Swap
-
-In addition to the stated configurations, we recommend having at least 2 GB of
-swap on your server, even if you currently have enough available memory. Having
-swap helps to reduce the chance of errors occurring if your available memory
-changes. We also recommend configuring the kernel's swappiness setting to a
-lower value (such as `10`) to make the most of your memory, while still having
-the swap available when needed. View the
-[Memory requirements](../../install/requirements.md#memory) for details.
+Before starting, see the [requirements](index.md#requirements) for reference architectures.
## Setup instructions
diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md
index 02739904f5e..71fe8b301e2 100644
--- a/doc/administration/reference_architectures/25k_users.md
+++ b/doc/administration/reference_architectures/25k_users.md
@@ -50,7 +50,9 @@ full list of reference architectures, see
- [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work.
4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section.
5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`.
-6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to the [Large Repositories](#large-repositories) for more info.
+6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large
+ repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to
+ [Large repositories](index.md#large-repositories) for more information.
<!-- markdownlint-enable MD029 -->
NOTE:
@@ -144,66 +146,7 @@ monitor .[#7FFFD4,norank]u--> elb
## Requirements
-Before starting, you should take note of the following requirements / guidance for this reference architecture.
-
-### Supported CPUs
-
-This reference architecture was built and tested on Google Cloud Platform (GCP) using the
-[Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms)
-CPU platform as a baseline ([Sysbench benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks)).
-
-Newer, similarly sized CPUs are supported and may have improved performance as a result. For Omnibus environments, ARM-based equivalents are also supported.
-
-NOTE:
-Any "burstable" instance types are not recommended due to inconsistent performance.
-
-### Supported infrastructure
-
-As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP, Azure) and their services,
-or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section.
-However, this does not constitute a guarantee for every potential permutation.
-
-See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
-
-### Additional workloads
-
-The Reference Architectures have been [designed and tested](index.md#validation-and-test-results) for standard GitLab setups with
-good headroom in mind to cover most scenarios. However, if any additional workloads are being added on the nodes,
-such as security software, you may still need to adjust the specs accordingly to compensate.
-
-This also applies for some GitLab features where it's possible to run custom scripts, for example [server hooks](../server_hooks.md).
-
-As a general rule, it's recommended to have robust monitoring in place to measure the impact of
-any additional workloads to inform any changes needed to be made.
-
-### Large repositories
-
-The Reference Architectures were tested with repositories of varying sizes that follow best practices.
-
-However, large repositories or monorepos (several gigabytes or more) can **significantly** impact the performance
-of Git and in turn the environment itself if best practices aren't being followed such as not storing
-binary or blob files in LFS. Repositories are at the core of any environment the consequences can be wide-ranging
-when they are not optimized. Some examples of this impact include [Git packing operations](https://git-scm.com/book/en/v2/Git-Internals-Packfiles)
-taking longer and consuming high CPU / Memory resources or Git checkouts taking longer that affect both users and
-CI pipelines alike.
-
-As such, large repositories come with notable cost and typically will require more resources to handle,
-significantly so in some cases. It's therefore **strongly** recommended then to review large repositories
-to ensure they maintain good health and reduce their size wherever possible.
-
-NOTE:
-If best practices aren't followed and large repositories are present on the environment,
-increased Gitaly specs may be required to ensure stable performance.
-
-Refer to the [Managing large repositories documentation](../../user/project/repository/managing_large_repositories.md)
-for more information and guidance.
-
-### Praefect PostgreSQL
-
-It's worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and
-that to achieve full High Availability a third-party PostgreSQL database solution will be required.
-We hope to offer a built in solutions for these restrictions in the future but in the meantime a non HA PostgreSQL server
-can be set up via Omnibus GitLab, which the above specs reflect. Refer to the following issues for more information: [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398).
+Before starting, see the [requirements](index.md#requirements) for reference architectures.
## Setup components
@@ -727,7 +670,7 @@ in the second step, do not supply the `EXTERNAL_URL` value.
PostgreSQL, with Patroni managing its failover, will default to use `pg_rewind` by default to handle conflicts.
Like most failover handling methods, this has a small chance of leading to data loss.
-Learn more about the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method).
+For more information, see the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method).
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
@@ -1228,19 +1171,25 @@ are supported and can be added if needed.
## Configure Gitaly Cluster
-[Gitaly Cluster](../gitaly/praefect.md) is a GitLab-provided and recommended
-fault tolerant solution for storing Git repositories.
-In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down.
+[Gitaly Cluster](../gitaly/praefect.md) is a GitLab-provided and recommended fault tolerant solution for storing Git
+repositories. In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being
+designated the primary, and failover occurs automatically if the primary node goes down.
-NOTE:
-Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster).
-For implementations with sharded Gitaly, use the same Gitaly specs. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section.
+Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management.
+Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster).
+
+For guidance on:
+
+- Implementing sharded Gitaly instead, follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md)
+ instead of this section. Use the same Gitaly specs.
+- Migrating existing repositories that aren't managed by Gitaly Cluster, see
+ [migrate to Gitaly Cluster](../gitaly/index.md#migrate-to-gitaly-cluster).
NOTE:
Gitaly has been designed and tested with repositories of varying sizes that follow best practices.
However, large repositories or monorepos not following these practices can significantly
impact Gitaly performance and requirements.
-Refer to the [Large Repositories](#large-repositories) for more info.
+Refer to [Large repositories](index.md#large-repositories) for more information.
The recommended cluster setup includes the following components:
@@ -1249,9 +1198,9 @@ The recommended cluster setup includes the following components:
- 1 Praefect PostgreSQL node: Database server for Praefect. A third-party solution
is required for Praefect database connections to be made highly available.
- 1 load balancer: A load balancer is required for Praefect. The
- [internal load balancer](#configure-the-internal-load-balancer) will be used.
+ [internal load balancer](#configure-the-internal-load-balancer) is used.
-This section will detail how to configure the recommended standard setup in order.
+This section details how to configure the recommended standard setup in order.
For more advanced setups refer to the [standalone Gitaly Cluster documentation](../gitaly/praefect.md).
### Configure Praefect PostgreSQL
@@ -1548,14 +1497,14 @@ requirements that are dependent on data and load.
NOTE:
Increased specs for Gitaly nodes may be required in some circumstances such as
-significantly large repositories or if any [additional workloads](#additional-workloads),
+significantly large repositories or if any [additional workloads](index.md#additional-workloads),
such as [server hooks](../server_hooks.md), have been added.
NOTE:
Gitaly has been designed and tested with repositories of varying sizes that follow best practices.
However, large repositories or monorepos not following these practices can significantly
impact Gitaly performance and requirements.
-Refer to the [Large Repositories](#large-repositories) for more info.
+Refer to [Large repositories](index.md#large-repositories) for more information.
Due to Gitaly having notable input and output requirements, we strongly
recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs
@@ -1774,7 +1723,10 @@ To configure Praefect with TLS:
Sidekiq requires connection to the [Redis](#configure-redis),
[PostgreSQL](#configure-postgresql) and [Gitaly](#configure-gitaly) instances.
-Because you must use [Object storage](#configure-the-object-storage) instead of NFS for data objects, the following
+It also requires a connection to [Object Storage](#configure-the-object-storage) as recommended.
+
+NOTE:
+[Because it's recommended to use Object storage](../object_storage.md) instead of NFS for data objects, the following
examples include the Object storage configuration.
- `10.6.0.101`: Sidekiq 1
@@ -1943,7 +1895,13 @@ run [multiple Sidekiq processes](../sidekiq/extra_sidekiq_processes.md).
## Configure GitLab Rails
This section describes how to configure the GitLab application (Rails) component.
-Because you must use [Object storage](#configure-the-object-storage) instead of NFS for data objects, the following
+
+Rails requires connections to the [Redis](#configure-redis),
+[PostgreSQL](#configure-postgresql) and [Gitaly](#configure-gitaly) instances.
+It also requires a connection to [Object Storage](#configure-the-object-storage) as recommended.
+
+NOTE:
+[Because it's recommended to use Object storage](../object_storage.md) instead of NFS for data objects, the following
examples include the Object storage configuration.
The following IPs will be used as an example:
@@ -2085,6 +2043,10 @@ On each node perform the following:
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+1. Copy the SSH host keys (all in the name format `/etc/ssh/ssh_host_*_key*`) from the first Omnibus node you configured and
+ add or replace the files of the same name on this server. This ensures host mismatch errors aren't thrown
+ for your users as they hit the load balanced Rails nodes. If this is the first Omnibus node you are configuring,
+ then you can skip this step.
1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run:
```shell
@@ -2219,16 +2181,10 @@ To configure the Monitoring node:
## Configure the object storage
-GitLab supports using an object storage service for holding numerous types of data.
-
-GitLab has been tested on a number of object storage providers:
-
-- [Amazon S3](https://aws.amazon.com/s3/)
-- [Google Cloud Storage](https://cloud.google.com/storage)
-- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces)
-- [Oracle Cloud Infrastructure](https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm)
-- [OpenStack Swift (S3 compatibility mode)](https://docs.openstack.org/swift/latest/s3_compat.html)
-- MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation.
+GitLab supports using an [object storage](../object_storage.md) service for holding numerous types of data.
+It's recommended over [NFS](../nfs.md) for data objects and in general it's better
+in larger setups as object storage is typically much more performant, reliable,
+and scalable.
There are two ways of specifying object storage configuration in GitLab:
@@ -2355,7 +2311,9 @@ services where applicable):
- [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work.
4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section.
5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`.
-6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to the [Large Repositories](#large-repositories) for more info.
+6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large
+ repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to
+ [Large repositories](index.md#large-repositories) for more information.
<!-- markdownlint-enable MD029 -->
NOTE:
diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md
index f41c8e9cb24..ee26504902c 100644
--- a/doc/administration/reference_architectures/2k_users.md
+++ b/doc/administration/reference_architectures/2k_users.md
@@ -42,11 +42,13 @@ For a full list of reference architectures, see
3. Can be optionally run on reputable third-party load balancing services (LB PaaS). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
- [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work.
4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section.
-5. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to the [Large Repositories](#large-repositories) for more info.
+5. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large
+ repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to
+ [Large repositories](index.md#large-repositories) for more information.
<!-- markdownlint-enable MD029 -->
NOTE:
-For all PaaS solutions that involve configuring instances, it is strongly recommended to implement a minimum of three nodes in three different availability zones to align with resilient cloud architecture practices.
+For all PaaS solutions that involve configuring instances, it's recommended to deploy them over multiple availability zones for resilience if desired.
```plantuml
@startuml 2k
@@ -80,59 +82,7 @@ monitor .[#7FFFD4,norank]u--> elb
## Requirements
-Before starting, you should take note of the following requirements / guidance for this reference architecture.
-
-### Supported CPUs
-
-This reference architecture was built and tested on Google Cloud Platform (GCP) using the
-[Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms)
-CPU platform as a baseline ([Sysbench benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks)).
-
-Newer, similarly sized CPUs are supported and may have improved performance as a result. For Omnibus environments, ARM-based equivalents are also supported.
-
-NOTE:
-Any "burstable" instance types are not recommended due to inconsistent performance.
-
-### Supported infrastructure
-
-As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP) and their services,
-or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section.
-However, this does not constitute a guarantee for every potential permutation.
-
-See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
-
-### Additional workloads
-
-The Reference Architectures have been [designed and tested](index.md#validation-and-test-results) for standard GitLab setups with
-good headroom in mind to cover most scenarios. However, if any additional workloads are being added on the nodes,
-such as security software, you may still need to adjust the specs accordingly to compensate.
-
-This also applies for some GitLab features where it's possible to run custom scripts, for example [server hooks](../server_hooks.md).
-
-As a general rule, it's recommended to have robust monitoring in place to measure the impact of
-any additional workloads to inform any changes needed to be made.
-
-### Large repositories
-
-The Reference Architectures were tested with repositories of varying sizes that follow best practices.
-
-However, large repositories or monorepos (several gigabytes or more) can **significantly** impact the performance
-of Git and in turn the environment itself if best practices aren't being followed such as not storing
-binary or blob files in LFS. Repositories are at the core of any environment the consequences can be wide-ranging
-when they are not optimized. Some examples of this impact include [Git packing operations](https://git-scm.com/book/en/v2/Git-Internals-Packfiles)
-taking longer and consuming high CPU / Memory resources or Git checkouts taking longer that affect both users and
-CI pipelines alike.
-
-As such, large repositories come with notable cost and typically will require more resources to handle,
-significantly so in some cases. It's therefore **strongly** recommended then to review large repositories
-to ensure they maintain good health and reduce their size wherever possible.
-
-NOTE:
-If best practices aren't followed and large repositories are present on the environment,
-increased Gitaly specs may be required to ensure stable performance.
-
-Refer to the [Managing large repositories documentation](../../user/project/repository/managing_large_repositories.md)
-for more information and guidance.
+Before starting, see the [requirements](index.md#requirements) for reference architectures.
## Setup components
@@ -460,14 +410,14 @@ specifically the number of projects and those projects' sizes.
NOTE:
Increased specs for Gitaly nodes may be required in some circumstances such as
-significantly large repositories or if any [additional workloads](#additional-workloads),
+significantly large repositories or if any [additional workloads](index.md#additional-workloads),
such as [server hooks](../server_hooks.md), have been added.
NOTE:
Gitaly has been designed and tested with repositories of varying sizes that follow best practices.
However, large repositories or monorepos not following these practices can significantly
impact Gitaly performance and requirements.
-Refer to the [Large Repositories](#large-repositories) for more info.
+Refer to [Large repositories](index.md#large-repositories) for more information.
Due to Gitaly having notable input and output requirements, we strongly
recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs
@@ -924,17 +874,10 @@ running [Prometheus](../monitoring/prometheus/index.md) and
## Configure the object storage
-GitLab supports using an object storage service for holding numerous types of data.
-
-GitLab has been tested on a number of object storage providers:
-
-- [Amazon S3](https://aws.amazon.com/s3/)
-- [Google Cloud Storage](https://cloud.google.com/storage)
-- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces)
-- [Oracle Cloud Infrastructure](https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm)
-- [OpenStack Swift (S3 compatibility mode)](https://docs.openstack.org/swift/latest/s3_compat.html)
-- [Azure Blob storage](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction)
-- MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation.
+GitLab supports using an [object storage](../object_storage.md) service for holding numerous types of data.
+It's recommended over [NFS](../nfs.md) for data objects and in general it's better
+in larger setups as object storage is typically much more performant, reliable,
+and scalable.
There are two ways of specifying object storage configuration in GitLab:
diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md
index 008b5ffcc0e..1d5dad02b18 100644
--- a/doc/administration/reference_architectures/3k_users.md
+++ b/doc/administration/reference_architectures/3k_users.md
@@ -59,7 +59,9 @@ For a full list of reference architectures, see
- [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work.
4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section.
5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`.
-6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to the [Large Repositories](#large-repositories) for more info.
+6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large
+ repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to
+ [Large repositories](index.md#large-repositories) for more information.
<!-- markdownlint-enable MD029 -->
NOTE:
@@ -150,66 +152,7 @@ monitor .[#7FFFD4,norank]u--> elb
## Requirements
-Before starting, you should take note of the following requirements / guidance for this reference architecture.
-
-### Supported CPUs
-
-This reference architecture was built and tested on Google Cloud Platform (GCP) using the
-[Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms)
-CPU platform as a baseline ([Sysbench benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks)).
-
-Newer, similarly sized CPUs are supported and may have improved performance as a result. For Omnibus environments, ARM-based equivalents are also supported.
-
-NOTE:
-Any "burstable" instance types are not recommended due to inconsistent performance.
-
-### Supported infrastructure
-
-As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP) and their services,
-or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section.
-However, this does not constitute a guarantee for every potential permutation.
-
-See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
-
-### Additional workloads
-
-The Reference Architectures have been [designed and tested](index.md#validation-and-test-results) for standard GitLab setups with
-good headroom in mind to cover most scenarios. However, if any additional workloads are being added on the nodes,
-such as security software, you may still need to adjust the specs accordingly to compensate.
-
-This also applies for some GitLab features where it's possible to run custom scripts, for example [server hooks](../server_hooks.md).
-
-As a general rule, it's recommended to have robust monitoring in place to measure the impact of
-any additional workloads to inform any changes needed to be made.
-
-### Large repositories
-
-The Reference Architectures were tested with repositories of varying sizes that follow best practices.
-
-However, large repositories or monorepos (several gigabytes or more) can **significantly** impact the performance
-of Git and in turn the environment itself if best practices aren't being followed such as not storing
-binary or blob files in LFS. Repositories are at the core of any environment the consequences can be wide-ranging
-when they are not optimized. Some examples of this impact include [Git packing operations](https://git-scm.com/book/en/v2/Git-Internals-Packfiles)
-taking longer and consuming high CPU / Memory resources or Git checkouts taking longer that affect both users and
-CI pipelines alike.
-
-As such, large repositories come with notable cost and typically will require more resources to handle,
-significantly so in some cases. It's therefore **strongly** recommended then to review large repositories
-to ensure they maintain good health and reduce their size wherever possible.
-
-NOTE:
-If best practices aren't followed and large repositories are present on the environment,
-increased Gitaly specs may be required to ensure stable performance.
-
-Refer to the [Managing large repositories documentation](../../user/project/repository/managing_large_repositories.md)
-for more information and guidance.
-
-### Praefect PostgreSQL
-
-It's worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and
-that to achieve full High Availability a third-party PostgreSQL database solution will be required.
-We hope to offer a built in solutions for these restrictions in the future but in the meantime a non HA PostgreSQL server
-can be set up via Omnibus GitLab, which the above specs reflect. Refer to the following issues for more information: [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398).
+Before starting, see the [requirements](index.md#requirements) for reference architectures.
## Setup components
@@ -1003,7 +946,7 @@ in the second step, do not supply the `EXTERNAL_URL` value.
PostgreSQL, with Patroni managing its failover, will default to use `pg_rewind` by default to handle conflicts.
Like most failover handling methods, this has a small chance of leading to data loss.
-Learn more about the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method).
+For more information, see the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method).
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
@@ -1164,18 +1107,25 @@ The following IPs will be used as an example:
## Configure Gitaly Cluster
-[Gitaly Cluster](../gitaly/praefect.md) is a GitLab provided and recommended fault tolerant solution for storing Git repositories.
-In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down.
+[Gitaly Cluster](../gitaly/praefect.md) is a GitLab-provided and recommended fault tolerant solution for storing Git
+repositories. In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being
+designated the primary, and failover occurs automatically if the primary node goes down.
-NOTE:
-Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster).
-For implementations with sharded Gitaly, use the same Gitaly specs. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section.
+Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management.
+Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster).
+
+For guidance on:
+
+- Implementing sharded Gitaly instead, follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md)
+ instead of this section. Use the same Gitaly specs.
+- Migrating existing repositories that aren't managed by Gitaly Cluster, see
+ [migrate to Gitaly Cluster](../gitaly/index.md#migrate-to-gitaly-cluster).
NOTE:
Gitaly has been designed and tested with repositories of varying sizes that follow best practices.
However, large repositories or monorepos not following these practices can significantly
impact Gitaly performance and requirements.
-Refer to the [Large Repositories](#large-repositories) for more info.
+Refer to [Large repositories](index.md#large-repositories) for more information.
The recommended cluster setup includes the following components:
@@ -1184,9 +1134,9 @@ The recommended cluster setup includes the following components:
- 1 Praefect PostgreSQL node: Database server for Praefect. A third-party solution
is required for Praefect database connections to be made highly available.
- 1 load balancer: A load balancer is required for Praefect. The
- [internal load balancer](#configure-the-internal-load-balancer) will be used.
+ [internal load balancer](#configure-the-internal-load-balancer) is used.
-This section will detail how to configure the recommended standard setup in order.
+This section details how to configure the recommended standard setup in order.
For more advanced setups refer to the [standalone Gitaly Cluster documentation](../gitaly/praefect.md).
### Configure Praefect PostgreSQL
@@ -1482,14 +1432,14 @@ requirements that are dependent on data and load.
NOTE:
Increased specs for Gitaly nodes may be required in some circumstances such as
-significantly large repositories or if any [additional workloads](#additional-workloads),
+significantly large repositories or if any [additional workloads](index.md#additional-workloads),
such as [server hooks](../server_hooks.md), have been added.
NOTE:
Gitaly has been designed and tested with repositories of varying sizes that follow best practices.
However, large repositories or monorepos not following these practices can significantly
impact Gitaly performance and requirements.
-Refer to the [Large Repositories](#large-repositories) for more info.
+Refer to the [Large repositories](index.md#large-repositories) for more information.
Due to Gitaly having notable input and output requirements, we strongly
recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs
@@ -1708,7 +1658,10 @@ To configure Praefect with TLS:
Sidekiq requires connection to the [Redis](#configure-redis),
[PostgreSQL](#configure-postgresql) and [Gitaly](#configure-gitaly) instances.
-Because you must use [Object storage](#configure-the-object-storage) instead of NFS for data objects, the following
+It also requires a connection to [Object Storage](#configure-the-object-storage) as recommended.
+
+NOTE:
+[Because it's recommended to use Object storage](../object_storage.md) instead of NFS for data objects, the following
examples include the Object storage configuration.
The following IPs will be used as an example:
@@ -1876,7 +1829,13 @@ run [multiple Sidekiq processes](../sidekiq/extra_sidekiq_processes.md).
## Configure GitLab Rails
This section describes how to configure the GitLab application (Rails) component.
-Because you must use [Object storage](#configure-the-object-storage) instead of NFS for data objects, the following
+
+Rails requires connections to the [Redis](#configure-redis),
+[PostgreSQL](#configure-postgresql) and [Gitaly](#configure-gitaly) instances.
+It also requires a connection to [Object Storage](#configure-the-object-storage) as recommended.
+
+NOTE:
+[Because it's recommended to use Object storage](../object_storage.md) instead of NFS for data objects, the following
examples include the Object storage configuration.
On each node perform the following:
@@ -2016,6 +1975,10 @@ On each node perform the following:
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+1. Copy the SSH host keys (all in the name format `/etc/ssh/ssh_host_*_key*`) from the first Omnibus node you configured and
+ add or replace the files of the same name on this server. This ensures host mismatch errors aren't thrown
+ for your users as they hit the load balanced Rails nodes. If this is the first Omnibus node you are configuring,
+ then you can skip this step.
1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run:
```shell
@@ -2165,16 +2128,10 @@ running [Prometheus](../monitoring/prometheus/index.md) and
## Configure the object storage
-GitLab supports using an object storage service for holding numerous types of data.
-
-GitLab has been tested on a number of object storage providers:
-
-- [Amazon S3](https://aws.amazon.com/s3/)
-- [Google Cloud Storage](https://cloud.google.com/storage)
-- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces)
-- [Oracle Cloud Infrastructure](https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm)
-- [OpenStack Swift (S3 compatibility mode)](https://docs.openstack.org/swift/latest/s3_compat.html)
-- MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation.
+GitLab supports using an [object storage](../object_storage.md) service for holding numerous types of data.
+It's recommended over [NFS](../nfs.md) for data objects and in general it's better
+in larger setups as object storage is typically much more performant, reliable,
+and scalable.
There are two ways of specifying object storage configuration in GitLab:
@@ -2324,7 +2281,9 @@ services where applicable):
- [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work.
4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section.
5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`.
-6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to the [Large Repositories](#large-repositories) for more info.
+6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large
+ repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to
+ [Large repositories](index.md#large-repositories) for more information.
<!-- markdownlint-enable MD029 -->
NOTE:
diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md
index 87d1408b568..3bcffa8f606 100644
--- a/doc/administration/reference_architectures/50k_users.md
+++ b/doc/administration/reference_architectures/50k_users.md
@@ -50,7 +50,9 @@ full list of reference architectures, see
- [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work.
4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section.
5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`.
-6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to the [Large Repositories](#large-repositories) for more info.
+6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large
+ repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to
+ [Large repositories](index.md#large-repositories) for more information.
<!-- markdownlint-enable MD029 -->
NOTE:
@@ -144,66 +146,7 @@ monitor .[#7FFFD4,norank]u--> elb
## Requirements
-Before starting, you should take note of the following requirements / guidance for this reference architecture.
-
-### Supported CPUs
-
-This reference architecture was built and tested on Google Cloud Platform (GCP) using the
-[Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms)
-CPU platform as a baseline ([Sysbench benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks)).
-
-Newer, similarly sized CPUs are supported and may have improved performance as a result. For Omnibus environments, ARM-based equivalents are also supported.
-
-NOTE:
-Any "burstable" instance types are not recommended due to inconsistent performance.
-
-### Supported infrastructure
-
-As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP, Azure) and their services,
-or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section.
-However, this does not constitute a guarantee for every potential permutation.
-
-See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
-
-### Additional workloads
-
-The Reference Architectures have been [designed and tested](index.md#validation-and-test-results) for standard GitLab setups with
-good headroom in mind to cover most scenarios. However, if any additional workloads are being added on the nodes,
-such as security software, you may still need to adjust the specs accordingly to compensate.
-
-This also applies for some GitLab features where it's possible to run custom scripts, for example [server hooks](../server_hooks.md).
-
-As a general rule, it's recommended to have robust monitoring in place to measure the impact of
-any additional workloads to inform any changes needed to be made.
-
-### Large repositories
-
-The Reference Architectures were tested with repositories of varying sizes that follow best practices.
-
-However, large repositories or monorepos (Several gigabytes or more) can **significantly** impact the performance
-of Git and in turn the environment itself if best practices aren't being followed such as not storing
-binary or blob files in LFS. Repositories are at the core of any environment the consequences can be wide-ranging
-when they are not optimized. Some examples of this impact include [Git packing operations](https://git-scm.com/book/en/v2/Git-Internals-Packfiles)
-taking longer and consuming high CPU / Memory resources or Git checkouts taking longer that affect both users and
-CI pipelines alike.
-
-As such, large repositories come with notable cost and typically will require more resources to handle,
-significantly so in some cases. It's therefore **strongly** recommended then to review large repositories
-to ensure they maintain good health and reduce their size wherever possible.
-
-NOTE:
-If best practices aren't followed and large repositories are present on the environment,
-increased Gitaly specs may be required to ensure stable performance.
-
-Refer to the [Managing large repositories documentation](../../user/project/repository/managing_large_repositories.md)
-for more information and guidance.
-
-### Praefect PostgreSQL
-
-It's worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and
-that to achieve full High Availability a third-party PostgreSQL database solution will be required.
-We hope to offer a built in solutions for these restrictions in the future but in the meantime a non HA PostgreSQL server
-can be set up via Omnibus GitLab, which the above specs reflect. Refer to the following issues for more information: [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398).
+Before starting, see the [requirements](index.md#requirements) for reference architectures.
## Setup components
@@ -720,7 +663,7 @@ in the second step, do not supply the `EXTERNAL_URL` value.
PostgreSQL, with Patroni managing its failover, will default to use `pg_rewind` by default to handle conflicts.
Like most failover handling methods, this has a small chance of leading to data loss.
-Learn more about the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method).
+For more information, see the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method).
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
@@ -1222,18 +1165,25 @@ Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.
## Configure Gitaly Cluster
-[Gitaly Cluster](../gitaly/praefect.md) is a GitLab provided and recommended fault tolerant solution for storing Git repositories.
-In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down.
+[Gitaly Cluster](../gitaly/praefect.md) is a GitLab-provided and recommended fault tolerant solution for storing Git
+repositories. In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being
+designated the primary, and failover occurs automatically if the primary node goes down.
-NOTE:
-Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster).
-For implementations with sharded Gitaly, use the same Gitaly specs. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section.
+Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management.
+Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster).
+
+For guidance on:
+
+- Implementing sharded Gitaly instead, follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md)
+ instead of this section. Use the same Gitaly specs.
+- Migrating existing repositories that aren't managed by Gitaly Cluster, see
+ [migrate to Gitaly Cluster](../gitaly/index.md#migrate-to-gitaly-cluster).
NOTE:
Gitaly has been designed and tested with repositories of varying sizes that follow best practices.
However, large repositories or monorepos not following these practices can significantly
impact Gitaly performance and requirements.
-Refer to the [Large Repositories](#large-repositories) for more info.
+Refer to [Large repositories](index.md#large-repositories) for more information.
The recommended cluster setup includes the following components:
@@ -1242,9 +1192,9 @@ The recommended cluster setup includes the following components:
- 1 Praefect PostgreSQL node: Database server for Praefect. A third-party solution
is required for Praefect database connections to be made highly available.
- 1 load balancer: A load balancer is required for Praefect. The
- [internal load balancer](#configure-the-internal-load-balancer) will be used.
+ [internal load balancer](#configure-the-internal-load-balancer) is used.
-This section will detail how to configure the recommended standard setup in order.
+This section details how to configure the recommended standard setup in order.
For more advanced setups refer to the [standalone Gitaly Cluster documentation](../gitaly/praefect.md).
### Configure Praefect PostgreSQL
@@ -1543,14 +1493,14 @@ requirements that are dependent on data and load.
NOTE:
Increased specs for Gitaly nodes may be required in some circumstances such as
-significantly large repositories or if any [additional workloads](#additional-workloads),
+significantly large repositories or if any [additional workloads](index.md#additional-workloads),
such as [server hooks](../server_hooks.md), have been added.
NOTE:
Gitaly has been designed and tested with repositories of varying sizes that follow best practices.
However, large repositories or monorepos not following these practices can significantly
impact Gitaly performance and requirements.
-Refer to the [Large Repositories](#large-repositories) for more info.
+Refer to [Large repositories](index.md#large-repositories) for more information.
Due to Gitaly having notable input and output requirements, we strongly
recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs
@@ -1769,7 +1719,10 @@ To configure Praefect with TLS:
Sidekiq requires connection to the [Redis](#configure-redis),
[PostgreSQL](#configure-postgresql) and [Gitaly](#configure-gitaly) instances.
-Because you must use [Object storage](#configure-the-object-storage) instead of NFS for data objects, the following
+It also requires a connection to [Object Storage](#configure-the-object-storage) as recommended.
+
+NOTE:
+[Because it's recommended to use Object storage](../object_storage.md) instead of NFS for data objects, the following
examples include the Object storage configuration.
- `10.6.0.101`: Sidekiq 1
@@ -1938,7 +1891,13 @@ run [multiple Sidekiq processes](../sidekiq/extra_sidekiq_processes.md).
## Configure GitLab Rails
This section describes how to configure the GitLab application (Rails) component.
-Because you must use [Object storage](#configure-the-object-storage) instead of NFS for data objects, the following
+
+Rails requires connections to the [Redis](#configure-redis),
+[PostgreSQL](#configure-postgresql) and [Gitaly](#configure-gitaly) instances.
+It also requires a connection to [Object Storage](#configure-the-object-storage) as recommended.
+
+NOTE:
+[Because it's recommended to use Object storage](../object_storage.md) instead of NFS for data objects, the following
examples include the Object storage configuration.
The following IPs will be used as an example:
@@ -2221,16 +2180,10 @@ To configure the Monitoring node:
## Configure the object storage
-GitLab supports using an object storage service for holding numerous types of data.
-
-GitLab has been tested on a number of object storage providers:
-
-- [Amazon S3](https://aws.amazon.com/s3/)
-- [Google Cloud Storage](https://cloud.google.com/storage)
-- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces)
-- [Oracle Cloud Infrastructure](https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm)
-- [OpenStack Swift (S3 compatibility mode)](https://docs.openstack.org/swift/latest/s3_compat.html)
-- MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation.
+GitLab supports using an [object storage](../object_storage.md) service for holding numerous types of data.
+It's recommended over [NFS](../nfs.md) for data objects and in general it's better
+in larger setups as object storage is typically much more performant, reliable,
+and scalable.
There are two ways of specifying object storage configuration in GitLab:
@@ -2357,7 +2310,9 @@ services where applicable):
- [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work.
4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section.
5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`.
-6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to the [Large Repositories](#large-repositories) for more info.
+6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large
+ repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to
+ [Large repositories](index.md#large-repositories) for more information.
<!-- markdownlint-enable MD029 -->
NOTE:
diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md
index 182edb82b5f..691f71289c3 100644
--- a/doc/administration/reference_architectures/5k_users.md
+++ b/doc/administration/reference_architectures/5k_users.md
@@ -56,7 +56,9 @@ costly-to-operate environment by using the
- [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work.
4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section.
5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`.
-6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to the [Large Repositories](#large-repositories) for more info.
+6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large
+ repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to
+ [Large repositories](index.md#large-repositories) for more information.
<!-- markdownlint-enable MD029 -->
NOTE:
@@ -147,66 +149,7 @@ monitor .[#7FFFD4,norank]u--> elb
## Requirements
-Before starting, you should take note of the following requirements / guidance for this reference architecture.
-
-### Supported CPUs
-
-This reference architecture was built and tested on Google Cloud Platform (GCP) using the
-[Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms)
-CPU platform as a baseline ([Sysbench benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks)).
-
-Newer, similarly sized CPUs are supported and may have improved performance as a result. For Omnibus environments, ARM-based equivalents are also supported.
-
-NOTE:
-Any "burstable" instance types are not recommended due to inconsistent performance.
-
-### Supported infrastructure
-
-As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP) and their services,
-or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section.
-However, this does not constitute a guarantee for every potential permutation.
-
-See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
-
-### Additional workloads
-
-The Reference Architectures have been [designed and tested](index.md#validation-and-test-results) for standard GitLab setups with
-good headroom in mind to cover most scenarios. However, if any additional workloads are being added on the nodes,
-such as security software, you may still need to adjust the specs accordingly to compensate.
-
-This also applies for some GitLab features where it's possible to run custom scripts, for example [server hooks](../server_hooks.md).
-
-As a general rule, it's recommended to have robust monitoring in place to measure the impact of
-any additional workloads to inform any changes needed to be made.
-
-### Large repositories
-
-The Reference Architectures were tested with repositories of varying sizes that follow best practices.
-
-However, large repositories or monorepos (Several gigabytes or more) can **significantly** impact the performance
-of Git and in turn the environment itself if best practices aren't being followed such as not storing
-binary or blob files in LFS. Repositories are at the core of any environment the consequences can be wide-ranging
-when they are not optimized. Some examples of this impact include [Git packing operations](https://git-scm.com/book/en/v2/Git-Internals-Packfiles)
-taking longer and consuming high CPU / Memory resources or Git checkouts taking longer that affect both users and
-CI pipelines alike.
-
-As such, large repositories come with notable cost and typically will require more resources to handle,
-significantly so in some cases. It's therefore **strongly** recommended then to review large repositories
-to ensure they maintain good health and reduce their size wherever possible.
-
-NOTE:
-If best practices aren't followed and large repositories are present on the environment,
-increased Gitaly specs may be required to ensure stable performance.
-
-Refer the [Managing large repositories documentation](../../user/project/repository/managing_large_repositories.md)
-for more information and guidance.
-
-### Praefect PostgreSQL
-
-It's worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and
-that to achieve full High Availability a third-party PostgreSQL database solution is required.
-We hope to offer a built in solutions for these restrictions in the future but in the meantime a non HA PostgreSQL server
-can be set up via Omnibus GitLab, which the above specs reflect. Refer to the following issues for more information: [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398).
+Before starting, see the [requirements](index.md#requirements) for reference architectures.
## Setup components
@@ -999,7 +942,7 @@ in the second step, do not supply the `EXTERNAL_URL` value.
PostgreSQL, with Patroni managing its failover, defaults to use `pg_rewind` by default to handle conflicts.
Like most failover handling methods, this has a small chance of leading to data loss.
-Learn more about the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method).
+For more information, see the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method).
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
@@ -1160,18 +1103,25 @@ The following IPs are used as an example:
## Configure Gitaly Cluster
-[Gitaly Cluster](../gitaly/praefect.md) is a GitLab provided and recommended fault tolerant solution for storing Git repositories.
-In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down.
+[Gitaly Cluster](../gitaly/praefect.md) is a GitLab-provided and recommended fault tolerant solution for storing Git
+repositories. In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being
+designated the primary, and failover occurs automatically if the primary node goes down.
-NOTE:
-Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster).
-For implementations with sharded Gitaly, use the same Gitaly specs. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section.
+Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management.
+Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster).
+
+For guidance on:
+
+- Implementing sharded Gitaly instead, follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md)
+ instead of this section. Use the same Gitaly specs.
+- Migrating existing repositories that aren't managed by Gitaly Cluster, see
+ [migrate to Gitaly Cluster](../gitaly/index.md#migrate-to-gitaly-cluster).
NOTE:
Gitaly has been designed and tested with repositories of varying sizes that follow best practices.
However, large repositories or monorepos not following these practices can significantly
impact Gitaly performance and requirements.
-Refer to the [Large Repositories](#large-repositories) for more info.
+Refer to [Large repositories](index.md#large-repositories) for more information.
The recommended cluster setup includes the following components:
@@ -1479,14 +1429,14 @@ requirements that are dependent on data and load.
NOTE:
Increased specs for Gitaly nodes may be required in some circumstances such as
-significantly large repositories or if any [additional workloads](#additional-workloads),
+significantly large repositories or if any [additional workloads](index.md#additional-workloads),
such as [server hooks](../server_hooks.md), have been added.
NOTE:
Gitaly has been designed and tested with repositories of varying sizes that follow best practices.
However, large repositories or monorepos not following these practices can significantly
impact Gitaly performance and requirements.
-Refer to the [Large Repositories](#large-repositories) for more info.
+Refer to [Large repositories](index.md#large-repositories) for more information.
Due to Gitaly having notable input and output requirements, we strongly
recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs
@@ -1703,9 +1653,12 @@ To configure Praefect with TLS:
## Configure Sidekiq
-Sidekiq requires connection to the [Redis](#configure-redis),
+Sidekiq requires connections to the [Redis](#configure-redis),
[PostgreSQL](#configure-postgresql) and [Gitaly](#configure-gitaly) instances.
-Because you must use [Object storage](#configure-the-object-storage) instead of NFS for data objects, the following
+It also requires a connection to [Object Storage](#configure-the-object-storage) as recommended.
+
+NOTE:
+[Because it's recommended to use Object storage](../object_storage.md) instead of NFS for data objects, the following
examples include the Object storage configuration.
- `10.6.0.71`: Sidekiq 1
@@ -1872,7 +1825,13 @@ run [multiple Sidekiq processes](../sidekiq/extra_sidekiq_processes.md).
## Configure GitLab Rails
This section describes how to configure the GitLab application (Rails) component.
-Because you must use [Object storage](#configure-the-object-storage) instead of NFS for data objects, the following
+
+Rails requires connections to the [Redis](#configure-redis),
+[PostgreSQL](#configure-postgresql) and [Gitaly](#configure-gitaly) instances.
+It also requires a connection to [Object Storage](#configure-the-object-storage) as recommended.
+
+NOTE:
+[Because it's recommended to use Object storage](../object_storage.md) instead of NFS for data objects, the following
examples include the Object storage configuration.
On each node perform the following:
@@ -2015,6 +1974,10 @@ On each node perform the following:
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+1. Copy the SSH host keys (all in the name format `/etc/ssh/ssh_host_*_key*`) from the first Omnibus node you configured and
+ add or replace the files of the same name on this server. This ensures host mismatch errors aren't thrown
+ for your users as they hit the load balanced Rails nodes. If this is the first Omnibus node you are configuring,
+ then you can skip this step.
1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run:
```shell
@@ -2164,16 +2127,10 @@ running [Prometheus](../monitoring/prometheus/index.md) and
## Configure the object storage
-GitLab supports using an object storage service for holding numerous types of data.
-
-GitLab has been tested on a number of object storage providers:
-
-- [Amazon S3](https://aws.amazon.com/s3/)
-- [Google Cloud Storage](https://cloud.google.com/storage)
-- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces)
-- [Oracle Cloud Infrastructure](https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm)
-- [OpenStack Swift (S3 compatibility mode)](https://docs.openstack.org/swift/latest/s3_compat.html)
-- MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation.
+GitLab supports using an [object storage](../object_storage.md) service for holding numerous types of data.
+It's recommended over [NFS](../nfs.md) for data objects and in general it's better
+in larger setups as object storage is typically much more performant, reliable,
+and scalable.
There are two ways of specifying object storage configuration in GitLab:
@@ -2299,7 +2256,9 @@ services where applicable):
- [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work.
4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section.
5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`.
-6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to the [Large Repositories](#large-repositories) for more info.
+6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large
+ repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to
+ [Large repositories](index.md#large-repositories) for more information.
<!-- markdownlint-enable MD029 -->
NOTE:
diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md
index 60258fb5a09..7b01efa183b 100644
--- a/doc/administration/reference_architectures/index.md
+++ b/doc/administration/reference_architectures/index.md
@@ -16,7 +16,7 @@ Depending on your workflow, the following recommended reference architectures
may need to be adapted accordingly. Your workload is influenced by factors
including how active your users are, how much automation you use, mirroring,
and repository/change size. Additionally, the displayed memory values are
-provided by [GCP machine types](https://cloud.google.com/compute/docs/machine-types).
+provided by [GCP machine types](https://cloud.google.com/compute/docs/machine-resource).
For different cloud vendors, attempt to select options that best match the
provided architecture.
@@ -43,29 +43,40 @@ The following Cloud Native Hybrid reference architectures, where select recommen
- [Up to 25,000 users](25k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative)
- [Up to 50,000 users](50k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative)
+## Before you start
+
+The first choice to consider is whether a Self Managed approach is correct for you and your requirements.
+
+Running any application in production is complex, and the same applies for GitLab. While we aim to make this as smooth as possible, there are still the general complexities. This depends on the design chosen, but typically you'll need to manage all aspects such as hardware, operating systems, networking, storage, security, GitLab itself, and more.
+
+As such, it's recommended that you have a working knowledge of running applications in production when deciding on going down this route. For users who want a more managed solution it's recommended to instead explore our other offerings such as [GitLab SaaS](../../subscriptions/gitlab_com/index.md) or [GitLab Dedicated](../../subscriptions/gitlab_dedicated/index.md).
+
## Deciding which architecture to use
The Reference Architectures are designed to strike a balance between two important factors--performance and resilience.
-While they are designed to make it easier to set up GitLab at scale, it can still be a challenge to know which one will meet your requirements.
+While they are designed to make it easier to set up GitLab at scale, it can still be a challenge to know which one meets your requirements.
-As a general guide, **the more performant and/or resilient you want your environment to be, the more involved it will be**.
+As a general guide, **the more performant and/or resilient you want your environment to be, the more complex it is**.
This section explains the designs you can choose from. It begins with the least complexity, goes to the most, and ends with a decision tree.
-### Backups
+### Standalone (non-HA)
+
+For environments serving 2,000 or fewer users, we generally recommend a standalone approach by deploying a non-highly available single or multi-node environment. With this approach, you can employ strategies such as [automated backups](../../raketasks/backup_gitlab.md#configuring-cron-to-make-daily-backups) for recovery to provide a good level of RPO / RTO while avoiding the complexities that come with HA.
-For environments serving 2,000 or fewer users we generally recommend that an [automated backup](../../raketasks/backup_gitlab.md#configuring-cron-to-make-daily-backups) strategy is used instead of HA.
+*[RTO]: Recovery time objective
+*[RPO]: Recovery point objective
-Backups can provide a good level of RPO / RTO while avoiding the complexities that come with HA.
+With standalone setups, especially single node environments, there are [various options available for installation](../../install/index.md) and management including [the ability to deploy directly via select cloud provider marketplaces](https://page.gitlab.com/cloud-partner-marketplaces.html) that reduce the complexity a little further.
### High Availability (HA)
-High Availability ensures every component in the GitLab setup can handle failures through various mechanisms. To achieve this however is involved, and the environments required can be sizable.
+High Availability ensures every component in the GitLab setup can handle failures through various mechanisms. To achieve this however is complex, and the environments required can be sizable.
-For environments serving 3,000 or more users we generally recommend that a HA strategy is used as at this level outages will have a bigger impact against more users. All the architectures in this range have HA built in by design for this reason.
+For environments serving 3,000 or more users we generally recommend that a HA strategy is used as at this level outages have a bigger impact against more users. All the architectures in this range have HA built in by design for this reason.
-For users who still need to have HA for a lower number of users this can also be achieved with an [adjusted 3K architecture as detailed here](3k_users.md#supported-modifications-for-lower-user-counts-ha).
+For users who still need to have HA for a lower number of users this can also be achieved with an adjusted [3K architecture](3k_users.md#supported-modifications-for-lower-user-counts-ha).
#### Do you need High Availability (HA)?
@@ -80,7 +91,7 @@ In general then, we'd only recommend you employ HA in the following scenarios:
#### Zero Downtime Upgrades
-[Zero Downtime Upgrades](../../update/zero_downtime.md) are available for standard Reference Architecture environments with HA (Cloud Native Hybrid is not supported at this time). This allows for an environment to stay up during an upgrade, but the process is more involved as a result and has some limitations as detailed in the documentation.
+[Zero Downtime Upgrades](../../update/zero_downtime.md) are available for standard Reference Architecture environments with HA (Cloud Native Hybrid is not supported at this time). This allows for an environment to stay up during an upgrade, but the process is more complex as a result and has some limitations as detailed in the documentation.
When going through this process it's worth noting that there may still be brief moments of downtime when the HA mechanisms tale effect.
@@ -90,13 +101,19 @@ In most cases the downtime required for doing an upgrade in general shouldn't be
As an additional layer of HA resilience you can deploy select components in Kubernetes, known as a Cloud Native Hybrid Reference Architecture.
-Note that this is an alternative and more **advanced** setup compared to a standard Reference Architecture. Running services in Kubernetes is well known to be complex. **This setup is only recommended** if you have strong working knowledge and experience in Kubernetes.
+This is an alternative and more **advanced** setup compared to a standard Reference Architecture. Running services in Kubernetes is well known to be complex. **This setup is only recommended** if you have strong working knowledge and experience in Kubernetes.
### GitLab Geo (Cross Regional Distribution / Disaster Recovery)
With [GitLab Geo](../geo/index.md) you can have both distributed environments in different regions and a full Disaster Recovery (DR) setup in place. With this setup you would have 2 or more separate environments, with one being a primary that gets replicated to the others. In the rare event the primary site went down completely you could fail over to one of the other environments.
-This is an **advanced and involved** setup and should only be undertaken if you have DR as a key requirement. Decisions then on how each environment are configured would also need to be taken, such as if each environment itself would be the full size and / or have HA.
+This is an **advanced and complex** setup and should only be undertaken if you have DR as a key requirement. Decisions then on how each environment are configured would also need to be taken, such as if each environment itself would be the full size and / or have HA.
+
+### Cloud provider services
+
+For all the previously described strategies, you can run select GitLab components on equivalent cloud provider services such as the PostgreSQL database or Redis.
+
+[For more information, see the recommended cloud providers and services](#recommended-cloud-providers-and-services).
### Decision Tree
@@ -105,14 +122,30 @@ Below you can find the above guidance in the form of a decision tree. It's recom
```mermaid
%%{init: { 'theme': 'base' } }%%
graph TD
- L1A(<b>What Reference Architecture should I use?</b>) --> L2A(More than 3000 users?)
- L2A -->|No| L3A("<a href=#do-you-need-high-availability-ha>Do you need HA?</a><br>(or Zero-Downtime Upgrades)") --> |Yes| L4A><b>Recommendation</b><br><br>3K architecture with HA<br>including supported modifications]
- L3A -->|No| L4B><b>Recommendation</b><br><br>Architecture closest to user<br>count with Backups]
- L2A -->|Yes| L3B[Do you have experience with<br/>and want additional resilience<br/>with select components in Kubernetes?]
- L3B -->|No| L4C><b>Recommendation</b><br><br>Architecture closest to user<br>count with HA]
- L3B -->|Yes| L4D><b>Recommendation</b><br><br>Cloud Native Hybrid architecture<br>closest to user count]
-
- L5A("<a href=#gitlab-geo-cross-regional-distribution-disaster-recovery>Do you need cross regional distribution or disaster recovery?"</a>) --> |Yes| L6A><b>Additional Recommendation</b><br><br> GitLab Geo]
+ L1A(<b>What Reference Architecture should I use?</b>)
+
+ L2A(3,000 users or more?)
+ L2B(2,000 users or less?)
+
+ L3A("<a href=#do-you-need-high-availability-ha>Do you need HA?</a><br>(or Zero-Downtime Upgrades)")
+ L3B[Do you have experience with<br/>and want additional resilience<br/>with select components in Kubernetes?]
+
+ L4A><b>Recommendation</b><br><br>3K architecture with HA<br>and supported reductions]
+ L4B><b>Recommendation</b><br><br>Architecture closest to user<br>count with HA]
+ L4C><b>Recommendation</b><br><br>Cloud Native Hybrid architecture<br>closest to user count]
+ L4D>"<b>Recommendation</b><br><br>Standalone 1K or 2K<br/>architecture with Backups"]
+
+ L1A --> L2A
+ L1A --> L2B
+ L2A -->|Yes| L3B
+ L3B -->|Yes| L4C
+ L3B -->|No| L4B
+
+ L2B --> L3A
+ L3A -->|Yes| L4A
+ L3A -->|No| L4D
+
+ L5A("<a href=#gitlab-geo-cross-regional-distribution-disaster--recovery>Do you need cross regional distribution or disaster recovery?"</a>) --> |Yes| L6A><b>Additional Recommendation</b><br><br> GitLab Geo]
L4A -.- L5A
L4B -.- L5A
L4C -.- L5A
@@ -122,6 +155,94 @@ classDef default fill:#FCA326
linkStyle default fill:none,stroke:#7759C2
```
+## Requirements
+
+Before implementing a reference architecture, refer to the following requirements and guidance.
+
+### Supported CPUs
+
+These reference architectures were built and tested on Google Cloud Platform (GCP) using the
+[Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms)
+CPU platform as a baseline ([Sysbench benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks)).
+
+Newer, similarly-sized CPUs are supported and may have improved performance as a result. For Omnibus GitLab environments,
+ARM-based equivalents are also supported.
+
+NOTE:
+Any "burstable" instance types are not recommended due to inconsistent performance.
+
+### Supported infrastructure
+
+As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP, Azure) and
+their services, or self managed (ESXi) that meet both:
+
+- The specifications detailed in each reference architecture.
+- Any requirements in this section.
+
+However, this does not constitute a guarantee for every potential permutation.
+
+See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
+
+### Additional workloads
+
+These reference architectures have been [designed and tested](index.md#validation-and-test-results) for standard GitLab
+setups with good headroom in mind to cover most scenarios.
+
+However, additional workloads can multiply the impact of operations by triggering follow-up actions.
+You may need to adjust the suggested specifications to compensate if you use, for example:
+
+- Security software on the nodes.
+- Hundreds of concurrent CI jobs for [large repositories](../../ci/large_repositories/index.md).
+- Custom scripts that [run at high frequency](../logs/log_parsing.md#print-top-api-user-agents).
+- [Integrations](../../integration/index.md) in many large projects.
+- [Server hooks](../server_hooks.md).
+- [System hooks](../system_hooks.md).
+
+As a general rule, you should have robust monitoring in place to measure the impact of any additional workloads to
+inform any changes needed to be made.
+
+### No swap
+
+Swap is not recommended in the reference architectures. It's a failsafe that impacts performance greatly. The
+reference architectures are designed to have memory headroom to avoid needing swap.
+
+### Large repositories
+
+The relevant reference architectures were tested with repositories of varying sizes that follow best practices.
+
+However, large repositories or monorepos (several gigabytes or more) can **significantly** impact the performance
+of Git and in turn the environment itself if best practices aren't being followed such as not storing binary or blob
+files in LFS.
+
+Repositories are at the core of any environment and the consequences can be wide-ranging when they are not optimized.
+Some examples of this impact include:
+
+- [Git packing operations](https://git-scm.com/book/en/v2/Git-Internals-Packfiles) taking longer and consuming high CPU
+ and memory resources.
+- Git checkouts taking longer that affect both users and CI/CD pipelines alike.
+
+As such, large repositories come with notable cost and typically require more resources to handle, (significantly more
+in some cases). You should review large repositories to ensure they maintain good health and reduce their size wherever
+possible.
+
+NOTE:
+If best practices aren't followed and large repositories are present on the environment, increased Gitaly specs may be
+required to ensure stable performance.
+
+Refer to the [Managing large repositories documentation](../../user/project/repository/managing_large_repositories.md)
+for more information and guidance.
+
+### Praefect PostgreSQL
+
+[Praefect requires its own database server](../gitaly/praefect.md#postgresql) and
+that to achieve full High Availability, a third-party PostgreSQL database solution is required.
+
+We hope to offer a built in solutions for these restrictions in the future but, in the meantime, a non HA PostgreSQL server
+can be set up using Omnibus GitLab, the specifications reflect. Refer to the following issues for more information:
+
+- [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919).
+- [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398).
+
## Recommended cloud providers and services
NOTE:
@@ -208,7 +329,7 @@ Several cloud provider services are known not to support the above or have been
- [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is incompatible and not supported. See [14.4.0](../../update/index.md#1440) for more details.
- [Azure Database for PostgreSQL Single Server](https://azure.microsoft.com/en-gb/products/postgresql/#overview) (Single / Flexible) is **strongly not recommended** for use due to notable performance / stability issues or missing functionality. See [Recommendation Notes for Azure](#recommendation-notes-for-azure) for more details.
- [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo.
- - Note that [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported.
+ - [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported.
### Recommendation notes for Azure
@@ -217,8 +338,8 @@ Due to performance issues that we found with several key Azure services, we only
In addition to the above, you should be aware of the additional specific guidance for Azure:
- **We outright strongly do not recommend [Azure Database for PostgreSQL Single Server](https://learn.microsoft.com/en-us/azure/postgresql/single-server/overview-single-server)** specifically due to significant performance and stability issues found. **For GitLab 14.0 and higher the service is not supported** due to it only supporting up to PostgreSQL 11.
- - A new service, [Azure Database for PostgreSQL Flexible Server](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/) has been released but due to it missing some functionality we don't recommend it at this time.
-- [Azure Blob Storage](https://azure.microsoft.com/en-gb/products/storage/blobs/) has been found to have performance limits that can impact production use at certain times. However, this has only been seen in larger architectures.
+ - A new service, [Azure Database for PostgreSQL Flexible Server](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/) has been released. [Internal testing](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/91) has shown that it does look to perform as expected, but this hasn't been validated in production, so generally isn't recommended at this time. Additionally, as it's a new service, you may find that it's missing some functionality depending on your requirements.
+- [Azure Blob Storage](https://azure.microsoft.com/en-gb/products/storage/blobs/) has been found to have [performance limits that can impact production use at certain times](https://gitlab.com/gitlab-org/gitlab/-/issues/344861). However, this has only been seen in our largest architectures (25k+) so far.
## Validation and test results
@@ -241,7 +362,7 @@ Testing occurs against all reference architectures and cloud providers in an aut
- The [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/gitlab-environment-toolkit) for building the environments.
- The [GitLab Performance Tool](https://gitlab.com/gitlab-org/quality/performance) for performance testing.
-Network latency on the test environments between components on all Cloud Providers were measured at <5 ms. Note that this is shared as an observation and not as an implicit recommendation.
+Network latency on the test environments between components on all Cloud Providers were measured at <5 ms. This is shared as an observation and not as an implicit recommendation.
We aim to have a "test smart" approach where architectures tested have a good range that can also apply to others. Testing focuses on 10k Omnibus on GCP as the testing has shown this is a good bellwether for the other architectures and cloud providers as well as Cloud Native Hybrids.
diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md
index 77a5eae3747..ab95887380d 100644
--- a/doc/administration/repository_storage_types.md
+++ b/doc/administration/repository_storage_types.md
@@ -208,7 +208,7 @@ CI/CD artifacts are:
#### LFS objects
[LFS Objects in GitLab](../topics/git/lfs/index.md) implement a similar
-storage pattern using two characters and two-level folders, following Git's own implementation:
+storage pattern using two characters and two-level folders, following the Git implementation:
```ruby
"shared/lfs-objects/#{oid[0..1}/#{oid[2..3]}/#{oid[4..-1]}"
diff --git a/doc/administration/sidekiq/extra_sidekiq_processes.md b/doc/administration/sidekiq/extra_sidekiq_processes.md
index d5007e9a3e9..7959d1a5ce7 100644
--- a/doc/administration/sidekiq/extra_sidekiq_processes.md
+++ b/doc/administration/sidekiq/extra_sidekiq_processes.md
@@ -55,7 +55,7 @@ To view the Sidekiq processes in GitLab:
By default each process defined under `sidekiq` starts with a number of threads
that equals the number of queues, plus one spare thread, up to a maximum of 50.
-For example, a process that handles all queues will use 50 threads by default.
+For example, a process that handles all queues uses 50 threads by default.
These threads run inside a single Ruby process, and each process can only use a
single CPU core. The usefulness of threading depends on the work having some
@@ -70,8 +70,9 @@ higher for mixed low-priority work. A reasonable starting range is `15` to `25`
for a non-specialized deployment.
We only recommend setting explicit concurrency by setting `min_concurrency` and
-`max_concurrency` to the same value. The two values are kept for backwards
-compatibility reasons, but for more predictable results, use the same value.
+`max_concurrency` to the same value. The two distinct settings are kept for
+backwards compatibility reasons, but for more predictable results use the same
+values – otherwise you might run into issues with Sidekiq jobs piling up.
For example, to set the concurrency to `20`:
@@ -89,7 +90,8 @@ For example, to set the concurrency to `20`:
```
`min_concurrency` and `max_concurrency` are independent; one can be set without
-the other. Setting `min_concurrency` to `0` disables the limit.
+the other. Setting `min_concurrency` to `0` disables the limit. Not explicitly
+setting `min_concurrency` is the same as setting it to `0`.
For each queue group, let `N` be one more than the number of queues. The
concurrency is set to:
@@ -117,7 +119,7 @@ for more details.
## Modify the check interval
-To modify Sidekiq's health check interval for the additional Sidekiq
+To modify the Sidekiq health check interval for the additional Sidekiq
processes:
1. Edit `/etc/gitlab/gitlab.rb`:
diff --git a/doc/administration/sidekiq/index.md b/doc/administration/sidekiq/index.md
index 7b3ecdd0890..bf858476c0c 100644
--- a/doc/administration/sidekiq/index.md
+++ b/doc/administration/sidekiq/index.md
@@ -184,8 +184,8 @@ Updates to example must be made at:
## Set number of Sidekiq queue processes to the same number as available CPUs
sidekiq['queue_groups'] = ['*'] * 4
- ## Set number of Sidekiq threads per queue process to the recommend number of 10
- sidekiq['max_concurrency'] = 10
+ ## Set number of Sidekiq threads per queue process to the recommend number of 20
+ sidekiq['max_concurrency'] = 20
```
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the GitLab instance and replace the file in the Sidekiq instance.
diff --git a/doc/administration/sidekiq/processing_specific_job_classes.md b/doc/administration/sidekiq/processing_specific_job_classes.md
index 080ad7d4eae..61a154c8db2 100644
--- a/doc/administration/sidekiq/processing_specific_job_classes.md
+++ b/doc/administration/sidekiq/processing_specific_job_classes.md
@@ -252,7 +252,7 @@ WARNING:
As described in [the concurrency section](extra_sidekiq_processes.md#manage-thread-counts-explicitly), we
recommend setting `min_concurrency` and `max_concurrency` to the same value. For example, if the number of queues
in a queue group entry is 1, while `min_concurrency` is set to `0`, and `max_concurrency` is set to `20`, the resulting
-concurrency will be set to `2` instead. A concurrency of `2` might be too low in most cases, except for very highly-CPU
+concurrency is set to `2` instead. A concurrency of `2` might be too low in most cases, except for very highly-CPU
bound tasks.
## Worker matching query
@@ -310,8 +310,8 @@ highest to lowest precedence:
and `query_b` are queries made up of the other operators here) includes
queues that match either query.
- `&` - the logical `AND` operator. For example, `query_a&query_b` (where
- `query_a` and `query_b` are queries made up of the other operators here) will
- only include queues that match both queries.
+ `query_a` and `query_b` are queries made up of the other operators here)
+ include only queues that match both queries.
- `!=` - the `NOT IN` operator. For example, `feature_category!=issue_tracking`
excludes all queues from the `issue_tracking` feature category.
- `=` - the `IN` operator. For example, `resource_boundary=cpu` includes all
diff --git a/doc/administration/sidekiq/sidekiq_job_migration.md b/doc/administration/sidekiq/sidekiq_job_migration.md
index b93d86d4c86..1332d9b35d8 100644
--- a/doc/administration/sidekiq/sidekiq_job_migration.md
+++ b/doc/administration/sidekiq/sidekiq_job_migration.md
@@ -28,11 +28,11 @@ Step 4 involves rewriting some Sidekiq job data for jobs that are already stored
- `gitlab:sidekiq:migrate_jobs:retry` for jobs to be retried.
- `gitlab:sidekiq:migrate_jobs:scheduled` for scheduled jobs.
-Queued jobs that are yet to be run can also be migrated with a Rake task:
+Queued jobs that are yet to be run can also be migrated with a Rake task ([available in GitLab 15.6](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/101348) and later):
- `gitlab:sidekiq:migrate_jobs:queued` for queued jobs to be performed asynchronously.
-Most of the time, running all three at the same time is the correct choice. There are three separate tasks to allow for more fine-grained control where needed. To run all three at once:
+Most of the time, running all three at the same time is the correct choice. There are three separate tasks to allow for more fine-grained control where needed. To run all three at once ([available in GitLab 15.6](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/101348) and later):
```shell
# omnibus-gitlab
diff --git a/doc/administration/sidekiq/sidekiq_memory_killer.md b/doc/administration/sidekiq/sidekiq_memory_killer.md
index 0876f98621d..cb27d44a2e6 100644
--- a/doc/administration/sidekiq/sidekiq_memory_killer.md
+++ b/doc/administration/sidekiq/sidekiq_memory_killer.md
@@ -4,22 +4,21 @@ group: Application Performance
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
-# Sidekiq MemoryKiller **(FREE SELF)**
+# Reducing memory use
The GitLab Rails application code suffers from memory leaks. For web requests
-this problem is made manageable using
-[`puma-worker-killer`](https://github.com/schneems/puma_worker_killer) which
-restarts Puma worker processes if it exceeds a memory limit. The Sidekiq
-MemoryKiller applies the same approach to the Sidekiq processes used by GitLab
+this problem is made manageable using a [supervision thread](../operations/puma.md#reducing-memory-use)
+that automatically restarts workers if they exceed a given resident set size (RSS) threshold
+for a certain amount of time.
+We use the same approach to the Sidekiq processes used by GitLab
to process background jobs.
-Unlike puma-worker-killer, which is enabled by default for all GitLab
-installations of GitLab 13.0 and later, the Sidekiq MemoryKiller is enabled by default
-_only_ for Omnibus packages. The reason for this is that the MemoryKiller
-relies on runit to restart Sidekiq after a memory-induced shutdown and GitLab
-installations from source do not all use runit or an equivalent.
+GitLab monitors the available RSS limit by default only for installations using
+the Linux packages (Omnibus) or Docker. The reason for this is that GitLab
+relies on runit to restart Sidekiq after a memory-induced shutdown, and GitLab
+self-compiled or Helm chart based installations don't use runit or an equivalent tool.
-With the default settings, the MemoryKiller causes a Sidekiq restart no
+With the default settings, Sidekiq restarts no
more often than once every 15 minutes, with the restart causing about one
minute of delay for incoming background jobs.
@@ -28,41 +27,75 @@ are cleanly terminated when Sidekiq is restarted, each Sidekiq process should be
run as a process group leader (for example, using `chpst -P`). If using Omnibus or the
`bin/background_jobs` script with `runit` installed, this is handled for you.
-## Configuring the MemoryKiller
-
-The MemoryKiller is controlled using environment variables.
-
-- `SIDEKIQ_MEMORY_KILLER_MAX_RSS` (KB): if this variable is set, and its value is greater
- than 0, the MemoryKiller is enabled. Otherwise the MemoryKiller is disabled.
-
- `SIDEKIQ_MEMORY_KILLER_MAX_RSS` defines the Sidekiq process allowed RSS.
-
- If the Sidekiq process exceeds the allowed RSS for longer than
- `SIDEKIQ_MEMORY_KILLER_GRACE_TIME` the graceful restart is triggered. If the
- Sidekiq process go below the allowed RSS within `SIDEKIQ_MEMORY_KILLER_GRACE_TIME`,
- the restart is aborted.
-
- The default value for Omnibus packages is set
- [in the Omnibus GitLab repository](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/attributes/default.rb).
-
-- `SIDEKIQ_MEMORY_KILLER_HARD_LIMIT_RSS` (KB): If the Sidekiq
- process RSS (expressed in kilobytes) exceeds `SIDEKIQ_MEMORY_KILLER_HARD_LIMIT_RSS`,
- an immediate graceful restart of Sidekiq is triggered.
-
-- `SIDEKIQ_MEMORY_KILLER_CHECK_INTERVAL`: Define how
- often to check process RSS, default to 3 seconds.
-
-- `SIDEKIQ_MEMORY_KILLER_GRACE_TIME`: defaults to 900 seconds (15 minutes).
- The usage of this variable is described as part of `SIDEKIQ_MEMORY_KILLER_MAX_RSS`.
-
-- `SIDEKIQ_MEMORY_KILLER_SHUTDOWN_WAIT`: defaults to 30 seconds. This defines the
- maximum time allowed for all Sidekiq jobs to finish. No new jobs are accepted
- during that time, and the process exits as soon as all jobs finish.
-
- If jobs do not finish during that time, the MemoryKiller interrupts all currently
- running jobs by sending `SIGTERM` to the Sidekiq process.
-
- If the process hard shutdown/restart is not performed by Sidekiq,
- the Sidekiq process is forcefully terminated after
- `Sidekiq[:timeout] + 2` seconds. An external supervision mechanism
- (for example, runit) must restart Sidekiq afterwards.
+## Configuring the limits
+
+Sidekiq memory limits are controlled using environment variables.
+
+- `SIDEKIQ_MEMORY_KILLER_MAX_RSS` (KB): defines the Sidekiq process soft limit for allowed RSS.
+ If the Sidekiq process RSS (expressed in kilobytes) exceeds `SIDEKIQ_MEMORY_KILLER_MAX_RSS`,
+ for longer than `SIDEKIQ_MEMORY_KILLER_GRACE_TIME`, the graceful restart is triggered.
+ If `SIDEKIQ_MEMORY_KILLER_MAX_RSS` is not set, or its value is set to 0, the soft limit is not monitored.
+ `SIDEKIQ_MEMORY_KILLER_MAX_RSS` defaults to `2000000`.
+
+- `SIDEKIQ_MEMORY_KILLER_GRACE_TIME`: defines the grace time period in seconds for which the Sidekiq process is allowed to run
+ above the allowed RSS soft limit. If the Sidekiq process goes below the allowed RSS (soft limit)
+ within `SIDEKIQ_MEMORY_KILLER_GRACE_TIME`, the restart is aborted. Default value is 900 seconds (15 minutes).
+
+- `SIDEKIQ_MEMORY_KILLER_HARD_LIMIT_RSS` (KB): defines the Sidekiq process hard limit for allowed RSS.
+ If the Sidekiq process RSS (expressed in kilobytes) exceeds `SIDEKIQ_MEMORY_KILLER_HARD_LIMIT_RSS`,
+ an immediate graceful restart of Sidekiq is triggered. If this value is not set, or set to 0,
+ the hard limit is not be monitored.
+
+- `SIDEKIQ_MEMORY_KILLER_CHECK_INTERVAL`: defines how often to check the process RSS. Defaults to 3 seconds.
+
+- `SIDEKIQ_MEMORY_KILLER_SHUTDOWN_WAIT`: defines the maximum time allowed for all Sidekiq jobs to finish.
+ No new jobs are accepted during that time. Defaults to 30 seconds.
+
+ If the process restart is not performed by Sidekiq, the Sidekiq process is forcefully terminated after
+ [Sidekiq shutdown timeout](https://github.com/mperham/sidekiq/wiki/Signals#term) (defaults to 25 seconds) +2 seconds.
+ If jobs do not finish during that time, all currently running jobs are interrupted with a `SIGTERM` signal
+ sent to the Sidekiq process.
+
+- `GITLAB_MEMORY_WATCHDOG_ENABLED`: enabled by default. Set the `GITLAB_MEMORY_WATCHDOG_ENABLED` to false, to use legacy
+ Daemon Sidekiq Memory Killer implementation used prior GitLab 15.9. Support for setting `GITLAB_MEMORY_WATCHDOG_ENABLED`
+ will be removed in GitLab 16.0.
+
+### Monitor worker restarts
+
+GitLab emits log events if workers are restarted due to high memory usage.
+
+The following is an example of one of these log events in `/var/log/gitlab/gitlab-rails/sidekiq_client.log`:
+
+```json
+{
+ "severity": "WARN",
+ "time": "2023-02-04T09:45:16.173Z",
+ "correlation_id": null,
+ "pid": 2725,
+ "worker_id": "sidekiq_1",
+ "memwd_handler_class": "Gitlab::Memory::Watchdog::SidekiqHandler",
+ "memwd_sleep_time_s": 3,
+ "memwd_rss_bytes": 1079683247,
+ "memwd_max_rss_bytes": 629145600,
+ "memwd_max_strikes": 5,
+ "memwd_cur_strikes": 6,
+ "message": "rss memory limit exceeded",
+ "running_jobs": [
+ {
+ jid: "83efb701c59547ee42ff7068",
+ worker_class: "Ci::DeleteObjectsWorker"
+ },
+ {
+ jid: "c3a74503dc2637f8f9445dd3",
+ worker_class: "Ci::ArchiveTraceWorker"
+ }
+ ]
+}
+```
+
+Where:
+
+- `memwd_rss_bytes` is the actual amount of memory consumed.
+- `memwd_max_rss_bytes` is the RSS limit set through `per_worker_max_memory_mb`.
+- `running jobs` lists the jobs that were running at the time when the process
+ exceeded the RSS limit and started a graceful restart.
diff --git a/doc/administration/sidekiq/sidekiq_troubleshooting.md b/doc/administration/sidekiq/sidekiq_troubleshooting.md
index b261e385949..8b95a9f6f0a 100644
--- a/doc/administration/sidekiq/sidekiq_troubleshooting.md
+++ b/doc/administration/sidekiq/sidekiq_troubleshooting.md
@@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w
Sidekiq is the background job processor GitLab uses to asynchronously run
tasks. When things go wrong it can be difficult to troubleshoot. These
situations also tend to be high-pressure because a production system job queue
-may be filling up. Users will notice when this happens because new branches
+may be filling up. Users notice when this happens because new branches
may not show up and merge requests may not be updated. The following are some
troubleshooting steps to help you diagnose the bottleneck.
@@ -105,7 +105,7 @@ Gather data on the state of the Sidekiq workers with the following Ruby script.
If the performance issue is intermittent:
- - Run this in a cron job every five minutes. Write the files to a location with enough space: allow for 500KB per file.
+ - Run this in a cron job every five minutes. Write the files to a location with enough space: allow for 500 KB per file.
- Refer back to the data to see what went wrong.
1. Analyze the output. The following commands assume that you have a directory of output files.
@@ -269,7 +269,7 @@ corresponding Ruby code where this is happening.
`gdb` can be another effective tool for debugging Sidekiq. It gives you a little
more interactive way to look at each thread and see what's causing problems.
-Attaching to a process with `gdb` suspends the normal operation
+Attaching to a process with `gdb` suspends the standard operation
of the process (Sidekiq does not process jobs while `gdb` is attached).
Start by attaching to the Sidekiq PID:
diff --git a/doc/administration/system_hooks.md b/doc/administration/system_hooks.md
index 038c26a9c2e..ddfa9fe9860 100644
--- a/doc/administration/system_hooks.md
+++ b/doc/administration/system_hooks.md
@@ -133,7 +133,7 @@ X-Gitlab-Event: System Hook
}
```
-Note that `project_rename` is not triggered if the namespace changes.
+`project_rename` is not triggered if the namespace changes.
Refer to `group_rename` and `user_rename` for that case.
**Project transferred:**
diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md
index 0cd34ca16df..ddee79046f6 100644
--- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md
+++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md
@@ -10,7 +10,7 @@ This was the GitLab Support Team's collection of information regarding the GitLa
console, for use while troubleshooting. It is listed here for posterity,
as most content has been moved to feature-specific troubleshooting pages and sections,
see epic [&8147](https://gitlab.com/groups/gitlab-org/-/epics/8147#tree).
-Please update your bookmarks accordingly.
+You may want to update your bookmarks accordingly.
If you are currently having an issue with GitLab,
it is highly recommended that you first check
diff --git a/doc/administration/troubleshooting/test_environments.md b/doc/administration/troubleshooting/test_environments.md
index c4d191f8c0f..63ccbce9480 100644
--- a/doc/administration/troubleshooting/test_environments.md
+++ b/doc/administration/troubleshooting/test_environments.md
@@ -87,7 +87,7 @@ docker.elastic.co/elasticsearch/elasticsearch:5.5.1
```
Then confirm it works in the browser at `curl "http://<IP_ADDRESS>:9200/_cat/health"`.
-Elasticsearch's default username is `elastic` and password is `changeme`.
+In Elasticsearch, the default username is `elastic`, and the default password is `changeme`.
### Kroki