diff options
Diffstat (limited to 'doc/administration')
88 files changed, 1884 insertions, 1579 deletions
diff --git a/doc/administration/audit_event_streaming.md b/doc/administration/audit_event_streaming.md index bbf4c0699ca..afc4926fec0 100644 --- a/doc/administration/audit_event_streaming.md +++ b/doc/administration/audit_event_streaming.md @@ -46,8 +46,8 @@ Users with the Owner role for a group can add streaming destinations for it: 1. Select **Add streaming destination** to show the section for adding destinations. 1. Enter the destination URL to add. 1. Optional. Locate the **Custom HTTP headers** table. -1. Ignore the **Active** checkbox because it isn't functional. To track progress on adding functionality to the **Active** checkbox, see the - [relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/361925). +1. Ignore the **Active** checkbox because it isn't functional. To track progress on adding functionality to the + **Active** checkbox, see [issue 367509](https://gitlab.com/gitlab-org/gitlab/-/issues/367509). 1. Select **Add header** to create a new name and value pair. Enter as many name and value pairs as required. You can add up to 20 headers per streaming destination. 1. After all headers have been filled out, select **Add** to add the new streaming destination. @@ -169,8 +169,8 @@ To update a streaming destinations custom HTTP headers: 1. To the right of the item, select **Edit** (**{pencil}**). 1. Locate the **Custom HTTP headers** table. 1. Locate the header that you wish to update. -1. Ignore the **Active** checkbox because it isn't functional. To track progress on adding functionality to the **Active** checkbox, see the - [relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/361925). +1. Ignore the **Active** checkbox because it isn't functional. To track progress on adding functionality to the + **Active** checkbox, see [issue 367509](https://gitlab.com/gitlab-org/gitlab/-/issues/367509). 1. Select **Add header** to create a new name and value pair. Enter as many name and value pairs as required. You can add up to 20 headers per streaming destination. 1. Select **Save** to update the streaming destination. diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index 47833224c0a..92bd73d1a75 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -216,6 +216,8 @@ The following actions on groups generate group audit events: [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/377877) in GitLab 15.6. - An environment is protected or unprotected. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216164) in GitLab 15.8. +- Changes to Code Suggestions. + [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/405295) in GitLab 15.11. ### Project events @@ -316,48 +318,62 @@ The following actions on projects generate project audit events: ### GitLab agent for Kubernetes events -The following actions on projects generate agent audit events: +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/382133) in GitLab 15.10. -- A cluster agent token is created. - Introduced in GitLab 15.9 -- A cluster agent token is revoked. - Introduced in GitLab 15.9 +GitLab generates audit events when a cluster agent token is created or revoked. ### Instance events **(PREMIUM SELF)** +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16826) in GitLab 13.5, audit events for failed second-factor authentication attempt. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276250) in GitLab 13.6, audit events for when a user is approved using the Admin Area. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276921) in GitLab 13.6, audit events for when a user's personal access token is successfully or unsuccessfully created or revoked. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/298783) in GitLab 13.9, audit events for when a user requests access to an instance or is rejected using the Admin Area. + The following user actions on a GitLab instance generate instance audit events: -- Sign-in events and the authentication type (such as standard, LDAP, or OmniAuth) -- Failed sign-ins -- Added SSH key -- Added or removed email -- Changed password -- Ask for password reset -- Grant OAuth access -- Started or stopped user impersonation -- Changed username. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7797) in GitLab 12.8. -- User was deleted. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/251) in GitLab 12.8. -- User was added. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/251) in GitLab 12.8. -- User requests access to an instance. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/298783) in GitLab 13.9. -- User was approved using the Admin Area. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276250) in GitLab 13.6. -- User was rejected using the Admin Area. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/298783) in GitLab 13.9. -- User was blocked using the Admin Area. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/251) in GitLab 12.8. -- User was blocked using the API. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25872) in GitLab 12.9. -- Failed second-factor authentication attempt. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16826) in - GitLab 13.5. -- A user's personal access token was successfully created or revoked. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276921) in GitLab 13.6. -- A failed attempt to create or revoke a user's personal access token. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276921) in GitLab 13.6. +- Sign-in events and the authentication type such as standard, LDAP, or OmniAuth. +- Failed sign-ins. +- Added SSH key. +- Added or removed email. +- Changed password. +- Ask for password reset. +- Grant OAuth access. +- Started or stopped user impersonation. +- Changed username. +- User was added or deleted. +- User requests access to an instance. +- User was approved, rejected, or blocked using the Admin Area. +- User was blocked using the API. +- Failed second-factor authentication attempt. +- A user's personal access token was successfully or unsuccessfully created or revoked. - Administrator added or removed. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/323905) in GitLab 14.1. - Removed SSH key. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/220127) in GitLab 14.1. - Added or removed GPG key. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/220127) in GitLab 14.1. - A user's two-factor authentication was disabled. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/238177) in GitLab 15.1. - Enabled Admin Mode. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/362101) in GitLab 15.7. +- All [group events](#group-events) and [project events](#project-events). +- User was unblocked using the Admin Area or API. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/115727) in GitLab 15.11. +- User was banned using the Admin Area or API. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/116103) in GitLab 15.11. +- User was unbanned using the Admin Area or API. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/116221) in GitLab 15.11. Instance events can also be accessed using the [Instance Audit Events API](../api/audit_events.md#instance-audit-events). +### GitLab Runner events + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/335509) in GitLab 14.8, audit events for when a runner is registered. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/349540) in GitLab 14.9, audit events for when a runner is unregistered. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/349542) in GitLab 14.9, audit events for when a runner is assigned to or unassigned from a project. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355637) in GitLab 15.0, audit events for when a runner registration token is reset. + +GitLab generates audit events for the following GitLab Runner actions: + +- Instance, group, or project runner is registered. +- Instance, group, or project runner is unregistered. +- Runner is assigned to or unassigned from a project. +- Instance, group, or project runner registration token is reset. + [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/102579) in GitLab 15.6. + ## "Deleted User" events Audit events created after users are deleted are created for "Deleted User". For example, if a deleted user's access to @@ -373,6 +389,7 @@ Some events are not tracked in audit events. The following epics and issues prop - [Group settings and activity](https://gitlab.com/groups/gitlab-org/-/epics/475). - [Instance-level settings and activity](https://gitlab.com/groups/gitlab-org/-/epics/476). - [Deployment Approval activity](https://gitlab.com/gitlab-org/gitlab/-/issues/354782). +- [Approval rules processing by a non GitLab user](https://gitlab.com/gitlab-org/gitlab/-/issues/407384). If you don't see the event you want in any of the epics, you can either: diff --git a/doc/administration/auth/atlassian.md b/doc/administration/auth/atlassian.md index 58412078802..45617b9965c 100644 --- a/doc/administration/auth/atlassian.md +++ b/doc/administration/auth/atlassian.md @@ -41,7 +41,7 @@ To enable the Atlassian OmniAuth provider for passwordless authentication you mu sudo -u git -H editor /home/git/gitlab/config/gitlab.yml ``` -1. Edit the [common configuration file settings](../../integration/omniauth.md#configure-common-settings) +1. Configure the [common settings](../../integration/omniauth.md#configure-common-settings) to add `atlassian_oauth2` as a single sign-on provider. This enables Just-In-Time account provisioning for users who do not have an existing GitLab account. diff --git a/doc/administration/auth/cognito.md b/doc/administration/auth/cognito.md index a73595c731b..cfac958e297 100644 --- a/doc/administration/auth/cognito.md +++ b/doc/administration/auth/cognito.md @@ -41,7 +41,7 @@ To enable AWS Cognito as an authentication provider, complete the following step ## Configure GitLab -1. Edit the [common configuration file settings](../../integration/omniauth.md#configure-common-settings) +1. Configure the [common settings](../../integration/omniauth.md#configure-common-settings) to add `cognito` as a single sign-on provider. This enables Just-In-Time account provisioning for users who do not have an existing GitLab account. 1. On your GitLab server, open the configuration file. diff --git a/doc/administration/auth/crowd.md b/doc/administration/auth/crowd.md index 277e59b1a8a..e403eb800d6 100644 --- a/doc/administration/auth/crowd.md +++ b/doc/administration/auth/crowd.md @@ -40,7 +40,7 @@ this provider also allows Crowd authentication for Git-over-https requests. sudo -u git -H editor config/gitlab.yml ``` -1. Edit the [common configuration file settings](../../integration/omniauth.md#configure-common-settings) +1. Configure the [common settings](../../integration/omniauth.md#configure-common-settings) to add `crowd` as a single sign-on provider. This enables Just-In-Time account provisioning for users who do not have an existing GitLab account. diff --git a/doc/administration/auth/index.md b/doc/administration/auth/index.md index 2b978dd1f2c..4a8e230a944 100644 --- a/doc/administration/auth/index.md +++ b/doc/administration/auth/index.md @@ -1,5 +1,4 @@ --- -comments: false type: index stage: Manage group: Authentication and Authorization @@ -34,3 +33,7 @@ For more information, see the links shown on this page for each external provide | **User Removal** | SCIM (remove user from top-level group) | LDAP (remove user from groups and block from the instance)<br>SCIM | 1. Using Just-In-Time (JIT) provisioning, user accounts are created when the user first signs in. + +## Test OIDC/OAuth in GitLab + +See [Test OIDC/OAuth in GitLab](test_oidc_oauth.md) to learn how to test OIDC/OAuth authentication in your GitLab instance using your client application. diff --git a/doc/administration/auth/jwt.md b/doc/administration/auth/jwt.md index 8c9800aa792..9994b374038 100644 --- a/doc/administration/auth/jwt.md +++ b/doc/administration/auth/jwt.md @@ -25,7 +25,7 @@ JWT provides you with a secret key for you to use. sudo -u git -H editor config/gitlab.yml ``` -1. Edit the [common configuration file settings](../../integration/omniauth.md#configure-common-settings) +1. Configure the [common settings](../../integration/omniauth.md#configure-common-settings) to add `jwt` as a single sign-on provider. This enables Just-In-Time account provisioning for users who do not have an existing GitLab account. 1. Add the provider configuration. diff --git a/doc/administration/auth/ldap/google_secure_ldap.md b/doc/administration/auth/ldap/google_secure_ldap.md index e0612099221..042a65be500 100644 --- a/doc/administration/auth/ldap/google_secure_ldap.md +++ b/doc/administration/auth/ldap/google_secure_ldap.md @@ -18,6 +18,8 @@ The steps below cover: - Configuring the Secure LDAP Client in the Google administrator console. - Required GitLab configuration. +Secure LDAP is only available on specific Google Workspace editions. For more information, see the [Google Secure LDAP service documentation](https://support.google.com/a/answer/9048516). + ## Configuring Google LDAP client 1. Go to <https://admin.google.com/Dashboard> and sign in as a Google Workspace domain administrator. @@ -88,7 +90,7 @@ values obtained during the LDAP client configuration earlier: encryption: 'simple_tls' verify_certificates: true retry_empty_result_with_codes: [80] - + base: "DC=example,DC=com" tls_options: cert: | -----BEGIN CERTIFICATE----- @@ -152,7 +154,7 @@ values obtained during the LDAP client configuration earlier: servers: main: # 'main' is the GitLab 'provider ID' of this LDAP server label: 'Google Secure LDAP' - + base: "DC=example,DC=com" host: 'ldap.google.com' port: 636 uid: 'uid' diff --git a/doc/administration/auth/ldap/ldap_synchronization.md b/doc/administration/auth/ldap/ldap_synchronization.md index 59ed3a4153d..f32a4af9e27 100644 --- a/doc/administration/auth/ldap/ldap_synchronization.md +++ b/doc/administration/auth/ldap/ldap_synchronization.md @@ -7,12 +7,16 @@ info: To determine the technical writer assigned to the Stage/Group associated w # LDAP synchronization **(PREMIUM SELF)** If you have [configured LDAP to work with GitLab](index.md), GitLab can automatically synchronize -users and groups. This process updates user and group information. +users and groups. + +LDAP synchronization updates user and group information for existing GitLab users that have an LDAP identity assigned. It does not create new GitLab users through LDAP. You can change when synchronization occurs. ## User sync +> Preventing LDAP username synchronization [introduced](<https://gitlab.com/gitlab-org/gitlab/-/issues/11336>) in GitLab 15.11. + Once per day, GitLab runs a worker to check and update GitLab users against LDAP. @@ -36,11 +40,109 @@ For more information, see [Bitmask Searches in LDAP](https://ctovswild.com/2009/ The process also updates the following user information: - Name. Because of a [sync issue](https://gitlab.com/gitlab-org/gitlab/-/issues/342598), `name` is not synchronized if - [**Prevent users from changing their profile name**](../../../user/admin_area/settings/account_and_limit_settings.md#disable-user-profile-name-changes) is enabled. + [**Prevent users from changing their profile name**](../../../user/admin_area/settings/account_and_limit_settings.md#disable-user-profile-name-changes) is enabled or `sync_name` is set to `false`. - Email address. - SSH public keys if `sync_ssh_keys` is set. - Kerberos identity if Kerberos is enabled. +### Synchronize LDAP username + +By default, GitLab synchronizes the LDAP username field. + +To prevent this synchronization, you can set `sync_name` to `false`. + +::Tabs + +:::TabTitle Linux package (Omnibus) + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_servers'] = { + 'main' => { + 'sync_name' => false, + } + } + ``` + +1. Save the file and reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +:::TabTitle Helm chart (Kubernetes) + +1. Export the Helm values: + + ```shell + helm get values gitlab > gitlab_values.yaml + ``` + +1. Edit `gitlab_values.yaml`: + + ```yaml + global: + appConfig: + ldap: + servers: + main: + sync_name: false + ``` + +1. Save the file and apply the new values: + + ```shell + helm upgrade -f gitlab_values.yaml gitlab gitlab/gitlab + ``` + +:::TabTitle Docker + +1. Edit `docker-compose.yml`: + + ```yaml + version: "3.6" + services: + gitlab: + environment: + GITLAB_OMNIBUS_CONFIG: | + gitlab_rails['ldap_servers'] = { + 'main' => { + 'sync_name' => false, + } + } + ``` + +1. Save the file and restart GitLab: + + ```shell + docker compose up -d + ``` + +:::TabTitle Self-compiled (source) + +1. Edit `/home/git/gitlab/config/gitlab.yml`: + + ```yaml + production: &base + ldap: + servers: + main: + sync_name: false + ``` + +1. Save the file and restart GitLab: + + ```shell + # For systems running systemd + sudo systemctl restart gitlab.target + + # For systems running SysV init + sudo service gitlab restart + ``` + +::EndTabs + ### Blocked users A user is blocked if either the: diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md index 4edc00b0d62..afc2f74201a 100644 --- a/doc/administration/auth/oidc.md +++ b/doc/administration/auth/oidc.md @@ -29,7 +29,7 @@ The OpenID Connect provides you with a client's details and secret for you to us sudo -u git -H editor config/gitlab.yml ``` -1. Edit the [common configuration file settings](../../integration/omniauth.md#configure-common-settings) +1. Configure the [common settings](../../integration/omniauth.md#configure-common-settings) to add `openid_connect` as a single sign-on provider. This enables Just-In-Time account provisioning for users who do not have an existing GitLab account. @@ -40,7 +40,7 @@ The OpenID Connect provides you with a client's details and secret for you to us ```ruby gitlab_rails['omniauth_providers'] = [ { - name: "openid_connect", + name: "openid_connect", # do not change this parameter label: "Provider name", # optional label for login button, defaults to "Openid Connect" icon: "<custom_provider_icon>", args: { @@ -66,7 +66,7 @@ The OpenID Connect provides you with a client's details and secret for you to us For installation from source: ```yaml - - { name: 'openid_connect', + - { name: 'openid_connect', # do not change this parameter label: 'Provider name', # optional label for login button, defaults to "Openid Connect" icon: '<custom_provider_icon>', args: { @@ -165,7 +165,7 @@ for more details: ```ruby gitlab_rails['omniauth_providers'] = [ { - name: "openid_connect", + name: "openid_connect", # do not change this parameter label: "Google OpenID", # optional label for login button, defaults to "Openid Connect" args: { name: "openid_connect", @@ -203,7 +203,7 @@ Example Omnibus configuration block: ```ruby gitlab_rails['omniauth_providers'] = [ { - name: "openid_connect", + name: "openid_connect", # do not change this parameter label: "Azure OIDC", # optional label for login button, defaults to "Openid Connect" args: { name: "openid_connect", @@ -335,7 +335,7 @@ but `LocalAccounts` authenticates against local Active Directory accounts. Befor ```ruby gitlab_rails['omniauth_providers'] = [ { - name: "openid_connect", + name: "openid_connect", # do not change this parameter label: "Azure B2C OIDC", # optional label for login button, defaults to "Openid Connect" args: { name: "openid_connect", @@ -395,7 +395,7 @@ Example Omnibus configuration block: ```ruby gitlab_rails['omniauth_providers'] = [ { - name: "openid_connect", + name: "openid_connect", # do not change this parameter label: "Keycloak", # optional label for login button, defaults to "Openid Connect" args: { name: "openid_connect", @@ -475,7 +475,7 @@ To use symmetric key encryption: ```ruby gitlab_rails['omniauth_providers'] = [ { - name: "openid_connect", + name: "openid_connect", # do not change this parameter label: "Keycloak", # optional label for login button, defaults to "Openid Connect" args: { name: "openid_connect", @@ -519,7 +519,7 @@ Example Omnibus GitLab configuration (file path: `/etc/gitlab/gitlab.rb`): ```ruby gitlab_rails['omniauth_providers'] = [ { - name: "openid_connect", + name: "openid_connect", # do not change this parameter label: "Casdoor", # optional label for login button, defaults to "Openid Connect" args: { name: "openid_connect", @@ -542,7 +542,7 @@ gitlab_rails['omniauth_providers'] = [ Example installations from source configuration (file path: `config/gitlab.yml`): ```yaml - - { name: 'openid_connect', + - { name: 'openid_connect', # do not change this parameter label: 'Casdoor', # optional label for login button, defaults to "Openid Connect" args: { name: 'openid_connect', diff --git a/doc/administration/auth/test_oidc_oauth.md b/doc/administration/auth/test_oidc_oauth.md new file mode 100644 index 00000000000..95cca1ced86 --- /dev/null +++ b/doc/administration/auth/test_oidc_oauth.md @@ -0,0 +1,57 @@ +--- +stage: Manage +group: Authentication and Authorization +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Test OIDC/OAuth in GitLab **(FREE)** + +To test OIDC/OAuth in GitLab, you must: + +1. [Enable OIDC/OAuth](#enable-oidcoauth-in-gitlab) +1. [Test OIDC/OAuth with your client application](#test-oidcoauth-with-your-client-application) +1. [Verify OIDC/OAuth authentication](#verify-oidcoauth-authentication) + +## Prerequisites + +Before you can test OIDC/OAuth on GitLab, you'll need the following: + +- Publicly accessible GitLab instance +- A client application that you want to use to test OIDC/OAuth +- A user account on the GitLab instance that you can use to log in and test OIDC/OAuth + +## Enable OIDC/OAuth in GitLab + +First, you must create OIDC/OAuth application on your GitLab instance. To do this: + +1. Sign in to GitLab as an administrator. +1. Select **Profile > Preferences > Applications**. +1. Fill in the details for your client application, including the name, redirect URI, and allowed scopes. +1. Make sure the `openid` scope is enabled. +1. Select **Save application** to create the new OAuth application. + +## Test OIDC/OAuth with your client application + +After you've created your OAuth application in GitLab, you can use it to test OIDC/OAuth: + +1. You can use <https://openidconnect.net> as the OIDC/OAuth playground. +1. Sign out of GitLab. +1. Visit your client application and initiate the OIDC/OAuth flow, using the GitLab OAuth application you created in the previous step. +1. Follow the prompts to sign in to GitLab and authorize the client application to access your GitLab account. +1. After you've completed the OIDC/OAuth flow, your client application should have received an access token that it can use to authenticate with GitLab. + +## Verify OIDC/OAuth authentication + +To verify that OIDC/OAuth authentication is working correctly on GitLab, you can perform the following checks: + +1. Check that the access token you received in the previous step is valid and can be used to authenticate with GitLab. You can do this by making a test API request to GitLab, using the access token to authenticate. For example: + + ```shell + curl --header "Authorization: Bearer <access_token>" https://mygitlabinstance.com/api/v4/user + ``` + + Replace `<access_token>` with the actual access token you received in the previous step. If the API request succeeds and returns information about the authenticated user, then OIDC/OAuth authentication is working correctly. + +1. Check that the scopes you specified in your OAuth application are being enforced correctly. You can do this by making API requests that require the specific scopes and checking that they succeed or fail as expected. + +That's it! With these steps, you should be able to test OIDC/OAuth authentication on your GitLab instance using your client application. diff --git a/doc/administration/clusters/kas.md b/doc/administration/clusters/kas.md index 7a289b6a500..6d6e8e5513c 100644 --- a/doc/administration/clusters/kas.md +++ b/doc/administration/clusters/kas.md @@ -1,6 +1,6 @@ --- -stage: Configure -group: Configure +stage: Deploy +group: Environments info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -43,6 +43,33 @@ To enable the agent server on a single node: For additional configuration options, see the **Enable GitLab KAS** section of the [`gitlab.rb.template`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/files/gitlab-config-template/gitlab.rb.template). +##### Configure KAS to listen on a UNIX socket + +If you use GitLab behind a proxy, KAS might not work correctly. You can resolve this issue on a single-node installation, you can configure KAS to listen on a UNIX socket. + +To configure KAS to listen on a UNIX socket: + +1. Create a directory for the KAS sockets: + + ```shell + sudo mkdir -p /var/opt/gitlab/gitlab-kas/sockets/ + ``` + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_kas['internal_api_listen_network'] = 'unix' + gitlab_kas['internal_api_listen_address'] = '/var/opt/gitlab/gitlab-kas/sockets/internal-api.socket' + gitlab_kas['private_api_listen_network'] = 'unix' + gitlab_kas['private_api_listen_address'] = '/var/opt/gitlab/gitlab-kas/sockets/private-api.socket' + gitlab_kas['env'] = { + 'SSL_CERT_DIR' => "/opt/gitlab/embedded/ssl/certs/", + 'OWN_PRIVATE_API_URL' => 'unix:///var/opt/gitlab/gitlab-kas/sockets/private-api.socket' + } + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + #### Enable on multiple nodes To enable the agent server on multiple nodes: diff --git a/doc/administration/compliance.md b/doc/administration/compliance.md index 592c110b446..9ae0e9d7790 100644 --- a/doc/administration/compliance.md +++ b/doc/administration/compliance.md @@ -18,25 +18,13 @@ standards or mandates from regulatory bodies. The following features help you define rules and policies to adhere to workflow requirements, separation of duties, and secure supply chain best practices: -- [**Credentials inventory**](../user/admin_area/credentials_inventory.md) (for - instances): With a credentials inventory, GitLab administrators can keep track - of the credentials used by all of the users in their GitLab instance. -- [**Granular user roles and flexible permissions**](../user/permissions.md) - (for instances, groups, and projects): Manage access and permissions with five - different user roles and settings for external users. Set permissions according - to people's role, rather than either read or write access to a repository. Don't - share the source code with people that only need access to the issue tracker. -- [**Merge request approvals**](../user/project/merge_requests/approvals/index.md) - (for instances, groups, and projects): Configure approvals required for - merge requests. -- [**Push rules**](../user/project/repository/push_rules.md) (for instances, groups, and - projects): Control pushes to your repositories. -- Separation of duties using [**protected branches**](../user/project/protected_branches.md#require-code-owner-approval-on-a-protected-branch) - and [**custom CI/CD configuration paths**](../ci/pipelines/settings.md#specify-a-custom-cicd-configuration-file) (for projects): Users can leverage the GitLab cross-project YAML configurations - to define deployers of code and developers of code. See how to use this setup - to define these roles in: - - The [Separation of Duties deploy project](https://gitlab.com/guided-explorations/separation-of-duties-deploy/blob/master/README.md). - - The [Separation of Duties project](https://gitlab.com/guided-explorations/separation-of-duties/blob/master/README.md). +| Feature | Instances | Groups | Projects | Description | +|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------|:-----------------------|:-----------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| [Credentials inventory](../user/admin_area/credentials_inventory.md) | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Keep track of the credentials used by all of the users in a GitLab instance. | +| [Granular user roles<br/>and flexible permissions](../user/permissions.md) | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Manage access and permissions with five different user roles and settings for external users. Set permissions according to people's role, rather than either read or write access to a repository. Don't share the source code with people that only need access to the issue tracker. | +| [Merge request approvals](../user/project/merge_requests/approvals/index.md) | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Configure approvals required for merge requests. | +| [Push rules](../user/project/repository/push_rules.md) | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Control pushes to your repositories. | +| Separation of duties using<br/>[protected branches](../user/project/protected_branches.md#require-code-owner-approval-on-a-protected-branch) and<br/>[custom CI/CD configuration paths](../ci/pipelines/settings.md#specify-a-custom-cicd-configuration-file) | **{dotted-circle}** No | **{dotted-circle}** No | **{check-circle}** Yes | Leverage the GitLab cross-project YAML configurations to define deployers of code and developers of code. See how to use this setup to define these roles in the [Separation of Duties deploy project](https://gitlab.com/guided-explorations/separation-of-duties-deploy/blob/master/README.md) and the [Separation of Duties project](https://gitlab.com/guided-explorations/separation-of-duties/blob/master/README.md). | ## Compliant workflow automation @@ -48,66 +36,42 @@ settings and automation to ensure that whatever a compliance team has configured stays configured and working correctly. These features can help you automate compliance: -- [**Compliance frameworks**](../user/group/compliance_frameworks.md) (for groups): Create a custom - compliance framework at the group level to describe the type of compliance requirements any child project needs to follow. -- [**Compliance pipelines**](../user/group/compliance_frameworks.md#compliance-pipelines) (for groups): Define a - pipeline configuration to run for any projects with a given compliance framework. +| Feature | Instances | Groups | Projects | Description | +|:------------------------------------------------------------------------------------|:-----------------------|:-----------------------|:-----------------------|:-------------------------------------------------------------------------------------------| +| [Compliance frameworks](../user/group/compliance_frameworks.md) | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Describe the type of compliance requirements projects must follow. | +| [Compliance pipelines](../user/group/compliance_frameworks.md#compliance-pipelines) | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Define a pipeline configuration to run for any projects with a given compliance framework. | ## Audit management An important part of any compliance program is being able to go back and understand -what happened, when it happened, and who was responsible. This is useful in audit +what happened, when it happened, and who was responsible. You can use this in audit situations as well as for understanding the root cause of issues when they occur. -It is useful to have both low-level, raw lists of audit data as well as high-level, + +It is helpful to have both low-level, raw lists of audit data as well as high-level, summary lists of audit data. Between these two, compliance teams can quickly identify if problems exist and then drill down into the specifics of those issues. These features can help provide visibility into GitLab and audit what is happening: -- [**Audit events**](audit_events.md) (for instances, groups, and projects): To - maintain the integrity of your code, audit events give administrators the - ability to view any modifications made within the GitLab server in an advanced - audit events system, so you can control, analyze, and track every change. -- [**Audit reports**](audit_reports.md) (for instances, groups, and projects): - Create and access reports based on the audit events that have occurred. Use - pre-built GitLab reports or the API to build your own. -- [**Auditor users**](auditor_users.md) (for instances): Auditor users are users - who are given read-only access to all projects, groups, and other resources on - the GitLab instance. -- [**Compliance report**](../user/compliance/compliance_report/index.md) (for - groups): Quickly get visibility into the compliance posture of your organization. +| Feature | Instances | Groups | Projects | Description | +|:-------------------------------------------------------------------|:-----------------------|:-----------------------|:-----------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| [Audit events](audit_events.md) | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | To maintain the integrity of your code, audit events give administrators the ability to view any modifications made in the GitLab server in an advanced audit events system, so you can control, analyze, and track every change. | +| [Audit reports](audit_reports.md) | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Create and access reports based on the audit events that have occurred. Use pre-built GitLab reports or the API to build your own. | +| [Auditor users](auditor_users.md) | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Auditor users are users who are given read-only access to all projects, groups, and other resources on the GitLab instance. | +| [Compliance report](../user/compliance/compliance_report/index.md) | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Quickly get visibility into the compliance posture of your organization. | ## Other compliance features These features can also help with compliance requirements: -- [**Email all users of a project, group, or entire server**](../user/admin_area/email_from_gitlab.md) - (for instances): An administrator can email groups of users based on project - or group membership, or email everyone using the GitLab instance. These emails - are great for scheduled maintenance or upgrades. -- [**Enforce ToS acceptance**](../user/admin_area/settings/terms.md) (for - instances): Enforce your users accepting new terms of service by blocking GitLab - traffic. -- [**External Status Checks**](../user/project/merge_requests/status_checks.md) - (for projects): Interface with third-party systems you already use during - development to ensure you remain compliant. -- [**Generate reports on permission levels of users**](../user/admin_area/index.md#user-permission-export) - (for instances): Administrators can generate a report listing all users' access - permissions for groups and projects in the instance. -- [**License compliance**](../user/compliance/license_compliance/index.md) (for - projects): Search dependencies for their licenses. This lets you determine if - the licenses of your project's dependencies are compatible with your project's - license. -- [**Lock project membership to group**](../user/group/access_and_permissions.md#prevent-members-from-being-added-to-projects-in-a-group) - (for groups): Group owners can prevent new members from being added to projects - within a group. -- [**LDAP group sync**](auth/ldap/ldap_synchronization.md#group-sync) (for - instances): Gives administrators the ability to automatically sync groups and - manage SSH keys, permissions, and authentication, so you can focus on building - your product, not configuring your tools. -- [**LDAP group sync filters**](auth/ldap/ldap_synchronization.md#group-sync) - (for instances): Gives more flexibility to synchronize with LDAP based on - filters, meaning you can leverage LDAP attributes to map GitLab permissions. -- [**Omnibus GitLab package supports log forwarding**](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-forwarding) - (for instances): Forward your logs to a central system. -- [**Restrict SSH Keys**](../security/ssh_keys_restrictions.md) (for instances): - Control the technology and key length of SSH keys used to access GitLab. +| Feature | Instances | Groups | Projects | Description | +|:------------------------------------------------------------------------------------------------------------------------------------|:-----------------------|:-----------------------|:-----------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| [Email all users of a project,<br/>group, or entire server](../user/admin_area/email_from_gitlab.md) | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Email groups of users based on project or group membership, or email everyone using the GitLab instance. These emails are great for scheduled maintenance or upgrades. | +| [Enforce ToS acceptance](../user/admin_area/settings/terms.md) | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Enforce your users accepting new terms of service by blocking GitLab traffic. | +| [External Status Checks](../user/project/merge_requests/status_checks.md) | **{dotted-circle}** No | **{dotted-circle}** No | **{check-circle}** Yes | Interface with third-party systems you already use during development to ensure you remain compliant. | +| [Generate reports on permission<br/>levels of users](../user/admin_area/index.md#user-permission-export) | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Generate a report listing all users' access permissions for groups and projects in the instance. | +| [License compliance](../user/compliance/license_compliance/index.md) | **{dotted-circle}** No | **{dotted-circle}** No | **{check-circle}** Yes | Search dependencies for their licenses. This lets you determine if the licenses of your project's dependencies are compatible with your project's license. | +| [Lock project membership to group](../user/group/access_and_permissions.md#prevent-members-from-being-added-to-projects-in-a-group) | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Group owners can prevent new members from being added to projects in a group. | +| [LDAP group sync](auth/ldap/ldap_synchronization.md#group-sync) | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Automatically synchronize groups and manage SSH keys, permissions, and authentication, so you can focus on building your product, not configuring your tools. | +| [LDAP group sync filters](auth/ldap/ldap_synchronization.md#group-sync) | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Gives more flexibility to synchronize with LDAP based on filters, meaning you can leverage LDAP attributes to map GitLab permissions. | +| [Omnibus GitLab package supports<br/>log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-forwarding) | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Forward your logs to a central system. | +| [Restrict SSH Keys](../security/ssh_keys_restrictions.md) | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Control the technology and key length of SSH keys used to access GitLab. | diff --git a/doc/administration/configure.md b/doc/administration/configure.md index bf618345a94..d68e81ebf69 100644 --- a/doc/administration/configure.md +++ b/doc/administration/configure.md @@ -7,44 +7,39 @@ type: reference # Configure your GitLab installation **(FREE SELF)** -Customize and configure your self-managed GitLab installation. Here are some quick links to get you started: +Customize and configure your self-managed GitLab installation. - [Authentication](auth/index.md) +- [CI/CD](../administration/cicd.md) - [Configuration](../user/admin_area/index.md) -- [Repository storage](repository_storage_paths.md) -- [Geo](geo/index.md) -- [Packages](packages/index.md) - -The following tables are intended to guide you to choose the right combination of capabilities based on your requirements. It is common to want the most -available, quickly recoverable, highly performant, and fully data resilient solution. However, there are tradeoffs. - -The tables lists features on the left and provides their capabilities to the right along with known trade-offs. - -## Gitaly Capabilities - -| | Availability | Recoverability | Data Resiliency | Performance | Risks/Trade-offs| -|-|--------------|----------------|-----------------|-------------|-----------------| -|Gitaly Cluster | Very high - tolerant of node failures | RTO for a single node of 10 s with no manual intervention | Data is stored on multiple nodes | Good - While writes may take slightly longer due to voting, read distribution improves read speeds | **Trade-off** - Slight decrease in write speed for redundant, strongly-consistent storage solution. **Risks** - [Does not support snapshot backups](gitaly/index.md#snapshot-backup-and-recovery-limitations), GitLab backup task can be slow for large data sets | -|Gitaly Shards | Single storage location is a single point of failure | Would need to restore only shards which failed | Single point of failure | Good - can allocate repositories to shards to spread load | **Trade-off** - Need to manually configure repositories into different shards to balance loads / storage space **Risks** - Single point of failure relies on recovery process when single-node failure occurs | -|Gitaly + NFS | Single storage location is a single point of failure | Single node failure requires restoration from backup | Single point of failure | Average - NFS is not ideally suited to large quantities of small reads / writes which can have a detrimental impact on performance | **Trade-off** - Familiar administration though NFS is not ideally suited to Git demands **Risks** - Many instances of NFS compatibility issues which provide very poor customer experiences | - -## Geo Capabilities - -If your availability needs to span multiple zones or multiple locations, read about [Geo](geo/index.md). - -| | Availability | Recoverability | Data Resiliency | Performance | Risks/Trade-offs| -|-|--------------|----------------|-----------------|-------------|-----------------| -|Geo| Depends on the architecture of the Geo site. It is possible to deploy secondaries in single and multiple node configurations. | Eventually consistent. Recovery point depends on replication lag, which depends on a number of factors such as network speeds. Geo supports failover from a primary to secondary site using manual commands that are scriptable. | Geo replicates 100% of planned data types and verifies 50%. See [limitations table](geo/replication/datatypes.md#limitations-on-replicationverification) for more detail. | Improves read/clone times for users of a secondary. | Geo is not intended to replace other backup/restore solutions. Because of replication lag and the possibility of replicating bad data from a primary, customers should also take regular backups of their primary site and test the restore process. | - -## Scenarios for failure modes and available mitigation paths - -The following table outlines failure modes and mitigation paths for the product offerings detailed in the tables above. Note - Gitaly Cluster install assumes an odd number replication factor of 3 or greater - -| Gitaly Mode | Loss of Single Gitaly Node | Application / Data Corruption | Regional Outage (Loss of Instance) | Notes | -| ----------- | -------------------------- | ----------------------------- | ---------------------------------- | ----- | -| Single Gitaly Node | Downtime - Must restore from backup | Downtime - Must restore from Backup | Downtime - Must wait for outage to end | | -| Single Gitaly Node + Geo Secondary | Downtime - Must restore from backup, can perform a manual failover to secondary | Downtime - Must restore from Backup, errors could have propagated to secondary | Manual intervention - failover to Geo secondary | | -| Sharded Gitaly Install | Partial Downtime - Only repositories on impacted node affected, must restore from backup | Partial Downtime - Only repositories on impacted node affected, must restore from backup | Downtime - Must wait for outage to end | | -| Sharded Gitaly Install + Geo Secondary | Partial Downtime - Only repositories on impacted node affected, must restore from backup, could perform manual failover to secondary for impacted repositories | Partial Downtime - Only repositories on impacted node affected, must restore from backup, errors could have propagated to secondary | Manual intervention - failover to Geo secondary | | -| Gitaly Cluster Install* | No Downtime - swaps repository primary to another node after 10 seconds | Not applicable; All writes are voted on by multiple Gitaly Cluster nodes | Downtime - Must wait for outage to end | Snapshot backups for Gitaly Cluster nodes not supported at this time | -| Gitaly Cluster Install* + Geo Secondary | No Downtime - swaps repository primary to another node after 10 seconds | Not applicable; All writes are voted on by multiple Gitaly Cluster nodes | Manual intervention - failover to Geo secondary | Snapshot backups for Gitaly Cluster nodes not supported at this time | +- [Consul](../administration/consul.md) +- [Environment variables](../administration/environment_variables.md) +- [File hooks](../administration/file_hooks.md) +- [Git protocol v2](../administration/git_protocol.md) +- [Incoming email](../administration/incoming_email.md) +- [Instance limits](../administration/instance_limits.md) +- [Instance Review](../administration/instance_review.md) +- [PostgreSQL](../administration/postgresql/index.md) +- [Load balancer](../administration/load_balancer.md) +- [NFS](../administration/nfs.md) +- [Postfix](../administration/reply_by_email_postfix_setup.md) +- [Redis](../administration/redis/index.md) +- [Sidekiq](../administration/sidekiq/index.md) +- [S/MIME signing](../administration/smime_signing_email.md) +- [Repository storage](../administration/repository_storage_paths.md) +- [Object storage](../administration/object_storage.md) +- [Merge request diffs storage](../administration/merge_request_diffs.md) +- [Static objects external storage](../administration/static_objects_external_storage.md) +- [Geo](../administration/geo/index.md) +- [Disaster recovery (Geo)](../administration/geo/disaster_recovery/index.md) +- [Agent server for Kubernetes](../administration/clusters/kas.md) +- [Server hooks](../administration/server_hooks.md) +- [Terraform state](../administration/terraform_state.md) +- [Terraform limits](../user/admin_area/settings/terraform_limits.md) +- [Packages](../administration/packages/index.md) +- [Web terminals](../administration/integration/terminal.md) +- [Wikis](../administration/wikis/index.md) +- [Invalidate Markdown cache](../administration/invalidate_markdown_cache.md) +- [Issue closing pattern](../administration/issue_closing_pattern.md) +- [Snippets](../administration/snippets/index.md) +- [Host the product documentation](../administration/docs_self_host.md) diff --git a/doc/administration/dedicated/index.md b/doc/administration/dedicated/index.md index 1991d0b64cc..25b53c97d8b 100644 --- a/doc/administration/dedicated/index.md +++ b/doc/administration/dedicated/index.md @@ -203,7 +203,7 @@ Prerequisites: To activate SAML for your GitLab Dedicated instance: -1. Read the [GitLab documentation about SAML](../../integration/saml.md#https://docs.gitlab.com/ee/integration/saml.html#configure-saml-on-your-idp) to gather all data your identity provider requires for configuration. You can also find some providers and their requirements in the [group SAML documentation](../../user/group/saml_sso/index.md#set-up-identity-provider). +1. Read the [GitLab documentation about SAML](../../integration/saml.md#https://docs.gitlab.com/ee/integration/saml.html#configure-saml-on-your-idp) to gather all data your identity provider requires for configuration. You can also find some providers and their requirements in the [group SAML documentation](../../user/group/saml_sso/index.md#set-up-your-identity-provider). 1. To make the necessary changes, include in your [support ticket](#configuration-changes) the desired [SAML configuration block](../../integration/saml.md#configure-saml-support-in-gitlab) that will be set on the GitLab application. At a minimum, GitLab needs the following information to enable SAML for your instance: - Assertion consumer service URL - Certificate fingerprint or certificate diff --git a/doc/administration/docs_self_host.md b/doc/administration/docs_self_host.md index 932ffb87288..a659508e37a 100644 --- a/doc/administration/docs_self_host.md +++ b/doc/administration/docs_self_host.md @@ -265,8 +265,5 @@ To upgrade to a later version [using your own web-server](#self-host-the-product If you self-host the product documentation: -- The version dropdown list displays additional versions that don't exist. Selecting - these versions displays a `404 Not Found` page. -- The search displays results from `docs.gitlab.com` and not the local site. - By default, the landing page redirects to the respective version (for example, `/14.5/`). This causes the landing page <https://docs.gitlab.com> to not be displayed. diff --git a/doc/administration/environment_variables.md b/doc/administration/environment_variables.md index b6a3a563ae1..a453d703a18 100644 --- a/doc/administration/environment_variables.md +++ b/doc/administration/environment_variables.md @@ -35,8 +35,8 @@ You can use the following environment variables to override certain values: | `GITLAB_ROOT_PASSWORD` | string | Sets the password for the `root` user on installation. | | `GITLAB_SHARED_RUNNERS_REGISTRATION_TOKEN` | string | Sets the initial registration token used for runners. | | `RAILS_ENV` | string | The Rails environment; can be one of `production`, `development`, `staging`, or `test`. | -| `UNSTRUCTURED_RAILS_LOG` | string | Enables the unstructured log in addition to JSON logs (defaults to `false`). | | `GITLAB_RAILS_CACHE_DEFAULT_TTL_SECONDS` | integer | The default TTL used for entries stored in the Rails-cache. Default is `28800`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/95042) in 15.3. | +| `GITLAB_CI_CONFIG_FETCH_TIMEOUT_SECONDS` | integer | Timeout for resolving remote includes in CI config in seconds. Must be between `0` and `60`. Default is `30`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/116383) in 15.11. | ## Adding more variables diff --git a/doc/administration/geo/disaster_recovery/background_verification.md b/doc/administration/geo/disaster_recovery/background_verification.md index 99ac95cd536..ea4523c66e6 100644 --- a/doc/administration/geo/disaster_recovery/background_verification.md +++ b/doc/administration/geo/disaster_recovery/background_verification.md @@ -6,12 +6,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Automatic background verification **(PREMIUM SELF)** -NOTE: -Automatic background verification of repositories and wikis was added in -GitLab EE 10.6 but is enabled by default only on GitLab EE 11.1. You can -disable or enable this feature manually by following -[these instructions](#disabling-or-enabling-the-automatic-background-verification). - Automatic background verification ensures that the transferred data matches a calculated checksum. If the checksum of the data on the **primary** site matches checksum of the data on the **secondary** site, the data transferred successfully. Following a planned failover, @@ -179,19 +173,5 @@ If the **primary** and **secondary** sites have a checksum verification mismatch ## Current limitations -Automatic background verification doesn't cover attachments, LFS objects, -job artifacts, and user uploads in file storage. You can keep track of the -progress to include them in [Geo: Verify all replicated data](https://gitlab.com/groups/gitlab-org/-/epics/1430). - -For now, you can verify their integrity -manually by following [these instructions](../../raketasks/check.md) on both -sites, and comparing the output between them. - -In GitLab EE 12.1, Geo calculates checksums for attachments, LFS objects, and -archived traces on secondary sites after the transfer, compares it with the -stored checksums, and rejects transfers if mismatched. Geo -currently does not support an automatic way to verify these data if they have -been synced before GitLab EE 12.1. - -Data in object storage is **not verified**, as the object store is responsible -for ensuring the integrity of the data. +For more information on what replication and verification methods are supported, +see the [supported Geo data types](../replication/datatypes.md). diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md index cef0101dd40..feb5e030b80 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md @@ -6,7 +6,7 @@ type: howto --- WARNING: -This runbook is in [**Alpha**](../../../../policy/alpha-beta-support.md#alpha-features). For complete, production-ready documentation, see the +This runbook is an [Experiment](../../../../policy/alpha-beta-support.md#experiment). For complete, production-ready documentation, see the [disaster recovery documentation](../index.md). # Disaster Recovery (Geo) promotion runbooks **(PREMIUM SELF)** diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md index 085765ec51e..19150027cca 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md @@ -6,7 +6,7 @@ type: howto --- WARNING: -This runbook is in [**Alpha**](../../../../policy/alpha-beta-support.md#alpha-features). For complete, production-ready documentation, see the +This runbook is an [Experiment](../../../../policy/alpha-beta-support.md#experiment). For complete, production-ready documentation, see the [disaster recovery documentation](../index.md). # Disaster Recovery (Geo) promotion runbooks **(PREMIUM SELF)** diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index feae2d49e8d..9fc2c05a492 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -304,7 +304,7 @@ If your **primary** site is using a [custom or self-signed certificate for inbou sudo gitlab-ctl reconfigure ``` -### Step 5. Enable Git access over HTTP/HTTPS +### Step 5. Enable Git access over HTTP/HTTPS and SSH Geo synchronizes repositories over HTTP/HTTPS, and therefore requires this clone method to be enabled. This is enabled by default, but if converting an existing site to Geo it should be checked: @@ -314,7 +314,10 @@ On the **primary** site: 1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Settings > General**. 1. Expand **Visibility and access controls**. -1. Ensure "Enabled Git access protocols" is set to either "Both SSH and HTTP(S)" or "Only HTTP(S)". +1. If using Git over SSH, then: + 1. Ensure "Enabled Git access protocols" is set to "Both SSH and HTTP(S)". + 1. Follow [Fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md) on both primary and secondary sites. +1. If not using Git over SSH, then set "Enabled Git access protocols" to "Only HTTP(S)". ### Step 6. Verify proper functioning of the **secondary** site diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index 95faafd073c..d6ce5ef5067 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -47,8 +47,8 @@ verification methods: | Blobs | Container registry _(object storage)_ | Geo with API/Managed/Docker API (*2*) | _SHA256 checksum_ | | Blobs | Package registry _(file system)_ | Geo with API | SHA256 checksum | | Blobs | Package registry _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | -| Blobs | Infrastructure registry _(file system)_ | Geo with API | SHA256 checksum | -| Blobs | Infrastructure registry _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | Terraform Module Registry _(file system)_ | Geo with API | SHA256 checksum | +| Blobs | Terraform Module Registry _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | Versioned Terraform State _(file system)_ | Geo with API | SHA256 checksum | | Blobs | Versioned Terraform State _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | External Merge Request Diffs _(file system)_ | Geo with API | SHA256 checksum | @@ -192,17 +192,17 @@ successfully, you must replicate their data using some other means. |:--------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------|:---------------------------------------------------------------------------|:--------------------------------------------------------------------|:----------------------------------------------------------------|:------| |[Application data in PostgreSQL](../../postgresql/index.md) | **Yes** (10.2) | **Yes** (10.2) | N/A | N/A | | |[Project repository](../../../user/project/repository/index.md) | **Yes** (10.2) | **Yes** (10.7) | N/A | N/A | | -|[Project wiki repository](../../../user/project/wiki/index.md) | **Yes** (10.2) | **Yes** (10.7) | N/A | N/A | | +|[Project wiki repository](../../../user/project/wiki/index.md) | **Yes** (10.2)<sup>2</sup> | **Yes** (10.7)<sup>2</sup> | N/A | N/A | Migrated to [self-service framework](../../../development/geo/framework.md) in 15.11. See GitLab issue [#367925](https://gitlab.com/gitlab-org/gitlab/-/issues/367925) for more details.<br /><br />Behind feature flag geo_project_wiki_repository_replication, enabled by default in (15.11). | |[Group wiki repository](../../../user/project/wiki/group.md) | [**Yes** (13.10)](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | No | N/A | N/A | Behind feature flag `geo_group_wiki_repository_replication`, enabled by default. | |[Uploads](../../uploads.md) | **Yes** (10.2) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Replication is behind the feature flag `geo_upload_replication`, enabled by default. Verification was behind the feature flag `geo_upload_verification`, removed in 14.8. | |[LFS objects](../../lfs/index.md) | **Yes** (10.2) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696).<br /><br />Replication is behind the feature flag `geo_lfs_object_replication`, enabled by default. Verification was behind the feature flag `geo_lfs_object_verification`, removed in 14.7. | |[Personal snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | N/A | N/A | | |[Project snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | N/A | N/A | | -|[CI job artifacts](../../../ci/pipelines/job_artifacts.md) | **Yes** (10.4) | **Yes** (14.10) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Verification is behind the feature flag `geo_job_artifact_replication`, enabled by default in 14.10. | +|[CI job artifacts](../../../ci/jobs/job_artifacts.md) | **Yes** (10.4) | **Yes** (14.10) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Verification is behind the feature flag `geo_job_artifact_replication`, enabled by default in 14.10. | |[CI Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/pipeline_artifact.rb) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Persists additional artifacts after a pipeline completes. | |[CI Secure Files](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/secure_file.rb) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [No](object_storage.md#verification-of-files-in-object-storage) | Verification is behind the feature flag `geo_ci_secure_file_replication`, enabled by default in 15.3. | -|[Container Registry](../../packages/container_registry.md) | **Yes** (12.3)* | **Yes** (15.10) | **Yes** (12.3)* | **Yes** (15.10) | See [instructions](container_registry.md) to set up the Container Registry replication. | -|[Infrastructure Registry](../../../user/packages/infrastructure_registry/index.md) | **Yes** (14.0) | **Yes** (14.0) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Container Registry](../../packages/container_registry.md) | **Yes** (12.3)<sup>1</sup> | **Yes** (15.10) | **Yes** (12.3)<sup>1</sup> | **Yes** (15.10) | See [instructions](container_registry.md) to set up the Container Registry replication. | +|[Terraform Module Registry](../../../user/packages/terraform_module_registry/index.md) | **Yes** (14.0) | **Yes** (14.0) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Behind feature flag `geo_package_file_replication`, enabled by default. | |[Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | N/A | N/A | Designs also require replication of LFS objects and Uploads. | |[Package Registry](../../../user/packages/package_registry/index.md) | **Yes** (13.2) | **Yes** (13.10) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Behind feature flag `geo_package_file_replication`, enabled by default. | |[Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | **Yes** (13.12) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Replication is behind the feature flag `geo_terraform_state_version_replication`, enabled by default. Verification was behind the feature flag `geo_terraform_state_version_verification`, which was removed in 14.0. | @@ -217,4 +217,6 @@ successfully, you must replicate their data using some other means. |[Dependency Proxy Images](../../../user/packages/dependency_proxy/index.md) | [**Yes** (15.7)](https://gitlab.com/groups/gitlab-org/-/epics/8833) | [**Yes** (15.7)](https://gitlab.com/groups/gitlab-org/-/epics/8833) | [**Yes** (15.7)](https://gitlab.com/groups/gitlab-org/-/epics/8833) | [No](object_storage.md#verification-of-files-in-object-storage) | | |[Vulnerability Export](../../../user/application_security/vulnerability_report/index.md#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | No | No | Not planned because they are ephemeral and sensitive information. They can be regenerated on demand. | -\* Migrated to [self-service framework](../../../development/geo/framework.md) in 15.5. See GitLab issue [#337436](https://gitlab.com/gitlab-org/gitlab/-/issues/337436) for more details. +<sup>1</sup> Migrated to [self-service framework](../../../development/geo/framework.md) in 15.5. See GitLab issue [#337436](https://gitlab.com/gitlab-org/gitlab/-/issues/337436) for more details. + +<sup>2</sup> Migrated to [self-service framework](../../../development/geo/framework.md) in 15.11. Behind feature flag `geo_project_wiki_repository_replication`, enabled by default. See GitLab issue [#367925](https://gitlab.com/gitlab-org/gitlab/-/issues/367925) for more details. diff --git a/doc/administration/geo/replication/img/adding_a_secondary_v13_3.png b/doc/administration/geo/replication/img/adding_a_secondary_v13_3.png Binary files differdeleted file mode 100644 index e43ace99666..00000000000 --- a/doc/administration/geo/replication/img/adding_a_secondary_v13_3.png +++ /dev/null diff --git a/doc/administration/geo/replication/single_sign_on.md b/doc/administration/geo/replication/single_sign_on.md index fc2f23552db..55e77d5657c 100644 --- a/doc/administration/geo/replication/single_sign_on.md +++ b/doc/administration/geo/replication/single_sign_on.md @@ -112,6 +112,15 @@ After configuring your SAML IdP to allow the secondary site's SAML callback URL, If you have configured SAML on the primary site correctly, then it should work on the secondary site without additional configuration. +## OpenID Connect + +If you use an [OpenID Connect (OIDC)](../../auth/oidc.md) OmniAuth provider, +in most cases, it should work without an issue: + +- **OIDC with Unified URL**: If you have configured OIDC on the primary site correctly, then it should work on the secondary site without additional configuration. +- **OIDC with separate URL with proxying disabled**: If you have configured OIDC on the primary site correctly, then it should work on the secondary site without additional configuration. +- **OIDC with separate URL with proxying enabled**: Geo with separate URL with proxying enabled does not support [OpenID Connect](../../auth/oidc.md). For more information, see [issue 396745](https://gitlab.com/gitlab-org/gitlab/-/issues/396745). + ## LDAP If you use LDAP on your **primary** site, you should also set up secondary LDAP servers on each **secondary** site. Otherwise, users cannot perform Git operations over HTTP(s) on the **secondary** site using HTTP basic authentication. However, users can still use Git with SSH and personal access tokens. diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index c18bb5b6351..8031e2f4896 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -959,7 +959,7 @@ This behavior affects only the following data types through GitLab 14.6: | Package Registry | 13.10 | | CI Pipeline Artifacts | 13.11 | | Terraform State Versions | 13.12 | -| Infrastructure Registry | 14.0 | +| Infrastructure Registry (renamed to Terraform Module Registry in GitLab 15.11) | 14.0 | | External MR diffs | 14.6 | | LFS Objects | 14.6 | | Pages Deployments | 14.6 | diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index 5f04326d49f..ff71ba4f94f 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -865,6 +865,7 @@ For each node running a Patroni instance on the secondary site: consul['configuration'] = { retry_join: %w[CONSUL_SECONDARY1_IP CONSUL_SECONDARY2_IP CONSUL_SECONDARY3_IP] } + consul['services'] = %w(postgresql) postgresql['md5_auth_cidr_addresses'] = [ 'PATRONI_SECONDARY1_IP/32', 'PATRONI_SECONDARY2_IP/32', 'PATRONI_SECONDARY3_IP/32', 'PATRONI_SECONDARY_PGBOUNCER/32', diff --git a/doc/administration/geo/setup/index.md b/doc/administration/geo/setup/index.md index 022d9c00772..64eb4fa5a6f 100644 --- a/doc/administration/geo/setup/index.md +++ b/doc/administration/geo/setup/index.md @@ -24,7 +24,6 @@ type: howto If you installed GitLab using the Omnibus packages (highly recommended): 1. [Set up the database replication](database.md) (`primary (read-write) <-> secondary (read-only)` topology). -1. [Configure fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md). This step is required and needs to be done on **both** the **primary** and **secondary** sites. 1. [Configure GitLab](../replication/configuration.md) to set the **primary** and **secondary** sites. 1. Optional: [Configure Object storage](../../object_storage.md) 1. Optional: [Configure a secondary LDAP server](../../auth/ldap/index.md) for the **secondary** sites. See [notes on LDAP](../index.md#ldap). diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md index 543f5b3c119..f3eb2de3f1d 100644 --- a/doc/administration/gitaly/configure_gitaly.md +++ b/doc/administration/gitaly/configure_gitaly.md @@ -6,25 +6,26 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Configure Gitaly **(FREE SELF)** -The Gitaly service itself is configured by using a [TOML configuration file](reference.md). +Configure Gitaly in one of two ways: -To change Gitaly settings: +::Tabs -**For Omnibus GitLab** +:::TabTitle Linux package (Omnibus) 1. Edit `/etc/gitlab/gitlab.rb` and add or change the [Gitaly settings](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/1dd07197c7e5ae23626aad5a4a070a800b670380/files/gitlab-config-template/gitlab.rb.template#L1622-1676). 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -**For installations from source** +:::TabTitle Self-compiled (source) 1. Edit `/home/git/gitaly/config.toml` and add or change the [Gitaly settings](https://gitlab.com/gitlab-org/gitaly/blob/master/config.toml.example). 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). +::EndTabs + The following configuration options are also available: - Enabling [TLS support](#enable-tls-support). -- Configuring the [number of `gitaly-ruby` workers](#configure-number-of-gitaly-ruby-workers). - Limiting [RPC concurrency](#limit-rpc-concurrency). ## About the Gitaly token @@ -138,7 +139,11 @@ Gitaly and GitLab use two shared secrets for authentication: - _Gitaly token_: used to authenticate gRPC requests to Gitaly - _GitLab Shell token_: used for authentication callbacks from GitLab Shell to the GitLab internal API -**For Omnibus GitLab** +Configure authentication in one of two ways: + +::Tabs + +:::TabTitle Linux package (Omnibus) To configure the _Gitaly token_, edit `/etc/gitlab/gitlab.rb`: @@ -167,7 +172,7 @@ Edit `/etc/gitlab/gitlab.rb`: gitlab_shell['secret_token'] = 'shellsecret' ``` -**For installations from source** +:::TabTitle Self-compiled (source) 1. Copy `/home/git/gitlab/.gitlab_shell_secret` from the Gitaly client to the same path on the Gitaly servers (and any other Gitaly clients). @@ -189,19 +194,26 @@ Edit `/etc/gitlab/gitlab.rb`: 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). -#### Configure Gitaly server - -**For Omnibus GitLab** +::EndTabs -1. Edit `/etc/gitlab/gitlab.rb`: +#### Configure Gitaly server <!-- Updates to example must be made at: + - https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab - https://gitlab.com/gitlab-org/gitlab/blob/master/doc/administration/gitaly/index.md#gitaly-server-configuration -- all reference architecture pages +- All reference architecture pages --> +Configure Gitaly server in one of two ways: + +::Tabs + +:::TabTitle Linux package (Omnibus) + +1. Edit `/etc/gitlab/gitlab.rb`: + ```ruby # Avoid running unnecessary services on the Gitaly server postgresql['enable'] = false @@ -294,7 +306,7 @@ Updates to example must be made at: - For GitLab 15.3 and later, run `sudo /opt/gitlab/embedded/bin/gitaly check /var/opt/gitlab/gitaly/config.toml`. - For GitLab 15.2 and earlier, run `sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml`. -**For installations from source** +:::TabTitle Self-compiled (source) 1. Edit `/home/git/gitaly/config.toml`: @@ -345,6 +357,8 @@ Updates to example must be made at: - For GitLab 15.3 and later, run `sudo /opt/gitlab/embedded/bin/gitaly check /var/opt/gitlab/gitaly/config.toml`. - For GitLab 15.2 and earlier, run `sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml`. +::EndTabs + WARNING: If directly copying repository data from a GitLab server to Gitaly, ensure that the metadata file, default path `/var/opt/gitlab/git-data/repositories/.gitaly-metadata`, is not included in the transfer. @@ -381,7 +395,11 @@ You can't define Gitaly servers with some as a local Gitaly server server (with `gitaly_address`) unless you use [mixed configuration](#mixed-configuration). -**For Omnibus GitLab** +Configure Gitaly clients in one of two ways: + +::Tabs + +:::TabTitle Linux package (Omnibus) 1. Edit `/etc/gitlab/gitlab.rb`: @@ -415,7 +433,7 @@ server (with `gitaly_address`) unless you use sudo gitlab-ctl tail gitaly ``` -**For installations from source** +:::TabTitle Self-compiled (source) 1. Edit `/home/git/gitlab/config/gitlab.yml`: @@ -451,6 +469,8 @@ server (with `gitaly_address`) unless you use tail -f /home/git/gitlab/log/gitaly.log ``` +::EndTabs + When you tail the Gitaly logs on your Gitaly server, you should see requests coming in. One sure way to trigger a Gitaly request is to clone a repository from GitLab over HTTP or HTTPS. @@ -532,9 +552,11 @@ Disabling Gitaly on the GitLab instance makes sense only when you run GitLab in Gitaly runs on a separate machine from the GitLab instance. Disabling Gitaly on all machines in the cluster is not a valid configuration (some machines much act as Gitaly servers). -To disable Gitaly on a GitLab server: +Disable Gitaly on a GitLab server in one of two ways: + +::Tabs -**For Omnibus GitLab** +:::TabTitle Linux package (Omnibus) 1. Edit `/etc/gitlab/gitlab.rb`: @@ -544,7 +566,7 @@ To disable Gitaly on a GitLab server: 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -**For installations from source** +:::TabTitle Self-compiled (source) 1. Edit `/etc/default/gitlab`: @@ -554,6 +576,8 @@ To disable Gitaly on a GitLab server: 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). +::EndTabs + ## Enable TLS support Gitaly supports TLS encryption. To communicate with a Gitaly instance that listens for secure @@ -585,9 +609,11 @@ If you use a load balancer, it must be able to negotiate HTTP/2 using the ALPN T ### Configure Gitaly with TLS -To configure Gitaly with TLS: +Configure Gitaly with TLS in one of two ways: + +::Tabs -**For Omnibus GitLab** +:::TabTitle Linux package (Omnibus) 1. Create certificates for Gitaly servers. 1. On the Gitaly clients, copy the certificates (or their certificate authority) into @@ -651,7 +677,7 @@ To configure Gitaly with TLS: 1. Saving the file. 1. [Reconfiguring GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -**For installations from source** +:::TabTitle Self-compiled (source) 1. Create certificates for Gitaly servers. 1. On the Gitaly clients, copy the certificates into the system trusted certificates: @@ -727,73 +753,13 @@ To configure Gitaly with TLS: 1. Saving the file. 1. [Restarting GitLab](../restart_gitlab.md#installations-from-source). +::EndTabs + ### Observe type of Gitaly connections For information on observing the type of Gitaly connections being served, see the [relevant documentation](monitoring.md#queries). -## `gitaly-ruby` - -Gitaly was developed to replace the Ruby application code in GitLab. - -To save time and avoid the risk of rewriting existing application logic, we chose to copy some -application code from GitLab into Gitaly. - -To be able to run that code, `gitaly-ruby` was created, which is a "sidecar" process for the main -Gitaly Go process. Some examples of things that are implemented in `gitaly-ruby` are: - -- RPCs that deal with wikis. -- RPCs that create commits on behalf of a user, such as merge commits. - -Recommended settings: - -- At least 300 MB memory per worker. -- No more than one worker per core. - -NOTE: -[Epic 2862](https://gitlab.com/groups/gitlab-org/-/epics/2862) proposes to remove `gitaly-ruby`. - -### Configure number of `gitaly-ruby` workers - -`gitaly-ruby` has much less capacity than Gitaly implemented in Go. If your Gitaly server has to handle lots of -requests, the default setting of having just one active `gitaly-ruby` sidecar might not be enough. - -If you see `ResourceExhausted` errors from Gitaly, it's very likely that you have not enough -`gitaly-ruby` capacity. - -You can increase the number of `gitaly-ruby` processes on your Gitaly server with the following -settings: - -**For Omnibus GitLab** - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitaly['configuration'] = { - # ... - 'gitaly-ruby': { - # ... - # - # Default is 2 workers. The minimum is 2; 1 worker is always reserved as - # a passive stand-by. - num_workers: 4 - }, - } - ``` - -1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - -**For installations from source** - -1. Edit `/home/git/gitaly/config.toml`: - - ```toml - [gitaly-ruby] - num_workers = 4 - ``` - -1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). - ## Limit RPC concurrency WARNING: @@ -1025,7 +991,11 @@ WARNING: Background repository optimization is an experimental feature and may place significant load on the host while running. Make sure to schedule this during off-peak hours and keep the duration short (for example, 30-60 minutes). -**For Omnibus GitLab** +Configure background repository optimization in one of two ways: + +::Tabs + +:::TabTitle Linux package (Omnibus) Edit `/etc/gitlab/gitlab.rb` and add: @@ -1042,7 +1012,7 @@ gitaly['configuration'] = { } ``` -**For installations from source** +:::TabTitle Self-compiled (source) Edit `/home/git/gitaly/config.toml` and add: @@ -1054,6 +1024,8 @@ duration = '30m' storages = ["default"] ``` +::EndTabs + ## Rotate Gitaly authentication token Rotating credentials in a production environment often requires downtime, causes outages, or both. @@ -1210,7 +1182,8 @@ automatically keep parts of the pack-objects cache files in RAM, making it faster. Because the pack-objects cache can lead to a significant increase in -disk write IO, it is off by default. +disk write IO, it is off by default. In GitLab 15.11 and later, +the write workload is approximately 50% lower, but the cache is still disabled by default. ### Configure the cache @@ -1222,6 +1195,7 @@ below. | `enabled` | `false` | Turns on the cache. When off, Gitaly runs a dedicated `git pack-objects` process for each request. | | `dir` | `<PATH TO FIRST STORAGE>/+gitaly/PackObjectsCache` | Local directory where cache files get stored. | | `max_age` | `5m` (5 minutes) | Cache entries older than this get evicted and removed from disk. | +| `min_occurrences` | 1 | Minimum times a key must occur before a cache entry is created. | In `/etc/gitlab/gitlab.rb`, set: @@ -1233,6 +1207,7 @@ gitaly['configuration'] = { enabled: true, # dir: '/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache', # max_age: '5m', + # min_occurrences: 1, }, } ``` @@ -1290,13 +1265,31 @@ a guarantee. Peak size may exceed this average. The `max_age` configuration setting lets you control the chance of a cache hit and the average amount of storage used by cache files. -Entries older than `max_age` get evicted from the in-memory metadata -store, and deleted from disk. +Entries older than `max_age` get deleted from the disk. Eviction does not interfere with ongoing requests. It is OK for `max_age` to be less than the time it takes to do a fetch over a slow connection because Unix filesystems do not truly delete a file until all processes that are reading the deleted file have closed it. +#### Minimum key occurrences `min_occurrences` + +> [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/2222) in GitLab 15.11. + +The `min_occurrences` setting controls how often an identical request +must occur before we create a new cache entry. The default value is `1`, +meaning that unique requests do not get written into the cache. + +If you: + +- Increase this number, your cache hit rate goes down and the +cache uses less disk space. +- Decrease this number, your cache hit +rate goes up and the cache uses more disk space. + +You should set `min_occurrences` to `1`. On GitLab.com, +going from 0 to 1 saved us 50% cache disk space while barely affecting +the cache hit rate. + ### Observe the cache The cache can be observed [using metrics](monitoring.md#pack-objects-cache) and in the following logged @@ -1515,10 +1508,14 @@ By default, Gitaly doesn't sign commits made using GitLab UI. For example, commi - Web IDE. - Merge requests. -You can configure Gitaly to sign commits made using GitLab UI. The commits show as unverified and signed by an unknown user. Support for improvements is -proposed in issue [19185](https://gitlab.com/gitlab-org/gitlab/-/issues/19185). +You can configure Gitaly to sign commits made with the GitLab UI. The commits show as unverified and signed by an unknown +user. Support for improvements is proposed in [issue 19185](https://gitlab.com/gitlab-org/gitlab/-/issues/19185). + +Configure Gitaly to sign commits made with the GitLab UI in one of two ways: -**For Omnibus GitLab** +::Tabs + +:::TabTitle Linux package (Omnibus) 1. [Create a GPG key](../../user/project/repository/gpg_signed_commits/index.md#create-a-gpg-key) and export it. For optimal performance, consider using an EdDSA key. @@ -1542,7 +1539,7 @@ proposed in issue [19185](https://gitlab.com/gitlab-org/gitlab/-/issues/19185). 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -**For installations from source** +:::TabTitle Self-compiled (source) 1. [Create a GPG key](../../user/project/repository/gpg_signed_commits/index.md#create-a-gpg-key) and export it. For optimal performance, consider using an EdDSA key. @@ -1560,3 +1557,60 @@ proposed in issue [19185](https://gitlab.com/gitlab-org/gitlab/-/issues/19185). ``` 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). + +::EndTabs + +## Generate configuration using an external command + +> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4828) in GitLab 15.11. + +You can generate parts of the Gitaly configuration using an external command. You might do this: + +- To configure nodes without having to distribute the full configuration to each of them. +- To configure using auto-discovery of the node's settings. For example, using DNS entries. +- To configure secrets at startup of the node, so that don't need to be visible in plain text. + +To generate configuration using an external command, you must provide a script that dumps the +desired configuration of the Gitaly node in JSON format to its standard output. + +For example, the following command configures the HTTP password used to connect to the +GitLab internal API using an AWS secret: + +```ruby +#!/usr/bin/env ruby +require 'json' +JSON.generate({"gitlab": {"http_settings": {"password": `aws get-secret-value --secret-id ...`}}}) +``` + +You must then make the script path known to Gitaly in one of two ways: + +::Tabs + +:::TabTitle Linux package (Omnibus) + +Edit `/etc/gitlab/gitlab.rb` and configure the `config_command`: + +```ruby +gitaly['configuration'] = { + config_command: '/path/to/config_command', +} +``` + +:::TabTitle Self-compiled (source) + +Edit `/home/git/gitaly/config.toml` and configure `config_command`: + +```toml +config_command = "/path/to/config_command" +``` + +::EndTabs + +After configuration, Gitaly executes the command on startup and parses its +standard output as JSON. The resulting configuration is then merged back into +the other Gitaly configuration. + +Gitaly fails to start up if either: + +- The configuration command fails. +- The output produced by the command cannot be parsed as valid JSON. diff --git a/doc/administration/gitaly/gitaly_geo_capabilities.md b/doc/administration/gitaly/gitaly_geo_capabilities.md new file mode 100644 index 00000000000..e4147eec162 --- /dev/null +++ b/doc/administration/gitaly/gitaly_geo_capabilities.md @@ -0,0 +1,41 @@ +--- +stage: Systems +group: Gitaly +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Gitaly and Geo capabilities + +It is common to want the most available, quickly recoverable, highly performant, +and fully resilient solution for your data. However, there are tradeoffs. + +The following tables are intended to guide you to choose the right combination of capabilities based on your requirements. + +## Gitaly capabilities + +| Capability | Availability | Recoverability | Data Resiliency | Performance | Risks/Trade-offs| +|------------|--------------|----------------|-----------------|-------------|-----------------| +|Gitaly Cluster | Very high - tolerant of node failures | RTO for a single node of 10 s with no manual intervention | Data is stored on multiple nodes | Good - While writes may take slightly longer due to voting, read distribution improves read speeds | **Trade-off** - Slight decrease in write speed for redundant, strongly-consistent storage solution. **Risks** - [Does not support snapshot backups](../gitaly/index.md#snapshot-backup-and-recovery-limitations), GitLab backup task can be slow for large data sets | +|Gitaly Shards | Single storage location is a single point of failure | Would need to restore only shards which failed | Single point of failure | Good - can allocate repositories to shards to spread load | **Trade-off** - Need to manually configure repositories into different shards to balance loads / storage space **Risks** - Single point of failure relies on recovery process when single-node failure occurs | +|Gitaly + NFS | Single storage location is a single point of failure | Single node failure requires restoration from backup | Single point of failure | Average - NFS is not ideally suited to large quantities of small reads / writes which can have a detrimental impact on performance | **Trade-off** - Familiar administration though NFS is not ideally suited to Git demands **Risks** - Many instances of NFS compatibility issues which provide very poor customer experiences | + +## Geo capabilities + +If your availability needs to span multiple zones or multiple locations, read about [Geo](../geo/index.md). + +| Capability | Availability | Recoverability | Data Resiliency | Performance | Risks/Trade-offs| +|------------|--------------|----------------|-----------------|-------------|-----------------| +|Geo| Depends on the architecture of the Geo site. It is possible to deploy secondaries in single and multiple node configurations. | Eventually consistent. Recovery point depends on replication lag, which depends on a number of factors such as network speeds. Geo supports failover from a primary to secondary site using manual commands that are scriptable. | Geo replicates 100% of planned data types and verifies 50%. See [limitations table](../geo/replication/datatypes.md#limitations-on-replicationverification) for more detail. | Improves read/clone times for users of a secondary. | Geo is not intended to replace other backup/restore solutions. Because of replication lag and the possibility of replicating bad data from a primary, customers should also take regular backups of their primary site and test the restore process. | + +## Scenarios for failure modes and available mitigation paths + +The following table outlines failure modes and mitigation paths for the product offerings detailed in the tables above. Note - Gitaly Cluster install assumes an odd number replication factor of 3 or greater + +| Gitaly Mode | Loss of Single Gitaly Node | Application / Data Corruption | Regional Outage (Loss of Instance) | Notes | +| ----------- | -------------------------- | ----------------------------- | ---------------------------------- | ----- | +| Single Gitaly Node | Downtime - Must restore from backup | Downtime - Must restore from Backup | Downtime - Must wait for outage to end | | +| Single Gitaly Node + Geo Secondary | Downtime - Must restore from backup, can perform a manual failover to secondary | Downtime - Must restore from Backup, errors could have propagated to secondary | Manual intervention - failover to Geo secondary | | +| Sharded Gitaly Install | Partial Downtime - Only repositories on impacted node affected, must restore from backup | Partial Downtime - Only repositories on impacted node affected, must restore from backup | Downtime - Must wait for outage to end | | +| Sharded Gitaly Install + Geo Secondary | Partial Downtime - Only repositories on impacted node affected, must restore from backup, could perform manual failover to secondary for impacted repositories | Partial Downtime - Only repositories on impacted node affected, must restore from backup, errors could have propagated to secondary | Manual intervention - failover to Geo secondary | | +| Gitaly Cluster Install* | No Downtime - swaps repository primary to another node after 10 seconds | Not applicable; All writes are voted on by multiple Gitaly Cluster nodes | Downtime - Must wait for outage to end | Snapshot backups for Gitaly Cluster nodes not supported at this time | +| Gitaly Cluster Install* + Geo Secondary | No Downtime - swaps repository primary to another node after 10 seconds | Not applicable; All writes are voted on by multiple Gitaly Cluster nodes | Manual intervention - failover to Geo secondary | Snapshot backups for Gitaly Cluster nodes not supported at this time | diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 34e72f3eb5a..36f7456f22e 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -69,7 +69,7 @@ the current status of these issues, refer to the referenced issues and epics. | Issue | Summary | How to avoid | |:--------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------| -| Gitaly Cluster + Geo - Issues retrying failed syncs | If Gitaly Cluster is used on a Geo secondary site, repositories that have failed to sync could continue to fail when Geo tries to resync them. Recovering from this state requires assistance from support to run manual steps. | No known solution prior to GitLab 15.0. In GitLab 15.0 to 15.2, enable the [`gitaly_praefect_generated_replica_paths` feature flag](#praefect-generated-replica-paths-gitlab-150-and-later). In GitLab 15.3, the feature flag is enabled by default. | +| Gitaly Cluster + Geo - Issues retrying failed syncs | If Gitaly Cluster is used on a Geo secondary site, repositories that have failed to sync could continue to fail when Geo tries to resync them. Recovering from this state requires assistance from support to run manual steps. | No known solution prior to GitLab 15.0. In GitLab 15.0 to 15.2, enable the [`gitaly_praefect_generated_replica_paths` feature flag](#praefect-generated-replica-paths-gitlab-150-and-later) on your Geo primary site. In GitLab 15.3, the feature flag is enabled by default. | | Praefect unable to insert data into the database due to migrations not being applied after an upgrade | If the database is not kept up to date with completed migrations, then the Praefect node is unable to perform standard operation. | Make sure the Praefect database is up and running with all migrations completed (For example: `/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migrate-status` should show a list of all applied migrations). Consider [requesting upgrade assistance](https://about.gitlab.com/support/scheduling-upgrade-assistance/) so your upgrade plan can be reviewed by support. | | Restoring a Gitaly Cluster node from a snapshot in a running cluster | Because the Gitaly Cluster runs with consistent state, introducing a single node that is behind results in the cluster not being able to reconcile the nodes data and other nodes data | Don't restore a single Gitaly Cluster node from a backup snapshot. If you must restore from backup:<br/><br/>1. [Shut down GitLab](../read_only_gitlab.md#shut-down-the-gitlab-ui).<br/>2. Snapshot all Gitaly Cluster nodes at the same time.<br/>3. Take a database dump of the Praefect database. | | Rebuilding or replacing an existing Gitaly Cluster node | There is no way to replace existing nodes in place because the Praefect database is relied on to determine the current state of each Gitaly node. Changing how Gitaly Cluster stores repositories is proposed in issue [4218](https://gitlab.com/gitlab-org/gitaly/-/issues/4218). Turning on [repository verification](praefect.md#repository-verification) is proposed in issue [4429](https://gitlab.com/gitlab-org/gitaly/-/issues/4429) so Praefect can detect that data is missing and needs to replicated to a new Gitaly node. | No known solution at this time. | diff --git a/doc/administration/gitaly/monitoring.md b/doc/administration/gitaly/monitoring.md index d4f3d931b03..a854a0a3b48 100644 --- a/doc/administration/gitaly/monitoring.md +++ b/doc/administration/gitaly/monitoring.md @@ -156,6 +156,7 @@ The following metrics are available from the `/metrics` endpoint: - `gitaly_praefect_replication_delay_bucket`, a histogram measuring how much time passes between when the replication job is created and when it starts. Available in GitLab 12.10 and later. - `gitaly_praefect_connections_total`, the total number of connections to Praefect. [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4220) in GitLab 14.7. +- `gitaly_praefect_method_types`, a count of accessor and mutator RPCs per node. To monitor [strong consistency](index.md#strong-consistency), you can use the following Prometheus metrics: diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index efaa8c1ee18..62773f66796 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -1033,7 +1033,7 @@ For more information on Gitaly server configuration, see our 1. Disable all other services by editing `/etc/gitlab/gitlab.rb`: ```ruby - # Disable all other services on the Praefect node + # Disable all other services on the Gitaly node postgresql['enable'] = false redis['enable'] = false nginx['enable'] = false @@ -1181,10 +1181,7 @@ scope of the GitLab documentation. NOTE: The load balancer must be configured to accept traffic from the Gitaly nodes in -addition to the GitLab nodes. Some requests handled by -[`gitaly-ruby`](configure_gitaly.md#gitaly-ruby) sidecar processes call into the main Gitaly -process. `gitaly-ruby` uses the Gitaly address set in the GitLab server's -`git_data_dirs` setting to make this connection. +addition to the GitLab nodes. We hope that if you're managing fault-tolerant systems like GitLab, you have a load balancer of choice already. Some examples include [HAProxy](https://www.haproxy.org/) diff --git a/doc/administration/gitaly/recovery.md b/doc/administration/gitaly/recovery.md index 331f9b5a956..7d27a633512 100644 --- a/doc/administration/gitaly/recovery.md +++ b/doc/administration/gitaly/recovery.md @@ -100,7 +100,7 @@ The following parameters are available: some assigned copies that are not available. NOTE: -`dataloss` is still in [Beta](../../policy/alpha-beta-support.md#beta-features) and the output format is subject to change. +`dataloss` is still in [Beta](../../policy/alpha-beta-support.md#beta) and the output format is subject to change. To check for repositories with outdated primaries or for unavailable repositories, run: diff --git a/doc/administration/gitaly/reference.md b/doc/administration/gitaly/reference.md index 8b4fc51bcdf..cc938c95548 100644 --- a/doc/administration/gitaly/reference.md +++ b/doc/administration/gitaly/reference.md @@ -155,36 +155,6 @@ Prometheus query to see the hit rate: sum(rate(gitaly_catfile_cache_total{type="hit"}[5m])) / sum(rate(gitaly_catfile_cache_total{type=~"(hit)|(miss)"}[5m])) ``` -### `gitaly-ruby` - -A Gitaly process uses one or more `gitaly-ruby` helper processes to -execute RPCs implemented in Ruby instead of Go. The `[gitaly-ruby]` -section of the configuration file contains settings for these helper processes. - -These processes are known to occasionally suffer from memory leaks. -Gitaly restarts its `gitaly-ruby` helpers when their memory exceeds the -`max_rss` limit. - -| Name | Type | Required | Description | -| ---- | ---- | -------- | ----------- | -| `dir` | string | yes | Path to where `gitaly-ruby` is installed (needed to boot the process).| -| `max_rss` | integer | no | Resident set size limit that triggers a `gitaly-ruby` restart, in bytes. Default is `200000000` (200 MB). | -| `graceful_restart_timeout` | string | no | Grace period before a `gitaly-ruby` process is forcibly terminated after exceeding `max_rss`. Default is `10m` (10 minutes).| -| `restart_delay` | string | no |Time that `gitaly-ruby` memory must remain high before a restart. Default is `5m` (5 minutes).| -| `num_workers` | integer | no |Number of `gitaly-ruby` worker processes. Try increasing this number in case of `ResourceExhausted` errors. Default is `2`, minimum is `2`.| -| `linguist_languages_path` | string | no | Override for dynamic `languages.json` discovery. Defaults to an empty string (use of dynamic discovery).| - -Example: - -```toml -[gitaly-ruby] -dir = "/home/git/gitaly/ruby" -max_rss = 200000000 -graceful_restart_timeout = "10m" -restart_delay = "5m" -num_workers = 2 -``` - ### GitLab Shell For historical reasons @@ -232,7 +202,6 @@ The following values configure logging in Gitaly under the `[logging]` section. | `level` | string | no | Log level: `debug`, `info`, `warn`, `error`, `fatal`, or `panic`. Default: `info`. | | `sentry_dsn` | string | no | Sentry DSN (Data Source Name) for exception monitoring. | | `sentry_environment` | string | no | [Sentry Environment](https://docs.sentry.io/product/sentry-basics/environments/) for exception monitoring. | -| `ruby_sentry_dsn` | string | no | Sentry DSN (Data Source Name) for `gitaly-ruby` exception monitoring. | While the main Gitaly application logs go to `stdout`, there are some extra log files that go to a configured directory, like the GitLab Shell logs. diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index a3db28b0d7b..2bb87b20058 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -85,7 +85,7 @@ check for an SSL or TLS problem: ``` Check whether `Verify return code` field indicates a -[known Omnibus GitLab configuration problem](https://docs.gitlab.com/omnibus/settings/ssl.html). +[known Omnibus GitLab configuration problem](https://docs.gitlab.com/omnibus/settings/ssl/index.html). If `openssl` succeeds but `gitlab-rake gitlab:gitaly:check` fails, check [certificate requirements](configure_gitaly.md#certificate-requirements) for Gitaly. @@ -123,27 +123,6 @@ sudo cat /proc/$PID/environ | tr '\0' '\n' | grep ^CORRELATION_ID= This method isn't reliable for `git cat-file` processes, because Gitaly internally pools and re-uses those across RPCs. -### Observing `gitaly-ruby` traffic - -[`gitaly-ruby`](configure_gitaly.md#gitaly-ruby) is an internal implementation detail of Gitaly, -so, there's not that much visibility into what goes on inside -`gitaly-ruby` processes. - -If you have Prometheus set up to scrape your Gitaly process, you can see -request rates and error codes for individual RPCs in `gitaly-ruby` by -querying `grpc_client_handled_total`. - -All gRPC calls made by `gitaly-ruby` itself are internal calls from the main Gitaly process to one of its `gitaly-ruby` -sidecars. - -Assuming your `grpc_client_handled_total` counter only observes Gitaly, -the following query shows you RPCs are (most likely) internally -implemented as calls to `gitaly-ruby`: - -```prometheus -sum(rate(grpc_client_handled_total[5m])) by (grpc_method) > 0 -``` - ### Repository changes fail with a `401 Unauthorized` error If you run Gitaly on its own server and notice these conditions: @@ -492,7 +471,7 @@ in sync so the token check succeeds. This check helps identify the root cause of `permission denied` [errors being logged by Praefect](#permission-denied-errors-appearing-in-gitaly-or-praefect-logs-when-accessing-repositories). -For offline environments where access to public [`pool.ntp.org`](https://pool.ntp.org) servers is not possible, the Praefect `check` sub-command fails this +For offline environments where access to public `pool.ntp.org` servers is not possible, the Praefect `check` sub-command fails this check with an error message similar to: ```plaintext diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index ea051e2067d..3b4eaeb161c 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -68,8 +68,7 @@ this method only supports replies, and not the other features of [incoming email ## Accepted headers -> - Accepting `Received` headers [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/81489) in GitLab 14.9 [with a flag](feature_flags.md) named `use_received_header_for_incoming_emails`. Enabled by default. -> - Accepting `Received` headers: [feature flag](feature_flags.md) named `use_received_header_for_incoming_emails` [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/362596) in GitLab 14.1. +> Accepting `Received` headers [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/81489) in GitLab 14.9. Email is processed correctly when a configured email address is present in one of the following headers (sorted in the order they are checked): diff --git a/doc/administration/index.md b/doc/administration/index.md index 68e16b56624..ded6eabdba7 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -7,235 +7,14 @@ description: 'Learn how to install, configure, update, and maintain your GitLab # Administer GitLab **(FREE SELF)** -If you use GitLab.com, only GitLab team members have access to administration tools and settings. -If you use a self-managed GitLab instance, learn how to administer it. +Learn how to administer a self-managed GitLab instance. -Only administrator users can access GitLab administration tools and settings. +- [Get started](../administration/get_started.md) +- [All feature flags](../user/feature_flags.md) +- [Configure your installation](../administration/configure.md) +- [Configure GitLab Dedicated](../administration/dedicated/index.md) +- [Maintain your installation](../administration/operations/index.md) +- [Secure your installation](../security/index.md) +- [Administer users](../user/profile/account/create_accounts.md) -## Available distributions - -GitLab has two product distributions available through [different subscriptions](https://about.gitlab.com/pricing/): - -- The open source [GitLab Community Edition (CE)](https://gitlab.com/gitlab-org/gitlab-foss). -- The open core [GitLab Enterprise Edition (EE)](https://gitlab.com/gitlab-org/gitlab). - -You can [install either GitLab CE or GitLab EE](https://about.gitlab.com/install/ce-or-ee/). -However, the features you have access to depend on your chosen subscription. - -GitLab Community Edition installations have access only to Free features. - -## Installing and maintaining GitLab - -Learn how to install, configure, update, and maintain your GitLab instance. - -### Installing GitLab - -- [Install](../install/index.md): Requirements, directory structures, and installation methods. -- [Reference architectures](reference_architectures/index.md): Add additional resources to support more users. -- [Installing GitLab on Amazon Web Services (AWS)](../install/aws/index.md): Set up GitLab on Amazon AWS. -- [Geo](geo/index.md): Replicate your GitLab instance to other geographic locations as a read-only fully operational version. -- [Disaster Recovery](geo/disaster_recovery/index.md): Quickly fail-over to a different site with minimal effort in a disaster situation. -- [Add License](../user/admin_area/license.md): Add a license at install time to unlock features that are in paid tiers of GitLab. - -### Configuring GitLab - -- [Adjust your instance's time zone](timezone.md): Customize the default time zone of GitLab. -- [System hooks](system_hooks.md): Notifications when users, projects and keys are changed. -- [Security](../security/index.md): Learn what you can do to further secure your GitLab instance. -- [Usage statistics, version check, and Service Ping](../user/admin_area/settings/usage_statistics.md): Enable or disable information about your instance to be sent to GitLab, Inc. -- [Global user settings](user_settings.md): Configure instance-wide user permissions. -- [Polling](polling.md): Configure how often the GitLab UI polls for updates. -- [GitLab Pages configuration](pages/index.md): Enable and configure GitLab Pages. -- [GitLab Pages configuration for GitLab source installations](pages/source.md): - Enable and configure GitLab Pages on [source installations](../install/installation.md#installation-from-source). -- [Uploads administration](uploads.md): Configure GitLab uploads storage. -- [Environment variables](environment_variables.md): Supported environment - variables that can be used to override their default values to configure - GitLab. -- [File hooks](file_hooks.md): With custom file hooks, GitLab administrators can - introduce custom integrations without modifying GitLab source code. -- [Enforcing Terms of Service](../user/admin_area/settings/terms.md) -- [Customer experience improvement and third-party offers](../user/admin_area/settings/third_party_offers.md) -- [Compliance](compliance.md): A collection of features from across the - application that you may configure to help ensure that your GitLab instance - and DevOps workflow meet compliance standards. -- [Diff limits](../user/admin_area/diff_limits.md): Configure the diff rendering - size limits of branch comparison pages. -- [Merge request diffs storage](merge_request_diffs.md): Configure merge - requests diffs external storage. -- [Broadcast Messages](../user/admin_area/broadcast_messages.md): Send messages - to GitLab users through the UI. -- [Elasticsearch](../integration/advanced_search/elasticsearch.md): Enable Elasticsearch to - empower advanced search. Use when you deal with a huge amount of data. -- [External Classification Policy Authorization](../user/admin_area/settings/external_authorization.md) -- [Add a license](../user/admin_area/license.md): Add a license to unlock - features that are in paid tiers of GitLab. -- [Admin Area](../user/admin_area/index.md): for self-managed instance-wide - configuration and maintenance. -- [S/MIME Signing](smime_signing_email.md): how to sign all outgoing notification - emails with S/MIME. -- [Enabling and disabling features flags](feature_flags.md): how to enable and - disable GitLab features deployed behind feature flags. -- [Application settings cache expiry interval](application_settings_cache.md) -- [Database Load Balancing](postgresql/database_load_balancing.md): Distribute database queries among multiple database servers. -- [Omnibus support for log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-shipping-gitlab-enterprise-edition-only). - -#### Customizing GitLab appearance - -- [Header logo](../user/admin_area/appearance.md#top-bar): Change the logo on all pages and email headers. -- [Favicon](../user/admin_area/appearance.md#favicon): Change the default favicon to your own logo. -- [Branded login page](../user/admin_area/appearance.md#sign-in--sign-up-pages): Customize the login page with your own logo, title, and description. -- ["New Project" page](../user/admin_area/appearance.md#new-project-pages): Customize the text to be displayed on the page that opens whenever your users create a new project. -- [Additional custom email text](../user/admin_area/settings/email.md#custom-additional-text): Add additional custom text to emails sent from GitLab. - -### Maintaining GitLab - -- [Rake tasks](../raketasks/index.md): Perform various tasks for maintenance, backups, automatic webhooks setup, and more. - - [Backup and restore](../raketasks/backup_restore.md): Backup and restore your GitLab instance. -- [Operations](operations/index.md): Keeping GitLab up and running (clean up Redis sessions, moving repositories, Sidekiq MemoryKiller, Puma). -- [Restart GitLab](restart_gitlab.md): Learn how to restart GitLab and its components. -- [Invalidate Markdown cache](invalidate_markdown_cache.md): Invalidate any cached Markdown. -- [Instance review](instance_review.md): Request a free review of your GitLab instance. - -#### Updating GitLab - -- [GitLab versions and maintenance policy](../policy/maintenance.md): Understand GitLab versions and releases (Major, Minor, Patch, Security), as well as update recommendations. -- [GitLab in maintenance mode](maintenance_mode/index.md): Put GitLab in maintenance mode. -- [Update GitLab](../update/index.md): Update guides to upgrade your installation to a new version. -- [Upgrading without downtime](../update/index.md#upgrading-without-downtime): Upgrade to a newer major, minor, or patch version of GitLab without taking your GitLab instance offline. - -### Upgrading or downgrading GitLab - -- [Upgrade from GitLab CE to GitLab EE](../update/index.md#upgrading-between-editions): Learn how to upgrade GitLab Community Edition to GitLab Enterprise Editions. -- [Downgrade from GitLab EE to GitLab CE](../downgrade_ee_to_ce/index.md): Learn how to downgrade GitLab Enterprise Editions to Community Edition. - -### GitLab platform integrations - -- [Mattermost](../integration/mattermost/index.md): Integrate with [Mattermost](https://mattermost.com), an open source, private cloud workplace for web messaging. -- [PlantUML](integration/plantuml.md): Create diagrams in AsciiDoc and Markdown documents - created in snippets, wikis, and repositories. -- [Web terminals](integration/terminal.md): Provide terminal access to your applications deployed to Kubernetes from GitLab CI/CD [environments](../ci/environments/index.md#web-terminals-deprecated). - -## User settings and permissions - -- [Creating users](../user/profile/account/create_accounts.md): Create users manually or through authentication integrations. -- [Libravatar](libravatar.md): Use Libravatar instead of Gravatar for user avatars. -- [Sign-up restrictions](../user/admin_area/settings/sign_up_restrictions.md): block email addresses of specific domains, or allow only specific domains. -- [Admin mode](../user/admin_area/settings/sign_in_restrictions.md#admin-mode): require that administrators authenticate separately to use administrative access, like `sudo`. -- [Access restrictions](../user/admin_area/settings/visibility_and_access_controls.md#configure-enabled-git-access-protocols): Define which Git access protocols can be used to talk to GitLab (SSH, HTTP, HTTPS). -- [Authentication and Authorization](auth/index.md): Configure external authentication with LDAP, SAML, CAS, and additional providers. - - [Sync LDAP](auth/ldap/index.md) - - [Kerberos authentication](../integration/kerberos.md) - - See also other [authentication](../topics/authentication/index.md#gitlab-administrators) topics (for example, enforcing 2FA). -- [Email users](../user/admin_area/email_from_gitlab.md): Email GitLab users from GitLab. -- [User Cohorts](../user/admin_area/user_cohorts.md): Display the monthly cohorts of new users and their activities over time. -- [Audit events](audit_events.md): View the changes made on the GitLab server for: - - Groups and projects. - - Instances. -- [Auditor users](auditor_users.md): Users with read-only access to all projects, groups, and other resources on the GitLab instance. -- [Incoming email](incoming_email.md): Configure incoming emails to allow - users to [reply by email](reply_by_email.md), create [issues by email](../user/project/issues/create_issues.md#by-sending-an-email) and - [merge requests by email](../user/project/merge_requests/creating_merge_requests.md#by-sending-an-email), and to enable [Service Desk](../user/project/service_desk.md). - - [Postfix for incoming email](reply_by_email_postfix_setup.md): Set up a - basic Postfix mail server with IMAP authentication on Ubuntu for incoming - emails. -- [Abuse reports](../user/admin_area/review_abuse_reports.md): View and resolve abuse reports from your users. -- [Credentials Inventory](../user/admin_area/credentials_inventory.md): With Credentials inventory, GitLab administrators can keep track of the credentials used by their users in their GitLab self-managed instance. - -## Project settings - -- [Issue closing pattern](issue_closing_pattern.md): Customize how to close an issue from commit messages. -- [Gitaly](gitaly/index.md): Configuring Gitaly, the Git repository storage service for GitLab. -- [Default labels](../user/admin_area/labels.md): Create labels that are automatically added to every new project. -- [Restrict the use of public or internal projects](../user/public_access.md#restrict-use-of-public-or-internal-projects): Restrict the use of visibility levels for users when they create a project or a snippet. -- [Custom project templates](../user/admin_area/custom_project_templates.md): Configure a set of projects to be used as custom templates when creating a new project. - -## Package Registry administration - -- [Container Registry](packages/container_registry.md): Configure GitLab to act as a registry for containers. -- [Package Registry](packages/index.md): Enable GitLab to act as a registry for packages. -- [Dependency Proxy](packages/dependency_proxy.md): Configure the Dependency Proxy, a local proxy for frequently used upstream images/packages. - -### Repository settings - -- [Repository checks](repository_checks.md): Check your repository for data corruption. -- [Repository storage paths](repository_storage_paths.md): Manage the paths used to store repositories. -- [Repository storage types](repository_storage_types.md): Information about the different repository storage types. -- [Repository storage Rake tasks](raketasks/storage.md): A collection of Rake tasks to list and migrate existing projects and attachments associated with it from Legacy storage to Hashed storage. -- [Limit repository size](../user/admin_area/settings/account_and_limit_settings.md): Set a hard limit for your repositories' size. -- [Static objects external storage](static_objects_external_storage.md): Set external storage for static objects in a repository. - -## CI/CD settings - -- [Enable or disable GitLab CI/CD](cicd.md#disable-gitlab-cicd-in-new-projects): Set new projects in your instance to have GitLab CI/CD enabled or disabled by default. -- [GitLab CI/CD administration settings](../user/admin_area/settings/continuous_integration.md): Enable or disable Auto DevOps site-wide and define the artifacts' max size and expiration time. -- [External Pipeline Validation](external_pipeline_validation.md): Enable, disable, and configure external pipeline validation. -- [Job artifacts](job_artifacts.md): Enable, disable, and configure job artifacts (a set of files and directories which are outputted by a job when it completes successfully). -- [Job logs](job_logs.md): Information about the job logs. -- [Register runners](../ci/runners/runners_scope.md): Learn how to register and configure runners. -- [Shared runners quota of CI/CD minutes](../ci/pipelines/cicd_minutes.md): Limit the usage of CI/CD minutes for shared runners. -- [Enable or disable Auto DevOps](../topics/autodevops/index.md#enable-or-disable-auto-devops): Enable or disable Auto DevOps for your instance. - -## Snippet settings - -- [Setting snippet content size limit](snippets/index.md): Set a maximum content size limit for snippets. - -## Wiki settings - -- [Setting wiki page content size limit](wikis/index.md): Set a maximum content size limit for wiki pages. - -## Git configuration options - -- [Git server hooks](server_hooks.md): Git server hooks (on the file system) for when webhooks aren't enough. Previously called server hooks. -- [Git LFS configuration](lfs/index.md): Learn how to configure LFS for GitLab. -- [Housekeeping](housekeeping.md): Keep your Git repositories tidy and fast. -- [Configuring Git Protocol v2](git_protocol.md): Git protocol version 2 support. - -## Monitoring GitLab - -- [Monitoring GitLab](monitoring/index.md): - - [Monitoring uptime](../user/admin_area/monitoring/health_check.md): Check the server status using the health check endpoint. - - [IP allowlist](monitoring/ip_allowlist.md): Monitor endpoints that provide health check information when probed. - - [Monitoring GitHub imports](monitoring/github_imports.md): The GitLab GitHub Importer displays Prometheus metrics to monitor the health and progress of the importer. - -### Performance Monitoring - -- [GitLab Performance Monitoring](monitoring/performance/index.md): - - [Enable Performance Monitoring](monitoring/performance/gitlab_configuration.md): Enable GitLab Performance Monitoring. - - [GitLab performance monitoring with Prometheus](monitoring/prometheus/index.md): Configure GitLab and Prometheus for measuring performance metrics. - - [GitLab performance monitoring with Grafana](monitoring/performance/grafana_configuration.md): Configure GitLab to visualize time series metrics through graphs and dashboards. - - [Performance Bar](monitoring/performance/performance_bar.md): Get performance information for the current page. - -## Troubleshooting - -- [Log system](logs/index.md): Where to look for logs. -- [Sidekiq Troubleshooting](sidekiq/sidekiq_troubleshooting.md): Debug when Sidekiq appears hung and is not processing jobs. -- [Navigating GitLab via Rails console](operations/rails_console.md) -- [GitLab application limits](instance_limits.md) -- [Responding to security incidents](../security/responding_to_security_incidents.md) - -### Support Team Documentation - -The GitLab Support Team has collected a lot of information about troubleshooting GitLab. -The following documents are used by the Support Team or by customers -with direct guidance from a Support Team member. GitLab administrators may find the -information useful for troubleshooting. However, if you are experiencing trouble with your -GitLab instance, you should check your [support options](https://about.gitlab.com/support/) -before referring to these documents. - -WARNING: -The commands in the following documentation might result in data loss or -other damage to a GitLab instance. They should be used only by experienced administrators -who are aware of the risks. - -- [Diagnostics tools](troubleshooting/diagnostics_tools.md) -- [Linux commands](troubleshooting/linux_cheat_sheet.md) -- [Troubleshooting Kubernetes](https://docs.gitlab.com/charts/troubleshooting/kubernetes_cheat_sheet.html) -- [Troubleshooting PostgreSQL](troubleshooting/postgresql.md) -- [Guide to test environments](troubleshooting/test_environments.md) (for Support Engineers) -- [Troubleshooting SSL](troubleshooting/ssl.md) -- Related links: - - [GitLab Developer Documentation](../development/index.md) - - [Repairing and recovering broken Git repositories](https://git.seveas.net/repairing-and-recovering-broken-git-repositories.html) - - [Testing with OpenSSL](https://www.feistyduck.com/library/openssl-cookbook/online/testing-with-openssl/index.html) - - [`strace` zine](https://wizardzines.com/zines/strace/) +Only GitLab team members have access to administration tools and settings on GitLab.com. diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index 2b2eefdb17c..c113d1aeff4 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -126,7 +126,7 @@ Read more about [import/export rate limits](../user/admin_area/settings/import_e Limit the maximum daily member invitations allowed per group hierarchy. -- GitLab.com: Free members may invite 20 members per day. +- GitLab.com: Free members may invite 20 members per day, Premium trial and Ultimate trial members may invite 50 members per day. - Self-managed: Invites are not limited. ### Webhook rate limit @@ -632,6 +632,42 @@ To update this limit to a new value on a self-managed installation, run the foll Plan.default.actual_limits.update!(ci_instance_level_variables: 30) ``` +### Number of group level variables + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/362227) in GitLab 15.7. + +The total number of group level CI/CD variables is limited at the instance level. +This limit is checked each time a new group level variable is created. If a new variable +would cause the total number of variables to exceed the limit, the new variable is not created. + +On self-managed instances this limit is defined for the `default` plan. By default, +this limit is set to `30000`. + +To update this limit to a new value on a self-managed installation, run the following in the +[GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): + +```ruby +Plan.default.actual_limits.update!(group_ci_variables: 40000) +``` + +### Number of project level variables + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/362227) in GitLab 15.7. + +The total number of project level CI/CD variables is limited at the instance level. +This limit is checked each time a new project level variable is created. If a new variable +would cause the total number of variables to exceed the limit, the new variable is not created. + +On self-managed instances this limit is defined for the `default` plan. By default, +this limit is set to `8000`. + +To update this limit to a new value on a self-managed installation, run the following in the +[GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): + +```ruby +Plan.default.actual_limits.update!(project_ci_variables: 10000) +``` + ### Maximum file size per type of artifact > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37226) in GitLab 13.3. diff --git a/doc/administration/integration/terminal.md b/doc/administration/integration/terminal.md index 0ffb83886ed..df9bb6b6e17 100644 --- a/doc/administration/integration/terminal.md +++ b/doc/administration/integration/terminal.md @@ -1,6 +1,6 @@ --- -stage: Configure -group: Configure +stage: Deploy +group: Environments info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 0a4213b7e2d..e697051f100 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Jobs artifacts administration **(FREE SELF)** This is the administration documentation. To learn how to use job artifacts in your GitLab CI/CD pipeline, -see the [job artifacts configuration documentation](../ci/pipelines/job_artifacts.md). +see the [job artifacts configuration documentation](../ci/jobs/job_artifacts.md). An artifact is a list of files and directories attached to a job after it finishes. This feature is enabled by default in all GitLab installations. @@ -174,7 +174,7 @@ In a multi-server setup you must use one of the options to [eliminate local disk usage for job logs](job_logs.md#prevent-local-disk-usage), or job logs could be lost. In GitLab 13.2 and later, you should use the -[consolidated object storage settings](object_storage.md#consolidated-object-storage-configuration). +[consolidated object storage settings](object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form). ### Migrating to object storage @@ -419,6 +419,8 @@ reasons are: to remove these. This script should always find work to do, as it also removes empty directories (see above). - [Artifact housekeeping was changed significantly](#artifacts-housekeeping-disabled-in-gitlab-146-to-152), and you might need to enable a feature flag to used the updated system. +- The [keep latest artifacts from most recent success jobs](../ci/jobs/job_artifacts.md#keep-artifacts-from-most-recent-successful-jobs) + feature is enabled. In these and other cases, identify the projects most responsible for disk space usage, figure out what types of artifacts are using the most @@ -461,7 +463,7 @@ To check if the feature flags are enabled: ``` These changes include switching artifacts from `unlocked` to `locked` if -they [should be retained](../ci/pipelines/job_artifacts.md#keep-artifacts-from-most-recent-successful-jobs). +they [should be retained](../ci/jobs/job_artifacts.md#keep-artifacts-from-most-recent-successful-jobs). Artifacts created before this feature was introduced have a status of `unknown`. After they expire, these artifacts are not processed by the new housekeeping jobs. @@ -673,7 +675,7 @@ If you need to manually remove job artifacts associated with multiple jobs while NOTE: This step also erases artifacts that users have chosen to - ["keep"](../ci/pipelines/job_artifacts.md#download-job-artifacts). + ["keep"](../ci/jobs/job_artifacts.md#download-job-artifacts). ```ruby builds_to_clear = builds_with_artifacts.where("finished_at < ?", 1.week.ago) @@ -779,7 +781,7 @@ In both cases, you might need to add `region` to the job artifact [object storag ### Job artifact upload fails with `500 Internal Server Error (Missing file)` -Bucket names that include folder paths are not supported with [consolidated object storage](object_storage.md#consolidated-object-storage-configuration). +Bucket names that include folder paths are not supported with [consolidated object storage](object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form). For example, `bucket/path`. If a bucket name has a path in it, you might receive an error similar to: ```plaintext @@ -807,3 +809,18 @@ To work around this issue, you can try: - For older kernels, using the `nolease` mount option to disable file leasing. For more information, [see the investigation details](https://gitlab.com/gitlab-org/gitlab/-/issues/389995). + +### Usage quota shows incorrect artifact storage usage + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/238536) in GitLab 14.10. + +Sometimes the [artifacts storage usage](../user/usage_quotas.md) displays an incorrect +value for the total storage space used by artifacts. To recalculate the artifact +usage statistics for all projects in the instance, you can run this background script: + +```shell +bin/rake 'gitlab:refresh_project_statistics_build_artifacts_size[file.csv]' +``` + +The artifact usage value can fluctuate to `0` while the script is running. After +recalculation, usage should display as expected again. diff --git a/doc/administration/lfs/index.md b/doc/administration/lfs/index.md index adfba28520e..ee5cd04f3d6 100644 --- a/doc/administration/lfs/index.md +++ b/doc/administration/lfs/index.md @@ -2,7 +2,6 @@ stage: Create group: Source Code info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" -disqus_identifier: 'https://docs.gitlab.com/ee/workflow/lfs/lfs_administration.html' --- # GitLab Git Large File Storage (LFS) Administration **(FREE SELF)** @@ -157,14 +156,14 @@ You can store LFS objects in remote object storage. This allows you to reduce reads and writes to the local disk, and free up disk space significantly. In GitLab 13.2 and later, you should use the -[consolidated object storage settings](../object_storage.md#consolidated-object-storage-configuration). +[consolidated object storage settings](../object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form). ### Migrating to object storage You can migrate the LFS objects from local storage to object storage. The processing is done in the background and requires **no downtime**. -1. [Configure the object storage](../object_storage.md#consolidated-object-storage-configuration). +1. [Configure the object storage](../object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form). 1. Migrate the LFS objects: ::Tabs diff --git a/doc/administration/logs/index.md b/doc/administration/logs/index.md index 1104412020a..c03971ab4c1 100644 --- a/doc/administration/logs/index.md +++ b/doc/administration/logs/index.md @@ -686,14 +686,6 @@ Unix timestamp format, and `gzip`-compressed (like `@1584057562.s`). This file is at `/var/log/gitlab/gitlab-rails/grpc.log` for Omnibus GitLab packages. Native [gRPC](https://grpc.io/) logging used by Gitaly. -### `gitaly_ruby_json.log` - -> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/2678) in GitLab 13.6. - -This file is at `/var/log/gitlab/gitaly/gitaly_ruby_json.log` and is -produced by [`gitaly-ruby`](../gitaly/reference.md#gitaly-ruby). It contains an -access log of gRPC calls made by Gitaly to `gitaly-ruby`. - ### `gitaly_hooks.log` This file is at `/var/log/gitlab/gitaly/gitaly_hooks.log` and is @@ -733,7 +725,7 @@ Depending on your installation method, this file is located at: - Omnibus GitLab: `/var/log/gitlab/gitlab-rails/importer.log` - Installations from source: `/home/git/gitlab/log/importer.log` -It logs the progress of the import process. +This file logs the progress of [project imports and migrations](../../user/project/import/index.md). ## `exporter.log` @@ -829,6 +821,8 @@ Depending on your installation method, this file is located at: - Omnibus GitLab: `/var/log/gitlab/gitlab-rails/migrations.log` - Installations from source: `/home/git/gitlab/log/migrations.log` +This file logs the progress of [database migrations](../raketasks/maintenance.md#display-status-of-database-migrations). + ## `mail_room_json.log` (default) > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/19186) in GitLab 12.6. diff --git a/doc/administration/logs/log_parsing.md b/doc/administration/logs/log_parsing.md index 84c517e6120..40d6059d246 100644 --- a/doc/administration/logs/log_parsing.md +++ b/doc/administration/logs/log_parsing.md @@ -158,7 +158,7 @@ CT: 297 ROUTE: /api/:version/projects/:id/repository/tags DURS: 731.39, CT: 190 ROUTE: /api/:version/projects/:id/repository/commits DURS: 1079.02, 979.68, 958.21 ``` -### Print top API user agents +#### Print top API user agents ```shell jq --raw-output '[.route, .ua] | @tsv' api_json.log | sort | uniq -c | sort -n @@ -180,9 +180,20 @@ if the output contains many: You can also [use `fast-stats top`](#parsing-gitlab-logs-with-jq) to extract performance statistics. +### Parsing `gitlab-rails/importer.log` + +To troubleshoot [project imports](../../administration/raketasks/project_import_export.md) or +[migrations](../../user/project/import/index.md), run this command: + +```shell +jq 'select(.project_path == "<namespace>/<project>").error_messages' importer.log +``` + +For common issues, see [troubleshooting](../../administration/raketasks/project_import_export.md#troubleshooting). + ### Parsing `gitlab-workhorse/current` -### Print top Workhorse user agents +#### Print top Workhorse user agents ```shell jq --raw-output '[.uri, .user_agent] | @tsv' current | sort | uniq -c | sort -n diff --git a/doc/administration/merge_request_diffs.md b/doc/administration/merge_request_diffs.md index 85677512860..3fbcdafd886 100644 --- a/doc/administration/merge_request_diffs.md +++ b/doc/administration/merge_request_diffs.md @@ -105,7 +105,7 @@ be configured already. ### Object Storage Settings In GitLab 13.2 and later, you should use the -[consolidated object storage settings](object_storage.md#consolidated-object-storage-configuration). +[consolidated object storage settings](object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form). This section describes the earlier configuration format. For source installations, these settings are nested under `external_diffs:` and diff --git a/doc/administration/monitoring/github_imports.md b/doc/administration/monitoring/github_imports.md index 097109585af..3d29626a132 100644 --- a/doc/administration/monitoring/github_imports.md +++ b/doc/administration/monitoring/github_imports.md @@ -1,6 +1,6 @@ --- -stage: Monitor -group: Respond +stage: Manage +group: Import info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/performance/index.md b/doc/administration/monitoring/performance/index.md index 89f1270532a..cbdd68804e8 100644 --- a/doc/administration/monitoring/performance/index.md +++ b/doc/administration/monitoring/performance/index.md @@ -1,6 +1,6 @@ --- -stage: Monitor -group: Respond +stage: Data Stores +group: Application Performance info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index c21c92a3d63..4a523ad75ec 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -1,6 +1,6 @@ --- -stage: Monitor -group: Respond +stage: Shared responsibility based on functional area +group: Shared responsibility based on functional area info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -30,6 +30,8 @@ For enabling and viewing metrics from Sidekiq nodes, see [Sidekiq metrics](#side ## Metrics available +> `caller_id` [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/392622) from `redis_hit_miss_operations_total` and `redis_cache_generation_duration_seconds` in GitLab 15.11. + The following metrics are available: | Metric | Type | Since | Description | Labels | @@ -133,6 +135,8 @@ The following metrics are available: | `gitlab_issuable_fast_count_by_state_total` | Counter | 13.5 | Total number of row count operations on issue/merge request list pages | | | `gitlab_issuable_fast_count_by_state_failures_total` | Counter | 13.5 | Number of soft-failed row count operations on issue/merge request list pages | | | `gitlab_ci_trace_finalize_duration_seconds` | Histogram | 13.6 | Duration of build trace chunks migration to object storage | | +| `gitlab_vulnerability_report_branch_comparison_real_duration_seconds` | Histogram | 15.11 | Execution duration of vulnerability report present on default branch sql query | | +| `gitlab_vulnerability_report_branch_comparison_cpu_duration_seconds` | Histogram | 15.11 | Execution duration of vulnerability report present on default branch sql query | | | `gitlab_external_http_total` | Counter | 13.8 | Total number of HTTP calls to external systems | `controller`, `action` | | `gitlab_external_http_duration_seconds` | Counter | 13.8 | Duration in seconds spent on each HTTP call to external systems | | | `gitlab_external_http_exception_total` | Counter | 13.8 | Total number of exceptions raised when making external HTTP calls | | @@ -151,8 +155,8 @@ The following metrics are available: | `gitlab_ci_build_trace_errors_total` | Counter | 14.4 | Total amount of different error types on a build trace | `error_reason` | | `gitlab_presentable_object_cacheless_render_real_duration_seconds` | Histogram | 15.3 | Duration of real time spent caching and representing specific web request objects | `controller`, `action` | | `cached_object_operations_total` | Counter | 15.3 | Total number of objects cached for specific web requests | `controller`, `action` | -| `redis_hit_miss_operations_total` | Counter | 15.6 | Total number of Redis cache hits and misses | `cache_hit`, `caller_id`, `cache_identifier`, `feature_category`, `backing_resource` | -| `redis_cache_generation_duration_seconds` | Histogram | 15.6 | Time to generate Redis cache | `cache_hit`, `caller_id`, `cache_identifier`, `feature_category`, `backing_resource` | +| `redis_hit_miss_operations_total` | Counter | 15.6 | Total number of Redis cache hits and misses | `cache_hit`, `cache_identifier`, `feature_category`, `backing_resource` | +| `redis_cache_generation_duration_seconds` | Histogram | 15.6 | Time to generate Redis cache | `cache_hit`, `cache_identifier`, `feature_category`, `backing_resource` | | `gitlab_diffs_reorder_real_duration_seconds` | Histogram | 15.8 | Duration in seconds spend on reordering of diff files on diffs batch request | `controller`, `action` | | `gitlab_diffs_collection_real_duration_seconds` | Histogram | 15.8 | Duration in seconds spent on querying merge request diff files on diffs batch request | `controller`, `action` | | `gitlab_diffs_comparison_real_duration_seconds` | Histogram | 15.8 | Duration in seconds spent on getting comparison data on diffs batch request | `controller`, `action` | @@ -171,7 +175,7 @@ The following metrics are available: The following metrics can be controlled by feature flags: -| Metric | Feature Flag | +| Metric | Feature flag | |:---------------------------------------------------------------|:-------------------------------------------------------------------| | `gitlab_view_rendering_duration_seconds` | `prometheus_metrics_view_instrumentation` | @@ -359,9 +363,20 @@ configuration option in `gitlab.yml`. These metrics are served from the | `geo_dependency_proxy_manifests_verification_total` | Gauge | 15.6 | Number of dependency proxy manifests to attempt to verify on secondary | `url` | | `geo_dependency_proxy_manifests_verified` | Gauge | 15.6 | Number of dependency proxy manifests successfully verified on secondary | `url` | | `geo_dependency_proxy_manifests_verification_failed` | Gauge | 15.6 | Number of dependency proxy manifests that failed verification on secondary | `url` | +| `geo_project_wiki_repositories` | Gauge | 15.10 | Number of Project Wiki Repositories on primary | `url` | +| `geo_project_wiki_repositories_checksum_total` | Gauge | 15.10 | Number of Project Wiki Repositories to checksum on primary | `url` | +| `geo_project_wiki_repositories_checksummed` | Gauge | 15.10 | Number of Project Wiki Repositories that successfully calculated the checksum on primary | `url` | +| `geo_project_wiki_repositories_checksum_failed` | Gauge | 15.10 | Number of Project Wiki Repositories that failed to calculate the checksum on primary | `url` | +| `geo_project_wiki_repositories_synced` | Gauge | 15.10 | Number of syncable Project Wiki Repositories synced on secondary | `url` | +| `geo_project_wiki_repositories_failed` | Gauge | 15.10 | Number of syncable Project Wiki Repositories failed to sync on secondary | `url` | +| `geo_project_wiki_repositories_registry` | Gauge | 15.10 | Number of Project Wiki Repositories in the registry | `url` | +| `geo_project_wiki_repositories_verification_total` | Gauge | 15.10 | Number of Project Wiki Repositories to attempt to verify on secondary | `url` | +| `geo_project_wiki_repositories_verified` | Gauge | 15.10 | Number of Project Wiki Repositories successfully verified on secondary | `url` | +| `geo_project_wiki_repositories_verification_failed` | Gauge | 15.10 | Number of Project Wiki Repositories that failed verification on secondary | `url` | | `gitlab_memwd_violations_total` | Counter | 15.9 | Total number of times a Sidekiq process violated a memory threshold | | | `gitlab_memwd_violations_handled_total` | Counter | 15.9 | Total number of times Sidekiq process memory violations were handled | | | `sidekiq_watchdog_running_jobs_total` | Counter | 15.9 | Current running jobs when RSS limit was reached | `worker_class` | +| `gitlab_maintenance_mode` | Gauge | 15.11 | Is GitLab Maintenance Mode enabled? | | ## Database load balancing metrics **(PREMIUM SELF)** diff --git a/doc/administration/monitoring/prometheus/node_exporter.md b/doc/administration/monitoring/prometheus/node_exporter.md index c2f84773b3a..337ab9becaa 100644 --- a/doc/administration/monitoring/prometheus/node_exporter.md +++ b/doc/administration/monitoring/prometheus/node_exporter.md @@ -1,6 +1,6 @@ --- -stage: Monitor -group: Respond +stage: Shared responsibility based on functional area +group: Shared responsibility based on functional area info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md index 56da3155fd9..f8d48832288 100644 --- a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md +++ b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md @@ -1,6 +1,6 @@ --- -stage: Monitor -group: Respond +stage: Data Stores +group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/prometheus/postgres_exporter.md b/doc/administration/monitoring/prometheus/postgres_exporter.md index 0458a4a5bae..1cae28a069f 100644 --- a/doc/administration/monitoring/prometheus/postgres_exporter.md +++ b/doc/administration/monitoring/prometheus/postgres_exporter.md @@ -1,6 +1,6 @@ --- -stage: Monitor -group: Respond +stage: Data Stores +group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/prometheus/redis_exporter.md b/doc/administration/monitoring/prometheus/redis_exporter.md index b50a2a4aebf..b17bf9787a7 100644 --- a/doc/administration/monitoring/prometheus/redis_exporter.md +++ b/doc/administration/monitoring/prometheus/redis_exporter.md @@ -1,6 +1,6 @@ --- -stage: Monitor -group: Respond +stage: Shared responsibility based on functional area +group: Shared responsibility based on functional area info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/prometheus/registry_exporter.md b/doc/administration/monitoring/prometheus/registry_exporter.md index b79e97bd937..3f4d6cfb14d 100644 --- a/doc/administration/monitoring/prometheus/registry_exporter.md +++ b/doc/administration/monitoring/prometheus/registry_exporter.md @@ -1,6 +1,6 @@ --- -stage: Monitor -group: Respond +stage: Package +group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md index 7349e8ad9ba..3e3712c9645 100644 --- a/doc/administration/nfs.md +++ b/doc/administration/nfs.md @@ -12,7 +12,7 @@ recommended for performance reasons. For data objects such as LFS, Uploads, and Artifacts, an [Object Storage service](object_storage.md) is recommended over NFS where possible, due to better performance. -When eliminating the usage of NFS, there are [additional steps you need to take](object_storage.md#other-alternatives-to-file-system-storage) +When eliminating the usage of NFS, there are [additional steps you need to take](object_storage.md#alternatives-to-file-system-storage) in addition to moving to Object Storage. File system performance can impact overall GitLab performance, especially for diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index 587da749766..848ee8de951 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -11,11 +11,24 @@ It's recommended over NFS and in general it's better in larger setups as object storage is typically much more performant, reliable, and scalable. +To configure the object storage, you have two options: + +- Recommended. [Configure a single storage connection for all object types](#configure-a-single-storage-connection-for-all-object-types-consolidated-form): + A single credential is shared by all supported object types. This is called the consolidated form. +- [Configure each object type to define its own storage connection](#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form): + Every object defines its own object storage connection and configuration. This is called the storage-specific form. + + If you already use the storage-specific form, see how to + [transition to the consolidated form](#transition-to-consolidated-form). + +If you store data locally, see how to +[migrate to object storage](#migrate-to-object-storage). + ## Supported object storage providers -GitLab is tightly integrated with `Fog`, so you can refer to its -[documentation](https://fog.io/about/provider_documentation.html) to check -which storage services can be integrated with GitLab. +GitLab is tightly integrated with the Fog library, so you can see which +[providers](https://fog.io/about/provider_documentation.html) can be used +with GitLab. Specifically, GitLab has been tested by vendors and customers on a number of object storage providers: @@ -23,57 +36,45 @@ Specifically, GitLab has been tested by vendors and customers on a number of obj is not supported, see [issue #335775](https://gitlab.com/gitlab-org/gitlab/-/issues/335775) for more information) - [Google Cloud Storage](https://cloud.google.com/storage) -- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces) +- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces) (S3 compatible) - [Oracle Cloud Infrastructure](https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift (S3 compatible mode)](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) +- [MinIO](https://min.io/) (S3 compatible) - On-premises hardware and appliances from various storage vendors, whose list is not officially established. -- MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. - -## Configure the object storage - -There are two ways of specifying object storage configuration in GitLab: -- [Consolidated form](#consolidated-object-storage-configuration): A single credential is - shared by all supported object types. -- [Storage-specific form](#storage-specific-configuration): Every object defines its - own object storage [connection and configuration](#connection-settings). - -For more information on the differences and to transition from one form to another, see -[Transition to consolidated form](#transition-to-consolidated-form). - -If you are currently storing data locally, see -[Migrate to object storage](#migrate-to-object-storage) for migration details. - -### Consolidated object storage configuration +## Configure a single storage connection for all object types (consolidated form) > [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4368) in GitLab 13.2. -Using the consolidated object storage configuration has a number of advantages: +Most types of objects, such as CI artifacts, LFS files, and upload attachments +can be saved in object storage by specifying a single credential for object +storage with multiple buckets. + +Configuring the object storage using the consolidated form has a number of advantages: - It can simplify your GitLab configuration since the connection details are shared across object types. - It enables the use of [encrypted S3 buckets](#encrypted-s3-buckets). - It [uploads files to S3 with proper `Content-MD5` headers](https://gitlab.com/gitlab-org/gitlab-workhorse/-/issues/222). -Because [direct upload mode](../development/uploads/index.md#direct-upload) -must be enabled, only the following providers can be used: +When the consolidated form is used, +[direct upload](../development/uploads/index.md#direct-upload) is enabled +automatically. Thus, only the following providers can be used: - [Amazon S3-compatible providers](#amazon-s3) - [Google Cloud Storage](#google-cloud-storage-gcs) - [Azure Blob storage](#azure-blob-storage) -When consolidated object storage is used, direct upload is enabled -automatically. For storage-specific -configuration, [direct upload may become the default](https://gitlab.com/gitlab-org/gitlab/-/issues/27331) -because it does not require a shared folder. +The consolidated form configuration can't be used for backups or +Mattermost. Backups can be configured with +[server side encryption](../raketasks/backup_gitlab.md#s3-encrypted-buckets) +separately. See the +[table for a complete list](#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form) +of supported object storage types. -Consolidated object storage configuration can't be used for backups or -Mattermost. See the [full table for a complete list](#storage-specific-configuration). -However, backups can be configured with [server side encryption](../raketasks/backup_gitlab.md#s3-encrypted-buckets) separately. - -Enabling consolidated object storage enables object storage for all object -types. If not all buckets are specified, `sudo gitlab-ctl reconfigure` may fail with the error like: +Enabling the consolidated form enables object storage for all object +types. If not all buckets are specified, you may see an error like: ```plaintext Object storage for <object type> must have a bucket specified @@ -82,20 +83,345 @@ Object storage for <object type> must have a bucket specified If you want to use local storage for specific object types, you can [disable object storage for specific features](#disable-object-storage-for-specific-features). -Most types of objects, such as CI artifacts, LFS files, and upload -attachments can be saved in object storage by specifying a single -credential for object storage with multiple buckets. +### Configure the common parameters -When the consolidated form is: +In the consolidated form, the `object_store` section defines a +common set of parameters. -- Used with an S3-compatible object storage, Workhorse uses its internal S3 client to - upload files. -- Not used with an S3-compatible object storage, Workhorse falls back to using - pre-signed URLs. +| Setting | Description | +|-------------------|-----------------------------------| +| `enabled` | Enable or disable object storage. | +| `proxy_download` | Set to `true` to [enable proxying all files served](#proxy-download). Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data. | +| `connection` | Various [connection options](#connection-settings) described below. | +| `storage_options` | Options to use when saving new objects, such as [server side encryption](#server-side-encryption-headers). Introduced in GitLab 13.3. | +| `objects` | [Object-specific configuration](#configure-the-parameters-of-each-object). | + +The following YAML is from the source +installation, to help you see the inheritance: + +```yaml + object_store: + enabled: true + proxy_download: true + connection: + provider: AWS + aws_access_key_id: <AWS_ACCESS_KEY_ID> + aws_secret_access_key: <AWS_SECRET_ACCESS_KEY> + objects: + ... +``` + +The Omnibus configuration maps directly to this: + +```ruby +gitlab_rails['object_store']['enabled'] = true +gitlab_rails['object_store']['proxy_download'] = true +gitlab_rails['object_store']['connection'] = { + 'provider' => 'AWS', + 'aws_access_key_id' => '<AWS_ACCESS_KEY_ID', + 'aws_secret_access_key' => '<AWS_SECRET_ACCESS_KEY>' +} +``` + +### Configure the parameters of each object + +Each object type must at least define the bucket name where it will be stored. + +The following table lists the valid `objects` that can be used: + +| Type | Description | +|--------------------|----------------------------------------------------------------------------| +| `artifacts` | [CI artifacts](job_artifacts.md) | +| `external_diffs` | [Merge request diffs](merge_request_diffs.md) | +| `uploads` | [User uploads](uploads.md) | +| `lfs` | [Git Large File Storage objects](lfs/index.md) | +| `packages` | [Project packages (for example, PyPI, Maven, or NuGet)](packages/index.md) | +| `dependency_proxy` | [Dependency Proxy](packages/dependency_proxy.md) | +| `terraform_state` | [Terraform state files](terraform_state.md) | +| `pages` | [Pages](pages/index.md) | + +Within each object type, three parameters can be defined: + +| Setting | Required? | Description | +|------------------|------------------------|-------------------------------------| +| `bucket` | **{check-circle}** Yes\* | Bucket name for the object type. Not required if `enabled` is set to `false`. | +| `enabled` | **{dotted-circle}** No | Overrides the [common parameter](#configure-the-common-parameters). | +| `proxy_download` | **{dotted-circle}** No | Overrides the [common parameter](#configure-the-common-parameters). | + +The following YAML shows how the `object_store` section defines +object-specific configuration block and how the `enabled` and +`proxy_download` flags can be overridden. The `bucket` is the only +required parameter within each type: + +```yaml + object_store: + connection: + ... + objects: + artifacts: + bucket: artifacts + proxy_download: false + external_diffs: + bucket: external-diffs + lfs: + bucket: lfs-objects + uploads: + bucket: uploads + packages: + bucket: packages + dependency_proxy: + enabled: false + bucket: dependency_proxy + terraform_state: + bucket: terraform + pages: + bucket: pages +``` + +This maps to this Omnibus GitLab configuration: + +```ruby +gitlab_rails['object_store']['objects']['artifacts']['bucket'] = 'artifacts' +gitlab_rails['object_store']['objects']['artifacts']['proxy_download'] = false +gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = 'external-diffs' +gitlab_rails['object_store']['objects']['lfs']['bucket'] = 'lfs-objects' +gitlab_rails['object_store']['objects']['uploads']['bucket'] = 'uploads' +gitlab_rails['object_store']['objects']['packages']['bucket'] = 'packages' +gitlab_rails['object_store']['objects']['dependency_proxy']['enabled'] = false +gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = 'dependency-proxy' +gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = 'terraform-state' +gitlab_rails['object_store']['objects']['pages']['bucket'] = 'pages' +``` + +#### Disable object storage for specific features + +As seen above, object storage can be disabled for specific types by +setting the `enabled` flag to `false`. For example, to disable object +storage for CI artifacts: + +```ruby +gitlab_rails['object_store']['objects']['artifacts']['enabled'] = false +``` + +A bucket is not needed if the feature is disabled entirely. For example, +no bucket is needed if CI artifacts are disabled with this setting: + +```ruby +gitlab_rails['artifacts_enabled'] = false +``` + +## Configure each object type to define its own storage connection (storage-specific form) + +With the storage-specific form, every object defines its own object +storage connection and configuration. If you're using GitLab 13.2 and later, +you should [transition to the consolidated form](#transition-to-consolidated-form). + +The use of [encrypted S3 buckets](#encrypted-s3-buckets) with non-consolidated form is not supported. +You may get [ETag mismatch errors](#etag-mismatch) if you use it. + +NOTE: +For the storage-specific form, +[direct upload may become the default](https://gitlab.com/gitlab-org/gitlab/-/issues/27331) +because it does not require a shared folder. + +For configuring object storage in GitLab 13.1 and earlier, or for storage types not +supported by consolidated form, refer to the following guides: + +| Object storage type | Supported by consolidated form? | +|---------------------|------------------------------------------| +| [Backups](../raketasks/backup_gitlab.md#upload-backups-to-a-remote-cloud-storage) | **{dotted-circle}** No | +| [Container Registry](packages/container_registry.md#use-object-storage) (optional feature) | **{dotted-circle}** No | +| [Mattermost](https://docs.mattermost.com/configure/file-storage-configuration-settings.html)| **{dotted-circle}** No | +| [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | **{dotted-circle}** No | +| [Job artifacts](job_artifacts.md#using-object-storage) including archived job logs | **{check-circle}** Yes | +| [LFS objects](lfs/index.md#storing-lfs-objects-in-remote-object-storage) | **{check-circle}** Yes | +| [Uploads](uploads.md#using-object-storage) | **{check-circle}** Yes | +| [Merge request diffs](merge_request_diffs.md#using-object-storage) | **{check-circle}** Yes | +| [Packages](packages/index.md#use-object-storage) (optional feature) | **{check-circle}** Yes | +| [Dependency Proxy](packages/dependency_proxy.md#using-object-storage) (optional feature) | **{check-circle}** Yes | +| [Terraform state files](terraform_state.md#using-object-storage) | **{check-circle}** Yes | +| [Pages content](pages/index.md#using-object-storage) | **{check-circle}** Yes | + +## Connection settings + +Both consolidated and storage-specific form must configure a connection. The following sections describe parameters that can be used +in the `connection` setting. + +### Amazon S3 + +The connection settings match those provided by [fog-aws](https://github.com/fog/fog-aws): + +| Setting | Description | Default | +|---------------------------------------------|------------------------------------|---------| +| `provider` | Always `AWS` for compatible hosts. | `AWS` | +| `aws_access_key_id` | AWS credentials, or compatible. | | +| `aws_secret_access_key` | AWS credentials, or compatible. | | +| `aws_signature_version` | AWS signature version to use. `2` or `4` are valid options. Digital Ocean Spaces and other providers may need `2`. | `4` | +| `enable_signature_v4_streaming` | Set to `true` to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be `false`. | `true` | +| `region` | AWS region. | | +| `host` | DEPRECATED: Use `endpoint` instead. S3 compatible host for when not using AWS. For example, `localhost` or `storage.example.com`. HTTPS and port 443 is assumed. | `s3.amazonaws.com` | +| `endpoint` | Can be used when configuring an S3 compatible service such as [MinIO](https://min.io), by entering a URL such as `http://127.0.0.1:9000`. This takes precedence over `host`. Always use `endpoint` for consolidated form. | (optional) | +| `path_style` | Set to `true` to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Set to `true` for using [MinIO](https://min.io). Leave as `false` for AWS S3. | `false`. | +| `use_iam_profile` | Set to `true` to use IAM profile instead of access keys. | `false` | +| `aws_credentials_refresh_threshold_seconds` | Sets the [automatic refresh threshold](https://github.com/fog/fog-aws#controlling-credential-refresh-time-with-iam-authentication) when using temporary credentials in IAM. | `15` | -See the section on [ETag mismatch errors](#etag-mismatch) for more details. +### Oracle Cloud S3 -#### Full example using Amazon S3 +Oracle Cloud S3 must be sure to use the following settings: + +| Setting | Value | +|---------------------------------|---------| +| `enable_signature_v4_streaming` | `false` | +| `path_style` | `true` | + +If `enable_signature_v4_streaming` is set to `true`, you may see the +following error in `production.log`: + +```plaintext +STREAMING-AWS4-HMAC-SHA256-PAYLOAD is not supported +``` + +### Google Cloud Storage (GCS) + +Here are the valid connection parameters for GCS: + +| Setting | Description | Example | +|------------------------------|-------------------|---------| +| `provider` | Provider name. | `Google` | +| `google_project` | GCP project name. | `gcp-project-12345` | +| `google_json_key_location` | JSON key path. | `/path/to/gcp-project-12345-abcde.json` | +| `google_json_key_string` | JSON key string. | `{ "type": "service_account", "project_id": "example-project-382839", ... }` | +| `google_application_default` | Set to `true` to use [Google Cloud Application Default Credentials](https://cloud.google.com/docs/authentication#adc) to locate service account credentials. | | + +GitLab reads the value of `google_json_key_location`, then `google_json_key_string`, and finally, `google_application_default`. +It uses the first of these settings that has a value. + +The service account must have permission to access the bucket. For more information, +see the [Cloud Storage authentication documentation](https://cloud.google.com/storage/docs/authentication). + +NOTE: +Bucket encryption with the [Cloud Key Management Service (KMS)](https://cloud.google.com/kms/docs) is not supported and results in [ETag mismatch errors](#etag-mismatch). + +#### GCS example + +For Omnibus installations, this is an example of the `connection` setting +in the consolidated form: + +```ruby +gitlab_rails['object_store']['connection'] = { + 'provider' => 'Google', + 'google_project' => '<GOOGLE PROJECT>', + 'google_json_key_location' => '<FILENAME>' +} +``` + +#### GCS example with ADC + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/275979) in GitLab 13.6. + +Google Cloud Application Default Credentials (ADC) are typically +used with GitLab to use the default service account. This eliminates the +need to supply credentials for the instance. For example, in the consolidated form: + +```ruby +gitlab_rails['object_store']['connection'] = { + 'provider' => 'Google', + 'google_project' => '<GOOGLE PROJECT>', + 'google_application_default' => true +} +``` + +If you use ADC, be sure that: + +- The service account that you use has the +[`iam.serviceAccounts.signBlob` permission](https://cloud.google.com/iam/docs/reference/credentials/rest/v1/projects.serviceAccounts/signBlob). + Typically this is done by granting the `Service Account Token Creator` role to the service account. +- Your virtual machines have the [correct access scopes to access Google Cloud APIs](https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances#changeserviceaccountandscopes). If the machines do not have the right scope, the error logs may show: + + ```markdown + Google::Apis::ClientError (insufficientPermissions: Request had insufficient authentication scopes.) + ``` + +### Azure Blob storage + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25877) in GitLab 13.4. + +Although Azure uses the word `container` to denote a collection of +blobs, GitLab standardizes on the term `bucket`. Be sure to configure +Azure container names in the `bucket` settings. + +Azure Blob storage can only be used with the [consolidated form](#configure-a-single-storage-connection-for-all-object-types-consolidated-form) +because a single set of credentials are used to access multiple +containers. The [storage-specific form](#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form) +is not supported. For more details, see [how to transition to consolidated form](#transition-to-consolidated-form). + +The following are the valid connection parameters for Azure. For more information, see the +[Azure Blob Storage documentation](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction). + +| Setting | Description | Example | +|------------------------------|----------------|-----------| +| `provider` | Provider name. | `AzureRM` | +| `azure_storage_account_name` | Name of the Azure Blob Storage account used to access the storage. | `azuretest` | +| `azure_storage_access_key` | Storage account access key used to access the container. This is typically a secret, 512-bit encryption key encoded in base64. | `czV2OHkvQj9FKEgrTWJRZVRoV21ZcTN0Nnc5eiRDJkYpSkBOY1JmVWpYbjJy\nNHU3eCFBJUQqRy1LYVBkU2dWaw==\n` | +| `azure_storage_domain` | Domain name used to contact the Azure Blob Storage API (optional). Defaults to `blob.core.windows.net`. Set this if you are using Azure China, Azure Germany, Azure US Government, or some other custom Azure domain. | `blob.core.windows.net` | + +- For Omnibus installations, this is an example of the `connection` setting + in the consolidated form: + + ```ruby + gitlab_rails['object_store']['connection'] = { + 'provider' => 'AzureRM', + 'azure_storage_account_name' => '<AZURE STORAGE ACCOUNT NAME>', + 'azure_storage_access_key' => '<AZURE STORAGE ACCESS KEY>', + 'azure_storage_domain' => '<AZURE STORAGE DOMAIN>' + } + ``` + +- For source installations, Workhorse also needs to be configured with Azure + credentials. This isn't needed in Omnibus installs, because the Workhorse + settings are populated from the previous settings. + + 1. Edit `/home/git/gitlab-workhorse/config.toml` and add or amend the following lines: + + ```toml + [object_storage] + provider = "AzureRM" + + [object_storage.azurerm] + azure_storage_account_name = "<AZURE STORAGE ACCOUNT NAME>" + azure_storage_access_key = "<AZURE STORAGE ACCESS KEY>" + ``` + + If you are using a custom Azure storage domain, + `azure_storage_domain` does **not** have to be set in the Workhorse + configuration. This information is exchanged in an API call between + GitLab Rails and Workhorse. + +### Storj Gateway (SJ) + +NOTE: +The Storj Gateway [does not support](https://github.com/storj/gateway-st/blob/4b74c3b92c63b5de7409378b0d1ebd029db9337d/docs/s3-compatibility.md) multi-threaded copying (see `UploadPartCopy` in the table). +While an implementation [is planned](https://github.com/storj/roadmap/issues/40), you must [disable multi-threaded copying](#multi-threaded-copying) until completion. + +The [Storj Network](https://www.storj.io/) provides an S3-compatible API gateway. Use the following configuration example: + +```ruby +gitlab_rails['object_store']['connection'] = { + 'provider' => 'AWS', + 'endpoint' => 'https://gateway.storjshare.io', + 'path_style' => true, + 'region' => 'eu1', + 'aws_access_key_id' => 'ACCESS_KEY', + 'aws_secret_access_key' => 'SECRET_KEY', + 'aws_signature_version' => 2, + 'enable_signature_v4_streaming' => false +} +``` + +The signature version must be `2`. Using v4 results in a HTTP 411 Length Required error. +For more information, see [issue #4419](https://gitlab.com/gitlab-org/gitlab/-/issues/4419). + +## Full example using the consolidated form and Amazon S3 The following example uses AWS S3 to enable object storage for all supported services: @@ -379,309 +705,7 @@ The following example uses AWS S3 to enable object storage for all supported ser ::EndTabs -#### Common parameters - -In the consolidated configuration, the `object_store` section defines a -common set of parameters. Here we use the YAML from the source -installation because it's easier to see the inheritance: - -```yaml - object_store: - enabled: true - proxy_download: true - connection: - provider: AWS - aws_access_key_id: <AWS_ACCESS_KEY_ID> - aws_secret_access_key: <AWS_SECRET_ACCESS_KEY> - objects: - ... -``` - -The Omnibus configuration maps directly to this: - -```ruby -gitlab_rails['object_store']['enabled'] = true -gitlab_rails['object_store']['proxy_download'] = true -gitlab_rails['object_store']['connection'] = { - 'provider' => 'AWS', - 'aws_access_key_id' => '<AWS_ACCESS_KEY_ID', - 'aws_secret_access_key' => '<AWS_SECRET_ACCESS_KEY>' -} -``` - -| Setting | Description | -|-------------------|-----------------------------------| -| `enabled` | Enable or disable object storage. | -| `proxy_download` | Set to `true` to [enable proxying all files served](#proxy-download). Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data. | -| `connection` | Various [connection options](#connection-settings) described below. | -| `storage_options` | Options to use when saving new objects, such as [server side encryption](#server-side-encryption-headers). Introduced in GitLab 13.3. | -| `objects` | [Object-specific configuration](#object-specific-configuration). | - -### Connection settings - -Both consolidated configuration form and storage-specific configuration form must configure a connection. The following sections describe parameters that can be used -in the `connection` setting. - -#### Amazon S3 - -The connection settings match those provided by [fog-aws](https://github.com/fog/fog-aws): - -| Setting | Description | Default | -|---------------------------------------------|------------------------------------|---------| -| `provider` | Always `AWS` for compatible hosts. | `AWS` | -| `aws_access_key_id` | AWS credentials, or compatible. | | -| `aws_secret_access_key` | AWS credentials, or compatible. | | -| `aws_signature_version` | AWS signature version to use. `2` or `4` are valid options. Digital Ocean Spaces and other providers may need `2`. | `4` | -| `enable_signature_v4_streaming` | Set to `true` to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be `false`. | `true` | -| `region` | AWS region. | | -| `host` | DEPRECATED: Use `endpoint` instead. S3 compatible host for when not using AWS. For example, `localhost` or `storage.example.com`. HTTPS and port 443 is assumed. | `s3.amazonaws.com` | -| `endpoint` | Can be used when configuring an S3 compatible service such as [MinIO](https://min.io), by entering a URL such as `http://127.0.0.1:9000`. This takes precedence over `host`. Always use `endpoint` for consolidated form. | (optional) | -| `path_style` | Set to `true` to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Set to `true` for using [MinIO](https://min.io). Leave as `false` for AWS S3. | `false`. | -| `use_iam_profile` | Set to `true` to use IAM profile instead of access keys. | `false` | -| `aws_credentials_refresh_threshold_seconds` | Sets the [automatic refresh threshold](https://github.com/fog/fog-aws#controlling-credential-refresh-time-with-iam-authentication) when using temporary credentials in IAM. | `15` | - -#### Oracle Cloud S3 - -Oracle Cloud S3 must be sure to use the following settings: - -| Setting | Value | -|---------------------------------|---------| -| `enable_signature_v4_streaming` | `false` | -| `path_style` | `true` | - -If `enable_signature_v4_streaming` is set to `true`, you may see the -following error in `production.log`: - -```plaintext -STREAMING-AWS4-HMAC-SHA256-PAYLOAD is not supported -``` - -#### Google Cloud Storage (GCS) - -Here are the valid connection parameters for GCS: - -| Setting | Description | Example | -|------------------------------|-------------------|---------| -| `provider` | Provider name. | `Google` | -| `google_project` | GCP project name. | `gcp-project-12345` | -| `google_json_key_location` | JSON key path. | `/path/to/gcp-project-12345-abcde.json` | -| `google_json_key_string` | JSON key string. | `{ "type": "service_account", "project_id": "example-project-382839", ... }` | -| `google_application_default` | Set to `true` to use [Google Cloud Application Default Credentials](https://cloud.google.com/docs/authentication#adc) to locate service account credentials. | | - -GitLab reads the value of `google_json_key_location`, then `google_json_key_string`, and finally, `google_application_default`. -It uses the first of these settings that has a value. - -The service account must have permission to access the bucket. For more information, -see the [Cloud Storage authentication documentation](https://cloud.google.com/storage/docs/authentication). - -NOTE: -Bucket encryption with the [Cloud Key Management Service (KMS)](https://cloud.google.com/kms/docs) is not supported and results in [ETag mismatch errors](#etag-mismatch). - -##### GCS example - -For Omnibus installations, this is an example of the `connection` setting -in the consolidated form: - -```ruby -gitlab_rails['object_store']['connection'] = { - 'provider' => 'Google', - 'google_project' => '<GOOGLE PROJECT>', - 'google_json_key_location' => '<FILENAME>' -} -``` - -##### GCS example with ADC - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/275979) in GitLab 13.6. - -Google Cloud Application Default Credentials (ADC) are typically -used with GitLab to use the default service account. This eliminates the -need to supply credentials for the instance. For example, in the consolidated form: - -```ruby -gitlab_rails['object_store']['connection'] = { - 'provider' => 'Google', - 'google_project' => '<GOOGLE PROJECT>', - 'google_application_default' => true -} -``` - -If you use ADC, be sure that: - -- The service account that you use has the -[`iam.serviceAccounts.signBlob` permission](https://cloud.google.com/iam/docs/reference/credentials/rest/v1/projects.serviceAccounts/signBlob). - Typically this is done by granting the `Service Account Token Creator` role to the service account. -- Your virtual machines have the [correct access scopes to access Google Cloud APIs](https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances#changeserviceaccountandscopes). If the machines do not have the right scope, the error logs may show: - - ```markdown - Google::Apis::ClientError (insufficientPermissions: Request had insufficient authentication scopes.) - ``` - -#### Azure Blob storage - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25877) in GitLab 13.4. - -Although Azure uses the word `container` to denote a collection of -blobs, GitLab standardizes on the term `bucket`. Be sure to configure -Azure container names in the `bucket` settings. - -Azure Blob storage can only be used with the [consolidated form](#consolidated-object-storage-configuration) -because a single set of credentials are used to access multiple -containers. The [storage-specific form](#storage-specific-configuration) -is not supported. For more details, see [how to transition to consolidated form](#transition-to-consolidated-form). - -The following are the valid connection parameters for Azure. For more information, see the -[Azure Blob Storage documentation](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction). - -| Setting | Description | Example | -|------------------------------|----------------|-----------| -| `provider` | Provider name. | `AzureRM` | -| `azure_storage_account_name` | Name of the Azure Blob Storage account used to access the storage. | `azuretest` | -| `azure_storage_access_key` | Storage account access key used to access the container. This is typically a secret, 512-bit encryption key encoded in base64. | `czV2OHkvQj9FKEgrTWJRZVRoV21ZcTN0Nnc5eiRDJkYpSkBOY1JmVWpYbjJy\nNHU3eCFBJUQqRy1LYVBkU2dWaw==\n` | -| `azure_storage_domain` | Domain name used to contact the Azure Blob Storage API (optional). Defaults to `blob.core.windows.net`. Set this if you are using Azure China, Azure Germany, Azure US Government, or some other custom Azure domain. | `blob.core.windows.net` | - -- For Omnibus installations, this is an example of the `connection` setting - in the consolidated form: - - ```ruby - gitlab_rails['object_store']['connection'] = { - 'provider' => 'AzureRM', - 'azure_storage_account_name' => '<AZURE STORAGE ACCOUNT NAME>', - 'azure_storage_access_key' => '<AZURE STORAGE ACCESS KEY>', - 'azure_storage_domain' => '<AZURE STORAGE DOMAIN>' - } - ``` - -- For source installations, Workhorse also needs to be configured with Azure - credentials. This isn't needed in Omnibus installs, because the Workhorse - settings are populated from the previous settings. - - 1. Edit `/home/git/gitlab-workhorse/config.toml` and add or amend the following lines: - - ```toml - [object_storage] - provider = "AzureRM" - - [object_storage.azurerm] - azure_storage_account_name = "<AZURE STORAGE ACCOUNT NAME>" - azure_storage_access_key = "<AZURE STORAGE ACCESS KEY>" - ``` - - If you are using a custom Azure storage domain, - `azure_storage_domain` does **not** have to be set in the Workhorse - configuration. This information is exchanged in an API call between - GitLab Rails and Workhorse. - -#### Storj Gateway (SJ) - -NOTE: -The Storj Gateway [does not support](https://github.com/storj/gateway-st/blob/4b74c3b92c63b5de7409378b0d1ebd029db9337d/docs/s3-compatibility.md) multi-threaded copying (see `UploadPartCopy` in the table). -While an implementation [is planned](https://github.com/storj/roadmap/issues/40), you must [disable multi-threaded copying](#multi-threaded-copying) until completion. - -The [Storj Network](https://www.storj.io/) provides an S3-compatible API gateway. Use the following configuration example: - -```ruby -gitlab_rails['object_store']['connection'] = { - 'provider' => 'AWS', - 'endpoint' => 'https://gateway.storjshare.io', - 'path_style' => true, - 'region' => 'eu1', - 'aws_access_key_id' => 'ACCESS_KEY', - 'aws_secret_access_key' => 'SECRET_KEY', - 'aws_signature_version' => 2, - 'enable_signature_v4_streaming' => false -} -``` - -The signature version must be `2`. Using v4 results in a HTTP 411 Length Required error. -For more information, see [issue #4419](https://gitlab.com/gitlab-org/gitlab/-/issues/4419). - -### Object-specific configuration - -The following YAML shows how the `object_store` section defines -object-specific configuration block and how the `enabled` and -`proxy_download` flags can be overridden. The `bucket` is the only -required parameter within each type: - -```yaml - object_store: - connection: - ... - objects: - artifacts: - bucket: artifacts - proxy_download: false - external_diffs: - bucket: external-diffs - lfs: - bucket: lfs-objects - uploads: - bucket: uploads - packages: - bucket: packages - dependency_proxy: - enabled: false - bucket: dependency_proxy - terraform_state: - bucket: terraform - pages: - bucket: pages -``` - -This maps to this Omnibus GitLab configuration: - -```ruby -gitlab_rails['object_store']['objects']['artifacts']['bucket'] = 'artifacts' -gitlab_rails['object_store']['objects']['artifacts']['proxy_download'] = false -gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = 'external-diffs' -gitlab_rails['object_store']['objects']['lfs']['bucket'] = 'lfs-objects' -gitlab_rails['object_store']['objects']['uploads']['bucket'] = 'uploads' -gitlab_rails['object_store']['objects']['packages']['bucket'] = 'packages' -gitlab_rails['object_store']['objects']['dependency_proxy']['enabled'] = false -gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = 'dependency-proxy' -gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = 'terraform-state' -gitlab_rails['object_store']['objects']['pages']['bucket'] = 'pages' -``` - -This is the list of valid `objects` that can be used: - -| Type | Description | -|--------------------|----------------------------------------------------------------------------| -| `artifacts` | [CI artifacts](job_artifacts.md) | -| `external_diffs` | [Merge request diffs](merge_request_diffs.md) | -| `uploads` | [User uploads](uploads.md) | -| `lfs` | [Git Large File Storage objects](lfs/index.md) | -| `packages` | [Project packages (for example, PyPI, Maven, or NuGet)](packages/index.md) | -| `dependency_proxy` | [Dependency Proxy](packages/dependency_proxy.md) | -| `terraform_state` | [Terraform state files](terraform_state.md) | -| `pages` | [Pages](pages/index.md) | - -Within each object type, three parameters can be defined: - -| Setting | Required? | Description | -|------------------|------------------------|-------------------------------------| -| `bucket` | **{check-circle}** Yes | Bucket name for the object storage. | -| `enabled` | **{dotted-circle}** No | Overrides the common parameter. | -| `proxy_download` | **{dotted-circle}** No | Overrides the common parameter. | - -#### Disable object storage for specific features - -As seen above, object storage can be disabled for specific types by -setting the `enabled` flag to `false`. For example, to disable object -storage for CI artifacts: - -```ruby -gitlab_rails['object_store']['objects']['artifacts']['enabled'] = false -``` - -A bucket is not needed if the feature is disabled entirely. For example, -no bucket is needed if CI artifacts are disabled with this setting: - -```ruby -gitlab_rails['artifacts_enabled'] = false -``` - -### Migrate to object storage +## Migrate to object storage To migrate existing local data to object storage see the following guides: @@ -694,7 +718,7 @@ To migrate existing local data to object storage see the following guides: - [Terraform state files](terraform_state.md#migrate-to-object-storage) - [Pages content](pages/index.md#migrate-pages-deployments-to-object-storage) -### Transition to consolidated form +## Transition to consolidated form Prior to GitLab 13.2: @@ -725,48 +749,11 @@ additional complexity and unnecessary redundancy. Since both GitLab Rails and Workhorse components need access to object storage, the consolidated form avoids excessive duplication of credentials. -The consolidated object storage configuration is used _only_ if all lines from +The consolidated form is used _only_ if all lines from the original form is omitted. To move to the consolidated form, remove the original configuration (for example, `artifacts_object_store_enabled`, or `uploads_object_store_connection`) -### Storage-specific configuration - -For configuring object storage in GitLab 13.1 and earlier, or for storage types not -supported by consolidated configuration form, refer to the following guides: - -| Object storage type | Supported by consolidated configuration? | -|---------------------|------------------------------------------| -| [Backups](../raketasks/backup_gitlab.md#upload-backups-to-a-remote-cloud-storage) | **{dotted-circle}** No | -| [Job artifacts](job_artifacts.md#using-object-storage) including archived job logs | **{check-circle}** Yes | -| [LFS objects](lfs/index.md#storing-lfs-objects-in-remote-object-storage) | **{check-circle}** Yes | -| [Uploads](uploads.md#using-object-storage) | **{check-circle}** Yes | -| [Container Registry](packages/container_registry.md#use-object-storage) (optional feature) | **{dotted-circle}** No | -| [Merge request diffs](merge_request_diffs.md#using-object-storage) | **{check-circle}** Yes | -| [Mattermost](https://docs.mattermost.com/configure/file-storage-configuration-settings.html)| **{dotted-circle}** No | -| [Packages](packages/index.md#use-object-storage) (optional feature) | **{check-circle}** Yes | -| [Dependency Proxy](packages/dependency_proxy.md#using-object-storage) (optional feature) | **{check-circle}** Yes | -| [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | **{dotted-circle}** No | -| [Terraform state files](terraform_state.md#using-object-storage) | **{check-circle}** Yes | -| [Pages content](pages/index.md#using-object-storage) | **{check-circle}** Yes | - -WARNING: -The use of [encrypted S3 buckets](#encrypted-s3-buckets) with non-consolidated configuration is not supported. -You may start getting [ETag mismatch errors](#etag-mismatch) if you use it. - -### Other alternatives to file system storage - -If you're working to [scale out](reference_architectures/index.md) your GitLab implementation, -or add fault tolerance and redundancy, you may be -looking at removing dependencies on block or network file systems. -See the following additional guides: - -1. Make sure the [`git` user home directory](https://docs.gitlab.com/omnibus/settings/configuration.html#moving-the-home-directory-for-a-user) is on local disk. -1. Configure [database lookup of SSH keys](operations/fast_ssh_key_lookup.md) - to eliminate the need for a shared `authorized_keys` file. -1. [Prevent local disk usage for job logs](job_logs.md#prevent-local-disk-usage). -1. [Disable Pages local storage](pages/index.md#disable-pages-local-storage). - ## Use Amazon instance profiles Instead of supplying AWS access and secret keys in object storage @@ -784,10 +771,10 @@ address must be added to the `no_proxy` list. ### Encrypted S3 buckets > - [Introduced](https://gitlab.com/gitlab-org/gitlab-workhorse/-/merge_requests/466) in GitLab 13.1 for instance profiles only and [S3 default encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/bucket-encryption.html). -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/34460) in GitLab 13.2 for static credentials when [consolidated object storage configuration](#consolidated-object-storage-configuration) and [S3 default encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/bucket-encryption.html) are used. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/34460) in GitLab 13.2 for static credentials when the [consolidated form](#configure-a-single-storage-connection-for-all-object-types-consolidated-form) and [S3 default encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/bucket-encryption.html) is used. When configured either with an instance profile or with the consolidated -object configuration, GitLab Workhorse properly uploads files to S3 +form, GitLab Workhorse properly uploads files to S3 buckets that have [SSE-S3 or SSE-KMS encryption enabled by default](https://docs.aws.amazon.com/kms/latest/developerguide/services-s3.html). Customer master keys (CMKs) and SSE-C encryption are [not supported since this requires sending the encryption keys in every request](https://gitlab.com/gitlab-org/gitlab/-/issues/226006). @@ -812,7 +799,7 @@ the Workhorse S3 client is enabled. One of the following two conditions must be fulfilled: - `use_iam_profile` is `true` in the connection settings. -- Consolidated object storage settings are in use. +- Consolidated form is in use. [ETag mismatch errors](#etag-mismatch) occur if server side encryption headers are used without enabling the Workhorse S3 client. @@ -889,6 +876,19 @@ After you have done at least one successful Rclone copy from the old location to 1. Perform a final `rclone sync` run, knowing that your users cannot add new objects so you do not leave any behind in the old bucket. 1. Update the object storage configuration of your GitLab server to use the new provider for `uploads`. +## Alternatives to file system storage + +If you're working to [scale out](reference_architectures/index.md) your GitLab implementation, +or add fault tolerance and redundancy, you may be +looking at removing dependencies on block or network file systems. +See the following additional guides: + +1. Make sure the [`git` user home directory](https://docs.gitlab.com/omnibus/settings/configuration.html#moving-the-home-directory-for-a-user) is on local disk. +1. Configure [database lookup of SSH keys](operations/fast_ssh_key_lookup.md) + to eliminate the need for a shared `authorized_keys` file. +1. [Prevent local disk usage for job logs](job_logs.md#prevent-local-disk-usage). +1. [Disable Pages local storage](pages/index.md#disable-pages-local-storage). + ## Troubleshooting ### Objects are not included in GitLab backups @@ -992,14 +992,14 @@ If you are seeing this ETag mismatch error with Amazon Web Services S3, it's likely this is due to [encryption settings on your bucket](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html). To fix this issue, you have two options: -- [Use the consolidated object configuration](#consolidated-object-storage-configuration). +- [Use the consolidated form](#configure-a-single-storage-connection-for-all-object-types-consolidated-form). - [Use Amazon instance profiles](#use-amazon-instance-profiles). The first option is recommended for MinIO. Otherwise, the [workaround for MinIO](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/1564#note_244497658) is to use the `--compat` parameter on the server. -Without consolidated object store configuration or instance profiles enabled, +Without the consolidated form or instance profiles enabled, GitLab Workhorse uploads files to S3 using pre-signed URLs that do not have a `Content-MD5` HTTP header computed for them. To ensure data is not corrupted, Workhorse checks that the MD5 hash of the data sent @@ -1007,9 +1007,14 @@ equals the ETag header returned from the S3 server. When encryption is enabled, this is not the case, which causes Workhorse to report an `ETag mismatch` error during an upload. -With the consolidated object configuration and instance profile, Workhorse has -S3 credentials so that it can compute the `Content-MD5` header. This -eliminates the need to compare ETag headers returned from the S3 server. +When the consolidated form is: + +- Used with an S3-compatible object storage or an istance profile, Workhorse + uses its internal S3 client which has S3 credentials so that it can compute + the `Content-MD5` header. This eliminates the need to compare ETag headers + returned from the S3 server. +- Not used with an S3-compatible object storage, Workhorse falls back to using + pre-signed URLs. Encrypting buckets with the GCS [Cloud Key Management Service (KMS)](https://cloud.google.com/kms/docs) is not supported and results in ETag mismatch errors. diff --git a/doc/administration/operations/gitlab_sshd.md b/doc/administration/operations/gitlab_sshd.md index 1153321267d..249d6232616 100644 --- a/doc/administration/operations/gitlab_sshd.md +++ b/doc/administration/operations/gitlab_sshd.md @@ -6,13 +6,13 @@ info: To determine the technical writer assigned to the Stage/Group associated w # `gitlab-sshd` **(FREE SELF)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299109) in GitLab 14.5 as an **Alpha** release for self-managed customers. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299109) in GitLab 14.5 as an Experiment for self-managed customers. > - Ready for production use with [Cloud Native GitLab in GitLab 15.1](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/2540) and [Omnibus GitLab in GitLab 15.9](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5937). `gitlab-sshd` is [a standalone SSH server](https://gitlab.com/gitlab-org/gitlab-shell/-/tree/main/internal/sshd) written in Go. It is provided as a part of the `gitlab-shell` package. It has a lower memory use as a OpenSSH alternative, and supports -[group access restriction by IP address](../../user/group/index.md) for applications +[group access restriction by IP address](../../user/group/access_and_permissions.md#restrict-group-access-by-ip-address) for applications running behind the proxy. `gitlab-sshd` is a lightweight alternative to OpenSSH for providing @@ -27,8 +27,9 @@ If you are considering switching from OpenSSH to `gitlab-sshd`, consider these c - `gitlab-sshd` supports the PROXY protocol. It can run behind proxy servers that rely on it, such as HAProxy. The PROXY protocol is not enabled by default, but [it can be enabled](#proxy-protocol-support). -- `gitlab-sshd` **does not** support SSH certificates. For more details, read - [issue #495](https://gitlab.com/gitlab-org/gitlab-shell/-/issues/495). +- `gitlab-sshd` **does not** support SSH certificates. For more details, see the + [confidential issue](../../user/project/issues/confidential_issues.md) + `https://gitlab.com/gitlab-org/gitlab-shell/-/issues/495`. ## Enable `gitlab-sshd` diff --git a/doc/administration/operations/index.md b/doc/administration/operations/index.md index b02aab738ea..867ef3236ee 100644 --- a/doc/administration/operations/index.md +++ b/doc/administration/operations/index.md @@ -4,30 +4,21 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Performing operations in GitLab **(FREE SELF)** +# Maintain your GitLab installation **(FREE SELF)** -Keep your GitLab instance up and running smoothly. +Keep your GitLab instance up and running. -- [Upgrading GitLab](../../update/index.md). -- [Rake tasks](../../raketasks/index.md): Tasks for common administration and operational processes such as - [cleaning up unneeded items from GitLab instance](../../raketasks/cleanup.md), integrity checks, - and more. -- [Moving repositories](moving_repositories.md): Moving all repositories managed - by GitLab to another file system or another server. -- [Sidekiq MemoryKiller](../sidekiq/sidekiq_memory_killer.md): Configure Sidekiq MemoryKiller - to restart Sidekiq. -- [Multiple Sidekiq processes](../sidekiq/extra_sidekiq_processes.md): Configure multiple Sidekiq processes to ensure certain queues always have dedicated workers, no matter the number of jobs that must be processed. **(FREE SELF)** -- [Puma](puma.md): Understand Puma and puma-worker-killer. -- [`gitlab-sshd`](gitlab_sshd.md): Use GitLab SSH daemon instead of OpenSSH. -- Speed up SSH operations by - [Authorizing SSH users via a fast, indexed lookup to the GitLab database](fast_ssh_key_lookup.md), and/or - by [doing away with user SSH keys stored on GitLab entirely in favor of SSH certificates](ssh_certificates.md). -- [File System Performance Benchmarking](filesystem_benchmarking.md): File system - performance can have a big impact on GitLab performance, especially for actions - that read or write Git repositories. This information helps benchmark - file system performance against known good and bad real-world systems. -- [The Rails Console](rails_console.md): Provides a way to interact with your GitLab instance from the command line. - Used for troubleshooting a problem or retrieving some data that can only be done through direct access to GitLab. -- [ChatOps Scripts](https://gitlab.com/gitlab-com/chatops): The GitLab.com Infrastructure team uses this repository to house - common ChatOps scripts they use to troubleshoot and maintain the production instance of GitLab.com. - These scripts can be used by administrators of GitLab instances of all sizes. +- [Housekeeping](../../administration/housekeeping.md) +- [Activate GitLab EE with license](../../user/admin_area/license_file.md) +- [Fast SSH key lookup](../../administration/operations/fast_ssh_key_lookup.md) +- [File system benchmarking](../../administration/operations/filesystem_benchmarking.md) +- [`gitlab-sshd`](../../administration/operations/gitlab_sshd.md) +- [Rails console](../../administration/operations/rails_console.md) +- [Use SSH certificates](../../administration/operations/ssh_certificates.md) +- [Enable encrypted configuration](../../administration/encrypted_configuration.md) +- [Rake tasks](../../raketasks/index.md) +- [Backup and restore](../../raketasks/backup_restore.md) +- [Inactive project deletion](../../administration/inactive_project_deletion.md) +- [Move repositories](../../administration/operations/moving_repositories.md) +- [Read-only state](../../administration/read_only_gitlab.md) +- [Restart GitLab](../../administration/restart_gitlab.md) diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md index f2f9f1cdcda..5ff9208ecb9 100644 --- a/doc/administration/operations/puma.md +++ b/doc/administration/operations/puma.md @@ -99,7 +99,7 @@ To change the worker timeout to 600 seconds: ## Disable Puma clustered mode in memory-constrained environments WARNING: -This is an experimental [Alpha feature](../../policy/alpha-beta-support.md#alpha-features) and subject to change without notice. The feature +This feature is an [Experiment](../../policy/alpha-beta-support.md#experiment) and subject to change without notice. The feature is not ready for production use. If you want to use this feature, you should test outside of production first. See the [known issues](#puma-single-mode-known-issues) for additional details. diff --git a/doc/administration/package_information/defaults.md b/doc/administration/package_information/defaults.md index 3b52b6bca82..932342a6a92 100644 --- a/doc/administration/package_information/defaults.md +++ b/doc/administration/package_information/defaults.md @@ -30,7 +30,7 @@ by default: | PgBouncer exporter | No | Port | X | 9188 | | GitLab Exporter | Yes | Port | X | 9168 | | Sidekiq exporter | Yes | Port | X | 8082 | -| Sidekiq health check | No | Port | X | 8092[^Sidekiq-health] | +| Sidekiq health check | Yes | Port | X | 8092[^Sidekiq-health] | | Web exporter | No | Port | X | 8083 | | Geo PostgreSQL | No | Socket | Port (5431) | X | | Redis Sentinel | No | Port | X | 26379 | diff --git a/doc/administration/package_information/supported_os.md b/doc/administration/package_information/supported_os.md index f92d57c0035..ea87dfb6e28 100644 --- a/doc/administration/package_information/supported_os.md +++ b/doc/administration/package_information/supported_os.md @@ -21,7 +21,6 @@ The following lists the currently supported OSs and their possible EOL dates. | CentOS 7 | GitLab CE / GitLab EE 7.10.0 | x86_64 | [CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | June 2024 | <https://wiki.centos.org/About/Product> | | Debian 10 | GitLab CE / GitLab EE 12.2.0 | amd64, arm64 | [Debian Install Documentation](https://about.gitlab.com/install/#debian) | 2024 | <https://wiki.debian.org/LTS> | | Debian 11 | GitLab CE / GitLab EE 14.6.0 | amd64, arm64 | [Debian Install Documentation](https://about.gitlab.com/install/#debian) | 2026 | <https://wiki.debian.org/LTS> | -| OpenSUSE 15.3 | GitLab CE / GitLab EE 14.5.0 | x86_64, aarch64 | [OpenSUSE Install Documentation](https://about.gitlab.com/install/#opensuse-leap) | Dec 2022 | <https://en.opensuse.org/Lifetime> | | OpenSUSE 15.4 | GitLab CE / GitLab EE 15.7.0 | x86_64, aarch64 | [OpenSUSE Install Documentation](https://about.gitlab.com/install/#opensuse-leap) | Nov 2023 | <https://en.opensuse.org/Lifetime> | | RHEL 8 | GitLab CE / GitLab EE 12.8.1 | x86_64, arm64 | [Use CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | May 2029 | [RHEL Details](https://access.redhat.com/support/policy/updates/errata/#Life_Cycle_Dates) | | SLES 12 | GitLab EE 9.0.0 | x86_64 | [Use OpenSUSE Install Documentation](https://about.gitlab.com/install/#opensuse-leap) | Oct 2027 | <https://www.suse.com/lifecycle/> | @@ -103,6 +102,7 @@ release for them can be found below: | Ubuntu 16.04 | [April 2021](https://ubuntu.com/info/release-end-of-life) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_13.12&dist=ubuntu%2Fxenial) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_13.12&dist=ubuntu%2Fxenial) 13.12 | | OpenSUSE 15.2 | [December 2021](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-14.7&dist=opensuse%2F15.2) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-14.7&dist=opensuse%2F15.2) 14.7 | | Debian 9 "Stretch" | [June 2022](https://lists.debian.org/debian-lts-announce/2022/07/msg00002.html) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_15.2&dist=debian%2Fstretch) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_15.2&dist=debian%2Fstretch) 15.2 | +| OpenSUSE 15.3 | [December 2022](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-15.10&dist=opensuse%2F15.3) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-15.10&dist=opensuse%2F15.3) 15.10 | NOTE: An exception to this deprecation policy is when we are unable to provide diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index 619ec2490a7..05d2f307f02 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -932,7 +932,7 @@ information, see the [Sidekiq configuration](../sidekiq/index.md) page. To reduce the amount of [Container Registry disk space used by a given project](#registry-disk-space-usage-by-project), -administrators can clean up image tags +administrators can setup cleanup policies and [run garbage collection](#container-registry-garbage-collection). ### Registry Disk Space Usage by Project diff --git a/doc/administration/packages/dependency_proxy.md b/doc/administration/packages/dependency_proxy.md index 78efd75bbe3..a29d6d93051 100644 --- a/doc/administration/packages/dependency_proxy.md +++ b/doc/administration/packages/dependency_proxy.md @@ -122,7 +122,7 @@ To change the local storage path: Instead of relying on the local storage, you can use an object storage to store the blobs of the Dependency Proxy. In GitLab 13.2 and later, you should use the -[consolidated object storage settings](../object_storage.md#consolidated-object-storage-configuration). +[consolidated object storage settings](../object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form). This section describes the earlier configuration format. [Migration steps still apply](#migrate-local-dependency-proxy-blobs-and-manifests-to-object-storage). [Read more about using object storage with GitLab](../object_storage.md). diff --git a/doc/administration/packages/index.md b/doc/administration/packages/index.md index 6e53d77109f..f0f238aa5ba 100644 --- a/doc/administration/packages/index.md +++ b/doc/administration/packages/index.md @@ -124,7 +124,7 @@ Instead of relying on the local storage, you can use an object storage to store packages. For more information, see how to use the -[consolidated object storage settings](../object_storage.md#consolidated-object-storage-configuration). +[consolidated object storage settings](../object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form). ### Migrate local packages to object storage diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index efeb19a8d5e..865d6e0bed7 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -1,8 +1,7 @@ --- -stage: Create -group: Editor +stage: Plan +group: Knowledge info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -description: 'Learn how to administer GitLab Pages.' --- # GitLab Pages administration **(FREE SELF)** @@ -247,20 +246,20 @@ control over how the Pages daemon runs and serves content in your environment. | `enable` | Enable or disable GitLab Pages on the current system. | | `external_http` | Configure Pages to bind to one or more secondary IP addresses, serving HTTP requests. Multiple addresses can be given as an array, along with exact ports, for example `['1.2.3.4', '1.2.3.5:8063']`. Sets value for `listen_http`. | | `external_https` | Configure Pages to bind to one or more secondary IP addresses, serving HTTPS requests. Multiple addresses can be given as an array, along with exact ports, for example `['1.2.3.4', '1.2.3.5:8063']`. Sets value for `listen_https`. | -| `server_shutdown_timeout` | GitLab Pages server shutdown timeout in seconds (default: 30 s). | -| `gitlab_client_http_timeout` | GitLab API HTTP client connection timeout in seconds (default: 10 s). | -| `gitlab_client_jwt_expiry` | JWT Token expiry time in seconds (default: 30 s). | -| `gitlab_cache_expiry` | The maximum time a domain's configuration is stored in the cache (default: 600 s). | -| `gitlab_cache_refresh` | The interval at which a domain's configuration is set to be due to refresh (default: 60 s). | -| `gitlab_cache_cleanup` | The interval at which expired items are removed from the cache (default: 60 s). | -| `gitlab_retrieval_timeout` | The maximum time to wait for a response from the GitLab API per request (default: 30 s). | -| `gitlab_retrieval_interval` | The interval to wait before retrying to resolve a domain's configuration via the GitLab API (default: 1 s). | +| `server_shutdown_timeout` | GitLab Pages server shutdown timeout in seconds (default: `30s`). | +| `gitlab_client_http_timeout` | GitLab API HTTP client connection timeout in seconds (default: `10s`). | +| `gitlab_client_jwt_expiry` | JWT Token expiry time in seconds (default: `30s`). | +| `gitlab_cache_expiry` | The maximum time a domain's configuration is stored in the cache (default: `600s`). | +| `gitlab_cache_refresh` | The interval at which a domain's configuration is set to be due to refresh (default: `60s`). | +| `gitlab_cache_cleanup` | The interval at which expired items are removed from the cache (default: `60s`). | +| `gitlab_retrieval_timeout` | The maximum time to wait for a response from the GitLab API per request (default: `30s`). | +| `gitlab_retrieval_interval` | The interval to wait before retrying to resolve a domain's configuration via the GitLab API (default: `1s`). | | `gitlab_retrieval_retries` | The maximum number of times to retry to resolve a domain's configuration via the API (default: 3). | | `domain_config_source` | This parameter was removed in 14.0, on earlier versions it can be used to enable and test API domain configuration source | | `gitlab_id` | The OAuth application public ID. Leave blank to automatically fill when Pages authenticates with GitLab. | | `gitlab_secret` | The OAuth application secret. Leave blank to automatically fill when Pages authenticates with GitLab. | | `auth_scope` | The OAuth application scope to use for authentication. Must match GitLab Pages OAuth application settings. Leave blank to use `api` scope by default. | -| `auth_cookie_session_timeout` | Authentication cookie session timeout in seconds (default: 600 s). A value of `0` means the cookie is deleted after the browser session ends. | +| `auth_cookie_session_timeout` | Authentication cookie session timeout in seconds (default: `10m`). A value of `0` means the cookie is deleted after the browser session ends. | | `gitlab_server` | Server to use for authentication when access control is enabled; defaults to GitLab `external_url`. | | `headers` | Specify any additional http headers that should be sent to the client with each response. Multiple headers can be given as an array, header and value as one string, for example `['my-header: myvalue', 'my-other-header: my-other-value']` | | `enable_disk` | Allows the GitLab Pages daemon to serve content from disk. Shall be disabled if shared disk storage isn't available. | @@ -514,7 +513,7 @@ internet connectivity is gated by a proxy. To use a proxy for GitLab Pages: ### Using a custom Certificate Authority (CA) When using certificates issued by a custom CA, [Access Control](../../user/project/pages/pages_access_control.md) and -the [online view of HTML job artifacts](../../ci/pipelines/job_artifacts.md#download-job-artifacts) +the [online view of HTML job artifacts](../../ci/jobs/job_artifacts.md#download-job-artifacts) fails to work if the custom CA is not recognized. This usually results in this error: @@ -540,22 +539,22 @@ archive. You can modify the cache behavior by changing the following configurati | Setting | Description | | ------- | ----------- | -| `zip_cache_expiration` | The cache expiration interval of ZIP archives. Must be greater than zero to avoid serving stale content. Default is 60 s. | -| `zip_cache_cleanup` | The interval at which archives are cleaned from memory if they have already expired. Default is 30 s. | -| `zip_cache_refresh` | The time interval in which an archive is extended in memory if accessed before `zip_cache_expiration`. This works together with `zip_cache_expiration` to determine if an archive is extended in memory. See the [example below](#zip-cache-refresh-example) for important details. Default is 30 s. | +| `zip_cache_expiration` | The cache expiration interval of ZIP archives. Must be greater than zero to avoid serving stale content. Default is `60s`. | +| `zip_cache_cleanup` | The interval at which archives are cleaned from memory if they have already expired. Default is `30s`. | +| `zip_cache_refresh` | The time interval in which an archive is extended in memory if accessed before `zip_cache_expiration`. This works together with `zip_cache_expiration` to determine if an archive is extended in memory. See the [example below](#zip-cache-refresh-example) for important details. Default is `30s`. | | `zip_open_timeout` | The maximum time allowed to open a ZIP archive. Increase this time for big archives or slow network connections, as doing so may affect the latency of serving Pages. Default is 30 s. | -| `zip_http_client_timeout` | The maximum time for the ZIP HTTP client. Default is 30 m. | +| `zip_http_client_timeout` | The maximum time for the ZIP HTTP client. Default is `30m`. | #### ZIP cache refresh example Archives are refreshed in the cache (extending the time they are held in memory) if they're accessed before `zip_cache_expiration`, and the time left before expiring is less than or equal to -`zip_cache_refresh`. For example, if `archive.zip` is accessed at time 0 s, it expires in 60 s (the -default for `zip_cache_expiration`). In the example below, if the archive is opened again after 15 s -it is **not** refreshed because the time left for expiry (45 s) is greater than `zip_cache_refresh` -(default 30 s). However, if the archive is accessed again after 45 s (from the first time it was +`zip_cache_refresh`. For example, if `archive.zip` is accessed at time `0s`, it expires in `60s` (the +default for `zip_cache_expiration`). In the example below, if the archive is opened again after `15s` +it is **not** refreshed because the time left for expiry (`45s`) is greater than `zip_cache_refresh` +(default `30s`). However, if the archive is accessed again after `45s` (from the first time it was opened) it's refreshed. This extends the time the archive remains in memory from -`45s + zip_cache_expiration (60s)`, for a total of 105 s. +`45s + zip_cache_expiration (60s)`, for a total of `105s`. After an archive reaches `zip_cache_expiration`, it's marked as expired and removed on the next `zip_cache_cleanup` interval. @@ -710,6 +709,15 @@ To set the maximum number of GitLab Pages custom domains for a project: 1. Enter a value for **Maximum number of custom domains per project**. Use `0` for unlimited domains. 1. Select **Save changes**. +## Set maximum number of files per GitLab Pages website + +The total number of file entries (including directories and symlinks) is limited to `200,000` per GitLab Pages website. + +You can update the limit in your self-managed instance using the +[GitLab Rails console](../operations/rails_console.md#starting-a-rails-console-session). + +For more information, see [GitLab application limits](../instance_limits.md#number-of-files-per-gitlab-pages-website). + ## Running GitLab Pages on a separate server You can run the GitLab Pages daemon on a separate server to decrease the load on @@ -818,7 +826,7 @@ From [GitLab 13.3 to GitLab 13.12](#domain-source-configuration-before-140) GitL Starting from [GitLab 14.0](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5993) GitLab Pages uses API by default and fails to start if it can't connect to it. -For common issues, see the [troubleshooting section](#failed-to-connect-to-the-internal-gitlab-api). +For common issues, see [troubleshooting](troubleshooting.md#failed-to-connect-to-the-internal-gitlab-api). For more details see this [blog post](https://about.gitlab.com/blog/2020/08/03/how-gitlab-pages-uses-the-gitlab-api-to-serve-content/). @@ -895,7 +903,7 @@ configuration is tried to be resolved automatically before reporting an error. > [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5577) in GitLab 13.6. -[Read more about using object storage with GitLab](../object_storage.md). +For more information, see [object storage](../object_storage.md). ### Object storage settings @@ -917,7 +925,7 @@ If you want to stop using and disconnect the NFS server, you need to #### S3-compatible connection settings In GitLab 13.2 and later, you should use the -[consolidated object storage settings](../object_storage.md#consolidated-object-storage-configuration). +[consolidated object storage settings](../object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form). This section describes the earlier configuration format. See [the available connection settings for different providers](../object_storage.md#connection-settings). @@ -1100,8 +1108,8 @@ In GitLab 14.0 a number of breaking changes were introduced which may require so The steps below describe the best way to migrate without causing any downtime for your GitLab instance. A GitLab instance running on a single server typically upgrades to 14.0 smoothly, and there should be minimal issues after the upgrade is complete. -Regardless, we recommend everyone follow the migration steps to ensure a successful upgrade. -If at any point you run into issues, consult the [troubleshooting section](#troubleshooting). +Regardless, you should follow the migration steps to ensure a successful upgrade. +For common issues, see [troubleshooting](troubleshooting.md). If your current GitLab version is lower than 13.12, then you must first update to 13.12. Updating directly to 14.0 is [not supported](../../update/index.md#upgrade-paths) @@ -1185,312 +1193,6 @@ Rate limits are enforced using the following: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -<!-- ## Troubleshooting - -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them in -the section below. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. - -Each scenario can be a third-level heading, for example `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> - -## Troubleshooting - -### How to see GitLab Pages logs - -You can see Pages daemon logs by running: - -```shell -sudo gitlab-ctl tail gitlab-pages -``` - -You can also find the log file in `/var/log/gitlab/gitlab-pages/current`. - -### `unsupported protocol scheme \"\""` - -If you see the following error: - -```plaintext -{"error":"failed to connect to internal Pages API: Get \"/api/v4/internal/pages/status\": unsupported protocol scheme \"\"","level":"warning","msg":"attempted to connect to the API","time":"2021-06-23T20:03:30Z"} -``` - -It means you didn't set the HTTP(S) protocol scheme in the Pages server settings. -To fix it: - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_pages['gitlab_server'] = "https://<your_pages_domain_name>" - gitlab_pages['internal_gitlab_server'] = "https://<your_pages_domain_name>" - ``` - -1. Reconfigure GitLab: - - ```shell - sudo gitlab-ctl reconfigure - ``` - -### 502 error when connecting to GitLab Pages proxy when server does not listen over IPv6 - -In some cases, NGINX might default to using IPv6 to connect to the GitLab Pages -service even when the server does not listen over IPv6. You can identify when -this is happening if you see something similar to the log entry below in the -`gitlab_pages_error.log`: - -```plaintext -2020/02/24 16:32:05 [error] 112654#0: *4982804 connect() failed (111: Connection refused) while connecting to upstream, client: 123.123.123.123, server: ~^(?<group>.*)\.pages\.example\.com$, request: "GET /-/group/project/-/jobs/1234/artifacts/artifact.txt HTTP/1.1", upstream: "http://[::1]:8090//-/group/project/-/jobs/1234/artifacts/artifact.txt", host: "group.example.com" -``` - -To resolve this, set an explicit IP and port for the GitLab Pages `listen_proxy` setting -to define the explicit address that the GitLab Pages daemon should listen on: - -```ruby -gitlab_pages['listen_proxy'] = '127.0.0.1:8090' -``` - -### Intermittent 502 errors or after a few days - -If you run Pages on a system that uses `systemd` and -[`tmpfiles.d`](https://www.freedesktop.org/software/systemd/man/tmpfiles.d.html), -you may encounter intermittent 502 errors trying to serve Pages with an error similar to: - -```plaintext -dial tcp: lookup gitlab.example.com on [::1]:53: dial udp [::1]:53: connect: no route to host" -``` - -GitLab Pages creates a [bind mount](https://man7.org/linux/man-pages/man8/mount.8.html) -inside `/tmp/gitlab-pages-*` that includes files like `/etc/hosts`. -However, `systemd` may clean the `/tmp/` directory on a regular basis so the DNS -configuration may be lost. - -To stop `systemd` from cleaning the Pages related content: - -1. Tell `tmpfiles.d` to not remove the Pages `/tmp` directory: - - ```shell - echo 'x /tmp/gitlab-pages-*' >> /etc/tmpfiles.d/gitlab-pages-jail.conf - ``` - -1. Restart GitLab Pages: - - ```shell - sudo gitlab-ctl restart gitlab-pages - ``` - -### Unable to access GitLab Pages - -If you can't access your GitLab Pages (such as receiving `502 Bad Gateway` errors, or a login loop) -and in your Pages log shows this error: - -```plaintext -"error":"retrieval context done: context deadline exceeded","host":"root.docs-cit.otenet.gr","level":"error","msg":"could not fetch domain information from a source" -``` - -1. Add the following to `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_pages['internal_gitlab_server'] = 'http://localhost:8080' - ``` - -1. Restart GitLab Pages: - - ```shell - sudo gitlab-ctl restart gitlab-pages - ``` - -### Failed to connect to the internal GitLab API - -If you see the following error: - -```plaintext -ERRO[0010] Failed to connect to the internal GitLab API after 0.50s error="failed to connect to internal Pages API: HTTP status: 401" -``` - -If you are [Running GitLab Pages on a separate server](#running-gitlab-pages-on-a-separate-server) -you must copy the `/etc/gitlab/gitlab-secrets.json` file -from the **GitLab server** to the **Pages server** after upgrading to GitLab 13.3, -as described in that section. - -Other reasons may include network connectivity issues between your -**GitLab server** and your **Pages server** such as firewall configurations or closed ports. -For example, if there is a connection timeout: - -```plaintext -error="failed to connect to internal Pages API: Get \"https://gitlab.example.com:3000/api/v4/internal/pages/status\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" -``` - -### Pages cannot communicate with an instance of the GitLab API - -If you use the default value for `domain_config_source=auto` and run multiple instances of GitLab -Pages, you may see intermittent 502 error responses while serving Pages content. You may also see -the following warning in the Pages logs: - -```plaintext -WARN[0010] Pages cannot communicate with an instance of the GitLab API. Please sync your gitlab-secrets.json file https://gitlab.com/gitlab-org/gitlab-pages/-/issues/535#workaround. error="pages endpoint unauthorized" -``` - -This can happen if your `gitlab-secrets.json` file is out of date between GitLab Rails and GitLab -Pages. Follow steps 8-10 of [Running GitLab Pages on a separate server](#running-gitlab-pages-on-a-separate-server), -in all of your GitLab Pages instances. - -### Intermittent 502 errors when using an AWS Network Load Balancer and GitLab Pages - -Connections will time out when using a Network Load Balancer with client IP preservation enabled and [the request is looped back to the source server](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-troubleshooting.html#loopback-timeout). -This can happen to GitLab instances with multiple servers -running both the core GitLab application and GitLab Pages. This can also happen when a single -container is running both the core GitLab application and GitLab Pages. - -AWS [recommends using an IP target type](https://aws.amazon.com/premiumsupport/knowledge-center/target-connection-fails-load-balancer/) -to resolve this issue. - -Turning off [client IP preservation](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html#client-ip-preservation) -may resolve this issue when the core GitLab application and GitLab Pages run on the same host or -container. - -### 500 error with `securecookie: failed to generate random iv` and `Failed to save the session` - -This problem most likely results from an [out-dated operating system](../package_information/supported_os.md#os-versions-that-are-no-longer-supported). -The [Pages daemon uses the `securecookie` library](https://gitlab.com/search?group_id=9970&project_id=734943&repository_ref=master&scope=blobs&search=securecookie&snippets=false) to get random strings via [`crypto/rand` in Go](https://pkg.go.dev/crypto/rand#pkg-variables). -This requires the `getrandom` system call or `/dev/urandom` to be available on the host OS. -Upgrading to an [officially supported operating system](https://about.gitlab.com/install/) is recommended. - -### The requested scope is invalid, malformed, or unknown - -This problem comes from the permissions of the GitLab Pages OAuth application. To fix it: - -1. On the top bar, select **Main menu > Admin**. -1. On the left sidebar, select **Applications > GitLab Pages**. -1. Edit the application. -1. Under **Scopes**, ensure that the `api` scope is selected. -1. Save your changes. - -When running a [separate Pages server](#running-gitlab-pages-on-a-separate-server), -this setting needs to be configured on the main GitLab server. - -### Workaround in case no wildcard DNS entry can be set - -If the wildcard DNS [prerequisite](#prerequisites) can't be met, you can still use GitLab Pages in a limited fashion: - -1. [Move](../../user/project/settings/index.md#transfer-a-project-to-another-namespace) - all projects you need to use Pages with into a single group namespace, for example `pages`. -1. Configure a [DNS entry](#dns-configuration) without the `*.`-wildcard, for example `pages.example.io`. -1. Configure `pages_external_url http://example.io/` in your `gitlab.rb` file. - Omit the group namespace here, because it automatically is prepended by GitLab. - -### Pages daemon fails with permission denied errors - -If `/tmp` is mounted with `noexec`, the Pages daemon fails to start with an error like: - -```plaintext -{"error":"fork/exec /gitlab-pages: permission denied","level":"fatal","msg":"could not create pages daemon","time":"2021-02-02T21:54:34Z"} -``` - -In this case, change `TMPDIR` to a location that is not mounted with `noexec`. Add the following to -`/etc/gitlab/gitlab.rb`: - -```ruby -gitlab_pages['env'] = {'TMPDIR' => '<new_tmp_path>'} -``` - -Once added, reconfigure with `sudo gitlab-ctl reconfigure` and restart GitLab with -`sudo gitlab-ctl restart`. - -### `The redirect URI included is not valid.` when using Pages Access Control - -You may see this error if `pages_external_url` was updated at some point of time. Verify the following: - -1. The **Callback URL**/Redirect URI in the GitLab Pages [System OAuth application](../../integration/oauth_provider.md#create-an-instance-wide-application) -is using the protocol (HTTP or HTTPS) that `pages_external_url` is configured to use. -1. The domain and path components of `Redirect URI` are valid: they should look like `projects.<pages_external_url>/auth`. - -### 500 error `cannot serve from disk` - -If you get a 500 response from Pages and encounter an error similar to: - -```plaintext -ERRO[0145] cannot serve from disk error="gitlab: disk access is disabled via enable-disk=false" project_id=27 source_path="file:///shared/pages/@hashed/67/06/670671cd97404156226e507973f2ab8330d3022ca96e0c93bdbdb320c41adcaf/pages_deployments/14/artifacts.zip" source_type=zip -``` - -It means that GitLab Rails is telling GitLab Pages to serve content from a location on disk, -however, GitLab Pages was configured to disable disk access. - -To enable disk access: - -1. Enable disk access for GitLab Pages in `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_pages['enable_disk'] = true - ``` - -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - -### `httprange: new resource 403` - -If you see an error similar to: - -```plaintext -{"error":"httprange: new resource 403: \"403 Forbidden\"","host":"root.pages.example.com","level":"error","msg":"vfs.Root","path":"/pages1/","time":"2021-06-10T08:45:19Z"} -``` - -And you run pages on the separate server syncing files via NFS, it may mean that -the shared pages directory is mounted on a different path on the main GitLab server and the -GitLab Pages server. - -In that case, it's highly recommended you to configure -[object storage and migrate any existing pages data to it](#using-object-storage). - -Alternatively, you can mount the GitLab Pages shared directory to the same path on -both servers. - -### GitLab Pages doesn't work after upgrading to GitLab 14.0 or above - -GitLab 14.0 introduces a number of changes to GitLab Pages which may require manual intervention. - -1. Firstly [follow the migration guide](#prepare-gitlab-pages-for-140). -1. Try to upgrade to GitLab 14.3 or above. Some of the issues were fixed in GitLab 14.1, 14.2 and 14.3. -1. If it doesn't work, see [GitLab Pages logs](#how-to-see-gitlab-pages-logs), and if you see any errors there then search them on this page. - -WARNING: -In GitLab 14.0-14.2 you can temporarily enable legacy storage and configuration mechanisms. - -To do that: - -1. Describe the issue you're seeing in the [migration feedback issue](https://gitlab.com/gitlab-org/gitlab/-/issues/331699). - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_pages['use_legacy_storage'] = true - ``` - -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - -### GitLab Pages deploy job fails with error "is not a recognized provider" - -If the **pages** job succeeds but the **deploy** job gives the error "is not a recognized provider": - - - -The error message `is not a recognized provider` could be coming from the `fog` gem that GitLab uses to connect to cloud providers for object storage. - -To fix that: - -1. Check your `gitlab.rb` file. If you have `gitlab_rails['pages_object_store_enabled']` enabled, but no bucket details have been configured, either: - - - Configure object storage for your Pages deployments, following the [S3-compatible connection settings](#s3-compatible-connection-settings) guide. - - Store your deployments locally, by commenting out that line. - -1. Save the changes you made to your `gitlab.rb` file, then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - -### 404 error `The page you're looking for could not be found` - -If you get a `404 Page Not Found` response from GitLab Pages: - -1. Check `.gitlab-ci.yml` contains the job `pages:`. -1. Check the current project's pipeline to confirm the job `pages:deploy` is being run. +## Related topics -Without the `pages:deploy` job, the updates to your GitLab Pages site are never published. +- [Troubleshooting GitLab Pages administration](troubleshooting.md) diff --git a/doc/administration/pages/source.md b/doc/administration/pages/source.md index db76d15ec58..457e92d96bd 100644 --- a/doc/administration/pages/source.md +++ b/doc/administration/pages/source.md @@ -1,6 +1,6 @@ --- -stage: Create -group: Editor +stage: Plan +group: Knowledge info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/administration/pages/troubleshooting.md b/doc/administration/pages/troubleshooting.md new file mode 100644 index 00000000000..6f00ae074f5 --- /dev/null +++ b/doc/administration/pages/troubleshooting.md @@ -0,0 +1,304 @@ +--- +stage: Plan +group: Knowledge +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Troubleshooting GitLab Pages administration **(FREE SELF)** + +This page contains a list of issues you might encounter when administering GitLab Pages. + +## How to see GitLab Pages logs + +You can see Pages daemon logs by running: + +```shell +sudo gitlab-ctl tail gitlab-pages +``` + +You can also find the log file in `/var/log/gitlab/gitlab-pages/current`. + +## `unsupported protocol scheme \"\""` + +If you see the following error: + +```plaintext +{"error":"failed to connect to internal Pages API: Get \"/api/v4/internal/pages/status\": unsupported protocol scheme \"\"","level":"warning","msg":"attempted to connect to the API","time":"2021-06-23T20:03:30Z"} +``` + +It means you didn't set the HTTP(S) protocol scheme in the Pages server settings. +To fix it: + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_pages['gitlab_server'] = "https://<your_gitlab_server_public_host_and_port>" + gitlab_pages['internal_gitlab_server'] = "https://<your_gitlab_server_private_host_and_port>" # optional, gitlab_pages['gitlab_server'] is used as default + ``` + +1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +## 502 error when connecting to GitLab Pages proxy when server does not listen over IPv6 + +In some cases, NGINX might default to using IPv6 to connect to the GitLab Pages +service even when the server does not listen over IPv6. You can identify when +this is happening if you see something similar to the log entry below in the +`gitlab_pages_error.log`: + +```plaintext +2020/02/24 16:32:05 [error] 112654#0: *4982804 connect() failed (111: Connection refused) while connecting to upstream, client: 123.123.123.123, server: ~^(?<group>.*)\.pages\.example\.com$, request: "GET /-/group/project/-/jobs/1234/artifacts/artifact.txt HTTP/1.1", upstream: "http://[::1]:8090//-/group/project/-/jobs/1234/artifacts/artifact.txt", host: "group.example.com" +``` + +To resolve this, set an explicit IP and port for the GitLab Pages `listen_proxy` setting +to define the explicit address that the GitLab Pages daemon should listen on: + +```ruby +gitlab_pages['listen_proxy'] = '127.0.0.1:8090' +``` + +## Intermittent 502 errors or after a few days + +If you run Pages on a system that uses `systemd` and +[`tmpfiles.d`](https://www.freedesktop.org/software/systemd/man/tmpfiles.d.html), +you may encounter intermittent 502 errors trying to serve Pages with an error similar to: + +```plaintext +dial tcp: lookup gitlab.example.com on [::1]:53: dial udp [::1]:53: connect: no route to host" +``` + +GitLab Pages creates a [bind mount](https://man7.org/linux/man-pages/man8/mount.8.html) +inside `/tmp/gitlab-pages-*` that includes files like `/etc/hosts`. +However, `systemd` may clean the `/tmp/` directory on a regular basis so the DNS +configuration may be lost. + +To stop `systemd` from cleaning the Pages related content: + +1. Tell `tmpfiles.d` to not remove the Pages `/tmp` directory: + + ```shell + echo 'x /tmp/gitlab-pages-*' >> /etc/tmpfiles.d/gitlab-pages-jail.conf + ``` + +1. Restart GitLab Pages: + + ```shell + sudo gitlab-ctl restart gitlab-pages + ``` + +## Unable to access GitLab Pages + +If you can't access your GitLab Pages (such as receiving `502 Bad Gateway` errors, or a login loop) +and in your Pages log shows this error: + +```plaintext +"error":"retrieval context done: context deadline exceeded","host":"root.docs-cit.otenet.gr","level":"error","msg":"could not fetch domain information from a source" +``` + +1. Add the following to `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_pages['internal_gitlab_server'] = 'http://localhost:8080' + ``` + +1. Restart GitLab Pages: + + ```shell + sudo gitlab-ctl restart gitlab-pages + ``` + +## Failed to connect to the internal GitLab API + +If you see the following error: + +```plaintext +ERRO[0010] Failed to connect to the internal GitLab API after 0.50s error="failed to connect to internal Pages API: HTTP status: 401" +``` + +If you are [Running GitLab Pages on a separate server](index.md#running-gitlab-pages-on-a-separate-server) +you must copy the `/etc/gitlab/gitlab-secrets.json` file +from the **GitLab server** to the **Pages server** after upgrading to GitLab 13.3, +as described in that section. + +Other reasons may include network connectivity issues between your +**GitLab server** and your **Pages server** such as firewall configurations or closed ports. +For example, if there is a connection timeout: + +```plaintext +error="failed to connect to internal Pages API: Get \"https://gitlab.example.com:3000/api/v4/internal/pages/status\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" +``` + +## Pages cannot communicate with an instance of the GitLab API + +If you use the default value for `domain_config_source=auto` and run multiple instances of GitLab +Pages, you may see intermittent 502 error responses while serving Pages content. You may also see +the following warning in the Pages logs: + +```plaintext +WARN[0010] Pages cannot communicate with an instance of the GitLab API. Please sync your gitlab-secrets.json file https://gitlab.com/gitlab-org/gitlab-pages/-/issues/535#workaround. error="pages endpoint unauthorized" +``` + +This can happen if your `gitlab-secrets.json` file is out of date between GitLab Rails and GitLab +Pages. Follow steps 8-10 of [Running GitLab Pages on a separate server](index.md#running-gitlab-pages-on-a-separate-server), +in all of your GitLab Pages instances. + +## Intermittent 502 errors when using an AWS Network Load Balancer and GitLab Pages + +Connections will time out when using a Network Load Balancer with client IP preservation enabled and [the request is looped back to the source server](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-troubleshooting.html#loopback-timeout). +This can happen to GitLab instances with multiple servers +running both the core GitLab application and GitLab Pages. This can also happen when a single +container is running both the core GitLab application and GitLab Pages. + +AWS [recommends using an IP target type](https://aws.amazon.com/premiumsupport/knowledge-center/target-connection-fails-load-balancer/) +to resolve this issue. + +Turning off [client IP preservation](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html#client-ip-preservation) +may resolve this issue when the core GitLab application and GitLab Pages run on the same host or +container. + +## 500 error with `securecookie: failed to generate random iv` and `Failed to save the session` + +This problem most likely results from an [out-dated operating system](../package_information/supported_os.md#os-versions-that-are-no-longer-supported). +The [Pages daemon uses the `securecookie` library](https://gitlab.com/search?group_id=9970&project_id=734943&repository_ref=master&scope=blobs&search=securecookie&snippets=false) to get random strings via [`crypto/rand` in Go](https://pkg.go.dev/crypto/rand#pkg-variables). +This requires the `getrandom` system call or `/dev/urandom` to be available on the host OS. +Upgrading to an [officially supported operating system](https://about.gitlab.com/install/) is recommended. + +## The requested scope is invalid, malformed, or unknown + +This problem comes from the permissions of the GitLab Pages OAuth application. To fix it: + +1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, select **Applications > GitLab Pages**. +1. Edit the application. +1. Under **Scopes**, ensure that the `api` scope is selected. +1. Save your changes. + +When running a [separate Pages server](index.md#running-gitlab-pages-on-a-separate-server), +this setting needs to be configured on the main GitLab server. + +## Workaround in case no wildcard DNS entry can be set + +If the wildcard DNS [prerequisite](index.md#prerequisites) can't be met, you can still use GitLab Pages in a limited fashion: + +1. [Move](../../user/project/settings/index.md#transfer-a-project-to-another-namespace) + all projects you need to use Pages with into a single group namespace, for example `pages`. +1. Configure a [DNS entry](index.md#dns-configuration) without the `*.`-wildcard, for example `pages.example.io`. +1. Configure `pages_external_url http://example.io/` in your `gitlab.rb` file. + Omit the group namespace here, because it automatically is prepended by GitLab. + +## Pages daemon fails with permission denied errors + +If `/tmp` is mounted with `noexec`, the Pages daemon fails to start with an error like: + +```plaintext +{"error":"fork/exec /gitlab-pages: permission denied","level":"fatal","msg":"could not create pages daemon","time":"2021-02-02T21:54:34Z"} +``` + +In this case, change `TMPDIR` to a location that is not mounted with `noexec`. Add the following to +`/etc/gitlab/gitlab.rb`: + +```ruby +gitlab_pages['env'] = {'TMPDIR' => '<new_tmp_path>'} +``` + +Once added, reconfigure with `sudo gitlab-ctl reconfigure` and restart GitLab with +`sudo gitlab-ctl restart`. + +## `The redirect URI included is not valid.` when using Pages Access Control + +You may see this error if `pages_external_url` was updated at some point of time. Verify the following: + +1. The **Callback URL**/Redirect URI in the GitLab Pages [System OAuth application](../../integration/oauth_provider.md#create-an-instance-wide-application) +is using the protocol (HTTP or HTTPS) that `pages_external_url` is configured to use. +1. The domain and path components of `Redirect URI` are valid: they should look like `projects.<pages_external_url>/auth`. + +## 500 error `cannot serve from disk` + +If you get a 500 response from Pages and encounter an error similar to: + +```plaintext +ERRO[0145] cannot serve from disk error="gitlab: disk access is disabled via enable-disk=false" project_id=27 source_path="file:///shared/pages/@hashed/67/06/670671cd97404156226e507973f2ab8330d3022ca96e0c93bdbdb320c41adcaf/pages_deployments/14/artifacts.zip" source_type=zip +``` + +It means that GitLab Rails is telling GitLab Pages to serve content from a location on disk, +however, GitLab Pages was configured to disable disk access. + +To enable disk access: + +1. Enable disk access for GitLab Pages in `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_pages['enable_disk'] = true + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +## `httprange: new resource 403` + +If you see an error similar to: + +```plaintext +{"error":"httprange: new resource 403: \"403 Forbidden\"","host":"root.pages.example.com","level":"error","msg":"vfs.Root","path":"/pages1/","time":"2021-06-10T08:45:19Z"} +``` + +And you run pages on the separate server syncing files via NFS, it may mean that +the shared pages directory is mounted on a different path on the main GitLab server and the +GitLab Pages server. + +In that case, it's highly recommended you to configure +[object storage and migrate any existing pages data to it](index.md#using-object-storage). + +Alternatively, you can mount the GitLab Pages shared directory to the same path on +both servers. + +## GitLab Pages doesn't work after upgrading to GitLab 14.0 or above + +GitLab 14.0 introduces a number of changes to GitLab Pages which may require manual intervention. + +1. Firstly [follow the migration guide](index.md#prepare-gitlab-pages-for-140). +1. Try to upgrade to GitLab 14.3 or above. Some of the issues were fixed in GitLab 14.1, 14.2 and 14.3. +1. If it doesn't work, see [GitLab Pages logs](#how-to-see-gitlab-pages-logs), and if you see any errors there then search them on this page. + +WARNING: +In GitLab 14.0-14.2 you can temporarily enable legacy storage and configuration mechanisms. + +To do that: + +1. Describe the issue you're seeing in the [migration feedback issue](https://gitlab.com/gitlab-org/gitlab/-/issues/331699). + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_pages['use_legacy_storage'] = true + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +## GitLab Pages deploy job fails with error "is not a recognized provider" + +If the **pages** job succeeds but the **deploy** job gives the error "is not a recognized provider": + + + +The error message `is not a recognized provider` could be coming from the `fog` gem that GitLab uses to connect to cloud providers for object storage. + +To fix that: + +1. Check your `gitlab.rb` file. If you have `gitlab_rails['pages_object_store_enabled']` enabled, but no bucket details have been configured, either: + + - Configure object storage for your Pages deployments, following the [S3-compatible connection settings](index.md#s3-compatible-connection-settings) guide. + - Store your deployments locally, by commenting out that line. + +1. Save the changes you made to your `gitlab.rb` file, then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +## 404 error `The page you're looking for could not be found` + +If you get a `404 Page Not Found` response from GitLab Pages: + +1. Check `.gitlab-ci.yml` contains the job `pages:`. +1. Check the current project's pipeline to confirm the job `pages:deploy` is being run. + +Without the `pages:deploy` job, the updates to your GitLab Pages site are never published. diff --git a/doc/administration/postgresql/database_load_balancing.md b/doc/administration/postgresql/database_load_balancing.md index 15129770888..d5cf93a135a 100644 --- a/doc/administration/postgresql/database_load_balancing.md +++ b/doc/administration/postgresql/database_load_balancing.md @@ -235,3 +235,8 @@ operation retries up to 3 times using an exponential back-off. When using load balancing, you should be able to safely restart a database server without it immediately leading to errors being presented to the users. + +### Development guide + +For detailed development guide on database load balancing, +see [the development documentation](../../development/database/load_balancing.md). diff --git a/doc/administration/postgresql/multiple_databases.md b/doc/administration/postgresql/multiple_databases.md index 188578a739c..857fd4fc9c5 100644 --- a/doc/administration/postgresql/multiple_databases.md +++ b/doc/administration/postgresql/multiple_databases.md @@ -15,7 +15,7 @@ By default, GitLab uses a single application database, referred to as the `main` To scale GitLab, you can configure GitLab to use multiple application databases. -Due to [known issues](#known-issues), configuring GitLab with multiple databases is in [**Alpha**](../../policy/alpha-beta-support.md#alpha-features). +Due to [known issues](#known-issues), configuring GitLab with multiple databases is an [Experiment](../../policy/alpha-beta-support.md#experiment). After you have set up multiple databases, GitLab uses a second application database for [CI/CD features](../../ci/index.md), referred to as the `ci` database. diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index 46890b0b2ca..8ac3ac40a75 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -1255,8 +1255,6 @@ To do the switch on **all** PgBouncer nodes: ``` 1. Run `gitlab-ctl reconfigure`. -1. You must also run `rm /var/opt/gitlab/consul/config.d/watcher_postgresql.json`. - This is a [known issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7293). #### Clean up diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md index 5c258d73fdb..055cf944046 100644 --- a/doc/administration/raketasks/maintenance.md +++ b/doc/administration/raketasks/maintenance.md @@ -11,7 +11,7 @@ GitLab provides Rake tasks for general maintenance. ## Gather GitLab and system information This command gathers information about your GitLab installation and the system it runs on. -These may be useful when asking for help or reporting issues. +These may be useful when asking for help or reporting issues. In a multi-node environment, run this command on nodes running GitLab Rails to avoid PostgreSQL socket errors. **Omnibus Installation** @@ -117,7 +117,7 @@ If you're running Geo, see also the [Geo Health check Rake task](../geo/replicat You may also have a look at our troubleshooting guides for: -- [GitLab](../index.md#troubleshooting) +- [GitLab](../troubleshooting/index.md) - [Omnibus GitLab](https://docs.gitlab.com/omnibus/index.html#troubleshooting) Additionally you should also [verify database values can be decrypted using the current secrets](check.md#verify-database-values-can-be-decrypted-using-the-current-secrets). diff --git a/doc/administration/raketasks/service_desk_email.md b/doc/administration/raketasks/service_desk_email.md index 10de379b1cd..1cbdec35171 100644 --- a/doc/administration/raketasks/service_desk_email.md +++ b/doc/administration/raketasks/service_desk_email.md @@ -12,7 +12,7 @@ The following are Service Desk email-related Rake tasks. ## Secrets -GitLab can use [Service Desk email](../../user/project/service_desk.md#configuring-a-custom-mailbox) secrets read from an encrypted file instead of storing them in plaintext in the file system. The following Rake tasks are provided for updating the contents of the encrypted file. +GitLab can use [Service Desk email](../../user/project/service_desk.md#configure-a-custom-mailbox) secrets read from an encrypted file instead of storing them in plaintext in the file system. The following Rake tasks are provided for updating the contents of the encrypted file. ### Show secret diff --git a/doc/administration/redis/troubleshooting.md b/doc/administration/redis/troubleshooting.md index 29407f65efd..fbfe52570c4 100644 --- a/doc/administration/redis/troubleshooting.md +++ b/doc/administration/redis/troubleshooting.md @@ -113,15 +113,19 @@ To make sure your configuration is correct: redis.info ``` - Keep this screen open and try to simulate a failover below. + Keep this screen open, and proceed to trigger a failover as described below. -1. To simulate a failover on primary Redis, SSH into the Redis server and run: +1. To trigger a failover on the primary Redis, SSH into the Redis server and run: ```shell # port must match your primary redis port, and the sleep time must be a few seconds bigger than defined one redis-cli -h localhost -p 6379 DEBUG sleep 20 ``` + WARNING: + This action affects services, and takes the instance down for up to 20 seconds. If successful, + it should recover after that. + 1. Then back in the Rails console from the first step, run: ```ruby diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 1934ca6bff0..e70d012507f 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -31,8 +31,8 @@ full list of reference architectures, see | Gitaly<sup>5 6</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | | Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | -| GitLab Rails | 3 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | +| Sidekiq<sup>7</sup> | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| GitLab Rails<sup>7</sup> | 3 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | | Object storage<sup>4</sup> | - | - | - | - | @@ -53,6 +53,8 @@ full list of reference architectures, see 6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to [Large repositories](index.md#large-repositories) for more information. +7. Can be placed in Auto Scaling Groups (ASGs) as the component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes). + However, for GitLab Rails certain processes like [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md) should be run on only one node. <!-- markdownlint-enable MD029 --> NOTE: @@ -417,6 +419,11 @@ spread connections equally in practice. ## Configure Consul +Next, we set up the Consul servers. + +NOTE: +Consul must be deployed in an odd number of 3 nodes or higher. This is to ensure the nodes can take votes as part of a quorum. + The following IPs will be used as an example: - `10.6.0.11`: Consul 1 @@ -801,6 +808,9 @@ Using [Redis](https://redis.io/) in scalable environment is possible using a **P topology with a [Redis Sentinel](https://redis.io/docs/manual/sentinel/) service to watch and automatically start the failover procedure. +NOTE: +Redis clusters must each be deployed in an odd number of 3 nodes or higher. This is to ensure Redis Sentinel can take votes as part of a quorum. This does not apply when configuring Redis externally, such as a cloud provider service. + Redis requires authentication if used with Sentinel. See [Redis Security](https://redis.io/docs/manual/security/) documentation for more information. We recommend using a combination of a Redis password and tight @@ -1336,6 +1346,9 @@ This is how this would work with a Omnibus GitLab PostgreSQL setup: Praefect is the router and transaction manager for Gitaly Cluster and all connections to Gitaly go through it. This section details how to configure it. +NOTE: +Consul must be deployed in an odd number of 3 nodes or higher. This is to ensure the nodes can take votes as part of a quorum. + Praefect requires several secret tokens to secure communications across the Cluster: - `<praefect_external_token>`: Used for repositories hosted on your Gitaly cluster and can only be accessed by Gitaly clients that carry this token. @@ -1510,9 +1523,7 @@ Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs should have a throughput of at least 8,000 input/output operations per second (IOPS) for read operations and 2,000 IOPS for -write operations. These IOPS values are initial recommendations, and may be -adjusted to greater or lesser values depending on the scale of your -environment's workload. If you're running the environment on a Cloud provider, +write operations. If you're running the environment on a Cloud provider, refer to their documentation about how to configure IOPS correctly. Gitaly servers must not be exposed to the public internet, as Gitaly's network @@ -2089,11 +2100,9 @@ the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html). sudo gitlab-rake gitlab:db:configure ``` - If you encounter a `rake aborted!` error message stating that PgBouncer is - failing to connect to PostgreSQL, it may be that your PgBouncer node's IP - address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` - on your database nodes. Before proceeding, see - [PgBouncer error `ERROR: pgbouncer cannot connect to server`](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). + Note that this requires the Rails node to be configured to connect to the primary database + directly, [bypassing PgBouncer](../postgresql/pgbouncer.md#procedure-for-bypassing-pgbouncer). + After migrations have completed, you must configure the node to pass through PgBouncer again. 1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). @@ -2189,15 +2198,15 @@ and scalable. There are two ways of specifying object storage configuration in GitLab: -- [Consolidated form](../object_storage.md#consolidated-object-storage-configuration): A single credential is +- [Consolidated form](../object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form): A single credential is shared by all supported object types. -- [Storage-specific form](../object_storage.md#storage-specific-configuration): Every object defines its +- [Storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form): Every object defines its own object storage [connection and configuration](../object_storage.md#connection-settings). The consolidated form is used in the following examples when available. NOTE: -When using the [storage-specific form](../object_storage.md#storage-specific-configuration) +When using the [storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form) in GitLab 14.x and earlier, you should enable [direct upload mode](../../development/uploads/index.md#direct-upload). The previous [background upload](../../development/uploads/index.md#direct-upload) mode, which was deprecated in 14.9, requires shared storage such as NFS. diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 08cb6e2cdff..19a79657218 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -31,8 +31,8 @@ full list of reference architectures, see | Gitaly<sup>5 6</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | | Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | -| GitLab Rails | 5 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | +| Sidekiq<sup>7</sup> | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| GitLab Rails<sup>7</sup> | 5 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | | Object storage<sup>4</sup> | - | - | - | - | @@ -53,6 +53,8 @@ full list of reference architectures, see 6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to [Large repositories](index.md#large-repositories) for more information. +7. Can be placed in Auto Scaling Groups (ASGs) as the component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes). + However, for GitLab Rails certain processes like [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md) should be run on only one node. <!-- markdownlint-enable MD029 --> NOTE: @@ -434,6 +436,11 @@ spread connections equally in practice. ## Configure Consul +Next, we set up the Consul servers. + +NOTE: +Consul must be deployed in an odd number of 3 nodes or higher. This is to ensure the nodes can take votes as part of a quorum. + The following IPs will be used as an example: - `10.6.0.11`: Consul 1 @@ -818,6 +825,9 @@ Using [Redis](https://redis.io/) in scalable environment is possible using a **P topology with a [Redis Sentinel](https://redis.io/docs/manual/sentinel/) service to watch and automatically start the failover procedure. +NOTE: +Redis clusters must each be deployed in an odd number of 3 nodes or higher. This is to ensure Redis Sentinel can take votes as part of a quorum. This does not apply when configuring Redis externally, such as a cloud provider service. + Redis requires authentication if used with Sentinel. See [Redis Security](https://redis.io/docs/manual/security/) documentation for more information. We recommend using a combination of a Redis password and tight @@ -1353,6 +1363,9 @@ This is how this would work with a Omnibus GitLab PostgreSQL setup: Praefect is the router and transaction manager for Gitaly Cluster and all connections to Gitaly go through it. This section details how to configure it. +NOTE: +Praefect must be deployed in an odd number of 3 nodes or higher. This is to ensure the nodes can take votes as part of a quorum. + Praefect requires several secret tokens to secure communications across the Cluster: - `<praefect_external_token>`: Used for repositories hosted on your Gitaly cluster and can only be accessed by Gitaly clients that carry this token. @@ -1527,9 +1540,7 @@ Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs should have a throughput of at least 8,000 input/output operations per second (IOPS) for read operations and 2,000 IOPS for -write operations. These IOPS values are initial recommendations, and may be -adjusted to greater or lesser values depending on the scale of your -environment's workload. If you're running the environment on a Cloud provider, +write operations. If you're running the environment on a Cloud provider, refer to their documentation about how to configure IOPS correctly. Gitaly servers must not be exposed to the public internet, as Gitaly's network @@ -2108,11 +2119,9 @@ the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html). sudo gitlab-rake gitlab:db:configure ``` - If you encounter a `rake aborted!` error message stating that PgBouncer is - failing to connect to PostgreSQL, it may be that your PgBouncer node's IP - address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` - on your database nodes. Before proceeding, see - [PgBouncer error `ERROR: pgbouncer cannot connect to server`](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). + Note that this requires the Rails node to be configured to connect to the primary database + directly, [bypassing PgBouncer](../postgresql/pgbouncer.md#procedure-for-bypassing-pgbouncer). + After migrations have completed, you must configure the node to pass through PgBouncer again. 1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). @@ -2207,15 +2216,15 @@ and scalable. There are two ways of specifying object storage configuration in GitLab: -- [Consolidated form](../object_storage.md#consolidated-object-storage-configuration): A single credential is +- [Consolidated form](../object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form): A single credential is shared by all supported object types. -- [Storage-specific form](../object_storage.md#storage-specific-configuration): Every object defines its +- [Storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form): Every object defines its own object storage [connection and configuration](../object_storage.md#connection-settings). The consolidated form is used in the following examples when available. NOTE: -When using the [storage-specific form](../object_storage.md#storage-specific-configuration) +When using the [storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form) in GitLab 14.x and earlier, you should enable [direct upload mode](../../development/uploads/index.md#direct-upload). The previous [background upload](../../development/uploads/index.md#direct-upload) mode, which was deprecated in 14.9, requires shared storage such as NFS. diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index d87e8270dcb..f31cf6f4ab6 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -26,7 +26,7 @@ For a full list of reference architectures, see | PostgreSQL<sup>1</sup> | 1 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | | Redis<sup>2</sup> | 1 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `m5.large` | `D2s v3` | | Gitaly<sup>5</sup> | 1 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| GitLab Rails | 2 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | +| GitLab Rails<sup>6</sup> | 2 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | | Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Object storage<sup>4</sup> | - | - | - | - | - | @@ -45,6 +45,8 @@ For a full list of reference architectures, see 5. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to [Large repositories](index.md#large-repositories) for more information. +6. Can be placed in Auto Scaling Groups (ASGs) as the component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes). + However, for GitLab Rails certain processes like [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md) should be run on only one node. <!-- markdownlint-enable MD029 --> NOTE: @@ -423,9 +425,7 @@ Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs should have a throughput of at least 8,000 input/output operations per second (IOPS) for read operations and 2,000 IOPS -for write operations. These IOPS values are initial recommendations, and may be -adjusted to greater or lesser values depending on the scale of your -environment's workload. If you're running the environment on a Cloud provider, +for write operations. If you're running the environment on a Cloud provider, refer to their documentation about how to configure IOPS correctly. Be sure to note the following items: @@ -774,11 +774,9 @@ the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html). sudo gitlab-rake gitlab:db:configure ``` - If you encounter a `rake aborted!` error message stating that PgBouncer is - failing to connect to PostgreSQL, it may be that your PgBouncer node's IP - address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` - on your database nodes. Before proceeding, see - [PgBouncer error `ERROR: pgbouncer cannot connect to server`](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). + Note that this requires the Rails node to be configured to connect to the primary database + directly, [bypassing PgBouncer](../postgresql/pgbouncer.md#procedure-for-bypassing-pgbouncer). + After migrations have completed, you must configure the node to pass through PgBouncer again. 1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). @@ -904,15 +902,15 @@ and scalable. There are two ways of specifying object storage configuration in GitLab: -- [Consolidated form](../object_storage.md#consolidated-object-storage-configuration): A single credential is +- [Consolidated form](../object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form): A single credential is shared by all supported object types. -- [Storage-specific form](../object_storage.md#storage-specific-configuration): Every object defines its +- [Storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form): Every object defines its own object storage [connection and configuration](../object_storage.md#connection-settings). The consolidated form is used in the following examples when available. NOTE: -When using the [storage-specific form](../object_storage.md#storage-specific-configuration) +When using the [storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form) in GitLab 14.x and earlier, you should enable [direct upload mode](../../development/uploads/index.md#direct-upload). The previous [background upload](../../development/uploads/index.md#direct-upload) mode, which was deprecated in 14.9, requires shared storage such as NFS. diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index c28799ed459..a01eaee10d3 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -40,8 +40,8 @@ For a full list of reference architectures, see | Gitaly<sup>5 6</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | | Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | -| GitLab Rails | 3 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | +| Sidekiq<sup>7</sup> | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | +| GitLab Rails<sup>7</sup> | 3 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | | Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Object storage<sup>4</sup> | - | - | - | - | @@ -62,6 +62,8 @@ For a full list of reference architectures, see 6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to [Large repositories](index.md#large-repositories) for more information. +7. Can be placed in Auto Scaling Groups (ASGs) as the component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes). + However, for GitLab Rails certain processes like [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md) should be run on only one node. <!-- markdownlint-enable MD029 --> NOTE: @@ -640,8 +642,13 @@ are supported and can be added if needed. ## Configure Consul and Sentinel -Now that the Redis servers are all set up, let's configure the Sentinel -servers. The following IPs will be used as an example: +Now that the Redis servers are all set up, let's configure the Consul + Sentinel +servers. + +NOTE: +Consul and Redis Sentinel must be deployed in an odd number of 3 nodes or higher. This is to ensure the nodes can take votes as part of a quorum. + +The following IPs will be used as an example: - `10.6.0.11`: Consul/Sentinel 1 - `10.6.0.12`: Consul/Sentinel 2 @@ -1288,6 +1295,9 @@ This is how this would work with a Omnibus GitLab PostgreSQL setup: Praefect is the router and transaction manager for Gitaly Cluster and all connections to Gitaly go through it. This section details how to configure it. +NOTE: +Praefect must be deployed in an odd number of 3 nodes or higher. This is to ensure the nodes can take votes as part of a quorum. + Praefect requires several secret tokens to secure communications across the Cluster: - `<praefect_external_token>`: Used for repositories hosted on your Gitaly cluster and can only be accessed by Gitaly clients that carry this token. @@ -1462,9 +1472,7 @@ Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs should have a throughput of at least 8,000 input/output operations per second (IOPS) for read operations and 2,000 IOPS for -write operations. These IOPS values are initial recommendations, and may be -adjusted to greater or lesser values depending on the scale of your -environment's workload. If you're running the environment on a Cloud provider, +write operations. If you're running the environment on a Cloud provider, refer to their documentation about how to configure IOPS correctly. Gitaly servers must not be exposed to the public internet, as Gitaly's network @@ -2073,11 +2081,9 @@ the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html). gitlab-rake gitlab:db:configure ``` - If you encounter a `rake aborted!` error message stating that PgBouncer is - failing to connect to PostgreSQL, it may be that your PgBouncer node's IP - address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` - on your database nodes. Before proceeding, see - [PgBouncer error `ERROR: pgbouncer cannot connect to server`](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). + Note that this requires the Rails node to be configured to connect to the primary database + directly, [bypassing PgBouncer](../postgresql/pgbouncer.md#procedure-for-bypassing-pgbouncer). + After migrations have completed, you must configure the node to pass through PgBouncer again. 1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). @@ -2180,15 +2186,15 @@ and scalable. There are two ways of specifying object storage configuration in GitLab: -- [Consolidated form](../object_storage.md#consolidated-object-storage-configuration): A single credential is +- [Consolidated form](../object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form): A single credential is shared by all supported object types. -- [Storage-specific form](../object_storage.md#storage-specific-configuration): Every object defines its +- [Storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form): Every object defines its own object storage [connection and configuration](../object_storage.md#connection-settings). The consolidated form is used in the following examples when available. NOTE: -When using the [storage-specific form](../object_storage.md#storage-specific-configuration) +When using the [storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form) in GitLab 14.x and earlier, you should enable [direct upload mode](../../development/uploads/index.md#direct-upload). The previous [background upload](../../development/uploads/index.md#direct-upload) mode, which was deprecated in 14.9, requires shared storage such as NFS. diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 3f2771dda29..d4423ef4f9f 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -31,8 +31,8 @@ full list of reference architectures, see | Gitaly<sup>5 6</sup> | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | `m5.16xlarge` | | Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | -| GitLab Rails | 12 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | +| Sidekiq<sup>7</sup> | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| GitLab Rails<sup>7</sup> | 12 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | | Object storage<sup>4</sup> | - | - | - | - | @@ -53,6 +53,8 @@ full list of reference architectures, see 6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to [Large repositories](index.md#large-repositories) for more information. +7. Can be placed in Auto Scaling Groups (ASGs) as the component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes). + However, for GitLab Rails certain processes like [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md) should be run on only one node. <!-- markdownlint-enable MD029 --> NOTE: @@ -426,6 +428,11 @@ spread connections equally in practice. ## Configure Consul +Next, we set up the Consul servers. + +NOTE: +Consul must be deployed in an odd number of 3 nodes or higher. This is to ensure the nodes can take votes as part of a quorum. + The following IPs will be used as an example: - `10.6.0.11`: Consul 1 @@ -811,6 +818,9 @@ Using [Redis](https://redis.io/) in scalable environment is possible using a **P topology with a [Redis Sentinel](https://redis.io/docs/manual/sentinel/) service to watch and automatically start the failover procedure. +NOTE: +Redis clusters must each be deployed in an odd number of 3 nodes or higher. This is to ensure Redis Sentinel can take votes as part of a quorum. This does not apply when configuring Redis externally, such as a cloud provider service. + Redis requires authentication if used with Sentinel. See [Redis Security](https://redis.io/docs/manual/security/) documentation for more information. We recommend using a combination of a Redis password and tight @@ -1349,6 +1359,9 @@ This is how this would work with a Omnibus GitLab PostgreSQL setup: Praefect is the router and transaction manager for Gitaly Cluster and all connections to Gitaly go through it. This section details how to configure it. +NOTE: +Praefect must be deployed in an odd number of 3 nodes or higher. This is to ensure the nodes can take votes as part of a quorum. + Praefect requires several secret tokens to secure communications across the Cluster: - `<praefect_external_token>`: Used for repositories hosted on your Gitaly cluster and can only be accessed by Gitaly clients that carry this token. @@ -1523,9 +1536,7 @@ Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs should have a throughput of at least 8,000 input/output operations per second (IOPS) for read operations and 2,000 IOPS for -write operations. These IOPS values are initial recommendations, and may be -adjusted to greater or lesser values depending on the scale of your -environment's workload. If you're running the environment on a Cloud provider, +write operations. If you're running the environment on a Cloud provider, refer to their documentation about how to configure IOPS correctly. Gitaly servers must not be exposed to the public internet, as Gitaly's network @@ -2107,11 +2118,9 @@ the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html). sudo gitlab-rake gitlab:db:configure ``` - If you encounter a `rake aborted!` error message stating that PgBouncer is - failing to connect to PostgreSQL, it may be that your PgBouncer node's IP - address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` - on your database nodes. Before proceeding, see - [PgBouncer error `ERROR: pgbouncer cannot connect to server`](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). + Note that this requires the Rails node to be configured to connect to the primary database + directly, [bypassing PgBouncer](../postgresql/pgbouncer.md#procedure-for-bypassing-pgbouncer). + After migrations have completed, you must configure the node to pass through PgBouncer again. 1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). @@ -2206,15 +2215,15 @@ and scalable. There are two ways of specifying object storage configuration in GitLab: -- [Consolidated form](../object_storage.md#consolidated-object-storage-configuration): A single credential is +- [Consolidated form](../object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form): A single credential is shared by all supported object types. -- [Storage-specific form](../object_storage.md#storage-specific-configuration): Every object defines its +- [Storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form): Every object defines its own object storage [connection and configuration](../object_storage.md#connection-settings). The consolidated form is used in the following examples when available. NOTE: -When using the [storage-specific form](../object_storage.md#storage-specific-configuration) +When using the [storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form) in GitLab 14.x and earlier, you should enable [direct upload mode](../../development/uploads/index.md#direct-upload). The previous [background upload](../../development/uploads/index.md#direct-upload) mode, which was deprecated in 14.9, requires shared storage such as NFS. diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 43d63d4f08e..627812f5647 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -37,8 +37,8 @@ costly-to-operate environment by using the | Gitaly<sup>5 6</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | | Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | -| GitLab Rails | 3 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | `c5.4xlarge` | +| Sidekiq<sup>7</sup> | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | +| GitLab Rails<sup>7</sup> | 3 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | `c5.4xlarge` | | Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Object storage<sup>4</sup> | - | - | - | - | @@ -59,6 +59,8 @@ costly-to-operate environment by using the 6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to [Large repositories](index.md#large-repositories) for more information. +7. Can be placed in Auto Scaling Groups (ASGs) as the component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes). + However, for GitLab Rails certain processes like [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md) should be run on only one node. <!-- markdownlint-enable MD029 --> NOTE: @@ -637,8 +639,13 @@ are supported and can be added if needed. ## Configure Consul and Sentinel -Now that the Redis servers are all set up, let's configure the Sentinel -servers. The following IPs are used as an example: +Now that the Redis servers are all set up, let's configure the Consul + Sentinel +servers. + +NOTE: +Consul and Redis Sentinel must be deployed in an odd number of 3 nodes or higher. This is to ensure the nodes can take votes as part of a quorum. + +The following IPs will be used as an example: - `10.6.0.11`: Consul/Sentinel 1 - `10.6.0.12`: Consul/Sentinel 2 @@ -1285,6 +1292,9 @@ This is how this would work with a Omnibus GitLab PostgreSQL setup: Praefect is the router and transaction manager for Gitaly Cluster and all connections to Gitaly go through it. This section details how to configure it. +NOTE: +Praefect must be deployed in an odd number of 3 nodes or higher. This is to ensure the nodes can take votes as part of a quorum. + Praefect requires several secret tokens to secure communications across the Cluster: - `<praefect_external_token>`: Used for repositories hosted on your Gitaly cluster and can only be accessed by Gitaly clients that carry this token. @@ -1459,9 +1469,7 @@ Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs should have a throughput of at least 8,000 input/output operations per second (IOPS) for read operations and 2,000 IOPS for -write operations. These IOPS values are initial recommendations, and may be -adjusted to greater or lesser values depending on the scale of your -environment's workload. If you're running the environment on a Cloud provider, +write operations. If you're running the environment on a Cloud provider, refer to their documentation about how to configure IOPS correctly. Gitaly servers must not be exposed to the public internet, as Gitaly's network @@ -2068,11 +2076,9 @@ the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html). gitlab-rake gitlab:db:configure ``` - If you encounter a `rake aborted!` error message stating that PgBouncer is - failing to connect to PostgreSQL, it may be that your PgBouncer node's IP - address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` - on your database nodes. Before proceeding, see - [PgBouncer error `ERROR: pgbouncer cannot connect to server`](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). + Note that this requires the Rails node to be configured to connect to the primary database + directly, [bypassing PgBouncer](../postgresql/pgbouncer.md#procedure-for-bypassing-pgbouncer). + After migrations have completed, you must configure the node to pass through PgBouncer again. 1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). @@ -2175,15 +2181,15 @@ and scalable. There are two ways of specifying object storage configuration in GitLab: -- [Consolidated form](../object_storage.md#consolidated-object-storage-configuration): A single credential is +- [Consolidated form](../object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form): A single credential is shared by all supported object types. -- [Storage-specific form](../object_storage.md#storage-specific-configuration): Every object defines its +- [Storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form): Every object defines its own object storage [connection and configuration](../object_storage.md#connection-settings). The consolidated form is used in the following examples when available. NOTE: -When using the [storage-specific form](../object_storage.md#storage-specific-configuration) +When using the [storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form) in GitLab 14.x and earlier, you should enable [direct upload mode](../../development/uploads/index.md#direct-upload). The previous [background upload](../../development/uploads/index.md#direct-upload) mode, which was deprecated in 14.9, requires shared storage such as NFS. diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index e4d07558306..f40c8fc3c67 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -47,9 +47,9 @@ The following Cloud Native Hybrid reference architectures, where select recommen The first choice to consider is whether a Self Managed approach is correct for you and your requirements. -Running any application in production is complex, and the same applies for GitLab. While we aim to make this as smooth as possible, there are still the general complexities. This depends on the design chosen, but typically you'll need to manage all aspects such as hardware, operating systems, networking, storage, security, GitLab itself, and more. +Running any application in production is complex, and the same applies for GitLab. While we aim to make this as smooth as possible, there are still the general complexities. This depends on the design chosen, but typically you'll need to manage all aspects such as hardware, operating systems, networking, storage, security, GitLab itself, and more. This includes both the initial setup of the environment and the longer term maintenance. -As such, it's recommended that you have a working knowledge of running applications in production when deciding on going down this route. For users who want a more managed solution it's recommended to instead explore our other offerings such as [GitLab SaaS](../../subscriptions/gitlab_com/index.md) or [GitLab Dedicated](../../subscriptions/gitlab_dedicated/index.md). +As such, it's recommended that you have a working knowledge of running and maintaining applications in production when deciding on going down this route. If you aren't in this position, our [Professional Services](https://about.gitlab.com/services/#implementation-services) team offers implementation services, but for those who want a more managed solution long term, it's recommended to instead explore our other offerings such as [GitLab SaaS](../../subscriptions/gitlab_com/index.md) or [GitLab Dedicated](../../subscriptions/gitlab_dedicated/index.md). ## Deciding which architecture to use @@ -72,15 +72,13 @@ With standalone setups, especially single node environments, there are [various ### High Availability (HA) -High Availability ensures every component in the GitLab setup can handle failures through various mechanisms. To achieve this however is complex, and the environments required can be sizable. +High Availability ensures every component in the GitLab setup can handle failures through various mechanisms. However, to achieve this is complex and the environments required can be sizable. For environments serving 3,000 or more users we generally recommend that a HA strategy is used as at this level outages have a bigger impact against more users. All the architectures in this range have HA built in by design for this reason. -For users who still need to have HA for a lower number of users this can also be achieved with an adjusted [3K architecture](3k_users.md#supported-modifications-for-lower-user-counts-ha). - #### Do you need High Availability (HA)? -As mentioned above, achieving HA does come at a cost. The environment's required are sizable as each component needs to be multiplied, which comes with additional actual and maintenance costs. +As mentioned above, achieving HA does come at a cost. The environment requirements are sizable as each component needs to be multiplied, which comes with additional actual and maintenance costs. For a lot of our customers with fewer than 3,000 users, we've found a backup strategy is sufficient and even preferable. While this does have a slower recovery time, it also means you have a much smaller architecture and less maintenance costs as a result. @@ -89,13 +87,17 @@ In general then, we'd only recommend you employ HA in the following scenarios: - When you have 3,000 or more users. - When GitLab being down would critically impact your workflow. +#### Scaled-down High Availability (HA) approaches + +If you still need to have HA for a lower number of users, this can be achieved with an adjusted [3K architecture](3k_users.md#supported-modifications-for-lower-user-counts-ha). + #### Zero Downtime Upgrades [Zero Downtime Upgrades](../../update/zero_downtime.md) are available for standard Reference Architecture environments with HA (Cloud Native Hybrid is not supported at this time). This allows for an environment to stay up during an upgrade, but the process is more complex as a result and has some limitations as detailed in the documentation. -When going through this process it's worth noting that there may still be brief moments of downtime when the HA mechanisms tale effect. +When going through this process it's worth noting that there may still be brief moments of downtime when the HA mechanisms take effect. -In most cases the downtime required for doing an upgrade in general shouldn't be substantial, so this is only recommended if it's a key requirement for you. +In most cases the downtime required for doing an upgrade shouldn't be substantial, so this is only recommended if it's a key requirement for you. ### Cloud Native Hybrid (Kubernetes HA) @@ -163,7 +165,7 @@ Before implementing a reference architecture, refer to the following requirement These reference architectures were built and tested on Google Cloud Platform (GCP) using the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) -CPU platform as a baseline ([Sysbench benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks)). +CPU platform as a lowest common denominator baseline ([Sysbench benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks)). Newer, similarly-sized CPUs are supported and may have improved performance as a result. For Omnibus GitLab environments, ARM-based equivalents are also supported. @@ -237,8 +239,8 @@ for more information and guidance. [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and that to achieve full High Availability, a third-party PostgreSQL database solution is required. -We hope to offer a built in solutions for these restrictions in the future but, in the meantime, a non HA PostgreSQL server -can be set up using Omnibus GitLab, the specifications reflect. Refer to the following issues for more information: +We hope to offer a built-in solution for these restrictions in the future. In the meantime, a non-HA PostgreSQL server +can be set up using Omnibus GitLab as the specifications reflect. Refer to the following issues for more information: - [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919). - [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398). @@ -324,10 +326,12 @@ When selecting a database service, it should run a standard, performant, and [su - Read Replicas for [Database Load Balancing](../postgresql/database_load_balancing.md). - Cross Region replication for [GitLab Geo](../geo/index.md). -Several cloud provider services are known not to support the above or have been found to have other issues and aren't recommended: +#### Unsupported database services + +Several database cloud provider services are known not to support the above or have been found to have other issues and aren't recommended: - [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is incompatible and not supported. See [14.4.0](../../update/index.md#1440) for more details. -- [Azure Database for PostgreSQL Single Server](https://azure.microsoft.com/en-gb/products/postgresql/#overview) (Single / Flexible) is **strongly not recommended** for use due to notable performance / stability issues or missing functionality. See [Recommendation Notes for Azure](#recommendation-notes-for-azure) for more details. +- [Azure Database for PostgreSQL Single Server](https://azure.microsoft.com/en-gb/products/postgresql/#overview) (Single / Flexible) is not supported for use due to notable performance / stability issues or missing functionality. See [Recommendation Notes for Azure](#recommendation-notes-for-azure) for more details. - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo. - [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported. @@ -337,10 +341,55 @@ Due to performance issues that we found with several key Azure services, we only In addition to the above, you should be aware of the additional specific guidance for Azure: -- **We outright strongly do not recommend [Azure Database for PostgreSQL Single Server](https://learn.microsoft.com/en-us/azure/postgresql/single-server/overview-single-server)** specifically due to significant performance and stability issues found. **For GitLab 14.0 and higher the service is not supported** due to it only supporting up to PostgreSQL 11. - - A new service, [Azure Database for PostgreSQL Flexible Server](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/) has been released. [Internal testing](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/91) has shown that it does look to perform as expected, but this hasn't been validated in production, so generally isn't recommended at this time. Additionally, as it's a new service, you may find that it's missing some functionality depending on your requirements. +- [Azure Database for PostgreSQL Single Server](https://azure.microsoft.com/en-gb/products/postgresql/#overview) (Single / Flexible) is not supported for use due to notable performance / stability issues or missing functionality. +- A new service, [Azure Database for PostgreSQL Flexible Server](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/) has been released. [Internal testing](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/91) has shown that it does look to perform as expected, but this hasn't been validated in production, so generally isn't recommended at this time. Additionally, as it's a new service, you may find that it's missing some functionality depending on your requirements. - [Azure Blob Storage](https://azure.microsoft.com/en-gb/products/storage/blobs/) has been found to have [performance limits that can impact production use at certain times](https://gitlab.com/gitlab-org/gitlab/-/issues/344861). However, this has only been seen in our largest architectures (25k+) so far. +## Deviating from the suggested reference architectures + +As a general guideline, the further away you move from the reference architectures, +the harder it is to get support for it. With any deviation, you're introducing +a layer of complexity that adds challenges to finding out where potential +issues might lie. + +The reference architectures use the official GitLab Linux packages (Omnibus +GitLab) or [Helm Charts](https://docs.gitlab.com/charts/) to install and configure the various components. The components are +installed on separate machines (virtualized or bare metal), with machine hardware +requirements listed in the "Configuration" column and equivalent VM standard sizes listed +in GCP/AWS/Azure columns of each [available reference architecture](#available-reference-architectures). + +Running components on Docker (including Docker Compose) with the same specs should be fine, as Docker is well known in terms of support. +However, it is still an additional layer and may still add some support complexities, such as not being able to run `strace` easily in containers. + +### Unsupported designs + +While we endeavour to try and have a good range of support for GitLab environment designs, there are certain approaches we know definitively not to work, and as a result are not supported. Those approaches are detailed in the following sections. + +#### Stateful components in Kubernetes + +[Running stateful components in Kubernetes, such as Gitaly Cluster, is not supported](https://docs.gitlab.com/charts/installation/#configure-the-helm-chart-to-use-external-stateful-data). + +Gitaly Cluster is only supported to be run on VMs as Git itself doesn't match well with the Kubernetes design and attempting to run it can lead to significant and complex issues. +[Refer to epic 6127 for more information](https://gitlab.com/groups/gitlab-org/-/epics/6127). + +This also applies to other third-party stateful components such as Postgres and Redis, but you can explore other third-party solutions for those components if desired such as supported Cloud Provider services unless called out specifically as unsupported. + +#### Autoscaling of stateful nodes + +As a general guidance, only _stateless_ components of GitLab can be run in Autoscaling groups, namely GitLab Rails +and Sidekiq. + +Other components that have state, such as Gitaly, are not supported in this fashion (for more information, see [issue 2997](https://gitlab.com/gitlab-org/gitaly/-/issues/2997)). + +This also applies to other third-party stateful components such as Postgres and Redis, but you can explore other third-party solutions for those components if desired such as supported Cloud Provider services unless called out specifically as unsupported. + +#### Spreading one environment over multiple data centers + +Deploying one GitLab environment over multiple data centers is not supported due to potential split brain edge cases +if a data center were to go down. For example, several components of the GitLab setup, namely Consul, Redis Sentinel and Praefect require an odd number quorum to function correctly and splitting over multiple data centers can impact this notably. + +For deploying GitLab over multiple data centers or regions we offer [GitLab Geo](https://about.gitlab.com/solutions/geo/) as a comprehensive solution. + ## Validation and test results The [Quality Engineering team](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/) @@ -427,34 +476,34 @@ table.test-coverage th { <tr> <th scope="row">1k</th> <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/1k">Weekly</a></td> - <td></td> - <td></td> - <td></td> - <td></td> + <td style="background-color:lightgrey"></td> + <td style="background-color:lightgrey"></td> + <td style="background-color:lightgrey"></td> + <td style="background-color:lightgrey"></td> </tr> <tr> <th scope="row">2k</th> <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/2k">Weekly</a></td> - <td></td> - <td></td> - <td></td> - <td></td> + <td style="background-color:lightgrey"></td> + <td style="background-color:lightgrey"></td> + <td style="background-color:lightgrey"></td> + <td><i>Planned</i></td> </tr> <tr> <th scope="row">3k</th> <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/3k">Weekly</a></td> - <td></td> - <td></td> + <td style="background-color:lightgrey"></td> + <td style="background-color:lightgrey"></td> <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/3k_hybrid_aws_services">Weekly</a></td> - <td></td> + <td style="background-color:lightgrey"></td> </tr> <tr> <th scope="row">5k</th> <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/5k">Weekly</a></td> - <td></td> - <td></td> - <td></td> - <td></td> + <td style="background-color:lightgrey"></td> + <td style="background-color:lightgrey"></td> + <td style="background-color:lightgrey"></td> + <td style="background-color:lightgrey"></td> </tr> <tr> <th scope="row">10k</th> @@ -462,29 +511,33 @@ table.test-coverage th { <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k_hybrid">Weekly</a></td> <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k_aws">Weekly</a></td> <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k_hybrid_aws_services">Weekly</a></td> - <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/10k">Ad-Hoc</a></td> + <td style="background-color:lightgrey"></td> </tr> <tr> <th scope="row">25k</th> <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/25k">Weekly</a></td> - <td></td> - <td></td> - <td></td> - <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/25k">Ad-Hoc</a></td> + <td style="background-color:lightgrey"></td> + <td style="background-color:lightgrey"></td> + <td style="background-color:lightgrey"></td> + <td style="background-color:lightgrey"></td> </tr> <tr> <th scope="row">50k</th> <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/50k">Weekly</a></td> - <td></td> - <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/50k">Ad-Hoc (inc Cloud Services)</a></td> - <td></td> - <td></td> + <td style="background-color:lightgrey"></td> + <td style="background-color:lightgrey"></td> + <td style="background-color:lightgrey"></td> + <td style="background-color:lightgrey"></td> </tr> </table> ## Cost to run -The following table details the cost to run the different reference architectures across GCP, AWS, and Azure. Bare-metal costs are not included here as it varies widely depending on each customer configuration. +As a starting point, the following table details initial costs for the different reference architectures across GCP, AWS, and Azure via Omnibus. + +NOTE: +Due to the nature of Cloud Native Hybrid, it's not possible to give a static cost calculation. +Bare-metal costs are also not included here as it varies widely depending on each configuration. <table class="test-coverage"> <col> @@ -492,91 +545,93 @@ The following table details the cost to run the different reference architecture <colgroup span="2"></colgroup> <tr> <th rowspan="2">Reference<br/>Architecture</th> - <th style="text-align: center" colspan="2" scope="colgroup">GCP</th> - <th style="text-align: center" colspan="2" scope="colgroup">AWS</th> - <th style="text-align: center" colspan="2" scope="colgroup">Azure</th> + <th style="text-align: center" scope="colgroup">GCP</th> + <th style="text-align: center" scope="colgroup">AWS</th> + <th style="text-align: center" scope="colgroup">Azure</th> </tr> <tr> <th scope="col">Omnibus</th> - <th scope="col">Cloud Native Hybrid</th> <th scope="col">Omnibus</th> - <th scope="col">Cloud Native Hybrid</th> <th scope="col">Omnibus</th> </tr> <tr> <th scope="row">1k</th> <td><a href="https://cloud.google.com/products/calculator#id=a6d6a94a-c7dc-4c22-85c4-7c5747f272ed">Calculated cost</a></td> - <td></td> <td><a href="https://calculator.aws/#/estimate?id=b51f178f4403b69a63f6eb33ea425f82de3bf249">Calculated cost</a></td> - <td></td> <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=1adf30bef7e34ceba9efa97c4470417b">Calculated cost</a></td> </tr> <tr> <th scope="row">2k</th> <td><a href="https://cloud.google.com/products/calculator#id=0d3aff1f-ea3d-43f9-aa59-df49d27c35ca">Calculated cost</a></td> - <td></td> <td><a href="https://calculator.aws/#/estimate?id=3b3e3b81953737132789591d3a5179521943f1c0">Calculated cost</a></td> - <td></td> <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=25f66c35ba454bb98fb4034a8a50bb8c">Calculated cost</a></td> </tr> <tr> <th scope="row">3k</th> <td><a href="https://cloud.google.com/products/calculator/#id=15fc2bd9-5b1c-479d-bc46-d5ce096b8107">Calculated cost</a></td> - <td></td> <td><a href="https://calculator.aws/#/estimate?id=7e94eb8712f6845fdeb05e61f459598a91dac3cb">Calculated cost</a></td> - <td></td> <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=24ac11fd947a4985ae9c9a5142649ad3">Calculated cost</a></td> </tr> <tr> <th scope="row">5k</th> <td><a href="https://cloud.google.com/products/calculator/#id=9a798136-53f2-4c35-be43-8e1e975a6663">Calculated cost</a></td> - <td></td> <td><a href="https://calculator.aws/#/estimate?id=ad4c9db623a214c92d780cd9dff33f444d62cf02">Calculated cost</a></td> - <td></td> <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=bcf23017ddfd40649fdc885cacd08d0c">Calculated cost</a></td> </tr> <tr> <th scope="row">10k</th> <td><a href="https://cloud.google.com/products/calculator#id=cbe61840-31a1-487f-88fa-631251c2fde5">Calculated cost</a></td> - <td></td> <td><a href="https://calculator.aws/#/estimate?id=3e2970f919915a6337acea76a9f07655a1ecda4a">Calculated cost</a></td> - <td></td> <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=5748068be4864af6a34efb1cde685fa1">Calculated cost</a></td> </tr> <tr> <th scope="row">25k</th> <td><a href="https://cloud.google.com/products/calculator#id=b4b8b587-508a-4433-adc8-dc506bbe924f">Calculated cost</a></td> - <td></td> <td><a href="https://calculator.aws/#/estimate?id=32acaeaa93366110cd5fbf98a66a8a141db7adcb">Calculated cost</a></td> - <td></td> <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=24f878f20ee64b5cb64de459d34c8128">Calculate cost</a></td> </tr> <tr> <th scope="row">50k</th> <td><a href="https://cloud.google.com/products/calculator/#id=48b4d817-d6cd-44b8-b069-0ba9a5d123ea">Calculated cost</a></td> - <td></td> <td><a href="https://calculator.aws/#/estimate?id=5a0bba1338e3577d627ec97833dbc80ac9615562">Calculated cost</a></td> - <td></td> <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=4dd065eea2194d70b44d6d897e81f460">Calculated cost</a></td> </tr> </table> -## Deviating from the suggested reference architectures +## Maintaining a Reference Architecture environment -As a general guideline, the further away you move from the Reference Architectures, -the harder it is to get support for it. With any deviation, you're introducing -a layer of complexity that adds challenges to finding out where potential -issues might lie. +Maintaining a Reference Architecture environment is generally the same as any other GitLab environment is generally covered in other sections of this documentation. -The reference architectures use the official GitLab Linux packages (Omnibus -GitLab) or [Helm Charts](https://docs.gitlab.com/charts/) to install and configure the various components. The components are -installed on separate machines (virtualized or bare metal), with machine hardware -requirements listed in the "Configuration" column and equivalent VM standard sizes listed -in GCP/AWS/Azure columns of each [available reference architecture](#available-reference-architectures). +In this section you'll find links to documentation for relevant areas as well as any specific Reference Architecture notes. -Running components on Docker (including Compose) with the same specs should be fine, as Docker is well known in terms of support. -However, it is still an additional layer and may still add some support complexities, such as not being able to run `strace` easily in containers. +### Upgrades + +Upgrades for a Reference Architecture environment is the same as any other GitLab environment. +The main [Upgrade GitLab](../../update/index.md) section has detailed steps on how to approach this. + +[Zero-downtime upgrades](#zero-downtime-upgrades) are also available. + +NOTE: +You should upgrade a Reference Architecture in the same order as you created it. + +### Scaling an environment + +Scaling a GitLab environment is designed to be as seamless as possible. + +In terms of the Reference Architectures, you would look to the next size and adjust accordingly. +Most setups would only need vertical scaling, but there are some specific areas that can be adjusted depending on the setup: + +- If you're scaling from a non-HA environment to an HA environment, various components are recommended to be deployed in their HA forms: + - Redis to multi-node Redis w/ Redis Sentinel + - Postgres to multi-node Postgres w/ Consul + PgBouncer + - Gitaly to Gitaly Cluster w/ Praefect +- From 10k users and higher, Redis is recommended to be split into multiple HA servers as it's single threaded. + +Conversely, if you have robust metrics in place that show the environment is over-provisioned you can apply the same process for +scaling downloads. It's recommended to take an iterative approach when scaling downwards however to ensure there are no issues. + +### How to monitor your environment -Other technologies, like [Docker swarm](https://docs.docker.com/engine/swarm/) -are not officially supported, but can be implemented at your own risk. In that -case, GitLab Support is not able to help you. +To monitor your GitLab environment, you can use the tools +[bundled with GitLab](../monitoring/index.md), but it's also possible to use third-party +options if desired. diff --git a/doc/administration/restart_gitlab.md b/doc/administration/restart_gitlab.md index 7996db3d1e1..e86541b7ced 100644 --- a/doc/administration/restart_gitlab.md +++ b/doc/administration/restart_gitlab.md @@ -126,3 +126,17 @@ kubectl delete pods -l release=<helm release name>,app=<component name> ``` The release name can be obtained from the output of the `helm list` command. + +## Docker installation + +If you change the configuration on your [Docker installation](../install/docker.md), for that change to take effect you must restart: + +- The main `gitlab` container. +- Any separate component containers. + +For example, if you deployed Sidekiq on a separate container, to restart the containers, run: + +```shell +sudo docker restart gitlab +sudo docker restart sidekiq +``` diff --git a/doc/administration/server_hooks.md b/doc/administration/server_hooks.md index 87ae6fdb003..5e9863447a1 100644 --- a/doc/administration/server_hooks.md +++ b/doc/administration/server_hooks.md @@ -2,7 +2,6 @@ stage: Systems group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -disqus_identifier: 'https://docs.gitlab.com/ee/administration/custom_hooks.html' --- # Git server hooks **(FREE SELF)** @@ -28,7 +27,43 @@ alternatives to server hooks include: [Geo](geo/index.md) doesn't replicate server hooks to secondary nodes. -## Create server hooks for a repository +## Set server hooks for a repository + +::Tabs + +:::TabTitle GitLab 15.11 and later + +> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4629) in GitLab 15.11, `hooks set` command replaces direct file system access. + +Prerequisites: + +- The [storage name and relative path](repository_storage_types.md#from-project-name-to-hashed-path) for the repository. + +To set server hooks for a repository: + +1. Create tarball containing custom hooks: + 1. Write the code to make the server hook function as expected. Git server hooks can be in any programming language. + Ensure the [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) at the top reflects the language type. For + example, if the script is in Ruby the shebang is probably `#!/usr/bin/env ruby`. + + - To create a single server hook, create a file with a name that matches the hook type. For example, for a + `pre-receive` server hook, the filename should be `pre-receive` with no extension. + - To create many server hooks, create a directory for the hooks that matches the hook type. For example, for a + `pre-receive` server hook, the directory name should be `pre-receive.d`. Put the files for the hook in that + directory. + + 1. Ensure the server hook files are executable and do not match the backup file pattern (`*~`). The server hooks be + in a `custom_hooks` directory that is at the root of the tarball. + 1. Create the custom hooks archive with the tar command. For example, `tar -cf custom_hooks.tar custom_hooks`. +1. Run the `hooks set` command with required options to set the Git hooks for the repository. For example, + `cat hooks.tar | gitaly hooks set --storage <storage> --repository <relative path> --config <config path>`. + + - A path to a valid Gitaly configuration for the node is required to connect to the node and provided to the `--config` flag. + - Custom hooks tarball must be passed via `stdin`. For example, `cat hooks.tar | gitaly hooks set --storage <storage> --repository <relative path> --config <config path>`. + +If you implemented the server hook code correctly, it should execute when the Git hook is next triggered. + +:::TabTitle GitLab 15.10 and earlier To create server hooks for a repository: @@ -56,6 +91,8 @@ To create server hooks for a repository: If the server hook code is properly implemented, it should execute when the Git hook is next triggered. +::EndTabs + ### Gitaly Cluster If you use [Gitaly Cluster](gitaly/index.md), the scripts must be copied to every Gitaly node that has a replica of the repository. Every Gitaly node @@ -113,6 +150,33 @@ To create a global server hook for all repositories: If the server hook code is properly implemented, it should execute when the Git hook is next triggered. Hooks are executed in alphabetical order by filename in the hook type subdirectories. +## Remove server hooks for a repository + +::Tabs + +:::TabTitle GitLab 15.11 and later + +> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4629) in GitLab 15.11, `hooks set` command replaces direct file system access. + +Prerequisites: + +- The [storage name and relative path](repository_storage_types.md#from-project-name-to-hashed-path) for the repository. + +To remove server hooks, pass an empty tarball to `hook set` to indicate that the repository should contain no hooks. For example: + +```shell +cat empty_hooks.tar | gitaly hooks set --storage <storage> --repository <relative path> --config <config path>`. +``` + +:::TabTitle GitLab 15.10 and earlier + +To remove server hooks: + +1. Go to the location of the repository on disk. +1. Delete the server hooks in the `custom_hooks` directory. + +::EndTabs + ## Chained server hooks GitLab can execute server hooks in a chain. GitLab searches for and executes server hooks in the following order: diff --git a/doc/administration/sidekiq/index.md b/doc/administration/sidekiq/index.md index 8645df55a62..6fb12aa6ef9 100644 --- a/doc/administration/sidekiq/index.md +++ b/doc/administration/sidekiq/index.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w You can configure an external Sidekiq instance by using the Sidekiq that's bundled in the GitLab package. Sidekiq requires connection to the Redis, PostgreSQL, and Gitaly instances. -## Configure TCP access for PostgreSQL, Gitaly, and Redis +## Configure TCP access for PostgreSQL, Gitaly, and Redis on the GitLab instance By default, GitLab uses UNIX sockets and is not set up to communicate via TCP. To change this: @@ -56,7 +56,6 @@ By default, GitLab uses UNIX sockets and is not set up to communicate via TCP. T redis['password'] = 'redis-password-goes-here' gitlab_rails['redis_password'] = 'redis-password-goes-here' - gitlab_rails['auto_migrate'] = false ``` 1. Run `reconfigure`: @@ -71,18 +70,6 @@ By default, GitLab uses UNIX sockets and is not set up to communicate via TCP. T sudo gitlab-ctl restart postgresql ``` -1. After the restart, set `auto_migrate` to `true` or comment to use the default settings: - - ```ruby - gitlab_rails['auto_migrate'] = true - ``` - -1. Run `reconfigure` again: - - ```shell - sudo gitlab-ctl reconfigure - ``` - ## Set up Sidekiq instance 1. SSH into the Sidekiq server. diff --git a/doc/administration/sidekiq/processing_specific_job_classes.md b/doc/administration/sidekiq/processing_specific_job_classes.md index 0e4a0662277..26c2804f130 100644 --- a/doc/administration/sidekiq/processing_specific_job_classes.md +++ b/doc/administration/sidekiq/processing_specific_job_classes.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: These are advanced settings. While they are used on GitLab.com, most GitLab -instances should add more processes that all listen to all queues. This is the +instances should only add more processes that listen to all queues. This is the same approach we take in our [Reference Architectures](../reference_architectures/index.md). GitLab has two options for creating Sidekiq processes that only handle specific diff --git a/doc/administration/silent_mode/index.md b/doc/administration/silent_mode/index.md new file mode 100644 index 00000000000..f07f1dde323 --- /dev/null +++ b/doc/administration/silent_mode/index.md @@ -0,0 +1,64 @@ +--- +stage: Systems +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# GitLab Silent Mode (Alpha) **(FREE SELF)** + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/9826) in GitLab 15.11. This feature is an [Experiment](../../policy/alpha-beta-support.md#experiment). + +Silent Mode allows you to suppress outbound communication, such as emails, from GitLab. Silent Mode is not intended to be used on environments which are in-use. Two use-cases are: + +- Validating Geo site promotion. You have a secondary Geo site as part of your [disaster recovery](../geo/disaster_recovery/index.md) solution. You want to regularly test promoting it to become a primary Geo site, as a best practice to ensure your disaster recovery plan actually works. But you don't want to actually perform an entire failover, since the primary site lives in a region which provides the lowest latency to your users. And you don't want to take downtime during every regular test. So, you let the primary site remain up, while you promote the secondary site. You start smoke testing the promoted site. But, the promoted site starts emailing users, the push mirrors push changes to external Git repositories, etc. This is where Silent Mode comes in. You can enable it as part of site promotion, to avoid this issue. +- Validating GitLab backups. You set up a testing instance to test that your backups restore successfully. As part of the restore, you enable Silent Mode, for example to avoid sending invalid emails to users. + +## Enable Silent Mode + +Prerequisites: + +- You must have administrator access. + +There are two ways to enable Silent Mode: + +- [**API**](../../api/settings.md): + + ```shell + curl --request PUT --header "PRIVATE-TOKEN:$ADMIN_TOKEN" "<gitlab-url>/api/v4/application/settings?silent_mode_enabled=true" + ``` + +- [**Rails console**](../operations/rails_console.md#starting-a-rails-console-session): + + ```ruby + ::Gitlab::CurrentSettings.update!(silent_mode_enabled: true) + ``` + +## Disable Silent Mode + +Prerequisites: + +- You must have administrator access. + +There are two ways to disable Silent Mode: + +- [**API**](../../api/settings.md): + + ```shell + curl --request PUT --header "PRIVATE-TOKEN:$ADMIN_TOKEN" "<gitlab-url>/api/v4/application/settings?silent_mode_enabled=false" + ``` + +- [**Rails console**](../operations/rails_console.md#starting-a-rails-console-session): + + ```ruby + ::Gitlab::CurrentSettings.update!(silent_mode_enabled: false) + ``` + +## Behavior of GitLab features in Silent Mode + +### Service Desk + +Incoming emails still raise issues, but the users who sent the emails to [Service Desk](../../user/project/service_desk.md) are not notified of issue creation or comments on their issues. + +### Outbound emails + +Outbound emails are suppressed. It may take up to a minute to take effect after enabling Silent Mode. [Issue 405433](https://gitlab.com/gitlab-org/gitlab/-/issues/405433) proposes removing this delay. diff --git a/doc/administration/static_objects_external_storage.md b/doc/administration/static_objects_external_storage.md index ef89e38be6f..73f44ed3889 100644 --- a/doc/administration/static_objects_external_storage.md +++ b/doc/administration/static_objects_external_storage.md @@ -1,6 +1,6 @@ --- stage: Create -group: Editor +group: IDE info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" type: reference --- diff --git a/doc/administration/terraform_state.md b/doc/administration/terraform_state.md index 12d88c7c74a..9a54eb86eeb 100644 --- a/doc/administration/terraform_state.md +++ b/doc/administration/terraform_state.md @@ -1,6 +1,6 @@ --- -stage: Configure -group: Configure +stage: Deploy +group: Environments info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -175,7 +175,7 @@ sudo find /var/opt/gitlab/gitlab-rails/shared/terraform_state -type f | grep -v ### S3-compatible connection settings In GitLab 13.2 and later, you should use the -[consolidated object storage settings](object_storage.md#consolidated-object-storage-configuration). +[consolidated object storage settings](object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form). This section describes the earlier configuration format. See [the available connection settings for different providers](object_storage.md#connection-settings). diff --git a/doc/administration/troubleshooting/index.md b/doc/administration/troubleshooting/index.md index db572f202ea..8be6abc180d 100644 --- a/doc/administration/troubleshooting/index.md +++ b/doc/administration/troubleshooting/index.md @@ -27,3 +27,28 @@ that you can check for feature-specific help, including helpful Rails commands. If you need a testing environment to troubleshoot, see the [apps for a testing environment](test_environments.md). + +## Support team troubleshooting info + +The GitLab Support Team has collected a lot of information about troubleshooting GitLab. +The following documents are used by the Support Team or by customers +with direct guidance from a Support Team member. GitLab administrators may find the +information useful for troubleshooting. However, if you are experiencing trouble with your +GitLab instance, you should check your [support options](https://about.gitlab.com/support/) +before referring to these documents. + +WARNING: +The commands in the following documentation might result in data loss or +other damage to a GitLab instance. They should be used only by experienced administrators +who are aware of the risks. + +- [Diagnostics tools](diagnostics_tools.md) +- [Linux commands](linux_cheat_sheet.md) +- [Troubleshooting Kubernetes](https://docs.gitlab.com/charts/troubleshooting/kubernetes_cheat_sheet.html) +- [Troubleshooting PostgreSQL](postgresql.md) +- [Guide to test environments](test_environments.md) (for Support Engineers) +- [Troubleshooting SSL](ssl.md) +- Related links: + - [Repairing and recovering broken Git repositories](https://git.seveas.net/repairing-and-recovering-broken-git-repositories.html) + - [Testing with OpenSSL](https://www.feistyduck.com/library/openssl-cookbook/online/testing-with-openssl/index.html) + - [`strace` zine](https://wizardzines.com/zines/strace/) diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index c4e2c352e9b..e614f0d38e3 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -65,7 +65,7 @@ This configuration relies on valid AWS credentials to be configured already. ### Object Storage Settings In GitLab 13.2 and later, you should use the -[consolidated object storage settings](object_storage.md#consolidated-object-storage-configuration). +[consolidated object storage settings](object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form). This section describes the earlier configuration format. For source installations the following settings are nested under `uploads:` and then `object_store:`. On Omnibus GitLab installs they are prefixed by `uploads_object_store_`. diff --git a/doc/administration/wikis/index.md b/doc/administration/wikis/index.md index f3442e23160..330e41ee880 100644 --- a/doc/administration/wikis/index.md +++ b/doc/administration/wikis/index.md @@ -1,7 +1,7 @@ --- type: reference, howto -stage: Create -group: Editor +stage: Plan +group: Knowledge info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- |
