diff options
Diffstat (limited to 'doc/administration')
49 files changed, 1679 insertions, 786 deletions
diff --git a/doc/administration/audit_event_streaming.md b/doc/administration/audit_event_streaming.md index 4d0e6518ebb..92ccbe8b1a6 100644 --- a/doc/administration/audit_event_streaming.md +++ b/doc/administration/audit_event_streaming.md @@ -373,7 +373,7 @@ Streamed audit events have a predictable schema in the body of the response. > - [Added `details.author_class` field](https://gitlab.com/gitlab-org/gitlab/-/issues/363876) in GitLab 15.3. > - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/101583) in GitLab 15.6. Feature flag `audit_event_streaming_git_operations` removed. -Streaming audit events can be sent when signed-in users push, pull, or clone a project's remote Git repositories: +Streaming audit events can be sent when authenticated users push, pull, or clone a project's remote Git repositories: - [Using SSH](../user/ssh.md). - Using HTTP or HTTPS. diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index 1951ab5e2c7..30e6ca44972 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -200,7 +200,7 @@ The following actions on groups generate group audit events: [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/353452) in GitLab 14.9. - Failed attempt to create a group deploy token. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/353452) in GitLab 14.9. -- [IP restrictions](../user/group/access_and_permissions.md#restrict-access-to-groups-by-ip-address) changed. +- [IP restrictions](../user/group/access_and_permissions.md#restrict-group-access-by-ip-address) changed. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/358986) in GitLab 15.0. - Changes to push rules. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/227629) in GitLab 15.0. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/356152) in GitLab 15.1, changes to the following merge diff --git a/doc/administration/auditor_users.md b/doc/administration/auditor_users.md index a047e7c1f0b..38c3a089756 100644 --- a/doc/administration/auditor_users.md +++ b/doc/administration/auditor_users.md @@ -55,6 +55,7 @@ If you are signed in with auditor access, you: you can push commits or comment on issues. - Can access the same resources using the GitLab UI or API. - Can't view the Admin Area, or perform any administration actions. +- Can't view job logs when [debug logging](../ci/variables/index.md#enable-debug-logging) is enabled. ## Maintain auditor users using API diff --git a/doc/administration/auth/ldap/google_secure_ldap.md b/doc/administration/auth/ldap/google_secure_ldap.md index 01197fdacdf..e0612099221 100644 --- a/doc/administration/auth/ldap/google_secure_ldap.md +++ b/doc/administration/auth/ldap/google_secure_ldap.md @@ -24,33 +24,33 @@ The steps below cover: 1. Go to **Apps > LDAP > Add Client**. -1. Provide an `LDAP client name` and an optional `Description`. Any descriptive - values are acceptable. For example, the name could be 'GitLab' and the - description could be 'GitLab LDAP Client'. Select the **Continue** button. +1. Provide an **LDAP client name** and an optional **Description**. Any descriptive + values are acceptable. For example, the name could be `GitLab` and the + description could be `GitLab LDAP Client`. Select **Continue**.  1. Set **Access Permission** according to your needs. You must choose either - 'Entire domain (GitLab)' or 'Selected organizational units' for both 'Verify user - credentials' and 'Read user information'. Select 'Add LDAP Client' + `Entire domain (GitLab)` or `Selected organizational units` for both **Verify user + credentials** and **Read user information**. Select **Add LDAP Client**. NOTE: If you plan to use GitLab [LDAP Group Sync](ldap_synchronization.md#group-sync) - , turn on 'Read group information'. + , turn on `Read group information`.  1. Download the generated certificate. This is required for GitLab to communicate with the Google Secure LDAP service. Save the downloaded certificates - for later use. After downloading, select the **Continue to Client Details** button. + for later use. After downloading, select **Continue to Client Details**. -1. Expand the **Service Status** section and turn the LDAP client 'ON for everyone'. - After selecting 'Save', select the 'Service Status' bar again to collapse +1. Expand the **Service Status** section and turn the LDAP client `ON for everyone`. + After selecting **Save**, select the **Service Status** bar again to collapse and return to the rest of the settings. -1. Expand the **Authentication** section and choose 'Generate New Credentials'. - Copy/note these credentials for later use. After selecting 'Close', select - the 'Authentication' bar again to collapse and return to the rest of the settings. +1. Expand the **Authentication** section and choose **Generate New Credentials**. + Copy/note these credentials for later use. After selecting **Close**, select + the **Authentication** bar again to collapse and return to the rest of the settings. Now the Google Secure LDAP Client configuration is finished. The screenshot below shows an example of the final settings. Continue on to configure GitLab. diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index 2cb9bac7af9..43d13b8ea32 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -69,93 +69,137 @@ You should only use LDAP integration if your LDAP users cannot: ## Configure LDAP -To configure LDAP integration, add your LDAP server settings in: +LDAP users must have a set email address, regardless of whether or not it's used +to sign in. -- `/etc/gitlab/gitlab.rb` for Omnibus GitLab instances. -- `/home/git/gitlab/config/gitlab.yml` for source install instances. +Here's an example of setting up LDAP with only the required options. -After configuring LDAP, to test the configuration, use the -[LDAP check Rake task](../../raketasks/ldap.md#check). +::Tabs -NOTE: -The `encryption` value `simple_tls` corresponds to 'Simple TLS' in the LDAP -library. `start_tls` corresponds to StartTLS, not to be confused with regular TLS. -Normally, if you specify `simple_tls` it is on port 636, while `start_tls` (StartTLS) -would be on port 389. `plain` also operates on port 389. Removed values: `tls` was replaced -with `start_tls` and `ssl` was replaced with `simple_tls`. +:::TabTitle Linux package (Omnibus) -LDAP users must have a set email address, regardless of whether or not it's used -to sign in. +1. Edit `/etc/gitlab/gitlab.rb`: -### Example Linux package (Omnibus) configuration - -This example shows a sample configuration for a GitLab instance that -was installed by using the Linux package (Omnibus): - -```ruby -gitlab_rails['ldap_enabled'] = true -gitlab_rails['prevent_ldap_sign_in'] = false -gitlab_rails['ldap_servers'] = { - 'main' => { - 'label' => 'LDAP', - 'host' => 'ldap.mydomain.com', - 'port' => 636, - 'uid' => 'sAMAccountName', - 'encryption' => 'simple_tls', - 'verify_certificates' => true, - 'bind_dn' => '_the_full_dn_of_the_user_you_will_bind_with', - 'password' => '_the_password_of_the_bind_user', - 'tls_options' => { - 'ca_file' => '', - 'ssl_version' => '', - 'ciphers' => '', - 'cert' => '', - 'key' => '' - }, - 'timeout' => 10, - 'active_directory' => true, - 'allow_username_or_email_login' => false, - 'block_auto_created_users' => false, - 'base' => 'dc=example,dc=com', - 'user_filter' => '', - 'attributes' => { - 'username' => ['uid', 'userid', 'sAMAccountName'], - 'email' => ['mail', 'email', 'userPrincipalName'], - 'name' => 'cn', - 'first_name' => 'givenName', - 'last_name' => 'sn' - }, - 'lowercase_usernames' => false, - - # EE Only - 'group_base' => '', - 'admin_group' => '', - 'external_groups' => [], - 'sync_ssh_keys' => false - } -} -``` + ```ruby + gitlab_rails['ldap_enabled'] = true + gitlab_rails['ldap_servers'] = { + 'main' => { + 'label' => 'LDAP', + 'host' => 'ldap.mydomain.com', + 'port' => 636, + 'uid' => 'sAMAccountName', + 'encryption' => 'simple_tls', + 'base' => 'dc=example,dc=com', + } + } + ``` + +1. Save the file and reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` -### Example Helm chart (Kubernetes) configuration +:::TabTitle Helm chart (Kubernetes) -View [how to configure LDAP for a GitLab instance that was installed by using the Helm chart](https://docs.gitlab.com/charts/charts/globals.html#ldap). +1. Export the Helm values: -### Example self-compiled (source) configuration + ```shell + helm get values gitlab > gitlab_values.yaml + ``` -This example shows a sample configuration for a GitLab instance that -was installed by using the self-compiled source: +1. Edit `gitlab_values.yaml`: -```yaml -production: - # snip... - ldap: - enabled: false - prevent_ldap_sign_in: false - servers: - main: - label: 'LDAP' - ... -``` + ```yaml + global: + appConfig: + ldap: + servers: + main: + label: 'LDAP' + host: 'ldap.mydomain.com' + port: 636 + uid: 'sAMAccountName' + base: 'dc=example,dc=com' + encryption: 'simple_tls' + ``` + +1. Save the file and apply the new values: + + ```shell + helm upgrade -f gitlab_values.yaml gitlab gitlab/gitlab + ``` + +For more information, see +[how to configure LDAP for a GitLab instance that was installed by using the Helm chart](https://docs.gitlab.com/charts/charts/globals.html#ldap). + +:::TabTitle Docker + +1. Edit `docker-compose.yml`: + + ```yaml + version: "3.6" + services: + gitlab: + image: 'gitlab/gitlab-ee:latest' + restart: always + hostname: 'gitlab.example.com' + environment: + GITLAB_OMNIBUS_CONFIG: | + gitlab_rails['ldap_enabled'] = true + gitlab_rails['ldap_servers'] = { + 'main' => { + 'label' => 'LDAP', + 'host' => 'ldap.mydomain.com', + 'port' => 636, + 'uid' => 'sAMAccountName', + 'encryption' => 'simple_tls', + 'base' => 'dc=example,dc=com', + } + } + ``` + +1. Save the file and restart GitLab: + + ```shell + docker compose up -d + ``` + +:::TabTitle Self-compiled (source) + +1. Edit `/home/git/gitlab/config/gitlab.yml`: + + ```yaml + production: &base + ldap: + enabled: true + servers: + main: + label: 'LDAP' + host: 'ldap.mydomain.com' + port: 636 + uid: 'sAMAccountName' + encryption: 'simple_tls' + base: 'dc=example,dc=com' + ``` + +1. Save the file and restart GitLab: + + ```shell + # For systems running systemd + sudo systemctl restart gitlab.target + + # For systems running SysV init + sudo service gitlab restart + ``` + +For more information about the various LDAP options, see the `ldap` setting in +[`gitlab.yml.example`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/gitlab.yml.example). + +::EndTabs + +After configuring LDAP, to test the configuration, use the +[LDAP check Rake task](../../raketasks/ldap.md#check). ### Basic configuration settings @@ -178,24 +222,17 @@ These configuration settings are available: | `uid` | The LDAP attribute that maps to the username that users use to sign in. Should be the attribute, not the value that maps to the `uid`. Does not affect the GitLab username (see [attributes section](#attribute-configuration-settings)). | **{check-circle}** Yes | `'sAMAccountName'` or `'uid'` or `'userPrincipalName'` | | `bind_dn` | The full DN of the user you bind with. | **{dotted-circle}** No | `'america\momo'` or `'CN=Gitlab,OU=Users,DC=domain,DC=com'` | | `password` | The password of the bind user. | **{dotted-circle}** No | `'your_great_password'` | -| `encryption` | Encryption method. The `method` key is deprecated in favor of `encryption`. | **{check-circle}** Yes | `'start_tls'` or `'simple_tls'` or `'plain'` | +| `encryption` | Encryption method. The `method` key is deprecated in favor of `encryption`. | **{check-circle}** Yes | `'start_tls'`, `'simple_tls'`, or `'plain'`. `simple_tls` corresponds to 'Simple TLS' in the LDAP library. `start_tls` corresponds to StartTLS, not to be confused with regular TLS. If you specify `simple_tls`, usually it's on port 636, while `start_tls` (StartTLS) would be on port 389. `plain` also operates on port 389. | | `verify_certificates` | Enables SSL certificate verification if encryption method is `start_tls` or `simple_tls`. If set to false, no validation of the LDAP server's SSL certificate is performed. Defaults to true. | **{dotted-circle}** No | boolean | | `timeout` | Set a timeout, in seconds, for LDAP queries. This helps avoid blocking a request if the LDAP server becomes unresponsive. A value of `0` means there is no timeout. (default: `10`) | **{dotted-circle}** No | `10` or `30` | | `active_directory` | This setting specifies if LDAP server is Active Directory LDAP server. For non-AD servers it skips the AD specific queries. If your LDAP server is not AD, set this to false. | **{dotted-circle}** No | boolean | | `allow_username_or_email_login` | If enabled, GitLab ignores everything after the first `@` in the LDAP username submitted by the user on sign-in. If you are using `uid: 'userPrincipalName'` on ActiveDirectory you must disable this setting, because the userPrincipalName contains an `@`. | **{dotted-circle}** No | boolean | | `block_auto_created_users` | To maintain tight control over the number of billable users on your GitLab installation, enable this setting to keep new users blocked until they have been cleared by an administrator (default: false). | **{dotted-circle}** No | boolean | | `base` | Base where we can search for users. | **{check-circle}** Yes | `'ou=people,dc=gitlab,dc=example'` or `'DC=mydomain,DC=com'` | -| `user_filter` | Filter LDAP users. Format: [RFC 4515](https://www.rfc-editor.org/rfc/rfc4515.html) Note: GitLab does not support `omniauth-ldap`'s custom filter syntax. | **{dotted-circle}** No | For examples, read [Examples of user filters](#examples-of-user-filters). | +| `user_filter` | Filter LDAP users. Format: [RFC 4515](https://www.rfc-editor.org/rfc/rfc4515.html) Note: GitLab does not support `omniauth-ldap`'s custom filter syntax. | **{dotted-circle}** No | Some examples of the `user_filter` field syntax:<br/><br/>- `'(employeeType=developer)'`<br/>- `'(&(objectclass=user)(|(samaccountname=momo)(samaccountname=toto)))'` | | `lowercase_usernames` | If enabled, GitLab converts the name to lower case. | **{dotted-circle}** No | boolean | | `retry_empty_result_with_codes` | An array of LDAP query response code that attempt to retry the operation if the result/content is empty. For Google Secure LDAP, set this value to `[80]`. | **{dotted-circle}** No | `[80]` | -#### Examples of user filters - -Some examples of the `user_filter` field syntax: - -- `'(employeeType=developer)'` -- `'(&(objectclass=user)(|(samaccountname=momo)(samaccountname=toto)))'` - ### SSL configuration settings These SSL configuration settings are available: @@ -247,35 +284,193 @@ If you have users on multiple LDAP servers, you can configure GitLab to use them alphanumeric characters. GitLab uses the provider ID to associate each user with a specific LDAP server. - For each entry, use a unique `label` value. These values are used for the tab names on the sign-in page. -#### Example of multiple LDAP servers - -The following example shows how to configure three LDAP servers in `gitlab.rb`: - -```ruby -gitlab_rails['ldap_enabled'] = true -gitlab_rails['ldap_servers'] = { - 'main' => { - 'label' => 'GitLab AD', - 'host' => 'ad.example.org', - 'port' => 636, - ... - }, - - 'secondary' => { - 'label' => 'GitLab Secondary AD', - 'host' => 'ad-secondary.example.net', - 'port' => 636, - ... - }, - - 'tertiary' => { - 'label' => 'GitLab Tertiary AD', - 'host' => 'ad-tertiary.example.net', - 'port' => 636, - ... - } -} -``` +The following example shows how to configure three LDAP servers with +minimal configuration: + +::Tabs + +:::TabTitle Linux package (Omnibus) + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_enabled'] = true + gitlab_rails['ldap_servers'] = { + 'main' => { + 'label' => 'GitLab AD', + 'host' => 'ad.mydomain.com', + 'port' => 636, + 'uid' => 'sAMAccountName', + 'encryption' => 'simple_tls', + 'base' => 'dc=example,dc=com', + }, + + 'secondary' => { + 'label' => 'GitLab Secondary AD', + 'host' => 'ad-secondary.mydomain.com', + 'port' => 636, + 'uid' => 'sAMAccountName', + 'encryption' => 'simple_tls', + 'base' => 'dc=example,dc=com', + }, + + 'tertiary' => { + 'label' => 'GitLab Tertiary AD', + 'host' => 'ad-tertiary.mydomain.com', + 'port' => 636, + 'uid' => 'sAMAccountName', + 'encryption' => 'simple_tls', + 'base' => 'dc=example,dc=com', + } + } + ``` + +1. Save the file and reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +:::TabTitle Helm chart (Kubernetes) + +1. Export the Helm values: + + ```shell + helm get values gitlab > gitlab_values.yaml + ``` + +1. Edit `gitlab_values.yaml`: + + ```yaml + global: + appConfig: + ldap: + servers: + main: + label: 'GitLab AD' + host: 'ad.mydomain.com' + port: 636 + uid: 'sAMAccountName' + base: 'dc=example,dc=com' + encryption: 'simple_tls' + secondary: + label: 'GitLab Secondary AD' + host: 'ad-secondary.mydomain.com' + port: 636 + uid: 'sAMAccountName' + base: 'dc=example,dc=com' + encryption: 'simple_tls' + tertiary: + label: 'GitLab Tertiary AD' + host: 'ad-tertiary.mydomain.com' + port: 636 + uid: 'sAMAccountName' + base: 'dc=example,dc=com' + encryption: 'simple_tls' + ``` + +1. Save the file and apply the new values: + + ```shell + helm upgrade -f gitlab_values.yaml gitlab gitlab/gitlab + ``` + +:::TabTitle Docker + +1. Edit `docker-compose.yml`: + + ```yaml + version: "3.6" + services: + gitlab: + image: 'gitlab/gitlab-ee:latest' + restart: always + hostname: 'gitlab.example.com' + environment: + GITLAB_OMNIBUS_CONFIG: | + gitlab_rails['ldap_enabled'] = true + gitlab_rails['ldap_servers'] = { + 'main' => { + 'label' => 'GitLab AD', + 'host' => 'ad.mydomain.com', + 'port' => 636, + 'uid' => 'sAMAccountName', + 'encryption' => 'simple_tls', + 'base' => 'dc=example,dc=com', + }, + + 'secondary' => { + 'label' => 'GitLab Secondary AD', + 'host' => 'ad-secondary.mydomain.com', + 'port' => 636, + 'uid' => 'sAMAccountName', + 'encryption' => 'simple_tls', + 'base' => 'dc=example,dc=com', + }, + + 'tertiary' => { + 'label' => 'GitLab Tertiary AD', + 'host' => 'ad-tertiary.mydomain.com', + 'port' => 636, + 'uid' => 'sAMAccountName', + 'encryption' => 'simple_tls', + 'base' => 'dc=example,dc=com', + } + } + ``` + +1. Save the file and restart GitLab: + + ```shell + docker compose up -d + ``` + +:::TabTitle Self-compiled (source) + +1. Edit `/home/git/gitlab/config/gitlab.yml`: + + ```yaml + production: &base + ldap: + enabled: true + servers: + main: + label: 'GitLab AD' + host: 'ad.mydomain.com' + port: 636 + uid: 'sAMAccountName' + base: 'dc=example,dc=com' + encryption: 'simple_tls' + secondary: + label: 'GitLab Secondary AD' + host: 'ad-secondary.mydomain.com' + port: 636 + uid: 'sAMAccountName' + base: 'dc=example,dc=com' + encryption: 'simple_tls' + tertiary: + label: 'GitLab Tertiary AD' + host: 'ad-tertiary.mydomain.com' + port: 636 + uid: 'sAMAccountName' + base: 'dc=example,dc=com' + encryption: 'simple_tls' + ``` + +1. Save the file and restart GitLab: + + ```shell + # For systems running systemd + sudo systemctl restart gitlab.target + + # For systems running SysV init + sudo service gitlab restart + ``` + +For more information about the various LDAP options, see the `ldap` setting in +[`gitlab.yml.example`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/gitlab.yml.example). + +::EndTabs This example results in a sign-in page with the following tabs: @@ -289,27 +484,100 @@ To limit all GitLab access to a subset of the LDAP users on your LDAP server, fi configured `base`. However, to further filter users if necessary, you can set up an LDAP user filter. The filter must comply with [RFC 4515](https://www.rfc-editor.org/rfc/rfc4515.html). -- Example user filter for Omnibus GitLab instances: +::Tabs - ```ruby - gitlab_rails['ldap_servers'] = { - 'main' => { - # snip... - 'user_filter' => '(employeeType=developer)' - } - } - ``` +:::TabTitle Linux package (Omnibus) -- Example user filter for source install instances: +1. Edit `/etc/gitlab/gitlab.rb`: - ```yaml - production: - ldap: - servers: - main: - # snip... - user_filter: '(employeeType=developer)' - ``` + ```ruby + gitlab_rails['ldap_servers'] = { + 'main' => { + 'user_filter' => '(employeeType=developer)' + } + } + ``` + +1. Save the file and reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +:::TabTitle Helm chart (Kubernetes) + +1. Export the Helm values: + + ```shell + helm get values gitlab > gitlab_values.yaml + ``` + +1. Edit `gitlab_values.yaml`: + + ```yaml + global: + appConfig: + ldap: + servers: + main: + user_filter: '(employeeType=developer)' + ``` + +1. Save the file and apply the new values: + + ```shell + helm upgrade -f gitlab_values.yaml gitlab gitlab/gitlab + ``` + +:::TabTitle Docker + +1. Edit `docker-compose.yml`: + + ```yaml + version: "3.6" + services: + gitlab: + image: 'gitlab/gitlab-ee:latest' + restart: always + hostname: 'gitlab.example.com' + environment: + GITLAB_OMNIBUS_CONFIG: | + gitlab_rails['ldap_servers'] = { + 'main' => { + 'user_filter' => '(employeeType=developer)' + } + } + ``` + +1. Save the file and restart GitLab: + + ```shell + docker compose up -d + ``` + +:::TabTitle Self-compiled (source) + +1. Edit `/home/git/gitlab/config/gitlab.yml`: + + ```yaml + production: &base + ldap: + servers: + main: + user_filter: '(employeeType=developer)' + ``` + +1. Save the file and restart GitLab: + + ```shell + # For systems running systemd + sudo systemctl restart gitlab.target + + # For systems running SysV init + sudo service gitlab restart + ``` + +::EndTabs To limit access to the nested members of an Active Directory group, use the following syntax: @@ -325,7 +593,7 @@ Support for nested members in the user filter shouldn't be confused with GitLab does not support the custom filter syntax used by OmniAuth LDAP. -#### Escape special characters +#### Escape special characters in `user_filter` The `user_filter` DN can contain special characters. For example: @@ -338,11 +606,11 @@ The `user_filter` DN can contain special characters. For example: - Open and close brackets: ```plaintext - OU=Gitlab (Inc),DC=gitlab,DC=com + OU=GitLab (Inc),DC=gitlab,DC=com ``` - These characters must be escaped as documented in - [RFC 4515](https://www.rfc-editor.org/rfc/rfc4515.html). +These characters must be escaped as documented in +[RFC 4515](https://www.rfc-editor.org/rfc/rfc4515.html#section-4). - Escape commas with `\2C`. For example: @@ -353,12 +621,12 @@ The `user_filter` DN can contain special characters. For example: - Escape open brackets with `\28` and close brackets with `\29`. For example: ```plaintext - OU=Gitlab \28Inc\29,DC=gitlab,DC=com + OU=GitLab \28Inc\29,DC=gitlab,DC=com ``` ### Enable LDAP username lowercase -Some LDAP servers, depending on their configurations, can return uppercase usernames. +Some LDAP servers, depending on their configuration, can return uppercase usernames. This can lead to several confusing issues such as creating links or namespaces with uppercase names. GitLab can automatically lowercase usernames provided by the LDAP server by enabling @@ -373,13 +641,67 @@ the configuration option `lowercase_usernames`. By default, this configuration o ```ruby gitlab_rails['ldap_servers'] = { 'main' => { - # snip... 'lowercase_usernames' => true } } ``` -1. [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +:::TabTitle Helm chart (Kubernetes) + +1. Export the Helm values: + + ```shell + helm get values gitlab > gitlab_values.yaml + ``` + +1. Edit `gitlab_values.yaml`: + + ```yaml + global: + appConfig: + ldap: + servers: + main: + lowercase_usernames: true + ``` + +1. Save the file and apply the new values: + + ```shell + helm upgrade -f gitlab_values.yaml gitlab gitlab/gitlab + ``` + +:::TabTitle Docker + +1. Edit `docker-compose.yml`: + + ```yaml + version: "3.6" + services: + gitlab: + image: 'gitlab/gitlab-ee:latest' + restart: always + hostname: 'gitlab.example.com' + environment: + GITLAB_OMNIBUS_CONFIG: | + gitlab_rails['ldap_servers'] = { + 'main' => { + 'lowercase_usernames' => true + } + } + ``` + +1. Save the file and restart GitLab: + + ```shell + docker compose up -d + ``` :::TabTitle Self-compiled (source) @@ -390,11 +712,18 @@ the configuration option `lowercase_usernames`. By default, this configuration o ldap: servers: main: - # snip... lowercase_usernames: true ``` -1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. +1. Save the file and restart GitLab: + + ```shell + # For systems running systemd + sudo systemctl restart gitlab.target + + # For systems running SysV init + sudo service gitlab restart + ``` ::EndTabs @@ -418,7 +747,11 @@ This does not disable using LDAP credentials for Git access. gitlab_rails['prevent_ldap_sign_in'] = true ``` -1. [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` :::TabTitle Helm chart (Kubernetes) @@ -443,6 +776,28 @@ This does not disable using LDAP credentials for Git access. helm upgrade -f gitlab_values.yaml gitlab gitlab/gitlab ``` +:::TabTitle Docker + +1. Edit `docker-compose.yml`: + + ```yaml + version: "3.6" + services: + gitlab: + image: 'gitlab/gitlab-ee:latest' + restart: always + hostname: 'gitlab.example.com' + environment: + GITLAB_OMNIBUS_CONFIG: | + gitlab_rails['prevent_ldap_sign_in'] = true + ``` + +1. Save the file and restart GitLab: + + ```shell + docker compose up -d + ``` + :::TabTitle Self-compiled (source) 1. Edit `config/gitlab.yaml`: @@ -453,41 +808,46 @@ This does not disable using LDAP credentials for Git access. prevent_ldap_sign_in: true ``` -1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. +1. Save the file and restart GitLab: + + ```shell + # For systems running systemd + sudo systemctl restart gitlab.target + + # For systems running SysV init + sudo service gitlab restart + ``` ::EndTabs ### Use encrypted credentials Instead of having the LDAP integration credentials stored in plaintext in the configuration files, you can optionally -use an encrypted file for the LDAP credentials. To use this feature, first you must enable -[GitLab encrypted configuration](../../encrypted_configuration.md). +use an encrypted file for the LDAP credentials. -The encrypted configuration for LDAP exists in an encrypted YAML file. By default the file is created at -`shared/encrypted_configuration/ldap.yaml.enc`. This location is configurable in the GitLab configuration. +Prerequisites: -The unencrypted contents of the file should be a subset of the secret settings from your `servers` block in the LDAP -configuration. +- To use encrypted credentials, you must first enable the + [encrypted configuration](../../encrypted_configuration.md). + +The encrypted configuration for LDAP exists in an encrypted YAML file. The +unencrypted contents of the file should be a subset of the secret settings from +your `servers` block in the LDAP configuration. The supported configuration items for the encrypted file are: - `bind_dn` - `password` -The encrypted contents can be configured with the [LDAP secret edit Rake command](../../raketasks/ldap.md#edit-secret). - ::Tabs :::TabTitle Linux package (Omnibus) -If initially your LDAP configuration looked like: - -1. In `/etc/gitlab/gitlab.rb`: +1. If initially your LDAP configuration in `/etc/gitlab/gitlab.rb` looked like: ```ruby gitlab_rails['ldap_servers'] = { 'main' => { - # snip... 'bind_dn' => 'admin', 'password' => '123' } @@ -500,7 +860,7 @@ If initially your LDAP configuration looked like: sudo gitlab-rake gitlab:ldap:secret:edit EDITOR=vim ``` -1. The unencrypted contents of the LDAP secret should be entered like: +1. Enter the unencrypted contents of the LDAP secret: ```yaml main: @@ -509,21 +869,69 @@ If initially your LDAP configuration looked like: ``` 1. Edit `/etc/gitlab/gitlab.rb` and remove the settings for `bind_dn` and `password`. +1. Save the file and reconfigure GitLab: -1. [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + ```shell + sudo gitlab-ctl reconfigure + ``` -:::TabTitle Self-compiled (source) +:::TabTitle Helm chart (Kubernetes) + +Use a Kubernetes secret to store the LDAP password. For more information, +read about [Helm LDAP secrets](https://docs.gitlab.com/charts/installation/secrets.html#ldap-password). -If initially your LDAP configuration looked like: +:::TabTitle Docker -1. In `config/gitlab.yaml`: +1. If initially your LDAP configuration in `docker-compose.yml` looked like: + + ```yaml + version: "3.6" + services: + gitlab: + image: 'gitlab/gitlab-ee:latest' + restart: always + hostname: 'gitlab.example.com' + environment: + GITLAB_OMNIBUS_CONFIG: | + gitlab_rails['ldap_servers'] = { + 'main' => { + 'bind_dn' => 'admin', + 'password' => '123' + } + } + ``` + +1. Get inside the container, and edit the encrypted secret: + + ```shell + sudo docker exec -t <container_name> bash + gitlab-rake gitlab:ldap:secret:edit EDITOR=vim + ``` + +1. Enter the unencrypted contents of the LDAP secret: + + ```yaml + main: + bind_dn: admin + password: '123' + ``` + +1. Edit `docker-compose.yml` and remove the settings for `bind_dn` and `password`. +1. Save the file and restart GitLab: + + ```shell + docker compose up -d + ``` + +:::TabTitle Self-compiled (source) + +1. If initially your LDAP configuration in `/home/git/gitlab/config/gitlab.yml` looked like: ```yaml production: ldap: servers: main: - # snip... bind_dn: admin password: '123' ``` @@ -534,7 +942,7 @@ If initially your LDAP configuration looked like: bundle exec rake gitlab:ldap:secret:edit EDITOR=vim RAILS_ENVIRONMENT=production ``` -1. The unencrypted contents of the LDAP secret should be entered like: +1. Enter the unencrypted contents of the LDAP secret: ```yaml main: @@ -542,9 +950,16 @@ If initially your LDAP configuration looked like: password: '123' ``` -1. Edit `config/gitlab.yaml` and remove the settings for `bind_dn` and `password`. +1. Edit `/home/git/gitlab/config/gitlab.yml` and remove the settings for `bind_dn` and `password`. +1. Save the file and restart GitLab: + + ```shell + # For systems running systemd + sudo systemctl restart gitlab.target -1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. + # For systems running SysV init + sudo service gitlab restart + ``` ::EndTabs diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md index 21ec4b293d4..95064b296af 100644 --- a/doc/administration/auth/ldap/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap/ldap-troubleshooting.md @@ -373,7 +373,7 @@ things to debug the situation. 1. Search for the user. 1. Open the user by selecting their name. Do not select **Edit**. 1. Select the **Identities** tab. There should be an LDAP identity with - an LDAP DN as the 'Identifier'. If not, this user hasn't signed in with + an LDAP DN as the `Identifier`. If not, this user hasn't signed in with LDAP yet and must do so first. - You've waited an hour or [the configured interval](ldap_synchronization.md#adjust-ldap-group-sync-schedule) for the group to sync. To speed up the process, either go to the GitLab group **Group information > Members** @@ -523,8 +523,8 @@ LDAP group lookups. The very last occurrence of this entry should indicate exactly which users GitLab believes should be added to the group. NOTE: -10 is 'Guest', 20 is 'Reporter', 30 is 'Developer', 40 is 'Maintainer' -and 50 is 'Owner'. +10 is `Guest`, 20 is `Reporter`, 30 is `Developer`, 40 is `Maintainer` +and 50 is `Owner`. ```shell Resolved 'my_group' group member access: {"uid=john0,ou=people,dc=example,dc=com"=>30, diff --git a/doc/administration/auth/ldap/ldap_synchronization.md b/doc/administration/auth/ldap/ldap_synchronization.md index 5c8c7f7baf1..cc300941470 100644 --- a/doc/administration/auth/ldap/ldap_synchronization.md +++ b/doc/administration/auth/ldap/ldap_synchronization.md @@ -366,7 +366,7 @@ group, GitLab revokes their `admin` role when syncing. ### Global group memberships lock -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1793) in GitLab 12.0. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/4354) in GitLab 12.0. GitLab administrators can prevent group members from inviting new members to subgroups that have their membership synchronized with LDAP. diff --git a/doc/administration/clusters/kas.md b/doc/administration/clusters/kas.md index 79dd69183a6..9f0f7e836f7 100644 --- a/doc/administration/clusters/kas.md +++ b/doc/administration/clusters/kas.md @@ -24,8 +24,6 @@ As a GitLab administrator, you can install the agent server: - For [Omnibus installations](#for-omnibus). - For [GitLab Helm Chart installations](#for-gitlab-helm-chart). -Or, you can [use an external agent server](#use-an-external-installation). - ### For Omnibus You can enable the agent server for [Omnibus](https://docs.gitlab.com/omnibus/) package installations on a single node, or on multiple nodes at once. @@ -60,6 +58,11 @@ To enable the agent server on multiple nodes: 'SSL_CERT_DIR' => "/opt/gitlab/embedded/ssl/certs/", 'OWN_PRIVATE_API_URL' => 'grpc://<ip_or_hostname_of_this_host>:8155' } + + gitlab_rails['gitlab_kas_enabled'] = true + gitlab_rails['gitlab_kas_external_url'] = 'wss://gitlab.example.com/-/kubernetes-agent/' + gitlab_rails['gitlab_kas_internal_url'] = 'grpc://kas.internal.gitlab.example.com' + gitlab_rails['gitlab_kas_external_k8s_proxy_url'] = 'https://gitlab.example.com/-/kubernetes-agent/' ``` In this configuration: @@ -68,8 +71,10 @@ To enable the agent server on multiple nodes: - `OWN_PRIVATE_API_URL` is the environment variable used by the KAS process for service discovery. You can set it to a hostname or IP address of the node you're configuring. The node must be reachable by other nodes in the cluster. - `gitlab_kas['api_secret_key']` is the shared secret used for authentication between KAS and GitLab. This value must be Base64-encoded and exactly 32 bytes long. - `gitlab_kas['private_api_secret_key']` is the shared secret used for authentication between different KAS instances. This value must be Base64-encoded and exactly 32 bytes long. + - `gitlab_rails['gitlab_kas_external_url']` is the user-facing URL for the in-cluster `agentk`. + - `gitlab_rails['gitlab_kas_internal_url']` is the internal URL the GitLab backend uses to communicate with KAS. + - `gitlab_rails['gitlab_kas_external_k8s_proxy_url']` is the user-facing URL for Kubernetes API proxying. -1. For each application node, follow the steps in [Use an external installation](../clusters/kas.md#use-an-external-installation). If the agent server is enabled on the application node, do not include `gitlab_kas['enable'] = false` in the configuration for that node. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). ### For GitLab Helm Chart @@ -100,30 +105,6 @@ For GitLab [Helm Chart](https://docs.gitlab.com/charts/) installations: For details, see [how to use the GitLab-KAS chart](https://docs.gitlab.com/charts/charts/gitlab/kas/). -### Use an external installation - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299850) in GitLab 13.10. - -Instead of installing the agent server, you can configure GitLab to use an external agent server. - -If you used the GitLab Helm Chart to install GitLab, see -[how to configure your external agent server](https://docs.gitlab.com/charts/charts/globals.html#external-kas). - -If you used the Omnibus packages: - -1. Edit `/etc/gitlab/gitlab.rb` and add the paths to your external agent server: - - ```ruby - gitlab_kas['enable'] = false - gitlab_kas['api_secret_key'] = 'Your shared secret between GitLab and KAS' - - gitlab_rails['gitlab_kas_enabled'] = true - gitlab_rails['gitlab_kas_external_url'] = 'wss://kas.gitlab.example.com' # User-facing URL for the in-cluster agentk - gitlab_rails['gitlab_kas_internal_url'] = 'grpc://kas.internal.gitlab.example.com' # Internal URL for the GitLab backend - ``` - -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - ## Troubleshooting If you have issues while using the agent server for Kubernetes, view the diff --git a/doc/administration/docs_self_host.md b/doc/administration/docs_self_host.md index 9021a795523..f049dafea76 100644 --- a/doc/administration/docs_self_host.md +++ b/doc/administration/docs_self_host.md @@ -91,7 +91,7 @@ Prerequisite: To host the product documentation site with GitLab Pages: -1. [Create a blank project](../user/project/working_with_projects.md#create-a-blank-project). +1. [Create a blank project](../user/project/index.md#create-a-blank-project). 1. Create a new or edit your existing `.gitlab-ci.yml` file, and add the following `pages` job, while ensuring the version is the same as your GitLab installation: diff --git a/doc/administration/geo/disaster_recovery/bring_primary_back.md b/doc/administration/geo/disaster_recovery/bring_primary_back.md index dda2c5d34d3..fe05b52cec9 100644 --- a/doc/administration/geo/disaster_recovery/bring_primary_back.md +++ b/doc/administration/geo/disaster_recovery/bring_primary_back.md @@ -26,6 +26,16 @@ Alternatively, you can [set up a new **secondary** GitLab instance](../setup/ind To bring the former **primary** site up to date: 1. SSH into the former **primary** site that has fallen behind. +1. Remove `/etc/gitlab/gitlab-cluster.json` if it exists. + + If the site to be re-added as a **secondary** site was promoted with the `gitlab-ctl geo promote` command, then it may contain a `/etc/gitlab/gitlab-cluster.json` file. For example during `gitlab-ctl reconfigure`, you may notice output like: + + ```plaintext + The 'geo_primary_role' is defined in /etc/gitlab/gitlab-cluster.json as 'true' and overrides the setting in the /etc/gitlab/gitlab.rb + ``` + + If so, then `/etc/gitlab/gitlab-cluster.json` must be deleted from every Sidekiq, PostgreSQL, Gitaly, and Rails node in the site, to make `/etc/gitlab/gitlab.rb` the single source of truth again. + 1. Make sure all the services are up: ```shell diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index a78350d9dba..b6703e8a5fb 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -460,9 +460,9 @@ required: ### Step 4. (Optional) Updating the primary domain DNS record -Updating the DNS records for the primary domain to point to the **secondary** site -to prevent the need to update all references to the primary domain to the -secondary domain, like changing Git remotes and API URLs. +Update DNS records for the primary domain to point to the **secondary** site. +This removes the need to update all references to the primary domain, for example +changing Git remotes and API URLs. 1. SSH into the **secondary** site and login as root: @@ -479,6 +479,21 @@ secondary domain, like changing Git remotes and API URLs. external_url 'https://<new_external_url>' ``` + If you provide GitLab with its certificate + [manually](https://docs.gitlab.com/omnibus/settings/ssl/index.html#configure-https-manually), + ensure: + + - The new URL is one of the subject alternative names: + + ```shell + /opt/gitlab/embedded/bin/openssl x509 -noout -dates -subject -issuer \ + -nameopt multiline -ext subjectAltName -in /etc/gitlab/ssl/new-gitlab.new-example.com.crt + ``` + + - The certificate and key filenames match the new `external_url`, + or those filenames are + [specified in `/etc/gitlab/gitlab.rb`](https://docs.gitlab.com/omnibus/settings/ssl/index.html#change-the-default-ssl-certificate-location). + NOTE: Changing `external_url` does not prevent access via the old secondary URL, as long as the secondary DNS records are still intact. diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index 68fd0c63e37..8728ab57bba 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -124,7 +124,9 @@ The following are required to run Geo: - Git 2.9 or later - Git-lfs 2.4.2 or later on the user side when using LFS - All sites must run [the same GitLab and PostgreSQL versions](setup/database.md#postgresql-replication). -- If using different operating system versions between Geo sites, [check OS locale data compatibility](replication/troubleshooting.md#check-os-locale-data-compatibility) across Geo sites. + - If using different operating system versions between Geo sites, + [check OS locale data compatibility](replication/troubleshooting.md#check-os-locale-data-compatibility) + across Geo sites to avoid silent corruption of database indexes. Additionally, check the GitLab [minimum requirements](../../install/requirements.md), and use the latest version of GitLab for a better experience. @@ -163,15 +165,6 @@ To update the internal URL of the primary Geo site: 1. Select **Edit** on the primary site. 1. Change the **Internal URL**, then select **Save changes**. -### LDAP - -We recommend that if you use LDAP on your **primary** site, you also set up secondary LDAP servers on each **secondary** site. Otherwise, users are unable to perform Git operations over HTTP(s) on the **secondary** site using HTTP Basic Authentication. However, Git via SSH and personal access tokens still works. - -NOTE: -It is possible for all **secondary** sites to share an LDAP server, but additional latency can be an issue. Also, consider what LDAP server is available in a [disaster recovery](disaster_recovery/index.md) scenario if a **secondary** site is promoted to be a **primary** site. - -Check for instructions on how to set up replication in your LDAP service. Instructions are different depending on the software or service used. For example, OpenLDAP provides [these instructions](https://www.openldap.org/doc/admin24/replication.html). - ### Geo Tracking Database The tracking database instance is used as metadata to control what needs to be updated on the disk of the local instance. For example: @@ -224,6 +217,8 @@ An [epic exists](https://gitlab.com/groups/gitlab-org/-/epics/4623) to fix this The only way to view designs replication data for a particular secondary site is to visit that secondary site directly. For example, `https://<IP of your secondary site>/admin/geo/replication/designs`. An [epic exists](https://gitlab.com/groups/gitlab-org/-/epics/4624) to fix this limitation. +Keep in mind that mentioned URLs don't work when [Admin Mode](../../user/admin_area/settings/sign_in_restrictions.md#admin-mode) is enabled. + ## Setup instructions For setup instructions, see [Setting up Geo](setup/index.md). @@ -292,6 +287,14 @@ For more information on how to replicate the Container Registry, see [Container For more information on using Geo proxying on secondary sites, see [Geo proxying for secondary sites](secondary_proxy/index.md). +### Single Sign On (SSO) + +For more information on configuring Single Sign-On (SSO), see [Geo with Single Sign-On (SSO)](replication/single_sign_on.md). + +#### LDAP + +For more information on configuring LDAP, see [Geo with Single Sign-On (SSO) > LDAP](replication/single_sign_on.md#ldap). + ### Security Review For more information on Geo security, see [Geo security review](replication/security_review.md). diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index d625b2a0324..79a3f65377b 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -104,7 +104,7 @@ keys must be manually replicated to the **secondary** site. 1. Make a backup of any existing SSH host keys: ```shell - find /etc/ssh -iname ssh_host_* -exec cp {} {}.backup.`date +%F` \; + find /etc/ssh -iname 'ssh_host_*' -exec cp {} {}.backup.`date +%F` \; ``` 1. Copy OpenSSH host keys from the **primary** site: diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index 52cd64b8f33..26acab510cf 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -64,7 +64,7 @@ verification methods: | Blobs | Alert Metric Images _(file system)_ | Geo with API | SHA256 checksum | | Blobs | Alert Metric Images _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | Dependency Proxy Images_(file system)_ | Geo with API | SHA256 checksum | -| Blobs | Dependency Proxy Images _(object_storage)_ | Geo with API/managed (*2*) | _Not implemented_ | +| Blobs | Dependency Proxy Images _(object storage)_ | Geo with API/managed (*2*) | _Not implemented_ | - (*1*): Redis replication can be used as part of HA with Redis sentinel. It's not used between Geo sites. - (*2*): Object storage replication can be performed by Geo or by your object storage provider/appliance diff --git a/doc/administration/geo/replication/location_aware_git_url.md b/doc/administration/geo/replication/location_aware_git_url.md index 460de5f3232..4a3f9c86041 100644 --- a/doc/administration/geo/replication/location_aware_git_url.md +++ b/doc/administration/geo/replication/location_aware_git_url.md @@ -57,7 +57,7 @@ routing configurations.  -1. Select the **Create traffic policy** button. +1. Select **Create traffic policy**.  diff --git a/doc/administration/geo/replication/remove_geo_site.md b/doc/administration/geo/replication/remove_geo_site.md index 4b9f31dc08c..9d92652daf4 100644 --- a/doc/administration/geo/replication/remove_geo_site.md +++ b/doc/administration/geo/replication/remove_geo_site.md @@ -11,7 +11,7 @@ type: howto 1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Geo > Nodes**. -1. Select the **Remove** button for the **secondary** site you want to remove. +1. For the **secondary** site you want to remove, select **Remove**. 1. Confirm by selecting **Remove** when the prompt appears. After the **secondary** site is removed from the Geo administration page, you must @@ -25,6 +25,9 @@ stop and uninstall this site. For each node on your secondary Geo site: 1. Uninstall GitLab: + NOTE: + If GitLab data has to be cleaned from the instance as well, see how to [uninstall the Linux package and all its data](https://docs.gitlab.com/omnibus/installation/#uninstall-the-linux-package-omnibus). + ```shell # Stop gitlab and remove its supervision process sudo gitlab-ctl uninstall diff --git a/doc/administration/geo/replication/single_sign_on.md b/doc/administration/geo/replication/single_sign_on.md new file mode 100644 index 00000000000..fc2f23552db --- /dev/null +++ b/doc/administration/geo/replication/single_sign_on.md @@ -0,0 +1,122 @@ +--- +stage: Systems +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: howto +--- + +# Geo with Single Sign On (SSO) **(PREMIUM SELF)** + +This documentation only discusses Geo-specific SSO considerations and configuration. For more information on general authentication, see [GitLab authentication and authorization](../../auth/index.md). + +## Configuring instance-wide SAML + +### Prerequisites + +[Instance-wide SAML](../../../integration/saml.md) must be working on your primary Geo site. + +You only configure SAML on the primary site. Configuring `gitlab_rails['omniauth_providers']` in `gitlab.rb` in a secondary site has no effect. + +### Determine the type of URL your secondary site uses + +How you configure instance-wide SAML differs depending on your secondary site configuration. Determine if your secondary site uses a: + +- [Unified URL](../secondary_proxy/index.md#set-up-a-unified-url-for-geo-sites), meaning the `external_url` exactly matches the `external_url` of the primary site. +- [Separate URL](../secondary_proxy/index.md#geo-proxying-with-separate-urls) with proxying enabled. Proxying is enabled by default in GitLab 15.1 and later. +- [Separate URL](../secondary_proxy/index.md#geo-proxying-with-separate-urls) with proxying disabled. + +### SAML with Unified URL + +If you have configured SAML on the primary site correctly, then it should work on the secondary site without additional configuration. + +### SAML with separate URL with proxying enabled + +If a secondary site uses a different `external_url` to the primary site, then configure your SAML Identity Provider (IdP) to allow the secondary site's SAML callback URL. For example, to configure Okta: + +1. [Sign in to Okta](https://www.okta.com/login/). +1. Go to **Okta Admin Dashboard** > **Applications** > **Your App Name** > **General**. +1. In **SAML Settings**, select **Edit**. +1. In **General Settings**, select **Next** to go to **SAML Settings**. +1. In **SAML Settings > General**, make sure the **Single sign-on URL** is your primary site's SAML callback URL. For example, `https://gitlab-primary.example.com/users/auth/saml/callback`. If it is not, enter your primary site's SAML callback URL into this field. +1. Select **Show Advanced Settings**. +1. In **Other Requestable SSO URLs**, enter your secondary site's SAML callback URL. For example, `https://gitlab-secondary.example.com/users/auth/saml/callback`. You can set **Index** to anything. +1. Select **Next** and then **Finish**. + +You must not specify `assertion_consumer_service_url` in the SAML provider configuration in `gitlab_rails['omniauth_providers']` in `gitlab.rb` of the primary site. For example: + +```ruby +gitlab_rails['omniauth_providers'] = [ + { + name: "saml", + label: "Okta", # optional label for login button, defaults to "Saml" + args: { + idp_cert_fingerprint: "B5:AD:AA:9E:3C:05:68:AD:3B:78:ED:31:99:96:96:43:9E:6D:79:96", + idp_sso_target_url: "https://<dev-account>.okta.com/app/dev-account_gitlabprimary_1/exk7k2gft2VFpVFXa5d1/sso/saml", + issuer: "https://<gitlab-primary>", + name_identifier_format: "urn:oasis:names:tc:SAML:2.0:nameid-format:persistent" + } + } +] +``` + +This configuration causes: + +- Both your sites to use `/users/auth/saml/callback` as their assertion consumer service (ACS) URL. +- The URL's host to be set to the corresponding site's host. + +You can check this by visiting each site's `/users/auth/saml/metadata` path. For example, visiting `https://gitlab-primary.example.com/users/auth/saml/metadata` may respond with: + +```xml +<md:EntityDescriptor ID="_b9e00d84-d34e-4e3d-95de-122e3c361617" entityID="https://gitlab-primary.example.com" + xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata" + xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion"> + <md:SPSSODescriptor AuthnRequestsSigned="false" WantAssertionsSigned="false" protocolSupportEnumeration="urn:oasis:names:tc:SAML:2.0:protocol"> + <md:NameIDFormat>urn:oasis:names:tc:SAML:2.0:nameid-format:persistent</md:NameIDFormat> + <md:AssertionConsumerService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" Location="https://gitlab-primary.example.com/users/auth/saml/callback" index="0" isDefault="true"/> + <md:AttributeConsumingService index="1" isDefault="true"> + <md:ServiceName xml:lang="en">Required attributes</md:ServiceName> + <md:RequestedAttribute FriendlyName="Email address" Name="email" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic" isRequired="false"/> + <md:RequestedAttribute FriendlyName="Full name" Name="name" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic" isRequired="false"/> + <md:RequestedAttribute FriendlyName="Given name" Name="first_name" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic" isRequired="false"/> + <md:RequestedAttribute FriendlyName="Family name" Name="last_name" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic" isRequired="false"/> + </md:AttributeConsumingService> + </md:SPSSODescriptor> +</md:EntityDescriptor> +``` + +Visiting `https://gitlab-secondary.example.com/users/auth/saml/metadata` may respond with: + +```xml +<md:EntityDescriptor ID="_bf71eb57-7490-4024-bfe2-54cec716d4bf" entityID="https://gitlab-primary.example.com" + xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata" + xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion"> + <md:SPSSODescriptor AuthnRequestsSigned="false" WantAssertionsSigned="false" protocolSupportEnumeration="urn:oasis:names:tc:SAML:2.0:protocol"> + <md:NameIDFormat>urn:oasis:names:tc:SAML:2.0:nameid-format:persistent</md:NameIDFormat> + <md:AssertionConsumerService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" Location="https://gitlab-secondary.example.com/users/auth/saml/callback" index="0" isDefault="true"/> + <md:AttributeConsumingService index="1" isDefault="true"> + <md:ServiceName xml:lang="en">Required attributes</md:ServiceName> + <md:RequestedAttribute FriendlyName="Email address" Name="email" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic" isRequired="false"/> + <md:RequestedAttribute FriendlyName="Full name" Name="name" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic" isRequired="false"/> + <md:RequestedAttribute FriendlyName="Given name" Name="first_name" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic" isRequired="false"/> + <md:RequestedAttribute FriendlyName="Family name" Name="last_name" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic" isRequired="false"/> + </md:AttributeConsumingService> + </md:SPSSODescriptor> +</md:EntityDescriptor> +``` + +The `Location` attribute of the `md:AssertionConsumerService` field points to `gitlab-secondary.example.com`. + +After configuring your SAML IdP to allow the secondary site's SAML callback URL, you should be able to sign in with SAML on your primary site as well as your secondary site. + +### SAML with separate URL with proxying disabled + +If you have configured SAML on the primary site correctly, then it should work on the secondary site without additional configuration. + +## LDAP + +If you use LDAP on your **primary** site, you should also set up secondary LDAP servers on each **secondary** site. Otherwise, users cannot perform Git operations over HTTP(s) on the **secondary** site using HTTP basic authentication. However, users can still use Git with SSH and personal access tokens. + +NOTE: +It is possible for all **secondary** sites to share an LDAP server, but additional latency can be an issue. Also, consider what LDAP server is available in a [disaster recovery](../disaster_recovery/index.md) scenario if a **secondary** site is promoted to be a **primary** site. + +Check your LDAP service documentation for instructions on how to set up replication in your LDAP service. The process differs depending on the software or service used. For example, OpenLDAP provides this [replication documentation](https://www.openldap.org/doc/admin24/replication.html). diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 1dcced781ce..2f2759d7339 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -1100,6 +1100,10 @@ On the **primary** site: 1. Ensure the **URL** field matches the value found in `/etc/gitlab/gitlab.rb` in `external_url "https://gitlab.example.com"` on the **Rails nodes of the secondary** site. +### Authenticating with SAML on the secondary site always lands on the primary site + +This [problem is usually encountered when upgrading to GitLab 15.1](version_specific_upgrades.md#upgrading-to-151). To fix this problem, see [configuring instance-wide SAML in Geo with Single Sign-On](single_sign_on.md#configuring-instance-wide-saml). + ## Fixing common errors This section documents common error messages reported in the Admin Area on the web interface, and how to fix them. @@ -1313,6 +1317,18 @@ registry = Geo::PackageFileRegistry.find(registry_id) registry.replicator.send(:download) ``` +#### Find registry records of blobs that failed to sync + +```ruby +Geo::PackageFileRegistry.failed +``` + +#### Find registry records of blobs that are missing on the primary site + +```ruby +Geo::PackageFileRegistry.where(last_sync_failure: 'The file is missing on the Geo primary site') +``` + #### Verify package files on the secondary manually This iterates over all package files on the secondary, looking at the @@ -1340,7 +1356,7 @@ status.keys.each {|key| puts "#{key} count: #{status[key].count}"} status ``` -### Reverify all uploads (or any SSF data type which is verified) +#### Reverify all uploads (or any SSF data type which is verified) 1. SSH into a GitLab Rails node in the primary Geo site. 1. Open [Rails console](../../../administration/operations/rails_console.md#starting-a-rails-console-session). @@ -1396,21 +1412,6 @@ registry = Geo::SnippetRepositoryRegistry.find(registry_id) registry.replicator.send(:sync_repository) ``` -### Find failed artifacts - -[Start a Rails console session](../../../administration/operations/rails_console.md#starting-a-rails-console-session) -to run the following commands: - -```ruby -Geo::JobArtifactRegistry.failed -``` - -#### Find `ID` of synced artifacts that are missing on primary - -```ruby -Geo::JobArtifactRegistry.synced.missing_on_primary.pluck(:artifact_id) -``` - ### Project or project wiki repositories #### Find repository verification failures @@ -1535,9 +1536,12 @@ If the above steps are **not successful**, proceed through the next steps: ## Check OS locale data compatibility -If different operating systems or different operating system versions are deployed across Geo sites, we recommend that you perform a locale data compatibility check setting up Geo. +If different operating systems or different operating system versions are deployed across Geo sites, you should perform a locale data compatibility check before setting up Geo. -Geo uses PostgreSQL and Streaming Replication to replicate data across Geo sites. PostgreSQL uses locale data provided by the operating system’s C library for sorting text. If the locale data in the C library is incompatible across Geo sites, erroneous query results that lead to [incorrect behavior on secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/360723). See [here](https://wiki.postgresql.org/wiki/Locale_data_changes) for more details. +Geo uses PostgreSQL and Streaming Replication to replicate data across Geo sites. PostgreSQL uses locale data provided by the operating system's C library for sorting text. If the locale data in the C library is incompatible across Geo sites, erroneous query results that lead to [incorrect behavior on secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/360723). + +For example, Ubuntu 18.04 (and earlier) and RHEL/Centos7 (and earlier) are incompatible with their later releases. +See the [PostgreSQL wiki for more details](https://wiki.postgresql.org/wiki/Locale_data_changes). On all hosts running PostgreSQL, across all Geo sites, run the following shell command: @@ -1561,4 +1565,8 @@ or the reverse order: If the output is identical on all hosts, then they running compatible versions of locale data. -If the output differs on some hosts, then PostgreSQL replication will not work properly. We advise that you select operating system versions that are compatible. +If the output differs on some hosts, PostgreSQL replication does not work properly: indexes are corrupted on the database replicas. You should select operating system versions that are compatible. + +A full index rebuild is required if the on-disk data is transferred 'at rest' to an operating system with an incompatible locale, or through replication. + +This check is also required when using a mixture of GitLab deployments. The locale might be different between an Linux package install, a GitLab Docker container, a Helm chart deployment, or external database services. diff --git a/doc/administration/geo/replication/version_specific_upgrades.md b/doc/administration/geo/replication/version_specific_upgrades.md index d981656f748..aa8e5d77f67 100644 --- a/doc/administration/geo/replication/version_specific_upgrades.md +++ b/doc/administration/geo/replication/version_specific_upgrades.md @@ -6,6 +6,13 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Version-specific upgrade instructions **(PREMIUM SELF)** +NOTE: +We're in the process of merging all the version-specific upgrade information +into a single page. See [epic 9581](https://gitlab.com/groups/gitlab-org/-/epics/9581) +for more information. For the time being, visit the +[general upgrade page](../../../update/index.md) +for the newest Geo version-specific upgrade instructions. + Review this page for upgrade instructions for your version. These steps accompany the [general steps](upgrading_the_geo_sites.md#general-upgrade-steps) for upgrading Geo sites. @@ -14,7 +21,7 @@ for upgrading Geo sites. [Geo proxying](../secondary_proxy/index.md) was [enabled by default for different URLs](https://gitlab.com/gitlab-org/gitlab/-/issues/346112) in 15.1. This may be a breaking change. If needed, you may [disable Geo proxying](../secondary_proxy/index.md#disable-geo-proxying). -If you are using SAML with different URLs, there is a [known issue which requires proxying to be disabled](https://gitlab.com/gitlab-org/gitlab/-/issues/377372). +If you are using SAML with different URLs, you must modify your SAML configuration and your Identity Provider configuration. For more information, see the [Geo with Single Sign-On (SSO) documentation](single_sign_on.md). ## Upgrading to 14.9 diff --git a/doc/administration/get_started.md b/doc/administration/get_started.md index 4099ddc16f8..2b9b5291c54 100644 --- a/doc/administration/get_started.md +++ b/doc/administration/get_started.md @@ -36,7 +36,7 @@ Watch an overview of [groups and projects](https://www.youtube.com/watch?v=cqb2m Get started: -- Create a [project](../user/project/working_with_projects.md#create-a-project). +- Create a [project](../user/project/index.md#create-a-project). - Create a [group](../user/group/manage.md#create-a-group). - [Add members](../user/group/manage.md#add-users-to-a-group) to the group. - Create a [subgroup](../user/group/subgroups/index.md#create-a-subgroup). @@ -55,7 +55,7 @@ Get started: You may need to import projects from external sources like GitHub, Bitbucket, or another instance of GitLab. Many external sources can be imported into GitLab. -- Review the [GitLab projects documentation](../user/project/index.md#project-integrations). +- Review the [GitLab projects documentation](../user/project/index.md). - Consider [repository mirroring](../user/project/repository/mirror/index.md)—an [alternative to project migrations](../ci/ci_cd_for_external_repos/index.md). - Check out our [migration index](../user/project/import/index.md) for documentation on common migration paths. - Schedule your project exports with our [import/export API](../api/project_import_export.md#schedule-an-export). diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 32b44552b1a..b2b6962a222 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -570,9 +570,9 @@ To migrate to Gitaly Cluster: [repository storage recommendations](praefect.md#repository-storage-recommendations). 1. Create and configure [Gitaly Cluster](praefect.md). 1. Configure the existing Gitaly instance [to use TCP](praefect.md#use-tcp-for-existing-gitlab-instances), if not already configured that way. -1. [Move the repositories](../operations/moving_repositories.md#move-repositories). To migrate to +1. [Move the repositories](../operations/moving_repositories.md#moving-repositories). To migrate to Gitaly Cluster, existing repositories stored outside Gitaly Cluster must be moved. There is no - automatic migration but the moves can be scheduled with the GitLab API. + automatic migration, but the moves can be scheduled with the GitLab API. Even if you don't use the `default` repository storage, you must ensure it is configured. [Read more about this limitation](configure_gitaly.md#gitlab-requires-a-default-repository-storage). @@ -583,7 +583,7 @@ If the limitations and tradeoffs of Gitaly Cluster are found to be not suitable off Gitaly Cluster to a sharded Gitaly instance: 1. Create and configure a new [Gitaly server](configure_gitaly.md#run-gitaly-on-its-own-server). -1. [Move the repositories](../operations/moving_repositories.md#move-repositories) to the newly created storage. You can +1. [Move the repositories](../operations/moving_repositories.md#moving-repositories) to the newly created storage. You can move them by shard or by group, which gives you the opportunity to spread them over multiple Gitaly servers. ## Direct access to Git in GitLab diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index 9cc93b21ae9..8b4750b299d 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -177,9 +177,10 @@ database on the same PostgreSQL server if using [Geo](../geo/index.md). The replication state is internal to each instance of GitLab and should not be replicated. -These instructions help set up a single PostgreSQL database, which creates a single point of -failure. To avoid this, you can configure your own clustered PostgreSQL. Support for PostgreSQL replication and failover using Omnibus GitLab is being tracked in -[a relevant epic](https://gitlab.com/groups/gitlab-org/-/epics/7814). +These instructions help set up a single PostgreSQL database, which creates a single point of failure. To avoid this, you can configure your own clustered +PostgreSQL. Support for PostgreSQL replication and failover using Omnibus GitLab is proposed in [epic 7814](https://gitlab.com/groups/gitlab-org/-/epics/7814). +Clustered database support for other databases (for example, Praefect and Geo databases) is proposed in +[issue 7292](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7292). The following options are available: diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index 7f5d4b9e443..4b8ed74ec62 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -169,7 +169,7 @@ Confirm the following are all true: - When any user adds or modifies a file from the repository using the GitLab UI, it immediately fails with a red `401 Unauthorized` banner. -- Creating a new project and [initializing it with a README](../../user/project/working_with_projects.md#create-a-blank-project) +- Creating a new project and [initializing it with a README](../../user/project/index.md#create-a-blank-project) successfully creates the project but doesn't create the README. - When [tailing the logs](https://docs.gitlab.com/omnibus/settings/logs.html#tail-logs-in-a-console-on-the-server) on a Gitaly client and reproducing the error, you get `401` errors diff --git a/doc/administration/housekeeping.md b/doc/administration/housekeeping.md index 584f06ef537..e1b26595bc4 100644 --- a/doc/administration/housekeeping.md +++ b/doc/administration/housekeeping.md @@ -34,7 +34,7 @@ Gitaly can perform housekeeping tasks in a Git repository in two ways: The "eager" housekeeping strategy executes housekeeping tasks in a repository independent of the repository state. This is the default strategy as used by the -[manual trigger](#manual-trigger) and the [push-based trigger](#push-based-trigger). +[manual trigger](#manual-trigger) and the push-based trigger. The eager housekeeping strategy is controlled by the GitLab application. Depending on the trigger that caused the housekeeping job to run, GitLab asks @@ -45,20 +45,14 @@ be slow. ### Heuristical housekeeping -> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/2634) in GitLab 14.9 for the [manual trigger](#manual-trigger) and the [push-based trigger](#push-based-trigger) [with a flag](feature_flags.md) named `optimized_housekeeping`. Enabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/2634) in GitLab 14.9 for the [manual trigger](#manual-trigger) and the push-based trigger [with a flag](feature_flags.md) named `optimized_housekeeping`. Enabled by default. > - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/353607) in GitLab 14.10. - -FLAG: -On self-managed GitLab, by default this feature is available. To hide the feature, ask an administrator to [disable the feature flag](feature_flags.md) named `optimize_repository`. - -To make it available, ask an administrator to [enable the feature flag](feature_flags.md) named `optimized_housekeeping`. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/107661) in GitLab 15.8. Feature flag `optimized_housekeeping` removed. The heuristical (or "opportunistic") housekeeping strategy analyzes the repository's state and executes housekeeping tasks only when it finds one or more data structures are insufficiently optimized. This is the strategy used by -[scheduled housekeeping](#scheduled-housekeeping). It can optionally be enabled -for the [manual trigger](#manual-trigger) and the [push-based trigger](#push-based-trigger) -by enabling the `optimized_housekeeping` feature flag. +[scheduled housekeeping](#scheduled-housekeeping). Heuristical housekeeping uses the following information to decide on the tasks it needs to run: @@ -99,7 +93,7 @@ There are different ways in which GitLab runs housekeeping tasks: - A project's administrator can [manually trigger](#manual-trigger) repository housekeeping tasks. -- GitLab can automatically schedule housekeeping tasks [after a number of Git pushes](#push-based-trigger). +- GitLab can automatically schedule housekeeping tasks after a number of Git pushes. - GitLab can [schedule a job](#scheduled-housekeeping) that runs housekeeping tasks for all repositories in a configurable time frame. @@ -120,65 +114,10 @@ To trigger housekeeping tasks manually: 1. Select **Run housekeeping**. This starts an asynchronous background worker for the project's repository. The -background worker executes `git gc`, which performs a number of optimizations. - -<!--- start_remove The following content will be removed on remove_date: '2023-04-22' --> - -### Push-based trigger - -FLAG: -On self-managed GitLab, by default this feature is not available and superseded by [heuristical housekeeping](#heuristical-housekeeping). It is planned to be removed in 15.8. To enable the feature, ask an administrator to [disable the feature flag](feature_flags.md) named `optimize_repository`. - -GitLab automatically runs repository housekeeping tasks after a configured -number of pushes: - -- [`git gc`](https://git-scm.com/docs/git-gc) runs a number of housekeeping tasks such as: - - Compressing Git objects to reduce disk space and increase performance. - - Removing unreachable objects that may have been created from changes to the repository, like force-overwriting branches. -- [`git repack`](https://git-scm.com/docs/git-repack) either: - - Runs an incremental repack, according to a [configured period](#configure-push-based-maintenance). This - packs all loose objects into a new packfile and prunes the now-redundant loose objects. - - Runs a full repack, according to a [configured period](#configure-push-based-maintenance). This repacks all - packfiles and loose objects into a single new packfile, and deletes the old now-redundant loose - objects and packfiles. It also optionally creates bitmaps for the new packfile. -- [`git pack-refs`](https://git-scm.com/docs/git-pack-refs) compresses references - stored as loose files into a single file. - -#### Configure push-based maintenance - -You can change how often these tasks run when pushes occur, or you can turn -them off entirely: - -1. On the top bar, select **Main menu > Admin**. -1. On the left sidebar, select **Settings > Repository**. -1. Expand **Repository maintenance**. -1. In the **Housekeeping** section, configure the housekeeping options. -1. Select **Save changes**. - -The following housekeeping options are available: - -- **Enable automatic repository housekeeping**: Regularly run housekeeping tasks. If you - keep this setting disabled for a long time, Git repository access on your GitLab server becomes - slower and your repositories use more disk space. -- **Incremental repack period**: Number of Git pushes after which an incremental `git repack` is - run. -- **Full repack period**: Number of Git pushes after which a full `git repack` is run. -- **Git GC period**: Number of Git pushes after which `git gc` is run. - -As an example, see the following scenario: - -- Incremental repack period: 10. -- Full repack period: 50. -- Git GC period: 200. - -When the: - -- `pushes_since_gc` value is 50, a `repack -A -l -d --pack-kept-objects` runs. -- `pushes_since_gc` value is 200, a `git gc` runs. +background worker asks Gitaly to perform a number of optimizations. Housekeeping also [removes unreferenced LFS files](../raketasks/cleanup.md#remove-unreferenced-lfs-files) -from your project on the same schedule as the `git gc` operation, freeing up storage space for your -project. +from your project every `200` push, freeing up storage space for your project. ### Scheduled housekeeping diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index 826340ad967..2093d55d8c0 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -304,7 +304,12 @@ gitlab_rails['incoming_email_mailbox_name'] = "inbox" # The IDLE command timeout. gitlab_rails['incoming_email_idle_timeout'] = 60 +# If you are using Microsoft Graph instead of IMAP, set this to false to retain +# messages in the inbox because deleted messages are auto-expunged after some time. +gitlab_rails['incoming_email_delete_after_delivery'] = true + # Whether to expunge (permanently remove) messages from the mailbox when they are marked as deleted after delivery +# Only applies to IMAP. Microsoft Graph will auto-expunge any deleted messages. gitlab_rails['incoming_email_expunge_deleted'] = true ``` @@ -342,7 +347,12 @@ incoming_email: # The IDLE command timeout. idle_timeout: 60 + # If you are using Microsoft Graph instead of IMAP, set this to false to retain + # messages in the inbox because deleted messages are auto-expunged after some time. + delete_after_delivery: true + # Whether to expunge (permanently remove) messages from the mailbox when they are marked as deleted after delivery + # Only applies to IMAP. Microsoft Graph will auto-expunge any deleted messages. expunge_deleted: true ``` @@ -386,7 +396,12 @@ gitlab_rails['incoming_email_mailbox_name'] = "inbox" # The IDLE command timeout. gitlab_rails['incoming_email_idle_timeout'] = 60 +# If you are using Microsoft Graph instead of IMAP, set this to false if you want to retain +# messages in the inbox because deleted messages are auto-expunged after some time. +gitlab_rails['incoming_email_delete_after_delivery'] = true + # Whether to expunge (permanently remove) messages from the mailbox when they are marked as deleted after delivery +# Only applies to IMAP. Microsoft Graph will auto-expunge any deleted messages. gitlab_rails['incoming_email_expunge_deleted'] = true ``` @@ -424,7 +439,12 @@ incoming_email: # The IDLE command timeout. idle_timeout: 60 + # If you are using Microsoft Graph instead of IMAP, set this to falseto retain + # messages in the inbox because deleted messages are auto-expunged after some time. + delete_after_delivery: true + # Whether to expunge (permanently remove) messages from the mailbox when they are marked as deleted after delivery + # Only applies to IMAP. Microsoft Graph will auto-expunge any deleted messages. expunge_deleted: true ``` @@ -467,6 +487,7 @@ gitlab_rails['incoming_email_port'] = 993 gitlab_rails['incoming_email_ssl'] = true # Whether to expunge (permanently remove) messages from the mailbox when they are marked as deleted after delivery +# Only applies to IMAP. Microsoft Graph will auto-expunge any deleted messages. gitlab_rails['incoming_email_expunge_deleted'] = true ``` @@ -497,6 +518,10 @@ incoming_email: # Whether the IMAP server uses SSL ssl: true + # If you are using Microsoft Graph instead of IMAP, set this to false to retain + # messages in the inbox since deleted messages are auto-expunged after some time. + delete_after_delivery: true + # Whether to expunge (permanently remove) messages from the mailbox when they are marked as deleted after delivery expunge_deleted: true ``` @@ -557,6 +582,10 @@ incoming_email: # Whether the IMAP server uses SSL ssl: true + # If you are using Microsoft Graph instead of IMAP, set this to false to retain + # messages in the inbox since deleted messages are auto-expunged after some time. + delete_after_delivery: true + # Whether to expunge (permanently remove) messages from the mailbox when they are marked as deleted after delivery expunge_deleted: true ``` @@ -812,6 +841,7 @@ gitlab_rails['incoming_email_address'] = "incoming+%{key}@example.onmicrosoft.co # Email account username gitlab_rails['incoming_email_email'] = "incoming@example.onmicrosoft.com" +gitlab_rails['incoming_email_delete_after_delivery'] = false gitlab_rails['incoming_email_inbox_method'] = 'microsoft_graph' gitlab_rails['incoming_email_inbox_options'] = { diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index 59746fc0f07..5215f0eaed2 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -912,7 +912,7 @@ An upper and lower limit applies to each of these: The lower limits result in additional diffs being collapsed. The higher limits prevent any more changes from rendering. For more information about these limits, -[read the development documentation](../development/diffs.md#diff-limits). +[read the development documentation](../development/merge_request_concepts/diffs/index.md#diff-limits). ### Merge request reports size limit diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index fb2f73876e8..816cfcb508d 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -16,26 +16,85 @@ finishes. This feature is enabled by default in all GitLab installations. To disable artifacts site-wide: -**In Omnibus installations:** +::Tabs -1. Edit `/etc/gitlab/gitlab.rb` and add the following line: +:::TabTitle Linux package (Omnibus) + +1. Edit `/etc/gitlab/gitlab.rb`: ```ruby gitlab_rails['artifacts_enabled'] = false ``` -1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +:::TabTitle Helm chart (Kubernetes) + +1. Export the Helm values: + + ```shell + helm get values gitlab > gitlab_values.yaml + ``` + +1. Edit `gitlab_values.yaml`: + + ```yaml + global: + appConfig: + artifacts: + enabled: false + ``` + +1. Save the file and apply the new values: + + ```shell + helm upgrade -f gitlab_values.yaml gitlab gitlab/gitlab + ``` + +:::TabTitle Docker + +1. Edit `docker-compose.yml`: -**In installations from source:** + ```yaml + version: "3.6" + services: + gitlab: + environment: + GITLAB_OMNIBUS_CONFIG: | + gitlab_rails['artifacts_enabled'] = false + ``` -1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: +1. Save the file and restart GitLab: + + ```shell + docker compose up -d + ``` + +:::TabTitle Self-compiled (source) + +1. Edit `/home/git/gitlab/config/gitlab.yml`: ```yaml - artifacts: - enabled: false + production: &base + artifacts: + enabled: false + ``` + +1. Save the file and restart GitLab: + + ```shell + # For systems running systemd + sudo systemctl restart gitlab.target + + # For systems running SysV init + sudo service gitlab restart ``` -1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. +::EndTabs ## Storing job artifacts @@ -48,45 +107,63 @@ Most artifacts are compressed by GitLab Runner before being sent to the coordina ### Using local storage -To change the location where the artifacts are stored locally, follow the steps -below. +If you're using the Linux package or have a self-compiled installation, you +can change the location where the artifacts are stored locally. + +NOTE: +For Docker installations, you can change the path where your data is mounted. +For the Helm chart, use +[object storage](https://docs.gitlab.com/charts/advanced/external-object-storage/). -**In Omnibus installations:** +::Tabs -_The artifacts are stored by default in -`/var/opt/gitlab/gitlab-rails/shared/artifacts`._ +:::TabTitle Linux package (Omnibus) -1. To change the storage path for example to `/mnt/storage/artifacts`, edit +The artifacts are stored by default in `/var/opt/gitlab/gitlab-rails/shared/artifacts`. + +1. To change the storage path, for example to `/mnt/storage/artifacts`, edit `/etc/gitlab/gitlab.rb` and add the following line: ```ruby gitlab_rails['artifacts_path'] = "/mnt/storage/artifacts" ``` -1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` -**In installations from source:** +:::TabTitle Self-compiled (source) -_The artifacts are stored by default in -`/home/git/gitlab/shared/artifacts`._ +The artifacts are stored by default in `/home/git/gitlab/shared/artifacts`. -1. To change the storage path for example to `/mnt/storage/artifacts`, edit +1. To change the storage path, for example to `/mnt/storage/artifacts`, edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: ```yaml - artifacts: - enabled: true - path: /mnt/storage/artifacts + production: &base + artifacts: + enabled: true + path: /mnt/storage/artifacts ``` -1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. +1. Save the file and restart GitLab: + + ```shell + # For systems running systemd + sudo systemctl restart gitlab.target + + # For systems running SysV init + sudo service gitlab restart + ``` + +::EndTabs ### Using object storage If you don't want to use the local disk where GitLab is installed to store the artifacts, you can use an object storage like AWS S3 instead. -This configuration relies on valid AWS credentials to be configured already. -Use an object storage option like AWS S3 to store job artifacts. If you configure GitLab to store artifacts on object storage, you may also want to [eliminate local disk usage for job logs](job_logs.md#prevent-local-disk-usage). @@ -96,149 +173,110 @@ WARNING: In a multi-server setup you must use one of the options to [eliminate local disk usage for job logs](job_logs.md#prevent-local-disk-usage), or job logs could be lost. -[Read more about using object storage with GitLab](object_storage.md). - -#### Object Storage Settings - In GitLab 13.2 and later, you should use the [consolidated object storage settings](object_storage.md#consolidated-object-storage-configuration). -This section describes the earlier configuration format. -For source installations the following settings are nested under `artifacts:` -and then `object_store:`. On Omnibus GitLab installs they are prefixed by -`artifacts_object_store_`. +### Migrating to object storage -| Setting | Default | Description | -|---------------------|---------|-------------| -| `enabled` | `false` | Enable or disable object storage. | -| `remote_directory` | | The bucket name where Artifacts are stored. Use the name only, do not include the path. | -| `proxy_download` | `false` | Set to `true` to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data. | -| `connection` | | Various connection options described below. | +You can migrate the job artifacts from local storage to object storage. The +processing is done in a background worker and requires **no downtime**. -#### Connection settings +1. [Configure the object storage](#using-object-storage). +1. Migrate the artifacts: -See [the available connection settings for different providers](object_storage.md#connection-settings). + ::Tabs -**In Omnibus installations:** + :::TabTitle Linux package (Omnibus) -_The artifacts are stored by default in -`/var/opt/gitlab/gitlab-rails/shared/artifacts`._ + ```shell + sudo gitlab-rake gitlab:artifacts:migrate + ``` -1. Edit `/etc/gitlab/gitlab.rb` and add the following lines, substituting - the values you want: + :::TabTitle Docker - ```ruby - gitlab_rails['artifacts_enabled'] = true - gitlab_rails['artifacts_object_store_enabled'] = true - gitlab_rails['artifacts_object_store_remote_directory'] = "artifacts" - gitlab_rails['artifacts_object_store_connection'] = { - 'provider' => 'AWS', - 'region' => 'eu-central-1', - 'aws_access_key_id' => 'AWS_ACCESS_KEY_ID', - 'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY' - } + ```shell + sudo docker exec -t <container name> gitlab-rake gitlab:artifacts:migrate ``` - NOTE: - If you're using AWS IAM profiles, omit the AWS access key and secret access - key/value pairs. For example: + :::TabTitle Self-compiled (source) - ```ruby - gitlab_rails['artifacts_object_store_connection'] = { - 'provider' => 'AWS', - 'region' => 'eu-central-1', - 'use_iam_profile' => true - } + ```shell + sudo -u git -H bundle exec rake gitlab:artifacts:migrate RAILS_ENV=production ``` -1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. [Migrate any existing local artifacts to the object storage](#migrating-to-object-storage). + ::EndTabs -**In installations from source:** +1. Optional. Track the progress and verify that all job artifacts migrated + successfully using the PostgreSQL console. + 1. Open a PostgreSQL console: -_The artifacts are stored by default in -`/home/git/gitlab/shared/artifacts`._ + ::Tabs -1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following - lines: + :::TabTitle Linux package (Omnibus) - ```yaml - artifacts: - enabled: true - object_store: - enabled: true - remote_directory: "artifacts" # The bucket name - connection: - provider: AWS # Only AWS supported at the moment - aws_access_key_id: AWS_ACCESS_KEY_ID - aws_secret_access_key: AWS_SECRET_ACCESS_KEY - region: eu-central-1 - ``` + ```shell + sudo gitlab-psql + ``` -1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. -1. [Migrate any existing local artifacts to the object storage](#migrating-to-object-storage). + :::TabTitle Docker -### Migrating to object storage + ```shell + sudo docker exec -it <container_name> /bin/bash + gitlab-psql + ``` -After [configuring the object storage](#using-object-storage), use the following task to -migrate existing job artifacts from the local storage to the remote storage. -The processing is done in a background worker and requires **no downtime**. + :::TabTitle Self-compiled (source) -**In Omnibus installations:** + ```shell + sudo -u git -H psql -d gitlabhq_production + ``` -```shell -gitlab-rake gitlab:artifacts:migrate -``` + ::EndTabs -**In installations from source:** + 1. Verify that all packages migrated to object storage with the following + SQL query. The number of `objectstg` should be the same as `total`: -```shell -sudo -u git -H bundle exec rake gitlab:artifacts:migrate RAILS_ENV=production -``` + ```shell + gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM ci_job_artifacts; -You can optionally track progress and verify that all job artifacts migrated successfully using the -[PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database): + total | filesystem | objectstg + ------+------------+----------- + 19 | 0 | 19 + ``` -- `sudo gitlab-rails dbconsole` for Omnibus GitLab 14.1 and earlier. -- `sudo gitlab-rails dbconsole --database main` for Omnibus GitLab 14.2 and later. -- `sudo -u git -H psql -d gitlabhq_production` for source-installed instances. +1. Verify that there are no files on disk in the `artifacts` directory: -Verify `objectstg` below (where `store=2`) has count of all job artifacts: + ::Tabs -```shell -gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM ci_job_artifacts; + :::TabTitle Linux package (Omnibus) -total | filesystem | objectstg -------+------------+----------- - 19 | 0 | 19 -``` + ```shell + sudo find /var/opt/gitlab/gitlab-rails/shared/artifacts -type f | grep -v tmp | wc -l + ``` -Verify that there are no files on disk in the `artifacts` folder: + :::TabTitle Docker -```shell -sudo find /var/opt/gitlab/gitlab-rails/shared/artifacts -type f | grep -v tmp | wc -l -``` + Assuming you mounted `/var/opt/gitlab` to `/srv/gitlab`: -In some cases, you need to run the [orphan artifact file cleanup Rake task](../raketasks/cleanup.md#remove-orphan-artifact-files) -to clean up orphaned artifacts. + ```shell + sudo find /srv/gitlab/gitlab-rails/shared/artifacts -type f | grep -v tmp | wc -l + ``` -WARNING: -JUnit test report artifact (`junit.xml.gz`) migration -[was not supported until GitLab 12.8](https://gitlab.com/gitlab-org/gitlab/-/issues/27698#note_317190991) -by the `gitlab:artifacts:migrate` Rake task. + :::TabTitle Self-compiled (source) -### Migrating from object storage to local storage + ```shell + sudo find /home/git/gitlab/shared/artifacts -type f | grep -v tmp | wc -l + ``` -**In Omnibus installations:** + ::EndTabs -To migrate back to local storage: +In some cases, you need to run the [orphan artifact file cleanup Rake task](../raketasks/cleanup.md#remove-orphan-artifact-files) +to clean up orphaned artifacts. -1. Run `gitlab-rake gitlab:artifacts:migrate_to_local`. -1. Disable object storage for artifacts in `gitlab.rb`: - - Set `gitlab_rails['artifacts_object_store_enabled'] = false`. - - Comment out all other `artifacts_object_store` settings, including the entire - `artifacts_object_store_connection` section, including the closing `}`. -1. [Reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure). +### Migrating from object storage to local storage + +To migrate back to local storage, you must +[selectively disable the artifacts storage](object_storage.md#selectively-disabling-object-storage). ## Expiring artifacts @@ -247,36 +285,93 @@ an expiry for the artifacts, they are marked for deletion right after that date Otherwise, they expire per the [default artifacts expiration setting](../user/admin_area/settings/continuous_integration.md). Artifacts are cleaned up by the `expire_build_artifacts_worker` cron job which Sidekiq -runs every 7 minutes (`*/7 * * * *`). +runs every 7 minutes (`*/7 * * * *` in [Cron](../topics/cron/index.md) syntax). + +To change the default schedule on which the artifacts are expired: -To change the default schedule on which the artifacts are expired, follow the -steps below. +::Tabs -**In Omnibus installations:** +:::TabTitle Linux package (Omnibus) -1. Edit `/etc/gitlab/gitlab.rb` and add the following line (or uncomment it if it already exists and is commented out), substituting - your schedule in cron syntax: +1. Edit `/etc/gitlab/gitlab.rb` and add the following line (or uncomment it if + it already exists and is commented out), substituting your schedule in cron + syntax: ```ruby gitlab_rails['expire_build_artifacts_worker_cron'] = "*/7 * * * *" ``` -1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and reconfigure GitLab: -**In installations from source:** + ```shell + sudo gitlab-ctl reconfigure + ``` + +:::TabTitle Helm chart (Kubernetes) + +1. Export the Helm values: -1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following - lines: + ```shell + helm get values gitlab > gitlab_values.yaml + ``` + +1. Edit `gitlab_values.yaml`: ```yaml - expire_build_artifacts_worker: - cron: "*/7 * * * *" + global: + appConfig: + cron_jobs: + expire_build_artifacts_worker: + cron: "*/7 * * * *" ``` -1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. +1. Save the file and apply the new values: + + ```shell + helm upgrade -f gitlab_values.yaml gitlab gitlab/gitlab + ``` + +:::TabTitle Docker + +1. Edit `docker-compose.yml`: + + ```yaml + version: "3.6" + services: + gitlab: + environment: + GITLAB_OMNIBUS_CONFIG: | + gitlab_rails['expire_build_artifacts_worker_cron'] = "*/7 * * * *" + ``` -If the `expire` directive is not set explicitly in your pipeline, artifacts expire per the -default artifacts expiration setting, which you can find in the [CI/CD Administration settings](../user/admin_area/settings/continuous_integration.md). +1. Save the file and restart GitLab: + + ```shell + docker compose up -d + ``` + +:::TabTitle Self-compiled (source) + +1. Edit `/home/git/gitlab/config/gitlab.yml`: + + ```yaml + production: &base + cron_jobs: + expire_build_artifacts_worker: + cron: "*/7 * * * *" + ``` + +1. Save the file and restart GitLab: + + ```shell + # For systems running systemd + sudo systemctl restart gitlab.target + + # For systems running SysV init + sudo service gitlab restart + ``` + +::EndTabs ## Set the maximum file size of the artifacts @@ -373,13 +468,41 @@ these artifacts are not processed by the new housekeeping jobs. You can check the database to confirm if your instance has artifacts with the `unknown` status: -1. Start a database console, on Omnibus: +1. Start a database console: + + ::Tabs + + :::TabTitle Linux package (Omnibus) ```shell sudo gitlab-psql ``` -1. Run this query: + :::TabTitle Helm chart (Kubernetes) + + ```shell + # Find the toolbox pod + kubectl --namespace <namespace> get pods -lapp=toolbox + # Connect to the PostgreSQL console + kubectl exec -it <toolbox-pod-name> -- /srv/gitlab/bin/rails dbconsole --include-password --database main + ``` + + :::TabTitle Docker + + ```shell + sudo docker exec -it <container_name> /bin/bash + gitlab-psql + ``` + + :::TabTitle Self-compiled (source) + + ```shell + sudo -u git -H psql -d gitlabhq_production + ``` + + ::EndTabs + +1. Run the following query: ```sql select expire_at, file_type, locked, count(*) from ci_job_artifacts @@ -652,7 +775,7 @@ review: {"error":"MissingRegion: could not find region configuration","level":"error","msg":"error uploading S3 session","time":"2021-03-16T22:10:55-04:00"} ``` -In both cases, you might need to add `region` to the job artifact [object storage configuration](#connection-settings). +In both cases, you might need to add `region` to the job artifact [object storage configuration](object_storage.md). ### Job artifact upload fails with `500 Internal Server Error (Missing file)` diff --git a/doc/administration/job_logs.md b/doc/administration/job_logs.md index 9b25c70716b..c8702260ccb 100644 --- a/doc/administration/job_logs.md +++ b/doc/administration/job_logs.md @@ -7,8 +7,6 @@ type: reference # Job logs **(FREE SELF)** -> [Renamed from job traces to job logs](https://gitlab.com/gitlab-org/gitlab/-/issues/29121) in GitLab 12.5. - Job logs are sent by a runner while it's processing a job. You can see logs in job pages, pipelines, email notifications, and so on. @@ -23,82 +21,125 @@ In the following table you can see the phases a log goes through: | 2: archiving | archived log | After a job is finished | Sidekiq moves log to artifacts folder | `#{ROOT_PATH}/gitlab-rails/shared/artifacts/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log` | | 3: uploading | archived log | After a log is archived | Sidekiq moves archived log to [object storage](#uploading-logs-to-object-storage) (if configured) | `#{bucket_name}/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log` | -The `ROOT_PATH` varies per environment. For Omnibus GitLab it -would be `/var/opt/gitlab`, and for installations from source -it would be `/home/git/gitlab`. +The `ROOT_PATH` varies per environment: + +- For the Linux package it's `/var/opt/gitlab`. +- For self-compiled installations it's `/home/git/gitlab`. ## Changing the job logs local location -To change the location where the job logs are stored, follow the steps below. +NOTE: +For Docker installations, you can change the path where your data is mounted. +For the Helm chart, use object storage. -**In Omnibus installations:** +To change the location where the job logs are stored: -1. Edit `/etc/gitlab/gitlab.rb` and add or amend the following line: +::Tabs - ```ruby - gitlab_ci['builds_directory'] = '/mnt/to/gitlab-ci/builds' - ``` +:::TabTitle Linux package (Omnibus) -1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the - changes to take effect. +1. Optional. If you have existing job logs, pause continuous integration data + processing by temporarily stopping Sidekiq: -Alternatively, if you have existing job logs you can follow -these steps to move the logs to a new location without losing any data. + ```shell + sudo gitlab-ctl stop sidekiq + ``` -1. Pause continuous integration data processing by updating this setting in `/etc/gitlab/gitlab.rb`. - Jobs in progress are not affected, based on how [data flow](#data-flow) works. +1. Set the new storage location in `/etc/gitlab/gitlab.rb`: ```ruby - sidekiq['queue_selector'] = true - sidekiq['queue_groups'] = [ - "feature_category!=continuous_integration" - ] + gitlab_ci['builds_directory'] = '/mnt/gitlab-ci/builds' ``` -1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the - changes to take effect. -1. Set the new storage location in `/etc/gitlab/gitlab.rb`: +1. Save the file and reconfigure GitLab: - ```ruby - gitlab_ci['builds_directory'] = '/mnt/to/gitlab-ci/builds' + ```shell + sudo gitlab-ctl reconfigure ``` -1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the - changes to take effect. 1. Use `rsync` to move job logs from the current location to the new location: ```shell - sudo rsync -avzh --remove-source-files --ignore-existing --progress /var/opt/gitlab/gitlab-ci/builds/ /mnt/to/gitlab-ci/builds` + sudo rsync -avzh --remove-source-files --ignore-existing --progress /var/opt/gitlab/gitlab-ci/builds/ /mnt/gitlab-ci/builds/ ``` Use `--ignore-existing` so you don't override new job logs with older versions of the same log. -1. Resume continuous integration data processing by editing `/etc/gitlab/gitlab.rb` and removing the `sidekiq` setting you updated earlier. -1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the - changes to take effect. + +1. If you opted to pause the continuous integration data processing, you can + start Sidekiq again: + + ```shell + sudo gitlab-ctl start sidekiq + ``` + 1. Remove the old job logs storage location: ```shell - sudo rm -rf /var/opt/gitlab/gitlab-ci/builds` + sudo rm -rf /var/opt/gitlab/gitlab-ci/builds ``` -**In installations from source:** +:::TabTitle Self-compiled (source) + +1. Optional. If you have existing job logs, pause continuous integration data + processing by temporarily stopping Sidekiq: + + ```shell + # For systems running systemd + sudo systemctl stop gitlab-sidekiq + + # For systems running SysV init + sudo service gitlab stop + ``` -1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: +1. Edit `/home/git/gitlab/config/gitlab.yml` to set the new storage location: ```yaml - gitlab_ci: - # The location where build logs are stored (default: builds/). - # Relative paths are relative to Rails.root. - builds_path: path/to/builds/ + production: &base + gitlab_ci: + builds_path: /mnt/gitlab-ci/builds + ``` + +1. Save the file and restart GitLab: + + ```shell + # For systems running systemd + sudo systemctl restart gitlab.target + + # For systems running SysV init + sudo service gitlab restart + ``` + +1. Use `rsync` to move job logs from the current location to the new location: + + ```shell + sudo rsync -avzh --remove-source-files --ignore-existing --progress /home/git/gitlab/builds/ /mnt/gitlab-ci/builds/ + ``` + + Use `--ignore-existing` so you don't override new job logs with older versions of the same log. + +1. If you opted to pause the continuous integration data processing, you can + start Sidekiq again: + + ```shell + # For systems running systemd + sudo systemctl start gitlab-sidekiq + + # For systems running SysV init + sudo service gitlab start + ``` + +1. Remove the old job logs storage location: + + ```shell + sudo rm -rf /home/git/gitlab/builds ``` -1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes - to take effect. +::EndTabs ## Uploading logs to object storage Archived logs are considered as [job artifacts](job_artifacts.md). -Therefore, when you [set up the object storage integration](job_artifacts.md#object-storage-settings), +Therefore, when you [set up the object storage integration](job_artifacts.md#using-object-storage), job logs are automatically migrated to it along with the other job artifacts. See "Phase 3: uploading" in [Data flow](#data-flow) to learn about the process. @@ -118,19 +159,45 @@ There isn't a way to automatically expire old job logs, but it's safe to remove them if they're taking up too much space. If you remove the logs manually, the job output in the UI is empty. -For example, to delete all job logs older than 60 days, run the following from a shell in your GitLab instance: +For example, to delete all job logs older than 60 days, run the following +command from a shell in your GitLab instance. + +NOTE: +For the Helm chart, use the storage management tools provided with your object +storage. WARNING: -This command permanently deletes the log files and is irreversible. +The following command permanently deletes the log files and is irreversible. + +::Tabs + +:::TabTitle Linux package (Omnibus) ```shell find /var/opt/gitlab/gitlab-rails/shared/artifacts -name "job.log" -mtime +60 -delete ``` -NOTE: -After execution, broken file references can be reported when running -[`sudo gitlab-rake gitlab:artifacts:check`](raketasks/check.md#uploaded-files-integrity). -For more information, see [delete references to missing artifacts](raketasks/check.md#delete-references-to-missing-artifacts). +:::TabTitle Docker + +Assuming you mounted `/var/opt/gitlab` to `/srv/gitlab`: + +```shell +find /srv/gitlab/gitlab-rails/shared/artifacts -name "job.log" -mtime +60 -delete +``` + +:::TabTitle Self-compiled (source) + +```shell +find /home/git/gitlab/shared/artifacts -name "job.log" -mtime +60 -delete +``` + +::EndTabs + +After the logs are deleted, you can find any broken file references by running +the Rake task that checks the +[integrity of the uploaded files](raketasks/check.md#uploaded-files-integrity). +For more information, see how to +[delete references to missing artifacts](raketasks/check.md#delete-references-to-missing-artifacts). ## Incremental logging architecture @@ -140,19 +207,48 @@ For more information, see [delete references to missing artifacts](raketasks/che > - [Recommended for production use with AWS S3](https://gitlab.com/gitlab-org/gitlab/-/issues/273498) in GitLab 13.7. > - To use in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-incremental-logging). -By default job logs are sent from the GitLab Runner in chunks and cached temporarily on disk -in `/var/opt/gitlab/gitlab-ci/builds` by Omnibus GitLab. After the job completes, -a background job archives the job log. The log is moved to `/var/opt/gitlab/gitlab-rails/shared/artifacts/` -by default, or to object storage if configured. - -In a [scaled-out architecture](reference_architectures/index.md) with Rails and Sidekiq running on more than one -server, these two locations on the file system have to be shared using NFS. +By default, job logs are sent from the GitLab Runner in chunks and cached +temporarily on disk. After the job completes, a background job archives the job +log. The log is moved to the artifacts directory by default, or to object +storage if configured. -To eliminate both file system requirements: +In a [scaled-out architecture](reference_architectures/index.md) with Rails and +Sidekiq running on more than one server, these two locations on the file system +have to be shared using NFS, which is not recommended. Instead: -1. Configure [object storage](job_artifacts.md#object-storage-settings) for storing archived job logs. +1. Configure [object storage](job_artifacts.md#using-object-storage) for storing archived job logs. 1. [Enable the incremental logging feature](#enable-or-disable-incremental-logging), which uses Redis instead of disk space for temporary caching of job logs. +### Enable or disable incremental logging + +Before you enable the feature flag: + +- Review [the limitations of incremental logging](#limitations). +- [Enable object storage](job_artifacts.md#using-object-storage). + +To enable incremental logging: + +1. Open a [Rails console](operations/rails_console.md#starting-a-rails-console-session). +1. Enable the feature flag: + + ```ruby + Feature.enable(:ci_enable_live_trace) + ``` + + Running jobs' logs continue to be written to disk, but new jobs use + incremental logging. + +To disable incremental logging: + +1. Open a [Rails console](operations/rails_console.md#starting-a-rails-console-session). +1. Disable the feature flag: + + ```ruby + Feature.disable(:ci_enable_live_trace) + ``` + + Running jobs continue to use incremental logging, but new jobs write to the disk. + ### Technical details The data flow is the same as described in the [data flow section](#data-flow) @@ -178,36 +274,7 @@ Here is the detailed data flow: ### Limitations - [Redis Cluster is not supported](https://gitlab.com/gitlab-org/gitlab/-/issues/224171). -- You must configure [object storage for CI/CD artifacts, logs, and builds](job_artifacts.md#object-storage-settings) +- You must configure [object storage for CI/CD artifacts, logs, and builds](job_artifacts.md#using-object-storage) before you enable the feature flag. After the flag is enabled, files cannot be written to disk, and there is no protection against misconfiguration. - There is [an epic tracking other potential limitations and improvements](https://gitlab.com/groups/gitlab-org/-/epics/3791). - -### Enable or disable incremental logging - -Incremental logging is under development, but [ready for production use as of GitLab 13.6](https://gitlab.com/groups/gitlab-org/-/epics/4275). It is -deployed behind a feature flag that is **disabled by default**. -[GitLab administrators with access to the GitLab Rails console](feature_flags.md) -can enable it. - -Before you enable the feature flag: - -- Review [the limitations of incremental logging](#limitations). -- [Enable object storage](job_artifacts.md#object-storage-settings). - -To enable incremental logging: - -```ruby -Feature.enable(:ci_enable_live_trace) -``` - -Running jobs' logs continue to be written to disk, but new jobs use -incremental logging. - -To disable incremental logging: - -```ruby -Feature.disable(:ci_enable_live_trace) -``` - -Running jobs continue to use incremental logging, but new jobs write to the disk. diff --git a/doc/administration/lfs/index.md b/doc/administration/lfs/index.md index cf80b05a5e0..302dc38cf28 100644 --- a/doc/administration/lfs/index.md +++ b/doc/administration/lfs/index.md @@ -10,206 +10,335 @@ disqus_identifier: 'https://docs.gitlab.com/ee/workflow/lfs/lfs_administration.h This page contains information about configuring Git LFS in self-managed GitLab instances. For user documentation about Git LFS, see [Git Large File Storage](../../topics/git/lfs/index.md). -LFS is enabled in GitLab self-managed instances by default. - -## Requirements +Prerequisites: - Users need to install [Git LFS client](https://git-lfs.github.com) version 1.0.1 or later. -## Configuration +## Enable or disable LFS + +LFS is enabled by default. To disable it: + +::Tabs + +:::TabTitle Linux package (Omnibus) + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + # Change to true to enable lfs - enabled by default if not defined + gitlab_rails['lfs_enabled'] = false + ``` + +1. Save the file and reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +:::TabTitle Helm chart (Kubernetes) + +1. Export the Helm values: + + ```shell + helm get values gitlab > gitlab_values.yaml + ``` + +1. Edit `gitlab_values.yaml`: + + ```yaml + global: + appConfig: + lfs: + enabled: false + ``` + +1. Save the file and apply the new values: + + ```shell + helm upgrade -f gitlab_values.yaml gitlab gitlab/gitlab + ``` + +:::TabTitle Docker + +1. Edit `docker-compose.yml`: + + ```yaml + version: "3.6" + services: + gitlab: + environment: + GITLAB_OMNIBUS_CONFIG: | + gitlab_rails['lfs_enabled'] = false + ``` + +1. Save the file and restart GitLab: + + ```shell + docker compose up -d + ``` + +:::TabTitle Self-compiled (source) + +1. Edit `/home/git/gitlab/config/gitlab.yml`: + + ```yaml + production: &base + lfs: + enabled: false + ``` + +1. Save the file and restart GitLab: + + ```shell + # For systems running systemd + sudo systemctl restart gitlab.target + + # For systems running SysV init + sudo service gitlab restart + ``` + +::EndTabs + +## Change local storage path Git LFS objects can be large in size. By default, they are stored on the server GitLab is installed on. -There are various configuration options to help GitLab server administrators: +NOTE: +For Docker installations, you can change the path where your data is mounted. +For the Helm chart, use +[object storage](https://docs.gitlab.com/charts/advanced/external-object-storage/). + +To change the default local storage path location: -- Enabling/disabling Git LFS support. -- Changing the location of LFS object storage. -- Setting up object storage supported by [Fog](https://fog.io/about/provider_documentation.html). +::Tabs -### Configuration for Omnibus installations +:::TabTitle Linux package (Omnibus) -In `/etc/gitlab/gitlab.rb`: +1. Edit `/etc/gitlab/gitlab.rb`: -```ruby -# Change to true to enable lfs - enabled by default if not defined -gitlab_rails['lfs_enabled'] = false + ```ruby + # /var/opt/gitlab/gitlab-rails/shared/lfs-objects by default. + gitlab_rails['lfs_storage_path'] = "/mnt/storage/lfs-objects" + ``` -# Optionally, change the storage path location. Defaults to -# `#{gitlab_rails['shared_path']}/lfs-objects`. Which evaluates to -# `/var/opt/gitlab/gitlab-rails/shared/lfs-objects` by default. -gitlab_rails['lfs_storage_path'] = "/mnt/storage/lfs-objects" -``` +1. Save the file and reconfigure GitLab: -After you update settings in `/etc/gitlab/gitlab.rb`, run [Omnibus GitLab reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). + ```shell + sudo gitlab-ctl reconfigure + ``` -### Configuration for installations from source +:::TabTitle Self-compiled (source) -In `config/gitlab.yml`: +1. Edit `/home/git/gitlab/config/gitlab.yml`: -```yaml -# Change to true to enable lfs - lfs: - enabled: false - storage_path: /mnt/storage/lfs-objects -``` + ```yaml + # /home/git/gitlab/shared/lfs-objects by default. + production: &base + lfs: + storage_path: /mnt/storage/lfs-objects + ``` + +1. Save the file and restart GitLab: + + ```shell + # For systems running systemd + sudo systemctl restart gitlab.target + + # For systems running SysV init + sudo service gitlab restart + ``` + +::EndTabs ## Storing LFS objects in remote object storage You can store LFS objects in remote object storage. This allows you to reduce reads and writes to the local disk, and free up disk space significantly. -GitLab is tightly integrated with `Fog`, so you can refer to its [documentation](https://fog.io/about/provider_documentation.html) -to check which storage services can be integrated with GitLab. -You can also use external object storage in a private local network. For example, -[MinIO](https://min.io/) is a standalone object storage service that works with GitLab instances. -[Read more about using object storage with GitLab](../object_storage.md). - -NOTE: In GitLab 13.2 and later, you should use the [consolidated object storage settings](../object_storage.md#consolidated-object-storage-configuration). -This section describes the earlier configuration format. [Migration steps still apply](#migrating-to-object-storage). -1. User pushes an `lfs` file to the GitLab instance. -1. GitLab-workhorse uploads the file directly to the external object storage. -1. GitLab-workhorse notifies GitLab-rails that the upload process is complete. +### Migrating to object storage + +You can migrate the LFS objects from local storage to object storage. The +processing is done in the background and requires **no downtime**. + +1. [Configure the object storage](../object_storage.md#consolidated-object-storage-configuration). +1. Migrate the LFS objects: + + ::Tabs -The following general settings are supported. + :::TabTitle Linux package (Omnibus) -| Setting | Description | Default | -|---------------------|-------------|---------| -| `enabled` | Enable/disable object storage. | `false` | -| `remote_directory` | The bucket name where LFS objects are stored. | | -| `proxy_download` | Set to true to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data. | `false` | -| `connection` | Various connection options described below. | | + ```shell + sudo gitlab-rake gitlab:lfs:migrate + ``` -See [the available connection settings for different providers](../object_storage.md#connection-settings). + :::TabTitle Docker -Here is a configuration example with S3. + ```shell + sudo docker exec -t <container name> gitlab-rake gitlab:lfs:migrate + ``` -### S3 for Omnibus installations + :::TabTitle Self-compiled (source) -On Omnibus GitLab installations, the settings are prefixed by `lfs_object_store_`: + ```shell + sudo -u git -H bundle exec rake gitlab:lfs:migrate RAILS_ENV=production + ``` -1. Edit `/etc/gitlab/gitlab.rb` and add the following lines, replacing values based on your needs: + ::EndTabs - ```ruby - gitlab_rails['lfs_object_store_enabled'] = true - gitlab_rails['lfs_object_store_remote_directory'] = "lfs-objects" - gitlab_rails['lfs_object_store_connection'] = { - 'provider' => 'AWS', - 'region' => 'eu-central-1', - 'aws_access_key_id' => '1ABCD2EFGHI34JKLM567N', - 'aws_secret_access_key' => 'abcdefhijklmnopQRSTUVwxyz0123456789ABCDE', - # The below options configure an S3 compatible host instead of AWS - 'host' => 'localhost', - 'endpoint' => 'http://127.0.0.1:9000', - 'path_style' => true - } +1. Optional. Track the progress and verify that all job LFS objects migrated + successfully using the PostgreSQL console. + 1. Open a PostgreSQL console: + + ::Tabs + + :::TabTitle Linux package (Omnibus) + + ```shell + sudo gitlab-psql + ``` + + :::TabTitle Docker + + ```shell + sudo docker exec -it <container_name> /bin/bash + gitlab-psql + ``` + + :::TabTitle Self-compiled (source) + + ```shell + sudo -u git -H psql -d gitlabhq_production + ``` + + ::EndTabs + + 1. Verify that all LFS files migrated to object storage with the following + SQL query. The number of `objectstg` should be the same as `total`: + + ```shell + gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM lfs_objects; + + total | filesystem | objectstg + ------+------------+----------- + 2409 | 0 | 2409 + ``` + +1. Verify that there are no files on disk in the `lfs-objects` directory: + + ::Tabs + + :::TabTitle Linux package (Omnibus) + + ```shell + sudo find /var/opt/gitlab/gitlab-rails/shared/lfs-objects -type f | grep -v tmp | wc -l ``` -1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. [Migrate any existing local LFS objects to the object storage](#migrating-to-object-storage). - New LFS objects are forwarded to object storage. + :::TabTitle Docker -### S3 for installations from source + Assuming you mounted `/var/opt/gitlab` to `/srv/gitlab`: -For source installations the settings are nested under `lfs:` and then -`object_store:`: + ```shell + sudo find /srv/gitlab/gitlab-rails/shared/lfs-objects -type f | grep -v tmp | wc -l + ``` -1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following - lines: + :::TabTitle Self-compiled (source) - ```yaml - lfs: - enabled: true - object_store: - enabled: false - remote_directory: lfs-objects # Bucket name - connection: - provider: AWS - aws_access_key_id: 1ABCD2EFGHI34JKLM567N - aws_secret_access_key: abcdefhijklmnopQRSTUVwxyz0123456789ABCDE - region: eu-central-1 - # Use the following options to configure an AWS compatible host such as Minio - host: 'localhost' - endpoint: 'http://127.0.0.1:9000' - path_style: true + ```shell + sudo find /home/git/gitlab/shared/lfs-objects -type f | grep -v tmp | wc -l ``` -1. Save the file, and then [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. -1. [Migrate any existing local LFS objects to the object storage](#migrating-to-object-storage). - New LFS objects are forwarded to object storage. + ::EndTabs -### Migrating to object storage +### Migrating back to local storage -**Option 1: Rake task** +NOTE: +For the Helm chart, you should use +[object storage](https://docs.gitlab.com/charts/advanced/external-object-storage/). -After [configuring the object storage](#storing-lfs-objects-in-remote-object-storage), use the following task to -migrate existing LFS objects from the local storage to the remote storage. -The processing is done in a background worker and requires **no downtime**. +To migrate back to local storage: -For Omnibus GitLab: +::Tabs -```shell -sudo gitlab-rake "gitlab:lfs:migrate" -``` +:::TabTitle Linux package (Omnibus) -For installations from source: +1. Migrate the LFS objects: -```shell -RAILS_ENV=production sudo -u git -H bundle exec rake gitlab:lfs:migrate -``` + ```shell + sudo gitlab-rake gitlab:lfs:migrate_to_local + ``` -You can optionally track progress and verify that all LFS objects migrated successfully using the -[PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database): +1. Edit `/etc/gitlab/gitlab.rb` and + [disable object storage](../object_storage.md#selectively-disabling-object-storage) + for LFS objects: -- `sudo gitlab-rails dbconsole` for Omnibus GitLab 14.1 and earlier. -- `sudo gitlab-rails dbconsole --database main` for Omnibus GitLab 14.2 and later. -- `sudo -u git -H psql -d gitlabhq_production` for source-installed instances. + ```ruby + gitlab_rails['object_store']['objects']['lfs']['enabled'] = false + ``` -Verify `objectstg` below (where `store=2`) has count of all LFS objects: +1. Save the file and reconfigure GitLab: -```shell -gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM lfs_objects; -``` + ```shell + sudo gitlab-ctl reconfigure + ``` -**Example Output** +:::TabTitle Docker -```shell -total | filesystem | objectstg -------+------------+----------- - 2409 | 0 | 2409 -``` +1. Migrate the LFS objects: + + ```shell + sudo docker exec -t <container name> gitlab-rake gitlab:lfs:migrate_to_local + ``` -Verify that there are no files on disk in the `objects` folder: +1. Edit `docker-compose.yml` and disable object storage for LFS objects: -```shell -sudo find /var/opt/gitlab/gitlab-rails/shared/lfs-objects -type f | grep -v tmp | wc -l -``` + ```yaml + version: "3.6" + services: + gitlab: + environment: + GITLAB_OMNIBUS_CONFIG: | + gitlab_rails['object_store']['objects']['lfs']['enabled'] = false + ``` -**Option 2: Rails console** +1. Save the file and restart GitLab: -Log into the Rails console: + ```shell + docker compose up -d + ``` -```shell -sudo gitlab-rails console -``` +:::TabTitle Self-compiled (source) -Upload LFS files manually +1. Migrate the LFS objects: -```ruby -LfsObject.where(file_store: [nil, 1]).find_each do |lfs_object| - lfs_object.file.migrate!(ObjectStorage::Store::REMOTE) if lfs_object.file.file.exists? -end -``` + ```shell + sudo -u git -H bundle exec rake gitlab:lfs:migrate_to_local RAILS_ENV=production + ``` -### Migrating back to local storage +1. Edit `/home/git/gitlab/config/gitlab.yml` and disable object storage for LFS objects: -To migrate back to local storage: + ```yaml + production: &base + object_store: + objects: + lfs: + enabled: false + ``` + +1. Save the file and restart GitLab: + + ```shell + # For systems running systemd + sudo systemctl restart gitlab.target + + # For systems running SysV init + sudo service gitlab restart + ``` -1. Run `rake gitlab:lfs:migrate_to_local` on your console. -1. Disable `object_storage` for LFS objects in `gitlab.rb`. Remember to restart GitLab afterwards. +::EndTabs ## Storage statistics diff --git a/doc/administration/load_balancer.md b/doc/administration/load_balancer.md index ad89d32183b..83b42295035 100644 --- a/doc/administration/load_balancer.md +++ b/doc/administration/load_balancer.md @@ -36,7 +36,7 @@ for details on managing SSL certificates and configuring NGINX. ### Load Balancers terminate SSL without backend SSL -Configure your load balancers to use the 'HTTP(S)' protocol rather than 'TCP'. +Configure your load balancers to use the `HTTP(S)` protocol rather than `TCP`. The load balancers is be responsible for managing SSL certificates and terminating SSL. @@ -47,7 +47,7 @@ for details. ### Load Balancers terminate SSL with backend SSL -Configure your load balancers to use the 'HTTP(S)' protocol rather than 'TCP'. +Configure your load balancers to use the `HTTP(S)` protocol rather than `TCP`. The load balancers is responsible for managing SSL certificates that end users see. diff --git a/doc/administration/logs/tracing_correlation_id.md b/doc/administration/logs/tracing_correlation_id.md index 906dcd3cea9..45c0ce37102 100644 --- a/doc/administration/logs/tracing_correlation_id.md +++ b/doc/administration/logs/tracing_correlation_id.md @@ -133,7 +133,7 @@ You can use the [performance bar](../monitoring/performance/performance_bar.md) To view the data, the correlation ID of the request must match the same session as the user viewing the performance bar. For API requests, this means that you must perform the request -using the session cookie of the signed-in user. +using the session cookie of the authenticated user. For example, if you want to view the database queries executed for the following API endpoint: diff --git a/doc/administration/merge_request_diffs.md b/doc/administration/merge_request_diffs.md index 25dba00beb9..85677512860 100644 --- a/doc/administration/merge_request_diffs.md +++ b/doc/administration/merge_request_diffs.md @@ -1,6 +1,6 @@ --- stage: Create -group: Editor +group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md index 92e9672cdb6..3dec34ebace 100644 --- a/doc/administration/monitoring/performance/grafana_configuration.md +++ b/doc/administration/monitoring/performance/grafana_configuration.md @@ -25,9 +25,9 @@ is `admin`. 1. Log in to Grafana as the administration user. 1. Select **Data Sources** from the **Configuration** menu. -1. Select the **Add data source** button. +1. Select **Add data source**. 1. Select the required data source type. For example, [Prometheus](../prometheus/index.md#prometheus-as-a-grafana-data-source). -1. Complete the details for the data source and select the **Save & Test** button. +1. Complete the details for the data source and select **Save & Test**. Grafana should indicate the data source is working. @@ -43,8 +43,8 @@ them: 1. Log in to Grafana as the administration user. 1. Select **Manage** from the **Dashboards** menu. - 1. Select the **Import** button, then the **Upload JSON file** button. - 1. Locate the JSON file to import and select **Choose for Upload**. Select the **Import** button. + 1. Select **Import**, then **Upload JSON file**. + 1. Locate the JSON file to import and select **Choose for Upload**. Select **Import**. 1. After the dashboard is imported, select the **Save dashboard** icon in the top bar. If you don't save the dashboard after importing it, the dashboard is removed diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index 53ee028bc32..29182077d66 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -153,6 +153,14 @@ The following metrics are available: | `cached_object_operations_total` | Counter | 15.3 | Total number of objects cached for specific web requests | `controller`, `action` | | `redis_hit_miss_operations_total` | Counter | 15.6 | Total number of Redis cache hits and misses | `cache_hit`, `caller_id`, `cache_identifier`, `feature_category`, `backing_resource` | | `redis_cache_generation_duration_seconds` | Histogram | 15.6 | Time to generate Redis cache | `cache_hit`, `caller_id`, `cache_identifier`, `feature_category`, `backing_resource` | +| `gitlab_diffs_reorder_real_duration_seconds` | Histogram | 15.8 | Duration in seconds spend on reordering of diff files on diffs batch request | `controller`, `action` | +| `gitlab_diffs_collection_real_duration_seconds` | Histogram | 15.8 | Duration in seconds spent on querying merge request diff files on diffs batch request | `controller`, `action` | +| `gitlab_diffs_comparison_real_duration_seconds` | Histogram | 15.8 | Duration in seconds spent on getting comparison data on diffs batch request | `controller`, `action` | +| `gitlab_diffs_unfoldable_positions_real_duration_seconds` | Histogram | 15.8 | Duration in seconds spent on getting unfoldable note positions on diffs batch request | `controller`, `action` | +| `gitlab_diffs_unfold_real_duration_seconds` | Histogram | 15.8 | Duration in seconds spent on unfolding positions on diffs batch request | `controller`, `action` | +| `gitlab_diffs_write_cache_real_duration_seconds` | Histogram | 15.8 | Duration in seconds spent on caching highlighted lines and stats on diffs batch request | `controller`, `action` | +| `gitlab_diffs_highlight_cache_decorate_real_duration_seconds` | Histogram | 15.8 | Duration in seconds spent on setting highlighted lines from cache on diffs batch request | `controller`, `action` | +| `gitlab_diffs_render_real_duration_seconds` | Histogram | 15.8 | Duration in seconds spent on serializing and rendering diffs on diffs batch request | `controller`, `action` | ## Metrics controlled by a feature flag diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index 0cae46faf28..6064ae7525e 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -13,7 +13,11 @@ typically much more performant, reliable, and scalable. ## Options -GitLab has been tested by vendors and customers on a number of object storage providers: +GitLab is tightly integrated with `Fog`, so you can refer to its +[documentation](https://fog.io/about/provider_documentation.html) to check +which storage services can be integrated with GitLab. + +Specifically, GitLab has been tested by vendors and customers on a number of object storage providers: - [Amazon S3](https://aws.amazon.com/s3/) - [Google Cloud Storage](https://cloud.google.com/storage) diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index 7aeb05457c0..a34b21e676a 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -188,7 +188,8 @@ file for the environment, as it isn't generated dynamically. ### Additional documentation -Additional technical documentation for `gitlab-sshd` may be found on the [GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell/-/blob/main/README.md) documentation page. +Additional technical documentation for `gitlab-sshd` may be found in the +[GitLab Shell documentation](../../development/gitlab_shell/index.md). ## Troubleshooting diff --git a/doc/administration/operations/moving_repositories.md b/doc/administration/operations/moving_repositories.md index 96c1fcc422d..5066f6d99d8 100644 --- a/doc/administration/operations/moving_repositories.md +++ b/doc/administration/operations/moving_repositories.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w You can move all repositories managed by GitLab to another file system or another server. -## Moving data within a GitLab instance +## Moving data in a GitLab instance The GitLab API is the recommended way to move Git repositories: @@ -28,10 +28,10 @@ For more information, see: querying and scheduling group repository moves **(PREMIUM SELF)**. - [Migrate to Gitaly Cluster](../gitaly/index.md#migrate-to-gitaly-cluster). -### Move Repositories +### Moving Repositories GitLab repositories can be associated with projects, groups, and snippets. Each of these types -have a separate API to schedule the respective repositories to move. To move all repositories +has a separate API to schedule the respective repositories to move. To move all repositories on a GitLab instance, each of these types must be scheduled to move for each storage. WARNING: @@ -41,7 +41,7 @@ To move repositories into a [Gitaly Cluster](../gitaly/index.md#gitaly-cluster) WARNING: Repositories can be **permanently deleted** by a call to `/projects/:project_id/repository_storage_moves` that attempts to move a project already stored in a Gitaly Cluster back into that cluster. -See [this issue for more details](https://gitlab.com/gitlab-org/gitaly/-/issues/3752). This was fixed in +See [this issue for more details](https://gitlab.com/gitlab-org/gitaly/-/issues/3752). This issue was fixed in GitLab 14.3.0 and backported to [14.2.4](https://about.gitlab.com/releases/2021/09/17/gitlab-14-2-4-released/), [14.1.6](https://about.gitlab.com/releases/2021/09/27/gitlab-14-1-6-released/), @@ -59,13 +59,16 @@ To move repositories: so that the new storages receives all new projects. This stops new projects from being created on existing storages while the migration is in progress. 1. Schedule repository moves for: - - [Projects](#bulk-schedule-project-moves). - - [Snippets](#bulk-schedule-snippet-moves). - - [Groups](#bulk-schedule-group-moves). **(PREMIUM SELF)** + - [All projects](#move-all-projects) or + [individual projects](../../api/project_repository_storage_moves.md#schedule-a-repository-storage-move-for-a-project). + - [All snippets](#move-all-snippets) or + [individual snippets](../../api/snippet_repository_storage_moves.md#schedule-a-repository-storage-move-for-a-snippet). + - [All groups](#move-all-groups) or + [individual groups](../../api/group_repository_storage_moves.md#schedule-a-repository-storage-move-for-a-group). **(PREMIUM SELF)** -### Bulk schedule project moves +#### Move all projects -Use the API to schedule project moves: +To move all projects by using the API: 1. [Schedule repository storage moves for all projects on a storage shard](../../api/project_repository_storage_moves.md#schedule-repository-storage-moves-for-all-projects-on-a-storage-shard) using the API. For example: @@ -100,9 +103,9 @@ Use the API to schedule project moves: 1. Repeat for each storage as required. -### Bulk schedule snippet moves +#### Move all snippets -Use the API to schedule snippet moves: +To move all snippets by using the API: 1. [Schedule repository storage moves for all snippets on a storage shard](../../api/snippet_repository_storage_moves.md#schedule-repository-storage-moves-for-all-snippets-on-a-storage-shard). For example: @@ -113,8 +116,8 @@ Use the API to schedule snippet moves: "https://gitlab.example.com/api/v4/snippet_repository_storage_moves" ``` -1. [Query the most recent repository moves](../../api/snippet_repository_storage_moves.md#retrieve-all-snippet-repository-storage-moves) -The response indicates either: +1. [Query the most recent repository moves](../../api/snippet_repository_storage_moves.md#retrieve-all-snippet-repository-storage-moves). + The response indicates either: - The moves have completed successfully. The `state` field is `finished`. - The moves are in progress. Re-query the repository move until it completes successfully. - The moves have failed. Most failures are temporary and are solved by rescheduling the move. @@ -129,12 +132,12 @@ The response indicates either: 1. Repeat for each storage as required. -### Bulk schedule group moves **(PREMIUM SELF)** +#### Move all groups **(PREMIUM SELF)** -Use the API to schedule group moves: +To move all groups by using the API: -1. [Schedule repository storage moves for all groups on a storage shard](../../api/group_repository_storage_moves.md#schedule-repository-storage-moves-for-all-groups-on-a-storage-shard) -. For example: +1. [Schedule repository storage moves for all groups on a storage shard](../../api/group_repository_storage_moves.md#schedule-repository-storage-moves-for-all-groups-on-a-storage-shard). + For example: ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ @@ -143,8 +146,8 @@ Use the API to schedule group moves: "https://gitlab.example.com/api/v4/group_repository_storage_moves" ``` -1. [Query the most recent repository moves](../../api/group_repository_storage_moves.md#retrieve-all-group-repository-storage-moves) -. The response indicates either: +1. [Query the most recent repository moves](../../api/group_repository_storage_moves.md#retrieve-all-group-repository-storage-moves). + The response indicates either: - The moves have completed successfully. The `state` field is `finished`. - The moves are in progress. Re-query the repository move until it completes successfully. - The moves have failed. Most failures are temporary and are solved by rescheduling the move. @@ -161,7 +164,7 @@ Use the API to schedule group moves: ## Migrating to another GitLab instance -[Using the API](#moving-data-within-a-gitlab-instance) isn't an option if you are migrating to a new +[Using the API](#moving-data-in-a-gitlab-instance) isn't an option if you are migrating to a new GitLab environment, for example: - From a single-node GitLab to a scaled-out architecture. @@ -185,7 +188,7 @@ Each of the approaches we list can or does overwrite data in the target director For either Gitaly or Gitaly Cluster targets, the GitLab [backup and restore capability](../../raketasks/backup_restore.md) should be used. Git repositories are accessed, managed, and stored on GitLab servers by Gitaly as a database. Data loss -can result from directly accessing and copying Gitaly's files using tools like `rsync`. +can result from directly accessing and copying Gitaly files using tools like `rsync`. - From GitLab 13.3, backup performance can be improved by [processing multiple repositories concurrently](../../raketasks/backup_gitlab.md#back-up-git-repositories-concurrently). diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md index af595cdf297..51d6e9ae1fd 100644 --- a/doc/administration/operations/puma.md +++ b/doc/administration/operations/puma.md @@ -14,24 +14,22 @@ features of GitLab. To reduce memory use, Puma forks worker processes. Each time a worker is created, it shares memory with the primary process. The worker uses additional memory only -when it changes or adds to its memory pages. +when it changes or adds to its memory pages. This can lead to Puma workers using +more physical memory over time as workers handle additional web requests. The amount of memory +used over time depends on the use of GitLab. The more features used by GitLab users, +the higher the expected memory use over time. -Memory use increases over time, but you can use Puma Worker Killer to recover memory. +To stop uncontrolled memory growth, the GitLab Rails application runs a supervision thread +that automatically restarts workers if they exceed a given resident set size (RSS) threshold +for a certain amount of time. -By default: - -- The [Puma Worker Killer](https://github.com/schneems/puma_worker_killer) restarts a worker if it - exceeds a [memory limit](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/cluster/puma_worker_killer_initializer.rb). -- Rolling restarts of Puma workers are performed every 12 hours. - -### Change the memory limit setting - -To change the memory limit setting: +GitLab sets a default of `1200Mb` for the memory limit. To override the default value, +set `per_worker_max_memory_mb` to the new RSS limit in megabytes: 1. Edit `/etc/gitlab/gitlab.rb`: ```ruby - puma['per_worker_max_memory_mb'] = 1024 + puma['per_worker_max_memory_mb'] = 1024 # 1GB ``` 1. Reconfigure GitLab: @@ -40,48 +38,40 @@ To change the memory limit setting: sudo gitlab-ctl reconfigure ``` -When workers are killed and replaced, capacity to run GitLab is reduced, -and CPU is consumed. Set `per_worker_max_memory_mb` to a higher value if the worker killer -is replacing workers too often. +When workers are restarted, capacity to run GitLab is reduced for a short +period of time. Set `per_worker_max_memory_mb` to a higher value if workers are replaced too often. Worker count is calculated based on CPU cores. A small GitLab deployment with 4-8 workers may experience performance issues if workers are being restarted too often (once or more per minute). -A higher value of `1200` or more would be beneficial if the server has free memory. +A higher value of `1200` or more could be beneficial if the server has free memory. -### Monitor worker memory +### Monitor worker restarts -The worker killer checks memory every 20 seconds. +GitLab emits log events if workers are restarted due to high memory use. -To monitor the worker killer, use [the Puma log](../logs/index.md#puma_stdoutlog) `/var/log/gitlab/puma/puma_stdout.log`. -For example: +The following is an example of one of these log events in `/var/log/gitlab/gitlab-rails/application_json.log`: -```plaintext -PumaWorkerKiller: Out of memory. 4 workers consuming total: 4871.23828125 MB -out of max: 4798.08 MB. Sending TERM to pid 26668 consuming 1001.00390625 MB. +```json +{ + "severity": "WARN", + "time": "2023-01-04T09:45:16.173Z", + "correlation_id": null, + "pid": 2725, + "worker_id": "puma_0", + "memwd_handler_class": "Gitlab::Memory::Watchdog::PumaHandler", + "memwd_sleep_time_s": 5, + "memwd_rss_bytes": 1077682176, + "memwd_max_rss_bytes": 629145600, + "memwd_max_strikes": 5, + "memwd_cur_strikes": 6, + "message": "rss memory limit exceeded" +} ``` -From this output: - -- The formula that calculates the maximum memory value results in workers - being killed before they reach the `per_worker_max_memory_mb` value. -- In GitLab 13.4 and earlier, the default values for the formula were 550 MB for the primary - and 850 MB for each worker. -- In GitLab 13.5 and later, the values are primary: 800 MB, worker: 1024 MB. -- The threshold for workers to be killed is set at 98% of the limit: - - ```plaintext - 0.98 * ( 800 + ( worker_processes * 1024MB ) ) - ``` - -- In the log output above, `0.98 * ( 800 + ( 4 * 1024 ) )` returns the - `max: 4798.08 MB` value. - -Increasing the maximum to `1200`, for example, would set a `max: 5488 MB` value. - -Workers use additional memory on top of the shared memory. The amount of memory -depends on a site's use of GitLab. +`memwd_rss_bytes` is the actual amount of memory consumed, and `memwd_max_rss_bytes` is the +RSS limit set through `per_worker_max_memory_mb`. ## Change the worker timeout @@ -146,7 +136,7 @@ for details. When running Puma in single mode, some features are not supported: - [Phased restart](https://gitlab.com/gitlab-org/gitlab/-/issues/300665) -- [Puma Worker Killer](https://gitlab.com/gitlab-org/gitlab/-/issues/300664) +- [Memory killers](#reducing-memory-use) To learn more, visit [epic 5303](https://gitlab.com/groups/gitlab-org/-/epics/5303). diff --git a/doc/administration/operations/rails_console.md b/doc/administration/operations/rails_console.md index efaf480c6df..f2143435755 100644 --- a/doc/administration/operations/rails_console.md +++ b/doc/administration/operations/rails_console.md @@ -31,6 +31,12 @@ Rails experience is useful but not required. sudo gitlab-rails console ``` +**For Docker installations** + +```shell +docker exec -it <container-id> gitlab-rails console +``` + **For installations from source** ```shell diff --git a/doc/administration/package_information/supported_os.md b/doc/administration/package_information/supported_os.md index fdbb17f3c71..9d1c8dcde5a 100644 --- a/doc/administration/package_information/supported_os.md +++ b/doc/administration/package_information/supported_os.md @@ -21,10 +21,11 @@ The following lists the currently supported OSs and their possible EOL dates. | CentOS 7 | GitLab CE / GitLab EE 7.10.0 | x86_64 | [CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | June 2024 | <https://wiki.centos.org/About/Product> | | Debian 10 | GitLab CE / GitLab EE 12.2.0 | amd64, arm64 | [Debian Install Documentation](https://about.gitlab.com/install/#debian) | 2024 | <https://wiki.debian.org/LTS> | | Debian 11 | GitLab CE / GitLab EE 14.6.0 | amd64, arm64 | [Debian Install Documentation](https://about.gitlab.com/install/#debian) | 2026 | <https://wiki.debian.org/LTS> | -| OpenSUSE 15.3 | GitLab CE / GitLab EE 14.5.0 | x86_64, aarch64 | [OpenSUSE Install Documentation](https://about.gitlab.com/install/#opensuse-leap-15-3) | Nov 2022 | <https://en.opensuse.org/Lifetime> | +| OpenSUSE 15.3 | GitLab CE / GitLab EE 14.5.0 | x86_64, aarch64 | [OpenSUSE Install Documentation](https://about.gitlab.com/install/#opensuse-leap) | Dec 2022 | <https://en.opensuse.org/Lifetime> | +| OpenSUSE 15.4 | GitLab CE / GitLab EE 15.7.0 | x86_64, aarch64 | [OpenSUSE Install Documentation](https://about.gitlab.com/install/#opensuse-leap) | Nov 2023 | <https://en.opensuse.org/Lifetime> | | RHEL 8 | GitLab CE / GitLab EE 12.8.1 | x86_64, arm64 | [Use CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | May 2029 | [RHEL Details](https://access.redhat.com/support/policy/updates/errata/#Life_Cycle_Dates) | -| SLES 12 | GitLab EE 9.0.0 | x86_64 | [Use OpenSUSE Install Documentation](https://about.gitlab.com/install/#opensuse-leap-15-3) | Oct 2027 | <https://www.suse.com/lifecycle/> | -| SLES 15 | GitLab EE 14.8.0 | x86_64 | [Use OpenSUSE Install Documentation](https://about.gitlab.com/install/#opensuse-leap-15-3) | Dec 2024 | <https://www.suse.com/lifecycle/> | +| SLES 12 | GitLab EE 9.0.0 | x86_64 | [Use OpenSUSE Install Documentation](https://about.gitlab.com/install/#opensuse-leap) | Oct 2027 | <https://www.suse.com/lifecycle/> | +| SLES 15 | GitLab EE 14.8.0 | x86_64 | [Use OpenSUSE Install Documentation](https://about.gitlab.com/install/#opensuse-leap) | Dec 2024 | <https://www.suse.com/lifecycle/> | | Oracle Linux | GitLab CE / GitLab EE 8.14.0 | x86_64 | [Use CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | Jul 2024 | <https://www.oracle.com/a/ocom/docs/elsp-lifetime-069338.pdf> | | Scientific Linux | GitLab CE / GitLab EE 8.14.0 | x86_64 | [Use CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | June 2024 | <https://scientificlinux.org/downloads/sl-versions/sl7/> | | Ubuntu 18.04 | GitLab CE / GitLab EE 10.7.0 | amd64 | [Ubuntu Install Documentation](https://about.gitlab.com/install/#ubuntu) | April 2023 | <https://wiki.ubuntu.com/Releases> | @@ -40,6 +41,9 @@ CentOS 8 was EOL on December 31, 2021. In GitLab 14.5 and later, We officially support all distributions that are binary compatible with Red Hat Enterprise Linux. This gives users a path forward for their CentOS 8 builds at its end of life. +NOTE: +The [CentOS major version and a minor version](https://en.wikipedia.org/wiki/CentOS#CentOS_releases) up to CentOS8 ([when CentOS Stream](https://en.wikipedia.org/wiki/CentOS#CentOS_Stream) was released) correspond to the set of major version and update versions of RHEL. + ## Update GitLab package sources after upgrading the OS After upgrading the Operating System (OS) as per its own documentation, diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index d76c728fb78..6446252e143 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -590,6 +590,13 @@ you can pull from the Container Registry, but you cannot push. #### Moving to Azure Object Storage +> The default configuration for the storage driver will be [changed](https://gitlab.com/gitlab-org/container-registry/-/issues/854) in GitLab 16.0. + +<!--- start_remove The following content will be removed on remove_date: '2023-10-22' --> +WARNING: +The default configuration for the storage driver will be [changed](https://gitlab.com/gitlab-org/container-registry/-/issues/854) in GitLab 16.0. The storage driver will use `/` as the default root directory. You can add `trimlegacyrootprefix: false` to your current configuration now to avoid any disruptions. For more information, see the [Container Registry configuration](https://gitlab.com/gitlab-org/container-registry/-/tree/master/docs-gitlab#azure-storage-driver) documentation. +<!--- end_remove --> + When moving from an existing file system or another object storage provider to Azure Object Storage, you must configure the registry to use the standard root directory. This configuration is done by setting [`trimlegacyrootprefix: true]`](https://gitlab.com/gitlab-org/container-registry/-/blob/a3f64464c3ec1c5a599c0a2daa99ebcbc0100b9a/docs-gitlab/README.md#azure-storage-driver) in the Azure storage driver section of the registry configuration. Without this configuration, the Azure storage driver uses `//` instead of `/` as the first section of the root path, rendering the migrated images inaccessible. @@ -758,6 +765,8 @@ project, you can [disable it from your project's settings](../../user/project/se ## Use an external container registry with GitLab as an auth endpoint +> Support for external container registries in GitLab is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/376217) in GitLab 15.8 and will be removed in GitLab 16.0. + If you use an external container registry, some features associated with the container registry may be unavailable or have [inherent risks](../../user/packages/container_registry/reduce_container_registry_storage.md#use-with-external-container-registries). @@ -777,7 +786,7 @@ auth: Without these entries, the registry logins cannot authenticate with GitLab. GitLab also remains unaware of -[nested image names](../../user/packages/container_registry/index.md#image-naming-convention) +[nested image names](../../user/packages/container_registry/index.md#naming-convention-for-your-container-images) under the project hierarchy, like `registry.example.com/group/project/image-name:tag` or `registry.example.com/group/project/my/image-name:tag`, and only recognizes diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index 2a28df96ef4..58003758c8a 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -185,7 +185,7 @@ you must also add the full paths as shown below: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). 1. If you're using [Pages Access Control](#access-control), update the redirect URI in the GitLab Pages -[System OAuth application](../../integration/oauth_provider.md#instance-wide-applications) +[System OAuth application](../../integration/oauth_provider.md#create-an-instance-wide-application) to use the HTTPS protocol. WARNING: @@ -384,7 +384,7 @@ then you need to also add the full paths as shown below: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). 1. If you're using [Pages Access Control](#access-control), update the redirect URI in the GitLab Pages -[System OAuth application](../../integration/oauth_provider.md#instance-wide-applications) +[System OAuth application](../../integration/oauth_provider.md#create-an-instance-wide-application) to use the HTTPS protocol. ### Custom domain verification @@ -485,7 +485,7 @@ this: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32095) in GitLab 12.7. You can enforce [Access Control](#access-control) for all GitLab Pages websites hosted -on your GitLab instance. By doing so, only logged-in users have access to them. +on your GitLab instance. By doing so, only authenticated users have access to them. This setting overrides Access Control set by users in individual projects. This can be helpful to restrict information published with Pages websites to the users @@ -1428,7 +1428,7 @@ Once added, reconfigure with `sudo gitlab-ctl reconfigure` and restart GitLab wi You may see this error if `pages_external_url` was updated at some point of time. Verify the following: -1. The **Callback URL**/Redirect URI in the GitLab Pages [System OAuth application](../../integration/oauth_provider.md#instance-wide-applications) +1. The **Callback URL**/Redirect URI in the GitLab Pages [System OAuth application](../../integration/oauth_provider.md#create-an-instance-wide-application) is using the protocol (HTTP or HTTPS) that `pages_external_url` is configured to use. 1. The domain and path components of `Redirect URI` are valid: they should look like `projects.<pages_external_url>/auth`. diff --git a/doc/administration/pages/source.md b/doc/administration/pages/source.md index e122d49a963..db76d15ec58 100644 --- a/doc/administration/pages/source.md +++ b/doc/administration/pages/source.md @@ -445,7 +445,7 @@ Pages access control is disabled by default. To enable it: ``` 1. [Restart GitLab](../restart_gitlab.md#installations-from-source). -1. Create a new [system OAuth application](../../integration/oauth_provider.md#user-owned-applications). +1. Create a new [system OAuth application](../../integration/oauth_provider.md#create-a-user-owned-application). This should be called `GitLab Pages` and have a `Redirect URL` of `https://projects.example.io/auth`. It does not need to be a "trusted" application, but it does need the `api` scope. diff --git a/doc/administration/postgresql/multiple_databases.md b/doc/administration/postgresql/multiple_databases.md index ffccbf861e7..25f148dbe84 100644 --- a/doc/administration/postgresql/multiple_databases.md +++ b/doc/administration/postgresql/multiple_databases.md @@ -118,6 +118,7 @@ the other way around. sudo gitlab-ctl start postgresql sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql -d template1 -c "CREATE DATABASE gitlabhq_production_ci OWNER gitlab;" sudo gitlab-rake db:schema:load:ci + ``` 1. Lock writes for `ci` tables in `main` database, and the other way around: diff --git a/doc/administration/raketasks/github_import.md b/doc/administration/raketasks/github_import.md index 61f6137e1ed..2e6a88a77e2 100644 --- a/doc/administration/raketasks/github_import.md +++ b/doc/administration/raketasks/github_import.md @@ -15,6 +15,11 @@ Bear in mind that the syntax is very specific. Remove any spaces in the argument before/after the brackets. Also, some shells (for example, Zsh) can interpret the open/close brackets (`[]`) separately. You may want to either escape the brackets or use double quotes. +Prerequisite: + +- At least the Maintainer role on the destination group to import to. Using the Developer role for this purpose was + [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/387891) in GitLab 15.8 and will be removed in GitLab 16.0. + ## Caveats If the GitHub [rate limit](https://docs.github.com/en/rest/rate-limit) is reached while @@ -34,7 +39,7 @@ bundle exec rake "import:github[access_token,root,foo/bar]" RAILS_ENV=production In this case, `access_token` is your GitHub personal access token, `root` is your GitLab username, and `foo/bar` is the new GitLab namespace/project -created from your GitHub project. Subgroups are also possible: `foo/foo/bar`. +created from your GitHub project. Subgroups are also possible: `foo/foo/bar`. The importer creates any missing intermediate namespaces (groups) if they do not exist. ## Importing a single project diff --git a/doc/administration/redis/replication_and_failover.md b/doc/administration/redis/replication_and_failover.md index 2ba19aa6f0a..2e56884e309 100644 --- a/doc/administration/redis/replication_and_failover.md +++ b/doc/administration/redis/replication_and_failover.md @@ -324,6 +324,14 @@ Read more about [roles](https://docs.gitlab.com/omnibus/roles/). sudo touch /etc/gitlab/skip-auto-reconfigure ``` +1. Only the primary GitLab application server should handle migrations. To + prevent database migrations from running on upgrade, add the following + configuration to your `/etc/gitlab/gitlab.rb` file: + + ```ruby + gitlab_rails['auto_migrate'] = false + ``` + 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Go through the steps again for all the other replica nodes. diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 88913eb1f7f..bb50f66aff0 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -1855,7 +1855,7 @@ Updates to example must be made at: # Set number of Sidekiq queue processes to the same number as available CPUs sidekiq['queue_groups'] = ['*'] * 4 - # Set number of Sidekiq threads per queue process to the recommend number of 20 + # Set number of Sidekiq threads per queue process to the recommended number of 20 sidekiq['max_concurrency'] = 20 # Monitoring @@ -1986,7 +1986,7 @@ On each node perform the following: {host: '10.6.0.53', port: 26379}, ] - ## Second cluster that will host the persistent queues, shared state, and actionable + ## Second cluster that will host the persistent queues, shared state, and actioncable gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' diff --git a/doc/administration/snippets/index.md b/doc/administration/snippets/index.md index 4bd03aeb8c8..613d161a64c 100644 --- a/doc/administration/snippets/index.md +++ b/doc/administration/snippets/index.md @@ -1,7 +1,7 @@ --- type: reference, howto stage: Create -group: Editor +group: Source Code info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" --- diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 7c5feb24e15..0cd34ca16df 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -74,7 +74,7 @@ Moved to [Geo replication troubleshooting](../geo/replication/troubleshooting.md ### Artifacts -Moved to [Geo replication troubleshooting](../geo/replication/troubleshooting.md#find-failed-artifacts). +Moved to [Geo replication troubleshooting](../geo/replication/troubleshooting.md#find-registry-records-of-blobs-that-failed-to-sync). ### Repository verification failures |