diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2021-09-20 16:18:24 +0300 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2021-09-20 16:18:24 +0300 |
| commit | 0653e08efd039a5905f3fa4f6e9cef9f5d2f799c (patch) | |
| tree | 4dcc884cf6d81db44adae4aa99f8ec1233a41f55 /doc/administration | |
| parent | 744144d28e3e7fddc117924fef88de5d9674fe4c (diff) | |
Add latest changes from gitlab-org/gitlab@14-3-stable-eev14.3.0-rc42
Diffstat (limited to 'doc/administration')
119 files changed, 1698 insertions, 492 deletions
diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index 48bd812c7f2..3ff5fb2635d 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -39,7 +39,7 @@ There are two kinds of events logged: ### Impersonation data -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/536) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.0. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/536) in GitLab 13.0. When a user is being [impersonated](../user/admin_area/index.md#user-impersonation), their actions are logged as audit events as usual, with two additional details: @@ -48,7 +48,7 @@ When a user is being [impersonated](../user/admin_area/index.md#user-impersonati  -### Group events **(PREMIUM)** +### Group events A user with: @@ -86,7 +86,7 @@ From there, you can see the following actions: Group events can also be accessed via the [Group Audit Events API](../api/audit_events.md#group-audit-events) -### Project events **(PREMIUM)** +### Project events A user with a Maintainer role (or above) can retrieve project audit events of all users. A user with a Developer role is limited to project audit events based on their individual actions. @@ -126,6 +126,10 @@ From there, you can see the following actions: - User password required for approvals was updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336211) in GitLab 14.2) - Permission to modify merge requests approval rules in merge requests was updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336211) in GitLab 14.2) - New approvals requirement when new commits are added to an MR was updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336211) in GitLab 14.2) +- When [strategies for feature flags](../operations/feature_flags.md#feature-flag-strategies) are changed ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68408) in GitLab 14.3) +- Allowing force push to protected branch changed ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/338873) in GitLab 14.3) +- Code owner approval requirement on merge requests targeting protected branch changed ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/338873) in GitLab 14.3) +- Users and groups allowed to merge and push to protected branch added or removed ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/338873) in GitLab 14.3) Project events can also be accessed via the [Project Audit Events API](../api/audit_events.md#project-audit-events). @@ -133,7 +137,7 @@ Project event queries are limited to a maximum of 30 days. ### Instance events **(PREMIUM SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2336) in [GitLab Premium](https://about.gitlab.com/pricing/) 9.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2336) in GitLab 9.3. Server-wide audit events introduce the ability to observe user actions across the entire instance of your GitLab server, making it easy to understand who @@ -143,7 +147,7 @@ Instance events do not include group or project audit events. To view the server-wide audit events: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Audit Events**. The following user actions are recorded: @@ -203,6 +207,8 @@ to request it, or you can [add it yourself](../development/audit_event_guide/). #### Repository push +> [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/337993) in GitLab 14.3. + The current architecture of audit events is not prepared to receive a very high amount of records. It may make the user interface for your project or audit events very busy, and the disk space consumed by the `audit_events` PostgreSQL table may increase considerably. It's disabled by default @@ -240,8 +246,8 @@ The search filters you can see depends on which audit level you are at. ## Export to CSV **(PREMIUM SELF)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1449) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.4. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/285441) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.7. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1449) in GitLab 13.4. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/285441) in GitLab 13.7. Export to CSV allows customers to export the current filter view of your audit events as a CSV file, which stores tabular data in plain text. The data provides a comprehensive view with respect to @@ -249,7 +255,7 @@ audit events. To export the Audit Events to CSV: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Audit Events**. 1. Select the available search [filters](#search). 1. Select **Export as CSV**. diff --git a/doc/administration/auditor_users.md b/doc/administration/auditor_users.md index 5f31ed709f2..5498ea9d4be 100644 --- a/doc/administration/auditor_users.md +++ b/doc/administration/auditor_users.md @@ -57,7 +57,7 @@ helpful: To create an Auditor user: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Overview > Users**. 1. Create a new user or edit an existing one, and in the **Access** section select Auditor. diff --git a/doc/administration/auth/atlassian.md b/doc/administration/auth/atlassian.md index 868482148e5..b58bbfa8eac 100644 --- a/doc/administration/auth/atlassian.md +++ b/doc/administration/auth/atlassian.md @@ -16,7 +16,7 @@ To enable the Atlassian OmniAuth provider for passwordless authentication you mu 1. Click **Create a new app**. 1. Choose an App Name, such as 'GitLab', and click **Create**. 1. Note the `Client ID` and `Secret` for the [GitLab configuration](#gitlab-configuration) steps. -1. In the left sidebar under **APIS AND FEATURES**, click **OAuth 2.0 (3LO)**. +1. On the left sidebar under **APIS AND FEATURES**, click **OAuth 2.0 (3LO)**. 1. Enter the GitLab callback URL using the format `https://gitlab.example.com/users/auth/atlassian_oauth2/callback` and click **Save changes**. 1. Click **+ Add** in the left sidebar under **APIS AND FEATURES**. 1. Click **Add** for **Jira platform REST API** and then **Configure**. diff --git a/doc/administration/auth/ldap/google_secure_ldap.md b/doc/administration/auth/ldap/google_secure_ldap.md index 55ccf6653a3..137f35986ac 100644 --- a/doc/administration/auth/ldap/google_secure_ldap.md +++ b/doc/administration/auth/ldap/google_secure_ldap.md @@ -87,6 +87,7 @@ values obtained during the LDAP client configuration earlier: password: 'd6V5H8nhMUW9AuDP25abXeLd' encryption: 'simple_tls' verify_certificates: true + retry_empty_result_with_codes: [80] tls_options: cert: | @@ -159,6 +160,7 @@ values obtained during the LDAP client configuration earlier: password: 'd6V5H8nhMUW9AuDP25abXeLd' encryption: 'simple_tls' verify_certificates: true + retry_empty_result_with_codes: [80] tls_options: cert: | @@ -213,7 +215,7 @@ values obtained during the LDAP client configuration earlier: ## Using encrypted credentials You can optionally store the `bind_dn` and `password` in a separate encrypted configuration file using the -[same steps as the regular LDAP integration](index.md#using-encrypted-credentials). +[same steps as the regular LDAP integration](index.md#use-encrypted-credentials). <!-- ## Troubleshooting diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index 63e3a0a3686..1992b450338 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -12,24 +12,22 @@ to support user authentication. This integration works with most LDAP-compliant directory servers, including: -- Microsoft Active Directory - - [Microsoft Active Directory Trusts](https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2008-R2-and-2008/cc771568(v=ws.10)) are not supported. -- Apple Open Directory -- Open LDAP -- 389 Server +- Microsoft Active Directory. + [Microsoft Active Directory Trusts](https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2008-R2-and-2008/cc771568(v=ws.10)) + are not supported. +- Apple Open Directory. +- Open LDAP. +- 389 Server. Users added through LDAP take a [licensed seat](../../../subscriptions/self_managed/index.md#billable-users). -GitLab Enterprise Editions (EE) include enhanced integration, -including group membership syncing and multiple LDAP server support. - ## Security GitLab assumes that LDAP users: - Are not able to change their LDAP `mail`, `email`, or `userPrincipalName` attributes. An LDAP user allowed to change their email on the LDAP server can potentially - [take over any account](#enabling-ldap-sign-in-for-existing-gitlab-users) + [take over any account](#enable-ldap-sign-in-for-existing-gitlab-users) on your GitLab server. - Have unique email addresses. If not, it's possible for LDAP users with the same email address to share the same GitLab account. @@ -42,7 +40,7 @@ the LDAP server, or share email addresses. Users deleted from the LDAP server are immediately blocked from signing in to GitLab. However, there's an LDAP check cache time of one hour (which is -[configurable](#adjusting-ldap-user-sync-schedule) for GitLab Premium users). +[configurable](#adjust-ldap-user-sync-schedule) for GitLab Premium users). This means users already signed-in or who are using Git over SSH can access GitLab for up to one hour. Manually block the user in the GitLab Admin Area to immediately block all access. @@ -53,7 +51,7 @@ LDAP-enabled users can authenticate with Git using their GitLab username or email and LDAP password, even if password authentication for Git is disabled in the application settings. -## Enabling LDAP sign-in for existing GitLab users +## Enable LDAP sign-in for existing GitLab users When a user signs in to GitLab with LDAP for the first time and their LDAP email address is the primary email address of an existing GitLab user, the @@ -74,7 +72,7 @@ See [Google Secure LDAP](google_secure_ldap.md) for detailed configuration instr ## Configuration -To enable LDAP integration you need to add your LDAP server settings in +To enable LDAP integration you must add your LDAP server settings in `/etc/gitlab/gitlab.rb` or `/home/git/gitlab/config/gitlab.yml` for Omnibus GitLab and installations from source respectively. @@ -155,7 +153,7 @@ production: ... ``` -### Basic Configuration Settings +### Basic configuration settings | Setting | Description | Required | Examples | |--------------------|-------------|----------|----------| @@ -169,12 +167,12 @@ production: | `verify_certificates` | Enables SSL certificate verification if encryption method is `start_tls` or `simple_tls`. Defaults to true. | **{dotted-circle}** No | boolean | | `timeout` | Set a timeout, in seconds, for LDAP queries. This helps avoid blocking a request if the LDAP server becomes unresponsive. A value of `0` means there is no timeout. (default: `10`) | **{dotted-circle}** No | `10` or `30` | | `active_directory` | This setting specifies if LDAP server is Active Directory LDAP server. For non-AD servers it skips the AD specific queries. If your LDAP server is not AD, set this to false. | **{dotted-circle}** No | boolean | -| `allow_username_or_email_login` | If enabled, GitLab ignores everything after the first `@` in the LDAP username submitted by the user on sign-in. If you are using `uid: 'userPrincipalName'` on ActiveDirectory you need to disable this setting, because the userPrincipalName contains an `@`. | **{dotted-circle}** No | boolean | +| `allow_username_or_email_login` | If enabled, GitLab ignores everything after the first `@` in the LDAP username submitted by the user on sign-in. If you are using `uid: 'userPrincipalName'` on ActiveDirectory you must disable this setting, because the userPrincipalName contains an `@`. | **{dotted-circle}** No | boolean | | `block_auto_created_users` | To maintain tight control over the number of billable users on your GitLab installation, enable this setting to keep new users blocked until they have been cleared by an administrator (default: false). | **{dotted-circle}** No | boolean | | `base` | Base where we can search for users. | **{check-circle}** Yes | `'ou=people,dc=gitlab,dc=example'` or `'DC=mydomain,DC=com'` | | `user_filter` | Filter LDAP users. Format: [RFC 4515](https://tools.ietf.org/search/rfc4515) Note: GitLab does not support `omniauth-ldap`'s custom filter syntax. | **{dotted-circle}** No | For examples, read [Examples of user filters](#examples-of-user-filters). | | `lowercase_usernames` | If enabled, GitLab converts the name to lower case. | **{dotted-circle}** No | boolean | -| `retry_empty_result_with_codes` | An array of LDAP query response code that will attempt to retrying the operation if the result/content is empty. | **{dotted-circle}** No | `[80]` | +| `retry_empty_result_with_codes` | An array of LDAP query response code that attempt to retry the operation if the result/content is empty. For Google Secure LDAP, set this value to `[80]`. | **{dotted-circle}** No | `[80]` | #### Examples of user filters @@ -183,17 +181,17 @@ Some examples of the `user_filter` field syntax: - `'(employeeType=developer)'` - `'(&(objectclass=user)(|(samaccountname=momo)(samaccountname=toto)))'` -### SSL Configuration Settings +### SSL configuration settings | Setting | Description | Required | Examples | |---------------|-------------|----------|----------| -| `ca_file` | Specifies the path to a file containing a PEM-format CA certificate, for example, if you need to use an internal CA. | **{dotted-circle}** No | `'/etc/ca.pem'` | +| `ca_file` | Specifies the path to a file containing a PEM-format CA certificate, for example, if you need an internal CA. | **{dotted-circle}** No | `'/etc/ca.pem'` | | `ssl_version` | Specifies the SSL version for OpenSSL to use, if the OpenSSL default is not appropriate. | **{dotted-circle}** No | `'TLSv1_1'` | | `ciphers` | Specific SSL ciphers to use in communication with LDAP servers. | **{dotted-circle}** No | `'ALL:!EXPORT:!LOW:!aNULL:!eNULL:!SSLv2'` | | `cert` | Client certificate. | **{dotted-circle}** No | `'-----BEGIN CERTIFICATE----- <REDACTED> -----END CERTIFICATE -----'` | | `key` | Client private key. | **{dotted-circle}** No | `'-----BEGIN PRIVATE KEY----- <REDACTED> -----END PRIVATE KEY -----'` | -### Attribute Configuration Settings +### Attribute configuration settings LDAP attributes that GitLab uses to create an account for the LDAP user. The specified attribute can either be the attribute name as a string (for example, `'mail'`), or an @@ -208,7 +206,7 @@ The user's LDAP sign-in is the attribute specified as `uid` above. | `first_name` | LDAP attribute for user first name. Used when the attribute configured for `name` does not exist. | **{dotted-circle}** No | `'givenName'` | | `last_name` | LDAP attribute for user last name. Used when the attribute configured for `name` does not exist. | **{dotted-circle}** No | `'sn'` | -### LDAP Sync Configuration Settings **(PREMIUM SELF)** +### LDAP Sync configuration settings **(PREMIUM SELF)** | Setting | Description | Required | Examples | |-------------------|-------------|----------|----------| @@ -261,7 +259,7 @@ Support for nested members in the user filter shouldn't be confused with GitLab does not support the custom filter syntax used by OmniAuth LDAP. -#### Escaping special characters +#### Escape special characters The `user_filter` DN can contain special characters. For example: @@ -292,7 +290,7 @@ The `user_filter` DN can contain special characters. For example: OU=Gitlab \28Inc\29,DC=gitlab,DC=com ``` -### Enabling LDAP username lowercase +### Enable LDAP username lowercase Some LDAP servers, depending on their configurations, can return uppercase usernames. This can lead to several confusing issues such as creating links or namespaces with uppercase names. @@ -362,10 +360,10 @@ This does not disable [using LDAP credentials for Git access](#git-password-auth 1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. -### Using encrypted credentials +### Use encrypted credentials Instead of having the LDAP integration credentials stored in plaintext in the configuration files, you can optionally -use an encrypted file for the LDAP credentials. To use this feature, you first need to enable +use an encrypted file for the LDAP credentials. To use this feature, first you must enable [GitLab encrypted configuration](../../encrypted_configuration.md). The encrypted configuration for LDAP exists in an encrypted YAML file. By default the file is created at @@ -451,7 +449,7 @@ If initially your LDAP configuration looked like: ## Encryption -### TLS Server Authentication +### TLS server authentication There are two encryption methods, `simple_tls` and `start_tls`. @@ -461,7 +459,7 @@ exchanged but no validation of the LDAP server's SSL certificate is performed. ### Limitations -#### TLS Client Authentication +#### TLS client authentication Not implemented by `Net::LDAP`. @@ -555,7 +553,7 @@ The LDAP sync process: - Updates existing users. - Creates new users on first sign in. -### Adjusting LDAP user sync schedule **(PREMIUM SELF)** +### Adjust LDAP user sync schedule **(PREMIUM SELF)** By default, GitLab runs a worker once per day at 01:30 a.m. server time to check and update GitLab users against LDAP. @@ -592,7 +590,7 @@ sync to run once every 12 hours at the top of the hour. If your LDAP supports the `memberof` property, when the user signs in for the first time GitLab triggers a sync for groups the user should be a member of. -That way they don't need to wait for the hourly sync to be granted +That way they don't have to wait for the hourly sync to be granted access to their groups and projects. A group sync process runs every hour on the hour, and `group_base` must be set @@ -635,10 +633,10 @@ following. 1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. -To take advantage of group sync, group owners or maintainers need to [create one -or more LDAP group links](#adding-group-links). +To take advantage of group sync, group owners or maintainers must [create one +or more LDAP group links](#add-group-links). -### Adding group links **(PREMIUM SELF)** +### Add group links **(PREMIUM SELF)** For information on adding group links by using CNs and filters, refer to the [GitLab groups documentation](../../../user/group/index.md#manage-group-memberships-via-ldap). @@ -702,15 +700,15 @@ When enabled, the following applies: - Users are not allowed to share project with other groups or invite members to a project created in a group. -To enable it you need to: +To enable it, you must: 1. [Enable LDAP](#configuration) -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > General**. 1. Expand the **Visibility and access controls** section. 1. Ensure the **Lock memberships to LDAP synchronization** checkbox is selected. -### Adjusting LDAP group sync schedule **(PREMIUM SELF)** +### Adjust LDAP group sync schedule **(PREMIUM SELF)** By default, GitLab runs a group sync process every hour, on the hour. The values shown are in cron format. If needed, you can use a diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md index 15e8496e915..1952e8afa97 100644 --- a/doc/administration/auth/ldap/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap/ldap-troubleshooting.md @@ -145,7 +145,7 @@ may see the following message: `Access denied for your LDAP account`. We have a workaround, based on toggling the access level of affected users: -1. As an administrator, on the top bar, select **Menu >** **{admin}** **Admin**. +1. As an administrator, on the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Overview > Users**. 1. Select the name of the affected user. 1. In the user's administrative page, press **Edit** on the top right of the page. @@ -203,7 +203,7 @@ field contains no data: To resolve this: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, go to **Settings > General**. 1. Expand both of the following: - **Account and limit**. @@ -336,7 +336,7 @@ Gitlab::Auth::Ldap::Person.find_by_uid('<uid>', adapter) ### Group memberships **(PREMIUM SELF)** -#### Membership(s) not granted **(PREMIUM SELF)** +#### Membership(s) not granted Sometimes you may think a particular user should be added to a GitLab group via LDAP group sync, but for some reason it's not happening. There are several @@ -345,10 +345,10 @@ things to check to debug the situation. - Ensure LDAP configuration has a `group_base` specified. [This configuration](index.md#group-sync) is required for group sync to work properly. - Ensure the correct [LDAP group link is added to the GitLab - group](index.md#adding-group-links). + group](index.md#add-group-links). - Check that the user has an LDAP identity: 1. Sign in to GitLab as an administrator user. - 1. On the top bar, select **Menu >** **{admin}** **Admin**. + 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Overview > Users**. 1. Search for the user. 1. Open the user by clicking their name. Do not click **Edit**. @@ -356,7 +356,7 @@ things to check to debug the situation. an LDAP DN as the 'Identifier'. If not, this user hasn't signed in with LDAP yet and must do so first. - You've waited an hour or [the configured - interval](index.md#adjusting-ldap-group-sync-schedule) for the group to + interval](index.md#adjust-ldap-group-sync-schedule) for the group to sync. To speed up the process, either go to the GitLab group **Group information > Members** and press **Sync now** (sync one group) or [run the group sync Rake task](../../raketasks/ldap.md#run-a-group-sync) (sync all groups). @@ -395,7 +395,7 @@ group sync](#sync-all-groups) in the rails console and [look through the output](#example-console-output-after-a-group-sync) to see what happens when GitLab syncs the `admin_group`. -#### Sync all groups **(PREMIUM SELF)** +#### Sync all groups NOTE: To sync all groups manually when debugging is unnecessary, [use the Rake @@ -413,7 +413,7 @@ LdapAllGroupsSyncWorker.new.perform Next, [learn how to read the output](#example-console-output-after-a-group-sync). -##### Example console output after a group sync **(PREMIUM SELF)** +##### Example console output after a group sync Like the output from the user sync, the output from the [manual group sync](#sync-all-groups) is also very verbose. However, it contains lots @@ -503,7 +503,7 @@ stating as such: No `admin_group` configured for 'ldapmain' provider. Skipping ``` -#### Sync one group **(PREMIUM SELF)** +#### Sync one group [Syncing all groups](#sync-all-groups) can produce a lot of noise in the output, which can be distracting when you're only interested in troubleshooting the memberships of @@ -525,7 +525,7 @@ EE::Gitlab::Auth::Ldap::Sync::Group.execute_all_providers(group) The output is similar to [that you get from syncing all groups](#example-console-output-after-a-group-sync). -#### Query a group in LDAP **(PREMIUM SELF)** +#### Query a group in LDAP When you'd like to confirm that GitLab can read a LDAP group and see all its members, you can run the following: diff --git a/doc/administration/auth/smartcard.md b/doc/administration/auth/smartcard.md index 07c29984552..7e2699d5eb3 100644 --- a/doc/administration/auth/smartcard.md +++ b/doc/administration/auth/smartcard.md @@ -28,7 +28,7 @@ GitLab supports two authentication methods: ### Authentication against a local database with X.509 certificates -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/726) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.6 as an experimental feature. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/726) in GitLab 11.6 as an experimental feature. WARNING: Smartcard authentication against local databases may change or be removed completely in future @@ -55,7 +55,7 @@ Certificate: ### Authentication against a local database with X.509 certificates and SAN extension -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8605) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8605) in GitLab 12.3. Smartcards with X.509 certificates using SAN extensions can be used to authenticate with GitLab. @@ -98,7 +98,7 @@ Certificate: ### Authentication against an LDAP server -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7693) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8 as an experimental feature. Smartcard authentication against an LDAP server may change or be removed completely in future releases. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7693) in GitLab 11.8 as an experimental feature. Smartcard authentication against an LDAP server may change or be removed completely in the future. GitLab implements a standard way of certificate matching following [RFC4523](https://tools.ietf.org/html/rfc4523). It uses the diff --git a/doc/administration/cicd.md b/doc/administration/cicd.md new file mode 100644 index 00000000000..89fc31822ee --- /dev/null +++ b/doc/administration/cicd.md @@ -0,0 +1,75 @@ +--- +stage: Verify +group: Pipeline Execution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: howto +--- + +# GitLab CI/CD instance configuration **(FREE SELF)** + +GitLab CI/CD is enabled by default in all new projects on an instance. You can configure +the instance to have [GitLab CI/CD disabled by default](#disable-gitlab-cicd-in-new-projects) +in new projects. + +You can still choose to [enable GitLab CI/CD in individual projects](../ci/enable_or_disable_ci.md#enable-cicd-in-a-project) +at any time. + +## Disable GitLab CI/CD in new projects + +You can set GitLab CI/CD to be disabled by default in all new projects by modifying the settings in: + +- `gitlab.yml` for source installations. +- `gitlab.rb` for Omnibus GitLab installations. + +Existing projects that already had CI/CD enabled are unchanged. Also, this setting only changes +the project default, so project owners can still enable CI/CD in the project settings. + +For installations from source: + +1. Open `gitlab.yml` with your editor and set `builds` to `false`: + + ```yaml + ## Default project features settings + default_projects_features: + issues: true + merge_requests: true + wiki: true + snippets: false + builds: false + ``` + +1. Save the `gitlab.yml` file. + +1. Restart GitLab: + + ```shell + sudo service gitlab restart + ``` + +For Omnibus GitLab installations: + +1. Edit `/etc/gitlab/gitlab.rb` and add this line: + + ```ruby + gitlab_rails['gitlab_default_projects_features_builds'] = false + ``` + +1. Save the `/etc/gitlab/gitlab.rb` file. + +1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/clusters/kas.md b/doc/administration/clusters/kas.md index 8e5c162001e..6afaff73396 100644 --- a/doc/administration/clusters/kas.md +++ b/doc/administration/clusters/kas.md @@ -129,3 +129,25 @@ or the path to `config.yaml` inside the project is not valid. To fix this, ensure that the paths to the configuration repository and to the `config.yaml` file are correct. + +### KAS logs - `dial tcp <GITLAB_INTERNAL_IP>:443: connect: connection refused` + +If you are running a self-managed GitLab instance and: + +- The instance isn't running behind an SSL-terminating proxy. +- The instance doesn't have HTTPS configured on the GitLab instance itself. +- The instance's hostname resolves locally to its internal IP address. + +You may see the following error when the KAS tries to connect to the GitLab API: + +```json +{"level":"error","time":"2021-08-16T14:56:47.289Z","msg":"GetAgentInfo()","correlation_id":"01FD7QE35RXXXX8R47WZFBAXTN","grpc_service":"gitlab.agent.reverse_tunnel.rpc.ReverseTunnel","grpc_method":"Connect","error":"Get \"https://gitlab.example.com/api/v4/internal/kubernetes/agent_info\": dial tcp 172.17.0.4:443: connect: connection refused"} +``` + +To fix this for [Omnibus](https://docs.gitlab.com/omnibus/) package installations, +set the following parameter in `/etc/gitlab/gitlab.rb` +(replacing `gitlab.example.com` with your GitLab instance's hostname): + +```ruby +gitlab_kas['gitlab_address'] = 'http://gitlab.example.com' +``` diff --git a/doc/administration/compliance.md b/doc/administration/compliance.md index e356ef0366b..1761af1ffa1 100644 --- a/doc/administration/compliance.md +++ b/doc/administration/compliance.md @@ -31,3 +31,4 @@ relevant compliance standards. |**[Compliance frameworks](../user/project/settings/index.md#compliance-frameworks)**<br>Create a custom compliance framework at the group level to describe the type of compliance requirements any child project needs to follow. | Premium+ | **{check-circle}** Yes | Group | |**[Compliance pipelines](../user/project/settings/index.md#compliance-pipeline-configuration)**<br>Define a pipeline configuration to run for any projects with a given compliance framework. | Ultimate | **{check-circle}** Yes | Group | |**[Compliance report](../user/compliance/compliance_report/index.md)**<br>Quickly get visibility into the compliance posture of your organization. | Ultimate | **{check-circle}** Yes | Group | +|**[External Status Checks](../user/project/merge_requests/status_checks.md)**<br>Interface with third party systems you already use during development to ensure you remain compliant. | Ultimate | **{check-circle}** Yes | Project | diff --git a/doc/administration/configure.md b/doc/administration/configure.md index 73fbf527fe1..d3e37b4a0ee 100644 --- a/doc/administration/configure.md +++ b/doc/administration/configure.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Configure your GitLab installation +# Configure your GitLab installation **(FREE SELF)** Customize and configure your self-managed GitLab installation. diff --git a/doc/administration/consul.md b/doc/administration/consul.md index c88047c4c61..72b3487a549 100644 --- a/doc/administration/consul.md +++ b/doc/administration/consul.md @@ -21,7 +21,7 @@ Before configuring Consul: 1. Review the [reference architecture](reference_architectures/index.md#available-reference-architectures) documentation to determine the number of Consul server nodes you should have. -1. If necessary, ensure the [appropriate ports are open](https://docs.gitlab.com/omnibus/package-information/defaults.html#ports) in your firewall. +1. If necessary, ensure the [appropriate ports are open](package_information/defaults.md#ports) in your firewall. ## Configure the Consul nodes diff --git a/doc/administration/database_load_balancing.md b/doc/administration/database_load_balancing.md index 7d17b22a4d7..45f27a8a8f2 100644 --- a/doc/administration/database_load_balancing.md +++ b/doc/administration/database_load_balancing.md @@ -117,9 +117,9 @@ For Sidekiq, we can define [data consistency](../development/sidekiq_style_guide.md#job-data-consistency-strategies) requirements for a specific job. -## Service Discovery +## Service Discovery **(PREMIUM SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/5883) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.0. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/5883) in GitLab 11.0. Service discovery allows GitLab to automatically retrieve a list of secondary databases to use, instead of having to manually specify these in the @@ -237,9 +237,9 @@ For example: {"severity":"INFO","time":"2019-09-02T12:12:01.728Z","correlation_id":"abcdefg","event":"host_online","message":"Host came back online","db_host":"111.222.333.444","db_port":null,"tag":"rails.database_load_balancing","environment":"production","hostname":"web-example-1","fqdn":"gitlab.example.com","path":null,"params":null} ``` -## Handling Stale Reads +## Handling Stale Reads **(PREMIUM SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3526) in [GitLab Premium](https://about.gitlab.com/pricing/) 10.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3526) in GitLab 10.3. To prevent reading from an outdated secondary the load balancer checks if it is in sync with the primary. If the data is determined to be recent enough the diff --git a/doc/administration/encrypted_configuration.md b/doc/administration/encrypted_configuration.md index 8afe30d20ab..9224def4a5a 100644 --- a/doc/administration/encrypted_configuration.md +++ b/doc/administration/encrypted_configuration.md @@ -11,8 +11,8 @@ type: reference GitLab can read settings for certain features from encrypted settings files. The supported features are: -- [LDAP `user_bn` and `password`](auth/ldap/index.md#using-encrypted-credentials) -- [SMTP `user_name` and `password`](raketasks/smtp.md#secrets) +- [LDAP `user_bn` and `password`](auth/ldap/index.md#use-encrypted-credentials). +- [SMTP `user_name` and `password`](raketasks/smtp.md#secrets). In order to enable the encrypted configuration settings, a new base key needs to be generated for `encrypted_settings_key_base`. The secret can be generated in the following ways: @@ -35,4 +35,4 @@ The new secret can be generated by running: bundle exec rake gitlab:env:info RAILS_ENV=production GITLAB_GENERATE_ENCRYPTED_SETTINGS_KEY_BASE=true ``` -This prints general information on the GitLab instance, but also causes the key to be generated in `<path-to-gitlab-rails>/config/secrets.yml` +This prints general information on the GitLab instance, but also causes the key to be generated in `<path-to-gitlab-rails>/config/secrets.yml`. diff --git a/doc/administration/geo/disaster_recovery/background_verification.md b/doc/administration/geo/disaster_recovery/background_verification.md index 65e0ffd4366..caa806c92c8 100644 --- a/doc/administration/geo/disaster_recovery/background_verification.md +++ b/doc/administration/geo/disaster_recovery/background_verification.md @@ -60,7 +60,7 @@ Feature.enable('geo_repository_verification') On the **primary** node: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Geo > Nodes**. 1. Expand **Verification information** tab for that node to view automatic checksumming status for repositories and wikis. Successes are shown in green, pending work @@ -70,7 +70,7 @@ On the **primary** node: On the **secondary** node: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Geo > Nodes**. 1. Expand **Verification information** tab for that node to view automatic checksumming status for repositories and wikis. Successes are shown in green, pending work @@ -89,7 +89,7 @@ in sync. ## Repository re-verification -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/8550) in GitLab Enterprise Edition 11.6. Available in [GitLab Premium](https://about.gitlab.com/pricing/). +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/8550) in GitLab 11.6. Due to bugs or transient infrastructure failures, it is possible for Git repositories to change unexpectedly without being marked for verification. @@ -100,7 +100,7 @@ increase load and vice versa. On the **primary** node: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Geo > Nodes**. 1. Select **Edit** for the **primary** node to customize the minimum re-verification interval: @@ -151,7 +151,7 @@ sudo gitlab-rake geo:verification:wiki:reset If the **primary** and **secondary** nodes have a checksum verification mismatch, the cause may not be apparent. To find the cause of a checksum mismatch: 1. On the **primary** node: - 1. On the top bar, select **Menu >** **{admin}** **Admin**. + 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Overview > Projects**. 1. Find the project that you want to check the checksum differences and select its name. diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index 8b55bebb42b..a7a64701cbd 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -111,7 +111,7 @@ ensure these processes are close to 100% as possible during active use. On the **secondary** node: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Geo > Nodes**. Replicated objects (shown in green) should be close to 100%, and there should be no failures (shown in red). If a large proportion of @@ -139,7 +139,7 @@ This [content was moved to another location](background_verification.md). On the **primary** node: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Messages**. 1. Add a message notifying users on the maintenance window. You can check under **Geo > Nodes** to estimate how long it @@ -152,7 +152,7 @@ To ensure that all data is replicated to a secondary site, updates (write reques be disabled on the **primary** site: 1. Enable [maintenance mode](../../maintenance_mode/index.md) on the **primary** node. -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Cron**. 1. Select `Disable All` to disable non-Geo periodic background jobs. @@ -165,7 +165,7 @@ be disabled on the **primary** site: 1. If you are manually replicating any data not managed by Geo, trigger the final replication process now. 1. On the **primary** node: - 1. On the top bar, select **Menu >** **{admin}** **Admin**. + 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Queues**, and wait for all queues except those with `geo` in the name to drop to 0. @@ -180,7 +180,7 @@ be disabled on the **primary** site: - The Geo log cursor is up to date (0 events behind). 1. On the **secondary** node: - 1. On the top bar, select **Menu >** **{admin}** **Admin**. + 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Queues**, and wait for all the `geo` queues to drop to 0 queued and 0 running jobs. diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md index 27990748071..4255fba83f6 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md @@ -65,7 +65,7 @@ promote a Geo replica and perform a failover. On the **secondary** node: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Geo > Nodes** to see its status. Replicated objects (shown in green) should be close to 100%, and there should be no failures (shown in red). If a large proportion of @@ -130,7 +130,7 @@ follow these steps to avoid unnecessary data loss: connection. 1. On the **primary** node: - 1. On the top bar, select **Menu >** **{admin}** **Admin**. + 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dhasboard, select **Cron**. 1. Select `Disable All` to disable any non-Geo periodic background jobs. @@ -148,7 +148,7 @@ follow these steps to avoid unnecessary data loss: [data not managed by Geo](../../replication/datatypes.md#limitations-on-replicationverification), trigger the final replication process now. 1. On the **primary** node: - 1. On the top bar, select **Menu >** **{admin}** **Admin**. + 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Queues**, and wait for all queues except those with `geo` in the name to drop to 0. @@ -163,7 +163,7 @@ follow these steps to avoid unnecessary data loss: - The Geo log cursor is up to date (0 events behind). 1. On the **secondary** node: - 1. On the top bar, select **Menu >** **{admin}** **Admin**. + 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Queues**, and wait for all the `geo` queues to drop to 0 queued and 0 running jobs. diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md index 9d5e65cd194..18923da1056 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md @@ -115,7 +115,7 @@ follow these steps to avoid unnecessary data loss: connection. 1. On the **primary** node: - 1. On the top bar, select **Menu >** **{admin}** **Admin**. + 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dhasboard, select **Cron**. 1. Select `Disable All` to disable any non-Geo periodic background jobs. @@ -133,7 +133,7 @@ follow these steps to avoid unnecessary data loss: [data not managed by Geo](../../replication/datatypes.md#limitations-on-replicationverification), trigger the final replication process now. 1. On the **primary** node: - 1. On the top bar, select **Menu >** **{admin}** **Admin**. + 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Queues**, and wait for all queues except those with `geo` in the name to drop to 0. @@ -148,7 +148,7 @@ follow these steps to avoid unnecessary data loss: - The Geo log cursor is up to date (0 events behind). 1. On the **secondary** node: - 1. On the top bar, select **Menu >** **{admin}** **Admin**. + 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Queues**, and wait for all the `geo` queues to drop to 0 queued and 0 running jobs. diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index 7175d41abd8..48091967189 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -7,12 +7,6 @@ type: howto # Geo **(PREMIUM SELF)** -> - Introduced in GitLab Enterprise Edition 8.9. -> - Using Geo in combination with -> [multi-node architectures](../reference_architectures/index.md) -> is considered **Generally Available** (GA) in -> [GitLab Premium](https://about.gitlab.com/pricing/) 10.4. - Geo is the solution for widely distributed development teams and for providing a warm-standby as part of a disaster recovery strategy. ## Overview @@ -144,7 +138,7 @@ The following table lists basic ports that must be open between the **primary** | 22 | 22 | TCP | | 5432 | | PostgreSQL | -See the full list of ports used by GitLab in [Package defaults](https://docs.gitlab.com/omnibus/package-information/defaults.html) +See the full list of ports used by GitLab in [Package defaults](../package_information/defaults.md) NOTE: [Web terminal](../../ci/environments/index.md#web-terminals) support requires your load balancer to correctly handle WebSocket connections. @@ -214,11 +208,11 @@ For information on configuring Geo, see [Geo configuration](replication/configur ### Updating Geo -For information on how to update your Geo site(s) to the latest GitLab version, see [Updating the Geo sites](replication/updating_the_geo_nodes.md). +For information on how to update your Geo site(s) to the latest GitLab version, see [Updating the Geo sites](replication/updating_the_geo_sites.md). ### Pausing and resuming replication -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35913) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.2. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35913) in GitLab 13.2. WARNING: In GitLab 13.2 and 13.3, promoting a secondary site to a primary while the @@ -230,7 +224,7 @@ WARNING: Pausing and resuming of replication is currently only supported for Geo installations using an Omnibus GitLab-managed database. External databases are currently not supported. -In some circumstances, like during [upgrades](replication/updating_the_geo_nodes.md) or a [planned failover](disaster_recovery/planned_failover.md), it is desirable to pause replication between the primary and secondary. +In some circumstances, like during [upgrades](replication/updating_the_geo_sites.md) or a [planned failover](disaster_recovery/planned_failover.md), it is desirable to pause replication between the primary and secondary. Pausing and resuming replication is done via a command line tool from the a node in the secondary site where the `postgresql` service is enabled. diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index 5b22741f578..88f1ad5b490 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -199,7 +199,7 @@ keys must be manually replicated to the **secondary** site. gitlab-ctl reconfigure ``` -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Geo > Sites**. 1. Select **New site**.  @@ -257,7 +257,7 @@ method to be enabled. This is enabled by default, but if converting an existing On the **primary** site: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > General**. 1. Expand **Visibility and access controls**. 1. Ensure "Enabled Git access protocols" is set to either "Both SSH and HTTP(S)" or "Only HTTP(S)". @@ -267,7 +267,7 @@ On the **primary** site: You can sign in to the **secondary** site with the same credentials you used with the **primary** site. After you sign in: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Geo > Sites**. 1. Verify that it's correctly identified as a **secondary** Geo site, and that Geo is enabled. @@ -338,7 +338,7 @@ when: ## Upgrading Geo -See the [updating the Geo sites document](updating_the_geo_nodes.md). +See the [updating the Geo sites document](updating_the_geo_sites.md). ## Troubleshooting diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index a696e5410e5..3f38436429a 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -47,12 +47,16 @@ verification methods: | Blobs | Container registry _(object storage)_ | Geo with API/Managed/Docker API (*2*) | _Not implemented_ | | Blobs | Package registry _(file system)_ | Geo with API | SHA256 checksum | | Blobs | Package registry _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | Infrastructure registry _(file system)_ | Geo with API | SHA256 checksum | +| Blobs | Infrastructure registry _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | Versioned Terraform State _(file system)_ | Geo with API | SHA256 checksum | | Blobs | Versioned Terraform State _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | External Merge Request Diffs _(file system)_ | Geo with API | _Not implemented_ | | Blobs | External Merge Request Diffs _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | Pipeline artifacts _(file system)_ | Geo with API | SHA256 checksum | | Blobs | Pipeline artifacts _(object storage)_ | Geo with API/Managed (*2*) | SHA256 checksum | +| Blobs | Pages _(file system)_ | Geo with API | _Not implemented_ | +| Blobs | Pages _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | - (*1*): Redis replication can be used as part of HA with Redis sentinel. It's not used between Geo sites. - (*2*): Object storage replication can be performed by Geo or by your object storage provider/appliance @@ -189,21 +193,15 @@ successfully, you must replicate their data using some other means. |[Job logs](../../job_logs.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. | |[Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | No | Disabled by default. See [instructions](docker_registry.md) to enable. | |[Content in object storage (beta)](object_storage.md) | **Yes** (12.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) | No | | -|[Infrastructure Registry for Terraform Module](../../../user/packages/terraform_module_registry/index.md) | **Yes** (14.0) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (14.0) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Infrastructure Registry](../../../user/packages/infrastructure_registry/index.md) | **Yes** (14.0) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (14.0) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | |[Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | No | Designs also require replication of LFS objects and Uploads. | -|[Package Registry for npm](../../../user/packages/npm_registry/index.md) | **Yes** (13.2) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | -|[Package Registry for Maven](../../../user/packages/maven_repository/index.md) | **Yes** (13.2) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | -|[Package Registry for Conan](../../../user/packages/conan_repository/index.md) | **Yes** (13.2) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | -|[Package Registry for NuGet](../../../user/packages/nuget_repository/index.md) | **Yes** (13.2) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | -|[Package Registry for PyPI](../../../user/packages/pypi_repository/index.md) | **Yes** (13.2) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | -|[Package Registry for Composer](../../../user/packages/composer_repository/index.md) | **Yes** (13.2) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | -|[Package Registry for generic packages](../../../user/packages/generic_packages/index.md) | **Yes** (13.5) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry](../../../user/packages/package_registry/index.md) | **Yes** (13.2) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | |[Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.12) | Via Object Storage provider if supported. Native Geo support (Beta). | Replication is behind the feature flag `geo_terraform_state_version_replication`, enabled by default. Verification was behind the feature flag `geo_terraform_state_version_verification`, which was removed in 14.0| |[External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Replication is behind the feature flag `geo_merge_request_diff_replication`, enabled by default. Verification is under development, behind the feature flag `geo_merge_request_diff_verification`, introduced in 14.0.| |[Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [**Yes** (14.2)](https://gitlab.com/groups/gitlab-org/-/epics/2810) | No | Verification was implemented behind the feature flag `geo_snippet_repository_verification` in 13.11, and the feature flag was removed in 14.2. | |[Server-side Git hooks](../../server_hooks.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1867) | No | No | | |[Elasticsearch integration](../../../integration/elasticsearch.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | No | | -|[GitLab Pages](../../pages/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | Via Object Storage provider if supported. **No** native Geo support (Beta). | | +|[GitLab Pages](../../pages/index.md) | [**Yes** (14.3)](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_pages_deployment_replication`, enabled by default. | |[Dependency proxy images](../../../user/packages/dependency_proxy/index.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/259694) | No | No | Blocked on [Geo: Secondary Mimicry](https://gitlab.com/groups/gitlab-org/-/epics/1528). Replication of this cache is not needed for Disaster Recovery purposes because it can be recreated from external sources. | |[Vulnerability Export](../../../user/application_security/vulnerability_report/#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | | Not planned because they are ephemeral and sensitive. They can be regenerated on demand. | diff --git a/doc/administration/geo/replication/disable_geo.md b/doc/administration/geo/replication/disable_geo.md index 485a5ee1950..ae2597ad649 100644 --- a/doc/administration/geo/replication/disable_geo.md +++ b/doc/administration/geo/replication/disable_geo.md @@ -36,7 +36,7 @@ to do that. To remove the **primary** site: 1. [Remove all secondary Geo sites](remove_geo_site.md) -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Geo > Nodes**. 1. Select **Remove** for the **primary** node. 1. Confirm by selecting **Remove** when the prompt appears. diff --git a/doc/administration/geo/replication/docker_registry.md b/doc/administration/geo/replication/docker_registry.md index 5cc4f66017b..4004ef3c17f 100644 --- a/doc/administration/geo/replication/docker_registry.md +++ b/doc/administration/geo/replication/docker_registry.md @@ -130,7 +130,7 @@ For each application and Sidekiq node on the **secondary** site: To verify Container Registry replication is working, on the **secondary** site: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Geo > Nodes**. The initial replication, or "backfill", is probably still in progress. diff --git a/doc/administration/geo/replication/faq.md b/doc/administration/geo/replication/faq.md index 28030dccb3b..70a6e506c28 100644 --- a/doc/administration/geo/replication/faq.md +++ b/doc/administration/geo/replication/faq.md @@ -52,7 +52,7 @@ query. ## Can I `git push` to a **secondary** site? -Yes! Pushing directly to a **secondary** site (for both HTTP and SSH, including Git LFS) was [introduced](https://about.gitlab.com/releases/2018/09/22/gitlab-11-3-released/) in [GitLab Premium](https://about.gitlab.com/pricing/#self-managed) 11.3. +Yes! Pushing directly to a **secondary** site (for both HTTP and SSH, including Git LFS) was [introduced](https://about.gitlab.com/releases/2018/09/22/gitlab-11-3-released/) in GitLab 11.3. ## How long does it take to have a commit replicated to a **secondary** site? diff --git a/doc/administration/geo/replication/object_storage.md b/doc/administration/geo/replication/object_storage.md index d3931c0ba80..1f799b30125 100644 --- a/doc/administration/geo/replication/object_storage.md +++ b/doc/administration/geo/replication/object_storage.md @@ -9,15 +9,19 @@ type: howto Geo can be used in combination with Object Storage (AWS S3, or other compatible object storage). -The storage method for files is recorded in the database, and the database is replicated -from the **primary** Geo site to the **secondary** Geo site, so the **secondary** Geo site -must match the storage method of the **primary** Geo site. -Therefore, if the **primary** Geo site uses object storage, the **secondary** Geo site must use it too. - Currently, **secondary** sites can use either: - The same storage bucket as the **primary** site. - A replicated storage bucket. +- Local storage, if the primary uses local storage. + +The storage method (local or object storage) for files is recorded in the database, and the database +is replicated from the **primary** Geo site to the **secondary** Geo site. + +When accessing an uploaded object, we get its storage method (local or object storage) from the +database, so the **secondary** Geo site must match the storage method of the **primary** Geo site. + +Therefore, if the **primary** Geo site uses object storage, the **secondary** Geo site must use it too. To have: @@ -38,7 +42,7 @@ whether they are stored on the local file system or in object storage. To enable GitLab replication: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Geo > Nodes**. 1. Select **Edit** on the **secondary** site. 1. In the **Synchronization Settings** section, find the **Allow this secondary node to replicate content on Object Storage** diff --git a/doc/administration/geo/replication/remove_geo_site.md b/doc/administration/geo/replication/remove_geo_site.md index 274eb28dbc9..b69dfe2e723 100644 --- a/doc/administration/geo/replication/remove_geo_site.md +++ b/doc/administration/geo/replication/remove_geo_site.md @@ -9,7 +9,7 @@ type: howto **Secondary** sites can be removed from the Geo cluster using the Geo administration page of the **primary** site. To remove a **secondary** site: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Geo > Nodes**. 1. Select the **Remove** button for the **secondary** site you want to remove. 1. Confirm by selecting **Remove** when the prompt appears. diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index e072696b37c..7b82d742bd5 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -27,7 +27,7 @@ Before attempting more advanced troubleshooting: On the **primary** node: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Geo > Nodes**. We perform the following health checks on each **secondary** node @@ -610,7 +610,7 @@ to start again from scratch, there are a few steps that can help you: ### Design repository failures on mirrored projects and project imports -On the top bar, under **Menu >** **{admin}** **Admin > Geo > Nodes**, +On the top bar, under **Menu > Admin > Geo > Nodes**, if the Design repositories progress bar shows `Synced` and `Failed` greater than 100%, and negative `Queued`, then the instance is likely affected by @@ -836,7 +836,7 @@ node's URL matches its external URL. On the **primary** node: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Geo > Nodes**. 1. Find the affected **secondary** site and select **Edit**. 1. Ensure the **URL** field matches the value found in `/etc/gitlab/gitlab.rb` diff --git a/doc/administration/geo/replication/tuning.md b/doc/administration/geo/replication/tuning.md index 9807f3e6444..bcff6181296 100644 --- a/doc/administration/geo/replication/tuning.md +++ b/doc/administration/geo/replication/tuning.md @@ -14,7 +14,7 @@ in the background. On the **primary** site: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Geo > Nodes**. 1. Select **Edit** of the secondary node you want to tune. 1. Under **Tuning settings**, there are several variables that can be tuned to diff --git a/doc/administration/geo/replication/updating_the_geo_nodes.md b/doc/administration/geo/replication/updating_the_geo_nodes.md index 03570048071..f07c8d547a4 100644 --- a/doc/administration/geo/replication/updating_the_geo_nodes.md +++ b/doc/administration/geo/replication/updating_the_geo_nodes.md @@ -1,52 +1,9 @@ --- -stage: Enablement -group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: howto +redirect_to: 'updating_the_geo_sites.md' +remove_date: '2021-11-23' --- -# Updating the Geo nodes **(PREMIUM SELF)** +This file was moved to [another location](updating_the_geo_sites.md). -WARNING: -Read these sections carefully before updating your Geo nodes. Not following -version-specific update steps may result in unexpected downtime. If you have -any specific questions, [contact Support](https://about.gitlab.com/support/#contact-support). - -Updating Geo nodes involves performing: - -1. [Version-specific update steps](version_specific_updates.md), depending on the - version being updated to or from. -1. [General update steps](#general-update-steps), for all updates. - -## General update steps - -NOTE: -These general update steps are not intended for [high-availability deployments](https://docs.gitlab.com/omnibus/update/README.html#multi-node--ha-deployment), and will cause downtime. If you want to avoid downtime, consider using [zero downtime updates](https://docs.gitlab.com/omnibus/update/README.html#zero-downtime-updates). - -To update the Geo nodes when a new GitLab version is released, update **primary** -and all **secondary** nodes: - -1. **Optional:** [Pause replication on each **secondary** node.](../index.md#pausing-and-resuming-replication) -1. Log into the **primary** node. -1. [Update GitLab on the **primary** node using Omnibus](https://docs.gitlab.com/omnibus/update/#update-using-the-official-repositories). -1. Log into each **secondary** node. -1. [Update GitLab on each **secondary** node using Omnibus](https://docs.gitlab.com/omnibus/update/#update-using-the-official-repositories). -1. If you paused replication in step 1, [resume replication on each **secondary**](../index.md#pausing-and-resuming-replication) -1. [Test](#check-status-after-updating) **primary** and **secondary** nodes, and check version in each. - -### Check status after updating - -Now that the update process is complete, you may want to check whether -everything is working correctly: - -1. Run the Geo Rake task on all nodes, everything should be green: - - ```shell - sudo gitlab-rake gitlab:geo:check - ``` - -1. Check the **primary** node's Geo dashboard for any errors. -1. Test the data replication by pushing code to the **primary** node and see if it - is received by **secondary** nodes. - -If you encounter any issues, see the [Geo troubleshooting guide](troubleshooting.md). +<!-- This redirect file can be deleted after <2021-11-23>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/geo/replication/updating_the_geo_sites.md b/doc/administration/geo/replication/updating_the_geo_sites.md new file mode 100644 index 00000000000..e82afe5f0d4 --- /dev/null +++ b/doc/administration/geo/replication/updating_the_geo_sites.md @@ -0,0 +1,54 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: howto +--- + +# Updating the Geo sites **(PREMIUM SELF)** + +WARNING: +Read these sections carefully before updating your Geo sites. Not following +version-specific update steps may result in unexpected downtime. If you have +any specific questions, [contact Support](https://about.gitlab.com/support/#contact-support). + +Updating Geo sites involves performing: + +1. [Version-specific update steps](version_specific_updates.md), depending on the + version being updated to or from. +1. [General update steps](#general-update-steps), for all updates. + +## General update steps + +NOTE: +These general update steps are not intended for multi-site deployments, +and will cause downtime. If you want to avoid downtime, consider using +[zero downtime upgrades](../../../update/zero_downtime.md#multi-node--ha-deployment-with-geo). + +To update the Geo sites when a new GitLab version is released, update **primary** +and all **secondary** sites: + +1. **Optional:** [Pause replication on each **secondary** sites.](../index.md#pausing-and-resuming-replication) +1. SSH into each node of the **primary** site. +1. [Upgrade GitLab on the **primary** site](../../../update/package/index.md#upgrade-using-the-official-repositories). +1. SSH into each node of **secondary** sites. +1. [Upgrade GitLab on each **secondary** site](../../../update/package/index.md#upgrade-using-the-official-repositories). +1. If you paused replication in step 1, [resume replication on each **secondary**](../index.md#pausing-and-resuming-replication) +1. [Test](#check-status-after-updating) **primary** and **secondary** sites, and check version in each. + +### Check status after updating + +Now that the update process is complete, you may want to check whether +everything is working correctly: + +1. Run the Geo Rake task on an application node for the primary and secondary sites. Everything should be green: + + ```shell + sudo gitlab-rake gitlab:geo:check + ``` + +1. Check the **primary** site's Geo dashboard for any errors. +1. Test the data replication by pushing code to the **primary** site and see if it + is received by **secondary** sites. + +If you encounter any issues, see the [Geo troubleshooting guide](troubleshooting.md). diff --git a/doc/administration/geo/replication/usage.md b/doc/administration/geo/replication/usage.md index 7fe8eec467e..f3c8f6ac759 100644 --- a/doc/administration/geo/replication/usage.md +++ b/doc/administration/geo/replication/usage.md @@ -11,7 +11,7 @@ type: howto After you set up the [database replication and configure the Geo nodes](../index.md#setup-instructions), use your closest GitLab site as you would do with the primary one. -You can push directly to a **secondary** site (for both HTTP, SSH including Git LFS), and the request will be proxied to the primary site instead ([introduced](https://about.gitlab.com/releases/2018/09/22/gitlab-11-3-released/) in [GitLab Premium](https://about.gitlab.com/pricing/#self-managed) 11.3). +You can push directly to a **secondary** site (for both HTTP, SSH including Git LFS), and the request will be proxied to the primary site instead ([introduced](https://about.gitlab.com/releases/2018/09/22/gitlab-11-3-released/) in GitLab 11.3). Example of the output you will see when pushing to a **secondary** site: @@ -33,3 +33,13 @@ you can't store credentials in the URL like `user:password@URL`. Instead, you ca for Unix-like operating systems or `_netrc` for Windows. In that case, the credentials will be stored as a plain text. If you're looking for a more secure way to store credentials, you can use [Git Credential Storage](https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage). + +## Fetch Go modules from Geo secondary sites + +Go modules can be pulled from secondary sites, with a number of limitations: + +- Git configuration (using `insteadOf`) is needed to fetch data from the Geo secondary site. +- For private projects, authentication details need to be specified in `~/.netrc`. + +Read more in the +[working with projects `go get` documentation](../../../user/project/working_with_projects.md#fetch-go-modules-from-geo-secondary-sites). diff --git a/doc/administration/geo/replication/version_specific_updates.md b/doc/administration/geo/replication/version_specific_updates.md index 155c06f90b8..84193e6baac 100644 --- a/doc/administration/geo/replication/version_specific_updates.md +++ b/doc/administration/geo/replication/version_specific_updates.md @@ -8,9 +8,44 @@ type: howto # Version-specific update instructions **(PREMIUM SELF)** Review this page for update instructions for your version. These steps -accompany the [general steps](updating_the_geo_nodes.md#general-update-steps) +accompany the [general steps](updating_the_geo_sites.md#general-update-steps) for updating Geo nodes. +## Updating to 14.1, 14.2, 14.3 + +We found an [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/336013) where the Container Registry replication wasn't fully working if you used multi-arch images. In case of a multi-arch image, only the primary architecture (for example `amd64`) would be replicated to the secondary node. This has been [fixed in GitLab 14.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67624) and was backported to 14.2 and 14.1, but manual steps are required to force a re-sync. + +You can check if you are affected by running: + +```shell +docker manifest inspect <SECONDARY_IMAGE_LOCATION> | jq '.mediaType' +``` + +Where `<SECONDARY_IMAGE_LOCATION>` is a container image on your secondary node. +If the output matches `application/vnd.docker.distribution.manifest.list.v2+json` +(there can be a `mediaType` entry at several levels, we only care about the top level entry), +then you don't need to do anything. + +Otherwise, on all your **secondary** nodes, in a [Rails console](../../operations/rails_console.md), run the following: + + ```ruby + list_type = 'application/vnd.docker.distribution.manifest.list.v2+json' + + Geo::ContainerRepositoryRegistry.synced.each do |gcr| + cr = gcr.container_repository + primary = Geo::ContainerRepositorySync.new(cr) + cr.tags.each do |tag| + primary_manifest = JSON.parse(primary.send(:client).repository_raw_manifest(cr.path, tag.name)) + next unless primary_manifest['mediaType'].eql?(list_type) + + cr.delete_tag_by_name(tag.name) + end + primary.execute + end + ``` + +If you are running a version prior to 14.1 and are using Geo and multi-arch containers in your Container Registry, we recommend [upgrading](updating_the_geo_sites.md) to at least GitLab 14.1. + ## Updating to GitLab 14.0/14.1 We found an issue where [Primary sites can not be removed from the UI](https://gitlab.com/gitlab-org/gitlab/-/issues/338231). @@ -277,7 +312,7 @@ GitLab 12.2 includes the following minor PostgreSQL updates: This update occurs even if major PostgreSQL updates are disabled. -Before [refreshing Foreign Data Wrapper during a Geo upgrade](https://docs.gitlab.com/omnibus/update/README.html#run-post-deployment-migrations-and-checks), +Before [refreshing Foreign Data Wrapper during a Geo upgrade](../../../update/zero_downtime.md#step-4-run-post-deployment-migrations-and-checks), restart the Geo tracking database: ```shell diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index fa343f7eb40..d72bb0737ae 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -501,6 +501,79 @@ two other clusters of nodes supporting a Geo **secondary** site. One for the main database and the other for the tracking database. For more information, see [High Availability with Omnibus GitLab](../../postgresql/replication_and_failover.md). +### Changing the replication password + +To change the password for the [replication user](https://wiki.postgresql.org/wiki/Streaming_Replication) +when using Omnibus-managed PostgreSQL instances: + +On the GitLab Geo **primary** server: + +1. The default value for the replication user is `gitlab_replicator`, but if you've set a custom replication + user in your `/etc/gitlab/gitlab.rb` under the `postgresql['sql_replication_user']` setting, make sure to + adapt the following instructions for your own user. + + Generate an MD5 hash of the desired password: + + ```shell + sudo gitlab-ctl pg-password-md5 gitlab_replicator + # Enter password: <your_password_here> + # Confirm password: <your_password_here> + # 950233c0dfc2f39c64cf30457c3b7f1e + ``` + + Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab_replicator` + postgresql['sql_replication_password'] = '<md5_hash_of_your_password>' + ``` + +1. Save the file and reconfigure GitLab to change the replication user's password in PostgreSQL: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. Restart PostgreSQL for the replication password change to take effect: + + ```shell + sudo gitlab-ctl restart postgresql + ``` + +Until the password is updated on any **secondary** servers, the [PostgreSQL log](../../logs.md#postgresql-logs) on +the secondaries will report the following error message: + +```console +FATAL: could not connect to the primary server: FATAL: password authentication failed for user "gitlab_replicator" +``` + +On all GitLab Geo **secondary** servers: + +1. The first step isn't necessary from a configuration perspective, since the hashed `'sql_replication_password'` + is not used on the GitLab Geo **secondary**. However in the event that **secondary** needs to be promoted + to the GitLab Geo **primary**, make sure to match the `'sql_replication_password'` in the secondary + server configuration. + + Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab_replicator` on the Geo primary + postgresql['sql_replication_password'] = '<md5_hash_of_your_password>' + ``` + +1. During the initial replication setup, the `gitlab-ctl replicate-geo-database` command writes the plaintext + password for the replication user account to two locations: + + - `gitlab-geo.conf`: Used by the PostgreSQL replication process, written to the PostgreSQL data + directory, by default at `/var/opt/gitlab/postgresql/data/gitlab-geo.conf`. + - `.pgpass`: Used by the `gitlab-psql` user, located by default at `/var/opt/gitlab/postgresql/.pgpass`. + + Update the plaintext password in both of these files, and restart PostgreSQL: + + ```shell + sudo gitlab-ctl restart postgresql + ``` + ## Multi-node database replication In GitLab 14.0, Patroni replaced `repmgr` as the supported @@ -521,7 +594,7 @@ For instructions about how to set up Patroni on the primary site, see the #### Configuring Patroni cluster for a Geo secondary site -In a Geo secondary site, the main PostgreSQL database is a read-only replica of the primary site’s PostgreSQL database. +In a Geo secondary site, the main PostgreSQL database is a read-only replica of the primary site's PostgreSQL database. If you are currently using `repmgr` on your Geo primary site, see [these instructions](#migrating-from-repmgr-to-patroni) for migrating from `repmgr` to Patroni. @@ -651,7 +724,7 @@ Refer to your preferred Load Balancer's documentation for further guidance. ##### Step 3. Configure a PgBouncer node on the secondary site A production-ready and highly available configuration requires at least -three Consul nodes, a minimum of one PgBouncer node, but it’s recommended to have +three Consul nodes, a minimum of one PgBouncer node, but it's recommended to have one per database node. An internal load balancer (TCP) is required when there is more than one PgBouncer service nodes. The internal load balancer provides a single endpoint for connecting to the PgBouncer cluster. For more information, diff --git a/doc/administration/geo/setup/external_database.md b/doc/administration/geo/setup/external_database.md index 56d196b54ec..13dbf9d1434 100644 --- a/doc/administration/geo/setup/external_database.md +++ b/doc/administration/geo/setup/external_database.md @@ -249,7 +249,7 @@ the tracking database on port 5432. gitlab-rake geo:db:create ``` -1. The reconfigure should automatically migrate the database. You can migrate the database manually if needed, for example if `gitlab_rails['auto_migrate'] = false`: +1. The reconfigure should automatically migrate the database. You can migrate the database manually if needed, for example if `geo_secondary['auto_migrate'] = false`: ```shell gitlab-rake geo:db:migrate diff --git a/doc/administration/get_started.md b/doc/administration/get_started.md index 6fe66aa1642..c0d5a45d8d1 100644 --- a/doc/administration/get_started.md +++ b/doc/administration/get_started.md @@ -137,7 +137,7 @@ Keep your configuration files and backup archives in a separate location to ensu You can restore a backup only to **the exact same version and type** (Community Edition/Enterprise Edition) of GitLab on which it was created. - Review the [Omnibus backup and restore documentation](https://docs.gitlab.com/omnibus/settings/backups). -- Review the [Helm Chart backup and restore documentation](https://docs.gitlab.com/charts/backup-restore). +- Review the [Helm Chart backup and restore documentation](https://docs.gitlab.com/charts/backup-restore/). ### Back up GitLab SaaS @@ -177,7 +177,7 @@ The EC2 instance meets the requirements for an application data backup by taking In general, if you're running GitLab on a virtualized server, you can create VM snapshots of the entire GitLab server. It is common for a VM snapshot to require you to power down the server. -#### Option 2: GitLab Geo +#### Option 2: GitLab Geo **(PREMIUM SELF)** Geo provides local, read-only instances of your GitLab instances. @@ -191,7 +191,7 @@ Learn more about [replication limitations](../administration/geo/replication/dat GitLab provides support for self-managed GitLab through different channels. -- Priority support: Premium and Ultimate self-managed customers receive priority support with tiered response times. +- Priority support: [Premium and Ultimate](https://about.gitlab.com/pricing/) self-managed customers receive priority support with tiered response times. Learn more about [upgrading to priority support](https://about.gitlab.com/support/#upgrading-to-priority-support). - Live upgrade assistance: Get one-on-one expert guidance during a production upgrade. With your **priority support plan**, you're eligible for a live, scheduled screen-sharing session with a member of our support team. diff --git a/doc/administration/git_protocol.md b/doc/administration/git_protocol.md index e3e2db81fb0..c8c532e9a01 100644 --- a/doc/administration/git_protocol.md +++ b/doc/administration/git_protocol.md @@ -99,7 +99,7 @@ $ GIT_TRACE_PACKET=1 git -c protocol.version=2 ls-remote https://your-gitlab-ins Verify Git v2 is used by the client: ```shell -GIT_SSH_COMMAND="ssh -v" git -c protocol.version=2 ls-remote ssh://git@your-gitlab-instance.com/group/repo.git 2>&1 |grep GIT_PROTOCOL +GIT_SSH_COMMAND="ssh -v" git -c protocol.version=2 ls-remote ssh://git@your-gitlab-instance.com/group/repo.git 2>&1 | grep GIT_PROTOCOL ``` You should see that the `GIT_PROTOCOL` environment variable is sent: diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md index 5e8cbac42c1..d0841f4e607 100644 --- a/doc/administration/gitaly/configure_gitaly.md +++ b/doc/administration/gitaly/configure_gitaly.md @@ -41,9 +41,8 @@ However, Gitaly can be deployed to its own server, which can benefit GitLab inst multiple machines. NOTE: -When configured to run on their own servers, Gitaly servers -[must be upgraded](https://docs.gitlab.com/omnibus/update/#upgrading-gitaly-servers) before Gitaly -clients in your cluster. +When configured to run on their own servers, Gitaly servers must be +[upgraded](../../update/package/index.md) before Gitaly clients in your cluster. The process for setting up Gitaly on its own server is: @@ -340,6 +339,12 @@ disable enforcement. For more information, see the documentation on configuring 1. Run `sudo -u git /home/git/gitaly/gitaly-hooks check /home/git/gitaly/config.toml` to confirm that Gitaly can perform callbacks to the GitLab internal API. +WARNING: +If directly copying repository data from a GitLab server to Gitaly, ensure that the metadata file, +default path `/var/opt/gitlab/git-data/repositories/.gitaly-metadata`, is not included in the transfer. +Copying this file causes GitLab to use the [Rugged patches](index.md#direct-access-to-git-in-gitlab) for repositories hosted on the Gitaly server, +leading to `Error creating pipeline` and `Commit not found` errors, or stale data. + ### Configure Gitaly clients As the final step, you must update Gitaly clients to switch from using local Gitaly service to use diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index bca83e903ac..7a7aac884ed 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -87,11 +87,8 @@ Gitaly comes pre-configured with Omnibus GitLab, which is a configuration - Omnibus GitLab installations for up to 2000 users, see [specific Gitaly configuration instructions](../reference_architectures/2k_users.md#configure-gitaly). - Source installations or custom Gitaly installations, see [Configure Gitaly](configure_gitaly.md). -GitLab installations for more than 2000 users should use Gitaly Cluster. - -NOTE: -If not set in GitLab, feature flags are read as false from the console and Gitaly uses their -default value. The default value depends on the GitLab version. +GitLab installations for more than 2000 active users performing daily Git write operation may be +best suited by using Gitaly Cluster. ## Gitaly Cluster @@ -137,7 +134,7 @@ In this example: - The [replication factor](#replication-factor) is `3`. There are three copies maintained of each repository. -The availability objectives for Gitaly clusters are: +The availability objectives for Gitaly clusters assuming a single node failure are: - **Recovery Point Objective (RPO):** Less than 1 minute. @@ -155,6 +152,58 @@ The availability objectives for Gitaly clusters are: Faster outage detection, to improve this speed to less than 1 second, is tracked [in this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/2608). +WARNING: +If complete cluster failure occurs, disaster recovery plans should be executed. These can affect the +RPO and RTO discussed above. + +### Architecture and configuration recommendations + +The following table provides recommendations based on your +requirements. Users means concurrent users actively performing +simultaneous Git Operations. + +Gitaly services are present in every GitLab installation and always coordinates Git repository storage and +retrieval. Gitaly can be as simple as a single background service when operating on a single instance Omnibus +GitLab (All of GitLab on one machine). Gitaly can be separated into it's own instance and it can be configured in +a full cluster configuration depending on scaling and availability requirements. + +The GitLab Reference Architectures provide guidance for what Gitaly configuration is advisable at each of the scales. +The Gitaly configuration is noted by the architecture diagrams and the table of required resources. + +| User Scaling | Reference Architecture To Use | GitLab Instance Configuration | Gitaly Configuration | Git Repository Storage | Instances Dedicated to Gitaly Services | +| ------------------------------------------------------------ | ------------------------------------------------------------ | -------------------------------------------- | ------------------------------- | ---------------------------------- | -------------------------------------- | +| Up to 1000 Users | [1K](../reference_architectures/1k_users.md) | Single Instance for all of GitLab | Already Integrated as a Service | Local Disk | 0 | +| Up to 2999 Users | [2K](../reference_architectures/2k_users.md) | Horizontally Scaled GitLab Instance (Non-HA) | Single Gitaly Server | Local Disk of Gitaly Instance | 1 | +| 3000 Users and Over | [3K](../reference_architectures/1k_users.md) | Horizontally Scaled GitLab Instance with HA | Gitaly Cluster | Local Disk of Each Gitaly Instance | 8 | +| RTO/RPO Requirements for AZ Failure Tolerance Regardless of User Scale | [3K (with downscaling)](../reference_architectures/3k_users.md) | Custom (1) | Gitaly Cluster | Local Disk of Each Gitaly Instance | 8 | + +1. If you feel that you need AZ Failure Tolerance for user scaling lower than 3K, please contact Customer Success + to discuss your RTO and RPO needs and what options exist to meet those objectives. + +WARNING: +At present, some [known database inconsistency issues](#known-issues-impacting-gitaly-cluster) +exist in Gitaly Cluster. It is our recommendation that for now, you remain on your current service. +We will adjust the date for NFS support removal if this applies to you. + +### Known issues impacting Gitaly Cluster + +The following table outlines current known issues impacting the use of Gitaly Cluster. For +the most up to date status of these issues, please refer to the referenced issues / epics. + +| Issue | Summary | +| Gitaly Cluster + Geo can cause database inconsistencies | There are some conditions during Geo replication that can cause database inconsistencies with Gitaly Cluster. These have been identified and are being resolved by updating Gitaly Cluster to [identify repositories with a unique and persistent identifier](https://gitlab.com/gitlab-org/gitaly/-/issues/3485). | +| Database inconsistencies due to repository access outside of Gitaly Cluster's control | Operations that write to the repository storage which do not go through normal Gitaly Cluster methods can cause database inconsistencies. These can include (but are not limited to) snapshot restoration for cluster node disks, node upgrades which modify files under Git control, or any other disk operation that may touch repository storage external to GitLab. The Gitaly team is actively working to provide manual commands to [reconcile the Praefect database with the repository storage](https://gitlab.com/groups/gitlab-org/-/epics/6723). | + +### Snapshot backup and recovery limitations + +Gitaly Cluster does not support snapshot backups because these can cause issues where the +Praefect database becomes out of sync with the disk storage. Because of how Praefect rebuilds +the replication metadata of Gitaly disk information during a restore, we recommend using the +[official backup and restore Rake tasks](../../raketasks/backup_restore.md). + +To track progress on work on a solution for manually re-synchronizing the Praefect database +with disk storage, see [this epic](https://gitlab.com/groups/gitlab-org/-/epics/6575). + ### Virtual storage Virtual storage makes it viable to have a single repository storage in GitLab to simplify repository @@ -306,7 +355,10 @@ For configuration information, see [Configure replication factor](praefect.md#co For more information on configuring Gitaly Cluster, see [Configure Gitaly Cluster](praefect.md). -### Migrate to Gitaly Cluster +## Migrate to Gitaly Cluster + +We recommend you migrate to Gitaly Cluster if your +[requirements recommend](#architecture-and-configuration-recommendations) Gitaly Cluster. Whether migrating to Gitaly Cluster because of [NFS support deprecation](index.md#nfs-deprecation-notice) or to move from single Gitaly nodes, the basic process involves: @@ -318,6 +370,11 @@ or to move from single Gitaly nodes, the basic process involves: Gitaly Cluster, existing repositories stored outside Gitaly Cluster must be moved. There is no automatic migration but the moves can be scheduled with the GitLab API. +WARNING: +At present, some [known database inconsistency issues](#known-issues-impacting-gitaly-cluster) +exist in Gitaly Cluster. It is our recommendation that for now, you remain on your current service. +We will adjust the date for NFS support removal if this applies to you. + ## Monitor Gitaly and Gitaly Cluster You can use the available logs and [Prometheus metrics](../monitoring/prometheus/index.md) to @@ -449,6 +506,8 @@ To monitor [strong consistency](#strong-consistency), you can use the following - `gitaly_hook_transaction_voting_delay_seconds`, the client-side delay introduced by waiting for the transaction to be committed. +You can also monitor the [Praefect logs](../logs.md#praefect-logs). + ## Do not bypass Gitaly GitLab doesn't advise directly accessing Gitaly repositories stored on disk with a Git client, @@ -539,12 +598,6 @@ To see if GitLab can access the repository file system directly, we use the foll Direct Git access is enable by default in Omnibus GitLab because it fills in the correct repository paths in the GitLab configuration file `config/gitlab.yml`. This satisfies the UUID check. -WARNING: -If directly copying repository data from a GitLab server to Gitaly, ensure that the metadata file, -default path `/var/opt/gitlab/git-data/repositories/.gitaly-metadata`, is not included in the transfer. -Copying this file causes GitLab to use the Rugged patches for repositories hosted on the Gitaly server, -leading to `Error creating pipeline` and `Commit not found` errors, or stale data. - ### Transition to Gitaly Cluster For the sake of removing complexity, we must remove direct Git access in GitLab. However, we can't diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index 4af7f1a58a5..eb666f1caf4 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -20,10 +20,6 @@ Configure Gitaly Cluster using either: Smaller GitLab installations may need only [Gitaly itself](index.md). -NOTE: -Upgrade instructions for Omnibus GitLab installations -[are available](https://docs.gitlab.com/omnibus/update/#gitaly-cluster). - ## Requirements The minimum recommended configuration for a Gitaly Cluster requires: @@ -279,7 +275,7 @@ you need to prepare PostgreSQL server with [setup instruction](#manual-database- ```ruby pgbouncer['databases'] = { - # Other database configuation including gitlabhq_production + # Other database configuration including gitlabhq_production ... praefect_production: { @@ -353,6 +349,7 @@ If there are multiple Praefect nodes: To complete this section you need a [configured PostgreSQL server](#postgresql), including: +WARNING: Praefect should be run on a dedicated node. Do not run Praefect on the application server, or a Gitaly node. @@ -994,7 +991,7 @@ Particular attention should be shown to: 1. Check that the Praefect storage is configured to store new repositories: - 1. On the top bar, select **Menu >** **{admin}** **Admin**. + 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Repository**. 1. Expand the **Repository storage** section. @@ -1574,3 +1571,28 @@ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.t - Replace the placeholder `<virtual-storage>` with the virtual storage containing the Gitaly node storage to be checked. - Replace the placeholder `<up-to-date-storage>` with the Gitaly storage name containing up to date repositories. - Replace the placeholder `<outdated-storage>` with the Gitaly storage name containing outdated repositories. + +### Manually remove repositories + +> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3767) in GitLab 14.3. + +The `remove-repository` Praefect sub-command removes repositories from a Gitaly Cluster. It removes +all state associated with a given repository including: + +- On-disk repositories on all relevant Gitaly nodes. +- Any database state tracked by Praefect. + +```shell +sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml remove-repository -virtual-storage <virtual-storage> -repository <repository> +``` + +- `-virtual-storage` is the virtual storage the repository is located in. +- `-repository` is the repository's relative path in the storage. + +Sometimes parts of the repository continue to exist after running `remove-repository`. This can be caused +because of: + +- A deletion error. +- An in-flight RPC call targeting the repository. + +If this occurs, run `remove-repository` again. diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index 3dd700968f9..1b53a0994f5 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -23,7 +23,7 @@ See also [Gitaly timeout](../../user/admin_area/settings/gitaly_timeouts.md) set When using standalone Gitaly servers, you must make sure they are the same version as GitLab to ensure full compatibility: -1. On the top bar, select **Menu >** **{admin}** **Admin** on your GitLab instance. +1. On the top bar, select **Menu > Admin** on your GitLab instance. 1. On the left sidebar, select **Overview > Gitaly Servers**. 1. Confirm all Gitaly servers indicate that they are up to date. @@ -226,8 +226,8 @@ update the secrets file on the Gitaly server to match the Gitaly client, then ### Repository pushes fail with a `deny updating a hidden ref` error Due to [a change](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3426) -introduced in GitLab 13.12, Gitaly has read-only, internal GitLab references that users are not -permitted to update. If you attempt to update internal references with `git push --mirror`, Git +introduced in GitLab 13.12, Gitaly has read-only, internal GitLab references that users are not +permitted to update. If you attempt to update internal references with `git push --mirror`, Git returns the rejection error, `deny updating a hidden ref`. The following references are read-only: @@ -243,7 +243,7 @@ To mirror-push branches and tags only, and avoid attempting to mirror-push prote git push origin +refs/heads/*:refs/heads/* +refs/tags/*:refs/tags/* ``` -Any other namespaces that the admin wants to push can be included there as well via additional patterns. +Any other namespaces that the admin wants to push can be included there as well via additional patterns. ### Command line tools cannot connect to Gitaly diff --git a/doc/administration/housekeeping.md b/doc/administration/housekeeping.md index 8f5bf2ee013..4de48aa3f14 100644 --- a/doc/administration/housekeeping.md +++ b/doc/administration/housekeeping.md @@ -27,7 +27,7 @@ GitLab automatically runs `git gc` and `git repack` on repositories after Git pu You can change how often this happens or turn it off: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Repository**. 1. Expand **Repository maintenance**. 1. In the **Housekeeping** section, configure the [housekeeping options](#housekeeping-options). @@ -62,6 +62,12 @@ Housekeeping also [removes unreferenced LFS files](../raketasks/cleanup.md#remov from your project on the same schedule as the `git gc` operation, freeing up storage space for your project. +WARNING: +Running `git gc` or `git repack` commands manually in the +[repository folder](repository_storage_types.md#from-project-name-to-hashed-path) +is discouraged. If the created pack files get incorrect access rights (that is, owned by the wrong user) +browsing to the project page might result in `404` and `503` errors. + ## How housekeeping handles pool repositories Housekeeping for pool repositories is handled differently from standard repositories. It is @@ -76,7 +82,7 @@ This is the current call stack by which it is invoked: 1. `ObjectPoolService#fetch` 1. `Gitaly::FetchIntoObjectPoolRequest` -To manually invoke it from a Rails console if needed, you can call +To manually invoke it from a [Rails console](operations/rails_console.md) if needed, you can call `project.pool_repository.object_pool.fetch`. This is a potentially long-running task, though Gitaly times out in about 8 hours. diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index c5cabc5794a..5b72583bc95 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -76,7 +76,7 @@ and use [an application password](https://support.google.com/mail/answer/185833) If you want to use Office 365, and two-factor authentication is enabled, make sure you're using an -[app password](https://docs.microsoft.com/en-us/azure/active-directory/user-help/multi-factor-authentication-end-user-app-passwords) +[app password](https://support.microsoft.com/en-us/account-billing/manage-app-passwords-for-two-step-verification-d6dc8c6d-4bf7-4851-ad95-6d07799387e9) instead of the regular password for the mailbox. To set up a basic Postfix mail server with IMAP access on Ubuntu, follow the @@ -464,7 +464,7 @@ Example configurations for Microsoft Office 365 with IMAP enabled. NOTE: As of September 2020 sub-addressing support -[has been added to Office 365](https://office365.uservoice.com/forums/273493-office-365-admin/suggestions/18612754-support-for-dynamic-email-aliases-in-office-36). This feature is not +[has been added to Office 365](https://support.microsoft.com/en-us/office/uservoice-pages-430e1a78-e016-472a-a10f-dc2a3df3450a). This feature is not enabled by default, and must be enabled through PowerShell. This series of PowerShell commands enables [sub-addressing](#email-sub-addressing) @@ -638,7 +638,7 @@ incoming_email: #### Microsoft Graph -> Introduced in [GitLab 13.11](https://gitlab.com/gitlab-org/gitlab/-/issues/214900). +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214900) in GitLab 13.11. GitLab can read incoming email using the Microsoft Graph API instead of IMAP. Because [Microsoft is deprecating IMAP usage with Basic Authentication](https://techcommunity.microsoft.com/t5/exchange-team-blog/announcing-oauth-2-0-support-for-imap-and-smtp-auth-protocols-in/ba-p/1330432), the Microsoft Graph API will soon be required for new Microsoft Exchange Online diff --git a/doc/administration/index.md b/doc/administration/index.md index 46624ab39a3..9412994edb7 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -111,7 +111,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. ### GitLab platform integrations -- [Mattermost](https://docs.gitlab.com/omnibus/gitlab-mattermost/): Integrate with [Mattermost](https://mattermost.com), an open source, private cloud workplace for web messaging. +- [Mattermost](../integration/mattermost/index.md): Integrate with [Mattermost](https://mattermost.com), an open source, private cloud workplace for web messaging. - [PlantUML](integration/plantuml.md): Create diagrams in AsciiDoc and Markdown documents created in snippets, wikis, and repositories. - [Web terminals](integration/terminal.md): Provide terminal access to your applications deployed to Kubernetes from GitLab CI/CD [environments](../ci/environments/index.md#web-terminals). @@ -164,16 +164,16 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Limit repository size](../user/admin_area/settings/account_and_limit_settings.md): Set a hard limit for your repositories' size. - [Static objects external storage](static_objects_external_storage.md): Set external storage for static objects in a repository. -## Continuous Integration settings +## CI/CD settings -- [Enable/disable GitLab CI/CD](../ci/enable_or_disable_ci.md#make-gitlab-cicd-disabled-by-default-in-new-projects): Enable or disable GitLab CI/CD for your instance. +- [Enable or disable GitLab CI/CD](cicd.md#disable-gitlab-cicd-in-new-projects): Set new projects in your instance to have GitLab CI/CD enabled or disabled by default. - [GitLab CI/CD administration settings](../user/admin_area/settings/continuous_integration.md): Enable or disable Auto DevOps site-wide and define the artifacts' max size and expiration time. - [External Pipeline Validation](external_pipeline_validation.md): Enable, disable, and configure external pipeline validation. - [Job artifacts](job_artifacts.md): Enable, disable, and configure job artifacts (a set of files and directories which are outputted by a job when it completes successfully). - [Job logs](job_logs.md): Information about the job logs. - [Register runners](../ci/runners/runners_scope.md): Learn how to register and configure runners. - [Shared runners pipelines quota](../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota): Limit the usage of pipeline minutes for shared runners. -- [Enable/disable Auto DevOps](../topics/autodevops/index.md#enable-or-disable-auto-devops): Enable or disable Auto DevOps for your instance. +- [Enable or disable Auto DevOps](../topics/autodevops/index.md#enable-or-disable-auto-devops): Enable or disable Auto DevOps for your instance. ## Snippet settings diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index 1ea2ec4c904..24ffee088f3 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -78,6 +78,16 @@ This setting limits the request rate on the Packages API per user or IP. For mor - **Default rate limit**: Disabled by default. +### Git LFS + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68642) in GitLab 14.3. + +This setting limits the request rate on the [Git LFS](../topics/git/lfs/index.md) +requests per user. For more information, read +[GitLab Git Large File Storage (LFS) Administration](../administration/lfs/index.md). + +- **Default rate limit**: Disabled by default. + ### Import/Export > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35728) in GitLab 13.2. @@ -387,6 +397,31 @@ To set this limit for a self-managed installation, run the following in the Plan.default.actual_limits.update!(ci_pipeline_schedules: 100) ``` +### Limit the number of pipelines created by a pipeline schedule per day + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/323066) in GitLab 14.0. + +You can limit the number of pipelines that pipeline schedules can trigger per day. + +Schedules that try to run pipelines more frequently than the limit are slowed to a maximum frequency. +The frequency is calculated by dividing 1440 (the number minutes in a day) by the +limit value. For example, for a maximum frequency of: + +- Once per minute, the limit must be `1440`. +- Once per 10 minutes, the limit must be `144`. +- Once per 60 minutes, the limit must be `24` + +There is no limit for self-managed instances by default. + +To set this limit to `1440` on a self-managed installation, run the following in the +[GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): + +```ruby +Plan.default.actual_limits.update!(ci_daily_pipeline_schedule_triggers: 1440) +``` + +This limit is [enabled on GitLab.com](../user/gitlab_com/index.md#gitlab-cicd). + ### Number of instance level variables > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216097) in GitLab 13.1. @@ -513,19 +548,61 @@ Update `ci_jobs_trace_size_limit` with the new value in megabytes: Plan.default.actual_limits.update!(ci_jobs_trace_size_limit: 125) ``` +### Maximum number of active DAST profile schedules per project + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68551) in GitLab 14.3. + +Limit the number of active DAST profile schedules per project. A DAST profile schedule can be active or inactive. + +You can change the limit in the [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session). +Update `dast_profile_schedules` with the new value: + +```ruby +Plan.default.actual_limits.update!(dast_profile_schedules: 50) +``` + +### Maximum size and depth of CI/CD configuration YAML files + +The default maximum size of a CI/CD configuration YAML file is 1 megabyte and the default depth is 100. + +You can change these limits in the [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session). +Update `max_yaml_size_bytes` with the new value in megabytes: + +```ruby +ApplicationSetting.update!(max_yaml_size_bytes: 2.megabytes) +``` + +Update `max_yaml_depth` with the new value in megabytes: + +```ruby +ApplicationSetting.update!(max_yaml_depth: 125) +``` + +To disable this limitation entirely, disable the feature flag in the console: + +```ruby +Feature.disable(:ci_yaml_limit_size) +``` + ## Instance monitoring and metrics -### Incident Management inbound alert limits +### Limit inbound incident management alerts > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17859) in GitLab 12.5. -Limiting inbound alerts for an incident reduces the number of alerts (issues) -that can be created within a period of time, which can help prevent overloading -your incident responders with duplicate issues. You can reduce the volume of -alerts in the following ways: +You can limit the number of inbound alerts for [incidents](../operations/incident_management/incidents.md) +that can be created in a period of time. The inbound [incident management](../operations/incident_management/index.md) +alert limit can help prevent overloading your incident responders by reducing the +number of alerts or duplicate issues. + +To set inbound incident management alert limits: -- Max requests per period per project, 3600 seconds by default. -- Rate limit period in seconds, 3600 seconds by default. +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Settings > Network**. +1. Expand General **Incident Management Limits**. +1. Select the **Enable Incident Management inbound alert limit** checkbox. +1. Optional. Input a custom value for **Maximum requests per project per rate limit period**. Default is 3600. +1. Optional. Input a custom value for **Rate limit period**. Default is 3600 seconds. ### Prometheus Alert JSON payloads @@ -577,9 +654,9 @@ panel_groups: See [Environment Dashboard](../ci/environments/environments_dashboard.md#adding-a-project-to-the-dashboard) for the maximum number of displayed projects. -## Environment data on Deploy Boards +## Environment data on deploy boards -[Deploy Boards](../user/project/deploy_boards.md) load information from Kubernetes about +[Deploy boards](../user/project/deploy_boards.md) load information from Kubernetes about Pods and Deployments. However, data over 10 MB for a certain environment read from Kubernetes won't be shown. @@ -645,7 +722,7 @@ indexed, which have a separate limit. For more information, read - For self-managed installations, the field length is unlimited by default. You can configure this limit for self-managed installations when you -[enable Elasticsearch](../integration/elasticsearch.md#enabling-advanced-search). +[enable Elasticsearch](../integration/elasticsearch.md#enable-advanced-search). Set the limit to `0` to disable it. ## Wiki limits @@ -733,7 +810,7 @@ Set the limit to `0` to allow any file size. When asking for versions of a given NuGet package name, the GitLab Package Registry returns a maximum of 300 versions. -## Branch retargeting on merge **(FREE SELF)** +## Branch retargeting on merge > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/320902) in GitLab 13.9. diff --git a/doc/administration/instance_review.md b/doc/administration/instance_review.md index f2ba3a5a562..b166bb32aa1 100644 --- a/doc/administration/instance_review.md +++ b/doc/administration/instance_review.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Instance Review **(FREE SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/6995) in [GitLab Free](https://about.gitlab.com/pricing/) 11.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/6995) in GitLab 11.3. If you run a medium-sized self-managed instance (50+ users) of a free version of GitLab, [either Community Edition or unlicensed Enterprise Edition](https://about.gitlab.com/install/ce-or-ee/), diff --git a/doc/administration/integration/kroki.md b/doc/administration/integration/kroki.md index 729894052b2..008d33c6c94 100644 --- a/doc/administration/integration/kroki.md +++ b/doc/administration/integration/kroki.md @@ -56,7 +56,7 @@ read the [Kroki installation](https://docs.kroki.io/kroki/setup/install/#_images You need to enable Kroki integration from Settings under Admin Area. To do that, log in with an administrator account and follow these steps: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. Go to **Settings > General**. 1. Expand the **Kroki** section. 1. Select **Enable Kroki** checkbox. diff --git a/doc/administration/integration/mailgun.md b/doc/administration/integration/mailgun.md new file mode 100644 index 00000000000..5a56aed4427 --- /dev/null +++ b/doc/administration/integration/mailgun.md @@ -0,0 +1,41 @@ +--- +stage: Growth +group: Expansion +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference, howto +--- + +# Mailgun and GitLab **(FREE SELF)** + +When you use Mailgun to send emails for your GitLab instance and [Mailgun](https://www.mailgun.com/) +integration is enabled and configured in GitLab, you can receive their webhook for +permanent invite email failures. To set up the integration, you must: + +1. [Configure your Mailgun domain](#configure-your-mailgun-domain). +1. [Enable Mailgun integration](#enable-mailgun-integration). + +After completing the integration, Mailgun `permanent_failure` webhooks are sent to your GitLab instance. + +## Configure your Mailgun domain + +Before you can enable Mailgun in GitLab, set up your own Mailgun permanent failure endpoint to receive the webhooks. + +Using the [Mailgun webhook guide](https://www.mailgun.com/blog/a-guide-to-using-mailguns-webhooks/): + +1. Add a webhook with the **Event type** set to **Permanent Failure**. +1. Fill in the URL of your instance and include the `/-/members/mailgun/permanent_failures` path. + - Example: `https://myinstance.gitlab.com/-/members/mailgun/permanent_failures` + +## Enable Mailgun integration + +After configuring your Mailgun domain for the permanent failures endpoint, +you're ready to enable the Mailgun integration: + +1. Sign in to GitLab as an [Administrator](../../user/permissions.md) user. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, go to **Settings > General** and expand the **Mailgun** section. +1. Select the **Enable Mailgun** check box. +1. Enter the Mailgun HTTP webhook signing key as described in + [the Mailgun documentation](https://documentation.mailgun.com/en/latest/user_manual.html#webhooks) and + shown in the [API security](https://app.mailgun.com/app/account/security/api_keys) section for your Mailgun account. +1. Select **Save changes**. diff --git a/doc/administration/integration/plantuml.md b/doc/administration/integration/plantuml.md index 9f83ba8e972..94fef89d966 100644 --- a/doc/administration/integration/plantuml.md +++ b/doc/administration/integration/plantuml.md @@ -206,8 +206,8 @@ stop; After configuring your local PlantUML server, you're ready to enable the PlantUML integration: 1. Sign in to GitLab as an [Administrator](../../user/permissions.md) user. -1. On the top bar, select **Menu >** **{admin}** **Admin**. -1. In the left sidebar, go to **Settings > General** and expand the **PlantUML** section. +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, go to **Settings > General** and expand the **PlantUML** section. 1. Select the **Enable PlantUML** checkbox. 1. Set the PlantUML instance as `https://gitlab.example.com/-/plantuml/`, and click **Save changes**. diff --git a/doc/administration/integration/terminal.md b/doc/administration/integration/terminal.md index 1be234c2771..882580b35b6 100644 --- a/doc/administration/integration/terminal.md +++ b/doc/administration/integration/terminal.md @@ -100,7 +100,7 @@ they receive a `Connection failed` message. By default, terminal sessions do not expire. To limit the terminal session lifetime in your GitLab instance: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. Select [**Settings > Web terminal**](../../user/admin_area/settings/index.md#general). 1. Set a `max session time`. diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 3b1d253b4b6..b4c16e007cc 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -435,6 +435,27 @@ Ci::JobArtifact.where(project: project).order(size: :desc).limit(50).map { |a| p You can change the number of job artifacts listed by modifying `.limit(50)` to the number you want. +#### List artifacts in a single project + +List the artifacts for a single project, sorted by artifact size. The output includes the: + +- ID of the job that created the artifact +- artifact size +- artifact file type +- artifact creation date +- on-disk location of the artifact + +```ruby +p = Project.find_by_id(:project ID) +arts = Ci::JobArtifact.where(project: p) + +list = arts.order('sort DESC').limit(50).each do |art| + puts "Job ID: #{art.job_id} - Size: #{art.size}b - Type: #{art.file_type} - Created: #{art.created_at} - File loc: #{art.file}" +end +``` + +To change the number of projects listed, change the number in `limit(50)`. + #### Delete job artifacts from jobs completed before a specific date WARNING: diff --git a/doc/administration/job_logs.md b/doc/administration/job_logs.md index a4fdf734c2e..64d9248cb16 100644 --- a/doc/administration/job_logs.md +++ b/doc/administration/job_logs.md @@ -138,7 +138,7 @@ For more information, see [delete references to missing artifacts](raketasks/che > - Enabled on GitLab.com. > - [Recommended for production use](https://gitlab.com/groups/gitlab-org/-/epics/4275) in GitLab 13.6. > - [Recommended for production use with AWS S3](https://gitlab.com/gitlab-org/gitlab/-/issues/273498) in GitLab 13.7. -> - To use in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-incremental-logging). **(FREE SELF)** +> - To use in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-incremental-logging). By default job logs are sent from the GitLab Runner in chunks and cached temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by Omnibus GitLab. After the job completes, @@ -183,7 +183,7 @@ Here is the detailed data flow: to disk, and there is no protection against misconfiguration. - There is [an epic tracking other potential limitations and improvements](https://gitlab.com/groups/gitlab-org/-/epics/3791). -### Enable or disable incremental logging **(FREE SELF)** +### Enable or disable incremental logging Incremental logging is under development, but [ready for production use as of GitLab 13.6](https://gitlab.com/groups/gitlab-org/-/epics/4275). It is deployed behind a feature flag that is **disabled by default**. diff --git a/doc/administration/logs.md b/doc/administration/logs.md index 883f1db8e09..990287e3907 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -50,6 +50,7 @@ except those captured by `runit`. | [Mailroom](#mail_room_jsonlog-default) | **{check-circle}** Yes | **{check-circle}** Yes | | [NGINX](#nginx-logs) | **{check-circle}** Yes | **{check-circle}** Yes | | [PostgreSQL Logs](#postgresql-logs) | **{dotted-circle}** No | **{check-circle}** Yes | +| [Praefect Logs](#praefect-logs) | **{dotted-circle}** Yes| **{check-circle}** Yes | | [Prometheus Logs](#prometheus-logs) | **{dotted-circle}** No | **{check-circle}** Yes | | [Puma](#puma-logs) | **{check-circle}** Yes | **{check-circle}** Yes | | [Redis Logs](#redis-logs) | **{dotted-circle}** No | **{check-circle}** Yes | @@ -63,9 +64,6 @@ Depending on your installation method, this file is located at: - Omnibus GitLab: `/var/log/gitlab/gitlab-rails/production_json.log` - Installations from source: `/home/git/gitlab/log/production_json.log` -When GitLab is running in an environment other than production, -the corresponding log file is shown here. - It contains a structured log for Rails controller requests received from GitLab, thanks to [Lograge](https://github.com/roidrage/lograge/). Requests from the API are logged to a separate file in `api_json.log`. @@ -216,9 +214,6 @@ Depending on your installation method, this file is located at: - Omnibus GitLab: `/var/log/gitlab/gitlab-rails/production.log` - Installations from source: `/home/git/gitlab/log/production.log` -When GitLab is running in an environment other than production, -the corresponding log file is shown here. - It contains information about all performed requests. You can see the URL and type of request, IP address, and what parts of code were involved to service this particular request. Also, you can see all SQL @@ -556,10 +551,9 @@ access to Git repositories. ### For GitLab versions 12.10 and up -For GitLab version 12.10 and later, there are two `gitlab-shell.log` files. Information containing `git-{upload-pack,receive-pack}` requests is at `/var/log/gitlab/gitlab-shell/gitlab-shell.log`. Information about hooks to -GitLab Shell from Gitaly is at `/var/log/gitlab/gitaly/gitlab-shell.log`. +GitLab Shell from Gitaly is at `/var/log/gitlab/gitaly/current`. Example log entries for `/var/log/gitlab/gitlab-shell/gitlab-shell.log`: @@ -585,7 +579,7 @@ Example log entries for `/var/log/gitlab/gitlab-shell/gitlab-shell.log`: } ``` -Example log entries for `/var/log/gitlab/gitaly/gitlab-shell.log`: +Example log entries for `/var/log/gitlab/gitaly/current`: ```json { @@ -668,6 +662,12 @@ This file is at `/var/log/gitlab/gitaly/gitaly_ruby_json.log` and is produced by [`gitaly-ruby`](gitaly/reference.md#gitaly-ruby). It contains an access log of gRPC calls made by Gitaly to `gitaly-ruby`. +### `gitaly_hooks.log` + +This file is at `/var/log/gitlab/gitaly/gitaly_hooks.log` and is +produced by `gitaly-hooks` command. It also contains records about +failures received during processing of the responses from GitLab API. + ## Puma Logs ### `puma_stdout.log` @@ -1063,6 +1063,12 @@ For Omnibus GitLab installations, GitLab Exporter logs are in `/var/log/gitlab/g For Omnibus GitLab installations, GitLab Kubernetes Agent Server logs are in `/var/log/gitlab/gitlab-kas/`. +## Praefect Logs + +For Omnibus GitLab installations, Praefect logs are in `/var/log/gitlab/praefect/`. + +GitLab also tracks [Prometheus metrics for Praefect](gitaly/#monitor-gitaly-cluster). + ## Performance bar stats > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/48149) in GitLab 13.7. diff --git a/doc/administration/maintenance_mode/index.md b/doc/administration/maintenance_mode/index.md index 7664c7c1751..39ee357cc2f 100644 --- a/doc/administration/maintenance_mode/index.md +++ b/doc/administration/maintenance_mode/index.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GitLab Maintenance Mode **(PREMIUM SELF)** -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2149) in GitLab Premium 13.9. +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2149) in GitLab 13.9. Maintenance Mode allows administrators to reduce write operations to a minimum while maintenance tasks are performed. The main goal is to block all external actions that change the internal state, including the PostgreSQL database, but especially files, Git repositories, Container repositories, and so on. @@ -21,7 +21,7 @@ Maintenance Mode allows most external actions that do not change internal state. There are three ways to enable Maintenance Mode as an administrator: - **Web UI**: - 1. On the top bar, select **Menu >** **{admin}** **Admin**. + 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > General**. 1. Expand **Maintenance Mode**, and toggle **Enable Maintenance Mode**. You can optionally add a message for the banner as well. @@ -45,7 +45,7 @@ There are three ways to enable Maintenance Mode as an administrator: There are three ways to disable Maintenance Mode: - **Web UI**: - 1. On the top bar, select **Menu >** **{admin}** **Admin**. + 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > General**. 1. Expand **Maintenance Mode**, and toggle **Enable Maintenance Mode**. You can optionally add a message for the banner as well. @@ -171,7 +171,7 @@ it is recommended that you disable all cron jobs except for those related to Geo To monitor queues and disable jobs: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Jobs**. ### Incident management diff --git a/doc/administration/merge_request_diffs.md b/doc/administration/merge_request_diffs.md index 1133bff204c..d6b9fa2b8d3 100644 --- a/doc/administration/merge_request_diffs.md +++ b/doc/administration/merge_request_diffs.md @@ -265,3 +265,9 @@ by assigning different processes to different parts of the table. The `BATCH` and `UPDATE_DELAY` parameters allow the speed of the migration to be traded off against concurrent access to the table. The `ANSI` parameter should be set to false if your terminal does not support ANSI escape codes. + +By default, `sudo` does not preserve existing environment variables. You should append them, rather than prefix them. + +```shell +sudo gitlab-rake gitlab:external_diffs:force_object_storage START_ID=59946109 END_ID=59946109 UPDATE_DELAY=5 +``` diff --git a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md index f8764468256..90d0b65dbe5 100644 --- a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md +++ b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md @@ -14,7 +14,7 @@ GitLab provides administrators insights into the health of their GitLab instance To provide a native experience (similar interacting with an application deployed using GitLab), a project called **Monitoring** is created: -- With [internal visibility](../../../public_access/public_access.md#internal-projects). +- With [internal visibility](../../../public_access/public_access.md#internal-projects-and-groups). - Under a group called **GitLab Instance**. The project is created specifically for visualizing and configuring the monitoring of your GitLab @@ -34,7 +34,7 @@ This project can be used to: ## Create the self monitoring project -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Metrics and profiling** and expand **Self monitoring**. 1. Toggle **Self monitoring** on. 1. After your GitLab instance creates the project, GitLab displays a link to the @@ -47,7 +47,7 @@ WARNING: Deleting the self monitoring project removes any changes made to the project. If you create the project again, it's created in its default state. -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, go to **Settings > Metrics and profiling** and expand **Self monitoring**. 1. Toggle **Self monitoring** off. 1. In the confirmation dialog that opens, click **Delete self monitoring project**. diff --git a/doc/administration/monitoring/performance/gitlab_configuration.md b/doc/administration/monitoring/performance/gitlab_configuration.md index e8abe2708c7..f316a75a868 100644 --- a/doc/administration/monitoring/performance/gitlab_configuration.md +++ b/doc/administration/monitoring/performance/gitlab_configuration.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w GitLab Performance Monitoring is disabled by default. To enable it and change any of its settings: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Metrics and profiling** (`/admin/application_settings/metrics_and_profiling`). 1. Add the necessary configuration changes. diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md index 0f40fd3c0e7..c37a264938e 100644 --- a/doc/administration/monitoring/performance/grafana_configuration.md +++ b/doc/administration/monitoring/performance/grafana_configuration.md @@ -62,7 +62,7 @@ repository. After setting up Grafana, you can enable a link to access it easily from the GitLab sidebar: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Metrics and profiling** and expand **Metrics - Grafana**. 1. Select the **Add a link to Grafana** checkbox. @@ -79,7 +79,7 @@ GitLab displays your link in the **Menu > Admin > Monitoring > Metrics Dashboard > [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5822) in GitLab 13.10. When setting up Grafana through the process above, no scope shows in the screen at -**Menu >** **{admin}** **Admin > Applications > GitLab Grafana**. However, the `read_user` scope is +**Menu > Admin > Applications > GitLab Grafana**. However, the `read_user` scope is required and is provided to the application automatically. Setting any scope other than `read_user` without also including `read_user` leads to this error when you try to log in using GitLab as the OAuth provider: diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index 6d9418133d8..ef4db93d5fc 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -72,7 +72,7 @@ From left to right, the performance bar displays: NOTE: Not all indicators are available in all environments. For instance, the memory view -requires running Ruby with [specific patches](https://gitlab.com/gitlab-org/gitlab-build-images/-/blob/master/patches/ruby/2.7.2/thread-memory-allocations-2.7.patch) +requires running Ruby with [specific patches](https://gitlab.com/gitlab-org/gitlab-build-images/-/blob/master/patches/ruby/2.7.4/thread-memory-allocations-2.7.patch) applied. When running GitLab locally using [GDK](../../../development/contributing/index.md#gitlab-development-kit), this is typically not the case and the memory view cannot be used. @@ -104,7 +104,7 @@ The performance bar is disabled by default for non-administrators. To enable it for a given group: 1. Sign in as a user with Administrator [role](../../../user/permissions.md). -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Metrics and profiling** (`admin/application_settings/metrics_and_profiling`), and expand **Profiling - Performance bar**. diff --git a/doc/administration/monitoring/prometheus/gitlab_exporter.md b/doc/administration/monitoring/prometheus/gitlab_exporter.md index 4ba4cad9143..d9852524aec 100644 --- a/doc/administration/monitoring/prometheus/gitlab_exporter.md +++ b/doc/administration/monitoring/prometheus/gitlab_exporter.md @@ -6,8 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GitLab exporter **(FREE SELF)** ->- Available since [Omnibus GitLab 8.17](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/1132). ->- Renamed from `GitLab monitor exporter` to `GitLab exporter` in [GitLab 12.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/16511). +> Renamed from `GitLab monitor exporter` to `GitLab exporter` in [GitLab 12.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/16511). The [GitLab exporter](https://gitlab.com/gitlab-org/gitlab-exporter) enables you to measure various GitLab metrics pulled from Redis and the database in Omnibus GitLab @@ -33,8 +32,8 @@ the GitLab exporter exposed at `localhost:9168`. ## Use a different Rack server ->- Introduced in [Omnibus GitLab 13.8](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4896). ->- WEBrick is now the default Rack server instead of Puma. +> - Introduced in [Omnibus GitLab 13.8](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4896). +> - WEBrick is now the default Rack server instead of Puma. By default, the GitLab exporter runs on [WEBrick](https://github.com/ruby/webrick), a single-threaded Ruby web server. You can choose a different Rack server that better matches your performance needs. diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index 459eb259498..c36d2b0f7a4 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w To enable the GitLab Prometheus metrics: 1. Log in to GitLab as a user with Administrator [role](../../../user/permissions.md). -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Metrics and profiling**. 1. Find the **Metrics - Prometheus** section, and select **Add link to Prometheus**. 1. [Restart GitLab](../../restart_gitlab.md#omnibus-gitlab-restart) for the changes to take effect. @@ -252,12 +252,26 @@ configuration option in `gitlab.yml`. These metrics are served from the | `geo_group_wiki_repositories_synced` | Gauge | 13.10 | Number of syncable group wikis synced on secondary | `url` | | `geo_group_wiki_repositories_failed` | Gauge | 13.10 | Number of syncable group wikis failed on secondary | `url` | | `geo_group_wiki_repositories_registry` | Gauge | 13.10 | Number of syncable group wikis in the registry | `url` | +| `geo_pages_deployments` | Gauge | 14.3 | Number of pages deployments on primary | `url` | +| `geo_pages_deployments_checksum_total` | Gauge | 14.3 | Number of pages deployments tried to checksum on primary | `url` | +| `geo_pages_deployments_checksummed` | Gauge | 14.3 | Number of pages deployments successfully checksummed on primary | `url` | +| `geo_pages_deployments_checksum_failed` | Gauge | 14.3 | Number of pages deployments failed to calculate the checksum on primary | `url` | +| `geo_pages_deployments_synced` | Gauge | 14.3 | Number of syncable pages deployments synced on secondary | `url` | +| `geo_pages_deployments_failed` | Gauge | 14.3 | Number of syncable pages deployments failed to sync on secondary | `url` | +| `geo_pages_deployments_registry` | Gauge | 14.3 | Number of pages deployments in the registry | `url` | +| `geo_pages_deployments_verification_total` | Gauge | 14.3 | Number of pages deployments verifications tried on secondary | `url` | +| `geo_pages_deployments_verified` | Gauge | 14.3 | Number of pages deployments verified on secondary | `url` | +| `geo_pages_deployments_verification_failed` | Gauge | 14.3 | Number of pages deployments verifications failed on secondary | `url` | | `limited_capacity_worker_running_jobs` | Gauge | 13.5 | Number of running jobs | `worker` | | `limited_capacity_worker_max_running_jobs` | Gauge | 13.5 | Maximum number of running jobs | `worker` | | `limited_capacity_worker_remaining_work_count` | Gauge | 13.5 | Number of jobs waiting to be enqueued | `worker` | | `destroyed_job_artifacts_count_total` | Counter | 13.6 | Number of destroyed expired job artifacts | | | `destroyed_pipeline_artifacts_count_total` | Counter | 13.8 | Number of destroyed expired pipeline artifacts | | | `gitlab_optimistic_locking_retries` | Histogram | 13.10 | Number of retry attempts to execute optimistic retry lock | | +| `geo_uploads` | Gauge | 14.1 | Number of uploads on primary | `url` | +| `geo_uploads_synced` | Gauge | 14.1 | Number of uploads synced on secondary | `url` | +| `geo_uploads_failed` | Gauge | 14.1 | Number of syncable uploads failed to sync on secondary | `url` | +| `geo_uploads_registry` | Gauge | 14.1 | Number of uploads in the registry | `url` | ## Database load balancing metrics **(PREMIUM SELF)** diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index dd81f71d418..e04aad9c6b8 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -55,8 +55,7 @@ To disable Prometheus and all of its exporters, as well as any added in the futu ### Changing the port and address Prometheus listens on WARNING: -The following change was added in [Omnibus GitLab 8.17](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/1261). Although possible, -it's not recommended to change the port Prometheus listens +Although possible, it's not recommended to change the port Prometheus listens on, as this might affect or conflict with other services running on the GitLab server. Proceed at your own risk. @@ -330,8 +329,6 @@ To add a Prometheus dashboard for a single server GitLab setup: ## GitLab metrics -> Introduced in GitLab 9.3. - GitLab monitors its own internal service metrics, and makes them available at the `/-/metrics` endpoint. Unlike other exporters, this endpoint requires authentication as it's available on the same URL and port as user traffic. Read more about the [GitLab Metrics](gitlab_metrics.md). @@ -380,9 +377,6 @@ The GitLab exporter allows you to measure various GitLab metrics, pulled from Re ## Configuring Prometheus to monitor Kubernetes -> - Introduced in GitLab 9.0. -> - Pod monitoring introduced in GitLab 9.4. - If your GitLab server is running within Kubernetes, Prometheus collects metrics from the Nodes and [annotated Pods](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config) in the cluster, including performance data on each container. This is particularly helpful if your CI/CD environments run in the same cluster, as you can use the [Prometheus project integration](../../../user/project/integrations/prometheus.md) to monitor them. To disable the monitoring of Kubernetes: diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index b8a6f2acc56..6c77576cb27 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -14,7 +14,7 @@ typically much more performant, reliable, and scalable. ## Options -GitLab has been tested on a number of object storage providers: +GitLab has been tested by vendors and customers on a number of object storage providers: - [Amazon S3](https://aws.amazon.com/s3/) - [Google Cloud Storage](https://cloud.google.com/storage) @@ -22,7 +22,7 @@ GitLab has been tested on a number of object storage providers: - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) -- On-premises hardware and appliances from various storage vendors. +- On-premises hardware and appliances from various storage vendors, whose list is not officially established. - MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. ### Known compatibility issues @@ -33,6 +33,10 @@ GitLab has been tested on a number of object storage providers: - Ceph S3 prior to [Kraken 11.0.2](https://ceph.com/releases/kraken-11-0-2-released/) does not support the [Upload Copy Part API](https://gitlab.com/gitlab-org/gitlab/-/issues/300604). You may need to [disable multi-threaded copying](#multi-threaded-copying). +- Amazon S3 [Object Lock](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html) + is not supported. Follow [issue #335775](https://gitlab.com/gitlab-org/gitlab/-/issues/335775) + for progress on enabling this option. + ## Configuration guides There are two ways of specifying object storage configuration in GitLab: @@ -70,6 +74,7 @@ because it does not require a shared folder. Consolidated object storage configuration can't be used for backups or Mattermost. See the [full table for a complete list](#storage-specific-configuration). +However, backups can be configured with [server side encryption](../raketasks/backup_restore.md#s3-encrypted-buckets) separately. Enabling consolidated object storage enables object storage for all object types. If you want to use local storage for specific object types, you can @@ -683,8 +688,8 @@ configuration. #### Encrypted S3 buckets -> - Introduced in [GitLab 13.1](https://gitlab.com/gitlab-org/gitlab-workhorse/-/merge_requests/466) for instance profiles only and [S3 default encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/bucket-encryption.html). -> - Introduced in [GitLab 13.2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/34460) for static credentials when [consolidated object storage configuration](#consolidated-object-storage-configuration) and [S3 default encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/bucket-encryption.html) are used. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-workhorse/-/merge_requests/466) in GitLab 13.1 for instance profiles only and [S3 default encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/bucket-encryption.html). +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/34460) in GitLab 13.2 for static credentials when [consolidated object storage configuration](#consolidated-object-storage-configuration) and [S3 default encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/bucket-encryption.html) are used. When configured either with an instance profile or with the consolidated object configuration, GitLab Workhorse properly uploads files to S3 @@ -695,7 +700,7 @@ supported since this requires sending the encryption keys in every request](http ##### Server-side encryption headers -> Introduced in [GitLab 13.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38240). +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38240) in GitLab 13.3. Setting a default encryption on an S3 bucket is the easiest way to enable encryption, but you may want to [set a bucket policy to ensure diff --git a/doc/administration/operations/cleaning_up_redis_sessions.md b/doc/administration/operations/cleaning_up_redis_sessions.md index 194dd8f39e2..ed5014b65e1 100644 --- a/doc/administration/operations/cleaning_up_redis_sessions.md +++ b/doc/administration/operations/cleaning_up_redis_sessions.md @@ -1,64 +1,9 @@ --- -stage: Enablement -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'index.md' +remove_date: '2021-12-10' --- -# Cleaning up stale Redis sessions +This document was moved to [another location](index.md). -Since version 6.2, GitLab stores web user sessions as key-value pairs in Redis. -Prior to GitLab 7.3, user sessions did not automatically expire from Redis. If -you have been running a large GitLab server (thousands of users) since before -GitLab 7.3 we recommend cleaning up stale sessions to compact the Redis -database after you upgrade to GitLab 7.3. You can also perform a cleanup while -still running GitLab 7.2 or older, but in that case new stale sessions will -start building up again after you clean up. - -In GitLab versions prior to 7.3.0, the session keys in Redis are 16-byte -hexadecimal values such as '976aa289e2189b17d7ef525a6702ace9'. Starting with -GitLab 7.3.0, the keys are -prefixed with `session:gitlab:`, so they would look like -`session:gitlab:976aa289e2189b17d7ef525a6702ace9`. Below we describe how to -remove the keys in the old format. - -NOTE: -The instructions below must be modified in accordance with your -configuration settings if you have used the advanced Redis -settings outlined in -[Configuration Files Documentation](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/README.md). - -First we define a shell function with the proper Redis connection details. - -```shell -rcli() { - # This example works for Omnibus installations of GitLab 7.3 or newer. For an - # installation from source you will have to change the socket path and the - # path to redis-cli. - sudo /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket "$@" -} - -# test the new shell function; the response should be PONG -rcli ping -``` - -Now we do a search to see if there are any session keys in the old format for -us to clean up. - -```shell -# returns the number of old-format session keys in Redis -rcli keys '*' | grep '^[a-f0-9]\{32\}$' | wc -l -``` - -If the number is larger than zero, you can proceed to expire the keys from -Redis. If the number is zero there is nothing to clean up. - -```shell -# Tell Redis to expire each matched key after 600 seconds. -rcli keys '*' | grep '^[a-f0-9]\{32\}$' | awk '{ print "expire", $0, 600 }' | rcli -# This will print '(integer) 1' for each key that gets expired. -``` - -Over the next 15 minutes (10 minutes expiry time plus 5 minutes Redis -background save interval) your Redis database will be compacted. If you are -still using GitLab 7.2, users who are not clicking around in GitLab during the -10 minute expiry window will be signed out of GitLab. +<!-- This redirect file can be deleted after 2021-12-10. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md index 2cc4e3a4551..31cbd6c457b 100644 --- a/doc/administration/operations/extra_sidekiq_processes.md +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -90,7 +90,7 @@ To start multiple processes: To view the Sidekiq processes in GitLab: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Jobs**. ## Negate settings diff --git a/doc/administration/operations/extra_sidekiq_routing.md b/doc/administration/operations/extra_sidekiq_routing.md index 6938f8a7012..bb8eb184302 100644 --- a/doc/administration/operations/extra_sidekiq_routing.md +++ b/doc/administration/operations/extra_sidekiq_routing.md @@ -40,6 +40,8 @@ In `/etc/gitlab/gitlab.rb`: ```ruby sidekiq['routing_rules'] = [ + # Do not re-route workers that require their own queue + ['tags=needs_own_queue', nil], # Route all non-CPU-bound workers that are high urgency to `high-urgency` queue ['resource_boundary!=cpu&urgency=high', 'high-urgency'], # Route all database, gitaly and global search workers that are throttled to `throttled` queue @@ -164,3 +166,32 @@ with the migration to avoid losing jobs entirely, especially in a system with long queues of jobs. The migration can be done by following the migration steps mentioned in [Sidekiq job migration](../../raketasks/sidekiq_job_migration.md) + +### Workers that cannot be migrated + +Some workers cannot share a queue with other workers - typically because +they check the size of their own queue - and so must be excluded from +this process. We recommend excluding these from any further worker +routing by adding a rule to keep them in their own queue, for example: + +```ruby +sidekiq['routing_rules'] = [ + ['tags=needs_own_queue', nil], + # ... +] +``` + +These queues will also need to be included in at least one [Sidekiq +queue group](extra_sidekiq_processes.md#start-multiple-processes). + +The following table shows the workers that should have their own queue: + +| Worker name | Queue name | GitLab issue | +| --- | --- | --- | +| `EmailReceiverWorker` | `email_receiver` | [`gitlab-com/gl-infra/scalability#1263`](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1263) | +| `ServiceDeskEmailReceiverWorker` | `service_desk_email_receiver` | [`gitlab-com/gl-infra/scalability#1263`](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1263) | +| `ProjectImportScheduleWorker` | `project_import_schedule` | [`gitlab-org/gitlab#340630`](https://gitlab.com/gitlab-org/gitlab/-/issues/340630) | +| `HashedStorage::MigratorWorker` | `hashed_storage:hashed_storage_migrator` | [`gitlab-org/gitlab#340629`](https://gitlab.com/gitlab-org/gitlab/-/issues/340629) | +| `HashedStorage::ProjectMigrateWorker` | `hashed_storage:hashed_storage_project_migrate` | [`gitlab-org/gitlab#340629`](https://gitlab.com/gitlab-org/gitlab/-/issues/340629) | +| `HashedStorage::ProjectRollbackWorker` | `hashed_storage:hashed_storage_project_rollback` | [`gitlab-org/gitlab#340629`](https://gitlab.com/gitlab-org/gitlab/-/issues/340629) | +| `HashedStorage::RollbackerWorker` | `hashed_storage:hashed_storage_rollbacker` | [`gitlab-org/gitlab#340629`](https://gitlab.com/gitlab-org/gitlab/-/issues/340629) | diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index 5c1271486c0..e30ad4d8248 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -106,7 +106,7 @@ users as long as a large file exists. To disable any more writes to the `authorized_keys` file: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Network**. 1. Expand **Performance optimization**. 1. Clear the **Write to "authorized_keys" file** checkbox. diff --git a/doc/administration/operations/index.md b/doc/administration/operations/index.md index 4b16c3b3a7e..7ccfa2739bb 100644 --- a/doc/administration/operations/index.md +++ b/doc/administration/operations/index.md @@ -8,11 +8,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w Keep your GitLab instance up and running smoothly. -- [Clean up Redis sessions](cleaning_up_redis_sessions.md): Prior to GitLab 7.3, - user sessions did not automatically expire from Redis. If - you have been running a large GitLab server (thousands of users) since before - GitLab 7.3 we recommend cleaning up stale sessions to compact the Redis - database after you upgrade to GitLab 7.3. - [Rake tasks](../../raketasks/index.md): Tasks for common administration and operational processes such as [cleaning up unneeded items from GitLab instance](../../raketasks/cleanup.md), integrity checks, and more. diff --git a/doc/administration/operations/moving_repositories.md b/doc/administration/operations/moving_repositories.md index 765cf64e735..61a9ec8e7d4 100644 --- a/doc/administration/operations/moving_repositories.md +++ b/doc/administration/operations/moving_repositories.md @@ -39,6 +39,11 @@ WARNING: To move repositories into a [Gitaly Cluster](../gitaly/index.md#gitaly-cluster) in GitLab versions 13.12 to 14.1, you must [enable the `gitaly_replicate_repository_direct_fetch` feature flag](../feature_flags.md). +WARNING: +Repositories can be **permanently deleted** by a call to `/projects/:project_id/repository_storage_moves` +that attempts to move a project already stored in a Gitaly Cluster back into that cluster. +See [this issue for more details](https://gitlab.com/gitlab-org/gitaly/-/issues/3752). + Each repository is made read-only for the duration of the move. The repository is not writable until the move has completed. diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md index e8477eaf686..775761d655d 100644 --- a/doc/administration/operations/puma.md +++ b/doc/administration/operations/puma.md @@ -36,6 +36,14 @@ For more details about the Puma configuration, see the ## Puma Worker Killer +Puma forks worker processes as part of a strategy to reduce memory use. + +Each time a worker is created, it shares memory with the primary process and +only uses additional memory when it makes changes or additions to its memory pages. + +Memory use by workers therefore increases over time, and Puma Worker Killer is the +mechanism that recovers this memory. + By default: - The [Puma Worker Killer](https://github.com/schneems/puma_worker_killer) restarts a worker if it @@ -56,6 +64,47 @@ To change the memory limit setting: sudo gitlab-ctl reconfigure ``` +There are costs associated with killing and replacing workers including +reduced capacity to run GitLab, and CPU that is consumed +restarting the workers. `per_worker_max_memory_mb` should be set to a +higher value if the worker killer is replacing workers too often. + +Worker count is calculated based on CPU cores, so a small GitLab deployment +with 4-8 workers may experience performance issues if workers are being restarted +frequently, once or more per minute. This is too often. + +A higher value of `1200` or more would be beneficial if the server has free memory. + +The worker killer checks every 20 seconds, and can be monitored using +[the Puma log](../logs.md#puma_stdoutlog) `/var/log/gitlab/puma/puma_stdout.log`. +For example, for GitLab 13.5: + +```plaintext +PumaWorkerKiller: Out of memory. 4 workers consuming total: 4871.23828125 MB +out of max: 4798.08 MB. Sending TERM to pid 26668 consuming 1001.00390625 MB. +``` + +From this output: + +- The formula that calculates the maximum memory value results in workers + being killed before they reach the `per_worker_max_memory_mb` value. +- The default values for the formula before GitLab 13.5 were 550MB for the primary + and `per_worker_max_memory_mb` specified 850MB for each worker. +- As of GitLab 13.5 the values are primary: 800MB, worker: 1024MB. +- The threshold for workers to be killed is set at 98% of the limit: + + ```plaintext + 0.98 * ( 800 + ( worker_processes * 1024MB ) ) + ``` + +- In the log output above, `0.98 * ( 800 + ( 4 * 1024 ) )` returns the + `max: 4798.08 MB` value. + +Increasing the maximum to `1200`, for example, would set a `max: 5488 MB` value. + +Workers use additional memory on top of the shared memory, how much +depends on a site's use of GitLab. + ## Worker timeout A [timeout of 60 seconds](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/initializers/rack_timeout.rb) @@ -95,7 +144,6 @@ considered as a fair tradeoff in a memory-constraint environment. When running Puma in Single mode, some features are not supported: -- Phased restart do not work: [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/300665) - [Phased restart](https://gitlab.com/gitlab-org/gitlab/-/issues/300665) - [Puma Worker Killer](https://gitlab.com/gitlab-org/gitlab/-/issues/300664) diff --git a/doc/administration/operations/sidekiq_memory_killer.md b/doc/administration/operations/sidekiq_memory_killer.md index 598baa4fcc7..e48ac6e65eb 100644 --- a/doc/administration/operations/sidekiq_memory_killer.md +++ b/doc/administration/operations/sidekiq_memory_killer.md @@ -4,7 +4,7 @@ group: Memory info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Sidekiq MemoryKiller +# Sidekiq MemoryKiller **(FREE SELF)** The GitLab Rails application code suffers from memory leaks. For web requests this problem is made manageable using diff --git a/doc/administration/operations/ssh_certificates.md b/doc/administration/operations/ssh_certificates.md index 374eebeb773..814e742b026 100644 --- a/doc/administration/operations/ssh_certificates.md +++ b/doc/administration/operations/ssh_certificates.md @@ -6,8 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # User lookup via OpenSSH's AuthorizedPrincipalsCommand **(FREE SELF)** -> [Available in](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/19911) GitLab -> Community Edition 11.2. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/19911) in GitLab 11.2. The default SSH authentication for GitLab requires users to upload their SSH public keys before they can use the SSH transport. diff --git a/doc/administration/operations/unicorn.md b/doc/administration/operations/unicorn.md deleted file mode 100644 index 6cee19186f9..00000000000 --- a/doc/administration/operations/unicorn.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'puma.md' -remove_date: '2021-08-26' ---- - -This file was moved to [another location](puma.md). - -<!-- This redirect file can be deleted after <2021-08-26>. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/package_information/defaults.md b/doc/administration/package_information/defaults.md new file mode 100644 index 00000000000..45bea065995 --- /dev/null +++ b/doc/administration/package_information/defaults.md @@ -0,0 +1,72 @@ +--- +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# Package defaults + +Unless configuration is specified in the `/etc/gitlab/gitlab.rb` file, +the package will assume the defaults as noted below. + +## Ports + +See the table below for the list of ports that the Omnibus GitLab assigns +by default: + +| Component | On by default | Communicates via | Alternative | Connection port | +| :----------------------------------------------------: | :------------: | :--------------: | :---------: | :------------------------------------: | +| <a name="gitlab-rails"></a> GitLab Rails | Yes | Port | X | 80 or 443 | +| <a name="gitlab-shell"></a> GitLab Shell | Yes | Port | X | 22 | +| <a name="postgresql"></a> PostgreSQL | Yes | Socket | Port (5432) | X | +| <a name="redis"></a> Redis | Yes | Socket | Port (6379) | X | +| <a name="puma"></a> Puma | Yes | Socket | Port (8080) | X | +| <a name="gitlab-workhorse"></a> GitLab Workhorse | Yes | Socket | Port (8181) | X | +| <a name="nginx-status"></a> NGINX status | Yes | Port | X | 8060 | +| <a name="prometheus"></a> Prometheus | Yes | Port | X | 9090 | +| <a name="node-exporter"></a> Node exporter | Yes | Port | X | 9100 | +| <a name="redis-exporter"></a> Redis exporter | Yes | Port | X | 9121 | +| <a name="postgres-exporter"></a> PostgreSQL exporter | Yes | Port | X | 9187 | +| <a name="pgbouncer-exporter"></a> PgBouncer exporter | No | Port | X | 9188 | +| <a name="gitlab-exporter"></a> GitLab Exporter | Yes | Port | X | 9168 | +| <a name="sidekiq-exporter"></a> Sidekiq exporter | Yes | Port | X | 8082 | +| <a name="puma-exporter"></a> Puma exporter | No | Port | X | 8083 | +| <a name="geo-postgresql"></a> Geo PostgreSQL | No | Socket | Port (5431) | X | +| <a name="redis-sentinel"></a> Redis Sentinel | No | Port | X | 26379 | +| <a name="incoming-email"></a> Incoming email | No | Port | X | 143 | +| <a name="elasticsearch"></a> Elastic search | No | Port | X | 9200 | +| <a name="gitlab-pages"></a> GitLab Pages | No | Port | X | 80 or 443 | +| <a name="gitlab-registry-web"></a> GitLab Registry | No* | Port | X | 80, 443 or 5050 | +| <a name="gitlab-registry"></a> GitLab Registry | No | Port | X | 5000 | +| <a name="ldap"></a> LDAP | No | Port | X | Depends on the component configuration | +| <a name="kerberos"></a> Kerberos | No | Port | X | 8443 or 8088 | +| <a name="omniauth"></a> OmniAuth | Yes | Port | X | Depends on the component configuration | +| <a name="smtp"></a> SMTP | No | Port | X | 465 | +| <a name="remote-syslog"></a> Remote syslog | No | Port | X | 514 | +| <a name="mattermost"></a> Mattermost | No | Port | X | 8065 | +| <a name="mattermost-web"></a> Mattermost | No | Port | X | 80 or 443 | +| <a name="pgbouncer"></a> PgBouncer | No | Port | X | 6432 | +| <a name="consul"></a> Consul | No | Port | X | 8300, 8301(UDP), 8500, 8600[^Consul-notes] | +| <a name="patroni"></a> Patroni | No | Port | X | 8008 | +| <a name="gitlab-kas"></a> GitLab KAS | No | Port | X | 8150 | +| <a name="gitaly"></a> Gitaly | No | Port | X | 8075 | + +Legend: + +- `Component` - Name of the component. +- `On by default` - Is the component running by default. +- `Communicates via` - How the component talks with the other components. +- `Alternative` - If it is possible to configure the component to use different type of communication. The type is listed with default port used in that case. +- `Connection port` - Port on which the component communicates. + +GitLab also expects a filesystem to be ready for the storage of Git repositories +and various other files. + +Note that if you are using NFS (Network File System), files will be carried +over a network which will require, based on implementation, ports `111` and +`2049` to be open. + +NOTE: +In some cases, the GitLab Registry will be automatically enabled by default. Please see [our documentation](../packages/container_registry.md) for more details + + [^Consul-notes]: If using additional Consul functionality, more ports may need to be opened. See the [official documentation](https://www.consul.io/docs/install/ports#ports-table) for the list. diff --git a/doc/administration/package_information/deprecated_os.md b/doc/administration/package_information/deprecated_os.md new file mode 100644 index 00000000000..251dbe1e20e --- /dev/null +++ b/doc/administration/package_information/deprecated_os.md @@ -0,0 +1,82 @@ +--- +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# OS Versions that are no longer supported + +GitLab provides omnibus packages for operating systems only until their +EOL (End-Of-Life). After the EOL date of the OS, GitLab will stop releasing +official packages. The list of deprecated operating systems and the final GitLab +release for them can be found below: + +| OS Version | End Of Life | Last supported GitLab version | +| --------------- | ---------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Raspbian Wheezy | [May 2015](https://downloads.raspberrypi.org/raspbian/images/raspbian-2015-05-07/) | [GitLab CE](https://packages.gitlab.com/app/gitlab/raspberry-pi2/search?q=gitlab-ce_8.17&dist=debian%2Fwheezy) 8.17 | +| OpenSUSE 13.2 | [January 2017](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-9.1&dist=opensuse%2F13.2) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-9.1&dist=opensuse%2F13.2) 9.1 | +| Ubuntu 12.04 | [April 2017](https://ubuntu.com/info/release-end-of-life) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_9.1&dist=ubuntu%2Fprecise) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_9.1&dist=ubuntu%2Fprecise) 9.1 | +| OpenSUSE 42.1 | [May 2017](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-9.3&dist=opensuse%2F42.1) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-9.3&dist=opensuse%2F42.1) 9.3 | +| OpenSUSE 42.2 | [January 2018](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-10.4&dist=opensuse%2F42.2) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-10.4&dist=opensuse%2F42.2) 10.4 | +| Debian Wheezy | [May 2018](https://www.debian.org/News/2018/20180601) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_11.6&dist=debian%2Fwheezy) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_11.6&dist=debian%2Fwheezy) 11.6 | +| Raspbian Jessie | [May 2017](https://downloads.raspberrypi.org/raspbian/images/raspbian-2017-07-05/) | [GitLab CE](https://packages.gitlab.com/app/gitlab/raspberry-pi2/search?q=gitlab-ce_11.7&dist=debian%2Fjessie) 11.7 | +| Ubuntu 14.04 | [April 2019](https://ubuntu.com/info/release-end-of-life) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_11.10&dist=ubuntu%2Ftrusty) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_11.10&dist=ubuntu%2Ftrusty) 11.10 | +| OpenSUSE 42.3 | [July 2019](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-12.1&dist=opensuse%2F42.3) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-12.1&dist=opensuse%2F42.3) 12.1 | +| OpenSUSE 15.0 | [December 2019](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-12.5&dist=opensuse%2F15.0) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-12.5&dist=opensuse%2F15.0) 12.5 | +| Raspbian Stretch | [June 2020](https://downloads.raspberrypi.org/raspbian/images/raspbian-2019-04-09/) | [GitLab CE](https://packages.gitlab.com/app/gitlab/raspberry-pi2/search?q=gitlab-ce_13.2&dist=raspbian%2Fstretch) 13.3 | +| Debian Jessie | [June 2020](https://www.debian.org/News/2020/20200709) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_13.2&dist=debian%2Fjessie) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_13.2&dist=debian%2Fjessie) 13.3 | +| CentOS 6 | [November 2020](https://wiki.centos.org/About/Product) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=13.6&filter=all&filter=all&dist=el%2F6) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=13.6&filter=all&filter=all&dist=el%2F6) 13.6 | +| OpenSUSE 15.1 | [November 2020](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-13.12&dist=opensuse%2F15.1) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-13.12&dist=opensuse%2F15.2) 13.12 | +| Ubuntu 16.04 | [April 2021](https://ubuntu.com/info/release-end-of-life) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_13.12&dist=ubuntu%2Fxenial) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_13.12&dist=ubuntu%2Fxenial) 13.12 | + +NOTE: +An exception to this deprecation policy is when we are unable to provide +packages for the next version of the operating system. The most common reason +for this our package repository provider, Packagecloud, not supporting newer +versions and hence we can't upload packages to it. + +## Update GitLab package sources after upgrading the OS + +After upgrading the Operating System (OS) as per its own documentation, +it may be necessary to also update the GitLab package source URL +in your package manager configuration. +If your package manager reports that no further updates are available, +although [new versions have been released](https://about.gitlab.com/releases/categories/releases/), repeat the +"Add the GitLab package repository" instructions +of the [Linux package install guide](https://about.gitlab.com/install/#content). +Future GitLab upgrades will now be fetched according to your upgraded OS. + +## Supported Operating Systems + +GitLab officially supports LTS versions of operating systems. While OSs like +Ubuntu have a clear distinction between LTS and non-LTS versions, there are +other OSs, openSUSE for example, that don't follow the LTS concept. Hence to +avoid confusion, the official policy is that at any point of time, all the +operating systems supported by GitLab are listed in the [installation +page](https://about.gitlab.com/install/). + +The following lists the currently supported OSs and their possible EOL dates. + +| OS Version | First supported GitLab version | Arch | OS EOL | Details | +| ---------------- | ------------------------------ | --------------- | ------------- | ------------------------------------------------------------ | +| CentOS 7 | GitLab CE / GitLab EE 7.10.0 | x86_64 | June 2024 | <https://wiki.centos.org/About/Product> | +| CentOS 8 | GitLab CE / GitLab EE 12.8.1 | x86_64, aarch64 | Dec 2021 | <https://wiki.centos.org/About/Product> | +| Debian 9 | GitLab CE / GitLab EE 9.3.0 | amd64 | 2022 | <https://wiki.debian.org/DebianReleases#Production_Releases> | +| Debian 10 | GitLab CE / GitLab EE 12.2.0 | amd64, arm64 | TBD | <https://wiki.debian.org/DebianReleases#Production_Releases> | +| OpenSUSE 15.2 | GitLab CE / GitLab EE 13.11.0 | x86_64, aarch64 | Dec 2021 | <https://en.opensuse.org/Lifetime> | +| SLES 12 | GitLab EE 9.0.0 | x86_64 | Oct 2027 | <https://www.suse.com/lifecycle/> | +| Ubuntu 18.04 | GitLab CE / GitLab EE 10.7.0 | amd64 | April 2023 | <https://wiki.ubuntu.com/Releases> | +| Ubuntu 20.04 | GitLab CE / GitLab EE 13.2.0 | amd64, arm64 | April 2025 | <https://wiki.ubuntu.com/Releases> | +| Raspbian Buster | GitLab CE 12.2.0 | armhf | 2022 | <https://wiki.debian.org/DebianReleases#Production_Releases> | + +### Packages for ARM64 + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-omnibus-builder/-/issues/27) in GitLab 13.4. + +GitLab provides arm64/aarch64 packages for some supported operating systems. +You can see if your operating system architecture is supported in the table +above. + +WARNING: +There are currently still some [known issues and limitation](https://gitlab.com/groups/gitlab-org/-/epics/4397) +running GitLab on ARM. diff --git a/doc/administration/package_information/deprecation_policy.md b/doc/administration/package_information/deprecation_policy.md new file mode 100644 index 00000000000..cc16661442a --- /dev/null +++ b/doc/administration/package_information/deprecation_policy.md @@ -0,0 +1,95 @@ +--- +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# Deprecation policy + +The Omnibus GitLab packages come with number of different libraries and services which offers users plethora of configuration options. + +As libraries and services get updated, their configuration options change +and become obsolete. To increase maintainability and preserve a working +setup, various configuration requires removal. + +## Configuration deprecation + +### Policy + +The Omnibus GitLab package will retain configuration for at least **one major** +version. We cannot guarantee that deprecated configuration +will be available in the next major release. See [example](#example) for more details. + +### Notice + +If the configuration becomes obsolete, we will announce the deprecation: + +- via release blog post on `https://about.gitlab.com/blog/`. The blog post item + will contain the deprecation notice together with the target removal date. +- via installation/reconfigure output (if applicable). +- via official documentation on `https://docs.gitlab.com/`. The documentation update will contain the corrected syntax (if applicable) or a date of configuration removal. + +### Procedure + +This section lists steps necessary for deprecating and removing configuration. + +We can differentiate two different types of configuration: + +- Sensitive: Configuration that can cause major service outage ( Data integrity, + installation integrity, preventing users from reaching the installation, etc.) +- Regular: Configuration that can make a feature unavailable but still makes the installation useable ( Change in default project/group settings, miscommunication with other components and similar ) + +We also need to differentiate deprecation and removal procedure. + +#### Deprecating configuration + +Deprecation procedure is similar for both `sensitive` and `regular` configuration. The only difference is in the removal target date. + +Common steps: + +1. Create an issue at the [Omnibus GitLab issue tracker](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues) with details on deprecation type and other necessary information. Apply the label `deprecation`. +1. Decide on the removal target for the deprecated configuration +1. Formulate deprecation notice for each item as noted in [Notice section](#notice) + +Removal target: + +For regular configuration, removal target should always be the date of the **next major** release. If the date is not known, you can reference the next major version. + +For sensitive configuration things are a bit more complicated. +We should aim to not remove sensitive configuration in the *next major* release if the next major release is 2 minor releases away (This number is chosen to match our security backport release policy). + +See the table below for some examples: + +| Config. type | Deprecation announced | Final minor release | Remove | +| -------- | -------- | -------- | -------- | +| Sensitive | 10.1.0 | 10.9.0 | 11.0.0 | +| Sensitive | 10.7.0 | 10.9.0 | 12.0.0 | +| Regular | 10.1.0 | 10.9.0 | 11.0.0 | +| Regular | 10.8.0 | 10.9.0 | 11.0.0 | + +#### Removing configuration + +When deprecation is announced and removal target set, the milestone for the issue +should be changed to match the removal target version. + +The final comment in the issue **has to have**: + +1. Text snippet for the release blog post section +1. Documentation MR ( or snippet ) for introducing the change +1. Draft MR removing the configuration OR details on what needs to be done. See [Adding deprecation messages](https://docs.gitlab.com/omnibus/development/adding-deprecation-messages.html) for more on this + +## Example + +User configuration available in `/etc/gitlab/gitlab.rb` was introduced in GitLab version 10.0, `gitlab_rails['configuration'] = true`. In GitLab version 10.4.0, a new change was introduced that requires rename of this configuration option. New configuration option is `gitlab_rails['better_configuration'] = true`. Development team will translate the old configuration into new one +and trigger a deprecation procedure. + +This means that these two configuration +options will both be valid through GitLab version 10. In other words, +if you still have `gitlab_rails['configuration'] = true` set in GitLab 10.8.0 +the feature will continue working the same way as if you had `gitlab_rails['better_configuration'] = true` set. +However, setting the old version of configuration will print out a deprecation +notice at the end of installation/upgrade/reconfigure run. + +With GitLab 11, `gitlab_rails['configuration'] = true` will no longer work and you will have to manually change the configuration in `/etc/gitlab/gitlab.rb` to the new valid config. +**Note** If this configuration option is sensitive and can put integrity of the installation or +data in danger, installation/upgrade will be aborted. diff --git a/doc/administration/package_information/index.md b/doc/administration/package_information/index.md new file mode 100644 index 00000000000..e18fb621b89 --- /dev/null +++ b/doc/administration/package_information/index.md @@ -0,0 +1,101 @@ +--- +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# Package information + +The Omnibus GitLab package is bundled with all dependencies required for GitLab +to function correctly. More details can be found +at [bundling dependencies document](omnibus_packages.md). + +## Package Version + +The released package versions are in the format `MAJOR.MINOR.PATCH-EDITION.OMNIBUS_RELEASE` + +| Component | Meaning | Example | +| --------- | ------- | ------- | +| MAJOR.MINOR.PATCH | The GitLab version this corresponds to | 13.3.0 | +| EDITION | The edition of GitLab this corresponds to | ee | +| OMNIBUS_RELEASE | The omnibus release. Usually, this will be 0. This will be incremented if we need to build a new package without changing the GitLab version. | 0 | + +## Licenses + +See [licensing](licensing.md) + +## Defaults + +The Omnibus GitLab package requires various configuration to get the +components in working order. +If the configuration is not provided, the package will use the default +values assumed in the package. + +These defaults are noted in the package [defaults document](defaults.md). + +## Checking the versions of bundled software + +Once the Omnibus GitLab package is installed, all versions of the bundled +libraries are located in `/opt/gitlab/version-manifest.txt`. + +If you don't have the package installed, you can always check the Omnibus GitLab +[source repository](https://gitlab.com/gitlab-org/omnibus-gitlab/tree/master), specifically the +[config directory](https://gitlab.com/gitlab-org/omnibus-gitlab/tree/master/config). + +For example, if you take a look at the `8-6-stable` branch, you can conclude that +8.6 packages were running [Ruby 2.1.8](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8-6-stable/config/projects/gitlab.rb#L48). +Or, that 8.5 packages were bundled with [NGINX 1.9.0](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8-5-stable/config/software/nginx.rb#L20). + +## Signatures of GitLab, Inc. provided packages + +Documentation on package signatures can be found at [Signed Packages](signed_packages.md) + +## Checking for newer configuration options on upgrade + +Configuration file in `/etc/gitlab/gitlab.rb` is created on initial installation +of the Omnibus GitLab package. On subsequent package upgrades, the configuration +file is not updated with new configuration. This is done in order to avoid +accidental overwrite of user configuration provided in `/etc/gitlab/gitlab.rb`. + +New configuration options are noted in the +[`gitlab.rb.template` file](https://gitlab.com/gitlab-org/omnibus-gitlab/raw/master/files/gitlab-config-template/gitlab.rb.template). + +The Omnibus GitLab package also provides convenience command which will +compare the existing user configuration with the latest version of the +template contained in the package. + +To view a diff between your configuration file and the latest version, run: + +```shell +sudo gitlab-ctl diff-config +``` + +_**Note:** This command is available from GitLab 8.17_ + +**Important:** If you are copy-pasting the output of this command into your +`/etc/gitlab/gitlab.rb` configuration file, make sure to omit leading `+` and `-` +on each line. + +## Init system detection + +Omnibus GitLab will attempt to query the underlaying system in order to +check which init system it uses. +This manifests itself as a `WARNING` during the `sudo gitlab-ctl reconfigure` +run. + +Depending on the init system, this `WARNING` can be one of: + +```plaintext +/sbin/init: unrecognized option '--version' +``` + +when the underlying init system *IS NOT* upstart. + +```plaintext + -.mount loaded active mounted / +``` + +when the underlying init system *IS* systemd. + +These warnings _can be safely ignored_. They are not suppressed because this +allows everyone to debug possible detection issues faster. diff --git a/doc/administration/package_information/licensing.md b/doc/administration/package_information/licensing.md new file mode 100644 index 00000000000..8557a94bf93 --- /dev/null +++ b/doc/administration/package_information/licensing.md @@ -0,0 +1,79 @@ +--- +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# Package Licensing + +## License + +While GitLab itself is MIT, the Omnibus GitLab sources are licensed under the Apache-2.0. + +## License file location + +Starting with version 8.11, the Omnibus GitLab package contains license +information of all software that is bundled within the package. + +After installing the package, licenses for each individual bundled library +can be found in `/opt/gitlab/LICENSES` directory. + +There is also one `LICENSE` file which contains all licenses compiled together. +This compiled license can be found in `/opt/gitlab/LICENSE` file. + +Starting with version 9.2, the Omnibus GitLab package ships a +`dependency_licenses.json` file containing version and license information of +all bundled software, including software libraries, Ruby gems that the rails +application uses, and JavaScript libraries that is required for the frontend +components. This file, being in JSON format, is easily machine parseable and +can be used for automated checks or validations. The file may be found at +`/opt/gitlab/dependency_licenses.json`. + +Starting with version 11.3, we have also made the license information available +online, at: <https://gitlab-org.gitlab.io/omnibus-gitlab/licenses.html> + +## Checking licenses + +The Omnibus GitLab package is made up of many pieces of software, comprising code +that is covered by many different licenses. Those licenses are provided and +compiled as stated above. + +Starting with version 8.13, GitLab has placed an additional step into +Omnibus GitLab. The `license_check` step calls +`lib/gitlab/tasks/license_check.rake`, which checks the compiled `LICENSE` file +against the current list of approved and questionable licenses as denoted in the +arrays at the top of the script. This script will output one of `Good`, +`Unknown` or `Check` for each piece of software that is a part of the +Omnibus GitLab package. + +- `Good`: denotes a license that is approved for all usage types, within GitLab and + Omnibus GitLab. +- `Unknown`: denotes a license that is not recognized in the list of 'good' or 'bad', + which should be immediately reviewed for implications of use. +- `Check`: denotes a license that has the potential be incompatible with GitLab itself, + and thus should be checked for how it is used as a part of the Omnibus GitLab package + to ensure compliance. + +This list is currently sourced from the [GitLab development documentation on licensing](https://gitlab.com/gitlab-org/gitlab-foss/blob/master/doc/development/licensing.md). +However, due to the nature of the Omnibus GitLab package the licenses may not apply +in the same way. Such as with `git` and `rsync`. See the [GNU License FAQ](https://www.gnu.org/licenses/gpl-faq.en.html#MereAggregation) + +## License acknowledgements + +### libjpeg-turbo - BSD 3-clause license + +This software is based in part on the work of the Independent JPEG Group. + +## Trademark Usage + +Within the GitLab documentation, reference to third party technology(ies) and/or trademarks of third party entities, may be made. The inclusion of reference to third party technology and/or entities is solely for the purposes of example(s) of how GitLab software may interact with, or be used in conjunction with, such third party technology. +All trademarks, materials, documentation, and other intellectual property remain the property of any/all such third party. + +### Trademark Requirements + +Use of GitLab Trademarks must be in compliance with the standards set forth in [our guidelines](https://about.gitlab.com/handbook/marketing/corporate-marketing/brand-activation/trademark-guidelines/) (as updated from time to time). +CHEF® and all Chef marks are owned by Progress Software Corporation and must be used in accordance with the [Progress Software Trademark Usage Policy](https://www.progress.com/legal/trademarks). + +When using a GitLab or 3rd party trademark in documentation, include the (R) symbol in the first instance, for example, "Chef(R) is used for configuring...." You may omit the symbol in subsequent instances. + +If a trademark owner requires a particular notice or trademark requirement, such notice or requirement should be stated above. diff --git a/doc/administration/package_information/omnibus_packages.md b/doc/administration/package_information/omnibus_packages.md new file mode 100644 index 00000000000..aa73534fc55 --- /dev/null +++ b/doc/administration/package_information/omnibus_packages.md @@ -0,0 +1,115 @@ +--- +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# Omnibus based packages and images + +Below you can find some basic information on why GitLab provides packages and +a Docker image that come with bundled dependencies. + +These methods are great for physical and virtual machine installations, and simple Docker installations. + +## Goals + +We have a few core goals with these packages: + +1. Extremely easy to install, upgrade, maintain. +1. Support for a wide variety of operating systems +1. Wide support of cloud service providers + +## Omnibus GitLab Architecture + +GitLab in its core is a Ruby on Rails project. However, GitLab as a whole +application is more complex and has multiple components. If these components are +not present or are incorrectly configured, GitLab will not work or it will work +unpredictably. + +The [GitLab Architecture Overview](../../development/architecture.md#gitlab-architecture-overview) shows some of these components and how they +interact. Each of these components needs to be configured and kept up to date. + +Most of the components also have external dependencies. For example, the Rails +application depends on a number of [Ruby gems](https://gitlab.com/gitlab-org/gitlab-foss/blob/master/Gemfile.lock). Some of these dependencies also +have their own external dependencies which need to be present on the Operating +System in order for them to function correctly. + +Furthermore, GitLab has a monthly release cycle requiring frequent maintenance +to stay up to date. + +All the things listed above present a challenge for the user maintaining the GitLab +installation. + +## External Software Dependencies + +For applications such as GitLab, external dependencies usually bring the following +challenges: + +- Keeping versions in sync between direct and indirect dependencies +- Availability of a version on a specific Operating System +- Version changes can introduce or remove previously used configuration +- Security implications when library is marked as vulnerable but does not have + a new version released yet + +Keep in mind that if a dependency exists on your Operating System, it does not +necessarily exist on other supported OSs. + +## Benefits + +A few benefits of a package with bundled dependencies: + +1. Minimal effort required to install GitLab. +1. Minimum configuration required to get GitLab up and running. +1. Minimum effort required to upgrade between GitLab versions. +1. Multiple platforms supported. +1. Maintenance on older platforms is greatly simplified. +1. Less effort to support potential issues. + +## Drawbacks + +Some drawbacks of a package with bundled dependencies: + +1. Duplication with possibly existing software. +1. Less flexibility in configuration. + +## Why would I install an omnibus package when I can use a system package? + +The answer can be simplified to: less maintenance required. Instead of handling +multiple packages that *can* break existing functionality if the versions are +not compatible, only handle one. + +Multiple packages require correct configuration in multiple locations. +Keeping configuration in sync can be error prone. + +If you have the skill set to maintain all current dependencies and enough time +to handle any future dependencies that might get introduced, the above listed +reasons might not be good enough for you to not use the omnibus package. + +There are two things to keep in mind before going down this route: + +1. Getting support for any problems + you encounter might be more difficult due to the number of possibilities that exist + when using a library version that is not tested by majority of users. +1. Omnibus package also allows shutting off of any services that you do not need, + if you need to run a component independently. For example, you can use a + [non-bundled PostgreSQL database](https://docs.gitlab.com/omnibus/settings/database.html#using-a-non-packaged-postgresql-database-management-server) with the omnibus package. + +Keep in mind that a non-standard solution like the omnibus package +might be a better fit when the application has a number of moving parts. + +## Docker image with multiple services + +[GitLab Docker image](../../install/docker.md#gitlab-docker-images) is based on the omnibus package. + +Considering that container spawned from this image contains multiple processes, +these types of containers are also referred to as 'fat containers'. + +There are reasons for and against an image like this, but they are similar to +what was noted above: + +1. Very simple to get started. +1. Upgrading to the latest version is extremely simple. +1. Running separate services in multiple containers and keeping them running + can be more complex and might not be required for a given install. + +This method is useful for organizations just getting started with containers and schedulers, and may not be ready for a more complex installation. This method is a great introduction, and will work well for smaller organizations. diff --git a/doc/administration/package_information/postgresql_versions.md b/doc/administration/package_information/postgresql_versions.md new file mode 100644 index 00000000000..89da4864872 --- /dev/null +++ b/doc/administration/package_information/postgresql_versions.md @@ -0,0 +1,42 @@ +--- +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# PostgreSQL versions shipped with Omnibus GitLab + +NOTE: +This table lists only GitLab versions where a significant change happened in the +package regarding PostgreSQL versions, not all. + +Usually, PostgreSQL versions change with major or minor GitLab releases. However, patch versions +of Omnibus GitLab sometimes update the patch level of PostgreSQL. + +For example: + +- Omnibus 12.7.6 shipped with PostgreSQL 9.6.14 and 10.9. +- Omnibus 12.7.7 shipped with PostgreSQL 9.6.17 and 10.12. + +[Find out which versions of PostgreSQL (and other components) ship with +each Omnibus GitLab release](https://gitlab-org.gitlab.io/omnibus-gitlab/licenses.html). + +Read more about update policies and warnings in the PostgreSQL +[upgrade docs](https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server). + +| GitLab version | PostgreSQL versions | Default version for fresh installs | Default version for upgrades | Notes | +| -------------- | --------------------- | ---------------------------------- | ---------------------------- | ----- | +| 14.1 | 12.6, 13.3 | 12.6 | 12.6 | PostgreSQL 13 available for fresh installations if not using Geo or High Availability. | +| 14.0 | 12.6 | 12.6 | 12.6 | HA installations with repmgr are no longer supported and will be prevented from upgrading to Omnibus GitLab 14.0 | +| 13.8 | 11.9, 12.4 | 12.4 | 12.4 | Package upgrades automatically performed PostgreSQL upgrade for nodes that are not part of a Geo or HA cluster.). | +| 13.7 | 11.9, 12.4 | 12.4 | 11.9 | For upgrades users can manually upgrade to 12.4 following the [upgrade docs](https://docs.gitlab.com/omnibus/settings/database.html#gitlab-133-and-later). | +| 13.4 | 11.9, 12.4 | 11.9 | 11.9 | Package upgrades aborted if users not running PostgreSQL 11 already | +| 13.3 | 11.7, 12.3 | 11.7 | 11.7 | Package upgrades aborted if users not running PostgreSQL 11 already | +| 13.0 | 11.7 | 11.7 | 11.7 | Package upgrades aborted if users not running PostgreSQL 11 already | +| 12.10 | 9.6.17, 10.12, and 11.7 | 11.7 | 11.7 | Package upgrades automatically performed PostgreSQL upgrade for nodes that are not part of a Geo or repmgr cluster. | +| 12.8 | 9.6.17, 10.12, and 11.7 | 10.12 | 10.12 | Users can manually upgrade to 11.7 following the upgrade docs. | +| 12.0 | 9.6.11 and 10.7 | 10.7 | 10.7 | Package upgrades automatically performed PostgreSQL upgrade. | +| 11.11 | 9.6.11 and 10.7 | 9.6.11 | 9.6.11 | Users can manually upgrade to 10.7 following the upgrade docs. | +| 10.0 | 9.6.3 | 9.6.3 | 9.6.3 | Package upgrades aborted if users still on 9.2. | +| 9.0 | 9.2.18 and 9.6.1 | 9.6.1 | 9.6.1 | Package upgrades automatically performed PostgreSQL upgrade. | +| 8.14 | 9.2.18 and 9.6.1 | 9.2.18 | 9.2.18 | Users can manually upgrade to 9.6 following the upgrade docs. | diff --git a/doc/administration/package_information/signed_packages.md b/doc/administration/package_information/signed_packages.md new file mode 100644 index 00000000000..fb994809460 --- /dev/null +++ b/doc/administration/package_information/signed_packages.md @@ -0,0 +1,25 @@ +--- +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# Package Signatures + +As of the release of GitLab 9.5 on August 22, 2017, GitLab provides signed Omnibus GitLab packages for RPM and DEB based distributions. This means that all packages provided on <https://packages.gitlab.com> are signed, starting with `9.5.0`, and all future versions of supported branches (e.g. `9.3.x` and `9.4.x` after August 22, 2017). Any package version prior to August 22, 2017, will not be signed. Please pass the appropriate argument to your package manager. (Example: `yum --nogpgcheck`) + +Omnibus GitLab packages produced by GitLab are created via the [Omnibus](https://github.com/chef/omnibus) tool, for which GitLab has added DEB signing via `debsigs` in [our own fork](https://gitlab.com/gitlab-org/omnibus). This addition, combined with the existing functionality of RPM signing, allows GitLab to provide signed packages for all supported distributions using DEB or RPM. + +These packages are produced by the GitLab CI process, as found in the [Omnibus GitLab project](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/.gitlab-ci.yml), prior to their delivery to <https://packages.gitlab.com> to ensure provide assurance that the packages are not altered prior to delivery to our community. + +## GnuPG Public Keys + +All packages are signed with [GnuPG](https://www.gnupg.org/), in a method appropriate for their format. The key used to sign these packages can be found on [pgp.mit.edu](https://pgp.mit.edu) at [0x3cfcf9baf27eab47](https://pgp.mit.edu/pks/lookup?op=vindex&search=0x3CFCF9BAF27EAB47) + +## Verifying Signatures + +Information on how to verify GitLab package signatures can be found in [Package Signatures](https://docs.gitlab.com/omnibus/update/package_signatures.html). + +## GPG Signature Management + +Information on how GitLab manages GPG keys for package signing can be found in [the runbooks](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/packaging/manage-package-signing-keys.md). diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index eb118709f94..1067474c8b4 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -321,7 +321,7 @@ The different supported drivers are: | Driver | Description | |--------------|--------------------------------------| | `filesystem` | Uses a path on the local file system | -| `Azure` | Microsoft Azure Blob Storage | +| `azure` | Microsoft Azure Blob Storage | | `gcs` | Google Cloud Storage | | `s3` | Amazon Simple Storage Service. Be sure to configure your storage bucket with the correct [S3 Permission Scopes](https://docs.docker.com/registry/storage-drivers/s3/#s3-permission-scopes). | | `swift` | OpenStack Swift Object Storage | @@ -1054,6 +1054,80 @@ PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin You may want to add the `-m` flag to [remove untagged manifests and unreferenced layers](#removing-untagged-manifests-and-unreferenced-layers). +## Configuring GitLab and Registry to run on separate nodes (Omnibus GitLab) + +By default, package assumes that both services are running on the same node. +In order to get GitLab and Registry to run on a separate nodes, separate configuration +is necessary for Registry and GitLab. + +### Configuring Registry + +Below you will find configuration options you should set in `/etc/gitlab/gitlab.rb`, +for Registry to run separately from GitLab: + +- `registry['registry_http_addr']`, default [set programmatically](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/gitlab/libraries/registry.rb#L50). Needs to be reachable by web server (or LB). +- `registry['token_realm']`, default [set programmatically](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/gitlab/libraries/registry.rb#L53). Specifies the endpoint to use to perform authentication, usually the GitLab URL. + This endpoint needs to be reachable by user. +- `registry['http_secret']`, [random string](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/gitlab/libraries/registry.rb#L32). A random piece of data used to sign state that may be stored with the client to protect against tampering. +- `registry['internal_key']`, default [automatically generated](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/gitlab/recipes/gitlab-rails.rb#L113-119). Contents of the key that GitLab uses to sign the tokens. They key gets created on the Registry server, but it won't be used there. +- `gitlab_rails['registry_key_path']`, default [set programmatically](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/gitlab/recipes/gitlab-rails.rb#L35). This is the path where `internal_key` contents will be written to disk. +- `registry['internal_certificate']`, default [automatically generated](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/registry/recipes/enable.rb#L60-66). Contents of the certificate that GitLab uses to sign the tokens. +- `registry['rootcertbundle']`, default [set programmatically](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/registry/recipes/enable.rb#L60). Path to certificate. This is the path where `internal_certificate` + contents will be written to disk. +- `registry['health_storagedriver_enabled']`, default [set programmatically](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-7-stable/files/gitlab-cookbooks/gitlab/libraries/registry.rb#L88). Configure whether health checks on the configured storage driver are enabled. +- `gitlab_rails['registry_issuer']`, [default value](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/gitlab/attributes/default.rb#L153). This setting needs to be set the same between Registry and GitLab. + +### Configuring GitLab + +Below you will find configuration options you should set in `/etc/gitlab/gitlab.rb`, +for GitLab to run separately from Registry: + +- `gitlab_rails['registry_enabled']`, must be set to `true`. This setting will + signal to GitLab that it should allow Registry API requests. +- `gitlab_rails['registry_api_url']`, default [set programmatically](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/gitlab/libraries/registry.rb#L52). This is the Registry URL used internally that users do not need to interact with, `registry['registry_http_addr']` with scheme. +- `gitlab_rails['registry_host']`, eg. `registry.gitlab.example`. Registry endpoint without the scheme, the address that gets shown to the end user. +- `gitlab_rails['registry_port']`. Registry endpoint port, visible to the end user. +- `gitlab_rails['registry_issuer']` must match the issuer in the Registry configuration. +- `gitlab_rails['registry_key_path']`, path to the key that matches the certificate on the + Registry side. +- `gitlab_rails['internal_key']`, contents of the key that GitLab uses to sign the tokens. + +## Architecture of GitLab Container Registry + +The GitLab registry is what users use to store their own Docker images. +Because of that the Registry is client facing, meaning that we expose it directly +on the web server (or load balancers, LB for short). + + + +The flow described by the diagram above: + +1. A user runs `docker login registry.gitlab.example` on their client. This reaches the web server (or LB) on port 443. +1. Web server connects to the Registry backend pool (by default, using port 5000). Since the user + didn’t provide a valid token, the Registry returns a 401 HTTP code and the URL (`token_realm` from + Registry configuration) where to get one. This points to the GitLab API. +1. The Docker client then connects to the GitLab API and obtains a token. +1. The API signs the token with the registry key and hands it to the Docker client +1. The Docker client now logs in again with the token received from the API. It can now push and pull Docker images. + +Reference: <https://docs.docker.com/registry/spec/auth/token/> + +### Communication between GitLab and Registry + +Registry doesn’t have a way to authenticate users internally so it relies on +GitLab to validate credentials. The connection between Registry and GitLab is +TLS encrypted. The key is used by GitLab to sign the tokens while the certificate +is used by Registry to validate the signature. By default, a self-signed certificate key pair is generated +for all installations. This can be overridden as needed. + +GitLab interacts with the Registry using the Registry private key. When a Registry +request goes out, a new short-living (10 minutes) namespace limited token is generated +and signed with the private key. +The Registry then verifies that the signature matches the registry certificate +specified in its configuration and allows the operation. +GitLab background jobs processing (through Sidekiq) also interacts with Registry. +These jobs talk directly to Registry in order to handle image deletion. + ## Troubleshooting Before diving in to the following sections, here's some basic troubleshooting: @@ -1122,7 +1196,7 @@ CI/CD > Container Registry > Authorization token duration (minutes)**. ### Docker login attempt fails with: 'token signed by untrusted key' -[Registry relies on GitLab to validate credentials](https://docs.gitlab.com/omnibus/architecture/registry/). +[Registry relies on GitLab to validate credentials](#architecture-of-gitlab-container-registry) If the registry fails to authenticate valid login attempts, you get the following error message: ```shell diff --git a/doc/administration/packages/dependency_proxy.md b/doc/administration/packages/dependency_proxy.md index c4906ef6d8e..32e7e191011 100644 --- a/doc/administration/packages/dependency_proxy.md +++ b/doc/administration/packages/dependency_proxy.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GitLab Dependency Proxy administration **(FREE SELF)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7934) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.11. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/273655) to [GitLab Free](https://about.gitlab.com/pricing/) in GitLab 13.6. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/273655) from GitLab Premium to GitLab Free in 13.6. GitLab can be used as a dependency proxy for a variety of common package managers. diff --git a/doc/administration/packages/img/gitlab-registry-architecture.png b/doc/administration/packages/img/gitlab-registry-architecture.png Binary files differnew file mode 100644 index 00000000000..742678d5e11 --- /dev/null +++ b/doc/administration/packages/img/gitlab-registry-architecture.png diff --git a/doc/administration/packages/index.md b/doc/administration/packages/index.md index 2c2e3fc0442..37473d35573 100644 --- a/doc/administration/packages/index.md +++ b/doc/administration/packages/index.md @@ -14,22 +14,17 @@ The Packages feature allows GitLab to act as a repository for the following: The Package Registry supports the following formats: -<div class="row"> -<div class="col-md-9"> -<table align="left" style="width:50%"> -<tr style="background:#dfdfdf"><th>Package type</th><th>GitLab version</th></tr> -<tr><td><a href="https://docs.gitlab.com/ee/user/packages/composer_repository/index.html">Composer</a></td><td>13.2+</td></tr> -<tr><td><a href="https://docs.gitlab.com/ee/user/packages/conan_repository/index.html">Conan</a></td><td>12.6+</td></tr> -<tr><td><a href="https://docs.gitlab.com/ee/user/packages/go_proxy/index.html">Go</a></td><td>13.1+</td></tr> -<tr><td><a href="https://docs.gitlab.com/ee/user/packages/maven_repository/index.html">Maven</a></td><td>11.3+</td></tr> -<tr><td><a href="https://docs.gitlab.com/ee/user/packages/npm_registry/index.html">npm</a></td><td>11.7+</td></tr> -<tr><td><a href="https://docs.gitlab.com/ee/user/packages/nuget_repository/index.html">NuGet</a></td><td>12.8+</td></tr> -<tr><td><a href="https://docs.gitlab.com/ee/user/packages/pypi_repository/index.html">PyPI</a></td><td>12.10+</td></tr> -<tr><td><a href="https://docs.gitlab.com/ee/user/packages/generic_packages/index.html">Generic packages</a></td><td>13.5+</td></tr> -<tr><td><a href="https://docs.gitlab.com/ee/user/packages/helm_repository/index.html">Helm Charts</a></td><td>14.1+</td></tr> -</table> -</div> -</div> +| Package type | GitLab version | +|-------------------------------------------------------------------|----------------| +| [Composer](../../user/packages/composer_repository/index.md) | 13.2+ | +| [Conan](../../user/packages/conan_repository/index.md) | 12.6+ | +| [Go](../../user/packages/go_proxy/index.md) | 13.1+ | +| [Maven](../../user/packages/maven_repository/index.md) | 11.3+ | +| [npm](../../user/packages/npm_registry/index.md) | 11.7+ | +| [NuGet](../../user/packages/nuget_repository/index.md) | 12.8+ | +| [PyPI](../../user/packages/pypi_repository/index.md) | 12.10+ | +| [Generic packages](../../user/packages/generic_packages/index.md) | 13.5+ | +| [Helm Charts](../../user/packages/helm_repository/index.md) | 14.1+ | ## Accepting contributions diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index 5aeb3eaef7f..8b7af5ee170 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -196,43 +196,6 @@ to use the HTTPS protocol. WARNING: Multiple wildcards for one instance is not supported. Only one wildcard per instance can be assigned. -### Additional configuration for Docker container - -The GitLab Pages daemon doesn't have permissions to bind mounts when it runs -in a Docker container. To overcome this issue, you must change the `chroot` -behavior: - -1. Edit `/etc/gitlab/gitlab.rb`. -1. Set the `inplace_chroot` to `true` for GitLab Pages: - - ```ruby - gitlab_pages['inplace_chroot'] = true - ``` - -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - -NOTE: -`inplace_chroot` option might not work with the other features, such as [Pages Access Control](#access-control). -The [GitLab Pages README](https://gitlab.com/gitlab-org/gitlab-pages#caveats) has more information about caveats and workarounds. - -### Jailing mechanism disabled by default for API-based configuration - -Starting from GitLab 14.1 the [jailing/chroot mechanism is disabled by default](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/589). -If you are using API-based configuration and the new [Zip storage architecture](#zip-storage) -there is nothing you need to do. - -If you run into any problems, [open a new issue](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/new) -and enable the jail again by setting the environment variable: - -1. Edit `/etc/gitlab/gitlab.rb`. -1. Set the `DAEMON_ENABLE_JAIL` environment variable to `true` for GitLab Pages: - - ```ruby - gitlab_pages['env']['DAEMON_ENABLE_JAIL'] = "true" - ``` - -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - ### Global settings Below is a table of all configuration settings known to Pages in Omnibus GitLab, @@ -270,7 +233,7 @@ control over how the Pages daemon runs and serves content in your environment. | `auth_scope` | The OAuth application scope to use for authentication. Must match GitLab Pages OAuth application settings. Leave blank to use `api` scope by default. | | `gitlab_server` | Server to use for authentication when access control is enabled; defaults to GitLab `external_url`. | | `headers` | Specify any additional http headers that should be sent to the client with each response. Multiple headers can be given as an array, header and value as one string, for example `['my-header: myvalue', 'my-other-header: my-other-value']` | -| `inplace_chroot` | On [systems that don't support bind-mounts](index.md#additional-configuration-for-docker-container), this instructs GitLab Pages to `chroot` into its `pages_path` directory. Some caveats exist when using in-place `chroot`; refer to the GitLab Pages [README](https://gitlab.com/gitlab-org/gitlab-pages/blob/master/README.md#caveats) for more information. | +| `inplace_chroot` | [REMOVED in GitLab 14.3.](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/561) On [systems that don't support bind-mounts](index.md#gitlab-pages-fails-to-start-in-docker-container), this instructs GitLab Pages to `chroot` into its `pages_path` directory. Some caveats exist when using in-place `chroot`; refer to the GitLab Pages [README](https://gitlab.com/gitlab-org/gitlab-pages/blob/master/README.md#caveats) for more information. | | `enable_disk` | Allows the GitLab Pages daemon to serve content from disk. Shall be disabled if shared disk storage isn't available. | | `insecure_ciphers` | Use default list of cipher suites, may contain insecure ones like 3DES and RC4. | | `internal_gitlab_server` | Internal GitLab server address used exclusively for API requests. Useful if you want to send that traffic over an internal load balancer. Defaults to GitLab `external_url`. | @@ -298,8 +261,9 @@ control over how the Pages daemon runs and serves content in your environment. | `pages_path` | The directory on disk where pages are stored, defaults to `GITLAB-RAILS/shared/pages`. | | **`pages_nginx[]`** | | | `enable` | Include a virtual host `server{}` block for Pages inside NGINX. Needed for NGINX to proxy traffic back to the Pages daemon. Set to `false` if the Pages daemon should directly receive all requests, for example, when using [custom domains](index.md#custom-domains). | -| `FF_ENABLE_REDIRECTS` | Feature flag to disable redirects (enabled by default). Read the [redirects documentation](../../user/project/pages/redirects.md#disable-redirects) for more information. | -| `use_legacy_storage` | Temporarily-introduced parameter allowing to use legacy domain configuration source and storage. [Will be removed in GitLab 14.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6166). | +| `FF_ENABLE_REDIRECTS` | Feature flag to enable/disable redirects (enabled by default). Read the [redirects documentation](../../user/project/pages/redirects.md#feature-flag-for-redirects) for more information. | +| `FF_ENABLE_PLACEHOLDERS` | Feature flag to enable/disable rewrites (disabled by default). Read the [redirects documentation](../../user/project/pages/redirects.md#feature-flag-for-rewrites) for more information. | +| `use_legacy_storage` | Temporarily-introduced parameter allowing to use legacy domain configuration source and storage. [Removed in 14.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6166). | --- @@ -393,7 +357,7 @@ adding a GitLab-controlled verification code to the DNS records for that domain. If your user base is private or otherwise trusted, you can disable the verification requirement: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Preferences**. 1. Expand **Pages**. 1. Clear the **Require users to prove ownership of custom domains** checkbox. @@ -410,7 +374,7 @@ sites served under a custom domain. To enable it: 1. Choose an email address on which you want to receive notifications about expiring domains. -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Preferences**. 1. Expand **Pages**. 1. Enter the email address for receiving notifications and accept Let's Encrypt's Terms of Service. @@ -465,7 +429,7 @@ pre-existing applications must modify the GitLab Pages OAuth application. Follow this: 1. Enable [access control](#access-control). -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Applications**. 1. Expand **GitLab Pages**. 1. Clear the `api` scope's checkbox and select the desired scope's checkbox (for example, @@ -484,7 +448,7 @@ This can be useful to preserve information published with Pages websites to the of your instance only. To do that: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Preferences**. 1. Expand **Pages**. 1. Select the **Disable public access to Pages sites** checkbox. @@ -522,7 +486,7 @@ Authority (CA) in the system certificate store. For Omnibus, this is fixed by [installing a custom CA in Omnibus GitLab](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). -### Zip serving and cache configuration +### ZIP serving and cache configuration > [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/392) in GitLab 13.7. @@ -530,19 +494,19 @@ WARNING: These are advanced settings. The recommended default values are set inside GitLab Pages. You should change these settings only if absolutely necessary. Use extreme caution. -GitLab Pages can serve content from zip archives through object storage (an +GitLab Pages can serve content from ZIP archives through object storage (an [issue](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/485) exists for supporting disk storage -as well). It uses an in-memory cache to increase the performance when serving content from a zip +as well). It uses an in-memory cache to increase the performance when serving content from a ZIP archive. You can modify the cache behavior by changing the following configuration flags. | Setting | Description | | ------- | ----------- | -| `zip_cache_expiration` | The cache expiration interval of zip archives. Must be greater than zero to avoid serving stale content. Default is 60s. | +| `zip_cache_expiration` | The cache expiration interval of ZIP archives. Must be greater than zero to avoid serving stale content. Default is 60s. | | `zip_cache_cleanup` | The interval at which archives are cleaned from memory if they have already expired. Default is 30s. | | `zip_cache_refresh` | The time interval in which an archive is extended in memory if accessed before `zip_cache_expiration`. This works together with `zip_cache_expiration` to determine if an archive is extended in memory. See the [example below](#zip-cache-refresh-example) for important details. Default is 30s. | -| `zip_open_timeout` | The maximum time allowed to open a zip archive. Increase this time for big archives or slow network connections, as doing so may affect the latency of serving Pages. Default is 30s. | +| `zip_open_timeout` | The maximum time allowed to open a ZIP archive. Increase this time for big archives or slow network connections, as doing so may affect the latency of serving Pages. Default is 30s. | -#### Zip cache refresh example +#### ZIP cache refresh example Archives are refreshed in the cache (extending the time they are held in memory) if they're accessed before `zip_cache_expiration`, and the time left before expiring is less than or equal to @@ -556,7 +520,7 @@ opened) it's refreshed. This extends the time the archive remains in memory from After an archive reaches `zip_cache_expiration`, it's marked as expired and removed on the next `zip_cache_cleanup` interval. - + ## Activate verbose logging for daemon @@ -665,7 +629,7 @@ Follow the steps below to configure the proxy listener of GitLab Pages. To set the global maximum pages size for a project: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Preferences**. 1. Expand **Pages**. 1. Edit the **Maximum size of pages**. @@ -742,6 +706,9 @@ database encryption. Proceed with caution. gitlab_pages['gitlab_server'] = 'http://<gitlab_server_IP_or_URL>' ``` +1. If you have custom UID/GID settings on the **GitLab server**, add them to the **Pages server** `/etc/gitlab/gitlab.rb` as well, + otherwise running a `gitlab-ctl reconfigure` on the **GitLab server** can change file ownership and cause Pages requests to fail. + 1. Create a backup of the secrets file on the **Pages server**: ```shell @@ -1047,7 +1014,7 @@ If you use [object storage](#using-object-storage), you can disable local storag Starting from GitLab 13.12, this setting also disables the [legacy storage](#migrate-legacy-storage-to-zip-storage), so if you were using NFS to serve Pages, you can completely disconnect from it. -## Migrate GitLab Pages to 14.0 +## Prepare GitLab Pages for 14.0 In GitLab 14.0 a number of breaking changes were introduced which may require some user intervention. The steps below describe the best way to migrate without causing any downtime for your GitLab instance. @@ -1104,6 +1071,9 @@ You can also find the log file in `/var/log/gitlab/gitlab-pages/current`. ### `open /etc/ssl/ca-bundle.pem: permission denied` +WARNING: +This issue is fixed in GitLab 14.3 and above, try upgrading GitLab first. + GitLab Pages runs inside a `chroot` jail, usually in a uniquely numbered directory like `/tmp/gitlab-pages-*`. @@ -1135,6 +1105,9 @@ sudo gitlab-ctl restart gitlab-pages ### `dial tcp: lookup gitlab.example.com` and `x509: certificate signed by unknown authority` +WARNING: +This issue is fixed in GitLab 14.3 and above, try upgrading GitLab first. + When setting both `inplace_chroot` and `access_control` to `true`, you might encounter errors like: ```plaintext @@ -1260,6 +1233,14 @@ For example, you can adapt the `rsync` strategy from the [moving repositories documentation](../operations/moving_repositories.md). Alternatively, run the CI pipelines of those projects that contain a `pages` job again. +## 404 or 500 error when accessing GitLab Pages in a Geo setup + +Pages sites are only available on the primary Geo site, while the codebase of the project is available on all sites. + +If you try to access a Pages page on a secondary site, you will get a 404 or 500 HTTP code depending on the access control. + +Read more which [features don't support Geo replication/verification](../geo/replication/datatypes.md#limitations-on-replicationverification). + ### Failed to connect to the internal GitLab API If you see the following error: @@ -1297,7 +1278,7 @@ in all of your GitLab Pages instances. ### 500 error with `securecookie: failed to generate random iv` and `Failed to save the session` -This problem most likely results from an [out-dated operating system](https://docs.gitlab.com/omnibus/package-information/deprecated_os.html). +This problem most likely results from an [out-dated operating system](../package_information/deprecated_os.md). The [Pages daemon uses the `securecookie` library](https://gitlab.com/search?group_id=9970&project_id=734943&repository_ref=master&scope=blobs&search=securecookie&snippets=false) to get random strings via [`crypto/rand` in Go](https://golang.org/pkg/crypto/rand/#pkg-variables). This requires the `getrandom` system call or `/dev/urandom` to be available on the host OS. Upgrading to an [officially supported operating system](https://about.gitlab.com/install/) is recommended. @@ -1306,7 +1287,7 @@ Upgrading to an [officially supported operating system](https://about.gitlab.com This problem comes from the permissions of the GitLab Pages OAuth application. To fix it: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Applications > GitLab Pages**. 1. Edit the application. 1. Under **Scopes**, ensure that the `api` scope is selected. @@ -1391,17 +1372,12 @@ both servers. GitLab 14.0 introduces a number of changes to GitLab Pages which may require manual intervention. -1. Firstly [follow the migration guide](#migrate-gitlab-pages-to-140). +1. Firstly [follow the migration guide](#prepare-gitlab-pages-for-140). +1. Try to upgrade to GitLab 14.3 or above. Some of the issues were fixed in GitLab 14.1, 14.2 and 14.3. 1. If it doesn't work, see [GitLab Pages logs](#how-to-see-gitlab-pages-logs), and if you see any errors there then search them on this page. -The most common problem is when using [`inplace_chroot`](#dial-tcp-lookup-gitlabexamplecom-and-x509-certificate-signed-by-unknown-authority). - -NOTE: -Starting from 14.1, the chroot/jailing mechanism is -[disabled by default for API-based configuration](#jailing-mechanism-disabled-by-default-for-api-based-configuration). - WARNING: -As the last resort you can temporarily enable legacy storage and configuration mechanisms. Support for them [will be removed in GitLab 14.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6166), so GitLab Pages will stop working if don't resolve the underlying issue. +In GitLab 14.0-14.2 you can temporarily enable legacy storage and configuration mechanisms. To do that: @@ -1414,3 +1390,25 @@ To do that: ``` 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +### GitLab Pages fails to start in Docker container + +WARNING: +This issue is fixed in GitLab 14.3 and above, try upgrading GitLab first. + +The GitLab Pages daemon doesn't have permissions to bind mounts when it runs +in a Docker container. To overcome this issue, you must change the `chroot` +behavior: + +1. Edit `/etc/gitlab/gitlab.rb`. +1. Set the `inplace_chroot` to `true` for GitLab Pages: + + ```ruby + gitlab_pages['inplace_chroot'] = true + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +NOTE: +`inplace_chroot` option might not work with the other features, such as [Pages Access Control](#access-control). +The [GitLab Pages README](https://gitlab.com/gitlab-org/gitlab-pages#caveats) has more information about caveats and workarounds. diff --git a/doc/administration/pages/source.md b/doc/administration/pages/source.md index b5c3330b2ce..278d792052a 100644 --- a/doc/administration/pages/source.md +++ b/doc/administration/pages/source.md @@ -473,7 +473,7 @@ The default for the maximum size of unpacked archives per project is 100 MB. To change this value: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Preferences**. 1. Expand **Pages**. 1. Update the value for **Maximum size of pages (MB)**. diff --git a/doc/administration/polling.md b/doc/administration/polling.md index 5c4ee837057..8bd28824f83 100644 --- a/doc/administration/polling.md +++ b/doc/administration/polling.md @@ -26,7 +26,7 @@ The default value (`1`) is recommended for the majority of GitLab installations. To adjust the polling interval multiplier: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Preferences**. 1. Expand **Polling interval multiplier**. 1. Set a value for the polling interval multiplier. This multiplier is applied to all resources at diff --git a/doc/administration/postgresql/pgbouncer.md b/doc/administration/postgresql/pgbouncer.md index 7171e90949e..e215622bbc7 100644 --- a/doc/administration/postgresql/pgbouncer.md +++ b/doc/administration/postgresql/pgbouncer.md @@ -173,7 +173,7 @@ ote_pid | tls Some database changes have to be done directly, and not through PgBouncer. Read more about the affected tasks: [database restores](../../raketasks/backup_restore.md#backup-and-restore-for-installations-using-pgbouncer) -and [GitLab upgrades](https://docs.gitlab.com/omnibus/update/README.html#use-postgresql-ha). +and [GitLab upgrades](../../update/zero_downtime.md#use-postgresql-ha). 1. To find the primary node, run the following on a database node: diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index 1308697c16e..2e0820b69c9 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -72,13 +72,13 @@ the PgBouncer service. ### Connection flow -Each service in the package comes with a set of [default ports](https://docs.gitlab.com/omnibus/package-information/defaults.html#ports). You may need to make specific firewall rules for the connections listed below: +Each service in the package comes with a set of [default ports](../package_information/defaults.md#ports). You may need to make specific firewall rules for the connections listed below: -- Application servers connect to either PgBouncer directly via its [default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#pgbouncer) or via a configured Internal Load Balancer (TCP) that serves multiple PgBouncers. -- PgBouncer connects to the primary database servers [PostgreSQL default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#postgresql) +- Application servers connect to either PgBouncer directly via its [default port](../package_information/defaults.md) or via a configured Internal Load Balancer (TCP) that serves multiple PgBouncers. +- PgBouncer connects to the primary database servers [PostgreSQL default port](../package_information/defaults.md) - Patroni actively manages the running PostgreSQL processes and configuration. -- PostgreSQL secondaries connect to the primary database servers [PostgreSQL default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#postgresql) -- Consul servers and agents connect to each others [Consul default ports](https://docs.gitlab.com/omnibus/package-information/defaults.html#consul) +- PostgreSQL secondaries connect to the primary database servers [PostgreSQL default port](../package_information/defaults.md) +- Consul servers and agents connect to each others [Consul default ports](../package_information/defaults.md) ## Setting it up @@ -183,7 +183,7 @@ Few things to remember about the service itself: - The service runs as the same system account as the database - In the package, this is by default `gitlab-psql` -- If you use a non-default user account for PgBouncer service (by default `pgbouncer`), you need to specify this username. +- If you use a non-default user account for PgBouncer service (by default `pgbouncer`), you need to specify this username. - Passwords are stored in the following locations: - `/etc/gitlab/gitlab.rb`: hashed, and in plain text - `/var/opt/gitlab/pgbouncer/pg_auth`: hashed @@ -306,7 +306,7 @@ If you enable Monitoring, it must be enabled on **all** database servers. #### Enable TLS support for the Patroni API By default, Patroni's [REST API](https://patroni.readthedocs.io/en/latest/rest_api.html#rest-api) is served over HTTP. -You have the option to enable TLS and use HTTPS over the same [port](https://docs.gitlab.com/omnibus/package-information/defaults.html#patroni). +You have the option to enable TLS and use HTTPS over the same [port](../package_information/defaults.md). To enable TLS, you need PEM-formatted certificate and private key files. Both files must be readable by the PostgreSQL user (`gitlab-psql` by default, or the one set by `postgresql['username']`): @@ -789,7 +789,7 @@ You do not need any special consideration for Patroni while provisioning your da Patroni monitors the cluster and handles any failover. When the primary node fails it works with Consul to notify PgBouncer. On failure, Patroni handles the transitioning of the old primary to a replica and rejoins it to the cluster automatically. -With Patroni, the connection flow is slightly different. Patroni on each node connects to Consul agent to join the cluster. Only after this point it decides if the node is the primary or a replica. Based on this decision, it configures and starts PostgreSQL which it communicates with directly over a Unix socket. This means that if the Consul cluster is not functional or does not have a leader, Patroni and by extension PostgreSQL does not start. Patroni also exposes a REST API which can be accessed via its [default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#patroni) +With Patroni, the connection flow is slightly different. Patroni on each node connects to Consul agent to join the cluster. Only after this point it decides if the node is the primary or a replica. Based on this decision, it configures and starts PostgreSQL which it communicates with directly over a Unix socket. This means that if the Consul cluster is not functional or does not have a leader, Patroni and by extension PostgreSQL does not start. Patroni also exposes a REST API which can be accessed via its [default port](../package_information/defaults.md) on each node. ### Check replication status diff --git a/doc/administration/pseudonymizer.md b/doc/administration/pseudonymizer.md index 533ebe0ad2f..da3a2e4b34c 100644 --- a/doc/administration/pseudonymizer.md +++ b/doc/administration/pseudonymizer.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Pseudonymizer **(ULTIMATE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/5532) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.1. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/5532) in GitLab 11.1. As the GitLab database hosts sensitive information, using it unfiltered for analytics implies high security requirements. To help alleviate this constraint, the Pseudonymizer diff --git a/doc/administration/raketasks/check.md b/doc/administration/raketasks/check.md index 56bf711f187..7cd7ecc26f7 100644 --- a/doc/administration/raketasks/check.md +++ b/doc/administration/raketasks/check.md @@ -204,12 +204,20 @@ See [LDAP Rake Tasks - LDAP Check](ldap.md#check) for details. The following are solutions to problems you might discover using the Rake tasks documented above. -### Dangling commits +### Dangling objects -`gitlab:git:fsck` can find dangling commits. To fix them, try -[enabling housekeeping](../housekeeping.md). +The `gitlab:git:fsck` task can find dangling objects such as: -If the issue persists, try triggering `gc` via the +```plaintext +dangling blob a12... +dangling commit b34... +dangling tag c56... +dangling tree d78... +``` + +To delete them, try [running housekeeping](../housekeeping.md). + +If the issue persists, try triggering garbage collection via the [Rails Console](../operations/rails_console.md#starting-a-rails-console-session): ```ruby @@ -217,6 +225,13 @@ p = Project.find_by_path("project-name") Repositories::HousekeepingService.new(p, :gc).execute ``` +If the dangling objects are younger than the 2 weeks default grace period, +and you don't want to wait until they expire automatically, run: + +```ruby +Repositories::HousekeepingService.new(p, :prune).execute +``` + ### Delete references to missing remote uploads `gitlab-rake gitlab:uploads:check VERBOSE=1` detects remote objects that do not exist because they were @@ -271,7 +286,7 @@ To delete these references to missing local artifacts (`job.log` files): ```ruby artifacts_deleted = 0 - ::Ci::JobArtifact.all.each do |artifact| ### Iterate artifacts + ::Ci::JobArtifact.find_each do |artifact| ### Iterate artifacts # next if artifact.file.filename != "job.log" ### Uncomment if only `job.log` files' references are to be processed next if artifact.file.exists? ### Skip if the file reference is valid artifacts_deleted += 1 diff --git a/doc/administration/raketasks/github_import.md b/doc/administration/raketasks/github_import.md index 0338732e886..f29e2a6c7f6 100644 --- a/doc/administration/raketasks/github_import.md +++ b/doc/administration/raketasks/github_import.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GitHub import **(FREE SELF)** -> [Introduced]( https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10308) in GitLab 9.1. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10308) in GitLab 9.1. To retrieve and import GitHub repositories, you need a [GitHub personal access token](https://github.com/settings/tokens). A username should be passed as the second argument to the Rake task, diff --git a/doc/administration/raketasks/ldap.md b/doc/administration/raketasks/ldap.md index d7a37d1df3a..585d254e41d 100644 --- a/doc/administration/raketasks/ldap.md +++ b/doc/administration/raketasks/ldap.md @@ -44,7 +44,7 @@ waiting for the next scheduled group sync to be run. NOTE: If you'd like to change the frequency at which a group sync is performed, -[adjust the cron schedule](../auth/ldap/index.md#adjusting-ldap-group-sync-schedule) +[adjust the cron schedule](../auth/ldap/index.md#adjust-ldap-group-sync-schedule) instead. **Omnibus Installation** @@ -151,7 +151,8 @@ sudo gitlab-rake gitlab:ldap:rename_provider[old_provider,new_provider] force=ye ## Secrets -GitLab can use [LDAP configuration secrets](../auth/ldap/index.md#using-encrypted-credentials) to read from an encrypted file. The following Rake tasks are provided for updating the contents of the encrypted file. +GitLab can use [LDAP configuration secrets](../auth/ldap/index.md#use-encrypted-credentials) to read from an encrypted file. +The following Rake tasks are provided for updating the contents of the encrypted file. ### Show secret diff --git a/doc/administration/raketasks/praefect.md b/doc/administration/raketasks/praefect.md index 5fe0546999b..d2fd4943c68 100644 --- a/doc/administration/raketasks/praefect.md +++ b/doc/administration/raketasks/praefect.md @@ -7,7 +7,7 @@ type: reference # Praefect Rake tasks **(FREE SELF)** -> [Introduced]( https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28369) in GitLab 12.10. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28369) in GitLab 12.10. Rake tasks are available for projects that have been created on Praefect storage. See the [Praefect documentation](../gitaly/praefect.md) for information on configuring Praefect. diff --git a/doc/administration/raketasks/project_import_export.md b/doc/administration/raketasks/project_import_export.md index 80321d75d66..e0ca7bfdeaf 100644 --- a/doc/administration/raketasks/project_import_export.md +++ b/doc/administration/raketasks/project_import_export.md @@ -52,7 +52,7 @@ Note the following: compatible as described in the [Version history](../../user/project/settings/import_export.md#version-history). - The project import option must be enabled: - 1. On the top bar, select **Menu >** **{admin}** **Admin**. + 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > General**. 1. Expand **Visibility and access controls**. 1. Under **Import sources**, check the "Project export enabled" option. diff --git a/doc/administration/raketasks/storage.md b/doc/administration/raketasks/storage.md index cee63a6cae5..017565e1b39 100644 --- a/doc/administration/raketasks/storage.md +++ b/doc/administration/raketasks/storage.md @@ -109,7 +109,7 @@ sudo gitlab-rake gitlab:storage:migrate_to_hashed ID_FROM=50 ID_TO=100 To monitor the progress in GitLab: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. Watch how long the `hashed_storage:hashed_storage_project_migrate` queue will take to finish. After it reaches zero, you can confirm every project diff --git a/doc/administration/redis/replication_and_failover_external.md b/doc/administration/redis/replication_and_failover_external.md index 234b1aa7fb9..c19e42a5f14 100644 --- a/doc/administration/redis/replication_and_failover_external.md +++ b/doc/administration/redis/replication_and_failover_external.md @@ -349,7 +349,7 @@ or a failover promotes a different **Primary** node. ```yaml production: - url: redis://:redi-password-goes-here@gitlab-redis/ + url: redis://:redis-password-goes-here@gitlab-redis/ sentinels: - host: 10.0.0.1 diff --git a/doc/administration/redis/troubleshooting.md b/doc/administration/redis/troubleshooting.md index 0c1046ca22d..6ab3d55e06a 100644 --- a/doc/administration/redis/troubleshooting.md +++ b/doc/administration/redis/troubleshooting.md @@ -73,7 +73,7 @@ there may be something wrong with your configuration files or it can be related to [this issue](https://github.com/redis/redis-rb/issues/531). You must make sure you are defining the same value in `redis['master_name']` -and `redis['master_pasword']` as you defined for your sentinel node. +and `redis['master_password']` as you defined for your sentinel node. The way the Redis connector `redis-rb` works with sentinel is a bit non-intuitive. We try to hide the complexity in omnibus, but it still requires diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index f9397e6dbca..0fd597e6a2d 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -1550,7 +1550,7 @@ To configure the Praefect nodes, on each one: # Configure the Consul agent consul['enable'] = true ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true + consul['monitoring_service_discovery'] = true # START user configuration # Please set the real values as explained in Required Information section @@ -2390,7 +2390,7 @@ The following tables and diagram detail the hybrid environment using the same fo as the normal environment above. First are the components that run in Kubernetes. The recommendation at this time is to -use Google Cloud’s Kubernetes Engine (GKE) and associated machine types, but the memory +use Google Cloud's Kubernetes Engine (GKE) and associated machine types, but the memory and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. diff --git a/doc/administration/reference_architectures/1k_users.md b/doc/administration/reference_architectures/1k_users.md index 18e34711953..ea40e150e58 100644 --- a/doc/administration/reference_architectures/1k_users.md +++ b/doc/administration/reference_architectures/1k_users.md @@ -13,7 +13,7 @@ full list of reference architectures, see If you need to serve up to 1,000 users and you don't have strict availability requirements, a single-node solution with [frequent backups](index.md#automated-backups) is appropriate for -many organizations . +many organizations. > - **Supported users (approximate):** 1,000 > - **High Availability:** No. For a highly-available environment, you can diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 65d59422da8..f500434d75b 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -2402,7 +2402,7 @@ The following tables and diagram detail the hybrid environment using the same fo as the normal environment above. First are the components that run in Kubernetes. The recommendation at this time is to -use Google Cloud’s Kubernetes Engine (GKE) and associated machine types, but the memory +use Google Cloud's Kubernetes Engine (GKE) and associated machine types, but the memory and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index 0af4dbc8a7f..99dd29c3a83 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -995,7 +995,7 @@ The following tables and diagram detail the hybrid environment using the same fo as the normal environment above. First are the components that run in Kubernetes. The recommendation at this time is to -use Google Cloud’s Kubernetes Engine (GKE) and associated machine types, but the memory +use Google Cloud's Kubernetes Engine (GKE) and associated machine types, but the memory and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index f4ae01c7442..da36968f053 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -2124,7 +2124,7 @@ The following tables and diagram detail the hybrid environment using the same fo as the normal environment above. First are the components that run in Kubernetes. The recommendation at this time is to -use Google Cloud’s Kubernetes Engine (GKE) and associated machine types, but the memory +use Google Cloud's Kubernetes Engine (GKE) and associated machine types, but the memory and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index b262545b27d..b071b48cbd9 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -2413,7 +2413,7 @@ The following tables and diagram detail the hybrid environment using the same fo as the normal environment above. First are the components that run in Kubernetes. The recommendation at this time is to -use Google Cloud’s Kubernetes Engine (GKE) and associated machine types, but the memory +use Google Cloud's Kubernetes Engine (GKE) and associated machine types, but the memory and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 666d18a66fc..4dfe628039a 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -2094,7 +2094,7 @@ The following tables and diagram detail the hybrid environment using the same fo as the normal environment above. First are the components that run in Kubernetes. The recommendation at this time is to -use Google Cloud’s Kubernetes Engine (GKE) and associated machine types, but the memory +use Google Cloud's Kubernetes Engine (GKE) and associated machine types, but the memory and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index 74d8bf39d03..4d95a61176b 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -139,8 +139,8 @@ to any of the [available reference architectures](#available-reference-architect > - Level of complexity: **Medium** > - Required domain knowledge: PostgreSQL, HAProxy, shared storage, distributed systems -GitLab supports [zero-downtime updates](https://docs.gitlab.com/omnibus/update/#zero-downtime-updates). -Single GitLab nodes can be updated with only a [few minutes of downtime](https://docs.gitlab.com/omnibus/update/README.html#single-node-deployment). +GitLab supports [zero-downtime upgrades](../../update/zero_downtime.md). +Single GitLab nodes can be updated with only a [few minutes of downtime](../../update/zero_downtime.md#single-node-deployment). To avoid this, we recommend to separate GitLab into several application nodes. As long as at least one of each component is online and capable of handling the instance's usage load, your team's productivity will not be interrupted during the update. diff --git a/doc/administration/reference_architectures/troubleshooting.md b/doc/administration/reference_architectures/troubleshooting.md index 61d9dfea2a2..aabf4809b4a 100644 --- a/doc/administration/reference_architectures/troubleshooting.md +++ b/doc/administration/reference_architectures/troubleshooting.md @@ -158,7 +158,7 @@ there may be something wrong with your configuration files or it can be related to [this issue](https://github.com/redis/redis-rb/issues/531). You must make sure you are defining the same value in `redis['master_name']` -and `redis['master_pasword']` as you defined for your sentinel node. +and `redis['master_password']` as you defined for your sentinel node. The way the Redis connector `redis-rb` works with sentinel is a bit non-intuitive. We try to hide the complexity in omnibus, but it still requires diff --git a/doc/administration/repository_checks.md b/doc/administration/repository_checks.md index ab203bb7993..0591f5b3c40 100644 --- a/doc/administration/repository_checks.md +++ b/doc/administration/repository_checks.md @@ -11,7 +11,7 @@ You can use [`git fsck`](https://git-scm.com/docs/git-fsck) to verify the integr committed to a repository. GitLab administrators can trigger this check for a project using the GitLab UI: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Overview > Projects**. 1. Select the project to check. 1. In the **Repository check** section, select **Trigger repository check**. @@ -25,7 +25,7 @@ This setting is off by default, because it can cause many false alarms. Instead of checking repositories manually, GitLab can be configured to run the checks periodically: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Repository** (`/admin/application_settings/repository`). 1. Expand the **Repository maintenance** section. 1. Enable **Enable repository checks**. @@ -50,7 +50,7 @@ disk at: If periodic repository checks cause false alarms, you can clear all repository check states: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Repository** (`/admin/application_settings/repository`). 1. Expand the **Repository maintenance** section. 1. Select **Clear all repository checks**. diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md index f4aed0f6a0f..e7edfb9f338 100644 --- a/doc/administration/repository_storage_paths.md +++ b/doc/administration/repository_storage_paths.md @@ -146,7 +146,7 @@ Omnibus stores the repositories in a `repositories` subdirectory of the `git-dat After you [configure](#configure-repository-storage-paths) multiple repository storage paths, you can choose where new repositories are stored: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Repository** and expand the **Repository storage** section. 1. Enter values in the **Storage nodes for new repositories** fields. diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md index f55bff1bf34..d2bab9a1e04 100644 --- a/doc/administration/repository_storage_types.md +++ b/doc/administration/repository_storage_types.md @@ -80,7 +80,7 @@ Administrators can look up a project's hashed path from its name or ID using: To look up a project's hash path in the Admin Area: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Overview > Projects** and select the project. The **Gitaly relative path** is displayed there and looks similar to: diff --git a/doc/administration/static_objects_external_storage.md b/doc/administration/static_objects_external_storage.md index 2f19a2e5058..21949388f19 100644 --- a/doc/administration/static_objects_external_storage.md +++ b/doc/administration/static_objects_external_storage.md @@ -16,8 +16,8 @@ storage such as a content delivery network (CDN). To configure external storage for static objects: -1. On the top bar, select **Menu >** **{admin}** **Admin**. -1. In the left sidebar, select **Settings > Repository**. +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Settings > Repository**. 1. Expand the **External storage for repository static objects** section. 1. Enter the base URL and an arbitrary token. When you [set up external storage](#set-up-external-storage), use a script that sets these values as `ORIGIN_HOSTNAME` and `STORAGE_TOKEN`. diff --git a/doc/administration/troubleshooting/diagnostics_tools.md b/doc/administration/troubleshooting/diagnostics_tools.md index fe85b5d5803..53d4810b920 100644 --- a/doc/administration/troubleshooting/diagnostics_tools.md +++ b/doc/administration/troubleshooting/diagnostics_tools.md @@ -23,8 +23,3 @@ running on. [strace-parser](https://gitlab.com/wchandler/strace-parser) is a small tool to analyze and summarize raw `strace` data. - -## Pritaly - -[Pritaly](https://gitlab.com/wchandler/pritaly) takes Gitaly logs and colorizes output -or converts the logs to JSON. diff --git a/doc/administration/troubleshooting/elasticsearch.md b/doc/administration/troubleshooting/elasticsearch.md index 79295856da8..cfce3b94554 100644 --- a/doc/administration/troubleshooting/elasticsearch.md +++ b/doc/administration/troubleshooting/elasticsearch.md @@ -196,7 +196,7 @@ Troubleshooting search result issues is rather straight forward on Elasticsearch The first step is to confirm GitLab is using Elasticsearch for the search function. To do this: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > General**, and then confirm the integration is enabled. 1. Confirm searches use Elasticsearch by accessing the rails console diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index c55efe78216..491db37d9da 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -243,7 +243,7 @@ project.update!(repository_read_only: true) ### Transfer project from one namespace to another ```ruby - p= Project.find_by_full_path('<project_path>') +p = Project.find_by_full_path('<project_path>') # To set the owner of the project current_user= p.creator @@ -416,9 +416,9 @@ p.create_wiki ### creates the wiki project on the filesystem ### In case of issue boards not loading properly and it's getting time out. We need to call the Issue Rebalancing service to fix this ```ruby -p=Project.find_by_full_path('PROJECT PATH') +p = Project.find_by_full_path('PROJECT PATH') -IssueRebalancingService.new(p.issues.take).execute +Issues::RelativePositionRebalancingService.new(p.root_namespace.all_projects).execute ``` ## Imports / Exports @@ -596,6 +596,17 @@ curl --silent --header "Private-Token: ********************" \ "https://gitlab.example.com/api/v4/users?per_page=100&active" | jq --compact-output '.[] | [.id,.name,.username]' ``` +### Update Daily Billable & Historical users + +```ruby +# Forces recount of historical (max) users +::HistoricalDataWorker.new.perform + +# Forces recount of daily billable users +identifier = Analytics::UsageTrends::Measurement.identifiers[:billable_users] +::Analytics::UsageTrends::CounterJobWorker.new.perform(identifier, User.minimum(:id), User.maximum(:id), Time.zone.now) +``` + ### Block or Delete Users that have no projects or groups ```ruby @@ -999,8 +1010,8 @@ This content has been moved to the [Troubleshooting Sidekiq docs](sidekiq.md). ### Get information about LFS objects and associated project ```ruby -o=LfsObject.find_by(oid: "<oid>") -p=Project.find(LfsObjectsProject.find_by_lfs_object_id(o.id).project_id) +o = LfsObject.find_by(oid: "<oid>") +p = Project.find(LfsObjectsProject.find_by_lfs_object_id(o.id).project_id) ``` You can then delete these records from the database with: diff --git a/doc/administration/troubleshooting/log_parsing.md b/doc/administration/troubleshooting/log_parsing.md index 0a73c61ae94..c5443c564f4 100644 --- a/doc/administration/troubleshooting/log_parsing.md +++ b/doc/administration/troubleshooting/log_parsing.md @@ -55,6 +55,12 @@ zcat @400000006026b71d1a7af804.s | (head -1; tail -1) | jq '.time' zcat some_json.log.25.gz | (head -1; tail -1) | jq '.time' ``` +#### Get activity for correlation ID across multiple JSON logs in chronological order + +```shell +grep -hR <correlationID> | jq -c -R 'fromjson?' | jq -C -s 'sort_by(.time)' | less -R +``` + ### Parsing `production_json.log` and `api_json.log` #### Find all requests with a 5XX status code diff --git a/doc/administration/troubleshooting/postgresql.md b/doc/administration/troubleshooting/postgresql.md index 994c194c6db..3df957bacf9 100644 --- a/doc/administration/troubleshooting/postgresql.md +++ b/doc/administration/troubleshooting/postgresql.md @@ -116,7 +116,7 @@ References: ERROR: deadlock detected ``` -Three applicable timeouts are identified in the issue [#1](https://gitlab.com/gitlab-org/gitlab/-/issues/30528); our recommended settings are as follows: +Three applicable timeouts are identified in the issue [#30528](https://gitlab.com/gitlab-org/gitlab/-/issues/30528); our recommended settings are as follows: ```ini deadlock_timeout = 5s @@ -124,7 +124,7 @@ statement_timeout = 15s idle_in_transaction_session_timeout = 60s ``` -Quoting from issue [#1](https://gitlab.com/gitlab-org/gitlab/-/issues/30528): +Quoting from issue [#30528](https://gitlab.com/gitlab-org/gitlab/-/issues/30528): > "If a deadlock is hit, and we resolve it through aborting the transaction after a short period, then the retry mechanisms we already have will make the deadlocked piece of work try again, and it's unlikely we'll deadlock multiple times in a row." @@ -146,7 +146,7 @@ PostgresSQL defaults: - `statement_timeout = 0` (never) - `idle_in_transaction_session_timeout = 0` (never) -Comments in issue [#1](https://gitlab.com/gitlab-org/gitlab/-/issues/30528) +Comments in issue [#30528](https://gitlab.com/gitlab-org/gitlab/-/issues/30528) indicate that these should both be set to at least a number of minutes for all Omnibus GitLab installations (so they don't hang indefinitely). However, 15s for statement_timeout is very short, and will only be effective if the diff --git a/doc/administration/troubleshooting/tracing_correlation_id.md b/doc/administration/troubleshooting/tracing_correlation_id.md index 1bb10e72290..3518f52e6f6 100644 --- a/doc/administration/troubleshooting/tracing_correlation_id.md +++ b/doc/administration/troubleshooting/tracing_correlation_id.md @@ -27,7 +27,7 @@ activity with the site that you're visiting. See the links below for network mon documentation for some popular browsers. - [Network Monitor - Firefox Developer Tools](https://developer.mozilla.org/en-US/docs/Tools/Network_Monitor) -- [Inspect Network Activity In Chrome DevTools](https://developer.chrome.com/docs/devtools/network) +- [Inspect Network Activity In Chrome DevTools](https://developer.chrome.com/docs/devtools/network/) - [Safari Web Development Tools](https://developer.apple.com/safari/tools/) - [Microsoft Edge Network panel](https://docs.microsoft.com/en-us/microsoft-edge/devtools-guide-chromium/network/) @@ -127,3 +127,11 @@ some sort of log aggregation software like Loki, ELK, Splunk, or others. You can use a tool like Ansible or PSSH (parallel SSH) that can execute identical commands across your servers in parallel, or craft your own solution. + +### Viewing the request in the Performance Bar + +You can use the [performance bar](../monitoring/performance/performance_bar.md) to view interesting data including calls made to SQL and Gitaly. + +To view the data, the correlation ID of the request must match the same session as the user +viewing the performance bar. For API requests, this means that you must perform the request +using the session cookie of the signed-in user. diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index 949687cfa0a..15ef024647c 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -53,8 +53,8 @@ _The uploads are stored by default in > **Notes:** > -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3867) in [GitLab Premium](https://about.gitlab.com/pricing/) 10.5. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17358) in [GitLab Free](https://about.gitlab.com/pricing/) 10.7. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3867) in GitLab 10.5. +> - [Moved](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17358) from GitLab Premium to GitLab Free in 10.7. > - Since version 11.1, we support direct_upload to S3. If you don't want to use the local disk where GitLab is installed to store the diff --git a/doc/administration/whats-new.md b/doc/administration/whats-new.md index d669d05e9f0..d3768907989 100644 --- a/doc/administration/whats-new.md +++ b/doc/administration/whats-new.md @@ -29,7 +29,7 @@ in the first patch release, such as `13.10.1`. You can configure the What's new variant: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Preferences**, then expand **What's new**. 1. Choose one of the following options: |
