diff options
Diffstat (limited to 'doc/administration')
179 files changed, 1381 insertions, 1532 deletions
diff --git a/doc/administration/application_settings_cache.md b/doc/administration/application_settings_cache.md new file mode 100644 index 00000000000..b6c09129c79 --- /dev/null +++ b/doc/administration/application_settings_cache.md @@ -0,0 +1,47 @@ +--- +stage: Enablement +group: Memory +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Changing application settings cache expiry interval **(FREE SELF)** + +Application settings are cached for 60 seconds by default which should work +for most installations. A higher value would mean a greater delay between +changing an application setting and noticing that change come into effect. +A value of `0` would result in the `application_settings` table being +loaded for every request causing extra load on Redis and/or PostgreSQL. +It is therefore recommended to keep the value above zero. + +## Change the application settings cache expiry + +To change the expiry value: + +**For Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['application_settings_cache_seconds'] = 60 + ``` + +1. Save the file, and reconfigure and restart GitLab for the changes to take effect: + + ```shell + gitlab-ctl reconfigure + gitlab-ctl restart + ``` + +--- + +**For installations from source** + +1. Edit `config/gitlab.yml`: + + ```yaml + gitlab: + application_settings_cache_seconds: 60 + ``` + +1. Save the file and [restart](restart_gitlab.md#installations-from-source) + GitLab for the changes to take effect. diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index 27f6bbcd028..6512879e8b8 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -4,7 +4,7 @@ group: Compliance info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Audit Events **(STARTER)** +# Audit Events **(PREMIUM)** GitLab offers a way to view the changes made within the GitLab server for owners and administrators on a [paid plan](https://about.gitlab.com/pricing/). @@ -42,7 +42,7 @@ There are two kinds of events logged: Impersonation is where an administrator uses credentials to perform an action as a different user. -### Group events **(STARTER)** +### Group events **(PREMIUM)** A user with a Owner role (or above) can retrieve group audit events of all users. A user with a Developer or Maintainer role is limited to group audit events based on their individual actions. @@ -72,7 +72,7 @@ From there, you can see the following actions: Group events can also be accessed via the [Group Audit Events API](../api/audit_events.md#group-audit-events) -### Project events **(STARTER)** +### Project events **(PREMIUM)** A user with a Maintainer role (or above) can retrieve project audit events of all users. A user with a Developer role is limited to project audit events based on their individual actions. @@ -99,13 +99,15 @@ From there, you can see the following actions: - Number of required approvals was updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7531) in GitLab 12.9) - Added or removed users and groups from project approval groups ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213603) in GitLab 13.2) - Project CI/CD variable added, removed, or protected status changed ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30857) in GitLab 13.4) -- User was approved via Admin Area ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276250) in GitLab 13.6) +- Project access token was successfully created or revoked ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230007) in GitLab 13.9) +- Failed attempt to create or revoke a project access token ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230007) in GitLab 13.9) +- When default branch changes for a project ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/52339) in GitLab 13.9) Project events can also be accessed via the [Project Audit Events API](../api/audit_events.md#project-audit-events). Project event queries are limited to a maximum of 30 days. -### Instance events **(PREMIUM ONLY)** +### Instance events **(PREMIUM SELF)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2336) in [GitLab Premium](https://about.gitlab.com/pricing/) 9.3. @@ -129,6 +131,9 @@ recorded: - Changed username ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7797) in GitLab 12.8) - User was deleted ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/251) in GitLab 12.8) - User was added ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/251) in GitLab 12.8) +- User requests access to an instance ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/298783) in GitLab 13.9) +- User was approved via Admin Area ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276250) in GitLab 13.6) +- User was rejected via Admin Area ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/298783) in GitLab 13.9) - User was blocked via Admin Area ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/251) in GitLab 12.8) - User was blocked via API ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25872) in GitLab 12.9) - Failed second-factor authentication attempt ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16826) in GitLab 13.5) @@ -189,7 +194,7 @@ The search filters you can see depends on which audit level you are at.  -## Export to CSV **(PREMIUM ONLY)** +## Export to CSV **(PREMIUM SELF)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1449) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.4. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/285441) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.7. diff --git a/doc/administration/auditor_users.md b/doc/administration/auditor_users.md index 7db20efb03f..b04ceebf645 100644 --- a/doc/administration/auditor_users.md +++ b/doc/administration/auditor_users.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Auditor users **(PREMIUM ONLY)** +# Auditor users **(PREMIUM SELF)** Auditor users are given read-only access to all projects, groups, and other resources on the GitLab instance. @@ -68,6 +68,8 @@ To create a new Auditor user: To revoke Auditor permissions from a user, make them a regular user by following the previous steps. +Additionally users can be set as an Auditor using [SAML groups](../integration/saml.md#auditor-groups). + ## Permissions and restrictions of an Auditor user An Auditor user should be able to access all projects and groups of a GitLab diff --git a/doc/administration/auth/README.md b/doc/administration/auth/README.md index cc3421d3133..69220113940 100644 --- a/doc/administration/auth/README.md +++ b/doc/administration/auth/README.md @@ -22,7 +22,8 @@ providers: - [Facebook](../../integration/facebook.md) - [GitHub](../../integration/github.md) - [GitLab.com](../../integration/gitlab.md) -- [Google](../../integration/google.md) +- [Google OAuth](../../integration/google.md) +- [Google Workspace SSO](../../integration/google_workspace_saml.md) - [JWT](jwt.md) - [Kerberos](../../integration/kerberos.md) - [LDAP](ldap/index.md): Includes Active Directory, Apple Open Directory, Open LDAP, @@ -31,9 +32,9 @@ providers: - [Okta](okta.md) - [Salesforce](../../integration/salesforce.md) - [SAML](../../integration/saml.md) -- [SAML for GitLab.com groups](../../user/group/saml_sso/index.md) **(SILVER ONLY)** +- [SAML for GitLab.com groups](../../user/group/saml_sso/index.md) **(PREMIUM SAAS)** - [Shibboleth](../../integration/shibboleth.md) -- [Smartcard](smartcard.md) **(PREMIUM ONLY)** +- [Smartcard](smartcard.md) **(PREMIUM SELF)** - [Twitter](../../integration/twitter.md) NOTE: diff --git a/doc/administration/auth/google_secure_ldap.md b/doc/administration/auth/google_secure_ldap.md deleted file mode 100644 index 37366b00f73..00000000000 --- a/doc/administration/auth/google_secure_ldap.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'ldap/google_secure_ldap.md' ---- - -This document was moved to [another location](ldap/google_secure_ldap.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md b/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md deleted file mode 100644 index ffce06afb63..00000000000 --- a/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: '../ldap/index.md' ---- - -This document was moved to [another location](../ldap/index.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/index.md b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/index.md deleted file mode 100644 index ffce06afb63..00000000000 --- a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/index.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: '../ldap/index.md' ---- - -This document was moved to [another location](../ldap/index.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/auth/img/okta_admin_panel_v13_9.png b/doc/administration/auth/img/okta_admin_panel_v13_9.png Binary files differnew file mode 100644 index 00000000000..bba859c80af --- /dev/null +++ b/doc/administration/auth/img/okta_admin_panel_v13_9.png diff --git a/doc/administration/auth/ldap-ee.md b/doc/administration/auth/ldap-ee.md deleted file mode 100644 index 6d56654a44b..00000000000 --- a/doc/administration/auth/ldap-ee.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'ldap/index.md' ---- - -This document was moved to [another location](ldap/index.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/auth/ldap-troubleshooting.md b/doc/administration/auth/ldap-troubleshooting.md deleted file mode 100644 index 1e02755e3e5..00000000000 --- a/doc/administration/auth/ldap-troubleshooting.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'ldap/ldap-troubleshooting.md' ---- - -This document was moved to [another location](ldap/ldap-troubleshooting.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/auth/ldap.md b/doc/administration/auth/ldap.md deleted file mode 100644 index 6d56654a44b..00000000000 --- a/doc/administration/auth/ldap.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'ldap/index.md' ---- - -This document was moved to [another location](ldap/index.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/auth/ldap/google_secure_ldap.md b/doc/administration/auth/ldap/google_secure_ldap.md index 6fecf74d935..2b75d864352 100644 --- a/doc/administration/auth/ldap/google_secure_ldap.md +++ b/doc/administration/auth/ldap/google_secure_ldap.md @@ -5,7 +5,7 @@ group: Access info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Google Secure LDAP **(CORE ONLY)** +# Google Secure LDAP **(FREE SELF)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/46391) in GitLab 11.9. diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index de0f123acf1..74621f9c1ba 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -29,7 +29,7 @@ stands for **Lightweight Directory Access Protocol**, which is a standard application protocol for accessing and maintaining distributed directory information services over an Internet Protocol (IP) network. -## Security **(CORE ONLY)** +## Security **(FREE SELF)** GitLab assumes that LDAP users: @@ -44,7 +44,7 @@ We recommend against using LDAP integration if your LDAP users are allowed to change their 'mail', 'email' or 'userPrincipalName' attribute on the LDAP server or share email addresses. -### User deletion **(CORE ONLY)** +### User deletion **(FREE SELF)** If a user is deleted from the LDAP server, they are also blocked in GitLab. Users are immediately blocked from logging in. However, there is an @@ -53,16 +53,16 @@ are already logged in or are using Git over SSH are be able to access GitLab for up to one hour. Manually block the user in the GitLab Admin Area to immediately block all access. -GitLab Enterprise Edition Starter supports a -[configurable sync time](#adjusting-ldap-user-sync-schedule). **(STARTER)** +GitLab Enterprise Edition Premium supports a +[configurable sync time](#adjusting-ldap-user-sync-schedule). **(PREMIUM)** -## Git password authentication **(CORE ONLY)** +## Git password authentication **(FREE SELF)** LDAP-enabled users can always authenticate with Git using their GitLab username or email and LDAP password, even if password authentication for Git is disabled in the application settings. -## Enabling LDAP sign-in for existing GitLab users **(CORE ONLY)** +## Enabling LDAP sign-in for existing GitLab users **(FREE SELF)** When a user signs in to GitLab with LDAP for the first time, and their LDAP email address is the primary email address of an existing GitLab user, then @@ -73,7 +73,7 @@ In other words, if an existing GitLab user wants to enable LDAP sign-in for themselves, they should check that their GitLab email address matches their LDAP email address, and then sign into GitLab via their LDAP credentials. -## Google Secure LDAP **(CORE ONLY)** +## Google Secure LDAP **(FREE SELF)** > Introduced in GitLab 11.9. @@ -81,7 +81,7 @@ LDAP email address, and then sign into GitLab via their LDAP credentials. LDAP service that can be configured with GitLab for authentication and group sync. See [Google Secure LDAP](google_secure_ldap.md) for detailed configuration instructions. -## Configuration **(CORE ONLY)** +## Configuration **(FREE SELF)** To enable LDAP integration you need to add your LDAP server settings in `/etc/gitlab/gitlab.rb` or `/home/git/gitlab/config/gitlab.yml` for Omnibus @@ -100,7 +100,7 @@ would be on port 389. `plain` also operates on port 389. Removed values: `tls` w LDAP users must have a set email address, regardless of whether or not it's used to sign in. -### Example Configurations **(CORE ONLY)** +### Example Configurations **(FREE SELF)** **Omnibus Configuration** @@ -163,7 +163,7 @@ production: ... ``` -### Basic Configuration Settings **(CORE ONLY)** +### Basic Configuration Settings **(FREE SELF)** | Setting | Description | Required | Examples | | ------- | ----------- | -------- | -------- | @@ -183,7 +183,7 @@ production: | `user_filter` | Filter LDAP users. Format: [RFC 4515](https://tools.ietf.org/search/rfc4515) Note: GitLab does not support `omniauth-ldap`'s custom filter syntax. | no | `'(employeeType=developer)'` or `'(&(objectclass=user)(|(samaccountname=momo)(samaccountname=toto)))'` | | `lowercase_usernames` | If lowercase_usernames is enabled, GitLab converts the name to lower case. | no | boolean | -### SSL Configuration Settings **(CORE ONLY)** +### SSL Configuration Settings **(FREE SELF)** | Setting | Description | Required | Examples | | ------- | ----------- | -------- | -------- | @@ -193,7 +193,7 @@ production: | `cert` | Client certificate | no | `'-----BEGIN CERTIFICATE----- <REDACTED> -----END CERTIFICATE -----'` | | `key` | Client private key | no | `'-----BEGIN PRIVATE KEY----- <REDACTED> -----END PRIVATE KEY -----'` | -### Attribute Configuration Settings **(CORE ONLY)** +### Attribute Configuration Settings **(FREE SELF)** LDAP attributes that GitLab uses to create an account for the LDAP user. The specified attribute can either be the attribute name as a string (for example, `'mail'`), or an array of attribute names to try in order (for example, `['mail', 'email']`). Note that the user's LDAP sign-in is the attribute specified as `uid` above. @@ -205,7 +205,7 @@ LDAP attributes that GitLab uses to create an account for the LDAP user. The spe | `first_name` | LDAP attribute for user first name. Used when the attribute configured for `name` does not exist. | no | `'givenName'` | | `last_name` | LDAP attribute for user last name. Used when the attribute configured for `name` does not exist. | no | `'sn'` | -### LDAP Sync Configuration Settings **(STARTER ONLY)** +### LDAP Sync Configuration Settings **(PREMIUM SELF)** | Setting | Description | Required | Examples | | ------- | ----------- | -------- | -------- | @@ -214,7 +214,7 @@ LDAP attributes that GitLab uses to create an account for the LDAP user. The spe | `external_groups` | An array of CNs of groups containing users that should be considered external. Note: Not `cn=interns` or the full DN. | no | `['interns', 'contractors']` | | `sync_ssh_keys` | The LDAP attribute containing a user's public SSH key. | no | `'sshPublicKey'` or false if not set | -### Set up LDAP user filter **(CORE ONLY)** +### Set up LDAP user filter **(FREE SELF)** If you want to limit all GitLab access to a subset of the LDAP users on your LDAP server, the first step should be to narrow the configured `base`. However, @@ -254,12 +254,12 @@ group, you can use the following syntax: For more information about this "LDAP_MATCHING_RULE_IN_CHAIN" filter, see the following [Microsoft Search Filter Syntax](https://docs.microsoft.com/en-us/windows/win32/adsi/search-filter-syntax) document. Support for nested members in the user filter should not be confused with -[group sync nested groups support](#supported-ldap-group-typesattributes). **(STARTER ONLY)** +[group sync nested groups support](#supported-ldap-group-typesattributes). **(PREMIUM SELF)** Please note that GitLab does not support the custom filter syntax used by OmniAuth LDAP. -#### Escaping special characters **(CORE ONLY)** +#### Escaping special characters **(FREE SELF)** The `user_filter` DN can contain special characters. For example: @@ -290,7 +290,7 @@ The `user_filter` DN can contain special characters. For example: OU=Gitlab \28Inc\29,DC=gitlab,DC=com ``` -### Enabling LDAP username lowercase **(CORE ONLY)** +### Enabling LDAP username lowercase **(FREE SELF)** Some LDAP servers, depending on their configurations, can return uppercase usernames. This can lead to several confusing issues such as creating links or namespaces with uppercase names. @@ -328,7 +328,7 @@ the configuration option `lowercase_usernames`. By default, this configuration o 1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. -### Disable LDAP web sign in **(CORE ONLY)** +### Disable LDAP web sign in **(FREE SELF)** It can be useful to prevent using LDAP credentials through the web UI when an alternative such as SAML is preferred. This allows LDAP to be used for group @@ -360,7 +360,7 @@ This does not disable [using LDAP credentials for Git access](#git-password-auth 1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. -### Using encrypted credentials **(CORE ONLY)** +### Using encrypted credentials **(FREE SELF)** Instead of having the LDAP integration credentials stored in plaintext in the configuration files, you can optionally use an encrypted file for the LDAP credentials. To use this feature, you first need to enable @@ -447,7 +447,7 @@ If initially your LDAP configuration looked like: 1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. -## Encryption **(CORE ONLY)** +## Encryption **(FREE SELF)** ### TLS Server Authentication @@ -467,7 +467,7 @@ You should disable anonymous LDAP authentication and enable simple or SASL authentication. The TLS client authentication setting in your LDAP server cannot be mandatory and clients cannot be authenticated with the TLS protocol. -## Multiple LDAP servers **(STARTER ONLY)** +## Multiple LDAP servers **(PREMIUM SELF)** With GitLab Enterprise Edition Starter, you can configure multiple LDAP servers that your GitLab instance connects to. @@ -515,7 +515,7 @@ gitlab_rails['ldap_servers'] = { If you configure multiple LDAP servers, use a unique naming convention for the `label` section of each entry. That label is used as the display name of the tab shown on the sign-in page. -## User sync **(STARTER ONLY)** +## User sync **(PREMIUM SELF)** Once per day, GitLab runs a worker to check and update GitLab users against LDAP. @@ -530,7 +530,12 @@ The process executes the following access checks: In Active Directory, a user is marked as disabled/blocked if the user account control attribute (`userAccountControl:1.2.840.113556.1.4.803`) has bit 2 set. -For more information, see <https://ctovswild.com/2009/09/03/bitmask-searches-in-ldap/> + +<!-- vale gitlab.Spelling = NO --> + +For more information, see [Bitmask Searches in LDAP](https://ctovswild.com/2009/09/03/bitmask-searches-in-ldap/). + +<!-- vale gitlab.Spelling = YES --> The user is set to an `ldap_blocked` state in GitLab if the previous conditions fail. This means the user is not able to sign in or push/pull code. @@ -546,7 +551,7 @@ The LDAP sync process: - Updates existing users. - Creates new users on first sign in. -### Adjusting LDAP user sync schedule **(STARTER ONLY)** +### Adjusting LDAP user sync schedule **(PREMIUM SELF)** By default, GitLab runs a worker once per day at 01:30 a.m. server time to check and update GitLab users against LDAP. @@ -579,7 +584,7 @@ sync to run once every 12 hours at the top of the hour. 1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. -## Group Sync **(STARTER ONLY)** +## Group Sync **(PREMIUM SELF)** If your LDAP supports the `memberof` property, when the user signs in for the first time GitLab triggers a sync for groups the user should be a member of. @@ -629,11 +634,11 @@ following. To take advantage of group sync, group owners or maintainers need to [create one or more LDAP group links](#adding-group-links). -### Adding group links **(STARTER ONLY)** +### Adding group links **(PREMIUM SELF)** For information on adding group links via CNs and filters, refer to [the GitLab groups documentation](../../../user/group/index.md#manage-group-memberships-via-ldap). -### Administrator sync **(STARTER ONLY)** +### Administrator sync **(PREMIUM SELF)** As an extension of group sync, you can automatically manage your global GitLab administrators. Specify a group CN for `admin_group` and all members of the @@ -677,7 +682,7 @@ group, as opposed to the full DN. 1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. -### Global group memberships lock **(STARTER ONLY)** +### Global group memberships lock **(PREMIUM SELF)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1793) in GitLab 12.0. @@ -696,7 +701,7 @@ To enable it you need to: 1. Navigate to **(admin)** **Admin Area > Settings -> Visibility and access controls**. 1. Make sure the "Lock memberships to LDAP synchronization" checkbox is enabled. -### Adjusting LDAP group sync schedule **(STARTER ONLY)** +### Adjusting LDAP group sync schedule **(PREMIUM SELF)** By default, GitLab runs a group sync process every hour, on the hour. The values shown are in cron format. If needed, you can use a @@ -735,7 +740,7 @@ sync to run once every 2 hours at the top of the hour. 1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. -### External groups **(STARTER ONLY)** +### External groups **(PREMIUM SELF)** Using the `external_groups` setting will allow you to mark all users belonging to these groups as [external users](../../../user/permissions.md#external-users). diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md index 1976bab03c6..5640e938651 100644 --- a/doc/administration/auth/ldap/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap/ldap-troubleshooting.md @@ -52,7 +52,7 @@ main: # 'main' is the GitLab 'provider ID' of this LDAP server admin_group: 'my_admin_group' ``` -#### Query LDAP **(STARTER ONLY)** +#### Query LDAP **(PREMIUM SELF)** The following allows you to perform a search in LDAP using the rails console. Depending on what you're trying to do, it may make more sense to query [a @@ -210,7 +210,7 @@ ldapsearch -H ldaps://$host:$port -D "$bind_dn" -y bind_dn_password.txt -b "$ba port. - We are assuming the password for the `bind_dn` user is in `bind_dn_password.txt`. -#### Sync all users **(STARTER ONLY)** +#### Sync all users **(PREMIUM SELF)** The output from a manual [user sync](index.md#user-sync) can show you what happens when GitLab tries to sync its users against LDAP. Enter the [rails console](#rails-console) @@ -225,7 +225,7 @@ LdapSyncWorker.new.perform Next, [learn how to read the output](#example-console-output-after-a-user-sync). -##### Example console output after a user sync **(STARTER ONLY)** +##### Example console output after a user sync **(PREMIUM SELF)** The output from a [manual user sync](#sync-all-users) will be very verbose, and a single user's successful sync can look like this: @@ -316,9 +316,9 @@ adapter = Gitlab::Auth::Ldap::Adapter.new('ldapmain') # If `main` is the LDAP pr Gitlab::Auth::Ldap::Person.find_by_uid('<uid>', adapter) ``` -### Group memberships **(STARTER ONLY)** +### Group memberships **(PREMIUM SELF)** -#### Membership(s) not granted **(STARTER ONLY)** +#### Membership(s) not granted **(PREMIUM SELF)** Sometimes you may think a particular user should be added to a GitLab group via LDAP group sync, but for some reason it's not happening. There are several @@ -376,7 +376,7 @@ group sync](#sync-all-groups) in the rails console and [look through the output](#example-console-output-after-a-group-sync) to see what happens when GitLab syncs the `admin_group`. -#### Sync all groups **(STARTER ONLY)** +#### Sync all groups **(PREMIUM SELF)** NOTE: To sync all groups manually when debugging is unnecessary, [use the Rake @@ -394,7 +394,7 @@ LdapAllGroupsSyncWorker.new.perform Next, [learn how to read the output](#example-console-output-after-a-group-sync). -##### Example console output after a group sync **(STARTER ONLY)** +##### Example console output after a group sync **(PREMIUM SELF)** Like the output from the user sync, the output from the [manual group sync](#sync-all-groups) will also be very verbose. However, it contains lots @@ -484,7 +484,7 @@ stating as such: No `admin_group` configured for 'ldapmain' provider. Skipping ``` -#### Sync one group **(STARTER ONLY)** +#### Sync one group **(PREMIUM SELF)** [Syncing all groups](#sync-all-groups) can produce a lot of noise in the output, which can be distracting when you're only interested in troubleshooting the memberships of @@ -506,7 +506,7 @@ EE::Gitlab::Auth::Ldap::Sync::Group.execute_all_providers(group) The output will be similar to [that you'd get from syncing all groups](#example-console-output-after-a-group-sync). -#### Query a group in LDAP **(STARTER ONLY)** +#### Query a group in LDAP **(PREMIUM SELF)** When you'd like to confirm that GitLab can read a LDAP group and see all its members, you can run the following: @@ -562,7 +562,7 @@ emails.each do |username, email| end ``` -You can then [run a UserSync](#sync-all-users) **(STARTER ONLY)** to sync the latest DN +You can then [run a UserSync](#sync-all-users) **(PREMIUM SELF)** to sync the latest DN for each of these users. ## Debugging Tools diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md index 158182edfb5..1ddf75e7c1b 100644 --- a/doc/administration/auth/oidc.md +++ b/doc/administration/auth/oidc.md @@ -175,6 +175,6 @@ If you're having trouble, here are some tips: OAuth2 access token if `client_auth_method` is not defined or if set to `basic`. If you are seeing 401 errors upon retrieving the `userinfo` endpoint, you may want to check your OpenID Web server configuration. For example, for - [oauth2-server-php](https://github.com/bshaffer/oauth2-server-php), you + [`oauth2-server-php`](https://github.com/bshaffer/oauth2-server-php), you may need to [add a configuration parameter to Apache](https://github.com/bshaffer/oauth2-server-php/issues/926#issuecomment-387502778). diff --git a/doc/administration/auth/okta.md b/doc/administration/auth/okta.md index 50dc3b58680..a014fe7404f 100644 --- a/doc/administration/auth/okta.md +++ b/doc/administration/auth/okta.md @@ -19,13 +19,13 @@ The following guidance is based on this Okta article, on adding a [SAML Applicat 1. On Okta admin section, make sure to select Classic UI view in the top left corner. From there, choose to **Add an App**. 1. When the app screen comes up you see another button to **Create an App** and choose SAML 2.0 on the next screen. -1. Now, very important, add a logo +1. Optionally you can add a logo (you can choose it from <https://about.gitlab.com/press/>). You'll have to crop and resize it. -1. Next, you'll need the to fill in the SAML general configuration. Here's an example +1. Next, you'll need the to fill in the SAML general configuration. Here's an example (showing the required URLs and attribute mapping): image. -  +  1. The last part of the configuration is the feedback section where you can just say you're a customer and creating an app for internal use. diff --git a/doc/administration/auth/smartcard.md b/doc/administration/auth/smartcard.md index 9790802e413..dfeee5e7ac4 100644 --- a/doc/administration/auth/smartcard.md +++ b/doc/administration/auth/smartcard.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Smartcard authentication **(PREMIUM ONLY)** +# Smartcard authentication **(PREMIUM SELF)** GitLab supports authentication using smartcards. diff --git a/doc/administration/availability/index.md b/doc/administration/availability/index.md deleted file mode 100644 index 4f934646ed6..00000000000 --- a/doc/administration/availability/index.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: ../reference_architectures/index.md ---- - -This document was moved to [another location](../reference_architectures/index.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/build_artifacts.md b/doc/administration/build_artifacts.md deleted file mode 100644 index e42f50b2e54..00000000000 --- a/doc/administration/build_artifacts.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'job_artifacts.md' ---- - -This document was moved to [job_artifacts](job_artifacts.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/compliance.md b/doc/administration/compliance.md index 6079e838ac6..b2c8d8982bd 100644 --- a/doc/administration/compliance.md +++ b/doc/administration/compliance.md @@ -15,9 +15,9 @@ relevant compliance standards. |Feature |GitLab tier |GitLab.com | Product level | | ---------| :--------: | :-------: | :-----------: | -|**[Restrict SSH Keys](../security/ssh_keys_restrictions.md)**<br>Control the technology and key length of SSH keys used to access GitLab|Core+||Instance| -|**[Granular user roles and flexible permissions](../user/permissions.md)**<br>Manage access and permissions with five different user roles and settings for external users. Set permissions according to people's role, rather than either read or write access to a repository. Don't share the source code with people that only need access to the issue tracker.|Core+|✓|Instance, Group, Project| -|**[Enforce TOS acceptance](../user/admin_area/settings/terms.md)**<br>Enforce your users accepting new terms of service by blocking GitLab traffic.|Core+||Instance| +|**[Restrict SSH Keys](../security/ssh_keys_restrictions.md)**<br>Control the technology and key length of SSH keys used to access GitLab|Free+||Instance| +|**[Granular user roles and flexible permissions](../user/permissions.md)**<br>Manage access and permissions with five different user roles and settings for external users. Set permissions according to people's role, rather than either read or write access to a repository. Don't share the source code with people that only need access to the issue tracker.|Free+|✓|Instance, Group, Project| +|**[Enforce TOS acceptance](../user/admin_area/settings/terms.md)**<br>Enforce your users accepting new terms of service by blocking GitLab traffic.|Free+||Instance| |**[Email all users of a project, group, or entire server](../tools/email.md)**<br>An admin can email groups of users based on project or group membership, or email everyone using the GitLab instance. This is great for scheduled maintenance or upgrades.|Starter+|||Instance |**[Omnibus package supports log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-forwarding)**<br>Forward your logs to a central system.|Starter+||Instance| |**[Lock project membership to group](../user/group/index.md#member-lock)**<br>Group owners can prevent new members from being added to projects within a group.|Starter+|✓|Group| @@ -26,4 +26,4 @@ relevant compliance standards. |**[Audit events](audit_events.md)**<br>To maintain the integrity of your code, GitLab Enterprise Edition Premium gives admins the ability to view any modifications made within the GitLab server in an advanced audit events system, so you can control, analyze, and track every change.|Premium+|✓|Instance, Group, Project| |**[Auditor users](auditor_users.md)**<br>Auditor users are users who are given read-only access to all projects, groups, and other resources on the GitLab instance.|Premium+||Instance| |**[Credentials inventory](../user/admin_area/credentials_inventory.md)**<br>With a credentials inventory, GitLab administrators can keep track of the credentials used by all of the users in their GitLab instance. |Ultimate||Instance| -|**Separation of Duties using [Protected branches](../user/project/protected_branches.md#protected-branches-approval-by-code-owners) and [custom CI Configuration Paths](../ci/pipelines/settings.md#custom-ci-configuration-path)**<br> GitLab Silver and Premium users can leverage the GitLab cross-project YAML configurations to define deployers of code and developers of code. View the [Separation of Duties Deploy Project](https://gitlab.com/guided-explorations/separation-of-duties-deploy/blob/master/README.md) and [Separation of Duties Project](https://gitlab.com/guided-explorations/separation-of-duties/blob/master/README.md) to see how to use this set up to define these roles.|Premium+|✓|Project| +|**Separation of Duties using [Protected branches](../user/project/protected_branches.md#protected-branches-approval-by-code-owners) and [custom CI Configuration Paths](../ci/pipelines/settings.md#custom-cicd-configuration-path)**<br> GitLab Premium users can leverage the GitLab cross-project YAML configurations to define deployers of code and developers of code. View the [Separation of Duties Deploy Project](https://gitlab.com/guided-explorations/separation-of-duties-deploy/blob/master/README.md) and [Separation of Duties Project](https://gitlab.com/guided-explorations/separation-of-duties/blob/master/README.md) to see how to use this set up to define these roles.|Premium+|✓|Project| diff --git a/doc/administration/consul.md b/doc/administration/consul.md index 5b577443c7c..dfc859e30c2 100644 --- a/doc/administration/consul.md +++ b/doc/administration/consul.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# How to set up Consul **(PREMIUM ONLY)** +# How to set up Consul **(PREMIUM SELF)** A Consul cluster consists of both [server and client agents](https://www.consul.io/docs/agent). diff --git a/doc/administration/container_registry.md b/doc/administration/container_registry.md deleted file mode 100644 index 52941556237..00000000000 --- a/doc/administration/container_registry.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'packages/container_registry.md' ---- - -This document was moved to [another location](packages/container_registry.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/custom_hooks.md b/doc/administration/custom_hooks.md deleted file mode 100644 index 0580540eef9..00000000000 --- a/doc/administration/custom_hooks.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'server_hooks.md' ---- - -This document was moved to [another location](server_hooks.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/database_load_balancing.md b/doc/administration/database_load_balancing.md index 45243afd8ac..fe30d98b92d 100644 --- a/doc/administration/database_load_balancing.md +++ b/doc/administration/database_load_balancing.md @@ -4,7 +4,7 @@ group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Database Load Balancing **(PREMIUM ONLY)** +# Database Load Balancing **(PREMIUM SELF)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/1283) in [GitLab Premium](https://about.gitlab.com/pricing/) 9.0. diff --git a/doc/administration/dependency_proxy.md b/doc/administration/dependency_proxy.md deleted file mode 100644 index c0e0643b5de..00000000000 --- a/doc/administration/dependency_proxy.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'packages/dependency_proxy.md' ---- - -This document was moved to [another location](packages/dependency_proxy.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/encrypted_configuration.md b/doc/administration/encrypted_configuration.md index 01a1cf66bdc..508f4314694 100644 --- a/doc/administration/encrypted_configuration.md +++ b/doc/administration/encrypted_configuration.md @@ -5,7 +5,7 @@ info: "To determine the technical writer assigned to the Stage/Group associated type: reference --- -# Encrypted Configuration **(CORE ONLY)** +# Encrypted Configuration **(FREE SELF)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/45712) in GitLab 13.7. @@ -34,4 +34,4 @@ The new secret can be generated by running: bundle exec rake gitlab:env:info RAILS_ENV=production GITLAB_GENERATE_ENCRYPTED_SETTINGS_KEY_BASE=true ``` -This will print general info on the GitLab instance, but will also cause the key to be generated in `<path-to-gitlab-rails>/config/secrets.yml` +This will print general information on the GitLab instance, but will also cause the key to be generated in `<path-to-gitlab-rails>/config/secrets.yml` diff --git a/doc/administration/external_database.md b/doc/administration/external_database.md deleted file mode 100644 index 7424b4e1e83..00000000000 --- a/doc/administration/external_database.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -redirect_to: 'postgresql/external.md' ---- - -# Configure GitLab using an external PostgreSQL service - -This content has been moved to a [new location](postgresql/external.md). diff --git a/doc/administration/feature_flags.md b/doc/administration/feature_flags.md index 272009c1f3a..55727822654 100644 --- a/doc/administration/feature_flags.md +++ b/doc/administration/feature_flags.md @@ -6,7 +6,7 @@ type: reference description: "GitLab administrator: enable and disable GitLab features deployed behind feature flags" --- -# Enable and disable GitLab features deployed behind feature flags **(CORE ONLY)** +# Enable and disable GitLab features deployed behind feature flags **(FREE SELF)** GitLab adopted [feature flags strategies](../development/feature_flags/index.md) to deploy features in an early stage of development so that they can be diff --git a/doc/administration/file_hooks.md b/doc/administration/file_hooks.md index 5de2e4aec3f..cfaeda546cc 100644 --- a/doc/administration/file_hooks.md +++ b/doc/administration/file_hooks.md @@ -19,19 +19,20 @@ directly to the GitLab source code and contribute back upstream. This way we can ensure functionality is preserved across versions and covered by tests. NOTE: -File hooks must be configured on the filesystem of the GitLab server. Only GitLab -server administrators will be able to complete these tasks. Explore -[system hooks](../system_hooks/system_hooks.md) or [webhooks](../user/project/integrations/webhooks.md) as an option if you do not have filesystem access. - -A file hook will run on each event so it's up to you to filter events or projects -within a file hook code. You can have as many file hooks as you want. Each file hook will -be triggered by GitLab asynchronously in case of an event. For a list of events +File hooks must be configured on the file system of the GitLab server. Only GitLab +server administrators can complete these tasks. Explore +[system hooks](../system_hooks/system_hooks.md) or [webhooks](../user/project/integrations/webhooks.md) +as an option if you do not have file system access. + +A file hook runs on each event. You can filter events or projects +in a file hook's code, and create many file hooks as you need. Each file hook is +triggered by GitLab asynchronously in case of an event. For a list of events see the [system hooks](../system_hooks/system_hooks.md) documentation. ## Setup The file hooks must be placed directly into the `file_hooks` directory, subdirectories -will be ignored. There is an +are ignored. There is an [`example` directory inside `file_hooks`](https://gitlab.com/gitlab-org/gitlab/tree/master/file_hooks/examples) where you can find some basic examples. @@ -52,23 +53,23 @@ Follow the steps below to set up a custom hook: in any language, and ensure the 'shebang' at the top properly reflects the language type. For example, if the script is in Ruby the shebang will probably be `#!/usr/bin/env ruby`. -1. The data to the file hook will be provided as JSON on STDIN. It will be exactly +1. The data to the file hook is provided as JSON on STDIN. It is exactly the same as for [system hooks](../system_hooks/system_hooks.md). -That's it! Assuming the file hook code is properly implemented, the hook will fire +That's it! Assuming the file hook code is properly implemented, the hook fires as appropriate. The file hooks file list is updated for each event, there is no need to restart GitLab to apply a new file hook. If a file hook executes with non-zero exit code or GitLab fails to execute it, a -message will be logged to: +message is logged to: - `gitlab-rails/plugin.log` in an Omnibus installation. - `log/plugin.log` in a source installation. ## Creating file hooks -Below is an example that will only response on the event `project_create` and -will inform the admins from the GitLab instance that a new project has been created. +This example responds only on the event `project_create`, and +the GitLab instance informs the administrators that a new project has been created. ```ruby #!/opt/gitlab/embedded/bin/ruby @@ -97,7 +98,7 @@ end Writing your own file hook can be tricky and it's easier if you can check it without altering the system. A Rake task is provided so that you can use it in a staging environment to test your file hook before using it in production. -The Rake task will use a sample data and execute each of file hook. The output +The Rake task uses a sample data and execute each of file hook. The output should be enough to determine if the system sees your file hook and if it was executed without errors. diff --git a/doc/administration/geo/disaster_recovery/background_verification.md b/doc/administration/geo/disaster_recovery/background_verification.md index f2854d0538b..ae81db8a8d0 100644 --- a/doc/administration/geo/disaster_recovery/background_verification.md +++ b/doc/administration/geo/disaster_recovery/background_verification.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Automatic background verification **(PREMIUM ONLY)** +# Automatic background verification **(PREMIUM SELF)** NOTE: Automatic background verification of repositories and wikis was added in diff --git a/doc/administration/geo/disaster_recovery/bring_primary_back.md b/doc/administration/geo/disaster_recovery/bring_primary_back.md index 3f86e03bd7f..d06f11bbe09 100644 --- a/doc/administration/geo/disaster_recovery/bring_primary_back.md +++ b/doc/administration/geo/disaster_recovery/bring_primary_back.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Bring a demoted primary node back online **(PREMIUM ONLY)** +# Bring a demoted primary node back online **(PREMIUM SELF)** After a failover, it is possible to fail back to the demoted **primary** node to restore your original configuration. This process consists of two steps: diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index 2b0a3e2f114..d62d7322ac0 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Disaster Recovery (Geo) **(PREMIUM ONLY)** +# Disaster Recovery (Geo) **(PREMIUM SELF)** Geo replicates your database, your Git repositories, and few other assets. We will support and replicate more data in the future, that will enable you to @@ -145,6 +145,13 @@ Note the following when promoting a secondary: a point-in-time recovery to the last known state. Data that was created on the primary while the secondary was paused will be lost. + NOTE: + In GitLab 13.7 and earlier, if you have a data type with zero items to sync, + this command reports `ERROR - Replication is not up-to-date` even if + replication is actually up-to-date. If replication and verification output + shows that it is complete, you can add `--skip-preflight-checks` to make the + command complete promotion. This bug was fixed in GitLab 13.8 and later. + To promote the secondary node to primary along with preflight checks: ```shell diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index 1468c5cd39d..1418ff77262 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Disaster recovery for planned failover **(PREMIUM ONLY)** +# Disaster recovery for planned failover **(PREMIUM SELF)** The primary use-case of Disaster Recovery is to ensure business continuity in the event of unplanned outage, but it can be used in conjunction with a planned @@ -45,6 +45,12 @@ be found in `/var/opt/gitlab/gitlab-rails/shared/pages` if using Omnibus). ## Preflight checks +NOTE: +In GitLab 13.7 and earlier, if you have a data type with zero items to sync, +this command reports `ERROR - Replication is not up-to-date` even if +replication is actually up-to-date. This bug was fixed in GitLab 13.8 and +later. + Run this command to list out all preflight checks and automatically check if replication and verification are complete before scheduling a planned failover to ensure the process will go smoothly: ```shell diff --git a/doc/administration/geo/disaster_recovery/promotion_runbook.md b/doc/administration/geo/disaster_recovery/promotion_runbook.md deleted file mode 100644 index eef1e7ae9dd..00000000000 --- a/doc/administration/geo/disaster_recovery/promotion_runbook.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: runbooks/planned_failover_single_node.md ---- - -This document was moved to [another location](runbooks/planned_failover_single_node.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md index f17990ce5f8..3227fafca0f 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md @@ -9,7 +9,7 @@ WARNING: This runbook is in **alpha**. For complete, production-ready documentation, see the [disaster recovery documentation](../index.md). -# Disaster Recovery (Geo) promotion runbooks **(PREMIUM ONLY)** +# Disaster Recovery (Geo) promotion runbooks **(PREMIUM SELF)** ## Geo planned failover for a multi-node configuration diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md index a2a9350e411..7f311d172ef 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md @@ -9,7 +9,7 @@ WARNING: This runbook is in **alpha**. For complete, production-ready documentation, see the [disaster recovery documentation](../index.md). -# Disaster Recovery (Geo) promotion runbooks **(PREMIUM ONLY)** +# Disaster Recovery (Geo) promotion runbooks **(PREMIUM SELF)** ## Geo planned failover for a single-node configuration @@ -233,12 +233,25 @@ To promote the secondary node: check if replication and verification are complete before scheduling a planned failover to ensure the process will go smoothly: + NOTE: + In GitLab 13.7 and earlier, if you have a data type with zero items to sync, + this command reports `ERROR - Replication is not up-to-date` even if + replication is actually up-to-date. This bug was fixed in GitLab 13.8 and + later. + ```shell gitlab-ctl promotion-preflight-checks ``` 1. Promote the **secondary**: + NOTE: + In GitLab 13.7 and earlier, if you have a data type with zero items to sync, + this command reports `ERROR - Replication is not up-to-date` even if + replication is actually up-to-date. If replication and verification output + shows that it is complete, you can add `--skip-preflight-checks` to make the + command complete promotion. This bug was fixed in GitLab 13.8 and later. + ```shell gitlab-ctl promote-to-primary-node ``` diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index 1985ea2e04c..296500e35e2 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Geo **(PREMIUM ONLY)** +# Geo **(PREMIUM SELF)** > - Introduced in GitLab Enterprise Edition 8.9. > - Using Geo in combination with diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index 5adf256f6fa..63c5b5d6c62 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Geo configuration **(PREMIUM ONLY)** +# Geo configuration **(PREMIUM SELF)** ## Configuring a new **secondary** node diff --git a/doc/administration/geo/replication/database.md b/doc/administration/geo/replication/database.md deleted file mode 100644 index 099a73e93d8..00000000000 --- a/doc/administration/geo/replication/database.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: '../setup/database.md' ---- - -This document was moved to [another location](../setup/index.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index 0d2922c3347..667dd83c3f1 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Geo data types support **(PREMIUM ONLY)** +# Geo data types support **(PREMIUM SELF)** A Geo data type is a specific class of data that is required by one or more GitLab features to store relevant information. @@ -35,21 +35,21 @@ verification methods: | Git | Object pools for forked project deduplication | Geo with Gitaly | _Not implemented_ | | Git | Project Snippets | Geo with Gitaly | _Not implemented_ | | Git | Personal Snippets | Geo with Gitaly | _Not implemented_ | -| Blobs | User uploads _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | User uploads _(file system)_ | Geo with API | _Not implemented_ | | Blobs | User uploads _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | -| Blobs | LFS objects _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | LFS objects _(file system)_ | Geo with API | _Not implemented_ | | Blobs | LFS objects _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | -| Blobs | CI job artifacts _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | CI job artifacts _(file system)_ | Geo with API | _Not implemented_ | | Blobs | CI job artifacts _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | -| Blobs | Archived CI build traces _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | Archived CI build traces _(file system)_ | Geo with API | _Not implemented_ | | Blobs | Archived CI build traces _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | -| Blobs | Container registry _(filesystem)_ | Geo with API/Docker API | _Not implemented_ | +| Blobs | Container registry _(file system)_ | Geo with API/Docker API | _Not implemented_ | | Blobs | Container registry _(object storage)_ | Geo with API/Managed/Docker API (*2*) | _Not implemented_ | -| Blobs | Package registry _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | Package registry _(file system)_ | Geo with API | _Not implemented_ | | Blobs | Package registry _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | -| Blobs | Versioned Terraform State _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | Versioned Terraform State _(file system)_ | Geo with API | _Not implemented_ | | Blobs | Versioned Terraform State _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | -| Blobs | External Merge Request Diffs _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | External Merge Request Diffs _(file system)_ | Geo with API | _Not implemented_ | | Blobs | External Merge Request Diffs _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | - (*1*): Redis replication can be used as part of HA with Redis sentinel. It's not used between Geo nodes. @@ -63,8 +63,8 @@ is responsible for allowing access and operations on the locally stored Git repo on a machine with a single disk, multiple disks mounted as a single mount-point (like with a RAID array), or using LVM. -It requires no special filesystem and can work with NFS or a mounted Storage Appliance (there may be -performance limitations when using a remote filesystem). +It requires no special file system and can work with NFS or a mounted Storage Appliance (there may be +performance limitations when using a remote file system). Communication is done via Gitaly's own gRPC API. There are three possible ways of synchronization: @@ -88,13 +88,13 @@ Both types will be synced to a secondary node. GitLab stores files and blobs such as Issue attachments or LFS objects into either: -- The filesystem in a specific location. +- The file system in a specific location. - An [Object Storage](../../object_storage.md) solution. Object Storage solutions can be: - Cloud based like Amazon S3 Google Cloud Storage. - Hosted by you (like MinIO). - A Storage Appliance that exposes an Object Storage-compatible API. -When using the filesystem store instead of Object Storage, you need to use network mounted filesystems +When using the file system store instead of Object Storage, you need to use network mounted file systems to run GitLab when using more than one server. With respect to replication and verification: @@ -144,9 +144,9 @@ The replication for some data types is behind a corresponding feature flag: > - They're enabled on GitLab.com. > - They can't be enabled or disabled per-project. > - They are recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable them](#enable-or-disable-replication-for-some-data-types). **(CORE ONLY)** +> - For GitLab self-managed instances, GitLab administrators can opt to [disable them](#enable-or-disable-replication-for-some-data-types). **(FREE SELF)** -#### Enable or disable replication (for some data types) **(CORE ONLY)** +#### Enable or disable replication (for some data types) Replication for some data types are released behind feature flags that are **enabled by default**. [GitLab administrators with access to the GitLab Rails console](../../feature_flags.md) can opt to disable it for your instance. You can find feature flag names of each of those data types in the notes column of the table below. @@ -173,7 +173,7 @@ successfully, you must replicate their data using some other means. | Feature | Replicated (added in GitLab version) | Verified (added in GitLab version) | Object Storage replication (see [Geo with Object Storage](object_storage.md)) | Notes | |:---------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|:----------------------------------------------------------|:-------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [Application data in PostgreSQL](../../postgresql/index.md) | **Yes** (10.2) | **Yes** (10.2) | No | | -| [Project repository](../../..//user/project/repository/) | **Yes** (10.2) | **Yes** (10.7) | No | | +| [Project repository](../../../user/project/repository/) | **Yes** (10.2) | **Yes** (10.7) | No | | | [Project wiki repository](../../../user/project/wiki/) | **Yes** (10.2) | **Yes** (10.7) | No | | [Group wiki repository](../../../user/group/index.md#group-wikis) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | No | | | [Uploads](../../uploads.md) | **Yes** (10.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | No | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them. | @@ -201,4 +201,4 @@ successfully, you must replicate their data using some other means. | [GitLab Pages](../../pages/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | No | | | [CI Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/pipeline_artifact.rb) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Persists additional artifacts after a pipeline completes | | [Dependency proxy images](../../../user/packages/dependency_proxy/index.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/259694) | No | No | Blocked on [Geo: Secondary Mimicry](https://gitlab.com/groups/gitlab-org/-/epics/1528). Note that replication of this cache is not needed for Disaster Recovery purposes because it can be recreated from external sources. | -| [Vulnerability Export](../../../user/application_security/security_dashboard/#export-vulnerabilities) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Not planned because they are ephemeral and sensitive. They can be regenerated on demand. | +| [Vulnerability Export](../../../user/application_security/vulnerability_report/#export-vulnerabilities) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Not planned because they are ephemeral and sensitive. They can be regenerated on demand. | diff --git a/doc/administration/geo/replication/disable_geo.md b/doc/administration/geo/replication/disable_geo.md index e8767a56266..a26adcefef5 100644 --- a/doc/administration/geo/replication/disable_geo.md +++ b/doc/administration/geo/replication/disable_geo.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Disabling Geo **(PREMIUM ONLY)** +# Disabling Geo **(PREMIUM SELF)** If you want to revert to a regular Omnibus setup after a test, or you have encountered a Disaster Recovery situation and you want to disable Geo momentarily, you can use these instructions to disable your diff --git a/doc/administration/geo/replication/docker_registry.md b/doc/administration/geo/replication/docker_registry.md index f92a878662e..1669abbc52a 100644 --- a/doc/administration/geo/replication/docker_registry.md +++ b/doc/administration/geo/replication/docker_registry.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Docker Registry for a secondary node **(PREMIUM ONLY)** +# Docker Registry for a secondary node **(PREMIUM SELF)** You can set up a [Docker Registry](https://docs.docker.com/registry/) on your **secondary** Geo node that mirrors the one on the **primary** Geo node. diff --git a/doc/administration/geo/replication/external_database.md b/doc/administration/geo/replication/external_database.md deleted file mode 100644 index dfaf6819fa0..00000000000 --- a/doc/administration/geo/replication/external_database.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: '../setup/external_database.md' ---- - -This document was moved to [another location](../setup/external_database.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/geo/replication/faq.md b/doc/administration/geo/replication/faq.md index ab8199c3717..73a36f5e674 100644 --- a/doc/administration/geo/replication/faq.md +++ b/doc/administration/geo/replication/faq.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Geo Frequently Asked Questions **(PREMIUM ONLY)** +# Geo Frequently Asked Questions **(PREMIUM SELF)** ## What are the minimum requirements to run Geo? diff --git a/doc/administration/geo/replication/geo_validation_tests.md b/doc/administration/geo/replication/geo_validation_tests.md index 2d701458e66..f050d3e708c 100644 --- a/doc/administration/geo/replication/geo_validation_tests.md +++ b/doc/administration/geo/replication/geo_validation_tests.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Geo validation tests **(PREMIUM ONLY)** +# Geo validation tests **(PREMIUM SELF)** The Geo team performs manual testing and validation on common deployment configurations to ensure that Geo works when upgrading between minor GitLab versions and major PostgreSQL database versions. @@ -53,7 +53,7 @@ The following are GitLab upgrade validation tests we performed. - Outcome: Partial success because we did not run the looping pipeline during the demo to validate zero-downtime. - Follow up issues: - - [Clarify hup Puma/Unicorn should include deploy node](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5460) + - [Clarify how Puma/Unicorn should include deploy node](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5460) - [Investigate MR creation failure after upgrade to 12.9.10](https://gitlab.com/gitlab-org/gitlab/-/issues/223282) Closed as false positive. ### February 2020 @@ -128,7 +128,7 @@ The following are PostgreSQL upgrade validation tests we performed. PostgreSQL 12 with a database cluster on the primary is not recommended until the issues are resolved. - Known issues for PostgreSQL clusters: - [Ensure Patroni detects PostgreSQL update](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5423) - - [Allow configuring permanent replication slots in patroni](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5628) + - [Allow configuring permanent replication slots in Patroni](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5628) ### August 2020 diff --git a/doc/administration/geo/replication/high_availability.md b/doc/administration/geo/replication/high_availability.md deleted file mode 100644 index 1da4246db1f..00000000000 --- a/doc/administration/geo/replication/high_availability.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'multiple_servers.md' ---- - -This document was moved to [another location](multiple_servers.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/geo/replication/location_aware_git_url.md b/doc/administration/geo/replication/location_aware_git_url.md index f2cf8c9bda0..0eea792d374 100644 --- a/doc/administration/geo/replication/location_aware_git_url.md +++ b/doc/administration/geo/replication/location_aware_git_url.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Location-aware Git remote URL with AWS Route53 **(PREMIUM ONLY)** +# Location-aware Git remote URL with AWS Route53 **(PREMIUM SELF)** You can provide GitLab users with a single remote URL that automatically uses the Geo node closest to them. This means users don't need to update their Git diff --git a/doc/administration/geo/replication/multiple_servers.md b/doc/administration/geo/replication/multiple_servers.md index 0ab972c9077..f83e0e14e54 100644 --- a/doc/administration/geo/replication/multiple_servers.md +++ b/doc/administration/geo/replication/multiple_servers.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Geo for multiple nodes **(PREMIUM ONLY)** +# Geo for multiple nodes **(PREMIUM SELF)** This document describes a minimal reference architecture for running Geo in a multi-node configuration. If your multi-node setup differs from the one diff --git a/doc/administration/geo/replication/object_storage.md b/doc/administration/geo/replication/object_storage.md index ce791d66b34..ad419f999b3 100644 --- a/doc/administration/geo/replication/object_storage.md +++ b/doc/administration/geo/replication/object_storage.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Geo with Object storage **(PREMIUM ONLY)** +# Geo with Object storage **(PREMIUM SELF)** Geo can be used in combination with Object Storage (AWS S3, or other compatible object storage). @@ -29,7 +29,7 @@ WARNING: This is a [**beta** feature](https://about.gitlab.com/handbook/product/#beta) and is not ready yet for production use at any scale. The main limitations are a lack of testing at scale and no verification of any replicated data. **Secondary** nodes can replicate files stored on the **primary** node regardless of -whether they are stored on the local filesystem or in object storage. +whether they are stored on the local file system or in object storage. To enable GitLab replication, you must: diff --git a/doc/administration/geo/replication/remove_geo_node.md b/doc/administration/geo/replication/remove_geo_node.md index 519b93b447b..fd74c224a7f 100644 --- a/doc/administration/geo/replication/remove_geo_node.md +++ b/doc/administration/geo/replication/remove_geo_node.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Removing secondary Geo nodes **(PREMIUM ONLY)** +# Removing secondary Geo nodes **(PREMIUM SELF)** **Secondary** nodes can be removed from the Geo cluster using the Geo admin page of the **primary** node. To remove a **secondary** node: diff --git a/doc/administration/geo/replication/security_review.md b/doc/administration/geo/replication/security_review.md index 7c15662340c..70b58582435 100644 --- a/doc/administration/geo/replication/security_review.md +++ b/doc/administration/geo/replication/security_review.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Geo security review (Q&A) **(PREMIUM ONLY)** +# Geo security review (Q&A) **(PREMIUM SELF)** The following security review of the Geo feature set focuses on security aspects of the feature as they apply to customers running their own GitLab instances. The review @@ -194,7 +194,7 @@ from [owasp.org](https://owasp.org/). administrator via SSH, and then back out to the **secondary** node in the same manner. In particular, this includes the PostgreSQL replication credentials and a secret key (`db_key_base`) which is used to decrypt certain columns in the database. - The `db_key_base` secret is stored unencrypted on the filesystem, in + The `db_key_base` secret is stored unencrypted on the file system, in `/etc/gitlab/gitlab-secrets.json`, along with a number of other secrets. There is no at-rest protection for them. @@ -217,7 +217,7 @@ from [owasp.org](https://owasp.org/). - **Secondary** nodes and **primary** nodes interact via HTTP/HTTPS (secured with JSON web tokens) and via PostgreSQL streaming replication. -- Within a **primary** node or **secondary** node, the SSOT is the filesystem and the database +- Within a **primary** node or **secondary** node, the SSOT is the file system and the database (including Geo tracking database on **secondary** node). The various internal components are orchestrated to make alterations to these stores. @@ -231,7 +231,7 @@ from [owasp.org](https://owasp.org/). ### What data is or may need to be encrypted and what key management requirements have been defined? -- Neither **primary** nodes or **secondary** nodes encrypt Git repository or filesystem data at +- Neither **primary** nodes or **secondary** nodes encrypt Git repository or file system data at rest. A subset of database columns are encrypted at rest using the `db_otp_key`. - A static secret shared across all hosts in a GitLab deployment. - In transit, data should be encrypted, although the application does permit @@ -287,5 +287,5 @@ from [owasp.org](https://owasp.org/). ### What application auditing requirements have been defined? How are audit and debug logs accessed, stored, and secured? -- Structured JSON log is written to the filesystem, and can also be ingested +- Structured JSON log is written to the file system, and can also be ingested into a Kibana installation for further analysis. diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 2c0160ed02a..3e9fb9d29e7 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Troubleshooting Geo **(PREMIUM ONLY)** +# Troubleshooting Geo **(PREMIUM SELF)** Setting up Geo requires careful attention to details and sometimes it's easy to miss a step. @@ -678,19 +678,19 @@ sudo /opt/gitlab/embedded/bin/gitlab-pg-ctl promote GitLab 12.9 and later are [unaffected by this error](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5147). -### Two-factor authentication is broken after a failover +### Message: `ERROR - Replication is not up-to-date` during `gitlab-ctl promotion-preflight-checks` -The setup instructions for Geo prior to 10.5 failed to replicate the -`otp_key_base` secret, which is used to encrypt the two-factor authentication -secrets stored in the database. If it differs between **primary** and **secondary** -nodes, users with two-factor authentication enabled won't be able to log in -after a failover. +In GitLab 13.7 and earlier, if you have a data type with zero items to sync, +this command reports `ERROR - Replication is not up-to-date` even if +replication is actually up-to-date. This bug was fixed in GitLab 13.8 and +later. -If you still have access to the old **primary** node, you can follow the -instructions in the -[Upgrading to GitLab 10.5](../replication/version_specific_updates.md#updating-to-gitlab-105) -section to resolve the error. Otherwise, the secret is lost and you'll need to -[reset two-factor authentication for all users](../../../security/two_factor_authentication.md#disabling-2fa-for-everyone). +### Message: `ERROR - Replication is not up-to-date` during `gitlab-ctl promote-to-primary-node` + +In GitLab 13.7 and earlier, if you have a data type with zero items to sync, +this command reports `ERROR - Replication is not up-to-date` even if +replication is actually up-to-date. If replication and verification output +shows that it is complete, you can add `--skip-preflight-checks` to make the command complete promotion. This bug was fixed in GitLab 13.8 and later. ## Expired artifacts diff --git a/doc/administration/geo/replication/tuning.md b/doc/administration/geo/replication/tuning.md index 183180e5ade..535d1053744 100644 --- a/doc/administration/geo/replication/tuning.md +++ b/doc/administration/geo/replication/tuning.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Tuning Geo **(PREMIUM ONLY)** +# Tuning Geo **(PREMIUM SELF)** ## Changing the sync/verification capacity values diff --git a/doc/administration/geo/replication/updating_the_geo_nodes.md b/doc/administration/geo/replication/updating_the_geo_nodes.md index ff3a1e8689b..0c68adf162d 100644 --- a/doc/administration/geo/replication/updating_the_geo_nodes.md +++ b/doc/administration/geo/replication/updating_the_geo_nodes.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Updating the Geo nodes **(PREMIUM ONLY)** +# Updating the Geo nodes **(PREMIUM SELF)** WARNING: Read these sections carefully before updating your Geo nodes. Not following diff --git a/doc/administration/geo/replication/using_a_geo_server.md b/doc/administration/geo/replication/using_a_geo_server.md index 743b1b384db..7578b6bf61a 100644 --- a/doc/administration/geo/replication/using_a_geo_server.md +++ b/doc/administration/geo/replication/using_a_geo_server.md @@ -7,7 +7,7 @@ type: howto <!-- Please update EE::GitLab::GeoGitAccess::GEO_SERVER_DOCS_URL if this file is moved) --> -# Using a Geo Server **(PREMIUM ONLY)** +# Using a Geo Server **(PREMIUM SELF)** After you set up the [database replication and configure the Geo nodes](../index.md#setup-instructions), use your closest GitLab node as you would a normal standalone GitLab instance. diff --git a/doc/administration/geo/replication/version_specific_updates.md b/doc/administration/geo/replication/version_specific_updates.md index 0d8eba555ab..be2ce0ac2c0 100644 --- a/doc/administration/geo/replication/version_specific_updates.md +++ b/doc/administration/geo/replication/version_specific_updates.md @@ -5,23 +5,37 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Version specific update instructions +# Version-specific update instructions -Check this document if it includes instructions for the version you are updating. -These steps go together with the [general steps](updating_the_geo_nodes.md#general-update-steps) +Review this page for update instructions for your version. These steps +accompany the [general steps](updating_the_geo_nodes.md#general-update-steps) for updating Geo nodes. +## Updating to GitLab 13.7 + +We've detected an issue with the `FetchRemove` call used by Geo secondaries. +This causes performance issues as we execute reference transaction hooks for +each updated reference. Delay any upgrade attempts until this is in the +[13.7.5 patch release.](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3002). +More details are available [in this issue](https://gitlab.com/gitlab-org/git/-/issues/79). + ## Updating to GitLab 13.5 -In GitLab 13.5, there is a [regression that prevents viewing a list of container repositories and registries](https://gitlab.com/gitlab-org/gitlab/-/issues/285475) on Geo secondaries. This issue is fixed in GitLab 13.6.1 and later. +GitLab 13.5 has a [regression that prevents viewing a list of container repositories and registries](https://gitlab.com/gitlab-org/gitlab/-/issues/285475) +on Geo secondaries. This issue is fixed in GitLab 13.6.1 and later. ## Updating to GitLab 13.3 -In GitLab 13.3, Geo removed the PostgreSQL [Foreign Data Wrapper](https://www.postgresql.org/docs/11/postgres-fdw.html) dependency for the tracking database. +In GitLab 13.3, Geo removed the PostgreSQL [Foreign Data Wrapper](https://www.postgresql.org/docs/11/postgres-fdw.html) +dependency for the tracking database. -The FDW server, user, and the extension will be removed during the upgrade process on each **secondary** node. The GitLab settings related to the FDW in the `/etc/gitlab/gitlab.rb` have been deprecated and can be safely removed. +The FDW server, user, and the extension will be removed during the upgrade +process on each secondary node. The GitLab settings related to the FDW in the +`/etc/gitlab/gitlab.rb` have been deprecated and can be safely removed. -There are some scenarios like using an external PostgreSQL instance for the tracking database where the FDW settings must be removed manually. Enter the PostgreSQL console of that instance and remove them: +There are some scenarios like using an external PostgreSQL instance for the +tracking database where the FDW settings must be removed manually. Enter the +PostgreSQL console of that instance and remove them: ```shell DROP SERVER gitlab_secondary CASCADE; @@ -36,7 +50,10 @@ upgrade to GitLab 13.4 or later. ## Updating to GitLab 13.2 -In GitLab 13.2, promoting a secondary node to a primary while the secondary is paused fails. **Do not pause replication before promoting a secondary.** If the node is paused, please resume before promoting. To avoid this issue, upgrade to GitLab 13.4 or later. +In GitLab 13.2, promoting a secondary node to a primary while the secondary is +paused fails. Do not pause replication before promoting a secondary. If the +node is paused, be sure to resume before promoting. To avoid this issue, +upgrade to GitLab 13.4 or later. ## Updating to GitLab 13.0 @@ -59,12 +76,13 @@ GitLab 12.9.0 through GitLab 12.9.3 are affected by [a bug that stops repository verification](https://gitlab.com/gitlab-org/gitlab/-/issues/213523). The issue is fixed in GitLab 12.9.4. Upgrade to GitLab 12.9.4 or later. -By default, GitLab 12.9 will attempt to automatically update the embedded -PostgreSQL server to 10.12 from 9.6, which requires downtime on secondaries -while reinitializing streaming replication. For the recommended procedure, see the +By default, GitLab 12.9 attempts to update the embedded PostgreSQL server +version from 9.6 to 10.12, which requires downtime on secondaries while +reinitializing streaming replication. For the recommended procedure, see the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). -This can be temporarily disabled by running the following before updating: +You can temporarily disable this behavior by running the following before +updating: ```shell sudo touch /etc/gitlab/disable-postgresql-upgrade @@ -72,12 +90,13 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade ## Updating to GitLab 12.8 -By default, GitLab 12.8 will attempt to automatically update the embedded -PostgreSQL server to 10.12 from 9.6, which requires downtime on secondaries -while reinitializing streaming replication. For the recommended procedure, see -the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). +By default, GitLab 12.8 attempts to update the embedded PostgreSQL server +version from 9.6 to 10.12, which requires downtime on secondaries while +reinitializing streaming replication. For the recommended procedure, see the +[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). -This can be temporarily disabled by running the following before updating: +You can temporarily disable this behavior by running the following before +updating: ```shell sudo touch /etc/gitlab/disable-postgresql-upgrade @@ -87,18 +106,17 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade WARNING: Only upgrade to GitLab 12.7.5 or later. Do not upgrade to versions 12.7.0 -through 12.7.4 because there is [an initialization order -bug](https://gitlab.com/gitlab-org/gitlab/-/issues/199672) that causes Geo -**secondaries** to set the incorrect database connection pool size. [The -fix](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/24021) was +through 12.7.4 because there is [an initialization order bug](https://gitlab.com/gitlab-org/gitlab/-/issues/199672) that causes Geo secondaries to set the incorrect database connection pool size. +[The fix](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/24021) was shipped in 12.7.5. -By default, GitLab 12.7 will attempt to automatically update the embedded -PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries -while reinitializing streaming replication. For the recommended procedure, see -the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). +By default, GitLab 12.7 attempts to update the embedded PostgreSQL server +version from 9.6 to 10.9, which requires downtime on secondaries while +reinitializing streaming replication. For the recommended procedure, see the +[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). -This can be temporarily disabled by running the following before updating: +You can temporarily disable this behavior by running the following before +updating: ```shell sudo touch /etc/gitlab/disable-postgresql-upgrade @@ -106,12 +124,13 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade ## Updating to GitLab 12.6 -By default, GitLab 12.6 will attempt to automatically update the embedded -PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries -while reinitializing streaming replication. For the recommended procedure, see -the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). +By default, GitLab 12.6 attempts to update the embedded PostgreSQL server +version from 9.6 to 10.9, which requires downtime on secondaries while +reinitializing streaming replication. For the recommended procedure, see the +[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). -This can be temporarily disabled by running the following before updating: +You can temporarily disable this behavior by running the following before +updating: ```shell sudo touch /etc/gitlab/disable-postgresql-upgrade @@ -119,12 +138,13 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade ## Updating to GitLab 12.5 -By default, GitLab 12.5 will attempt to automatically update the embedded -PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries -while reinitializing streaming replication. For the recommended procedure, see -the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). +By default, GitLab 12.5 attempts to update the embedded PostgreSQL server +version from 9.6 to 10.9, which requires downtime on secondaries while +reinitializing streaming replication. For the recommended procedure, see the +[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). -This can be temporarily disabled by running the following before updating: +You can temporarily disable this behavior by running the following before +updating: ```shell sudo touch /etc/gitlab/disable-postgresql-upgrade @@ -132,12 +152,13 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade ## Updating to GitLab 12.4 -By default, GitLab 12.4 will attempt to automatically update the embedded -PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries -while reinitializing streaming replication. For the recommended procedure, see -the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). +By default, GitLab 12.4 attempts to update the embedded PostgreSQL server +version from 9.6 to 10.9, which requires downtime on secondaries while +reinitializing streaming replication. For the recommended procedure, see the +[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). -This can be temporarily disabled by running the following before updating: +You can temporarily disable this behavior by running the following before +updating: ```shell sudo touch /etc/gitlab/disable-postgresql-upgrade @@ -146,9 +167,9 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade ## Updating to GitLab 12.3 WARNING: -If the existing PostgreSQL server version is 9.6.x, it is recommended to -upgrade to GitLab 12.4 or later. By default, GitLab 12.3 attempts to update the -embedded PostgreSQL server from 9.6 to 10.9. In certain circumstances, it will +If the existing PostgreSQL server version is 9.6.x, we recommend upgrading to +GitLab 12.4 or later. By default, GitLab 12.3 attempts to update the embedded +PostgreSQL server version from 9.6 to 10.9. In certain circumstances, it can fail. For more information, see the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). @@ -160,23 +181,23 @@ For the recommended procedure, see the ## Updating to GitLab 12.2 WARNING: -If the existing PostgreSQL server version is 9.6.x, it is recommended to -upgrade to GitLab 12.4 or later. By default, GitLab 12.2 attempts to update the -embedded PostgreSQL server from 9.6 to 10.9. In certain circumstances, it will +If the existing PostgreSQL server version is 9.6.x, we recommend upgrading to +GitLab 12.4 or later. By default, GitLab 12.2 attempts to update the embedded +PostgreSQL server version from 9.6 to 10.9. In certain circumstances, it can fail. For more information, see the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). -Additionally, if the PostgreSQL upgrade does not fail, a successful upgrade +Additionally, if the PostgreSQL upgrade doesn't fail, a successful upgrade requires downtime for secondaries while reinitializing streaming replication. For the recommended procedure, see the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). GitLab 12.2 includes the following minor PostgreSQL updates: -- To version `9.6.14` if you run PostgreSQL 9.6. -- To version `10.9` if you run PostgreSQL 10. +- To version `9.6.14`, if you run PostgreSQL 9.6. +- To version `10.9`, if you run PostgreSQL 10. -This update will occur even if major PostgreSQL updates are disabled. +This update occurs even if major PostgreSQL updates are disabled. Before [refreshing Foreign Data Wrapper during a Geo upgrade](https://docs.gitlab.com/omnibus/update/README.html#run-post-deployment-migrations-and-checks), restart the Geo tracking database: @@ -185,14 +206,15 @@ restart the Geo tracking database: sudo gitlab-ctl restart geo-postgresql ``` -The restart avoids a version mismatch when PostgreSQL tries to load the FDW extension. +The restart avoids a version mismatch when PostgreSQL tries to load the FDW +extension. ## Updating to GitLab 12.1 WARNING: -If the existing PostgreSQL server version is 9.6.x, it is recommended to -upgrade to GitLab 12.4 or later. By default, GitLab 12.1 attempts to update the -embedded PostgreSQL server from 9.6 to 10.9. In certain circumstances, it will +If the existing PostgreSQL server version is 9.6.x, we recommend upgrading to +GitLab 12.4 or later. By default, GitLab 12.1 attempts to update the embedded +PostgreSQL server version from 9.6 to 10.9. In certain circumstances, it can fail. For more information, see the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). @@ -206,419 +228,11 @@ For the recommended procedure, see the WARNING: This version is affected by a [bug that results in new LFS objects not being replicated to Geo secondary nodes](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). -The issue is fixed in GitLab 12.1; be sure to upgrade to GitLab 12.1 or later. +The issue is fixed in GitLab 12.1. Be sure to upgrade to GitLab 12.1 or later. ## Updating to GitLab 11.11 WARNING: This version is affected by a [bug that results in new LFS objects not being replicated to Geo secondary nodes](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). -The issue is fixed in GitLab 12.1; be sure to upgrade to GitLab 12.1 or later. - -## Updating to GitLab 10.8 - -Before 10.8, broadcast messages would not propagate without flushing -the cache on the **secondary** nodes. This has been fixed in 10.8, but -requires one last cache flush on each **secondary** node: - -```shell -sudo gitlab-rake cache:clear -``` - -## Updating to GitLab 10.6 - -In 10.4, we started to recommend that you define a password for database user (`gitlab`). - -We now require this change as we use this password to enable the Foreign Data Wrapper, as a way to optimize -the Geo Tracking Database. We are also improving security by disabling the use of **trust** -authentication method. - -1. **(primary)** Login to your **primary** node and run: - - ```shell - gitlab-ctl pg-password-md5 gitlab - # Enter password: <your_password_here> - # Confirm password: <your_password_here> - # fca0b89a972d69f00eb3ec98a5838484 - ``` - - Copy the generated hash and edit `/etc/gitlab/gitlab.rb`: - - ```ruby - # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` - postgresql['sql_user_password'] = '<md5_hash_of_your_password>' - - # Every node that runs Unicorn or Sidekiq needs to have the database - # password specified as below. - # This must be present in all application nodes. - gitlab_rails['db_password'] = '<your_password_here>' - ``` - - Still in the configuration file, locate and remove the `trust_auth_cidr_address`: - - ```ruby - postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','1.2.3.4/32'] # <- Remove this - ``` - -1. **(primary)** Reconfigure and restart: - - ```shell - sudo gitlab-ctl reconfigure - sudo gitlab-ctl restart - ``` - -1. **(secondary)** Login to all **secondary** nodes and edit `/etc/gitlab/gitlab.rb`: - - ```ruby - # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` - postgresql['sql_user_password'] = '<md5_hash_of_your_password>' - - # Every node that runs Unicorn or Sidekiq needs to have the database - # password specified as below. - # This must be present in all application nodes. - gitlab_rails['db_password'] = '<your_password_here>' - - # Enable Foreign Data Wrapper - geo_secondary['db_fdw'] = true - - # Secondary address in CIDR format, for example '5.6.7.8/32' - postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32'] - ``` - - Still in the configuration file, locate and remove the `trust_auth_cidr_address`: - - ```ruby - postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','5.6.7.8/32'] # <- Remove this - ``` - -1. **(secondary)** Reconfigure and restart: - - ```shell - sudo gitlab-ctl reconfigure - sudo gitlab-ctl restart - ``` - -## Updating to GitLab 10.5 - -For Geo Disaster Recovery to work with minimum downtime, your **secondary** node -should use the same set of secrets as the **primary** node. However, setup instructions -prior to the 10.5 release only synchronized the `db_key_base` secret. - -To rectify this error on existing installations, you should **overwrite** the -contents of `/etc/gitlab/gitlab-secrets.json` on each **secondary** node with the -contents of `/etc/gitlab/gitlab-secrets.json` on the **primary** node, then run the -following command on each **secondary** node: - -```shell -sudo gitlab-ctl reconfigure -``` - -If you do not perform this step, you may find that two-factor authentication -[is broken following DR](troubleshooting.md#two-factor-authentication-is-broken-after-a-failover). - -To prevent SSH requests to the newly promoted **primary** node from failing -due to SSH host key mismatch when updating the **primary** node domain's DNS record -you should perform the step to [Manually replicate **primary** SSH host keys](configuration.md#step-2-manually-replicate-the-primary-nodes-ssh-host-keys) in each -**secondary** node. - -## Updating to GitLab 10.3 - -### Support for SSH repository synchronization removed - -In GitLab 10.2, synchronizing secondaries over SSH was deprecated. In 10.3, -support is removed entirely. All installations will switch to the HTTP/HTTPS -cloning method instead. Before updating, ensure that all your Geo nodes are -configured to use this method and that it works for your installation. In -particular, ensure that [Git access over HTTP/HTTPS is enabled](configuration.md#step-5-enable-git-access-over-httphttps). - -Synchronizing repositories over the public Internet using HTTP is insecure, so -you should ensure that you have HTTPS configured before updating. Note that -file synchronization is **also** insecure in these cases! - -## Updating to GitLab 10.2 - -### Secure PostgreSQL replication - -Support for TLS-secured PostgreSQL replication has been added. If you are -currently using PostgreSQL replication across the open internet without an -external means of securing the connection (e.g., a site-to-site VPN), then you -should immediately reconfigure your **primary** and **secondary** PostgreSQL instances -according to the [updated instructions](../setup/database.md). - -If you *are* securing the connections externally and wish to continue doing so, -ensure you include the new option `--sslmode=prefer` in future invocations of -`gitlab-ctl replicate-geo-database`. - -### HTTPS repository sync - -Support for replicating repositories and wikis over HTTP/HTTPS has been added. -Replicating over SSH has been deprecated, and support for this option will be -removed in a future release. - -To switch to HTTP/HTTPS replication, log into the **primary** node as an admin and visit -**Admin Area > Geo** (`/admin/geo/nodes`). For each **secondary** node listed, -press the "Edit" button, change the "Repository cloning" setting from -"SSH (deprecated)" to "HTTP/HTTPS", and press "Save changes". This should take -effect immediately. - -Any new secondaries should be created using HTTP/HTTPS replication - this is the -default setting. - -After you've verified that HTTP/HTTPS replication is working, you should remove -the now-unused SSH keys from your secondaries, as they may cause problems if the -**secondary** node if ever promoted to a **primary** node: - -1. **(secondary)** Login to **all** your **secondary** nodes and run: - - ```ruby - sudo -u git -H rm ~git/.ssh/id_rsa ~git/.ssh/id_rsa.pub - ``` - -### Hashed Storage - -WARNING: -Hashed storage is in **Alpha**. It is considered experimental and not -production-ready. See [Hashed Storage](../../repository_storage_types.md) for more detail. - -If you previously enabled Hashed Storage and migrated all your existing -projects to Hashed Storage, disabling hashed storage will not migrate projects -to their previous project based storage path. As such, once enabled and -migrated we recommend leaving Hashed Storage enabled. - -## Updating to GitLab 10.1 - -WARNING: -Hashed storage is in **Alpha**. It is considered experimental and not -production-ready. See [Hashed Storage](../../repository_storage_types.md) for more detail. - -[Hashed storage](../../repository_storage_types.md) was introduced in -GitLab 10.0, and a [migration path](../../raketasks/storage.md) for -existing repositories was added in GitLab 10.1. - -## Updating to GitLab 10.0 - -In GitLab 10.0 and later, we require all **Geo** systems to [use SSH key lookups via -the database](../../operations/fast_ssh_key_lookup.md) to avoid having to maintain consistency of the -`authorized_keys` file for SSH access. Failing to do this will prevent users -from being able to clone via SSH. - -Note that in older versions of Geo, attachments downloaded on the **secondary** -nodes would be saved to the wrong directory. We recommend that you do the -following to clean this up. - -On the **secondary** Geo nodes, run as root: - -```shell -mv /var/opt/gitlab/gitlab-rails/working /var/opt/gitlab/gitlab-rails/working.old -mkdir /var/opt/gitlab/gitlab-rails/working -chmod 700 /var/opt/gitlab/gitlab-rails/working -chown git:git /var/opt/gitlab/gitlab-rails/working -``` - -You may delete `/var/opt/gitlab/gitlab-rails/working.old` any time. - -Once this is done, we advise restarting GitLab on the **secondary** nodes for the -new working directory to be used: - -```shell -sudo gitlab-ctl restart -``` - -## Updating from GitLab 9.3 or older - -If you started running Geo on GitLab 9.3 or older, we recommend that you -resync your **secondary** PostgreSQL databases to use replication slots. If you -started using Geo with GitLab 9.4 or 10.x, no further action should be -required because replication slots are used by default. However, if you -started with GitLab 9.3 and updated later, you should still follow the -instructions below. - -When in doubt, it doesn't hurt to do a resync. The easiest way to do this in -Omnibus is the following: - -1. Make sure you have Omnibus GitLab on the **primary** server. -1. Run `gitlab-ctl reconfigure` and `gitlab-ctl restart postgresql`. This will enable replication slots on the **primary** database. -1. Check the steps about defining `postgresql['sql_user_password']`, `gitlab_rails['db_password']`. -1. Make sure `postgresql['max_replication_slots']` matches the number of **secondary** Geo nodes locations. -1. Install GitLab on the **secondary** server. -1. Re-run the [database replication process](../setup/database.md#step-3-initiate-the-replication-process). - -## Updating to GitLab 9.0 - -> **IMPORTANT**: -With GitLab 9.0, the PostgreSQL version is updated to 9.6 and manual steps are -required to update the **secondary** nodes and keep the Streaming Replication -working. Downtime is required, so plan ahead. - -The following steps apply only if you update from a 8.17 GitLab version to -9.0+. For previous versions, update to GitLab 8.17 first before attempting to -update to 9.0+. - ---- - -Make sure to follow the steps in the exact order as they appear below and pay -extra attention in what node (either **primary** or **secondary**) you execute them! Each step -is prepended with the relevant node for better clarity: - -1. **(secondary)** Log in to **all** your **secondary** nodes and stop all services: - - ```ruby - sudo gitlab-ctl stop - ``` - -1. **(secondary)** Make a backup of the `recovery.conf` file on **all** - **secondary** nodes to preserve PostgreSQL's credentials: - - ```shell - sudo cp /var/opt/gitlab/postgresql/data/recovery.conf /var/opt/gitlab/ - ``` - -1. **(primary)** Update the **primary** node to GitLab 9.0 following the - [regular update docs](../../../update/README.md). At the end of the - update, the **primary** node will be running with PostgreSQL 9.6. - -1. **(primary)** To prevent a de-synchronization of the repository replication, - stop all services except `postgresql` as we will use it to re-initialize the - **secondary** node's database: - - ```shell - sudo gitlab-ctl stop - sudo gitlab-ctl start postgresql - ``` - -1. **(secondary)** Run the following steps on each of the **secondary** nodes: - - 1. **(secondary)** Stop all services: - - ```shell - sudo gitlab-ctl stop - ``` - - 1. **(secondary)** Prevent running database migrations: - - ```shell - sudo touch /etc/gitlab/skip-auto-migrations - ``` - - 1. **(secondary)** Move the old database to another directory: - - ```shell - sudo mv /var/opt/gitlab/postgresql{,.bak} - ``` - - 1. **(secondary)** Update to GitLab 9.0 following the [regular update docs](../../../update/README.md). - At the end of the update, the node will be running with PostgreSQL 9.6. - - 1. **(secondary)** Make sure all services are up: - - ```shell - sudo gitlab-ctl start - ``` - - 1. **(secondary)** Reconfigure GitLab: - - ```shell - sudo gitlab-ctl reconfigure - ``` - - 1. **(secondary)** Run the PostgreSQL upgrade command: - - ```shell - sudo gitlab-ctl pg-upgrade - ``` - - 1. **(secondary)** See the stored credentials for the database that you will - need to re-initialize the replication: - - ```shell - sudo grep -s primary_conninfo /var/opt/gitlab/recovery.conf - ``` - - 1. **(secondary)** Save the snippet below in a file, let's say `/tmp/replica.sh`. Modify the - embedded paths if necessary: - - ```shell - #!/bin/bash - - PORT="5432" - USER="gitlab_replicator" - echo --------------------------------------------------------------- - echo WARNING: Make sure this script is run from the secondary server - echo --------------------------------------------------------------- - echo - echo Enter the IP or FQDN of the primary PostgreSQL server - read HOST - echo Enter the password for $USER@$HOST - read -s PASSWORD - echo Enter the required sslmode - read SSLMODE - - echo Stopping PostgreSQL and all GitLab services - sudo service gitlab stop - sudo service postgresql stop - - echo Backing up postgresql.conf - sudo -u postgres mv /var/opt/gitlab/postgresql/data/postgresql.conf /var/opt/gitlab/postgresql/ - - echo Cleaning up old cluster directory - sudo -u postgres rm -rf /var/opt/gitlab/postgresql/data - - echo Starting base backup as the replicator user - echo Enter the password for $USER@$HOST - sudo -u postgres /opt/gitlab/embedded/bin/pg_basebackup -h $HOST -D /var/opt/gitlab/postgresql/data -U gitlab_replicator -v -x -P - - echo Writing recovery.conf file - sudo -u postgres bash -c "cat > /var/opt/gitlab/postgresql/data/recovery.conf <<- _EOF1_ - standby_mode = 'on' - primary_conninfo = 'host=$HOST port=$PORT user=$USER password=$PASSWORD sslmode=$SSLMODE' - _EOF1_ - " - - echo Restoring postgresql.conf - sudo -u postgres mv /var/opt/gitlab/postgresql/postgresql.conf /var/opt/gitlab/postgresql/data/ - - echo Starting PostgreSQL - sudo service postgresql start - ``` - - 1. **(secondary)** Run the recovery script using the credentials from the - previous step: - - ```shell - sudo bash /tmp/replica.sh - ``` - - 1. **(secondary)** Reconfigure GitLab: - - ```shell - sudo gitlab-ctl reconfigure - ``` - - 1. **(secondary)** Start all services: - - ```shell - sudo gitlab-ctl start - ``` - - 1. **(secondary)** Repeat the steps for the remaining **secondary** nodes. - -1. **(primary)** After all **secondary** nodes are updated, start all services in - **primary** node: - - ```shell - sudo gitlab-ctl start - ``` - -### Update tracking database on **secondary** node - -After updating a **secondary** node, you might need to run migrations on the -tracking database. The tracking database was added in GitLab 9.1, and is -required in GitLab 10.0 and later. - -1. Run database migrations on tracking database: - - ```shell - sudo gitlab-rake geo:db:migrate - ``` - -1. Repeat this step for each **secondary** node. +The issue is fixed in GitLab 12.1. Be sure to upgrade to GitLab 12.1 or later. diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index 95f67b20ab5..c9fd94bf2b2 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Geo database replication **(PREMIUM ONLY)** +# Geo database replication **(PREMIUM SELF)** NOTE: If your GitLab installation uses external (not managed by Omnibus) PostgreSQL @@ -472,7 +472,7 @@ information, see [High Availability with Omnibus GitLab](../../postgresql/replic ## Patroni support Support for Patroni is intended to replace `repmgr` as a -[highly availabile PostgreSQL solution](../../postgresql/replication_and_failover.md) +[highly available PostgreSQL solution](../../postgresql/replication_and_failover.md) on the primary node, but it can also be used for PostgreSQL HA on a secondary node. @@ -489,7 +489,7 @@ This experimental implementation has the following limitations: - Whenever `gitlab-ctl reconfigure` runs on a Patroni Leader instance, there's a chance the node will be demoted due to the required short-time restart. To avoid this, you can pause auto-failover by running `gitlab-ctl patroni pause`. - After a reconfigure, it unpauses on its own. + After a reconfigure, it resumes on its own. For instructions about how to set up Patroni on the primary node, see the [PostgreSQL replication and failover with Omnibus GitLab](../../postgresql/replication_and_failover.md#patroni) page. @@ -644,8 +644,8 @@ With Patroni it's now possible to support that. In order to migrate the existing 1. [Configure a permanent replication slot](#step-1-configure-patroni-permanent-replication-slot-on-the-primary-site). 1. [Configure a Standby Cluster](#step-2-configure-a-standby-cluster-on-the-secondary-site) on that single node machine. - -You will end up with a "Standby Cluster" with a single node. That allows you to later on add additional patroni nodes + +You will end up with a "Standby Cluster" with a single node. That allows you to later on add additional Patroni nodes by following the same instructions above. ## Troubleshooting diff --git a/doc/administration/geo/setup/external_database.md b/doc/administration/geo/setup/external_database.md index 1fb41fbb53a..8e7d8049467 100644 --- a/doc/administration/geo/setup/external_database.md +++ b/doc/administration/geo/setup/external_database.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Geo with external PostgreSQL instances **(PREMIUM ONLY)** +# Geo with external PostgreSQL instances **(PREMIUM SELF)** This document is relevant if you are using a PostgreSQL instance that is *not managed by Omnibus*. This includes cloud-managed instances like AWS RDS, or diff --git a/doc/administration/git_annex.md b/doc/administration/git_annex.md index e21a8ac7e84..07b7b1730e3 100644 --- a/doc/administration/git_annex.md +++ b/doc/administration/git_annex.md @@ -38,7 +38,7 @@ very fast file copying tool. ## GitLab git-annex Configuration -`git-annex` is disabled by default in GitLab. Below you will find the +`git-annex` is disabled by default in GitLab. Below are the configuration options required to enable it. ### Requirements @@ -60,7 +60,7 @@ sudo yum install epel-release && sudo yum install git-annex ### Configuration for Omnibus packages For Omnibus GitLab packages, only one configuration setting is needed. -The Omnibus package will internally set the correct options in all locations. +The Omnibus package internally sets the correct options in all locations. 1. In `/etc/gitlab/gitlab.rb` add the following line: @@ -148,15 +148,15 @@ To example.com:group/project.git ok ``` -Your files can be found in the `master` branch, but you'll notice that there -are more branches created by the `annex sync` command. +Your files can be found in the `master` branch, but more branches are created +by the `annex sync` command. -Git Annex will also create a new directory at `.git/annex/` and will record the +Git Annex creates a new directory at `.git/annex/` and records the tracked files in the `.git/config` file. The files you assign to be tracked -with `git-annex` will not affect the existing `.git/config` records. The files +with `git-annex` don't affect the existing `.git/config` records. The files are turned into symbolic links that point to data in `.git/annex/objects/`. -The `debian.iso` file in the example will contain the symbolic link: +The `debian.iso` file in the example contain the symbolic link: ```plaintext .git/annex/objects/ZW/1k/SHA256E-s82701--6384039733b5035b559efd5a2e25a493ab6e09aabfd5162cc03f6f0ec238429d.png/SHA256E-s82701--6384039733b5035b559efd5a2e25a493ab6e09aabfd5162cc03f6f0ec238429d.iso @@ -167,7 +167,7 @@ repository. --- -Downloading a single large file is also very simple: +You can download a single large file with these commands: ```shell git clone git@gitlab.example.com:group/project.git @@ -185,7 +185,7 @@ git annex sync --content # sync Git branches and download all the large files ``` By using `git-annex` without GitLab, anyone that can access the server can also -access the files of all projects, but GitLab Annex ensures that you can only +access the files of all projects. GitLab Annex ensures that you can only access files of projects you have access to (developer, maintainer, or owner role). ## How it works diff --git a/doc/administration/git_protocol.md b/doc/administration/git_protocol.md index ca4fa0549d0..e034314f6ce 100644 --- a/doc/administration/git_protocol.md +++ b/doc/administration/git_protocol.md @@ -29,11 +29,11 @@ server to accept the `GIT_PROTOCOL` environment. In installations using [GitLab Helm Charts](https://docs.gitlab.com/charts/) and [All-in-one Docker image](https://docs.gitlab.com/omnibus/docker/), the SSH -service is already configured to accept the `GIT_PROTOCOL` environment and users +service is already configured to accept the `GIT_PROTOCOL` environment. Users need not do anything more. -For Omnibus GitLab and installations from source, you have to manually update -the SSH configuration of your server by adding the line below to the `/etc/ssh/sshd_config` file: +For Omnibus GitLab and installations from source, update +the SSH configuration of your server manually by adding this line to the `/etc/ssh/sshd_config` file: ```plaintext AcceptEnv GIT_PROTOCOL diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 9577fb40abe..6738b912dab 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -40,7 +40,7 @@ The following is a high-level architecture overview of how Gitaly is used. ## Configure Gitaly -The Gitaly service itself is configured via a [TOML configuration file](reference.md). +The Gitaly service itself is configured by using a [TOML configuration file](reference.md). To change Gitaly settings: @@ -121,7 +121,7 @@ The following list depicts the network architecture of Gitaly: - GitLab Shell. - Elasticsearch indexer. - Gitaly itself. -- A Gitaly server must be able to make RPC calls **to itself** via its own +- A Gitaly server must be able to make RPC calls **to itself** by using its own `(Gitaly address, Gitaly token)` pair as specified in `/config/gitlab.yml`. - Authentication is done through a static token which is shared among the Gitaly and GitLab Rails nodes. @@ -502,8 +502,8 @@ If it's excluded, default Git storage directory is used for that storage shard. ### Disable Gitaly where not required (optional) -If you are running Gitaly [as a remote service](#run-gitaly-on-its-own-server), you may want to -disable the local Gitaly service that runs on your GitLab server by default and have it only running +If you run Gitaly [as a remote service](#run-gitaly-on-its-own-server), consider +disabling the local Gitaly service that runs on your GitLab server by default, and only run it where required. Disabling Gitaly on the GitLab instance only makes sense when you run GitLab in a custom cluster configuration, where @@ -538,7 +538,7 @@ To disable Gitaly on a GitLab server: > - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/3160) in GitLab 13.6, outgoing TLS connections to GitLab provide client certificates if configured. Gitaly supports TLS encryption. To communicate with a Gitaly instance that listens for secure -connections, you must use `tls://` URL scheme in the `gitaly_address` of the corresponding +connections, use the `tls://` URL scheme in the `gitaly_address` of the corresponding storage entry in the GitLab configuration. Gitaly provides the same server certificates as client certificates in TLS @@ -935,12 +935,13 @@ Note that `enforced="true"` means that authentication is being enforced. ## Direct Git access bypassing Gitaly -While it is possible to access Gitaly repositories stored on disk directly with a Git client, -it is not advisable because Gitaly is being continuously improved and changed. Theses improvements may invalidate assumptions, resulting in performance degradation, instability, and even data loss. +GitLab doesn't advise directly accessing Gitaly repositories stored on disk with +a Git client, because Gitaly is being continuously improved and changed. These +improvements may invalidate assumptions, resulting in performance degradation, instability, and even data loss. Gitaly has optimizations, such as the [`info/refs` advertisement cache](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/design_diskcache.md), -that rely on Gitaly controlling and monitoring access to repositories via the +that rely on Gitaly controlling and monitoring access to repositories by using the official gRPC interface. Likewise, Praefect has optimizations, such as fault tolerance and distributed reads, that depend on the gRPC interface and database to determine repository state. @@ -979,11 +980,11 @@ lookup. Even when Gitaly is able to re-use an already-running `git` process (for a commit), you still have: - The cost of a network roundtrip to Gitaly. -- Within Gitaly, a write/read roundtrip on the Unix pipes that connect Gitaly to the `git` process. +- Inside Gitaly, a write/read roundtrip on the Unix pipes that connect Gitaly to the `git` process. Using GitLab.com to measure, we reduced the number of Gitaly calls per request until the loss of Rugged's efficiency was no longer felt. It also helped that we run Gitaly itself directly on the Git -file severs, rather than via NFS mounts. This gave us a speed boost that counteracted the negative +file severs, rather than by using NFS mounts. This gave us a speed boost that counteracted the negative effect of not using Rugged anymore. Unfortunately, other deployments of GitLab could not remove NFS like we did on GitLab.com, and they @@ -1018,7 +1019,7 @@ The result of these checks is cached. To see if GitLab can access the repository file system directly, we use the following heuristic: - Gitaly ensures that the file system has a metadata file in its root with a UUID in it. -- Gitaly reports this UUID to GitLab via the `ServerInfo` RPC. +- Gitaly reports this UUID to GitLab by using the `ServerInfo` RPC. - GitLab Rails tries to read the metadata file directly. If it exists, and if the UUID's match, assume we have direct access. @@ -1085,7 +1086,7 @@ app nodes). ### Client side gRPC logs Gitaly uses the [gRPC](https://grpc.io/) RPC framework. The Ruby gRPC -client has its own log file which may contain useful information when +client has its own log file which may contain debugging information when you are seeing Gitaly errors. You can control the log level of the gRPC client with the `GRPC_LOG_LEVEL` environment variable. The default level is `WARN`. @@ -1100,7 +1101,7 @@ sudo GRPC_TRACE=all GRPC_VERBOSITY=DEBUG gitlab-rake gitlab:gitaly:check Sometimes you need to find out which Gitaly RPC created a particular Git process. -One method for doing this is via `DEBUG` logging. However, this needs to be enabled +One method for doing this is by using `DEBUG` logging. However, this needs to be enabled ahead of time and the logs produced are quite verbose. A lightweight method for doing this correlation is by inspecting the environment @@ -1137,16 +1138,19 @@ sum(rate(grpc_client_handled_total[5m])) by (grpc_method) > 0 ### Repository changes fail with a `401 Unauthorized` error -If you're running Gitaly on its own server and notice that users can -successfully clone and fetch repositories (via both SSH and HTTPS), but can't -push to them or make changes to the repository in the web UI without getting a -`401 Unauthorized` message, then it's possible Gitaly is failing to authenticate -with the Gitaly client due to having the [wrong secrets file](#configure-gitaly-servers). +If you run Gitaly on its own server and notice these conditions: + +- Users can successfully clone and fetch repositories by using both SSH and HTTPS. +- Users can't push to repositories, or receive a `401 Unauthorized` message when attempting to + make changes to them in the web UI. + +Gitaly may be failing to authenticate with the Gitaly client because it has the +[wrong secrets file](#configure-gitaly-servers). Confirm the following are all true: - When any user performs a `git push` to any repository on this Gitaly server, it - fails with the following error (note the `401 Unauthorized`): + fails with a `401 Unauthorized` error: ```shell remote: GitLab: 401 Unauthorized @@ -1158,7 +1162,7 @@ Confirm the following are all true: - When any user adds or modifies a file from the repository using the GitLab UI, it immediately fails with a red `401 Unauthorized` banner. - Creating a new project and [initializing it with a README](../../gitlab-basics/create-project.md#blank-projects) - successfully creates the project but doesn't create the README. + successfully creates the project, but doesn't create the README. - When [tailing the logs](https://docs.gitlab.com/omnibus/settings/logs.html#tail-logs-in-a-console-on-the-server) on a Gitaly client and reproducing the error, you get `401` errors when reaching the `/api/v4/internal/allowed` endpoint: @@ -1229,11 +1233,11 @@ update the secrets file on the Gitaly server to match the Gitaly client, then ### Command line tools cannot connect to Gitaly -If you are having trouble connecting to a Gitaly server with command line (CLI) tools, +If you can't connect to a Gitaly server with command line (CLI) tools, and certain actions result in a `14: Connect Failed` error message, -it means that gRPC cannot reach your Gitaly server. +gRPC cannot reach your Gitaly server. -Verify that you can reach Gitaly via TCP: +Verify you can reach Gitaly by using TCP: ```shell sudo gitlab-rake gitlab:tcp_check[GITALY_SERVER_IP,GITALY_LISTEN_PORT] diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index fe8b3e5f566..b4fd607c4be 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -5,12 +5,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Gitaly Cluster **(CORE ONLY)** +# Gitaly Cluster **(FREE SELF)** [Gitaly](index.md), the service that provides storage for Git repositories, can be run in a clustered configuration to increase fault tolerance. In this configuration, every Git repository is stored on every Gitaly node in the -cluster. Multiple clusters (or shards) can be configured. +cluster. Multiple clusters (or storage shards) can be configured. NOTE: Technical support for Gitaly clusters is limited to GitLab Premium and Ultimate @@ -21,7 +21,7 @@ component for running a Gitaly Cluster.  -Using a Gitaly Cluster increase fault tolerance by: +Using a Gitaly Cluster increases fault tolerance by: - Replicating write operations to warm standby Gitaly nodes. - Detecting Gitaly node failures. @@ -53,7 +53,7 @@ Gitaly Cluster supports: - Reporting of possible data loss if replication queue is non-empty. - Marking repositories as [read only](#read-only-mode) if data loss is detected to prevent data inconsistencies. -Follow the [HA Gitaly epic](https://gitlab.com/groups/gitlab-org/-/epics/1489) +Follow the [Gitaly Cluster epic](https://gitlab.com/groups/gitlab-org/-/epics/1489) for improvements including [horizontally distributing reads](https://gitlab.com/groups/gitlab-org/-/epics/2013). @@ -65,7 +65,7 @@ Gitaly Cluster and [Geo](../geo/index.md) both provide redundancy. However the r not aware when Gitaly Cluster is used. - Geo provides [replication](../geo/index.md) and [disaster recovery](../geo/disaster_recovery/index.md) for an entire instance of GitLab. Users know when they are using Geo for - [replication](../geo/index.md). Geo [replicates multiple datatypes](../geo/replication/datatypes.md#limitations-on-replicationverification), + [replication](../geo/index.md). Geo [replicates multiple data types](../geo/replication/datatypes.md#limitations-on-replicationverification), including Git data. The following table outlines the major differences between Gitaly Cluster and Geo: @@ -80,23 +80,65 @@ For more information, see: - [Gitaly architecture](index.md#architecture). - Geo [use cases](../geo/index.md#use-cases) and [architecture](../geo/index.md#architecture). -## Cluster or shard +## Where Gitaly Cluster fits + +GitLab accesses [repositories](../../user/project/repository/index.md) through the configured +[repository storages](../repository_storage_paths.md). Each new repository is stored on one of the +repository storages based on their configured weights. Each repository storage is either: + +- A Gitaly storage served directly by Gitaly. These map to a directory on the file system of a + Gitaly node. +- A [virtual storage](#virtual-storage-or-direct-gitaly-storage) served by Praefect. A virtual + storage is a cluster of Gitaly storages that appear as a single repository storage. + +Virtual storages are a feature of Gitaly Cluster. They support replicating the repositories to +multiple storages for fault tolerance. Virtual storages can improve performance by distributing +requests across Gitaly nodes. Their distributed nature makes it viable to have a single repository +storage in GitLab to simplify repository management. + +## Components of Gitaly Cluster + +Gitaly Cluster consists of multiple components: + +- [Load balancer](#load-balancer) for distributing requests and providing fault-tolerant access to + Praefect nodes. +- [Praefect](#praefect) nodes for managing the cluster and routing requests to Gitaly nodes. +- [PostgreSQL database](#postgresql) for persisting cluster metadata and [PgBouncer](#pgbouncer), + recommended for pooling Praefect's database connections. +- [Gitaly](index.md) nodes to provide repository storage and Git access. + + + +In this example: + +- Repositories are stored on a virtual storage called `storage-1`. +- Three Gitaly nodes provide `storage-1` access: `gitaly-1`, `gitaly-2`, and `gitaly-3`. +- The three Gitaly nodes store data on their file systems. + +### Virtual storage or direct Gitaly storage Gitaly supports multiple models of scaling: - Clustering using Gitaly Cluster, where each repository is stored on multiple Gitaly nodes in the cluster. Read requests are distributed between repository replicas and write requests are - broadcast to repository replicas. -- Sharding using [repository storage paths](../repository_storage_paths.md), where each repository - is stored on the assigned Gitaly node. All requests are routed to this node. + broadcast to repository replicas. GitLab accesses virtual storage. +- Direct access to Gitaly storage using [repository storage paths](../repository_storage_paths.md), + where each repository is stored on the assigned Gitaly node. All requests are routed to this node. + +The following is Gitaly set up to use direct access to Gitaly instead of Gitaly Cluster: + + -| Cluster | Shard | -|:--------------------------------------------------|:----------------------------------------------| -|  |  | +In this example: -Generally, Gitaly Cluster can replace sharded configurations, at the expense of additional storage -needed to store each repository on multiple Gitaly nodes. The benefit of using Gitaly Cluster over -sharding is: +- Each repository is stored on one of three Gitaly storages: `storage-1`, `storage-2`, + or `storage-3`. +- Each storage is serviced by a Gitaly node. +- The three Gitaly nodes store data in three separate hashed storage locations. + +Generally, virtual storage with Gitaly Cluster can replace direct Gitaly storage configurations, at +the expense of additional storage needed to store each repository on multiple Gitaly nodes. The +benefit of using Gitaly Cluster over direct Gitaly storage is: - Improved fault tolerance, because each Gitaly node has a copy of every repository. - Improved resource utilization, reducing the need for over-provisioning for shard-specific peak @@ -105,7 +147,7 @@ sharding is: replicas. - Simpler management, because all Gitaly nodes are identical. -Under some workloads, CPU and memory requirements may require a large fleet of Gitaly nodes and it +Under some workloads, CPU and memory requirements may require a large fleet of Gitaly nodes. It can be uneconomical to have one to one replication factor. A hybrid approach can be used in these instances, where each shard is configured as a smaller @@ -168,7 +210,7 @@ If you are using Google Cloud Platform, SoftLayer, or any other vendor that prov The communication between components is secured with different secrets, which are described below. Before you begin, generate a unique secret for each, and -make note of it. This makes it easy to replace these placeholder tokens +make note of it. This enables you to replace these placeholder tokens with secure tokens as you complete the setup process. 1. `GITLAB_SHELL_SECRET_TOKEN`: this is used by Git hooks to make callback HTTP @@ -260,13 +302,12 @@ this, set the corresponding IP or host address of the PgBouncer instance in - `praefect['database_port']`, for the port. Because PgBouncer manages resources more efficiently, Praefect still requires a -direct connection to the PostgreSQL database because it uses +direct connection to the PostgreSQL database. It uses the [LISTEN](https://www.postgresql.org/docs/11/sql-listen.html) -functionality that is [not supported](https://www.pgbouncer.org/features.html) by +feature that is [not supported](https://www.pgbouncer.org/features.html) by PgBouncer with `pool_mode = transaction`. - -Therefore, `praefect['database_host_no_proxy']` and `praefect['database_port_no_proxy']` -should be set to a direct connection and not a PgBouncer connection. +Set `praefect['database_host_no_proxy']` and `praefect['database_port_no_proxy']` +to a direct connection, and not a PgBouncer connection. Save the changes to `/etc/gitlab/gitlab.rb` and [reconfigure Praefect](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -774,7 +815,7 @@ configuration. ### Load Balancer -In a highly available Gitaly configuration, a load balancer is needed to route +In a fault-tolerant Gitaly configuration, a load balancer is needed to route internal traffic from the GitLab application to the Praefect nodes. The specifics on which load balancer to use or the exact configuration is beyond the scope of the GitLab documentation. @@ -786,7 +827,7 @@ addition to the GitLab nodes. Some requests handled by process. `gitaly-ruby` uses the Gitaly address set in the GitLab server's `git_data_dirs` setting to make this connection. -We hope that if you’re managing HA systems like GitLab, you have a load balancer +We hope that if you’re managing fault-tolerant systems like GitLab, you have a load balancer of choice already. Some examples include [HAProxy](https://www.haproxy.org/) (open-source), [Google Internal Load Balancer](https://cloud.google.com/load-balancing/docs/internal/), [AWS Elastic Load Balancer](https://aws.amazon.com/elasticloadbalancing/), F5 @@ -960,14 +1001,14 @@ To get started quickly: gitlab-ctl reconfigure ``` -1. Set the Grafana admin password. This command prompts you to enter a new +1. Set the Grafana administrator password. This command prompts you to enter a new password: ```shell gitlab-ctl set-grafana-password ``` -1. In your web browser, open `/-/grafana` (e.g. +1. In your web browser, open `/-/grafana` (such as `https://gitlab.example.com/-/grafana`) on your GitLab server. Login using the password you set, and the username `admin`. @@ -975,7 +1016,7 @@ To get started quickly: 1. Go to **Explore** and query `gitlab_build_info` to verify that you are getting metrics from all your machines. -Congratulations! You've configured an observable highly available Praefect +Congratulations! You've configured an observable fault-tolerant Praefect cluster. ## Distributed reads @@ -983,18 +1024,12 @@ cluster. > - Introduced in GitLab 13.1 in [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga) with feature flag `gitaly_distributed_reads` set to disabled. > - [Made generally available and enabled by default](https://gitlab.com/gitlab-org/gitaly/-/issues/2951) in GitLab 13.3. > - [Disabled by default](https://gitlab.com/gitlab-org/gitaly/-/issues/3178) in GitLab 13.5. +> - [Enabled by default](https://gitlab.com/gitlab-org/gitaly/-/issues/3334) in GitLab 13.8. Praefect supports distribution of read operations across Gitaly nodes that are configured for the virtual node. -The feature is disabled by default. To enable distributed reads, the `gitaly_distributed_reads` -[feature flag](../feature_flags.md) must be enabled in a Ruby console: - -```ruby -Feature.enable(:gitaly_distributed_reads) -``` - -If enabled, all RPCs marked with `ACCESSOR` option like +All RPCs marked with `ACCESSOR` option like [GetBlob](https://gitlab.com/gitlab-org/gitaly/-/blob/v12.10.6/proto/blob.proto#L16) are redirected to an up to date and healthy Gitaly node. @@ -1025,9 +1060,8 @@ Praefect guarantees eventual consistency by replicating all writes to secondary after the write to the primary Gitaly node has happened. Praefect can instead provide strong consistency by creating a transaction and writing -changes to all Gitaly nodes at once. Strong consistency is currently in -[alpha](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga) and not enabled by -default. If enabled, transactions are only available for a subset of RPCs. For more +changes to all Gitaly nodes at once. +If enabled, transactions are only available for a subset of RPCs. For more information, see the [strong consistency epic](https://gitlab.com/groups/gitlab-org/-/epics/1189). To enable strong consistency: @@ -1092,7 +1126,7 @@ specific storage nodes to host a repository. support configuring a default replication factor for a virtual storage. The default replication factor is applied to every newly-created repository. -Prafect does not store the actual replication factor, but assigns enough storages to host the repository +Praefect does not store the actual replication factor, but assigns enough storages to host the repository so the desired replication factor is met. If a storage node is later removed from the virtual storage, the replication factor of repositories assigned to the storage is decreased accordingly. @@ -1171,8 +1205,8 @@ To enable writes again, an administrator can: ### Check for data loss -The Praefect `dataloss` sub-command identifies replicas that are likely to be outdated. This is -useful for identifying potential data loss after a failover. The following parameters are +The Praefect `dataloss` sub-command identifies replicas that are likely to be outdated. This can help +identify potential data loss after a failover. The following parameters are available: - `-virtual-storage` that specifies which virtual storage to check. The default behavior is to @@ -1196,7 +1230,7 @@ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.t ``` Repositories which have assigned storage nodes that contain an outdated copy of the repository are listed -in the output. A number of useful information is printed for each repository: +in the output. This information is printed for each repository: - A repository's relative path to the storage directory identifies each repository and groups the related information. @@ -1213,7 +1247,7 @@ in the output. A number of useful information is printed for each repository: Whether a replica is assigned to host the repository is listed with each replica's status. `assigned host` is printed next to replicas which are assigned to store the repository. The text is omitted if the replica contains a copy of -the repository but is not assigned to store the repository. Such replicas won't be kept in-sync by Praefect but may +the repository but is not assigned to store the repository. Such replicas aren't kept in-sync by Praefect, but may act as replication sources to bring assigned replicas up to date. Example output: @@ -1282,7 +1316,7 @@ To check a project's repository checksums across on all Gitaly nodes, run the ### Enable writes or accept data loss -Praefect provides the following subcommands to re-enable writes: +Praefect provides the following sub-commands to re-enable writes: - In GitLab 13.2 and earlier, `enable-writes` to re-enable virtual storage for writes after data recovery attempts. @@ -1324,7 +1358,7 @@ These tools reconcile the outdated repositories to bring them fully up to date a Praefect automatically reconciles repositories that are not up to date. By default, this is done every five minutes. For each outdated repository on a healthy Gitaly node, the Praefect picks a -random, fully up to date replica of the repository on another healthy Gitaly node to replicate from. A +random, fully up-to-date replica of the repository on another healthy Gitaly node to replicate from. A replication job is scheduled only if there are no other replication jobs pending for the target repository. @@ -1383,7 +1417,7 @@ To move repositories to Gitaly Cluster: - The moves are in progress. Re-query the repository move until it completes successfully. - The moves have failed. Most failures are temporary and are solved by rescheduling the move. -1. Once the moves are complete, [query projects](../../api/projects.md#list-all-projects) +1. After the moves are complete, [query projects](../../api/projects.md#list-all-projects) using the API to confirm that all projects have moved. No projects should be returned with `repository_storage` field set to the old storage. diff --git a/doc/administration/gitaly/reference.md b/doc/administration/gitaly/reference.md index 5a004d97220..5105b9ed0d4 100644 --- a/doc/administration/gitaly/reference.md +++ b/doc/administration/gitaly/reference.md @@ -95,13 +95,13 @@ key_path = '/home/git/key.pem' ### Storage -GitLab repositories are grouped into directories known as "storages" -(e.g., `/home/git/repositories`) containing bare repositories managed -by GitLab with names (e.g., `default`). +GitLab repositories are grouped into directories known as storages, such as +`/home/git/repositories`. They contain bare repositories managed +by GitLab with names, such as `default`. These names and paths are also defined in the `gitlab.yml` configuration file of -GitLab. When you run Gitaly on the same machine as GitLab, which is the default -and recommended configuration, storage paths defined in Gitaly's `config.toml` +GitLab. When you run Gitaly on the same machine as GitLab (the default +and recommended configuration) storage paths defined in Gitaly's `config.toml` must match those in `gitlab.yml`. | Name | Type | Required | Description | @@ -232,9 +232,9 @@ The following values configure logging in Gitaly under the `[logging]` section. | ---- | ---- | -------- | ----------- | | `format` | string | no | Log format: `text` or `json`. Default: `text`. | | `level` | string | no | Log level: `debug`, `info`, `warn`, `error`, `fatal`, or `panic`. Default: `info`. | -| `sentry_dsn` | string | no | Sentry DSN for exception monitoring. | +| `sentry_dsn` | string | no | Sentry DSN (Data Source Name) for exception monitoring. | | `sentry_environment` | string | no | [Sentry Environment](https://docs.sentry.io/product/sentry-basics/environments/) for exception monitoring. | -| `ruby_sentry_dsn` | string | no | Sentry DSN for `gitaly-ruby` exception monitoring. | +| `ruby_sentry_dsn` | string | no | Sentry DSN (Data Source Name) for `gitaly-ruby` exception monitoring. | While the main Gitaly application logs go to `stdout`, there are some extra log files that go to a configured directory, like the GitLab Shell logs. diff --git a/doc/administration/housekeeping.md b/doc/administration/housekeeping.md index 178fc438df2..8f369a05fbf 100644 --- a/doc/administration/housekeeping.md +++ b/doc/administration/housekeeping.md @@ -49,7 +49,7 @@ It is ultimately performed by the Gitaly RPC `FetchIntoObjectPool`. This is the current call stack by which it is invoked: 1. `Repositories::HousekeepingService#execute_gitlab_shell_gc` -1. `GitGarbageCollectWorker#perform` +1. `Projects::GitGarbageCollectWorker#perform` 1. `Projects::GitDeduplicationService#fetch_from_source` 1. `ObjectPool#fetch` 1. `ObjectPoolService#fetch` diff --git a/doc/administration/img/time_zone_settings.png b/doc/administration/img/time_zone_settings.png Binary files differnew file mode 100644 index 00000000000..73961b1090c --- /dev/null +++ b/doc/administration/img/time_zone_settings.png diff --git a/doc/administration/index.md b/doc/administration/index.md index f071fde2faa..f70948c63df 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -5,7 +5,7 @@ info: "To determine the technical writer assigned to the Stage/Group associated description: 'Learn how to install, configure, update, and maintain your GitLab instance.' --- -# Administrator documentation **(CORE ONLY)** +# Administrator documentation **(FREE SELF)** Learn how to administer your self-managed GitLab instance. @@ -17,7 +17,7 @@ GitLab has two product distributions available through [different subscriptions] You can [install either GitLab CE or GitLab EE](https://about.gitlab.com/install/ce-or-ee/). However, the features you have access to depend on your chosen [subscription](https://about.gitlab.com/pricing/). -GitLab Community Edition installations have access only to Core features. +GitLab Community Edition installations have access only to Free features. Non-administrator users can't access GitLab administration tools and settings. @@ -80,6 +80,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. emails with S/MIME. - [Enabling and disabling features flags](feature_flags.md): how to enable and disable GitLab features deployed behind feature flags. +- [Application settings cache expiry interval](application_settings_cache.md) #### Customizing GitLab appearance @@ -101,6 +102,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. #### Updating GitLab - [GitLab versions and maintenance policy](../policy/maintenance.md): Understand GitLab versions and releases (Major, Minor, Patch, Security), as well as update recommendations. +- [GitLab in maintenance mode](maintenance_mode/index.md): Put GitLab in maintenance mode. - [Update GitLab](../update/README.md): Update guides to upgrade your installation to a new version. - [Upgrading without downtime](../update/README.md#upgrading-without-downtime): Upgrade to a newer major, minor, or patch version of GitLab without taking your GitLab instance offline. - [Migrate your GitLab CI/CD data to another version of GitLab](../migrate_ci_to_ce/README.md): If you have an old GitLab installation (older than 8.0), follow this guide to migrate your existing GitLab CI/CD data to another version of GitLab. @@ -186,7 +188,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. ## Git configuration options -- [Server hooks](server_hooks.md): Server hooks (on the filesystem) for when webhooks aren't enough. +- [Server hooks](server_hooks.md): Server hooks (on the file system) for when webhooks aren't enough. - [Git LFS configuration](lfs/index.md): Learn how to configure LFS for GitLab. - [Housekeeping](housekeeping.md): Keep your Git repositories tidy and fast. - [Configuring Git Protocol v2](git_protocol.md): Git protocol version 2 support. diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index 0eede5f3ca5..882928a3628 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -127,6 +127,15 @@ It's possible that this limit will be changed to a lower number in the future. - **Max size:** ~1 million characters / ~1 MB +## Size of commit titles and descriptions + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292039) in GitLab 13.9 + +Commits with arbitrarily large messages may be pushed to GitLab, but when +displaying commits, titles (the first line of the commit message) will be +limited to 1KiB, and descriptions (the rest of the message) will be limited to +1MiB. + ## Number of issues in the milestone overview > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/39453) in GitLab 12.10. diff --git a/doc/administration/instance_review.md b/doc/administration/instance_review.md index 8f33ddf3ca5..56a305fa4da 100644 --- a/doc/administration/instance_review.md +++ b/doc/administration/instance_review.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Instance Review -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/6995) in [GitLab Core](https://about.gitlab.com/pricing/) 11.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/6995) in [GitLab Free](https://about.gitlab.com/pricing/) 11.3. If you run a medium-sized self-managed instance (50+ users) of a free version of GitLab, [either Community Edition or unlicensed Enterprise Edition](https://about.gitlab.com/install/ce-or-ee/), diff --git a/doc/administration/integration/kroki.md b/doc/administration/integration/kroki.md index dead5640873..5e458399773 100644 --- a/doc/administration/integration/kroki.md +++ b/doc/administration/integration/kroki.md @@ -1,4 +1,10 @@ -# Kroki diagrams **(CORE ONLY)** +--- +stage: Create +group: Ecosystem +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Kroki diagrams **(FREE SELF)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/241744) in GitLab 13.7. @@ -24,6 +30,8 @@ The **Kroki URL** is the hostname of the server running the container. The [`yuzutech/kroki`](https://hub.docker.com/r/yuzutech/kroki) image contains the following diagrams libraries out-of-the-box: +<!-- vale gitlab.Spelling = NO --> + - [Bytefield](https://bytefield-svg.deepsymmetry.org/) - [Ditaa](http://ditaa.sourceforge.net) - [Erd](https://github.com/BurntSushi/erd) @@ -37,6 +45,8 @@ The [`yuzutech/kroki`](https://hub.docker.com/r/yuzutech/kroki) image contains t - [Vega-Lite](https://github.com/vega/vega-lite) - [WaveDrom](https://wavedrom.com/) +<!-- vale gitlab.Spelling = YES --> + If you want to use additional diagram libraries, read the [Kroki installation](https://docs.kroki.io/kroki/setup/install/#_images) to learn how to start Kroki companion containers. @@ -138,8 +148,12 @@ Rel(banking_system, mainframe, "Uses")  +<!-- vale gitlab.Spelling = NO --> + **Nomnoml** +<!-- vale gitlab.Spelling = YES --> + ```plaintext [nomnoml] .... @@ -159,4 +173,4 @@ Rel(banking_system, mainframe, "Uses") .... ``` - + diff --git a/doc/administration/integration/plantuml.md b/doc/administration/integration/plantuml.md index cd61dc9a2bf..5f6222f1169 100644 --- a/doc/administration/integration/plantuml.md +++ b/doc/administration/integration/plantuml.md @@ -10,13 +10,13 @@ type: reference, howto > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/8537) in GitLab 8.16. When [PlantUML](https://plantuml.com) integration is enabled and configured in -GitLab we are able to create simple diagrams in AsciiDoc and Markdown documents +GitLab you can create diagrams in AsciiDoc and Markdown documents created in snippets, wikis, and repositories. ## PlantUML Server -Before you can enable PlantUML in GitLab; you need to set up your own PlantUML -server that will generate the diagrams. +Before you can enable PlantUML in GitLab; set up your own PlantUML +server to generate the diagrams. ### Docker @@ -26,12 +26,11 @@ With Docker, you can just run a container like this: docker run -d --name plantuml -p 8080:8080 plantuml/plantuml-server:tomcat ``` -The **PlantUML URL** will be the hostname of the server running the container. +The **PlantUML URL** is the hostname of the server running the container. -When running GitLab in Docker, it will need to have access to the PlantUML container. -The easiest way to achieve that is by using [Docker Compose](https://docs.docker.com/compose/). - -A simple `docker-compose.yml` file would be: +When running GitLab in Docker, it must have access to the PlantUML container. +You can achieve that by using [Docker Compose](https://docs.docker.com/compose/). +A basic `docker-compose.yml` file could contain: ```yaml version: "3" @@ -47,13 +46,12 @@ services: container_name: plantuml ``` -In this scenario, PlantUML will be accessible for GitLab at the URL +In this scenario, PlantUML is accessible to GitLab at the URL `http://plantuml:8080/`. ### Debian/Ubuntu -Installing and configuring your -own PlantUML server is easy in Debian/Ubuntu distributions using Tomcat. +You can also install and configure a PlantUML server in Debian/Ubuntu distributions using Tomcat. First you need to create a `plantuml.war` file from the source code: @@ -64,8 +62,7 @@ cd plantuml-server mvn package ``` -The above sequence of commands will generate a WAR file that can be deployed -using Tomcat: +The above sequence of commands generates a `.war` file you can deploy with Tomcat: ```shell sudo apt-get install tomcat8 @@ -74,17 +71,18 @@ sudo chown tomcat8:tomcat8 /var/lib/tomcat8/webapps/plantuml.war sudo service tomcat8 restart ``` -Once the Tomcat service restarts the PlantUML service will be ready and +After the Tomcat service restarts, the PlantUML service is ready and listening for requests on port 8080: ```plaintext http://localhost:8080/plantuml ``` -you can change these defaults by editing the `/etc/tomcat8/server.xml` file. +To change these defaults, edit the `/etc/tomcat8/server.xml` file. -Note that the default URL is different than when using the Docker-based image, -where the service is available at the root of URL with no relative path. Adjust +NOTE: +The default URL is different when using this approach. The Docker-based image +makes the service available at the root URL, with no relative path. Adjust the configuration below accordingly. ### Making local PlantUML accessible using custom GitLab setup @@ -112,7 +110,7 @@ To activate the changes, run the following command: sudo gitlab-ctl reconfigure ``` -Note that the redirection through GitLab **must** be configured +Note that the redirection through GitLab must be configured when running [GitLab with TLS](https://docs.gitlab.com/omnibus/settings/ssl.html) due to PlantUML's use of the insecure HTTP protocol. Newer browsers such as [Google Chrome 86+](https://www.chromestatus.com/feature/4926989725073408) @@ -120,7 +118,7 @@ do not load insecure HTTP resources on a page served over HTTPS. ### Security -PlantUML has features that allows fetching network resources. +PlantUML has features that allow fetching network resources. ```plaintext @startuml @@ -136,18 +134,18 @@ stop; ## GitLab You need to enable PlantUML integration from Settings under Admin Area. To do -that, login with an Admin account and do following: +that, sign in with an Administrator account, and then do following: -- In GitLab, go to **Admin Area > Settings > General**. -- Expand the **PlantUML** section. -- Check **Enable PlantUML** checkbox. -- Set the PlantUML instance as `https://gitlab.example.com/-/plantuml/`. +1. In GitLab, go to **Admin Area > Settings > General**. +1. Expand the **PlantUML** section. +1. Select the **Enable PlantUML** check box. +1. Set the PlantUML instance as `https://gitlab.example.com/-/plantuml/`. NOTE: If you are using a PlantUML server running v1.2020.9 and above (for example, [plantuml.com](https://plantuml.com)), set the `PLANTUML_ENCODING` -environment variable to enable the `deflate` compression. On Omnibus, -this can be done set in `/etc/gitlab.rb`: +environment variable to enable the `deflate` compression. On Omnibus GitLab, +this can be set in `/etc/gitlab.rb`: ```ruby gitlab_rails['env'] = { 'PLANTUML_ENCODING' => 'deflate' } @@ -191,9 +189,11 @@ our AsciiDoc snippets, wikis, and repositories using delimited blocks: Alice -> Bob: hi ``` - You can also use the `uml::` directive for compatibility with [`sphinxcontrib-plantuml`](https://pypi.org/project/sphinxcontrib-plantuml/), but please note that we currently only support the `caption` option. + You can also use the `uml::` directive for compatibility with + [`sphinxcontrib-plantuml`](https://pypi.org/project/sphinxcontrib-plantuml/), + but GitLab only supports the `caption` option. -The above blocks will be converted to an HTML image tag with source pointing to the +The above blocks are converted to an HTML image tag with source pointing to the PlantUML instance. If the PlantUML server is correctly configured, this should render a nice diagram instead of the block: @@ -202,12 +202,18 @@ Bob -> Alice : hello Alice -> Bob : hi ``` -Inside the block you can add any of the supported diagrams by PlantUML such as -[Sequence](https://plantuml.com/sequence-diagram), [Use Case](https://plantuml.com/use-case-diagram), -[Class](https://plantuml.com/class-diagram), [Activity](https://plantuml.com/activity-diagram-legacy), -[Component](https://plantuml.com/component-diagram), [State](https://plantuml.com/state-diagram), -and [Object](https://plantuml.com/object-diagram) diagrams. You do not need to use the PlantUML -diagram delimiters `@startuml`/`@enduml` as these are replaced by the AsciiDoc `plantuml` block. +Inside the block you can add any of the diagrams PlantUML supports, such as: + +- [Sequence](https://plantuml.com/sequence-diagram) +- [Use Case](https://plantuml.com/use-case-diagram) +- [Class](https://plantuml.com/class-diagram) +- [Activity](https://plantuml.com/activity-diagram-legacy) +- [Component](https://plantuml.com/component-diagram) +- [State](https://plantuml.com/state-diagram), +- [Object](https://plantuml.com/object-diagram) + +You do not need to use the PlantUML +diagram delimiters `@startuml`/`@enduml`, as these are replaced by the AsciiDoc `plantuml` block. Some parameters can be added to the AsciiDoc block definition: @@ -217,4 +223,4 @@ Some parameters can be added to the AsciiDoc block definition: - `width`: Width attribute added to the image tag. - `height`: Height attribute added to the image tag. -Markdown does not support any parameters and will always use PNG format. +Markdown does not support any parameters and always uses PNG format. diff --git a/doc/administration/integration/terminal.md b/doc/administration/integration/terminal.md index f4c242b6e72..0f26eb83d17 100644 --- a/doc/administration/integration/terminal.md +++ b/doc/administration/integration/terminal.md @@ -8,14 +8,14 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/7690) in GitLab 8.15. -NOTE: -Only project maintainers and owners can access web terminals. - With the introduction of the [Kubernetes integration](../../user/project/clusters/index.md), -GitLab gained the ability to store and use credentials for a Kubernetes cluster. -One of the things it uses these credentials for is providing access to +GitLab can store and use credentials for a Kubernetes cluster. +GitLab uses these credentials to provide access to [web terminals](../../ci/environments/index.md#web-terminals) for environments. +NOTE: +Only project maintainers and owners can access web terminals. + ## How it works A detailed overview of the architecture of web terminals and how they work @@ -53,15 +53,13 @@ detail below. NOTE: AWS Elastic Load Balancers (ELBs) do not support web sockets. -AWS Application Load Balancers (ALBs) must be used if you want web terminals -to work. See [AWS Elastic Load Balancing Product Comparison](https://aws.amazon.com/elasticloadbalancing/features/#compare) +If you want web terminals to work, use AWS Application Load Balancers (ALBs). +Read [AWS Elastic Load Balancing Product Comparison](https://aws.amazon.com/elasticloadbalancing/features/#compare) for more information. As web terminals use WebSockets, every HTTP/HTTPS reverse proxy in front of -Workhorse needs to be configured to pass the `Connection` and `Upgrade` headers -through to the next one in the chain. If you installed GitLab using Omnibus, or -from source, starting with GitLab 8.15, this should be done by the default -configuration, so there's no need for you to do anything. +Workhorse must be configured to pass the `Connection` and `Upgrade` headers +to the next one in the chain. GitLab is configured by default to do so. However, if you run a [load balancer](../load_balancer.md) in front of GitLab, you may need to make some changes to your configuration. These @@ -73,17 +71,17 @@ guides document the necessary steps for a selection of popular reverse proxies: - [Varnish](https://varnish-cache.org/docs/4.1/users-guide/vcl-example-websockets.html) Workhorse doesn't let WebSocket requests through to non-WebSocket endpoints, so -it's safe to enable support for these headers globally. If you'd rather had a -narrower set of rules, you can restrict it to URLs ending with `/terminal.ws` -(although this may still have a few false positives). +it's safe to enable support for these headers globally. If you prefer a +narrower set of rules, you can restrict it to URLs ending with `/terminal.ws`. +This approach may still result in a few false positives. If you installed from source, or have made any configuration changes to your Omnibus installation before upgrading to 8.15, you may need to make some changes -to your configuration. See the [Upgrading Community Edition and Enterprise -Edition from source](../../update/upgrading_from_source.md#nginx-configuration) -document for more details. +to your configuration. Read +[Upgrading Community Edition and Enterprise Edition from source](../../update/upgrading_from_source.md#nginx-configuration) +for more details. -If you'd like to disable web terminal support in GitLab, just stop passing +To disable web terminal support in GitLab, stop passing the `Connection` and `Upgrade` hop-by-hop headers in the *first* HTTP reverse proxy in the chain. For most users, this is the NGINX server bundled with Omnibus GitLab, in which case, you need to: @@ -104,4 +102,6 @@ they receive a `Connection failed` message. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/8413) in GitLab 8.17. Terminal sessions, by default, do not expire. -You can limit terminal session lifetime in your GitLab instance. To do so, navigate to [**Admin Area > Settings > Web terminal**](../../user/admin_area/settings/index.md#general), and set a `max session time`. +You can limit terminal session lifetime in your GitLab instance. To do so, +go to [**Admin Area > Settings > Web terminal**](../../user/admin_area/settings/index.md#general), +and set a `max session time`. diff --git a/doc/administration/invalidate_markdown_cache.md b/doc/administration/invalidate_markdown_cache.md index 75bee6e0c9a..266e9d44ff7 100644 --- a/doc/administration/invalidate_markdown_cache.md +++ b/doc/administration/invalidate_markdown_cache.md @@ -8,10 +8,10 @@ type: reference # Invalidate Markdown Cache For performance reasons, GitLab caches the HTML version of Markdown text -(e.g. issue and merge request descriptions, comments). It's possible -that these cached versions become outdated, for example -when the `external_url` configuration option is changed - causing links -in the cached text to refer to the old URL. +in fields like comments, issue descriptions, and merge request descriptions. These +cached versions can become outdated, such as +when the `external_url` configuration option is changed. Links +in the cached text would still refer to the old URL. To avoid this problem, the administrator can invalidate the existing cache by increasing the `local_markdown_version` setting in application settings. This can diff --git a/doc/administration/issue_closing_pattern.md b/doc/administration/issue_closing_pattern.md index 6a5b90ae963..32d367bb15e 100644 --- a/doc/administration/issue_closing_pattern.md +++ b/doc/administration/issue_closing_pattern.md @@ -5,7 +5,7 @@ info: "To determine the technical writer assigned to the Stage/Group associated type: reference --- -# Issue closing pattern **(CORE ONLY)** +# Issue closing pattern **(FREE SELF)** NOTE: This is the administration documentation. There is a separate [user documentation](../user/project/issues/managing_issues.md#closing-issues-automatically) diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 5d07e7d1b26..ba2cb05b449 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -91,7 +91,7 @@ _The artifacts are stored by default in > [GitLab Premium](https://about.gitlab.com/pricing/) 9.4. > - Since version 9.5, artifacts are [browsable](../ci/pipelines/job_artifacts.md#browsing-artifacts), > when object storage is enabled. 9.4 lacks this feature. -> - Since version 10.6, available in [GitLab Core](https://about.gitlab.com/pricing/) +> - Since version 10.6, available in [GitLab Free](https://about.gitlab.com/pricing/). > - Since version 11.0, we support `direct_upload` to S3. If you don't want to use the local disk where GitLab is installed to store the @@ -375,7 +375,7 @@ default artifacts expiration setting, which you can find in the [CI/CD Admin set > Introduced in GitLab 10.3. To disable [the dependencies validation](../ci/yaml/README.md#when-a-dependent-job-fails), -you can enable the `ci_disable_validates_dependencies` feature flag from a Rails console. +you can enable the `ci_validate_build_dependencies_override` feature flag from a Rails console. **In Omnibus installations:** @@ -388,7 +388,7 @@ you can enable the `ci_disable_validates_dependencies` feature flag from a Rails 1. Enable the feature flag to disable the validation: ```ruby - Feature.enable(:ci_disable_validates_dependencies) + Feature.enable(:ci_validate_build_dependencies_override) ``` **In installations from source:** @@ -403,7 +403,7 @@ you can enable the `ci_disable_validates_dependencies` feature flag from a Rails 1. Enable the feature flag to disable the validation: ```ruby - Feature.enable(:ci_disable_validates_dependencies) + Feature.enable(:ci_validate_build_dependencies_override) ``` ## Set the maximum file size of the artifacts diff --git a/doc/administration/job_logs.md b/doc/administration/job_logs.md index b2c6864e671..1afadcaf668 100644 --- a/doc/administration/job_logs.md +++ b/doc/administration/job_logs.md @@ -73,7 +73,7 @@ these steps to move the logs to a new location without losing any data. ``` Use `--ignore-existing` so you don't override new job logs with older versions of the same log. -1. Unpause continuous integration data processing by editing `/etc/gitlab/gitlab.rb` and removing the `sidekiq` setting you updated earlier. +1. Resume continuous integration data processing by editing `/etc/gitlab/gitlab.rb` and removing the `sidekiq` setting you updated earlier. 1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Remove the old job logs storage location: diff --git a/doc/administration/job_traces.md b/doc/administration/job_traces.md deleted file mode 100644 index 2dbcf5d6705..00000000000 --- a/doc/administration/job_traces.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'job_logs.md' ---- - -This document was moved to [another location](job_logs.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/lfs/index.md b/doc/administration/lfs/index.md index bc349536a0c..862c26abac8 100644 --- a/doc/administration/lfs/index.md +++ b/doc/administration/lfs/index.md @@ -6,17 +6,17 @@ type: reference, howto disqus_identifier: 'https://docs.gitlab.com/ee/workflow/lfs/lfs_administration.html' --- -# GitLab Git Large File Storage (LFS) Administration **(CORE ONLY)** +# GitLab Git Large File Storage (LFS) Administration **(FREE SELF)** > - Git LFS is supported in GitLab starting with version 8.2. > - Support for object storage, such as AWS S3, was introduced in 10.0. > - LFS is enabled in GitLab self-managed instances by default. -Documentation on how to use Git LFS are under [Managing large binary files with Git LFS doc](../../topics/git/lfs/index.md). +Documentation about how to use Git LFS are under [Managing large binary files with Git LFS doc](../../topics/git/lfs/index.md). ## Requirements -- Users need to install [Git LFS client](https://git-lfs.github.com) version 1.0.1 and up. +- Users need to install [Git LFS client](https://git-lfs.github.com) version 1.0.1 or later. ## Configuration @@ -25,9 +25,9 @@ GitLab is installed on. There are various configuration options to help GitLab server administrators: -- Enabling/disabling Git LFS support -- Changing the location of LFS object storage -- Setting up object storage supported by [Fog](http://fog.io/about/provider_documentation.html) +- Enabling/disabling Git LFS support. +- Changing the location of LFS object storage. +- Setting up object storage supported by [Fog](http://fog.io/about/provider_documentation.html). ### Configuration for Omnibus installations @@ -43,7 +43,7 @@ gitlab_rails['lfs_enabled'] = false gitlab_rails['lfs_storage_path'] = "/mnt/storage/lfs-objects" ``` -After you update settings in `/etc/gitlab/gitlab.rb`, make sure to run [Omnibus GitLab reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). +After you update settings in `/etc/gitlab/gitlab.rb`, run [Omnibus GitLab reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). ### Configuration for installations from source @@ -58,14 +58,12 @@ In `config/gitlab.yml`: ## Storing LFS objects in remote object storage -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/2760) in [GitLab Premium](https://about.gitlab.com/pricing/) 10.0. Brought to GitLab Core in 10.7. - -It is possible to store LFS objects in remote object storage which allows you -to offload local hard disk R/W operations, and free up disk space significantly. +You can store LFS objects in remote object storage. This allows you +to offload reads and writes to the local disk, and free up disk space significantly. GitLab is tightly integrated with `Fog`, so you can refer to its [documentation](http://fog.io/about/provider_documentation.html) to check which storage services can be integrated with GitLab. You can also use external object storage in a private local network. For example, -[MinIO](https://min.io/) is a standalone object storage service, is easy to set up, and works well with GitLab instances. +[MinIO](https://min.io/) is a standalone object storage service that works with GitLab instances. GitLab provides two different options for the uploading mechanism: "Direct upload" and "Background upload". @@ -78,26 +76,26 @@ This section describes the earlier configuration format. **Option 1. Direct upload** -1. User pushes an `lfs` file to the GitLab instance -1. GitLab-workhorse uploads the file directly to the external object storage -1. GitLab-workhorse notifies GitLab-rails that the upload process is complete +1. User pushes an `lfs` file to the GitLab instance. +1. GitLab-workhorse uploads the file directly to the external object storage. +1. GitLab-workhorse notifies GitLab-rails that the upload process is complete. **Option 2. Background upload** -1. User pushes an `lfs` file to the GitLab instance -1. GitLab-rails stores the file in the local file storage -1. GitLab-rails then uploads the file to the external object storage asynchronously +1. User pushes an `lfs` file to the GitLab instance. +1. GitLab-rails stores the file in the local file storage. +1. GitLab-rails then uploads the file to the external object storage asynchronously. The following general settings are supported. -| Setting | Description | Default | -|---------|-------------|---------| -| `enabled` | Enable/disable object storage | `false` | -| `remote_directory` | The bucket name where LFS objects will be stored| | -| `direct_upload` | Set to true to enable direct upload of LFS without the need of local shared storage. Option may be removed once we decide to support only single storage for all files. | `false` | -| `background_upload` | Set to false to disable automatic upload. Option may be removed once upload is direct to S3 | `true` | -| `proxy_download` | Set to true to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` | -| `connection` | Various connection options described below | | +| Setting | Description | Default | +|---------------------|-------------|---------| +| `enabled` | Enable/disable object storage. | `false` | +| `remote_directory` | The bucket name where LFS objects are stored. | | +| `direct_upload` | Set to true to enable direct upload of LFS without the need of local shared storage. Option may be removed after we decide to support only single storage for all files. | `false` | +| `background_upload` | Set to false to disable automatic upload. Option may be removed once upload is direct to S3. | `true` | +| `proxy_download` | Set to true to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data. | `false` | +| `connection` | Various connection options described below. | | See [the available connection settings for different providers](../object_storage.md#connection-settings). @@ -131,10 +129,9 @@ end ### S3 for Omnibus installations -On Omnibus installations, the settings are prefixed by `lfs_object_store_`: +On Omnibus GitLab installations, the settings are prefixed by `lfs_object_store_`: -1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with - the values you want: +1. Edit `/etc/gitlab/gitlab.rb` and add the following lines, replacing values based on your needs: ```ruby gitlab_rails['lfs_object_store_enabled'] = true @@ -151,17 +148,17 @@ On Omnibus installations, the settings are prefixed by `lfs_object_store_`: } ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Migrate any existing local LFS objects to the object storage: ```shell gitlab-rake gitlab:lfs:migrate ``` - This will migrate existing LFS objects to object storage. New LFS objects - will be forwarded to object storage unless + This migrates existing LFS objects to object storage. New LFS objects + are forwarded to object storage unless `gitlab_rails['lfs_object_store_background_upload']` and `gitlab_rails['lfs_object_store_direct_upload']` is set to `false`. -1. Optional: Verify all files migrated properly. +1. (Optional) Verify all files migrated properly. From [PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database) (`sudo gitlab-psql -d gitlabhq_production`) verify `objectstg` below (where `file_store=2`) has count of all artifacts: @@ -204,17 +201,17 @@ For source installations the settings are nested under `lfs:` and then path_style: true ``` -1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. +1. Save the file, and then [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. 1. Migrate any existing local LFS objects to the object storage: ```shell sudo -u git -H bundle exec rake gitlab:lfs:migrate RAILS_ENV=production ``` - This will migrate existing LFS objects to object storage. New LFS objects - will be forwarded to object storage unless `background_upload` and `direct_upload` is set to + This migrates existing LFS objects to object storage. New LFS objects + are forwarded to object storage unless `background_upload` and `direct_upload` is set to `false`. -1. Optional: Verify all files migrated properly. +1. (Optional) Verify all files migrated properly. From PostgreSQL console (`sudo -u git -H psql -d gitlabhq_production`) verify `objectstg` below (where `file_store=2`) has count of all artifacts: ```shell @@ -233,7 +230,7 @@ For source installations the settings are nested under `lfs:` and then ### Migrating back to local storage -In order to migrate back to local storage: +To migrate back to local storage: 1. Set both `direct_upload` and `background_upload` to `false` under the LFS object storage settings. Don't forget to restart GitLab. 1. Run `rake gitlab:lfs:migrate_to_local` on your console. @@ -241,17 +238,18 @@ In order to migrate back to local storage: ## Storage statistics -You can see the total storage used for LFS objects on groups and projects -in the administration area, as well as through the [groups](../../api/groups.md) -and [projects APIs](../../api/projects.md). +You can see the total storage used for LFS objects on groups and projects: + +- In the administration area. +- In the [groups](../../api/groups.md) and [projects APIs](../../api/projects.md). ## Troubleshooting: `Google::Apis::TransmissionError: execution expired` If LFS integration is configured with Google Cloud Storage and background uploads (`background_upload: true` and `direct_upload: false`), Sidekiq workers may encounter this error. This is because the uploading timed out with very large files. -LFS files up to 6Gb can be uploaded without any extra steps, otherwise you need to use the following workaround. +LFS files up to 6 GB can be uploaded without any extra steps, otherwise you need to use the following workaround. -Log into Rails console: +Sign in to Rails console: ```shell sudo gitlab-rails console @@ -282,6 +280,6 @@ See more information in [!19581](https://gitlab.com/gitlab-org/gitlab-foss/-/mer - Support for removing unreferenced LFS objects was added in 8.14 onward. - LFS authentications via SSH was added with GitLab 8.12. -- Only compatible with the Git LFS client versions 1.1.0 and up, or 1.0.2. -- The storage statistics currently count each LFS object multiple times for +- Only compatible with the Git LFS client versions 1.1.0 and later, or 1.0.2. +- The storage statistics count each LFS object multiple times for every project linking to it. diff --git a/doc/administration/lfs/lfs_administration.md b/doc/administration/lfs/lfs_administration.md deleted file mode 100644 index a275efce5bb..00000000000 --- a/doc/administration/lfs/lfs_administration.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'index.md' ---- - -This document was moved to [another location](index.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/lfs/manage_large_binaries_with_git_lfs.md b/doc/administration/lfs/manage_large_binaries_with_git_lfs.md deleted file mode 100644 index 69c401acb86..00000000000 --- a/doc/administration/lfs/manage_large_binaries_with_git_lfs.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: '../../topics/git/lfs/index.md' ---- - -This document was moved to [another location](../../topics/git/lfs/index.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/lfs/migrate_from_git_annex_to_git_lfs.md b/doc/administration/lfs/migrate_from_git_annex_to_git_lfs.md deleted file mode 100644 index 16095c1dbf6..00000000000 --- a/doc/administration/lfs/migrate_from_git_annex_to_git_lfs.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: '../../topics/git/lfs/migrate_from_git_annex_to_git_lfs.md' ---- - -This document was moved to [another location](../../topics/git/lfs/migrate_from_git_annex_to_git_lfs.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/logs.md b/doc/administration/logs.md index 3d5ba903941..36172869493 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -4,7 +4,7 @@ group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Log system +# Log system **(FREE SELF)** GitLab has an advanced log system where everything is logged, so you can analyze your instance using various system log files. In addition to @@ -381,10 +381,11 @@ only. For example: } ``` -## `audit_json.log` +## `audit_json.log` **(FREE)** NOTE: -Most log entries only exist in [GitLab Starter](https://about.gitlab.com/pricing/), however a few exist in GitLab Core. +GitLab Free tracks a small number of different audit events. +[GitLab Premium](https://about.gitlab.com/pricing/) tracks many more. This file lives in `/var/log/gitlab/gitlab-rails/audit_json.log` for Omnibus GitLab packages or in `/home/git/gitlab/log/audit_json.log` for @@ -772,7 +773,7 @@ are generated: - For Omnibus GitLab packages, in `/var/log/gitlab/gitlab-rails/web_exporter.log`. - For installations from source, in `/home/git/gitlab/log/web_exporter.log`. -## `database_load_balancing.log` **(PREMIUM ONLY)** +## `database_load_balancing.log` **(PREMIUM SELF)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/15442) in GitLab 12.3. @@ -782,7 +783,7 @@ It's stored at: - `/var/log/gitlab/gitlab-rails/database_load_balancing.log` for Omnibus GitLab packages. - `/home/git/gitlab/log/database_load_balancing.log` for installations from source. -## `elasticsearch.log` **(STARTER ONLY)** +## `elasticsearch.log` **(PREMIUM SELF)** > Introduced in GitLab 12.6. @@ -856,7 +857,7 @@ For example: { "severity":"INFO", "time":"2020-04-22T16:04:50.691Z","correlation_id":"04f1366e-57a1-45b8-88c1-b00b23dc3616","class":"Projects::ImportExport::ExportService","current_user":"John Doe","project_full_path":"group1/test-export","file_path":"/path/to/archive","gc_stats":{"count":{"before":127,"after":127,"diff":0},"heap_allocated_pages":{"before":10369,"after":10369,"diff":0},"heap_sorted_length":{"before":10369,"after":10369,"diff":0},"heap_allocatable_pages":{"before":0,"after":0,"diff":0},"heap_available_slots":{"before":4226409,"after":4226409,"diff":0},"heap_live_slots":{"before":2542709,"after":2641420,"diff":98711},"heap_free_slots":{"before":1683700,"after":1584989,"diff":-98711},"heap_final_slots":{"before":0,"after":0,"diff":0},"heap_marked_slots":{"before":2542704,"after":2542704,"diff":0},"heap_eden_pages":{"before":10369,"after":10369,"diff":0},"heap_tomb_pages":{"before":0,"after":0,"diff":0},"total_allocated_pages":{"before":10369,"after":10369,"diff":0},"total_freed_pages":{"before":0,"after":0,"diff":0},"total_allocated_objects":{"before":24896308,"after":24995019,"diff":98711},"total_freed_objects":{"before":22353599,"after":22353599,"diff":0},"malloc_increase_bytes":{"before":140032,"after":6650240,"diff":6510208},"malloc_increase_bytes_limit":{"before":25804104,"after":25804104,"diff":0},"minor_gc_count":{"before":94,"after":94,"diff":0},"major_gc_count":{"before":33,"after":33,"diff":0},"remembered_wb_unprotected_objects":{"before":34284,"after":34284,"diff":0},"remembered_wb_unprotected_objects_limit":{"before":68568,"after":68568,"diff":0},"old_objects":{"before":2404725,"after":2404725,"diff":0},"old_objects_limit":{"before":4809450,"after":4809450,"diff":0},"oldmalloc_increase_bytes":{"before":140032,"after":6650240,"diff":6510208},"oldmalloc_increase_bytes_limit":{"before":68537556,"after":68537556,"diff":0}},"time_to_finish":0.12298400001600385,"number_of_sql_calls":70,"memory_usage":"0.0 MiB","label":"process_48616"} ``` -## `geo.log` **(PREMIUM ONLY)** +## `geo.log` **(PREMIUM SELF)** > Introduced in 9.5. @@ -973,9 +974,13 @@ For Omnibus GitLab installations, Redis logs reside in `/var/log/gitlab/redis/`. For Omnibus GitLab installations, Alertmanager logs reside in `/var/log/gitlab/alertmanager/`. +<!-- vale gitlab.Spelling = NO --> + ## Crond Logs -For Omnibus GitLab installations, `crond` logs reside in `/var/log/gitlab/crond/`. +For Omnibus GitLab installations, crond logs reside in `/var/log/gitlab/crond/`. + +<!-- vale gitlab.Spelling = YES --> ## Grafana Logs @@ -983,7 +988,7 @@ For Omnibus GitLab installations, Grafana logs reside in `/var/log/gitlab/grafan ## LogRotate Logs -For Omnibus GitLab installations, logrotate logs reside in `/var/log/gitlab/logrotate/`. +For Omnibus GitLab installations, `logrotate` logs reside in `/var/log/gitlab/logrotate/`. ## GitLab Monitor Logs @@ -1009,7 +1014,7 @@ installations from source. Performance bar statistics (currently only duration of SQL queries) are recorded in that file. For example: ```json -{"severity":"INFO","time":"2020-12-04T09:29:44.592Z","correlation_id":"33680b1490ccd35981b03639c406a697","filename":"app/models/ci/pipeline.rb","filenum":"395","method":"each_with_object","request_id":"rYHomD0VJS4","duration_ms":26.889,"type": "sql"} +{"severity":"INFO","time":"2020-12-04T09:29:44.592Z","correlation_id":"33680b1490ccd35981b03639c406a697","filename":"app/models/ci/pipeline.rb","method_path":"app/models/ci/pipeline.rb:each_with_object","request_id":"rYHomD0VJS4","duration_ms":26.889,"count":2,"type": "sql"} ``` These statistics are logged on .com only, disabled on self-deployments. @@ -1026,14 +1031,14 @@ GitLab Support often asks for one of these, and maintains the required tools. ### Briefly tail the main logs If the bug or error is readily reproducible, save the main GitLab logs -[to a file](troubleshooting/linux_cheat_sheet.md#files--dirs) while reproducing the +[to a file](troubleshooting/linux_cheat_sheet.md#files-and-directories) while reproducing the problem a few times: ```shell sudo gitlab-ctl tail | tee /tmp/<case-ID-and-keywords>.log ``` -Conclude the log gathering with <kbd>Ctrl</kbd> + <kbd>C</kbd>. +Conclude the log gathering with <kbd>Control</kbd> + <kbd>C</kbd>. ### GitLabSOS diff --git a/doc/administration/maintenance_mode/index.md b/doc/administration/maintenance_mode/index.md new file mode 100644 index 00000000000..c37eff53549 --- /dev/null +++ b/doc/administration/maintenance_mode/index.md @@ -0,0 +1,137 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# GitLab in maintenance mode **(PREMIUM SELF)** + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2149) in GitLab Premium 13.9. + +In maintenance mode, most write operations are disabled at the application level. +This means that GitLab is effectively in a read-only mode for all non-administrative +users (administrators are still able to edit application settings). Regular users +are able to log in to GitLab, view the interface and perform other read-only +operations, such as `git clone` or `git pull`. + +## Enable maintenance mode + +There are three ways to enable maintenance mode as an administrator: + +- **Web UI**: + 1. Navigate to the **Admin Area > Application settings > General** and toggle + the maintenance mode. You can optionally add a message for the banner as well. + 1. Click **Save** for the changes to take effect. + +- **API**: + + ```shell + curl --request PUT --header "PRIVATE-TOKEN:$ADMIN_TOKEN" "<gitlab-url>/api/v4/application/settings?maintenance_mode=true" + ``` + +- [**Rails console**](../operations/rails_console.md#starting-a-rails-console-session): + + ```ruby + ::Gitlab::CurrentSettings.update_attributes!(maintenance_mode: true) + ::Gitlab::CurrentSettings.update_attributes!(maintenance_mode_message: "New message") + ``` + +## Disable maintenance mode + +There are three ways to disable maintenance mode: + +- **Web UI**: + 1. Navigate to the **Admin Area > Application settings > General** and toggle + the maintenance mode. You can optionally add a message for the banner as well. + 1. Click **Save** for the changes to take effect. + +- **API**: + + ```shell + curl --request PUT --header "PRIVATE-TOKEN:$ADMIN_TOKEN" "<gitlab-url>/api/v4/application/settings?maintenance_mode=false" + ``` + +- [**Rails console**](../operations/rails_console.md#starting-a-rails-console-session): + + ```ruby + ::Gitlab::CurrentSettings.update_attributes!(maintenance_mode: false) + ``` + +## Behavior in maintenance mode + +When maintenance mode is enabled, a banner is displayed at the top of the page. +The banner can be customized with a specific message. + +An error is displayed when a user tries to perform a write operation that isn't allowed. +The API will return a 403 or 503 error for failed write requests. + +NOTE: +In some cases, the visual feedback from an action could be misleading, for example +when starring a project, the **Star** button changes to show the **Unstar** action, +however, this is only the frontend update, and it doesn't take into account the +failed status of the POST request. These visual bugs are to be fixed +[in follow-up iterations](https://gitlab.com/gitlab-org/gitlab/-/issues/295197). + +### Application settings + +In maintenance mode, admins can edit the application settings. This will allow +them to disable maintenance mode after it's been enabled. + +### Logging in/out + +All users can log in and out of the GitLab instance. + +### CI/CD + +In maintenance mode: + +- No new jobs are started. Already running jobs stay in 'running' + status but their logs are no longer updated. +- If the job has been in 'running' state for longer than the project's time limit, + it will **not** time out. +- Pipelines cannot be started, retried or canceled in maintenance mode. + No new jobs can be created either. + +Once maintenance mode is disabled, new jobs are picked up again. The jobs that were +in the running state before enabling maintenance mode, will resume, and their logs +will start getting updated again. + +### Git actions + +All read-only Git operations will continue to work in maintenance mode, for example +`git clone` and `git pull`, but all write operations will fail, both through the CLI +and Web IDE. + +Geo secondaries are read-only instances that allow Git pushes because they are +proxied to the primary instance. However, in maintenance mode, Git pushes to +both primary and secondaries will fail. + +### Merge requests, issues, epics + +All write actions except those mentioned above will fail. So, in maintenance mode, a user cannot update merge requests, issues, etc. + +### Container Registry + +In maintenance mode, `docker push` is blocked, but `docker pull` is available. + +### Auto Deploys + +It is recommended to disable auto deploys during maintenance mode, and enable +them once maintenance mode is disabled. + +### Background jobs + +Background jobs (cron jobs, Sidekiq) will continue running as is, because maintenance +mode does not disable any background jobs. + +[During a planned Geo failover](../geo/disaster_recovery/planned_failover.md#prevent-updates-to-the-primary-node), +it is recommended that you disable all cron jobs except for those related to Geo. + +You can monitor queues and disable jobs in **Admin Area > Monitoring > Background Jobs**. + +### Geo secondaries + +The maintenance mode setting will be propagated to the secondary as they sync up. +It is important that you do not disable replication before enabling maintenance mode. + +Replication and verification will continue to work in maintenance mode. diff --git a/doc/administration/maven_packages.md b/doc/administration/maven_packages.md deleted file mode 100644 index 690b6785941..00000000000 --- a/doc/administration/maven_packages.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'packages/index.md' ---- - -This document was moved to [another location](packages/index.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/maven_repository.md b/doc/administration/maven_repository.md deleted file mode 100644 index 690b6785941..00000000000 --- a/doc/administration/maven_repository.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'packages/index.md' ---- - -This document was moved to [another location](packages/index.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/merge_request_diffs.md b/doc/administration/merge_request_diffs.md index 92fdba6216c..1133bff204c 100644 --- a/doc/administration/merge_request_diffs.md +++ b/doc/administration/merge_request_diffs.md @@ -5,7 +5,7 @@ info: "To determine the technical writer assigned to the Stage/Group associated type: reference --- -# Merge request diffs storage **(CORE ONLY)** +# Merge request diffs storage **(FREE SELF)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/52568) in GitLab 11.8. @@ -31,8 +31,8 @@ that only [stores outdated diffs](#alternative-in-database-storage) outside of d gitlab_rails['external_diffs_enabled'] = true ``` -1. _The external diffs will be stored in - `/var/opt/gitlab/gitlab-rails/shared/external-diffs`._ To change the path, +1. The external diffs are stored in + `/var/opt/gitlab/gitlab-rails/shared/external-diffs`. To change the path, for example, to `/mnt/storage/external-diffs`, edit `/etc/gitlab/gitlab.rb` and add the following line: @@ -52,8 +52,8 @@ that only [stores outdated diffs](#alternative-in-database-storage) outside of d enabled: true ``` -1. _The external diffs will be stored in - `/home/git/gitlab/shared/external-diffs`._ To change the path, for example, +1. The external diffs are stored in + `/home/git/gitlab/shared/external-diffs`. To change the path, for example, to `/mnt/storage/external-diffs`, edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: @@ -68,7 +68,7 @@ that only [stores outdated diffs](#alternative-in-database-storage) outside of d ## Using object storage WARNING: -Currently migrating to object storage is **non-reversible** +Migrating to object storage is not reversible. Instead of storing the external diffs on disk, we recommended the use of an object store like AWS S3 instead. This configuration relies on valid AWS credentials to @@ -114,7 +114,7 @@ then `object_store:`. On Omnibus installations, they are prefixed by | Setting | Description | Default | |---------|-------------|---------| | `enabled` | Enable/disable object storage | `false` | -| `remote_directory` | The bucket name where external diffs will be stored| | +| `remote_directory` | The bucket name where external diffs are stored| | | `direct_upload` | Set to `true` to enable direct upload of external diffs without the need of local shared storage. Option may be removed once we decide to support only single storage for all files. | `false` | | `background_upload` | Set to `false` to disable automatic upload. Option may be removed once upload is direct to S3 | `true` | | `proxy_download` | Set to `true` to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` | @@ -141,7 +141,7 @@ See [the available connection settings for different providers](object_storage.m } ``` - Note that, if you are using AWS IAM profiles, be sure to omit the + If you are using AWS IAM profiles, omit the AWS access key and secret access key/value pairs. For example: ```ruby @@ -206,8 +206,8 @@ To enable this feature, perform the following steps: 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. -With this feature enabled, diffs will initially stored in the database, rather -than externally. They will be moved to external storage once any of these +With this feature enabled, diffs are initially stored in the database, rather +than externally. They are moved to external storage after any of these conditions become true: - A newer version of the merge request diff exists @@ -225,15 +225,15 @@ of some merge request diffs when [external diffs in object storage](#object-stor were enabled. This mainly affected imported merge requests, and was resolved with [this merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/31005). -If you are using object storage, have never used on-disk storage for external -diffs, the "changes" tab for some merge requests fails to load with a 500 error, +If you are using object storage, or have never used on-disk storage for external +diffs, the **Changes** tab for some merge requests fails to load with a 500 error, and the exception for that error is of this form: ```plain Errno::ENOENT (No such file or directory @ rb_sysopen - /var/opt/gitlab/gitlab-rails/shared/external-diffs/merge_request_diffs/mr-6167082/diff-8199789) ``` -Then you are affected by this issue. Since it's not possible to safely determine +Then you are affected by this issue. Because it's not possible to safely determine all these conditions automatically, we've provided a Rake task in GitLab v13.2.0 that you can run manually to correct the data: diff --git a/doc/administration/monitoring/github_imports.md b/doc/administration/monitoring/github_imports.md index cf7c105e8cf..736ff299a86 100644 --- a/doc/administration/monitoring/github_imports.md +++ b/doc/administration/monitoring/github_imports.md @@ -4,7 +4,7 @@ group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Monitoring GitHub imports +# Monitoring GitHub imports **(FREE SELF)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/14731) in GitLab 10.2. diff --git a/doc/administration/monitoring/gitlab_instance_administration_project/index.md b/doc/administration/monitoring/gitlab_instance_administration_project/index.md deleted file mode 100644 index 59837dcdb20..00000000000 --- a/doc/administration/monitoring/gitlab_instance_administration_project/index.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: '../gitlab_self_monitoring_project/index.md' ---- - -This document was moved to [another location](../gitlab_self_monitoring_project/index.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md index 13d42ca2ee6..f7eb03b75ef 100644 --- a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md +++ b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md @@ -4,7 +4,7 @@ group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab self monitoring project +# GitLab self monitoring project **(FREE SELF)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32351) in GitLab 12.7, behind a disabled feature flag (`self_monitoring_project`). > - The feature flag was removed and the Self Monitoring Project was [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/198511) in GitLab 12.8. diff --git a/doc/administration/monitoring/index.md b/doc/administration/monitoring/index.md index 68dbe9f3cf9..ac09acf2d2e 100644 --- a/doc/administration/monitoring/index.md +++ b/doc/administration/monitoring/index.md @@ -4,7 +4,7 @@ group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Monitoring GitLab +# Monitoring GitLab **(FREE SELF)** Explore our features to monitor your GitLab instance: diff --git a/doc/administration/monitoring/ip_whitelist.md b/doc/administration/monitoring/ip_whitelist.md index 83fbec86f57..91f9c5d560f 100644 --- a/doc/administration/monitoring/ip_whitelist.md +++ b/doc/administration/monitoring/ip_whitelist.md @@ -4,7 +4,7 @@ group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# IP whitelist +# IP whitelist **(FREE SELF)** > Introduced in GitLab 9.4. diff --git a/doc/administration/monitoring/performance/gitlab_configuration.md b/doc/administration/monitoring/performance/gitlab_configuration.md index dd4481434a6..03eba67292d 100644 --- a/doc/administration/monitoring/performance/gitlab_configuration.md +++ b/doc/administration/monitoring/performance/gitlab_configuration.md @@ -4,22 +4,24 @@ group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab Configuration +# GitLab Configuration **(FREE SELF)** GitLab Performance Monitoring is disabled by default. To enable it and change any of its settings: 1. Navigate to **Admin Area > Settings > Metrics and profiling** - (`/admin/application_settings/metrics_and_profiling`): - -  - -1. You must restart all GitLab processes for the changes to take effect: + (`/admin/application_settings/metrics_and_profiling`). +1. Add the necessary configuration changes. +1. Restart all GitLab for the changes to take effect: - For Omnibus GitLab installations: `sudo gitlab-ctl restart` - For installations from source: `sudo service gitlab restart` -## Pending Migrations +NOTE: +Removed [in GitLab 13.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/30786). Use the +[Prometheus integration](../prometheus/index.md) instead. + +## Pending migrations When any migrations are pending, the metrics are disabled until the migrations have been performed. diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md index e76e81c6499..e6dda2ac87a 100644 --- a/doc/administration/monitoring/performance/grafana_configuration.md +++ b/doc/administration/monitoring/performance/grafana_configuration.md @@ -4,7 +4,7 @@ group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Grafana Configuration +# Grafana Configuration **(FREE SELF)** [Grafana](https://grafana.com/) is a tool that enables you to visualize time series metrics through graphs and dashboards. GitLab writes performance data to Prometheus, @@ -17,42 +17,38 @@ or Grafana supplies package repositories (Yum/Apt) for easy installation. See [Grafana installation documentation](https://grafana.com/docs/grafana/latest/installation/) for detailed steps. -Before starting Grafana for the first time, set the admin user +Before starting Grafana for the first time, set the administration user and password in `/etc/grafana/grafana.ini`. If you don't, the default password is `admin`. ## Configuration -1. Log in to Grafana as the admin user. -1. Expand the menu by clicking the Grafana logo in the top left corner. -1. Choose **Data Sources** from the menu. -1. Click **Add new** in the top bar: -  -1. Edit the data source to fit your needs: -  -1. Click **Save**. +1. Log in to Grafana as the administration user. +1. Select **Data Sources** from the **Configuration** menu. +1. Select the **Add data source** button. +1. Select the required data source type. For example, [Prometheus](../prometheus/index.md#prometheus-as-a-grafana-data-source). +1. Complete the details for the data source and select the **Save & Test** button. + +Grafana should indicate the data source is working. ## Import Dashboards You can now import a set of default dashboards to start displaying useful information. GitLab has published a set of default -[Grafana dashboards](https://gitlab.com/gitlab-org/grafana-dashboards) to get you started. -Clone the repository, or download a ZIP file or tarball, then follow these steps to import each -JSON file individually: +[Grafana dashboards](https://gitlab.com/gitlab-org/grafana-dashboards) to get you started. To use +them: -1. Log in to Grafana as the admin user. -1. Open the dashboard dropdown menu and click **Import**: -  -1. Click **Choose file**, and browse to the location where you downloaded or - cloned the dashboard repository. Select a JSON file to import: -  -1. After the dashboard is imported, click the **Save dashboard** icon in the top bar: -  +1. Clone the repository, or download a ZIP file or tarball. +1. Follow these steps to import each JSON file individually: - If you don't save the dashboard after importing it, the dashboard is removed - when you navigate away from the page. + 1. Log in to Grafana as the administration user. + 1. Select **Manage** from the **Dashboards** menu. + 1. Select the **Import** button, then the **Upload JSON file** button. + 1. Locate the JSON file to import and select **Choose for Upload**. Select the **Import** button. + 1. After the dashboard is imported, select the **Save dashboard** icon in the top bar. -Repeat this process for each dashboard you wish to import. +If you don't save the dashboard after importing it, the dashboard is removed +when you navigate away from the page. Repeat this process for each dashboard you wish to import. Alternatively, you can import all the dashboards into your Grafana instance. For more information about this process, see the @@ -103,7 +99,7 @@ sudo mv /var/opt/gitlab/grafana/data.bak.xxxx/ /var/opt/gitlab/grafana/data/ However, you should **not** reinstate your old data _except_ under one of the following conditions: -1. If you're certain that you changed your default admin password when you enabled Grafana. +1. If you're certain that you changed your default administration password when you enabled Grafana. 1. If you run GitLab in a private network, accessed only by trusted users, and your Grafana login page has not been exposed to the internet. @@ -121,8 +117,6 @@ existing data and dashboards. For more information and further mitigation details, please refer to our [blog post on the security release](https://about.gitlab.com/releases/2019/08/12/critical-security-release-gitlab-12-dot-1-dot-6-released/). ---- - Read more on: - [Introduction to GitLab Performance Monitoring](index.md) diff --git a/doc/administration/monitoring/performance/img/grafana_dashboard_dropdown.png b/doc/administration/monitoring/performance/img/grafana_dashboard_dropdown.png Binary files differdeleted file mode 100644 index 51eef90068d..00000000000 --- a/doc/administration/monitoring/performance/img/grafana_dashboard_dropdown.png +++ /dev/null diff --git a/doc/administration/monitoring/performance/img/grafana_dashboard_import.png b/doc/administration/monitoring/performance/img/grafana_dashboard_import.png Binary files differdeleted file mode 100644 index fd639ee0eb8..00000000000 --- a/doc/administration/monitoring/performance/img/grafana_dashboard_import.png +++ /dev/null diff --git a/doc/administration/monitoring/performance/img/grafana_data_source_configuration.png b/doc/administration/monitoring/performance/img/grafana_data_source_configuration.png Binary files differdeleted file mode 100644 index a98e0ed1e7d..00000000000 --- a/doc/administration/monitoring/performance/img/grafana_data_source_configuration.png +++ /dev/null diff --git a/doc/administration/monitoring/performance/img/grafana_data_source_empty.png b/doc/administration/monitoring/performance/img/grafana_data_source_empty.png Binary files differdeleted file mode 100644 index 549ada8343e..00000000000 --- a/doc/administration/monitoring/performance/img/grafana_data_source_empty.png +++ /dev/null diff --git a/doc/administration/monitoring/performance/img/grafana_save_icon.png b/doc/administration/monitoring/performance/img/grafana_save_icon.png Binary files differdeleted file mode 100644 index 68a071f5ae2..00000000000 --- a/doc/administration/monitoring/performance/img/grafana_save_icon.png +++ /dev/null diff --git a/doc/administration/monitoring/performance/img/metrics_gitlab_configuration_settings.png b/doc/administration/monitoring/performance/img/metrics_gitlab_configuration_settings.png Binary files differdeleted file mode 100644 index 13bfd097b81..00000000000 --- a/doc/administration/monitoring/performance/img/metrics_gitlab_configuration_settings.png +++ /dev/null diff --git a/doc/administration/monitoring/performance/img/performance_bar_external_http_calls.png b/doc/administration/monitoring/performance/img/performance_bar_external_http_calls.png Binary files differnew file mode 100644 index 00000000000..45f2b0956e9 --- /dev/null +++ b/doc/administration/monitoring/performance/img/performance_bar_external_http_calls.png diff --git a/doc/administration/monitoring/performance/index.md b/doc/administration/monitoring/performance/index.md index 072baa16e29..cdf78811092 100644 --- a/doc/administration/monitoring/performance/index.md +++ b/doc/administration/monitoring/performance/index.md @@ -4,7 +4,7 @@ group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab Performance Monitoring +# GitLab Performance Monitoring **(FREE SELF)** GitLab comes with its own application performance measuring system as of GitLab 8.4, called "GitLab Performance Monitoring". GitLab Performance Monitoring is available in both the diff --git a/doc/administration/monitoring/performance/influxdb_configuration.md b/doc/administration/monitoring/performance/influxdb_configuration.md deleted file mode 100644 index d793e014a18..00000000000 --- a/doc/administration/monitoring/performance/influxdb_configuration.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: 'prometheus.md' ---- - -Support for InfluxDB was removed in GitLab 13.0. Use [Prometheus](prometheus.md) for performance monitoring. diff --git a/doc/administration/monitoring/performance/influxdb_schema.md b/doc/administration/monitoring/performance/influxdb_schema.md deleted file mode 100644 index d793e014a18..00000000000 --- a/doc/administration/monitoring/performance/influxdb_schema.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: 'prometheus.md' ---- - -Support for InfluxDB was removed in GitLab 13.0. Use [Prometheus](prometheus.md) for performance monitoring. diff --git a/doc/administration/monitoring/performance/introduction.md b/doc/administration/monitoring/performance/introduction.md deleted file mode 100644 index a275efce5bb..00000000000 --- a/doc/administration/monitoring/performance/introduction.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'index.md' ---- - -This document was moved to [another location](index.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index a214660bd5c..0b67f8c11ba 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -4,7 +4,7 @@ group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Performance Bar +# Performance Bar **(FREE SELF)** You can display the GitLab Performance Bar to see statistics for the performance of a page. When activated, it looks as follows: @@ -31,6 +31,10 @@ From left to right, it displays:  - **Elasticsearch calls**: the time taken (in milliseconds) and the total number of Elasticsearch calls. Click to display a modal window with more details. +- **External HTTP calls**: the time taken (in milliseconds) and the total + number of external calls to other systems. Click to display a modal window + with more details +  - **Load timings** of the page: if your browser supports load timings (Chromium and Chrome) several values in milliseconds, separated by slashes. Click to display a modal window with more details. The values, from left to right: diff --git a/doc/administration/monitoring/performance/prometheus.md b/doc/administration/monitoring/performance/prometheus.md deleted file mode 100644 index 2978de865e2..00000000000 --- a/doc/administration/monitoring/performance/prometheus.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: '../prometheus/index.md' ---- - -This document was moved to [another location](../prometheus/index.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/monitoring/performance/request_profiling.md b/doc/administration/monitoring/performance/request_profiling.md index 23af50dec11..f0ac8f991d3 100644 --- a/doc/administration/monitoring/performance/request_profiling.md +++ b/doc/administration/monitoring/performance/request_profiling.md @@ -4,7 +4,7 @@ group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Request Profiling +# Request Profiling **(FREE SELF)** To profile a request: diff --git a/doc/administration/monitoring/prometheus/gitlab_exporter.md b/doc/administration/monitoring/prometheus/gitlab_exporter.md index 5add842bccc..589fa799cca 100644 --- a/doc/administration/monitoring/prometheus/gitlab_exporter.md +++ b/doc/administration/monitoring/prometheus/gitlab_exporter.md @@ -4,7 +4,7 @@ group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab exporter +# GitLab exporter **(FREE SELF)** >- Available since [Omnibus GitLab 8.17](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/1132). >- Renamed from `GitLab monitor exporter` to `GitLab exporter` in [GitLab 12.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/16511). diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index 4493c1677bc..cd26ef1aef2 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -4,7 +4,7 @@ group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab Prometheus metrics +# GitLab Prometheus metrics **(FREE SELF)** To enable the GitLab Prometheus metrics: @@ -33,87 +33,95 @@ For enabling and viewing metrics from Sidekiq nodes, see [Sidekiq metrics](#side The following metrics are available: -| Metric | Type | Since | Description | Labels | -|:---------------------------------------------------------------|:----------|-----------------------:|:----------------------------------------------------------------------------------------------------|:----------------------------------------------------| -| `gitlab_banzai_cached_render_real_duration_seconds` | Histogram | 9.4 | Duration of rendering Markdown into HTML when cached output exists | `controller`, `action` | -| `gitlab_banzai_cacheless_render_real_duration_seconds` | Histogram | 9.4 | Duration of rendering Markdown into HTML when cached output does not exist | `controller`, `action` | -| `gitlab_cache_misses_total` | Counter | 10.2 | Cache read miss | `controller`, `action` | -| `gitlab_cache_operation_duration_seconds` | Histogram | 10.2 | Cache access time | | -| `gitlab_cache_operations_total` | Counter | 12.2 | Cache operations by controller or action | `controller`, `action`, `operation` | -| `gitlab_ci_pipeline_creation_duration_seconds` | Histogram | 13.0 | Time in seconds it takes to create a CI/CD pipeline | | -| `gitlab_ci_pipeline_size_builds` | Histogram | 13.1 | Total number of builds within a pipeline grouped by a pipeline source | `source` | -| `job_waiter_started_total` | Counter | 12.9 | Number of batches of jobs started where a web request is waiting for the jobs to complete | `worker` | -| `job_waiter_timeouts_total` | Counter | 12.9 | Number of batches of jobs that timed out where a web request is waiting for the jobs to complete | `worker` | -| `gitlab_database_transaction_seconds` | Histogram | 12.1 | Time spent in database transactions, in seconds | | -| `gitlab_method_call_duration_seconds` | Histogram | 10.2 | Method calls real duration | `controller`, `action`, `module`, `method` | -| `gitlab_page_out_of_bounds` | Counter | 12.8 | Counter for the PageLimiter pagination limit being hit | `controller`, `action`, `bot` | -| `gitlab_rails_queue_duration_seconds` | Histogram | 9.4 | Measures latency between GitLab Workhorse forwarding a request to Rails | | -| `gitlab_sql_duration_seconds` | Histogram | 10.2 | SQL execution time, excluding `SCHEMA` operations and `BEGIN` / `COMMIT` | | -| `gitlab_ruby_threads_max_expected_threads` | Gauge | 13.3 | Maximum number of threads expected to be running and performing application work | -| `gitlab_ruby_threads_running_threads` | Gauge | 13.3 | Number of running Ruby threads by name | -| `gitlab_transaction_cache_<key>_count_total` | Counter | 10.2 | Counter for total Rails cache calls (per key) | | -| `gitlab_transaction_cache_<key>_duration_total` | Counter | 10.2 | Counter for total time (seconds) spent in Rails cache calls (per key) | | -| `gitlab_transaction_cache_count_total` | Counter | 10.2 | Counter for total Rails cache calls (aggregate) | | -| `gitlab_transaction_cache_duration_total` | Counter | 10.2 | Counter for total time (seconds) spent in Rails cache calls (aggregate) | | -| `gitlab_transaction_cache_read_hit_count_total` | Counter | 10.2 | Counter for cache hits for Rails cache calls | `controller`, `action` | -| `gitlab_transaction_cache_read_miss_count_total` | Counter | 10.2 | Counter for cache misses for Rails cache calls | `controller`, `action` | -| `gitlab_transaction_duration_seconds` | Histogram | 10.2 | Duration for all transactions (`gitlab_transaction_*` metrics) | `controller`, `action` | -| `gitlab_transaction_event_build_found_total` | Counter | 9.4 | Counter for build found for API /jobs/request | | -| `gitlab_transaction_event_build_invalid_total` | Counter | 9.4 | Counter for build invalid due to concurrency conflict for API /jobs/request | | -| `gitlab_transaction_event_build_not_found_cached_total` | Counter | 9.4 | Counter for cached response of build not found for API /jobs/request | | -| `gitlab_transaction_event_build_not_found_total` | Counter | 9.4 | Counter for build not found for API /jobs/request | | -| `gitlab_transaction_event_change_default_branch_total` | Counter | 9.4 | Counter when default branch is changed for any repository | | -| `gitlab_transaction_event_create_repository_total` | Counter | 9.4 | Counter when any repository is created | | -| `gitlab_transaction_event_etag_caching_cache_hit_total` | Counter | 9.4 | Counter for ETag cache hit. | `endpoint` | -| `gitlab_transaction_event_etag_caching_header_missing_total` | Counter | 9.4 | Counter for ETag cache miss - header missing | `endpoint` | -| `gitlab_transaction_event_etag_caching_key_not_found_total` | Counter | 9.4 | Counter for ETag cache miss - key not found | `endpoint` | -| `gitlab_transaction_event_etag_caching_middleware_used_total` | Counter | 9.4 | Counter for ETag middleware accessed | `endpoint` | -| `gitlab_transaction_event_etag_caching_resource_changed_total` | Counter | 9.4 | Counter for ETag cache miss - resource changed | `endpoint` | -| `gitlab_transaction_event_fork_repository_total` | Counter | 9.4 | Counter for repository forks (RepositoryForkWorker). Only incremented when source repository exists | | -| `gitlab_transaction_event_import_repository_total` | Counter | 9.4 | Counter for repository imports (RepositoryImportWorker) | | -| `gitlab_transaction_event_push_branch_total` | Counter | 9.4 | Counter for all branch pushes | | -| `gitlab_transaction_event_push_commit_total` | Counter | 9.4 | Counter for commits | `branch` | -| `gitlab_transaction_event_push_tag_total` | Counter | 9.4 | Counter for tag pushes | | -| `gitlab_transaction_event_rails_exception_total` | Counter | 9.4 | Counter for number of rails exceptions | | -| `gitlab_transaction_event_receive_email_total` | Counter | 9.4 | Counter for received emails | `handler` | -| `gitlab_transaction_event_remote_mirrors_failed_total` | Counter | 10.8 | Counter for failed remote mirrors | | -| `gitlab_transaction_event_remote_mirrors_finished_total` | Counter | 10.8 | Counter for finished remote mirrors | | -| `gitlab_transaction_event_remote_mirrors_running_total` | Counter | 10.8 | Counter for running remote mirrors | | -| `gitlab_transaction_event_remove_branch_total` | Counter | 9.4 | Counter when a branch is removed for any repository | | -| `gitlab_transaction_event_remove_repository_total` | Counter | 9.4 | Counter when a repository is removed | | -| `gitlab_transaction_event_remove_tag_total` | Counter | 9.4 | Counter when a tag is remove for any repository | | -| `gitlab_transaction_event_sidekiq_exception_total` | Counter | 9.4 | Counter of Sidekiq exceptions | | -| `gitlab_transaction_event_stuck_import_jobs_total` | Counter | 9.4 | Count of stuck import jobs | `projects_without_jid_count`, `projects_with_jid_count` | -| `gitlab_transaction_event_update_build_total` | Counter | 9.4 | Counter for update build for API `/jobs/request/:id` | | -| `gitlab_transaction_new_redis_connections_total` | Counter | 9.4 | Counter for new Redis connections | | -| `gitlab_transaction_queue_duration_total` | Counter | 9.4 | Duration jobs were enqueued before processing | | -| `gitlab_transaction_rails_queue_duration_total` | Counter | 9.4 | Measures latency between GitLab Workhorse forwarding a request to Rails | `controller`, `action` | -| `gitlab_transaction_view_duration_total` | Counter | 9.4 | Duration for views | `controller`, `action`, `view` | -| `gitlab_view_rendering_duration_seconds` | Histogram | 10.2 | Duration for views (histogram) | `controller`, `action`, `view` | -| `http_requests_total` | Counter | 9.4 | Rack request count | `method`, `status` | -| `http_request_duration_seconds` | Histogram | 9.4 | HTTP response time from rack middleware | `method` | -| `gitlab_transaction_db_count_total` | Counter | 13.1 | Counter for total number of SQL calls | `controller`, `action` | -| `gitlab_transaction_db_write_count_total` | Counter | 13.1 | Counter for total number of write SQL calls | `controller`, `action` | -| `gitlab_transaction_db_cached_count_total` | Counter | 13.1 | Counter for total number of cached SQL calls | `controller`, `action` | -| `http_elasticsearch_requests_duration_seconds` **(STARTER)** | Histogram | 13.1 | Elasticsearch requests duration during web transactions | `controller`, `action` | -| `http_elasticsearch_requests_total` **(STARTER)** | Counter | 13.1 | Elasticsearch requests count during web transactions | `controller`, `action` | -| `pipelines_created_total` | Counter | 9.4 | Counter of pipelines created | | -| `rack_uncaught_errors_total` | Counter | 9.4 | Rack connections handling uncaught errors count | | -| `user_session_logins_total` | Counter | 9.4 | Counter of how many users have logged in since GitLab was started or restarted | | -| `upload_file_does_not_exist` | Counter | 10.7 in EE, 11.5 in CE | Number of times an upload record could not find its file | | -| `failed_login_captcha_total` | Gauge | 11.0 | Counter of failed CAPTCHA attempts during login | | -| `successful_login_captcha_total` | Gauge | 11.0 | Counter of successful CAPTCHA attempts during login | | -| `auto_devops_pipelines_completed_total` | Counter | 12.7 | Counter of completed Auto DevOps pipelines, labeled by status | | -| `gitlab_metrics_dashboard_processing_time_ms` | Summary | 12.10 | Metrics dashboard processing time in milliseconds | service, stages | -| `action_cable_active_connections` | Gauge | 13.4 | Number of ActionCable WS clients currently connected | `server_mode` | -| `action_cable_pool_min_size` | Gauge | 13.4 | Minimum number of worker threads in ActionCable thread pool | `server_mode` | -| `action_cable_pool_max_size` | Gauge | 13.4 | Maximum number of worker threads in ActionCable thread pool | `server_mode` | -| `action_cable_pool_current_size` | Gauge | 13.4 | Current number of worker threads in ActionCable thread pool | `server_mode` | -| `action_cable_pool_largest_size` | Gauge | 13.4 | Largest number of worker threads observed so far in ActionCable thread pool | `server_mode` | -| `action_cable_pool_pending_tasks` | Gauge | 13.4 | Number of tasks waiting to be executed in ActionCable thread pool | `server_mode` | -| `action_cable_pool_tasks_total` | Gauge | 13.4 | Total number of tasks executed in ActionCable thread pool | `server_mode` | -| `gitlab_issuable_fast_count_by_state_total` | Counter | 13.5 | Total number of row count operations on issue/merge request list pages | | -| `gitlab_issuable_fast_count_by_state_failures_total` | Counter | 13.5 | Number of soft-failed row count operations on issue/merge request list pages | | +| Metric | Type | Since | Description | Labels | +|:---------------------------------------------------------------|:----------|------:|:------------------------------------------------------------------------------------------------------|:--------------------------------------------------------| +| `gitlab_banzai_cached_render_real_duration_seconds` | Histogram | 9.4 | Duration of rendering Markdown into HTML when cached output exists | `controller`, `action` | +| `gitlab_banzai_cacheless_render_real_duration_seconds` | Histogram | 9.4 | Duration of rendering Markdown into HTML when cached output does not exist | `controller`, `action` | +| `gitlab_cache_misses_total` | Counter | 10.2 | Cache read miss | `controller`, `action` | +| `gitlab_cache_operation_duration_seconds` | Histogram | 10.2 | Cache access time | | +| `gitlab_cache_operations_total` | Counter | 12.2 | Cache operations by controller or action | `controller`, `action`, `operation` | +| `gitlab_ci_pipeline_creation_duration_seconds` | Histogram | 13.0 | Time in seconds it takes to create a CI/CD pipeline | | +| `gitlab_ci_pipeline_size_builds` | Histogram | 13.1 | Total number of builds within a pipeline grouped by a pipeline source | `source` | +| `job_waiter_started_total` | Counter | 12.9 | Number of batches of jobs started where a web request is waiting for the jobs to complete | `worker` | +| `job_waiter_timeouts_total` | Counter | 12.9 | Number of batches of jobs that timed out where a web request is waiting for the jobs to complete | `worker` | +| `gitlab_database_transaction_seconds` | Histogram | 12.1 | Time spent in database transactions, in seconds | | +| `gitlab_method_call_duration_seconds` | Histogram | 10.2 | Method calls real duration | `controller`, `action`, `module`, `method` | +| `gitlab_page_out_of_bounds` | Counter | 12.8 | Counter for the PageLimiter pagination limit being hit | `controller`, `action`, `bot` | +| `gitlab_rails_queue_duration_seconds` | Histogram | 9.4 | Measures latency between GitLab Workhorse forwarding a request to Rails | | +| `gitlab_sql_duration_seconds` | Histogram | 10.2 | SQL execution time, excluding `SCHEMA` operations and `BEGIN` / `COMMIT` | | +| `gitlab_ruby_threads_max_expected_threads` | Gauge | 13.3 | Maximum number of threads expected to be running and performing application work | | +| `gitlab_ruby_threads_running_threads` | Gauge | 13.3 | Number of running Ruby threads by name | | +| `gitlab_transaction_cache_<key>_count_total` | Counter | 10.2 | Counter for total Rails cache calls (per key) | | +| `gitlab_transaction_cache_<key>_duration_total` | Counter | 10.2 | Counter for total time (seconds) spent in Rails cache calls (per key) | | +| `gitlab_transaction_cache_count_total` | Counter | 10.2 | Counter for total Rails cache calls (aggregate) | | +| `gitlab_transaction_cache_duration_total` | Counter | 10.2 | Counter for total time (seconds) spent in Rails cache calls (aggregate) | | +| `gitlab_transaction_cache_read_hit_count_total` | Counter | 10.2 | Counter for cache hits for Rails cache calls | `controller`, `action` | +| `gitlab_transaction_cache_read_miss_count_total` | Counter | 10.2 | Counter for cache misses for Rails cache calls | `controller`, `action` | +| `gitlab_transaction_duration_seconds` | Histogram | 10.2 | Duration for all transactions (`gitlab_transaction_*` metrics) | `controller`, `action` | +| `gitlab_transaction_event_build_found_total` | Counter | 9.4 | Counter for build found for API /jobs/request | | +| `gitlab_transaction_event_build_invalid_total` | Counter | 9.4 | Counter for build invalid due to concurrency conflict for API /jobs/request | | +| `gitlab_transaction_event_build_not_found_cached_total` | Counter | 9.4 | Counter for cached response of build not found for API /jobs/request | | +| `gitlab_transaction_event_build_not_found_total` | Counter | 9.4 | Counter for build not found for API /jobs/request | | +| `gitlab_transaction_event_change_default_branch_total` | Counter | 9.4 | Counter when default branch is changed for any repository | | +| `gitlab_transaction_event_create_repository_total` | Counter | 9.4 | Counter when any repository is created | | +| `gitlab_transaction_event_etag_caching_cache_hit_total` | Counter | 9.4 | Counter for ETag cache hit. | `endpoint` | +| `gitlab_transaction_event_etag_caching_header_missing_total` | Counter | 9.4 | Counter for ETag cache miss - header missing | `endpoint` | +| `gitlab_transaction_event_etag_caching_key_not_found_total` | Counter | 9.4 | Counter for ETag cache miss - key not found | `endpoint` | +| `gitlab_transaction_event_etag_caching_middleware_used_total` | Counter | 9.4 | Counter for ETag middleware accessed | `endpoint` | +| `gitlab_transaction_event_etag_caching_resource_changed_total` | Counter | 9.4 | Counter for ETag cache miss - resource changed | `endpoint` | +| `gitlab_transaction_event_fork_repository_total` | Counter | 9.4 | Counter for repository forks (RepositoryForkWorker). Only incremented when source repository exists | | +| `gitlab_transaction_event_import_repository_total` | Counter | 9.4 | Counter for repository imports (RepositoryImportWorker) | | +| `gitlab_transaction_event_patch_hard_limit_bytes_hit_total` | Counter | 13.9 | Counter for diff patch size limit hits | | +| `gitlab_transaction_event_push_branch_total` | Counter | 9.4 | Counter for all branch pushes | | +| `gitlab_transaction_event_push_commit_total` | Counter | 9.4 | Counter for commits | `branch` | +| `gitlab_transaction_event_push_tag_total` | Counter | 9.4 | Counter for tag pushes | | +| `gitlab_transaction_event_rails_exception_total` | Counter | 9.4 | Counter for number of rails exceptions | | +| `gitlab_transaction_event_receive_email_total` | Counter | 9.4 | Counter for received emails | `handler` | +| `gitlab_transaction_event_remote_mirrors_failed_total` | Counter | 10.8 | Counter for failed remote mirrors | | +| `gitlab_transaction_event_remote_mirrors_finished_total` | Counter | 10.8 | Counter for finished remote mirrors | | +| `gitlab_transaction_event_remote_mirrors_running_total` | Counter | 10.8 | Counter for running remote mirrors | | +| `gitlab_transaction_event_remove_branch_total` | Counter | 9.4 | Counter when a branch is removed for any repository | | +| `gitlab_transaction_event_remove_repository_total` | Counter | 9.4 | Counter when a repository is removed | | +| `gitlab_transaction_event_remove_tag_total` | Counter | 9.4 | Counter when a tag is remove for any repository | | +| `gitlab_transaction_event_sidekiq_exception_total` | Counter | 9.4 | Counter of Sidekiq exceptions | | +| `gitlab_transaction_event_stuck_import_jobs_total` | Counter | 9.4 | Count of stuck import jobs | `projects_without_jid_count`, `projects_with_jid_count` | +| `gitlab_transaction_event_update_build_total` | Counter | 9.4 | Counter for update build for API `/jobs/request/:id` | | +| `gitlab_transaction_new_redis_connections_total` | Counter | 9.4 | Counter for new Redis connections | | +| `gitlab_transaction_queue_duration_total` | Counter | 9.4 | Duration jobs were enqueued before processing | | +| `gitlab_transaction_rails_queue_duration_total` | Counter | 9.4 | Measures latency between GitLab Workhorse forwarding a request to Rails | `controller`, `action` | +| `gitlab_transaction_view_duration_total` | Counter | 9.4 | Duration for views | `controller`, `action`, `view` | +| `gitlab_view_rendering_duration_seconds` | Histogram | 10.2 | Duration for views (histogram) | `controller`, `action`, `view` | +| `http_requests_total` | Counter | 9.4 | Rack request count | `method`, `status` | +| `http_request_duration_seconds` | Histogram | 9.4 | HTTP response time from rack middleware | `method` | +| `gitlab_transaction_db_count_total` | Counter | 13.1 | Counter for total number of SQL calls | `controller`, `action` | +| `gitlab_transaction_db_write_count_total` | Counter | 13.1 | Counter for total number of write SQL calls | `controller`, `action` | +| `gitlab_transaction_db_cached_count_total` | Counter | 13.1 | Counter for total number of cached SQL calls | `controller`, `action` | +| `http_elasticsearch_requests_duration_seconds` **(PREMIUM)** | Histogram | 13.1 | Elasticsearch requests duration during web transactions | `controller`, `action` | +| `http_elasticsearch_requests_total` **(PREMIUM)** | Counter | 13.1 | Elasticsearch requests count during web transactions | `controller`, `action` | +| `pipelines_created_total` | Counter | 9.4 | Counter of pipelines created | | +| `rack_uncaught_errors_total` | Counter | 9.4 | Rack connections handling uncaught errors count | | +| `user_session_logins_total` | Counter | 9.4 | Counter of how many users have logged in since GitLab was started or restarted | | +| `upload_file_does_not_exist` | Counter | 10.7 | Number of times an upload record could not find its file. Made available in all tiers in GitLab 11.5. | | +| `failed_login_captcha_total` | Gauge | 11.0 | Counter of failed CAPTCHA attempts during login | | +| `successful_login_captcha_total` | Gauge | 11.0 | Counter of successful CAPTCHA attempts during login | | +| `auto_devops_pipelines_completed_total` | Counter | 12.7 | Counter of completed Auto DevOps pipelines, labeled by status | | +| `gitlab_metrics_dashboard_processing_time_ms` | Summary | 12.10 | Metrics dashboard processing time in milliseconds | service, stages | +| `action_cable_active_connections` | Gauge | 13.4 | Number of ActionCable WS clients currently connected | `server_mode` | +| `action_cable_pool_min_size` | Gauge | 13.4 | Minimum number of worker threads in ActionCable thread pool | `server_mode` | +| `action_cable_pool_max_size` | Gauge | 13.4 | Maximum number of worker threads in ActionCable thread pool | `server_mode` | +| `action_cable_pool_current_size` | Gauge | 13.4 | Current number of worker threads in ActionCable thread pool | `server_mode` | +| `action_cable_pool_largest_size` | Gauge | 13.4 | Largest number of worker threads observed so far in ActionCable thread pool | `server_mode` | +| `action_cable_pool_pending_tasks` | Gauge | 13.4 | Number of tasks waiting to be executed in ActionCable thread pool | `server_mode` | +| `action_cable_pool_tasks_total` | Gauge | 13.4 | Total number of tasks executed in ActionCable thread pool | `server_mode` | +| `gitlab_issuable_fast_count_by_state_total` | Counter | 13.5 | Total number of row count operations on issue/merge request list pages | | +| `gitlab_issuable_fast_count_by_state_failures_total` | Counter | 13.5 | Number of soft-failed row count operations on issue/merge request list pages | | +| `gitlab_external_http_total` | Counter | 13.8 | Total number of HTTP calls to external systems | `controller`, `action` | +| `gitlab_external_http_duration_seconds` | Counter | 13.8 | Duration in seconds spent on each HTTP call to external systems | | +| `gitlab_external_http_exception_total` | Counter | 13.8 | Total number of exceptions raised when making external HTTP calls | | +| `ci_report_parser_duration_seconds` | Histogram | 13.9 | Time to parse CI/CD report artifacts | `parser` | +| `pipeline_graph_link_calculation_duration_seconds` | Histogram | 13.9 | Total time spent calculating links, in seconds | `project` | +| `pipeline_graph_links_total` | Histogram | 13.9 | Number of links per graph | `project` | +| `pipeline_graph_links_per_job_ratio` | Histogram | 13.9 | Ratio of links to job per graph | `project` | ## Metrics controlled by a feature flag @@ -212,7 +220,7 @@ configuration option in `gitlab.yml`. These metrics are served from the | `destroyed_job_artifacts_count_total` | Counter | 13.6 | Number of destroyed expired job artifacts | | | `destroyed_pipeline_artifacts_count_total` | Counter | 13.8 | Number of destroyed expired pipeline artifacts | | -## Database load balancing metrics **(PREMIUM ONLY)** +## Database load balancing metrics **(PREMIUM SELF)** The following metrics are available: @@ -220,7 +228,7 @@ The following metrics are available: |:--------------------------------- |:--------- |:------------------------------------------------------------- |:-------------------------------------- | | `db_load_balancing_hosts` | Gauge | [12.3](https://gitlab.com/gitlab-org/gitlab/-/issues/13630) | Current number of load balancing hosts | -## Database partitioning metrics **(PREMIUM ONLY)** +## Database partitioning metrics **(PREMIUM SELF)** The following metrics are available: diff --git a/doc/administration/monitoring/prometheus/gitlab_monitor_exporter.md b/doc/administration/monitoring/prometheus/gitlab_monitor_exporter.md deleted file mode 100644 index 8fa2e16dcd0..00000000000 --- a/doc/administration/monitoring/prometheus/gitlab_monitor_exporter.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'gitlab_exporter.md' ---- - -This document was moved to [another location](gitlab_exporter.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index 80ddb87e727..23bffae99cf 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -4,7 +4,12 @@ group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Monitoring GitLab with Prometheus +# Monitoring GitLab with Prometheus **(FREE SELF)** + +[Prometheus](https://prometheus.io) is a powerful time-series monitoring service, providing a flexible +platform for monitoring GitLab and other software products. +GitLab provides out-of-the-box monitoring with Prometheus, providing easy +access to high quality time-series monitoring of GitLab services. > **Notes:** > @@ -16,11 +21,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - Prometheus and its exporters don't authenticate users, and are available > to anyone who can access them. -[Prometheus](https://prometheus.io) is a powerful time-series monitoring service, providing a flexible -platform for monitoring GitLab and other software products. -GitLab provides out of the box monitoring with Prometheus, providing easy -access to high quality time-series monitoring of GitLab services. - ## Overview Prometheus works by periodically connecting to data sources and collecting their diff --git a/doc/administration/monitoring/prometheus/node_exporter.md b/doc/administration/monitoring/prometheus/node_exporter.md index f27912d5806..6efc9d67e8d 100644 --- a/doc/administration/monitoring/prometheus/node_exporter.md +++ b/doc/administration/monitoring/prometheus/node_exporter.md @@ -4,7 +4,7 @@ group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Node exporter +# Node exporter **(FREE SELF)** The [node exporter](https://github.com/prometheus/node_exporter) enables you to measure various machine resources such as memory, disk and CPU utilization. diff --git a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md index 0dffad59793..df2aa08efaa 100644 --- a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md +++ b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md @@ -4,7 +4,7 @@ group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# PgBouncer exporter +# PgBouncer exporter **(FREE SELF)** > Introduced in [Omnibus GitLab 11.0](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/2493). diff --git a/doc/administration/monitoring/prometheus/postgres_exporter.md b/doc/administration/monitoring/prometheus/postgres_exporter.md index 8bc61ff53bb..8b6d3d82018 100644 --- a/doc/administration/monitoring/prometheus/postgres_exporter.md +++ b/doc/administration/monitoring/prometheus/postgres_exporter.md @@ -4,7 +4,7 @@ group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# PostgreSQL Server Exporter +# PostgreSQL Server Exporter **(FREE SELF)** The [PostgreSQL Server Exporter](https://github.com/wrouesnel/postgres_exporter) allows you to export various PostgreSQL metrics. diff --git a/doc/administration/monitoring/prometheus/redis_exporter.md b/doc/administration/monitoring/prometheus/redis_exporter.md index 03c215625ec..e9d656a6493 100644 --- a/doc/administration/monitoring/prometheus/redis_exporter.md +++ b/doc/administration/monitoring/prometheus/redis_exporter.md @@ -4,7 +4,7 @@ group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Redis exporter +# Redis exporter **(FREE SELF)** The [Redis exporter](https://github.com/oliver006/redis_exporter) enables you to measure various [Redis](https://redis.io) metrics. For more information on what is exported, diff --git a/doc/administration/monitoring/prometheus/registry_exporter.md b/doc/administration/monitoring/prometheus/registry_exporter.md index 009d94491f3..d2bd86aa908 100644 --- a/doc/administration/monitoring/prometheus/registry_exporter.md +++ b/doc/administration/monitoring/prometheus/registry_exporter.md @@ -4,7 +4,7 @@ group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Registry exporter +# Registry exporter **(FREE SELF)** > [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/2884) in GitLab 11.9. diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md index bddf770180b..a7009b702ed 100644 --- a/doc/administration/nfs.md +++ b/doc/administration/nfs.md @@ -20,10 +20,10 @@ From GitLab 14.0, technical support for NFS for Git repositories will no longer be provided. Upgrade to [Gitaly Cluster](gitaly/praefect.md) as soon as possible. -Filesystem performance can impact overall GitLab performance, especially for +File system performance can impact overall GitLab performance, especially for actions that read or write to Git repositories. For steps you can use to test -filesystem performance, see -[Filesystem Performance Benchmarking](operations/filesystem_benchmarking.md). +file system performance, see +[File system Performance Benchmarking](operations/filesystem_benchmarking.md). ## Known kernel version incompatibilities @@ -408,7 +408,7 @@ For supported database architecture, see our documentation about ### Finding the requests that are being made to NFS In case of NFS-related problems, it can be helpful to trace -the filesystem requests that are being made by using `perf`: +the file system requests that are being made by using `perf`: ```shell sudo perf trace -e 'nfs4:*' -p $(pgrep -fd ',' puma && pgrep -fd ',' unicorn) diff --git a/doc/administration/npm_registry.md b/doc/administration/npm_registry.md deleted file mode 100644 index 690b6785941..00000000000 --- a/doc/administration/npm_registry.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'packages/index.md' ---- - -This document was moved to [another location](packages/index.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index 999d9b87363..d1b460df273 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -551,17 +551,17 @@ supported by consolidated configuration form, refer to the following guides: | [Merge request diffs](merge_request_diffs.md#using-object-storage) | Yes | | [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | | [Packages](packages/index.md#using-object-storage) (optional feature) | Yes | -| [Dependency Proxy](packages/dependency_proxy.md#using-object-storage) (optional feature) **(PREMIUM ONLY)** | Yes | -| [Pseudonymizer](pseudonymizer.md#configuration) (optional feature) **(ULTIMATE ONLY)** | No | +| [Dependency Proxy](packages/dependency_proxy.md#using-object-storage) (optional feature) **(PREMIUM SELF)** | Yes | +| [Pseudonymizer](pseudonymizer.md#configuration) (optional feature) **(ULTIMATE SELF)** | No | | [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | | [Terraform state files](terraform_state.md#using-object-storage) | Yes | | [GitLab Pages content](pages/index.md#using-object-storage) | Yes | -### Other alternatives to filesystem storage +### Other alternatives to file system storage If you're working to [scale out](reference_architectures/index.md) your GitLab implementation, or add fault tolerance and redundancy, you may be -looking at removing dependencies on block or network filesystems. +looking at removing dependencies on block or network file systems. See the following additional guides and [note that Pages requires disk storage](#gitlab-pages-requires-nfs): diff --git a/doc/administration/operations.md b/doc/administration/operations.md deleted file mode 100644 index cfabb5ec27d..00000000000 --- a/doc/administration/operations.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'operations/index.md' ---- - -This document was moved to [another location](operations/index.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md index 1b7c0edab04..0157ea8f430 100644 --- a/doc/administration/operations/extra_sidekiq_processes.md +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Run multiple Sidekiq processes **(CORE ONLY)** +# Run multiple Sidekiq processes **(FREE SELF)** GitLab allows you to start multiple Sidekiq processes. These processes can be used to consume a dedicated set @@ -27,7 +27,7 @@ can be started. ## Start multiple processes > - [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4006) in GitLab 12.10, starting multiple processes with Sidekiq cluster. -> - [Sidekiq cluster moved](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181) to GitLab [Core](https://about.gitlab.com/pricing/#self-managed) in GitLab 12.10. +> - [Sidekiq cluster moved](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181) to GitLab [Free](https://about.gitlab.com/pricing/) in GitLab 12.10. > - [Sidekiq cluster became default](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4140) in GitLab 13.0. To start multiple processes: @@ -113,7 +113,7 @@ you list: ## Queue selector > - [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/45) in [GitLab Starter](https://about.gitlab.com/pricing/) 12.8. -> - [Sidekiq cluster including queue selector moved](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181) to GitLab [Core](https://about.gitlab.com/pricing/#self-managed) in GitLab 12.10. +> - [Sidekiq cluster including queue selector moved](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181) to GitLab [Free](https://about.gitlab.com/pricing/) in GitLab 12.10. > - [Renamed from `experimental_queue_selector` to `queue_selector`](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/147) in GitLab 13.6. In addition to selecting queues by name, as above, the `queue_selector` diff --git a/doc/administration/operations/filesystem_benchmarking.md b/doc/administration/operations/filesystem_benchmarking.md index 9072214eb02..6119fcdae45 100644 --- a/doc/administration/operations/filesystem_benchmarking.md +++ b/doc/administration/operations/filesystem_benchmarking.md @@ -4,15 +4,15 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Filesystem Performance Benchmarking +# File system Performance Benchmarking -Filesystem performance has a big impact on overall GitLab performance, +File system performance has a big impact on overall GitLab performance, especially for actions that read or write to Git repositories. This information -will help benchmark filesystem performance against known good and bad real-world +will help benchmark file system performance against known good and bad real-world systems. -Normally when talking about filesystem performance the biggest concern is -with Network Filesystems (NFS). However, even some local disks can have slow +Normally when talking about file system performance the biggest concern is +with Network File Systems (NFS). However, even some local disks can have slow I/O. The information on this page can be used for either scenario. ## Executing benchmarks @@ -77,7 +77,7 @@ available on the system. It's possible to receive good results on this test but still have poor performance due to read speed and various other factors. -The following one-line commands provide a quick benchmark for filesystem write and read +The following one-line commands provide a quick benchmark for file system write and read performance. This will write 1,000 small files to the directory in which it is executed, and then read the same 1,000 files. @@ -125,4 +125,4 @@ sys 0m1.663s ``` From experience with multiple customers, this task should take under 10 -seconds to indicate good filesystem performance. +seconds to indicate good file system performance. diff --git a/doc/administration/operations/index.md b/doc/administration/operations/index.md index b15417ea8d9..eb1854fee10 100644 --- a/doc/administration/operations/index.md +++ b/doc/administration/operations/index.md @@ -20,17 +20,17 @@ Keep your GitLab instance up and running smoothly. by GitLab to another file system or another server. - [Sidekiq MemoryKiller](sidekiq_memory_killer.md): Configure Sidekiq MemoryKiller to restart Sidekiq. -- [Multiple Sidekiq processes](extra_sidekiq_processes.md): Configure multiple Sidekiq processes to ensure certain queues always have dedicated workers, no matter the number of jobs that need to be processed. **(CORE ONLY)** +- [Multiple Sidekiq processes](extra_sidekiq_processes.md): Configure multiple Sidekiq processes to ensure certain queues always have dedicated workers, no matter the number of jobs that need to be processed. **(FREE SELF)** - [Puma](puma.md): Understand Puma and puma-worker-killer. - [Unicorn](unicorn.md): Understand Unicorn and unicorn-worker-killer. - Speed up SSH operations by [Authorizing SSH users via a fast, indexed lookup to the GitLab database](fast_ssh_key_lookup.md), and/or by [doing away with user SSH keys stored on GitLab entirely in favor of SSH certificates](ssh_certificates.md). -- [Filesystem Performance Benchmarking](filesystem_benchmarking.md): Filesystem +- [File System Performance Benchmarking](filesystem_benchmarking.md): File system performance can have a big impact on GitLab performance, especially for actions that read or write Git repositories. This information will help benchmark - filesystem performance against known good and bad real-world systems. + file system performance against known good and bad real-world systems. - [The Rails Console](rails_console.md): Provides a way to interact with your GitLab instance from the command line. Used for troubleshooting a problem or retrieving some data that can only be done through direct access to GitLab. - [ChatOps Scripts](https://gitlab.com/gitlab-com/chatops): The GitLab.com Infrastructure team uses this repository to house diff --git a/doc/administration/operations/moving_repositories.md b/doc/administration/operations/moving_repositories.md index 029f3bb01ed..9997dcb18dd 100644 --- a/doc/administration/operations/moving_repositories.md +++ b/doc/administration/operations/moving_repositories.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Moving repositories managed by GitLab **(CORE ONLY)** +# Moving repositories managed by GitLab **(FREE SELF)** Sometimes you need to move all repositories managed by GitLab to another file system or another server. @@ -20,8 +20,8 @@ The GitLab API is the recommended way to move Git repositories: For more information, see: -- [Configuring additional storage for Gitaly](../gitaly/index.md#network-architecture). Within this - example, additional storage called `storage1` and `storage2` is configured. +- [Configuring additional storage for Gitaly](../gitaly/index.md#network-architecture). This + example configures additional storage called `storage1` and `storage2`. - [The API documentation](../../api/project_repository_storage_moves.md) details the endpoints for querying and scheduling project repository moves. - [The API documentation](../../api/snippet_repository_storage_moves.md) details the endpoints for @@ -38,7 +38,7 @@ Read more in the [API documentation for projects](../../api/project_repository_s GitLab environment, for example: - From a single-node GitLab to a scaled-out architecture. -- From a GitLab instance in your private datacenter to a cloud provider. +- From a GitLab instance in your private data center to a cloud provider. The rest of the document looks at some of the ways you can copy all your repositories from @@ -103,8 +103,8 @@ Using `rsync` to migrate Git data can cause data loss and repository corruption. If the target directory already contains a partial / outdated copy of the repositories it may be wasteful to copy all the data again with `tar`. In this scenario it is better to use `rsync`. This utility -is either already installed on your system or easily installable -via `apt`, `yum`, and so on. +is either already installed on your system, or installable +by using `apt` or `yum`. ```shell sudo -u git sh -c 'rsync -a --delete /var/opt/gitlab/git-data/repositories/. \ @@ -112,7 +112,7 @@ sudo -u git sh -c 'rsync -a --delete /var/opt/gitlab/git-data/repositories/. \ ``` The `/.` in the command above is very important, without it you can -easily get the wrong directory structure in the target directory. +get the wrong directory structure in the target directory. If you want to see progress, replace `-a` with `-av`. #### Single `rsync` to another server @@ -135,20 +135,23 @@ WARNING: Using `rsync` to migrate Git data can cause data loss and repository corruption. [These instructions are being reviewed](https://gitlab.com/gitlab-org/gitlab/-/issues/270422). -Every time you start an `rsync` job it has to inspect all files in -the source directory, all files in the target directory, and then -decide what files to copy or not. If the source or target directory -has many contents this startup phase of `rsync` can become a burden -for your GitLab server. In cases like this you can make `rsync`'s -life easier by dividing its work in smaller pieces, and sync one -repository at a time. +Every time you start an `rsync` job it must: + +- Inspect all files in the source directory. +- Inspect all files in the target directory. +- Decide whether or not to copy files. + +If the source or target directory +has many contents, this startup phase of `rsync` can become a burden +for your GitLab server. You can reduce the workload of `rsync` by dividing its +work in smaller pieces, and sync one repository at a time. In addition to `rsync` we use [GNU Parallel](http://www.gnu.org/software/parallel/). -This utility is not included in GitLab so you need to install it yourself with `apt` -or `yum`. Also note that the GitLab scripts we used below were added in GitLab 8.1. +This utility is not included in GitLab, so you must install it yourself with `apt` +or `yum`. -**This process does not clean up repositories at the target location that no -longer exist at the source.** +This process does not clean up repositories at the target location that no +longer exist at the source. #### Parallel `rsync` for all repositories known to GitLab @@ -218,8 +221,8 @@ Using `rsync` to migrate Git data can cause data loss and repository corruption. [These instructions are being reviewed](https://gitlab.com/gitlab-org/gitlab/-/issues/270422). Suppose you have already done one sync that started after 2015-10-1 12:00 UTC. -Then you might only want to sync repositories that were changed via GitLab -_after_ that time. You can use the `SINCE` variable to tell `rake +Then you might only want to sync repositories that were changed by using GitLab +after that time. You can use the `SINCE` variable to tell `rake gitlab:list_repos` to only print repositories with recent activity. ```shell diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md index 44ac014650e..3b676010bfe 100644 --- a/doc/administration/operations/puma.md +++ b/doc/administration/operations/puma.md @@ -31,7 +31,7 @@ Unicorn in GitLab 14.0. When switching to Puma, Unicorn server configuration will _not_ carry over automatically, due to differences between the two application servers. For Omnibus-based deployments, see [Configuring Puma Settings](https://docs.gitlab.com/omnibus/settings/puma.html#configuring-puma-settings). -For Helm based deployments, see the [Webservice Chart documentation](https://docs.gitlab.com/charts/charts/gitlab/webservice/index.html). +For Helm based deployments, see the [`webservice` chart documentation](https://docs.gitlab.com/charts/charts/gitlab/webservice/index.html). Additionally we strongly recommend that multi-node deployments [configure their load balancers to use the readiness check](../load_balancer.md#readiness-check) due to a difference between Unicorn and Puma in how they handle connections during a restart of the service. diff --git a/doc/administration/operations/speed_up_ssh.md b/doc/administration/operations/speed_up_ssh.md deleted file mode 100644 index 2f3cf40a222..00000000000 --- a/doc/administration/operations/speed_up_ssh.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'fast_ssh_key_lookup.md' ---- - -This document was moved to [another location](fast_ssh_key_lookup.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/packages.md b/doc/administration/packages.md deleted file mode 100644 index 690b6785941..00000000000 --- a/doc/administration/packages.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'packages/index.md' ---- - -This document was moved to [another location](packages/index.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index ab6202fef4c..63bb969afce 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -312,14 +312,14 @@ configuration. The different supported drivers are: -| Driver | Description | -|------------|-------------------------------------| -| filesystem | Uses a path on the local filesystem | -| Azure | Microsoft Azure Blob Storage | -| gcs | Google Cloud Storage | -| s3 | Amazon Simple Storage Service. Be sure to configure your storage bucket with the correct [S3 Permission Scopes](https://docs.docker.com/registry/storage-drivers/s3/#s3-permission-scopes). | -| swift | OpenStack Swift Object Storage | -| oss | Aliyun OSS | +| Driver | Description | +|--------------|--------------------------------------| +| `filesystem` | Uses a path on the local file system | +| `Azure` | Microsoft Azure Blob Storage | +| `gcs` | Google Cloud Storage | +| `s3` | Amazon Simple Storage Service. Be sure to configure your storage bucket with the correct [S3 Permission Scopes](https://docs.docker.com/registry/storage-drivers/s3/#s3-permission-scopes). | +| `swift` | OpenStack Swift Object Storage | +| `oss` | Aliyun OSS | Although most S3 compatible services (like [MinIO](https://min.io/)) should work with the Container Registry, we only guarantee support for AWS S3. Because we cannot assert the correctness of third-party S3 implementations, we can debug issues, but we cannot patch the registry unless an issue is reproducible against an AWS S3 bucket. diff --git a/doc/administration/packages/dependency_proxy.md b/doc/administration/packages/dependency_proxy.md index 9e315722c24..16cfdc8c784 100644 --- a/doc/administration/packages/dependency_proxy.md +++ b/doc/administration/packages/dependency_proxy.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GitLab Dependency Proxy administration > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7934) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.11. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/273655) to [GitLab Core](https://about.gitlab.com/pricing/) in GitLab 13.6. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/273655) to [GitLab Free](https://about.gitlab.com/pricing/) in GitLab 13.6. GitLab can be used as a dependency proxy for a variety of common package managers. diff --git a/doc/administration/packages/index.md b/doc/administration/packages/index.md index 0f391371a6a..25fce08d3d2 100644 --- a/doc/administration/packages/index.md +++ b/doc/administration/packages/index.md @@ -35,6 +35,8 @@ The Package Registry supports the following formats: The below table lists formats that are not supported, but are accepting Community contributions for. Consider contributing to GitLab. This [development documentation](../../development/packages.md) guides you through the process. +<!-- vale gitlab.Spelling = NO --> + | Format | Status | | ------ | ------ | | Chef | [#36889](https://gitlab.com/gitlab-org/gitlab/-/issues/36889) | @@ -51,6 +53,8 @@ guides you through the process. | Terraform | [WIP: Merge Request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18834) | | Vagrant | [#36899](https://gitlab.com/gitlab-org/gitlab/-/issues/36899) | +<!-- vale gitlab.Spelling = YES --> + ## Enabling the Packages feature NOTE: diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index 9e949468805..2b7a7300ad8 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -190,7 +190,7 @@ outside world. ### Additional configuration for Docker container The GitLab Pages daemon doesn't have permissions to bind mounts when it runs -in a Docker container. To overcome this issue, you must change the chroot +in a Docker container. To overcome this issue, you must change the `chroot` behavior: 1. Edit `/etc/gitlab/gitlab.rb`. @@ -222,7 +222,7 @@ control over how the Pages daemon runs and serves content in your environment. | `api_secret_key` | Full path to file with secret key used to authenticate with the GitLab API. Auto-generated when left unset. | `artifacts_server` | Enable viewing [artifacts](../job_artifacts.md) in GitLab Pages. | `artifacts_server_timeout` | Timeout (in seconds) for a proxied request to the artifacts server. -| `artifacts_server_url` | API URL to proxy artifact requests to. Defaults to GitLab `external URL` + `/api/v4`, for example `https://gitlab.com/api/v4`. +| `artifacts_server_url` | API URL to proxy artifact requests to. Defaults to GitLab `external URL` + `/api/v4`, for example `https://gitlab.com/api/v4`. When running a [separate Pages server](#running-gitlab-pages-on-a-separate-server), this URL must point to the main GitLab server's API. | `auth_redirect_uri` | Callback URL for authenticating with GitLab. Defaults to project's subdomain of `pages_external_url` + `/auth`. | `auth_secret` | Secret key for signing authentication requests. Leave blank to pull automatically from GitLab during OAuth registration. | `dir` | Working directory for configuration and secrets files. @@ -236,7 +236,7 @@ control over how the Pages daemon runs and serves content in your environment. | `gitlab_secret` | The OAuth application secret. Leave blank to automatically fill when Pages authenticates with GitLab. | `gitlab_server` | Server to use for authentication when access control is enabled; defaults to GitLab `external_url`. | `headers` | Specify any additional http headers that should be sent to the client with each response. -| `inplace_chroot` | On [systems that don't support bind-mounts](index.md#additional-configuration-for-docker-container), this instructs GitLab Pages to chroot into its `pages_path` directory. Some caveats exist when using inplace chroot; refer to the GitLab Pages [README](https://gitlab.com/gitlab-org/gitlab-pages/blob/master/README.md#caveats) for more information. +| `inplace_chroot` | On [systems that don't support bind-mounts](index.md#additional-configuration-for-docker-container), this instructs GitLab Pages to `chroot` into its `pages_path` directory. Some caveats exist when using in-place `chroot`; refer to the GitLab Pages [README](https://gitlab.com/gitlab-org/gitlab-pages/blob/master/README.md#caveats) for more information. | `insecure_ciphers` | Use default list of cipher suites, may contain insecure ones like 3DES and RC4. | `internal_gitlab_server` | Internal GitLab server address used exclusively for API requests. Useful if you want to send that traffic over an internal load balancer. Defaults to GitLab `external_url`. | `listen_proxy` | The addresses to listen on for reverse-proxy requests. Pages binds to these addresses' network sockets and receives incoming requests from them. Sets the value of `proxy_pass` in `$nginx-dir/conf/gitlab-pages.conf`. @@ -538,7 +538,7 @@ the below steps to do a no downtime transfer to a new storage location. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). 1. Verify Pages are still being served up as expected. -1. Unpause Pages deployments by removing from `/etc/gitlab/gitlab.rb` the `sidekiq` setting set above. +1. Resume Pages deployments by removing from `/etc/gitlab/gitlab.rb` the `sidekiq` setting set above. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). 1. Trigger a new Pages deployment and verify it's working as expected. 1. Remove the old Pages storage location: `sudo rm -rf /var/opt/gitlab/gitlab-rails/shared/pages` @@ -573,7 +573,7 @@ You can configure the maximum size of the unpacked archive per project in **Admin Area > Settings > Preferences > Pages**, in **Maximum size of pages (MB)**. The default is 100MB. -### Override maximum pages size per project or group **(PREMIUM ONLY)** +### Override maximum pages size per project or group **(PREMIUM SELF)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16610) in GitLab 12.7. @@ -629,7 +629,7 @@ database encryption. Proceed with caution. on the **Pages server** and configure this share to allow access from your main **GitLab server**. Note that the example there is more general and - shares several sub-directories from `/home` to several `/nfs/home` mountpoints. + shares several sub-directories from `/home` to several `/nfs/home` mount points. For our Pages-specific example here, we instead share only the default GitLab Pages folder `/var/opt/gitlab/gitlab-rails/shared/pages` from the **Pages server** and we mount it to `/mnt/pages` @@ -730,6 +730,13 @@ or report an issue. ### Object storage settings +WARNING: +With the following settings, Pages uses both NFS and Object Storage locations when deploying the +site. **Do not remove the existing NFS mount used by Pages** when applying these settings. For more +information, see the epics +[3901](https://gitlab.com/groups/gitlab-org/-/epics/3901#how-to-test-object-storage-integration-in-beta) +and [3910](https://gitlab.com/groups/gitlab-org/-/epics/3910). + The following settings are: - Nested under `pages:` and then `object_store:` on source installations. @@ -818,7 +825,7 @@ but commented out to help encourage others to add to it in the future. --> ### `open /etc/ssl/ca-bundle.pem: permission denied` -GitLab Pages runs inside a chroot jail, usually in a uniquely numbered directory like +GitLab Pages runs inside a `chroot` jail, usually in a uniquely numbered directory like `/tmp/gitlab-pages-*`. Within the jail, a bundle of trusted certificates is @@ -828,7 +835,7 @@ from `/opt/gitlab/embedded/ssl/certs/cacert.pem` as part of starting up Pages. If the permissions on the source file are incorrect (they should be `0644`), then -the file inside the chroot jail is also wrong. +the file inside the `chroot` jail is also wrong. Pages logs errors in `/var/log/gitlab/gitlab-pages/current` like: @@ -837,8 +844,8 @@ x509: failed to load system roots and no roots provided open /etc/ssl/ca-bundle.pem: permission denied ``` -The use of a chroot jail makes this error misleading, as it is not -referring to `/etc/ssl` on the root filesystem. +The use of a `chroot` jail makes this error misleading, as it is not +referring to `/etc/ssl` on the root file system. The fix is to correct the source file permissions and restart Pages: @@ -862,8 +869,8 @@ open /opt/gitlab/embedded/ssl/certs/cacert.pem: no such file or directory x509: certificate signed by unknown authority ``` -The reason for those errors is that the files `resolv.conf` and `ca-bundle.pem` are missing inside the chroot. -The fix is to copy the host's `/etc/resolv.conf` and the GitLab certificate bundle inside the chroot: +The reason for those errors is that the files `resolv.conf` and `ca-bundle.pem` are missing inside the `chroot`. +The fix is to copy the host's `/etc/resolv.conf` and the GitLab certificate bundle inside the `chroot`: ```shell sudo mkdir -p /var/opt/gitlab/gitlab-rails/shared/pages/etc/ssl @@ -895,7 +902,7 @@ gitlab_pages['listen_proxy'] = '127.0.0.1:8090' ### 404 error after transferring project to a different group or user If you encounter a `404 Not Found` error a Pages site after transferring a project to -another group or user, you must trigger adomain configuration update for Pages. To do +another group or user, you must trigger a domain configuration update for Pages. To do so, write something in the `.update` file. The Pages daemon monitors for changes to this file, and reloads the configuration when changes occur. @@ -928,11 +935,25 @@ For example, if there is a connection timeout: error="failed to connect to internal Pages API: Get \"https://gitlab.example.com:3000/api/v4/internal/pages/status\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" ``` +### Pages cannot communicate with an instance of the GitLab API + +If you use the default value for `domain_config_source=auto` and run multiple instances of GitLab +Pages, you may see intermittent 502 error responses while serving Pages content. You may also see +the following warning in the Pages logs: + +```plaintext +WARN[0010] Pages cannot communicate with an instance of the GitLab API. Please sync your gitlab-secrets.json file https://gitlab.com/gitlab-org/gitlab-pages/-/issues/535#workaround. error="pages endpoint unauthorized" +``` + +This can happen if your `gitlab-secrets.json` file is out of date between GitLab Rails and GitLab +Pages. Follow steps 8-10 of [Running GitLab Pages on a separate server](#running-gitlab-pages-on-a-separate-server), +in all of your GitLab Pages instances. + ### 500 error with `securecookie: failed to generate random iv` and `Failed to save the session` This problem most likely results from an [out-dated operating system](https://docs.gitlab.com/omnibus/package-information/deprecated_os.html). -The [Pages daemon uses the `securecookie` library](https://gitlab.com/search?group_id=9970&project_id=734943&repository_ref=master&scope=blobs&search=securecookie&snippets=false) to get random strings via [crypto/rand in Go](https://golang.org/pkg/crypto/rand/#pkg-variables). -This requires the `getrandom` syscall or `/dev/urandom` to be available on the host OS. +The [Pages daemon uses the `securecookie` library](https://gitlab.com/search?group_id=9970&project_id=734943&repository_ref=master&scope=blobs&search=securecookie&snippets=false) to get random strings via [`crypto/rand` in Go](https://golang.org/pkg/crypto/rand/#pkg-variables). +This requires the `getrandom` system call or `/dev/urandom` to be available on the host OS. Upgrading to an [officially supported operating system](https://about.gitlab.com/install/) is recommended. ### The requested scope is invalid, malformed, or unknown @@ -940,6 +961,8 @@ Upgrading to an [officially supported operating system](https://about.gitlab.com This problem comes from the permissions of the GitLab Pages OAuth application. To fix it, go to **Admin > Applications > GitLab Pages** and edit the application. Under **Scopes**, ensure that the `api` scope is selected and save your changes. +When running a [separate Pages server](#running-gitlab-pages-on-a-separate-server), +this setting needs to be configured on the main GitLab server. ### Workaround in case no wildcard DNS entry can be set @@ -948,3 +971,21 @@ If the wildcard DNS [prerequisite](#prerequisites) can't be met, you can still u 1. [Move](../../user/project/settings/index.md#transferring-an-existing-project-into-another-namespace) all projects you need to use Pages with into a single group namespace, for example `pages`. 1. Configure a [DNS entry](#dns-configuration) without the `*.`-wildcard, for example `pages.example.io`. + +### Pages daemon fails with permission denied errors + +If `/tmp` is mounted with `noexec`, the Pages daemon fails to start with an error like: + +```plaintext +{"error":"fork/exec /gitlab-pages: permission denied","level":"fatal","msg":"could not create pages daemon","time":"2021-02-02T21:54:34Z"} +``` + +In this case, change `TMPDIR` to a location that is not mounted with `noexec`. Add the following to +`/etc/gitlab/gitlab.rb`: + +```ruby +gitlab_pages['env'] = {'TMPDIR' => '<new_tmp_path>'} +``` + +Once added, reconfigure with `sudo gitlab-ctl reconfigure` and restart GitLab with +`sudo gitlab-ctl restart`. diff --git a/doc/administration/pages/source.md b/doc/administration/pages/source.md index 0aebeaf2ebe..d1a8b703860 100644 --- a/doc/administration/pages/source.md +++ b/doc/administration/pages/source.md @@ -407,7 +407,7 @@ Pages access control is disabled by default. To enable it: ``` 1. [Restart GitLab](../restart_gitlab.md#installations-from-source). -1. Create a new [system OAuth application](../../integration/oauth_provider.md#adding-an-application-through-the-profile). +1. Create a new [system OAuth application](../../integration/oauth_provider.md#add-an-application-through-the-profile). This should be called `GitLab Pages` and have a `Redirect URL` of `https://projects.example.io/auth`. It does not need to be a "trusted" application, but it does need the `api` scope. diff --git a/doc/administration/plugins.md b/doc/administration/plugins.md deleted file mode 100644 index f5d4b3aa2ff..00000000000 --- a/doc/administration/plugins.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'file_hooks.md' ---- - -This document was moved to [File Hooks](file_hooks.md), after the Plugins feature was renamed to File Hooks. - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/postgresql/index.md b/doc/administration/postgresql/index.md index 0e6fd757421..eabb396aeab 100644 --- a/doc/administration/postgresql/index.md +++ b/doc/administration/postgresql/index.md @@ -11,7 +11,7 @@ In this section, you'll be guided through configuring a PostgreSQL database to be used with GitLab in one of our [reference architectures](../reference_architectures/index.md). There are essentially three setups to choose from. -## PostgreSQL replication and failover with Omnibus GitLab **(PREMIUM ONLY)** +## PostgreSQL replication and failover with Omnibus GitLab **(PREMIUM SELF)** This setup is for when you have installed GitLab using the [Omnibus GitLab **Enterprise Edition** (EE) package](https://about.gitlab.com/install/?version=ee). @@ -21,7 +21,7 @@ the package, so you can it to set up the whole PostgreSQL infrastructure (primar [> Read how to set up PostgreSQL replication and failover using Omnibus GitLab](replication_and_failover.md) -## Standalone PostgreSQL using Omnibus GitLab **(CORE ONLY)** +## Standalone PostgreSQL using Omnibus GitLab **(FREE SELF)** This setup is for when you have installed the [Omnibus GitLab packages](https://about.gitlab.com/install/) (CE or EE), @@ -29,7 +29,7 @@ to use the bundled PostgreSQL having only its service enabled. [> Read how to set up a standalone PostgreSQL instance using Omnibus GitLab](standalone.md) -## Provide your own PostgreSQL instance **(CORE ONLY)** +## Provide your own PostgreSQL instance **(FREE SELF)** This setup is for when you have installed GitLab using the [Omnibus GitLab packages](https://about.gitlab.com/install/) (CE or EE), diff --git a/doc/administration/postgresql/pgbouncer.md b/doc/administration/postgresql/pgbouncer.md index 951edfeaec2..791ca64b001 100644 --- a/doc/administration/postgresql/pgbouncer.md +++ b/doc/administration/postgresql/pgbouncer.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Working with the bundled PgBouncer service **(PREMIUM ONLY)** +# Working with the bundled PgBouncer service **(PREMIUM SELF)** [PgBouncer](http://www.pgbouncer.org/) is used to seamlessly migrate database connections between servers in a failover scenario. Additionally, it can be used diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index 75d0c558962..e80e38fe5d1 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -4,7 +4,7 @@ group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# PostgreSQL replication and failover with Omnibus GitLab **(PREMIUM ONLY)** +# PostgreSQL replication and failover with Omnibus GitLab **(PREMIUM SELF)** This document focuses on configuration supported with [GitLab Premium](https://about.gitlab.com/pricing/), using the Omnibus GitLab package. If you're a Community Edition or Starter user, consider using a cloud hosted solution. @@ -46,7 +46,7 @@ Each database node runs three services: `PostgreSQL` - The database itself. -`Patroni` - Communicates with other patroni services in the cluster and handles +`Patroni` - Communicates with other Patroni services in the cluster and handles failover when issues with the leader server occurs. The failover procedure consists of: diff --git a/doc/administration/postgresql/standalone.md b/doc/administration/postgresql/standalone.md index 30495824cd2..cca46a2ea8c 100644 --- a/doc/administration/postgresql/standalone.md +++ b/doc/administration/postgresql/standalone.md @@ -4,7 +4,7 @@ group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Standalone PostgreSQL using Omnibus GitLab **(CORE ONLY)** +# Standalone PostgreSQL using Omnibus GitLab **(FREE SELF)** If you wish to have your database service hosted separately from your GitLab application servers, you can do this using the PostgreSQL binaries packaged diff --git a/doc/administration/raketasks/check.md b/doc/administration/raketasks/check.md index 41defd89df5..5ec268ac769 100644 --- a/doc/administration/raketasks/check.md +++ b/doc/administration/raketasks/check.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Integrity check Rake task **(CORE ONLY)** +# Integrity check Rake task **(FREE SELF)** GitLab provides Rake tasks to check the integrity of various components. diff --git a/doc/administration/raketasks/doctor.md b/doc/administration/raketasks/doctor.md index c80f580cce6..ec3f7835b9c 100644 --- a/doc/administration/raketasks/doctor.md +++ b/doc/administration/raketasks/doctor.md @@ -4,7 +4,7 @@ group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Doctor Rake tasks **(CORE ONLY)** +# Doctor Rake tasks **(FREE SELF)** This is a collection of tasks to help investigate and repair problems caused by data integrity issues. diff --git a/doc/administration/raketasks/geo.md b/doc/administration/raketasks/geo.md index 492fe4b52ca..3112e5f61b1 100644 --- a/doc/administration/raketasks/geo.md +++ b/doc/administration/raketasks/geo.md @@ -4,7 +4,7 @@ group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Geo Rake Tasks **(PREMIUM ONLY)** +# Geo Rake Tasks **(PREMIUM SELF)** The following Rake tasks are for [Geo installations](../geo/index.md). diff --git a/doc/administration/raketasks/github_import.md b/doc/administration/raketasks/github_import.md index 630570cb81f..c29865be56c 100644 --- a/doc/administration/raketasks/github_import.md +++ b/doc/administration/raketasks/github_import.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitHub import **(CORE ONLY)** +# GitHub import **(FREE SELF)** > [Introduced]( https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10308) in GitLab 9.1. diff --git a/doc/administration/raketasks/ldap.md b/doc/administration/raketasks/ldap.md index 7dafda89e3c..531e9e89020 100644 --- a/doc/administration/raketasks/ldap.md +++ b/doc/administration/raketasks/ldap.md @@ -4,7 +4,7 @@ group: Access info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# LDAP Rake tasks **(CORE ONLY)** +# LDAP Rake tasks **(FREE SELF)** The following are LDAP-related Rake tasks. @@ -34,7 +34,7 @@ limit by passing a number to the check task: rake gitlab:ldap:check[50] ``` -## Run a group sync **(STARTER ONLY)** +## Run a group sync **(PREMIUM SELF)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/14735) in [GitLab Starter](https://about.gitlab.com/pricing/) 12.2. @@ -233,9 +233,9 @@ It can also be used as a receiving application for content encrypted with a KMS: gcloud kms decrypt --key my-key --keyring my-test-kms --plaintext-file=- --ciphertext-file=my-file --location=us-west1 | sudo gitlab-rake gitlab:ldap:secret:write ``` -**gcloud secret integration example** +**Google Cloud secret integration example** -It can also be used as a receiving application for secrets out of gcloud: +It can also be used as a receiving application for secrets out of Google Cloud: ```shell gcloud secrets versions access latest --secret="my-test-secret" > $1 | sudo gitlab-rake gitlab:ldap:secret:write diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md index 26381434ad4..85e905702c8 100644 --- a/doc/administration/raketasks/maintenance.md +++ b/doc/administration/raketasks/maintenance.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Maintenance Rake tasks **(CORE ONLY)** +# Maintenance Rake tasks **(FREE SELF)** GitLab provides Rake tasks for general maintenance. @@ -64,9 +64,10 @@ Repository storage paths: GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell ``` -## Show GitLab license information **(STARTER ONLY)** +## Show GitLab license information **(PREMIUM SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20501) in GitLab Starter 12.6. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20501) in GitLab 12.6. +> - [Moved](../../subscriptions/bronze_starter.md) to GitLab Premium in 13.9. This command shows information about your [GitLab license](../../user/admin_area/license.md) and how many seats are used. It is only available on GitLab Enterprise @@ -325,6 +326,39 @@ sudo gitlab-rake db:migrate After the command completes, run `sudo gitlab-rake db:migrate:status` to check if all migrations are completed (have an `up` status). +## Rebuild database indexes + +WARNING: +This is an experimental feature that isn't enabled by default. + +Database indexes can be rebuilt regularly to reclaim space and maintain healthy levels of index bloat over time. + +In order to rebuild the two indexes with the highest estimated bloat, use the following Rake task: + +```shell +sudo gitlab-rake gitlab:db:reindex +``` + +In order to target a specific index, use the following Rake task: + +```shell +sudo gitlab-rake gitlab:db:reindex['public.a_specific_index'] +``` + +The following index types are not supported: + +1. Unique and primary key indexes +1. Indexes used for constraint exclusion +1. Partitioned indexes +1. Expression indexes + +Optionally, this Rake task sends annotations to a Grafana (4.6 or later) endpoint. Use the following custom environment variables in order to enable annotations: + +1. `GRAFANA_API_URL` - Grafana's base URL, for example `http://some-host:3000`. +1. `GRAFANA_API_KEY` - Grafana API key with at least `Editor role`. + +You can also [enable reindexing as a regular cron job](https://docs.gitlab.com/omnibus/settings/database.html#automatic-database-reindexing). + ## Import common metrics Sometimes you may need to re-import the common metrics that power the Metrics dashboards. diff --git a/doc/administration/raketasks/praefect.md b/doc/administration/raketasks/praefect.md index 17c3e44bb7e..5fe0546999b 100644 --- a/doc/administration/raketasks/praefect.md +++ b/doc/administration/raketasks/praefect.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Praefect Rake tasks **(CORE ONLY)** +# Praefect Rake tasks **(FREE SELF)** > [Introduced]( https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28369) in GitLab 12.10. diff --git a/doc/administration/raketasks/project_import_export.md b/doc/administration/raketasks/project_import_export.md index 0ea0bb3c28f..0fe15f2b5ba 100644 --- a/doc/administration/raketasks/project_import_export.md +++ b/doc/administration/raketasks/project_import_export.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Project import/export administration **(CORE ONLY)** +# Project import/export administration **(FREE SELF)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/3050) in GitLab 8.9. > - From GitLab 11.3, import/export can use object storage automatically. diff --git a/doc/administration/raketasks/storage.md b/doc/administration/raketasks/storage.md index 158b541b36b..7fedcbf3c1a 100644 --- a/doc/administration/raketasks/storage.md +++ b/doc/administration/raketasks/storage.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Repository storage Rake tasks **(CORE ONLY)** +# Repository storage Rake tasks **(FREE SELF)** This is a collection of Rake tasks to help you list and migrate existing projects and their attachments to the new diff --git a/doc/administration/raketasks/uploads/migrate.md b/doc/administration/raketasks/uploads/migrate.md index 2f51bf9357f..ab0a51ba8d6 100644 --- a/doc/administration/raketasks/uploads/migrate.md +++ b/doc/administration/raketasks/uploads/migrate.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Uploads migrate Rake tasks **(CORE ONLY)** +# Uploads migrate Rake tasks **(FREE SELF)** There is a Rake task for migrating uploads between different storage types. diff --git a/doc/administration/raketasks/uploads/sanitize.md b/doc/administration/raketasks/uploads/sanitize.md index 637992d52ca..7dc813de14e 100644 --- a/doc/administration/raketasks/uploads/sanitize.md +++ b/doc/administration/raketasks/uploads/sanitize.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Uploads sanitize Rake tasks **(CORE ONLY)** +# Uploads sanitize Rake tasks **(FREE SELF)** In GitLab 11.9 and later, EXIF data is automatically stripped from JPG or TIFF image uploads. diff --git a/doc/administration/read_only_gitlab.md b/doc/administration/read_only_gitlab.md index d1cb53d0090..23f55cb8463 100644 --- a/doc/administration/read_only_gitlab.md +++ b/doc/administration/read_only_gitlab.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Place GitLab into a read-only state **(CORE ONLY)** +# Place GitLab into a read-only state **(FREE SELF)** WARNING: This document should be used as a temporary solution. diff --git a/doc/administration/redis/replication_and_failover.md b/doc/administration/redis/replication_and_failover.md index 1abc52b65cc..20a9fbd7d68 100644 --- a/doc/administration/redis/replication_and_failover.md +++ b/doc/administration/redis/replication_and_failover.md @@ -5,7 +5,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Redis replication and failover with Omnibus GitLab **(PREMIUM ONLY)** +# Redis replication and failover with Omnibus GitLab **(PREMIUM SELF)** NOTE: This is the documentation for the Omnibus GitLab packages. For using your own diff --git a/doc/administration/redis/replication_and_failover_external.md b/doc/administration/redis/replication_and_failover_external.md index 3bcefe2ef13..1b2e93cccd9 100644 --- a/doc/administration/redis/replication_and_failover_external.md +++ b/doc/administration/redis/replication_and_failover_external.md @@ -5,7 +5,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Redis replication and failover providing your own instance **(CORE ONLY)** +# Redis replication and failover providing your own instance **(FREE SELF)** If you’re hosting GitLab on a cloud provider, you can optionally use a managed service for Redis. For example, AWS offers ElastiCache that runs Redis. diff --git a/doc/administration/redis/standalone.md b/doc/administration/redis/standalone.md index 0e7eca84d17..42482475d26 100644 --- a/doc/administration/redis/standalone.md +++ b/doc/administration/redis/standalone.md @@ -5,7 +5,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Standalone Redis using Omnibus GitLab **(CORE ONLY)** +# Standalone Redis using Omnibus GitLab **(FREE SELF)** The Omnibus GitLab package can be used to configure a standalone Redis server. In this configuration, Redis is not scaled, and represents a single diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index afa53b5efa8..f9765183e64 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -5,7 +5,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Reference architecture: up to 10,000 users **(PREMIUM ONLY)** +# Reference architecture: up to 10,000 users **(PREMIUM SELF)** This page describes GitLab reference architecture for up to 10,000 users. For a full list of reference architectures, see @@ -17,21 +17,21 @@ full list of reference architectures, see | Service | Nodes | Configuration | GCP | AWS | Azure | |--------------------------------------------|-------------|-------------------------|-----------------|-------------|----------| -| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Consul | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| PostgreSQL | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Redis - Cache | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis - Queues / Shared State | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | t3.small | B1MS | -| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | t3.small | B1MS | -| Gitaly | 2 (minimum) | 16 vCPU, 60 GB memory | n1-standard-16 | m5.4xlarge | D16s v3 | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| GitLab Rails | 3 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | -| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Consul | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| PostgreSQL | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Redis - Cache | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | +| Redis - Queues / Shared State | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | +| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | `t3.small` | B1MS | +| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | `t3.small` | B1MS | +| Gitaly | 2 (minimum) | 16 vCPU, 60 GB memory | n1-standard-16 | `m5.4xlarge` | D16s v3 | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | +| GitLab Rails | 3 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | `c5.9xlarge` | F32s v2 | +| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | | Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | ```mermaid stateDiagram-v2 @@ -1928,7 +1928,7 @@ GitLab has been tested on a number of object storage providers: - [Google Cloud Storage](https://cloud.google.com/storage) - [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces/) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) -- [Openstack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) +- [OpenStack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) - On-premises hardware and appliances from various storage vendors. - MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. @@ -1957,7 +1957,7 @@ on what features you intend to use: | [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | | [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | | [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) | Yes | -| [Pseudonymizer](../pseudonymizer.md#configuration) (optional feature) **(ULTIMATE ONLY)** | No | +| [Pseudonymizer](../pseudonymizer.md#configuration) (optional feature) **(ULTIMATE SELF)** | No | | [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | | [Terraform state files](../terraform_state.md#using-object-storage) | Yes | @@ -1983,7 +1983,7 @@ work. </a> </div> -## Configure Advanced Search **(STARTER ONLY)** +## Configure Advanced Search **(PREMIUM SELF)** You can leverage Elasticsearch and [enable Advanced Search](../../integration/elasticsearch.md) for faster, more advanced code search across your entire GitLab instance. diff --git a/doc/administration/reference_architectures/1k_users.md b/doc/administration/reference_architectures/1k_users.md index 161964353f5..138d3c34389 100644 --- a/doc/administration/reference_architectures/1k_users.md +++ b/doc/administration/reference_architectures/1k_users.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Reference architecture: up to 1,000 users **(CORE ONLY)** +# Reference architecture: up to 1,000 users **(FREE SELF)** This page describes GitLab reference architecture for up to 1,000 users. For a full list of reference architectures, see @@ -21,8 +21,8 @@ many organizations . | Users | Configuration | GCP | AWS | Azure | |--------------|-------------------------|----------------|-----------------|----------------| -| Up to 500 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | -| Up to 1,000 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | +| Up to 500 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | +| Up to 1,000 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | `c5.2xlarge` | F8s v2 | The Google Cloud Platform (GCP) architectures were built and tested using the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) @@ -45,9 +45,9 @@ To install GitLab for this default reference architecture, use the standard You can also optionally configure GitLab to use an [external PostgreSQL service](../postgresql/external.md) or an [external object storage service](../object_storage.md) for added -performance and reliability at a reduced complexity cost. +performance and reliability at an increased complexity cost. -## Configure Advanced Search **(STARTER ONLY)** +## Configure Advanced Search **(PREMIUM SELF)** You can leverage Elasticsearch and [enable Advanced Search](../../integration/elasticsearch.md) for faster, more advanced code search across your entire GitLab instance. diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index d96e93d4ab4..66a9c4adbac 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -5,7 +5,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Reference architecture: up to 25,000 users **(PREMIUM ONLY)** +# Reference architecture: up to 25,000 users **(PREMIUM SELF)** This page describes GitLab reference architecture for up to 25,000 users. For a full list of reference architectures, see @@ -17,21 +17,21 @@ full list of reference architectures, see | Service | Nodes | Configuration | GCP | AWS | Azure | |-----------------------------------------|-------------|-------------------------|-----------------|-------------|----------| -| External load balancing node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | -| Consul | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| PostgreSQL | 3 | 8 vCPU, 30 GB memory | n1-standard-8 | m5.2xlarge | D8s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Internal load balancing node | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.large | F2s v2 | -| Redis - Cache | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis - Queues / Shared State | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | t3.small | B1MS | -| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | t3.small | B1MS | -| Gitaly | 2 (minimum) | 32 vCPU, 120 GB memory | n1-standard-32 | m5.8xlarge | D32s v3 | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| GitLab Rails | 5 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | -| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| External load balancing node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | +| Consul | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| PostgreSQL | 3 | 8 vCPU, 30 GB memory | n1-standard-8 | `m5.2xlarge` | D8s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Internal load balancing node | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | `c5.large` | F2s v2 | +| Redis - Cache | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | +| Redis - Queues / Shared State | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | +| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | `t3.small` | B1MS | +| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | `t3.small` | B1MS | +| Gitaly | 2 (minimum) | 32 vCPU, 120 GB memory | n1-standard-32 | `m5.8xlarge` | D32s v3 | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | +| GitLab Rails | 5 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | `c5.9xlarge` | F32s v2 | +| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | | Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | ```mermaid stateDiagram-v2 @@ -1928,7 +1928,7 @@ GitLab has been tested on a number of object storage providers: - [Google Cloud Storage](https://cloud.google.com/storage) - [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces/) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) -- [Openstack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) +- [OpenStack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) - On-premises hardware and appliances from various storage vendors. - MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. @@ -1957,7 +1957,7 @@ on what features you intend to use: | [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | | [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | | [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) | Yes | -| [Pseudonymizer](../pseudonymizer.md#configuration) (optional feature) **(ULTIMATE ONLY)** | No | +| [Pseudonymizer](../pseudonymizer.md#configuration) (optional feature) **(ULTIMATE SELF)** | No | | [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | | [Terraform state files](../terraform_state.md#using-object-storage) | Yes | @@ -1983,7 +1983,7 @@ work. </a> </div> -## Configure Advanced Search **(STARTER ONLY)** +## Configure Advanced Search **(PREMIUM SELF)** You can leverage Elasticsearch and [enable Advanced Search](../../integration/elasticsearch.md) for faster, more advanced code search across your entire GitLab instance. diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index 3d38586fa62..1d11f972c2a 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -5,7 +5,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Reference architecture: up to 2,000 users **(CORE ONLY)** +# Reference architecture: up to 2,000 users **(FREE SELF)** This page describes GitLab reference architecture for up to 2,000 users. For a full list of reference architectures, see @@ -18,14 +18,14 @@ For a full list of reference architectures, see | Service | Nodes | Configuration | GCP | AWS | Azure | |------------------------------------------|--------|-------------------------|----------------|--------------|---------| -| Load balancer | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| PostgreSQL | 1 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | -| Redis | 1 | 1 vCPU, 3.75 GB memory | n1-standard-1 | m5.large | D2s v3 | -| Gitaly | 1 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| GitLab Rails | 2 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | -| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Load balancer | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| PostgreSQL | 1 | 2 vCPU, 7.5 GB memory | n1-standard-2 | `m5.large` | D2s v3 | +| Redis | 1 | 1 vCPU, 3.75 GB memory | n1-standard-1 | `m5.large` | D2s v3 | +| Gitaly | 1 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | +| GitLab Rails | 2 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | `c5.2xlarge` | F8s v2 | +| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | | Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | ```mermaid stateDiagram-v2 @@ -789,7 +789,7 @@ running [Prometheus](../monitoring/prometheus/index.md) and gitlab_exporter['enable'] = false ``` -1. Prometheus also needs some scrape configs to pull all the data from the various +1. Prometheus also needs some scrape configurations to pull all the data from the various nodes where we configured exporters. Assuming that your nodes' IPs are: ```plaintext @@ -879,7 +879,7 @@ GitLab has been tested on a number of object storage providers: - [Google Cloud Storage](https://cloud.google.com/storage) - [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces/) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) -- [Openstack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) +- [OpenStack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) - On-premises hardware and appliances from various storage vendors. - MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. @@ -908,7 +908,7 @@ on what features you intend to use: | [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | | [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | | [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) | Yes | -| [Pseudonymizer](../pseudonymizer.md#configuration) (optional feature) **(ULTIMATE ONLY)** | No | +| [Pseudonymizer](../pseudonymizer.md#configuration) (optional feature) **(ULTIMATE SELF)** | No | | [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | | [Terraform state files](../terraform_state.md#using-object-storage) | Yes | @@ -932,7 +932,7 @@ functioning backups is encountered. </a> </div> -## Configure Advanced Search **(STARTER ONLY)** +## Configure Advanced Search **(PREMIUM SELF)** You can leverage Elasticsearch and [enable Advanced Search](../../integration/elasticsearch.md) for faster, more advanced code search across your entire GitLab instance. diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index 648fac8025f..0fc48657c05 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -5,7 +5,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Reference architecture: up to 3,000 users **(PREMIUM ONLY)** +# Reference architecture: up to 3,000 users **(PREMIUM SELF)** This page describes GitLab reference architecture for up to 3,000 users. It is designed to help your organization achieve a @@ -25,18 +25,18 @@ For a full list of reference architectures, see | Service | Nodes | Configuration | GCP | AWS | Azure | |--------------------------------------------|-------------|-----------------------|----------------|-------------|---------| -| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Redis | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | -| Consul + Sentinel | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| PostgreSQL | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Gitaly | 2 (minimum) | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | -| GitLab Rails | 3 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | -| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Redis | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | `m5.large` | D2s v3 | +| Consul + Sentinel | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| PostgreSQL | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | `m5.large` | D2s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Gitaly | 2 (minimum) | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | +| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | n1-standard-2 | `m5.large` | D2s v3 | +| GitLab Rails | 3 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | `c5.2xlarge` | F8s v2 | +| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | | Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | ```mermaid stateDiagram-v2 @@ -1621,7 +1621,7 @@ GitLab has been tested on a number of object storage providers: - [Google Cloud Storage](https://cloud.google.com/storage) - [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces/) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) -- [Openstack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) +- [OpenStack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) - On-premises hardware and appliances from various storage vendors. - MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. @@ -1650,7 +1650,7 @@ on what features you intend to use: | [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | | [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | | [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) | Yes | -| [Pseudonymizer](../pseudonymizer.md#configuration) (optional feature) **(ULTIMATE ONLY)** | No | +| [Pseudonymizer](../pseudonymizer.md#configuration) (optional feature) **(ULTIMATE SELF)** | No | | [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | | [Terraform state files](../terraform_state.md#using-object-storage) | Yes | @@ -1676,7 +1676,7 @@ work. </a> </div> -## Configure Advanced Search **(STARTER ONLY)** +## Configure Advanced Search **(PREMIUM SELF)** You can leverage Elasticsearch and [enable Advanced Search](../../integration/elasticsearch.md) for faster, more advanced code search across your entire GitLab instance. diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 093869d331b..7eb56f62774 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -5,7 +5,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Reference architecture: up to 50,000 users **(PREMIUM ONLY)** +# Reference architecture: up to 50,000 users **(PREMIUM SELF)** This page describes GitLab reference architecture for up to 50,000 users. For a full list of reference architectures, see @@ -17,21 +17,21 @@ full list of reference architectures, see | Service | Nodes | Configuration | GCP | AWS | Azure | |-----------------------------------------|-------------|-------------------------|-----------------|--------------|----------| -| External load balancing node | 1 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | -| Consul | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| PostgreSQL | 3 | 16 vCPU, 60 GB memory | n1-standard-16 | m5.4xlarge | D16s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Internal load balancing node | 1 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | -| Redis - Cache | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis - Queues / Shared State | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | t3.small | B1MS | -| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | t3.small | B1MS | -| Gitaly | 2 (minimum) | 64 vCPU, 240 GB memory | n1-standard-64 | m5.16xlarge | D64s v3 | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| GitLab Rails | 12 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | -| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| External load balancing node | 1 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | `c5.2xlarge` | F8s v2 | +| Consul | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| PostgreSQL | 3 | 16 vCPU, 60 GB memory | n1-standard-16 | `m5.4xlarge` | D16s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Internal load balancing node | 1 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | `c5.2xlarge` | F8s v2 | +| Redis - Cache | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | +| Redis - Queues / Shared State | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | +| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | `t3.small` | B1MS | +| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | `t3.small` | B1MS | +| Gitaly | 2 (minimum) | 64 vCPU, 240 GB memory | n1-standard-64 | `m5.16xlarge` | D64s v3 | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | +| GitLab Rails | 12 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | `c5.9xlarge` | F32s v2 | +| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | | Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | ```mermaid stateDiagram-v2 @@ -1928,7 +1928,7 @@ GitLab has been tested on a number of object storage providers: - [Google Cloud Storage](https://cloud.google.com/storage) - [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces/) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) -- [Openstack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) +- [OpenStack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) - On-premises hardware and appliances from various storage vendors. - MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. @@ -1957,7 +1957,7 @@ on what features you intend to use: | [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | | [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | | [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) | Yes | -| [Pseudonymizer](../pseudonymizer.md#configuration) (optional feature) **(ULTIMATE ONLY)** | No | +| [Pseudonymizer](../pseudonymizer.md#configuration) (optional feature) **(ULTIMATE SELF)** | No | | [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | | [Terraform state files](../terraform_state.md#using-object-storage) | Yes | @@ -1983,7 +1983,7 @@ work. </a> </div> -## Configure Advanced Search **(STARTER ONLY)** +## Configure Advanced Search **(PREMIUM SELF)** You can leverage Elasticsearch and [enable Advanced Search](../../integration/elasticsearch.md) for faster, more advanced code search across your entire GitLab instance. diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 16ad866a108..d617920e2e4 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -5,7 +5,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Reference architecture: up to 5,000 users **(PREMIUM ONLY)** +# Reference architecture: up to 5,000 users **(PREMIUM SELF)** This page describes GitLab reference architecture for up to 5,000 users. For a full list of reference architectures, see @@ -24,18 +24,18 @@ costly-to-operate environment by using the | Service | Nodes | Configuration | GCP | AWS | Azure | |--------------------------------------------|-------------|-------------------------|----------------|-------------|----------| -| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Redis | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | -| Consul + Sentinel | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| PostgreSQL | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Gitaly | 2 (minimum) | 8 vCPU, 30 GB memory | n1-standard-8 | m5.2xlarge | D8s v3 | -| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | -| GitLab Rails | 3 | 16 vCPU, 14.4 GB memory | n1-highcpu-16 | c5.4xlarge | F16s v2 | -| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Redis | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | `m5.large` | D2s v3 | +| Consul + Sentinel | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| PostgreSQL | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | `m5.large` | D2s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Gitaly | 2 (minimum) | 8 vCPU, 30 GB memory | n1-standard-8 | `m5.2xlarge` | D8s v3 | +| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | n1-standard-2 | `m5.large` | D2s v3 | +| GitLab Rails | 3 | 16 vCPU, 14.4 GB memory | n1-highcpu-16 | `c5.4xlarge` | F16s v2 | +| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | | Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | ```mermaid stateDiagram-v2 @@ -1620,7 +1620,7 @@ GitLab has been tested on a number of object storage providers: - [Google Cloud Storage](https://cloud.google.com/storage) - [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces/) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) -- [Openstack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) +- [OpenStack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) - On-premises hardware and appliances from various storage vendors. - MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. @@ -1649,7 +1649,7 @@ on what features you intend to use: | [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | | [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | | [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) | Yes | -| [Pseudonymizer](../pseudonymizer.md#configuration) (optional feature) **(ULTIMATE ONLY)** | No | +| [Pseudonymizer](../pseudonymizer.md#configuration) (optional feature) **(ULTIMATE SELF)** | No | | [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | | [Terraform state files](../terraform_state.md#using-object-storage) | Yes | @@ -1675,7 +1675,7 @@ work. </a> </div> -## Configure Advanced Search **(STARTER ONLY)** +## Configure Advanced Search **(PREMIUM SELF)** You can leverage Elasticsearch and [enable Advanced Search](../../integration/elasticsearch.md) for faster, more advanced code search across your entire GitLab instance. diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index b90b8d67b68..035811cd494 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -91,7 +91,7 @@ Also, not implementing extra servers for GitLab doesn't necessarily mean that yo more downtime. Depending on your needs and experience level, single servers can have more actual perceived uptime for your users. -### Automated backups **(CORE ONLY)** +### Automated backups **(FREE SELF)** > - Level of complexity: **Low** > - Required domain knowledge: PostgreSQL, GitLab configurations, Git @@ -102,7 +102,7 @@ this can be an optimal solution if you don't have strict requirements. [Automated backups](../../raketasks/backup_restore.md#configuring-cron-to-make-daily-backups) is the least complex to setup. This provides a point-in-time recovery of a predetermined schedule. -### Traffic load balancer **(STARTER ONLY)** +### Traffic load balancer **(PREMIUM SELF)** > - Level of complexity: **Medium** > - Required domain knowledge: HAProxy, shared storage, distributed systems @@ -124,7 +124,7 @@ to the default installation: For more details on how to configure a traffic load balancer with GitLab, you can refer to any of the [available reference architectures](#available-reference-architectures) with more than 1,000 users. -### Zero downtime updates **(STARTER ONLY)** +### Zero downtime updates **(PREMIUM SELF)** > - Level of complexity: **Medium** > - Required domain knowledge: PostgreSQL, HAProxy, shared storage, distributed systems @@ -134,7 +134,7 @@ Single GitLab nodes can be updated with only a [few minutes of downtime](https:/ To avoid this, we recommend to separate GitLab into several application nodes. As long as at least one of each component is online and capable of handling the instance's usage load, your team's productivity will not be interrupted during the update. -### Automated database failover **(PREMIUM ONLY)** +### Automated database failover **(PREMIUM SELF)** > - Level of complexity: **High** > - Required domain knowledge: PgBouncer, Repmgr or Patroni, shared storage, distributed systems @@ -145,7 +145,7 @@ cluster management and failover policies. [PgBouncer in conjunction with Repmgr or Patroni](../postgresql/replication_and_failover.md) is recommended. -### Instance level replication with GitLab Geo **(PREMIUM ONLY)** +### Instance level replication with GitLab Geo **(PREMIUM SELF)** > - Level of complexity: **Very High** > - Required domain knowledge: Storage replication diff --git a/doc/administration/repository_checks.md b/doc/administration/repository_checks.md index 6f7a72a0ef2..6c7069a5ac6 100644 --- a/doc/administration/repository_checks.md +++ b/doc/administration/repository_checks.md @@ -14,25 +14,24 @@ integrity of all data committed to a repository. GitLab administrators can trigger such a check for a project via the project page under the admin panel. The checks run asynchronously so it may take a few minutes before the check result is visible on the project admin page. If the -checks failed you can see their output on in the [`repocheck.log` -file.](logs.md#repochecklog) +checks failed you can see their output on in the +[`repocheck.log` file.](logs.md#repochecklog) -NOTE: -It is OFF by default because it still causes too many false alarms. +This setting is off by default, because it can cause many false alarms. ## Periodic checks When enabled, GitLab periodically runs a repository check on all project repositories and wiki repositories in order to detect data corruption. -A project will be checked no more than once per month. If any projects -fail their repository checks all GitLab administrators will receive an email +A project is checked no more than once per month. If any projects +fail their repository checks all GitLab administrators receive an email notification of the situation. This notification is sent out once a week, by default, midnight at the start of Sunday. Repositories with known check failures can be found at `/admin/projects?last_repository_check_failed=1`. ## Disabling periodic checks -You can disable the periodic checks on the 'Settings' page of the admin +You can disable the periodic checks on the **Settings** page of the admin panel. ## What to do if a check failed @@ -40,9 +39,9 @@ panel. If the repository check fails for some repository you should look up the error in the [`repocheck.log` file](logs.md#repochecklog) on disk: -- `/var/log/gitlab/gitlab-rails` for Omnibus installations +- `/var/log/gitlab/gitlab-rails` for Omnibus GitLab installations - `/home/git/gitlab/log` for installations from source If the periodic repository check causes false alarms, you can clear all repository check states by -navigating to **Admin Area > Settings > Repository** +going to **Admin Area > Settings > Repository** (`/admin/application_settings/repository`) and clicking **Clear all repository checks**. diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md index c71d1a5714c..1c6218b22be 100644 --- a/doc/administration/repository_storage_paths.md +++ b/doc/administration/repository_storage_paths.md @@ -40,9 +40,9 @@ storage2: ## Configure GitLab -In order for [backups](../raketasks/backup_restore.md) to work correctly, the storage path must **not** be a -mount point and the GitLab user should have correct permissions for the parent -directory of the path. In Omnibus GitLab this is taken care of automatically, +For [backups](../raketasks/backup_restore.md) to work correctly, the storage path cannot be a +mount point, and the GitLab user must have correct permissions for the parent +directory of the path. Omnibus GitLab takes care of these issues for you, but for source installations you should be extra careful. The thing is that for compatibility reasons `gitlab.yml` has a different @@ -52,22 +52,26 @@ indicate `git_data_dirs`, which for the example above would be `/home/git`. Then, Omnibus creates a `repositories` directory under that path to use with `gitlab.yml`. -This little detail matters because while restoring a backup, the current -contents of `/home/git/repositories` [are moved to](https://gitlab.com/gitlab-org/gitlab/blob/033e5423a2594e08a7ebcd2379bd2331f4c39032/lib/backup/repository.rb#L54-56) `/home/git/repositories.old`, -so if `/home/git/repositories` is the mount point, then `mv` would be moving +WARNING: +This detail matters because while restoring a backup, the current +contents of `/home/git/repositories` +[are moved to](https://gitlab.com/gitlab-org/gitlab/blob/033e5423a2594e08a7ebcd2379bd2331f4c39032/lib/backup/repository.rb#L54-56) +`/home/git/repositories.old`. +If `/home/git/repositories` is the mount point, then `mv` would be moving things between mount points, and bad things could happen. Ideally, -`/home/git` would be the mount point, so then things would be moving within the -same mount point. This is guaranteed with Omnibus installations (because they -don't specify the full repository path but the parent path), but not for source -installations. +`/home/git` would be the mount point, so things would remain inside the +same mount point. Omnibus installations guarantee this, because they +don't specify the full repository path but instead the parent path, but source +installations do not. -Now that you've read that big fat warning above, let's edit the configuration -files and add the full paths of the alternative repository storage paths. In +Next, edit the configuration +files, and add the full paths of the alternative repository storage paths. In the example below, we add two more mount points that are named `nfs_1` and `nfs_2` respectively. NOTE: -This example uses NFS. We do not recommend using EFS for storage as it may impact GitLab performance. See the [relevant documentation](nfs.md#avoid-using-awss-elastic-file-system-efs) for more details. +This example uses NFS. We do not recommend using EFS for storage as it may impact GitLab performance. Read +the [relevant documentation](nfs.md#avoid-using-awss-elastic-file-system-efs) for more details. **For installations from source** @@ -112,7 +116,7 @@ working, you can remove the `repos_path` line. ## Choose where new repositories are stored -Once you set the multiple storage paths, you can choose where new repositories +After you set the multiple storage paths, you can choose where new repositories are stored in the Admin Area under **Settings > Repository > Repository storage > Storage nodes for new repositories**. Each storage can be assigned a weight from 0-100. When a new project is created, these diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md index 6ec43a8ce06..52e4db6ca31 100644 --- a/doc/administration/repository_storage_types.md +++ b/doc/administration/repository_storage_types.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference, howto --- -# Repository storage types **(CORE ONLY)** +# Repository storage types **(FREE SELF)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/28283) in GitLab 10.0. > - Hashed storage became the default for new installations in GitLab 12.0 @@ -38,13 +38,13 @@ been disabled. Hashed storage is the storage behavior we rolled out with 10.0. Instead of coupling project URL and the folder structure where the repository is -stored on disk, we are coupling a hash, based on the project's ID. This makes +stored on disk, we couple a hash based on the project's ID. This makes the folder structure immutable, and therefore eliminates any requirement to synchronize state from URLs to disk structure. This means that renaming a group, user, or project costs only the database transaction, and takes effect immediately. -The hash also helps to spread the repositories more evenly on the disk, so the +The hash also helps spread the repositories more evenly on the disk. The top-level directory contains fewer folders than the total number of top-level namespaces. @@ -136,8 +136,8 @@ when housekeeping is run on the source project. ### Hashed storage coverage migration Files stored in an S3-compatible endpoint do not have the downsides -mentioned earlier, if they are not prefixed with `#{namespace}/#{project_name}`, -which is true for CI Cache and LFS Objects. +mentioned earlier, if they are not prefixed with `#{namespace}/#{project_name}`. +This is true for CI Cache and LFS Objects. In the table below, you can find the coverage of the migration to the hashed storage. @@ -161,7 +161,8 @@ When avatar is replaced, `Upload` model is destroyed and a new one takes place w #### CI artifacts -CI Artifacts are S3 compatible since **9.4** (GitLab Premium), and available in GitLab Core since **10.6**. +CI Artifacts are S3 compatible since **9.4** (GitLab Premium), and available in GitLab Free since +**10.6**. #### LFS objects @@ -194,10 +195,10 @@ reasons, GitLab replicated the same mapping structure from the projects URLs: - Project's repository: `#{namespace}/#{project_name}.git` - Project's wiki: `#{namespace}/#{project_name}.wiki.git` -This structure made it simple to migrate from existing solutions to GitLab and -easy for Administrators to find where the repository is stored. +This structure enables you to migrate from existing solutions to GitLab, and +for Administrators to find where the repository is stored. -On the other hand this has some drawbacks: +This approach also has some drawbacks: Storage location concentrates a huge number of top-level namespaces. The impact can be reduced by the introduction of @@ -211,4 +212,4 @@ is at that same URL today. Any change in the URL needs to be reflected on disk (when groups / users or projects are renamed). This can add a lot of load in big installations, -especially if using any type of network based filesystem. +especially if using any type of network based file system. diff --git a/doc/administration/repository_storages.md b/doc/administration/repository_storages.md deleted file mode 100644 index 93d0ee3cac9..00000000000 --- a/doc/administration/repository_storages.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'repository_storage_paths.md' ---- - -This document was moved to [another location](repository_storage_paths.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/restart_gitlab.md b/doc/administration/restart_gitlab.md index 4f104c6a63f..69b3ae5282f 100644 --- a/doc/administration/restart_gitlab.md +++ b/doc/administration/restart_gitlab.md @@ -140,7 +140,7 @@ your server in `/etc/init.d/gitlab`. --- -If you are using other init systems, like systemd, you can check the +If you are using other init systems, like `systemd`, you can check the [GitLab Recipes](https://gitlab.com/gitlab-org/gitlab-recipes/tree/master/init) repository for some unofficial services. These are **not** officially supported so use them at your own risk. diff --git a/doc/administration/scaling/index.md b/doc/administration/scaling/index.md deleted file mode 100644 index 4f934646ed6..00000000000 --- a/doc/administration/scaling/index.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: ../reference_architectures/index.md ---- - -This document was moved to [another location](../reference_architectures/index.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/server_hooks.md b/doc/administration/server_hooks.md index 645d3bc4e9f..93b899d5146 100644 --- a/doc/administration/server_hooks.md +++ b/doc/administration/server_hooks.md @@ -6,7 +6,7 @@ type: reference, howto disqus_identifier: 'https://docs.gitlab.com/ee/administration/custom_hooks.html' --- -# Server hooks **(CORE ONLY)** +# Server hooks **(FREE SELF)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/196051) in GitLab 12.8 replacing Custom Hooks. diff --git a/doc/administration/snippets/index.md b/doc/administration/snippets/index.md index 0bf0eb3b1ed..61d0b1ec50a 100644 --- a/doc/administration/snippets/index.md +++ b/doc/administration/snippets/index.md @@ -5,7 +5,7 @@ group: Editor info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" --- -# Snippets settings **(CORE ONLY)** +# Snippets settings **(FREE SELF)** Adjust the snippets' settings of your GitLab instance. @@ -18,7 +18,7 @@ abuse of the feature. The default value is **52428800 Bytes** (50 MB). ### How does it work? -The content size limit will be applied when a snippet is created or updated. +The content size limit is applied when a snippet is created or updated. This limit doesn't affect existing snippets until they're updated and their content changes. @@ -60,8 +60,8 @@ To retrieve the current value, start the Rails console and run: #### Through the API -The process to set the snippets size limit through the Application Settings API is -exactly the same as you would do to [update any other setting](../../api/settings.md#change-application-settings). +To set the snippets size limit through the Application Settings API (similar to +[updating any other setting](../../api/settings.md#change-application-settings)), use this command: ```shell curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/application/settings?snippet_size_limit=52428800" diff --git a/doc/administration/static_objects_external_storage.md b/doc/administration/static_objects_external_storage.md index b10af12de67..947e2d43fe9 100644 --- a/doc/administration/static_objects_external_storage.md +++ b/doc/administration/static_objects_external_storage.md @@ -9,8 +9,8 @@ type: reference > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/31025) in GitLab 12.3. -GitLab can be configured to serve repository static objects (for example, archives or raw blobs) from an external -storage, such as a Content Delivery Network (CDN). +You can configure GitLab to serve repository static objects, like archives or raw blobs, +from an external storage, such as a Content Delivery Network (CDN). ## Configuring @@ -19,19 +19,22 @@ To configure external storage for static objects: 1. Navigate to **Admin Area > Settings > Repository**. 1. Expand the **Repository static objects** section. 1. Enter the base URL and an arbitrary token. When you [set up external storage](#set-up-external-storage), -you'll use a script that uses these values as `ORIGIN_HOSTNAME` and `STORAGE_TOKEN`. + use a script that sets these values as `ORIGIN_HOSTNAME` and `STORAGE_TOKEN`. The token is required to distinguish requests coming from the external storage, so users don't -circumvent the external storage and go for the application directly. The token is expected to be -set in the `X-Gitlab-External-Storage-Token` header in requests originating from the external -storage. +circumvent the external storage and access the application directly. GitLab expects +this token to be set in the `X-Gitlab-External-Storage-Token` header in requests +originating from the external storage. ## Serving private static objects -GitLab will append a user-specific token for static object URLs that belong to private projects, -so an external storage can be authenticated on behalf of the user. When processing requests originating -from the external storage, GitLab will look for the token in the `token` query parameter or in -the `X-Gitlab-Static-Object-Token` header to check the user's ability to access the requested object. +GitLab appends a user-specific token for static object URLs belonging to private projects, +so an external storage can be authenticated on the user's behalf. When processing requests originating +from the external storage, GitLab checks the following places to confirm the user may +access the requested object: + +- The `token` query parameter. +- The `X-Gitlab-Static-Object-Token` header. ## Requests flow example @@ -66,8 +69,8 @@ other CDNs or Function as a Service (FaaS) systems should work using the same pr 1. In the following script, set the following values for the first two constants: - `ORIGIN_HOSTNAME`: the hostname of your GitLab installation. - - `STORAGE_TOKEN`: any arbitrary secure token (e.g. you can get one by running - `pwgen -cn1 64` on a UNIX machine). Save this token for the admin panel, as + - `STORAGE_TOKEN`: any arbitrary secure token. You can get a token by running + `pwgen -cn1 64` on a UNIX machine. Save this token for the admin panel, as described in the [configuring](#configuring) section. ```javascript diff --git a/doc/administration/terraform_state.md b/doc/administration/terraform_state.md index be5647aa133..5cc377a45fc 100644 --- a/doc/administration/terraform_state.md +++ b/doc/administration/terraform_state.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Terraform state administration (alpha) +# Terraform state administration (alpha) **(CORE)** > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2673) in GitLab 12.10. @@ -47,7 +47,7 @@ Terraform state files are stored locally, follow the steps below. 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. -## Using object storage **(CORE ONLY)** +## Using object storage **(FREE SELF)** Instead of storing Terraform state files on disk, we recommend the use of [one of the supported object storage options](object_storage.md#options). This configuration relies on valid credentials to @@ -100,6 +100,11 @@ See [the available connection settings for different providers](object_storage.m ``` 1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Migrate any existing local states to the object storage (GitLab 13.9 and later): + + ```shell + gitlab-rake gitlab:terraform_states:migrate + ``` **In installations from source:** @@ -120,3 +125,8 @@ See [the available connection settings for different providers](object_storage.m ``` 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. +1. Migrate any existing local states to the object storage (GitLab 13.9 and later): + + ```shell + sudo -u git -H bundle exec rake gitlab:terraform_states:migrate RAILS_ENV=production + ``` diff --git a/doc/administration/timezone.md b/doc/administration/timezone.md index 798e7f5050c..6f460ed0ea8 100644 --- a/doc/administration/timezone.md +++ b/doc/administration/timezone.md @@ -41,3 +41,18 @@ After adding the configuration parameter, reconfigure and restart your GitLab in gitlab-ctl reconfigure gitlab-ctl restart ``` + +## Changing time zone per user + +To allow users to change the time zone in their profile, the feature flag `user_time_settings` should be enabled: + +1. [Start a Rails console session](operations/rails_console.md). +1. Enable the feature flag: + + ```ruby + Feature.enable(:user_time_settings) + ``` + +1. You should now be able to see the timezone dropdown in the users' **Settings > Profile** page. + +  diff --git a/doc/administration/troubleshooting/debug.md b/doc/administration/troubleshooting/debug.md index 8c8fa25aa5e..7b57cdbf17f 100644 --- a/doc/administration/troubleshooting/debug.md +++ b/doc/administration/troubleshooting/debug.md @@ -126,7 +126,7 @@ an SMTP server, but you're not seeing mail delivered. Here's how to check the se For more advanced issues, `gdb` is a must-have tool for debugging issues. -### The GNU Project Debugger (gdb) +### The GNU Project Debugger (GDB) To install on Ubuntu/Debian: @@ -140,9 +140,13 @@ On CentOS: sudo yum install gdb ``` +<!-- vale gitlab.Spelling = NO --> + ### rbtrace -GitLab 11.2 ships with [rbtrace](https://github.com/tmm1/rbtrace), which +<!-- vale gitlab.Spelling = YES --> + +GitLab 11.2 ships with [`rbtrace`](https://github.com/tmm1/rbtrace), which allows you to trace Ruby code, view all running threads, take memory dumps, and more. However, this is not enabled by default. To enable it, define the `ENABLE_RBTRACE` variable to the environment. For example, in Omnibus: @@ -175,7 +179,7 @@ downtime. Otherwise skip to the next section. 1. Load the problematic URL 1. Run `sudo gdb -p <PID>` to attach to the Unicorn process. -1. In the gdb window, type: +1. In the GDB window, type: ```plaintext call (void) rb_backtrace() @@ -210,7 +214,7 @@ downtime. Otherwise skip to the next section. ``` Note that if the Unicorn process terminates before you are able to run these -commands, gdb will report an error. To buy more time, you can always raise the +commands, GDB will report an error. To buy more time, you can always raise the Unicorn timeout. For omnibus users, you can edit `/etc/gitlab/gitlab.rb` and increase it from 60 seconds to 300: @@ -246,7 +250,7 @@ separate Rails process to debug the issue: ``` 1. In a new window, run `top`. It should show this Ruby process using 100% CPU. Write down the PID. -1. Follow step 2 from the previous section on using gdb. +1. Follow step 2 from the previous section on using GDB. ### GitLab: API is not accessible @@ -279,4 +283,4 @@ The output in `/tmp/unicorn.txt` may help diagnose the root cause. ## More information - [Debugging Stuck Ruby Processes](https://blog.newrelic.com/engineering/debugging-stuck-ruby-processes-what-to-do-before-you-kill-9/) -- [Cheatsheet of using gdb and Ruby processes](gdb-stuck-ruby.txt) +- [Cheat sheet of using GDB and Ruby processes](gdb-stuck-ruby.txt) diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 4a112733bfa..c654e029a9c 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# GitLab Rails Console Cheat Sheet **(CORE ONLY)** +# GitLab Rails Console Cheat Sheet **(FREE SELF)** This is the GitLab Support Team's collection of information regarding the GitLab Rails console, for use while troubleshooting. It is listed here for transparency, diff --git a/doc/administration/troubleshooting/group_saml_scim.md b/doc/administration/troubleshooting/group_saml_scim.md index e9dbbfbde12..f0d44a578ff 100644 --- a/doc/administration/troubleshooting/group_saml_scim.md +++ b/doc/administration/troubleshooting/group_saml_scim.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Troubleshooting Group SAML and SCIM **(SILVER ONLY)** +# Troubleshooting Group SAML and SCIM **(PREMIUM SAAS)** These are notes and screenshots regarding Group SAML and SCIM that the GitLab Support Team sometimes uses while troubleshooting, but which do not fit into the official documentation. GitLab is making this public, so that anyone can make use of the Support team’s collected knowledge. diff --git a/doc/administration/troubleshooting/index.md b/doc/administration/troubleshooting/index.md index 67115ce31c0..63b056df8b7 100644 --- a/doc/administration/troubleshooting/index.md +++ b/doc/administration/troubleshooting/index.md @@ -13,7 +13,7 @@ in case something goes wrong: - [Diagnostics tools](diagnostics_tools.md) - [Elasticsearch](elasticsearch.md) - [GitLab Rails console cheat sheet](gitlab_rails_cheat_sheet.md) -- [Group SAML and SCIM troubleshooting](group_saml_scim.md) **(SILVER ONLY)** +- [Group SAML and SCIM troubleshooting](group_saml_scim.md) **(PREMIUM SAAS)** - [Kubernetes cheat sheet](kubernetes_cheat_sheet.md) - [Linux cheat sheet](linux_cheat_sheet.md) - [Parsing GitLab logs with `jq`](log_parsing.md) diff --git a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md index 07a7baf338b..7b8828dc14e 100644 --- a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md +++ b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md @@ -70,7 +70,7 @@ and they will assist you with any issues you are having. kubectl logs <pod-name> --previous ``` - No logs are kept in the containers/pods themselves. Everything is written to stdout. + No logs are kept in the containers/pods themselves. Everything is written to `stdout`. This is the principle of Kubernetes, read [Twelve-factor app](https://12factor.net/) for details. @@ -88,7 +88,7 @@ and they will assist you with any issues you are having. - Minimal configuration that can be used to [test a Kubernetes Helm chart](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/620). -- Tailing logs of a separate pod. An example for a Webservice pod: +- Tailing logs of a separate pod. An example for a `webservice` pod: ```shell kubectl logs gitlab-webservice-54fbf6698b-hpckq -c webservice diff --git a/doc/administration/troubleshooting/linux_cheat_sheet.md b/doc/administration/troubleshooting/linux_cheat_sheet.md index c61a78624c3..c4e991ccc1b 100644 --- a/doc/administration/troubleshooting/linux_cheat_sheet.md +++ b/doc/administration/troubleshooting/linux_cheat_sheet.md @@ -55,7 +55,7 @@ chown root:git <file_or_dir> chmod u+x <file> ``` -### Files & Dirs +### Files and directories ```shell # create a new directory and all subdirectories @@ -202,7 +202,7 @@ or you can build it from source if you have the Rust compiler. First run the tool with no arguments other than the strace output filename to get a summary of the top processes sorted by time spent actively performing tasks. You -can also sort based on total time, # of syscalls made, PID #, and # of child processes +can also sort based on total time, # of system calls made, PID #, and # of child processes using the `-S` or `--sort` flag. The number of results defaults to 25 processes, but can be changed using the `-c`/`--count` option. See `--help` for full details. @@ -220,7 +220,7 @@ Top 25 PIDs ... ``` -Based on the summary, you can then view the details of syscalls made by one or more +Based on the summary, you can then view the details of system calls made by one or more processes using the `-p`/`--pid` for a specific process, or `-s`/`--stats` flags for a sorted list. `--stats` takes the same sorting and count options as summary. @@ -266,7 +266,7 @@ Rough numbers for calls to `open` and `openat` (used to access files) on various Slow storage can cause the dreaded `DeadlineExceeded` error in Gitaly. Also [see this entry](../operations/filesystem_benchmarking.md) -in the handbook for quick tests customers can perform to check their filesystem performance. +in the handbook for quick tests customers can perform to check their file system performance. Keep in mind that timing information from `strace` is often somewhat inaccurate, so small differences should not be considered significant. @@ -304,6 +304,24 @@ whois <ip_address> | grep -i "orgname\|netname" # Curl headers with redirect curl --head --location "https://example.com" + +# Test if a host is reachable on the network. `ping6` works on IPv6 networks. +ping example.com + +# Show the route taken to a host. `traceroute6` works on IPv6 networks. +traceroute example.com +mtr example.com + +# List details of network interfaces +ip address + +# Check local DNS settings +cat /etc/hosts +cat /etc/resolv.conf +systemd-resolve --status + +# Capture traffic to/from a host +sudo tcpdump host www.example.com ``` ## Package Management diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index 49af8358e29..d3a9777775f 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Uploads administration **(CORE ONLY)** +# Uploads administration **(FREE SELF)** Uploads represent all user data that may be sent to GitLab as a single file. As an example, avatars and notes' attachments are uploads. Uploads are integral to GitLab functionality, and therefore cannot be disabled. @@ -49,12 +49,12 @@ _The uploads are stored by default in 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. -## Using object storage **(CORE ONLY)** +## Using object storage **(FREE SELF)** > **Notes:** > > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3867) in [GitLab Premium](https://about.gitlab.com/pricing/) 10.5. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17358) in [GitLab Core](https://about.gitlab.com/pricing/) 10.7. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17358) in [GitLab Free](https://about.gitlab.com/pricing/) 10.7. > - Since version 11.1, we support direct_upload to S3. If you don't want to use the local disk where GitLab is installed to store the @@ -112,7 +112,24 @@ _The uploads are stored by default in ``` 1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate:all` Rake task](raketasks/uploads/migrate.md). +1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate` Rake task](raketasks/uploads/migrate.md). +1. Optional: Verify all files migrated properly. + From [PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database) + (`sudo gitlab-psql -d gitlabhq_production`) verify `objectstg` below (where `store=2`) has count of all artifacts: + + ```shell + gitlabhq_production=# SELECT count(*) AS total, sum(case when store = '1' then 1 else 0 end) AS filesystem, sum(case when store = '2' then 1 else 0 end) AS objectstg FROM uploads; + + total | filesystem | objectstg + ------+------------+----------- + 2409 | 0 | 2409 + ``` + + Verify no files on disk in `artifacts` folder: + + ```shell + sudo find /var/opt/gitlab/gitlab-rails/uploads -type f | grep -v tmp | wc -l + ``` **In installations from source:** @@ -136,6 +153,22 @@ _The uploads are stored by default in 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. 1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate:all` Rake task](raketasks/uploads/migrate.md). +1. Optional: Verify all files migrated properly. + From PostgreSQL console (`sudo -u git -H psql -d gitlabhq_production`) verify `objectstg` below (where `file_store=2`) has count of all artifacts: + + ```shell + gitlabhq_production=# SELECT count(*) AS total, sum(case when store = '1' then 1 else 0 end) AS filesystem, sum(case when store = '2' then 1 else 0 end) AS objectstg FROM uploads; + + total | filesystem | objectstg + ------+------------+----------- + 2409 | 0 | 2409 + ``` + + Verify no files on disk in `artifacts` folder: + + ```shell + sudo find /var/opt/gitlab/gitlab-rails/uploads -type f | grep -v tmp | wc -l + ``` #### OpenStack example @@ -162,6 +195,23 @@ _The uploads are stored by default in 1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate:all` Rake task](raketasks/uploads/migrate.md). +1. Optional: Verify all files migrated properly. + From [PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database) + (`sudo gitlab-psql -d gitlabhq_production`) verify `objectstg` below (where `store=2`) has count of all artifacts: + + ```shell + gitlabhq_production=# SELECT count(*) AS total, sum(case when store = '1' then 1 else 0 end) AS filesystem, sum(case when store = '2' then 1 else 0 end) AS objectstg FROM uploads; + + total | filesystem | objectstg + ------+------------+----------- + 2409 | 0 | 2409 + ``` + + Verify no files on disk in `artifacts` folder: + + ```shell + sudo find /var/opt/gitlab/gitlab-rails/uploads -type f | grep -v tmp | wc -l + ``` --- @@ -193,3 +243,19 @@ _The uploads are stored by default in 1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate:all` Rake task](raketasks/uploads/migrate.md). +1. Optional: Verify all files migrated properly. + From PostgreSQL console (`sudo -u git -H psql -d gitlabhq_production`) verify `objectstg` below (where `file_store=2`) has count of all artifacts: + + ```shell + gitlabhq_production=# SELECT count(*) AS total, sum(case when store = '1' then 1 else 0 end) AS filesystem, sum(case when store = '2' then 1 else 0 end) AS objectstg FROM uploads; + + total | filesystem | objectstg + ------+------------+----------- + 2409 | 0 | 2409 + ``` + + Verify no files on disk in `artifacts` folder: + + ```shell + sudo find /var/opt/gitlab/gitlab-rails/uploads -type f | grep -v tmp | wc -l + ``` diff --git a/doc/administration/user_settings.md b/doc/administration/user_settings.md index 9892d2a0764..681ce87edb6 100644 --- a/doc/administration/user_settings.md +++ b/doc/administration/user_settings.md @@ -1,6 +1,6 @@ --- -stage: Enablement -group: Distribution +stage: Manage +group: Access info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -10,7 +10,8 @@ GitLab administrators can modify user settings for the entire GitLab instance. ## Disallow users creating top-level groups -By default, new users can create top-level groups. To disable this, modify the appropriate configuration file. +By default, new users can create top-level groups. To disable this, modify the appropriate configuration file, +and then [reconfigure and restart GitLab](restart_gitlab.md). For Omnibus installations, add the following to `/etc/gitlab/gitlab.rb`: @@ -26,7 +27,8 @@ For source installations, uncomment the following line in `config/gitlab.yml`: ## Disallow users changing usernames -By default, new users can change their usernames. To disable this, modify the appropriate configuration file. +By default, new users can change their usernames. To disable this, modify the appropriate configuration file, +and then [reconfigure and restart GitLab](restart_gitlab.md). For Omnibus installations, add the following to `/etc/gitlab/gitlab.rb`: diff --git a/doc/administration/wikis/index.md b/doc/administration/wikis/index.md index 026f9b6f471..57bbe913216 100644 --- a/doc/administration/wikis/index.md +++ b/doc/administration/wikis/index.md @@ -1,11 +1,11 @@ --- type: reference, howto stage: Create -group: Knowledge +group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Wiki settings **(CORE ONLY)** +# Wiki settings **(FREE SELF)** Adjust the wiki settings of your GitLab instance. @@ -18,24 +18,24 @@ abuse of the feature. The default value is **52428800 Bytes** (50 MB). ### How does it work? -The content size limit will be applied when a wiki page is created or updated -through the GitLab UI or API. Local changes pushed via Git will not be validated. +The content size limit is applied when a wiki page is created or updated +through the GitLab UI or API. Local changes pushed via Git are not validated. -In order not to break any existing wiki pages, the limit doesn't have any -effect on them until a wiki page is edited again and the content changes. +To break any existing wiki pages, the limit doesn't take effect until a wiki page +is edited again and the content changes. ### Wiki page content size limit configuration This setting is not available through the [Admin Area settings](../../user/admin_area/settings/index.md). -In order to configure this setting, use either the Rails console +To configure this setting, use either the Rails console or the [Application settings API](../../api/settings.md). NOTE: -The value of the limit **must** be in bytes. The minimum value is 1024 bytes. +The value of the limit must be in bytes. The minimum value is 1024 bytes. #### Through the Rails console -The steps to configure this setting through the Rails console are: +To configure this setting through the Rails console: 1. Start the Rails console: @@ -61,14 +61,14 @@ To retrieve the current value, start the Rails console and run: #### Through the API -The process to set the wiki page size limit through the Application Settings API is -exactly the same as you would do to [update any other setting](../../api/settings.md#change-application-settings). +To set the wiki page size limit through the Application Settings API, use a command, +as you would to [update any other setting](../../api/settings.md#change-application-settings): ```shell curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/application/settings?wiki_page_max_content_bytes=52428800" ``` -You can also use the API to [retrieve the current value](../../api/settings.md#get-current-application-settings). +You can also use the API to [retrieve the current value](../../api/settings.md#get-current-application-settings): ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/application/settings" |
