diff options
Diffstat (limited to 'doc/administration')
215 files changed, 2837 insertions, 1304 deletions
diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index 67a3a3c1539..7436661920a 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -1,7 +1,7 @@ --- stage: Manage group: Compliance -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Audit Events **(STARTER)** @@ -44,10 +44,10 @@ Impersonation is where an administrator uses credentials to perform an action as ### Group events **(STARTER)** -NOTE: **Note:** -You need Owner [permissions](../user/permissions.md) to view the group Audit Events page. +A user with a Owner role (or above) can retrieve group audit events of all users. +A user with a Developer or Maintainer role is limited to group audit events based on their individual actions. -To view a group's audit events, navigate to **Group > Settings > Audit Events**. +To view a group's audit events, navigate to **Group > Security & Compliance > Audit Events**. From there, you can see the following actions: - Group name or path changed. @@ -74,10 +74,10 @@ Group events can also be accessed via the [Group Audit Events API](../api/audit_ ### Project events **(STARTER)** -NOTE: **Note:** -You need Maintainer [permissions](../user/permissions.md) or higher to view the project Audit Events page. +A user with a Maintainer role (or above) can retrieve project audit events of all users. +A user with a Developer role is limited to project audit events based on their individual actions. -To view a project's audit events, navigate to **Project > Settings > Audit Events**. +To view a project's audit events, navigate to **Project > Security & Compliance > Audit Events**. From there, you can see the following actions: - Added or removed deploy keys @@ -107,11 +107,11 @@ Project events can also be accessed via the [Project Audit Events API](../api/au > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2336) in [GitLab Premium](https://about.gitlab.com/pricing/) 9.3. -Server-wide audit logging introduces the ability to observe user actions across +Server-wide audit events introduce the ability to observe user actions across the entire instance of your GitLab server, making it easy to understand who changed what and when for audit purposes. -To view the server-wide administrator log, visit **Admin Area > Monitoring > Audit Log**. +To view the server-wide administrator log, visit **Admin Area > Monitoring > Audit Events**. In addition to the group and project events, the following user actions are also recorded: @@ -133,12 +133,6 @@ recorded: - A user's personal access token was successfully created or revoked ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276921) in GitLab 13.6) - A failed attempt to create or revoke a user's personal access token ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276921) in GitLab 13.6) -It's possible to filter particular actions by choosing an audit data type from -the filter dropdown box. You can further filter by specific group, project, or user -(for authentication events). - - - Instance events can also be accessed via the [Instance Audit Events API](../api/audit_events.md#instance-audit-events). ### Missing events @@ -156,11 +150,11 @@ on adding these events into GitLab: #### Repository push The current architecture of audit events is not prepared to receive a very high amount of records. -It may make the user interface for your project or audit logs very busy, and the disk space consumed by the +It may make the user interface for your project or audit events very busy, and the disk space consumed by the `audit_events` PostgreSQL table may increase considerably. It's disabled by default to prevent performance degradations on GitLab instances with very high Git write traffic. -In an upcoming release, Audit Logs for Git push events will be enabled +In an upcoming release, Audit Events for Git push events will be enabled by default. Follow [#7865](https://gitlab.com/gitlab-org/gitlab/-/issues/7865) for updates. If you still wish to enable **Repository push** events in your instance, follow @@ -180,42 +174,37 @@ the steps bellow. Feature.enable(:repository_push_audit_event) ``` -## Retention policy +## Search + +The search filters you can see depends on which audit level you are at. -On GitLab.com, Audit Event records become subject to deletion after 400 days, or when your license is downgraded to a tier that does not include access to Audit Events. Data that is subject to deletion will be deleted at GitLab's discretion, possibly without additional notice. +| Filter | Available options | +| ------ | ----------------- | +| Scope (Project level) | A specific user who performed the action. | +| Scope (Group level) | A specific user (in a group) who performed the action. | +| Scope (Instance level) | A specific group, project, or user that the action was scoped to. | +| Date range | Either via the date range buttons or pickers (maximum range of 31 days). Default is from the first day of the month to today's date. | -If you require a longer retention period, you should independently archive your Audit Event data, which you can retrieve through the [Audit Events API](../api/audit_events.md). + ## Export to CSV **(PREMIUM ONLY)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1449) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.4. -> - It's [deployed behind a feature flag](../user/feature_flags.md), disabled by default. -> - It's disabled on GitLab.com. -> - It's not recommended for production use. -> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-audit-log-export-to-csv). **(PREMIUM ONLY)** - -CAUTION: **Warning:** -This feature might not be available to you. Check the **version history** note above for details. -If available, you can enable it with a [feature flag](#enable-or-disable-audit-log-export-to-csv). - -Export to CSV allows customers to export the current filter view of your audit log as a -CSV file, -which stores tabular data in plain text. The data provides a comprehensive view with respect to +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/285441) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.7. + +Export to CSV allows customers to export the current filter view of your audit events as a +CSV file, which stores tabular data in plain text. The data provides a comprehensive view with respect to audit events. -To export the Audit Log to CSV, navigate to -**{monitor}** **Admin Area > Monitoring > Audit Log** +To export the Audit Events to CSV, navigate to +**{monitor}** **Admin Area > Monitoring > Audit Events** -1. Click in the field **Search**. -1. In the dropdown menu that appears, select the event type that you want to filter by. -1. Select the preferred date range. +1. Select the available search [filters](#search). 1. Click **Export as CSV**. - - ### Sort -Exported events are always sorted by `ID` in ascending order. +Exported events are always sorted by `created_at` in ascending order. ### Format @@ -228,8 +217,8 @@ The first row contains the headers, which are listed in the following table alon | Author ID | ID of the author | | Author Name | Full name of the author | | Entity ID | ID of the scope | -| Entity Type | Type of the entity (`Project`/`Group`/`User`) | -| Entity Path | Path of the entity | +| Entity Type | Type of the scope (`Project`/`Group`/`User`) | +| Entity Path | Path of the scope | | Target ID | ID of the target | | Target Type | Type of the target | | Target Details | Details of the target | @@ -239,24 +228,5 @@ The first row contains the headers, which are listed in the following table alon ### Limitation -The Audit Log CSV file size is limited to a maximum of `100,000` events. +The Audit Events CSV file is limited to a maximum of `100,000` events. The remaining records are truncated when this limit is reached. - -### Enable or disable Audit Log Export to CSV - -The Audit Log Export to CSV is under development and not ready for production use. It is -deployed behind a feature flag that is **disabled by default**. -[GitLab administrators with access to the GitLab Rails console](../administration/feature_flags.md) -can enable it. - -To enable it: - -```ruby -Feature.enable(:audit_log_export_csv) -``` - -To disable it: - -```ruby -Feature.disable(:audit_log_export_csv) -``` diff --git a/doc/administration/audit_reports.md b/doc/administration/audit_reports.md index 9c772302375..2721ee39b60 100644 --- a/doc/administration/audit_reports.md +++ b/doc/administration/audit_reports.md @@ -1,7 +1,7 @@ --- stage: Manage group: Compliance -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments description: 'Learn how to create evidence artifacts typically requested by a 3rd party auditor.' --- diff --git a/doc/administration/auditor_users.md b/doc/administration/auditor_users.md index c41065abd17..7db20efb03f 100644 --- a/doc/administration/auditor_users.md +++ b/doc/administration/auditor_users.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Auditor users **(PREMIUM ONLY)** @@ -58,7 +58,7 @@ helpful: To create a new Auditor user: 1. Create a new user or edit an existing one by navigating to - **Admin Area > Users**. You will find the option of the access level in + **Admin Area > Users**. The option of the access level is located in the 'Access' section.  diff --git a/doc/administration/auth/README.md b/doc/administration/auth/README.md index cf82454cfd2..cc3421d3133 100644 --- a/doc/administration/auth/README.md +++ b/doc/administration/auth/README.md @@ -3,7 +3,7 @@ comments: false type: index stage: Manage group: Access -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # GitLab authentication and authorization @@ -36,5 +36,5 @@ providers: - [Smartcard](smartcard.md) **(PREMIUM ONLY)** - [Twitter](../../integration/twitter.md) -NOTE: **Note:** +NOTE: UltraAuth has removed their software which supports OmniAuth integration. We have therefore removed all references to UltraAuth integration. diff --git a/doc/administration/auth/atlassian.md b/doc/administration/auth/atlassian.md index 3a1f5eeb0c2..9c5da558b0d 100644 --- a/doc/administration/auth/atlassian.md +++ b/doc/administration/auth/atlassian.md @@ -2,7 +2,7 @@ type: reference stage: Manage group: Access -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Atlassian OmniAuth Provider diff --git a/doc/administration/auth/authentiq.md b/doc/administration/auth/authentiq.md index 56f6fddc1af..dbc5e446287 100644 --- a/doc/administration/auth/authentiq.md +++ b/doc/administration/auth/authentiq.md @@ -2,7 +2,7 @@ type: reference stage: Manage group: Access -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Authentiq OmniAuth Provider diff --git a/doc/administration/auth/cognito.md b/doc/administration/auth/cognito.md index e777acd0d32..f264321ea6c 100644 --- a/doc/administration/auth/cognito.md +++ b/doc/administration/auth/cognito.md @@ -2,7 +2,7 @@ type: concepts, howto stage: Manage group: Access -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Amazon Web Services Cognito diff --git a/doc/administration/auth/crowd.md b/doc/administration/auth/crowd.md index 6e3dbcb68b3..440d956fc8f 100644 --- a/doc/administration/auth/crowd.md +++ b/doc/administration/auth/crowd.md @@ -2,12 +2,13 @@ type: reference stage: Manage group: Access -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Atlassian Crowd OmniAuth Provider -Authenticate to GitLab using the Atlassian Crowd OmniAuth provider. +Authenticate to GitLab using the Atlassian Crowd OmniAuth provider. Enabling +this provider also allows Crowd authentication for Git-over-https requests. ## Configure a new Crowd application diff --git a/doc/administration/auth/google_secure_ldap.md b/doc/administration/auth/google_secure_ldap.md index d30efc6d3cc..37366b00f73 100644 --- a/doc/administration/auth/google_secure_ldap.md +++ b/doc/administration/auth/google_secure_ldap.md @@ -3,3 +3,6 @@ redirect_to: 'ldap/google_secure_ldap.md' --- This document was moved to [another location](ldap/google_secure_ldap.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md b/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md index 40d021e180c..ffce06afb63 100644 --- a/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md +++ b/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md @@ -3,3 +3,6 @@ redirect_to: '../ldap/index.md' --- This document was moved to [another location](../ldap/index.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/index.md b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/index.md index 40d021e180c..ffce06afb63 100644 --- a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/index.md +++ b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/index.md @@ -3,3 +3,6 @@ redirect_to: '../ldap/index.md' --- This document was moved to [another location](../ldap/index.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/auth/jwt.md b/doc/administration/auth/jwt.md index d3a77fdaca5..448922230e7 100644 --- a/doc/administration/auth/jwt.md +++ b/doc/administration/auth/jwt.md @@ -2,7 +2,7 @@ type: reference stage: Manage group: Access -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # JWT OmniAuth provider @@ -62,7 +62,7 @@ JWT will provide you with a secret key for you to use. } ``` - NOTE: **Note:** + NOTE: For more information on each configuration option refer to the [OmniAuth JWT usage documentation](https://github.com/mbleigh/omniauth-jwt#usage). diff --git a/doc/administration/auth/ldap-ee.md b/doc/administration/auth/ldap-ee.md index f5565628af1..6d56654a44b 100644 --- a/doc/administration/auth/ldap-ee.md +++ b/doc/administration/auth/ldap-ee.md @@ -3,3 +3,6 @@ redirect_to: 'ldap/index.md' --- This document was moved to [another location](ldap/index.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/auth/ldap-troubleshooting.md b/doc/administration/auth/ldap-troubleshooting.md index a553f449abb..1e02755e3e5 100644 --- a/doc/administration/auth/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap-troubleshooting.md @@ -3,3 +3,6 @@ redirect_to: 'ldap/ldap-troubleshooting.md' --- This document was moved to [another location](ldap/ldap-troubleshooting.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/auth/ldap.md b/doc/administration/auth/ldap.md index f5565628af1..6d56654a44b 100644 --- a/doc/administration/auth/ldap.md +++ b/doc/administration/auth/ldap.md @@ -3,3 +3,6 @@ redirect_to: 'ldap/index.md' --- This document was moved to [another location](ldap/index.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/auth/ldap/google_secure_ldap.md b/doc/administration/auth/ldap/google_secure_ldap.md index 90f0e681dd1..dcfb77f277e 100644 --- a/doc/administration/auth/ldap/google_secure_ldap.md +++ b/doc/administration/auth/ldap/google_secure_ldap.md @@ -2,7 +2,7 @@ type: reference stage: Manage group: Access -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Google Secure LDAP **(CORE ONLY)** @@ -34,7 +34,7 @@ The steps below cover: 'Entire domain (GitLab)' or 'Selected organizational units' for both 'Verify user credentials' and 'Read user information'. Select 'Add LDAP Client' - TIP: **Tip:** + NOTE: If you plan to use GitLab [LDAP Group Sync](index.md#group-sync) , turn on 'Read group information'. @@ -210,6 +210,11 @@ values obtained during the LDAP client configuration earlier: 1. Save the file and [restart](../../restart_gitlab.md#installations-from-source) GitLab for the changes to take effect. +## Using encrypted credentials + +You can optionally store the `bind_dn` and `password` in a separate encrypted configuration file using the +[same steps as the regular LDAP integration](index.md#using-encrypted-credentials). + <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index 9d88d66bf46..4dedaba3754 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -2,7 +2,7 @@ type: reference stage: Manage group: Access -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # General LDAP Setup @@ -46,10 +46,10 @@ the LDAP server or share email addresses. ### User deletion **(CORE ONLY)** -If a user is deleted from the LDAP server, they will be blocked in GitLab as -well. Users will be immediately blocked from logging in. However, there is an +If a user is deleted from the LDAP server, they are also blocked in GitLab. +Users are immediately blocked from logging in. However, there is an LDAP check cache time of one hour (see note) which means users that -are already logged in or are using Git over SSH will still be able to access +are already logged in or are using Git over SSH are be able to access GitLab for up to one hour. Manually block the user in the GitLab Admin Area to immediately block all access. @@ -66,8 +66,8 @@ in the application settings. When a user signs in to GitLab with LDAP for the first time, and their LDAP email address is the primary email address of an existing GitLab user, then -the LDAP DN will be associated with the existing user. If the LDAP email -attribute is not found in GitLab's database, a new user is created. +the LDAP DN is associated with the existing user. If the LDAP email +attribute is not found in the GitLab user database, a new user is created. In other words, if an existing GitLab user wants to enable LDAP sign-in for themselves, they should check that their GitLab email address matches their @@ -91,10 +91,10 @@ There is a Rake task to check LDAP configuration. After configuring LDAP using the documentation below, see [LDAP check Rake task](../../raketasks/check.md#ldap-check) for information on the LDAP check Rake task. -NOTE: **Note:** +NOTE: The `encryption` value `simple_tls` corresponds to 'Simple TLS' in the LDAP library. `start_tls` corresponds to StartTLS, not to be confused with regular TLS. -Normally, if you specify `simple_tls` it will be on port 636, while `start_tls` (StartTLS) +Normally, if you specify `simple_tls` it is on port 636, while `start_tls` (StartTLS) would be on port 389. `plain` also operates on port 389. Removed values: `tls` was replaced with `start_tls` and `ssl` was replaced with `simple_tls`. LDAP users must have a set email address, regardless of whether or not it's used @@ -167,27 +167,27 @@ production: | Setting | Description | Required | Examples | | ------- | ----------- | -------- | -------- | -| `label` | A human-friendly name for your LDAP server. It will be displayed on your sign-in page. | yes | `'Paris'` or `'Acme, Ltd.'` | +| `label` | A human-friendly name for your LDAP server. It is displayed on your sign-in page. | yes | `'Paris'` or `'Acme, Ltd.'` | | `host` | IP address or domain name of your LDAP server. | yes | `'ldap.mydomain.com'` | | `port` | The port to connect with on your LDAP server. Always an integer, not a string. | yes | `389` or `636` (for SSL) | -| `uid` | LDAP attribute for username. Should be the attribute, not the value that maps to the `uid`. | yes | `'sAMAccountName'`, `'uid'`, `'userPrincipalName'` | -| `bind_dn` | The full DN of the user you will bind with. | no | `'america\momo'` or `'CN=Gitlab,OU=Users,DC=domain,DC=com'` | +| `uid` | LDAP attribute for username. Should be the attribute, not the value that maps to the `uid`. | yes | `'sAMAccountName'` or `'uid'` or `'userPrincipalName'` | +| `bind_dn` | The full DN of the user you bind with. | no | `'america\momo'` or `'CN=Gitlab,OU=Users,DC=domain,DC=com'` | | `password` | The password of the bind user. | no | `'your_great_password'` | | `encryption` | Encryption method. The `method` key is deprecated in favor of `encryption`. | yes | `'start_tls'` or `'simple_tls'` or `'plain'` | | `verify_certificates` | Enables SSL certificate verification if encryption method is `start_tls` or `simple_tls`. Defaults to true. | no | boolean | -| `timeout` | Set a timeout, in seconds, for LDAP queries. This helps avoid blocking a request if the LDAP server becomes unresponsive. A value of 0 means there is no timeout. | no | `10` or `30` | +| `timeout` | Set a timeout, in seconds, for LDAP queries. This helps avoid blocking a request if the LDAP server becomes unresponsive. A value of `0` means there is no timeout. (default: `10`) | no | `10` or `30` | | `active_directory` | This setting specifies if LDAP server is Active Directory LDAP server. For non-AD servers it skips the AD specific queries. If your LDAP server is not AD, set this to false. | no | boolean | -| `allow_username_or_email_login` | If enabled, GitLab will ignore everything after the first `@` in the LDAP username submitted by the user on sign-in. If you are using `uid: 'userPrincipalName'` on ActiveDirectory you need to disable this setting, because the userPrincipalName contains an `@`. | no | boolean | -| `block_auto_created_users` | To maintain tight control over the number of active users on your GitLab installation, enable this setting to keep new users blocked until they have been cleared by the admin (default: false). | no | boolean | +| `allow_username_or_email_login` | If enabled, GitLab ignores everything after the first `@` in the LDAP username submitted by the user on sign-in. If you are using `uid: 'userPrincipalName'` on ActiveDirectory you need to disable this setting, because the userPrincipalName contains an `@`. | no | boolean | +| `block_auto_created_users` | To maintain tight control over the number of billable users on your GitLab installation, enable this setting to keep new users blocked until they have been cleared by an administrator (default: false). | no | boolean | | `base` | Base where we can search for users. | yes | `'ou=people,dc=gitlab,dc=example'` or `'DC=mydomain,DC=com'` | | `user_filter` | Filter LDAP users. Format: [RFC 4515](https://tools.ietf.org/search/rfc4515) Note: GitLab does not support `omniauth-ldap`'s custom filter syntax. | no | `'(employeeType=developer)'` or `'(&(objectclass=user)(|(samaccountname=momo)(samaccountname=toto)))'` | -| `lowercase_usernames` | If lowercase_usernames is enabled, GitLab will lower case the username. | no | boolean | +| `lowercase_usernames` | If lowercase_usernames is enabled, GitLab converts the name to lower case. | no | boolean | ### SSL Configuration Settings **(CORE ONLY)** | Setting | Description | Required | Examples | | ------- | ----------- | -------- | -------- | -| `ca_file` | Specifies the path to a file containing a PEM-format CA certificate, e.g. if you need to use an internal CA. | no | `'/etc/ca.pem'` | +| `ca_file` | Specifies the path to a file containing a PEM-format CA certificate, for example, if you need to use an internal CA. | no | `'/etc/ca.pem'` | | `ssl_version` | Specifies the SSL version for OpenSSL to use, if the OpenSSL default is not appropriate. | no | `'TLSv1_1'` | | `ciphers` | Specific SSL ciphers to use in communication with LDAP servers. | no | `'ALL:!EXPORT:!LOW:!aNULL:!eNULL:!SSLv2'` | | `cert` | Client certificate | no | `'-----BEGIN CERTIFICATE----- <REDACTED> -----END CERTIFICATE -----'` | @@ -195,11 +195,11 @@ production: ### Attribute Configuration Settings **(CORE ONLY)** -LDAP attributes that GitLab will use to create an account for the LDAP user. The specified attribute can either be the attribute name as a string (e.g. `'mail'`), or an array of attribute names to try in order (e.g. `['mail', 'email']`). Note that the user's LDAP sign-in will always be the attribute specified as `uid` above. +LDAP attributes that GitLab uses to create an account for the LDAP user. The specified attribute can either be the attribute name as a string (for example, `'mail'`), or an array of attribute names to try in order (for example, `['mail', 'email']`). Note that the user's LDAP sign-in is the attribute specified as `uid` above. | Setting | Description | Required | Examples | | ------- | ----------- | -------- | -------- | -| `username` | The username will be used in paths for the user's own projects (like `gitlab.example.com/username/project`) and when mentioning them in issues, merge request and comments (like `@username`). If the attribute specified for `username` contains an email address, the GitLab username will be the part of the email address before the `@`. | no | `['uid', 'userid', 'sAMAccountName']` | +| `username` | The username is used in paths for the user's own projects (like `gitlab.example.com/username/project`) and when mentioning them in issues, merge request and comments (like `@username`). If the attribute specified for `username` contains an email address, the GitLab username is part of the email address before the `@`. | no | `['uid', 'userid', 'sAMAccountName']` | | `email` | LDAP attribute for user email. | no | `['mail', 'email', 'userPrincipalName']` | | `name` | LDAP attribute for user display name. If no full name could be found at the attribute specified for `name`, the full name is determined using the attributes specified for `first_name` and `last_name`. | no | `'cn'` or `'displayName'` | | `first_name` | LDAP attribute for user first name. | no | `'givenName'` | @@ -335,7 +335,7 @@ an alternative such as SAML is preferred. This allows LDAP to be used for group sync, while also allowing your SAML identity provider to handle additional checks like custom 2FA. -When LDAP web sign in is disabled, users will not see a **LDAP** tab on the sign in page. +When LDAP web sign in is disabled, users don't see an **LDAP** tab on the sign in page. This does not disable [using LDAP credentials for Git access](#git-password-authentication). **Omnibus configuration** @@ -360,6 +360,93 @@ This does not disable [using LDAP credentials for Git access](#git-password-auth 1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. +### Using encrypted credentials **(CORE ONLY)** + +Instead of having the LDAP integration credentials stored in plaintext in the configuration files, you can optionally +use an encrypted file for the LDAP credentials. To use this feature, you first need to enable +[GitLab encrypted configuration](../../encrypted_configuration.md). + +The encrypted configuration for LDAP exists in an encrypted YAML file. By default the file will be created at +`shared/encrypted_configuration/ldap.yaml.enc`. This location is configurable in the GitLab configuration. + +The unencrypted contents of the file should be a subset of the secret settings from your `servers` block in the LDAP +configuration. + +The supported configuration items for the encrypted file are: + +- `bind_dn` +- `password` + +The encrypted contents can be configured with the [LDAP secret edit Rake command](../../raketasks/ldap.md#edit-secret). + +**Omnibus configuration** + +If initially your LDAP configuration looked like: + +1. In `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_servers'] = { + 'main' => { + # snip... + 'bind_dn' => 'admin', + 'password' => '123' + } + } + ``` + +1. Edit the encrypted secret: + + ```shell + sudo gitlab-rake gitlab:ldap:secret:edit EDITOR=vim + ``` + +1. The unencrypted contents of the LDAP secret should be entered like: + + ```yaml + main: + bind_dn: admin + password: '123' + ``` + +1. Edit `/etc/gitlab/gitlab.rb` and remove the settings for `user_bn` and `password`. + +1. [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +**Source configuration** + +If initially your LDAP configuration looked like: + +1. In `config/gitlab.yaml`: + + ```yaml + production: + ldap: + servers: + main: + # snip... + bind_dn: admin + password: '123' + ``` + +1. Edit the encrypted secret: + + ```shell + bundle exec rake gitlab:ldap:secret:edit EDITOR=vim RAILS_ENVIRONMENT=production + ``` + +1. The unencrypted contents of the LDAP secret should be entered like: + + ```yaml + main: + bind_dn: admin + password: '123' + ``` + +1. Edit `config/gitlab.yaml` and remove the settings for `user_bn` and `password`. + +1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. + ## Encryption **(CORE ONLY)** ### TLS Server Authentication @@ -383,7 +470,7 @@ be mandatory and clients cannot be authenticated with the TLS protocol. ## Multiple LDAP servers **(STARTER ONLY)** With GitLab Enterprise Edition Starter, you can configure multiple LDAP servers -that your GitLab instance will connect to. +that your GitLab instance connects to. To add another LDAP server: @@ -391,7 +478,7 @@ To add another LDAP server: 1. Edit them to match the additional LDAP server. Be sure to choose a different provider ID made of letters a-z and numbers 0-9. -This ID will be stored in the database so that GitLab can remember which LDAP +This ID is stored in the database so that GitLab can remember which LDAP server a user belongs to.  @@ -437,7 +524,7 @@ The process executes the following access checks: - Ensure the user is still present in LDAP. - If the LDAP server is Active Directory, ensure the user is active (not - blocked/disabled state). This will only be checked if + blocked/disabled state). This is checked only if `active_directory: true` is set in the LDAP configuration. In Active Directory, a user is marked as disabled/blocked if the user @@ -446,7 +533,7 @@ has bit 2 set. For more information, see <https://ctovswild.com/2009/09/03/bitmask-searches-in-ldap/> The user is set to an `ldap_blocked` state in GitLab if the previous conditions -fail. This means the user won't be able to sign in or push/pull code. +fail. This means the user is not able to sign in or push/pull code. The process also updates the following user information: @@ -495,18 +582,18 @@ sync to run once every 12 hours at the top of the hour. ## Group Sync **(STARTER ONLY)** If your LDAP supports the `memberof` property, when the user signs in for the -first time GitLab will trigger a sync for groups the user should be a member of. +first time GitLab triggers a sync for groups the user should be a member of. That way they don't need to wait for the hourly sync to be granted access to their groups and projects. -A group sync process will run every hour on the hour, and `group_base` must be set +A group sync process runs every hour on the hour, and `group_base` must be set in LDAP configuration for LDAP synchronizations based on group CN to work. This allows GitLab group membership to be automatically updated based on LDAP group members. The `group_base` configuration should be a base LDAP 'container', such as an 'organization' or 'organizational unit', that contains LDAP groups that should be available to GitLab. For example, `group_base` could be -`ou=groups,dc=example,dc=com`. In the configuration file it will look like the +`ou=groups,dc=example,dc=com`. In the configuration file it looks like the following. **Omnibus configuration** @@ -539,7 +626,7 @@ following. 1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. -To take advantage of group sync, group owners or maintainers will need to [create one +To take advantage of group sync, group owners or maintainers need to [create one or more LDAP group links](#adding-group-links). ### Adding group links **(STARTER ONLY)** @@ -550,11 +637,11 @@ For information on adding group links via CNs and filters, refer to [the GitLab As an extension of group sync, you can automatically manage your global GitLab administrators. Specify a group CN for `admin_group` and all members of the -LDAP group will be given administrator privileges. The configuration will look +LDAP group will be given administrator privileges. The configuration looks like the following. -NOTE: **Note:** -Administrators will not be synced unless `group_base` is also +NOTE: +Administrators are not synced unless `group_base` is also specified alongside `admin_group`. Also, only specify the CN of the admin group, as opposed to the full DN. @@ -615,7 +702,7 @@ By default, GitLab runs a group sync process every hour, on the hour. The values shown are in cron format. If needed, you can use a [Crontab Generator](http://www.crontabgenerator.com). -CAUTION: **Important:** +WARNING: Do not start the sync process too frequently as this could lead to multiple syncs running concurrently. This is primarily a concern for installations with a large number of LDAP users. Please review the @@ -691,10 +778,10 @@ There is a lot going on with group sync 'under the hood'. This section outlines what LDAP queries are executed and what behavior you can expect from group sync. -Group member access will be downgraded from a higher level if their LDAP group +Group member access are downgraded from a higher level if their LDAP group membership changes. For example, if a user has 'Owner' rights in a group and the next group sync reveals they should only have 'Developer' privileges, their -access will be adjusted accordingly. The only exception is if the user is the +access is adjusted accordingly. The only exception is if the user is the *last* owner in a group. Groups need at least one owner to fulfill administrative duties. @@ -716,13 +803,13 @@ are defined as one of the mentioned attributes. This also means GitLab supports Microsoft Active Directory, Apple Open Directory, Open LDAP, and 389 Server. Other LDAP servers should work, too. -Active Directory also supports nested groups. Group sync will recursively -resolve membership if `active_directory: true` is set in the configuration file. +Active Directory also supports nested groups. Group sync recursively +resolves membership if `active_directory: true` is set in the configuration file. ##### Nested group memberships Nested group memberships are resolved only if the nested group -is found within the configured `group_base`. For example, if GitLab sees a +is found in the configured `group_base`. For example, if GitLab sees a nested group with DN `cn=nested_group,ou=special_groups,dc=example,dc=com` but the configured `group_base` is `ou=groups,dc=example,dc=com`, `cn=nested_group` is ignored. @@ -731,7 +818,7 @@ is ignored. - Each LDAP group is queried a maximum of one time with base `group_base` and filter `(cn=<cn_from_group_link>)`. -- If the LDAP group has the `memberuid` attribute, GitLab will execute another +- If the LDAP group has the `memberuid` attribute, GitLab executes another LDAP query per member to obtain each user's full DN. These queries are executed with base `base`, scope 'base object', and a filter depending on whether `user_filter` is set. Filter may be `(uid=<uid_from_group>)` or a @@ -750,9 +837,9 @@ LDAP group links each: - Subsequent syncs (checking membership, no writes) took 15 minutes These metrics are meant to provide a baseline and performance may vary based on -any number of factors. This was a pretty extreme benchmark and most instances will -not have near this many users or groups. Disk speed, database performance, -network and LDAP server response time will affect these metrics. +any number of factors. This was an extreme benchmark and most instances don't +have near this many users or groups. Disk speed, database performance, +network and LDAP server response time affects these metrics. ## Troubleshooting diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md index 8920479949d..1976bab03c6 100644 --- a/doc/administration/auth/ldap/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap/ldap-troubleshooting.md @@ -2,7 +2,7 @@ type: reference stage: Manage group: Access -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # LDAP Troubleshooting for Administrators @@ -378,7 +378,7 @@ GitLab syncs the `admin_group`. #### Sync all groups **(STARTER ONLY)** -NOTE: **Note:** +NOTE: To sync all groups manually when debugging is unnecessary, [use the Rake task](../../raketasks/ldap.md#run-a-group-sync) instead. @@ -429,7 +429,7 @@ and more DNs may be added, or existing entries modified, based on additional LDAP group lookups. The very last occurrence of this entry should indicate exactly which users GitLab believes should be added to the group. -NOTE: **Note:** +NOTE: 10 is 'Guest', 20 is 'Reporter', 30 is 'Developer', 40 is 'Maintainer' and 50 is 'Owner'. @@ -673,7 +673,7 @@ adfind -h ad.example.org:636 -ssl -u "CN=GitLabSRV,CN=Users,DC=GitLab,DC=org" -u ### Rails console -CAUTION: **Caution:** +WARNING: It is very easy to create, read, modify, and destroy data with the rails console. Be sure to run commands exactly as listed. diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md index 794733f860c..158182edfb5 100644 --- a/doc/administration/auth/oidc.md +++ b/doc/administration/auth/oidc.md @@ -2,7 +2,7 @@ type: reference stage: Manage group: Access -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # OpenID Connect OmniAuth provider @@ -81,7 +81,7 @@ The OpenID Connect will provide you with a client details and secret for you to } ``` - NOTE: **Note:** + NOTE: For more information on each configuration option refer to the [OmniAuth OpenID Connect usage documentation](https://github.com/m0n9oose/omniauth_openid_connect#usage) and the [OpenID Connect Core 1.0 specification](https://openid.net/specs/openid-connect-core-1_0.html). diff --git a/doc/administration/auth/okta.md b/doc/administration/auth/okta.md index 06b50be2647..50dc3b58680 100644 --- a/doc/administration/auth/okta.md +++ b/doc/administration/auth/okta.md @@ -2,7 +2,7 @@ type: reference stage: Manage group: Access -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Okta SSO provider @@ -158,7 +158,7 @@ You might want to try this out on an incognito browser window. ## Configuring groups -NOTE: **Note:** +NOTE: Make sure the groups exist and are assigned to the Okta app. You can take a look of the [SAML documentation](../../integration/saml.md#saml-groups) on configuring groups. diff --git a/doc/administration/auth/smartcard.md b/doc/administration/auth/smartcard.md index d2937d36a35..9790802e413 100644 --- a/doc/administration/auth/smartcard.md +++ b/doc/administration/auth/smartcard.md @@ -1,7 +1,7 @@ --- stage: Manage group: Access -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -30,7 +30,7 @@ GitLab supports two authentication methods: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/726) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.6 as an experimental feature. -CAUTION: **Caution:** +WARNING: Smartcard authentication against local databases may change or be removed completely in future releases. @@ -60,7 +60,7 @@ Certificate: Smartcards with X.509 certificates using SAN extensions can be used to authenticate with GitLab. -NOTE: **Note:** +NOTE: This is an experimental feature. Smartcard authentication against local databases may change or be removed completely in future releases. @@ -211,7 +211,7 @@ attribute. As a prerequisite, you must use an LDAP server that: client_certificate_required_port: 3443 ``` - NOTE: **Note:** + NOTE: Assign a value to at least one of the following variables: `client_certificate_required_host` or `client_certificate_required_port`. diff --git a/doc/administration/availability/index.md b/doc/administration/availability/index.md index 748373c8941..4f934646ed6 100644 --- a/doc/administration/availability/index.md +++ b/doc/administration/availability/index.md @@ -3,3 +3,6 @@ redirect_to: ../reference_architectures/index.md --- This document was moved to [another location](../reference_architectures/index.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/build_artifacts.md b/doc/administration/build_artifacts.md index 85ae2946bc8..e42f50b2e54 100644 --- a/doc/administration/build_artifacts.md +++ b/doc/administration/build_artifacts.md @@ -3,3 +3,6 @@ redirect_to: 'job_artifacts.md' --- This document was moved to [job_artifacts](job_artifacts.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/compliance.md b/doc/administration/compliance.md index 4cb2c002015..85b72d7b18f 100644 --- a/doc/administration/compliance.md +++ b/doc/administration/compliance.md @@ -1,26 +1,29 @@ --- stage: Manage group: Compliance -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Compliance features -You can configure the following GitLab features to help ensure that your GitLab instance meets common compliance standards. Click a feature name for further documentation. +You can configure the following GitLab features to help ensure that your GitLab +instance meets common compliance standards. Click a feature name for additional +documentation. -GitLabâs [security features](../security/README.md) may also help you meet relevant compliance standards. +The [security features](../security/README.md) in GitLab may also help you meet +relevant compliance standards. |Feature |GitLab tier |GitLab.com | | ---------| :--------: | :-------: | |**[Restrict SSH Keys](../security/ssh_keys_restrictions.md)**<br>Control the technology and key length of SSH keys used to access GitLab|Core+|| |**[Granular user roles and flexible permissions](../user/permissions.md)**<br>Manage access and permissions with five different user roles and settings for external users. Set permissions according to people's role, rather than either read or write access to a repository. Don't share the source code with people that only need access to the issue tracker.|Core+|â| |**[Enforce TOS acceptance](../user/admin_area/settings/terms.md)**<br>Enforce your users accepting new terms of service by blocking GitLab traffic.|Core+|| -|**[Email all users of a project, group, or entire server](../user/admin_area/settings/terms.md)**<br>An admin can email groups of users based on project or group membership, or email everyone using the GitLab instance. This is great for scheduled maintenance or upgrades.|Starter+|| +|**[Email all users of a project, group, or entire server](../tools/email.md)**<br>An admin can email groups of users based on project or group membership, or email everyone using the GitLab instance. This is great for scheduled maintenance or upgrades.|Starter+|| |**[Omnibus package supports log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-forwarding)**<br>Forward your logs to a central system.|Starter+|| |**[Lock project membership to group](../user/group/index.md#member-lock)**<br>Group owners can prevent new members from being added to projects within a group.|Starter+|â| |**[LDAP group sync](auth/ldap/index.md#group-sync)**<br>GitLab Enterprise Edition gives admins the ability to automatically sync groups and manage SSH keys, permissions, and authentication, so you can focus on building your product, not configuring your tools.|Starter+|| |**[LDAP group sync filters](auth/ldap/index.md#group-sync)**<br>GitLab Enterprise Edition Premium gives more flexibility to synchronize with LDAP based on filters, meaning you can leverage LDAP attributes to map GitLab permissions.|Premium+|| -|**[Audit logs](audit_events.md)**<br>To maintain the integrity of your code, GitLab Enterprise Edition Premium gives admins the ability to view any modifications made within the GitLab server in an advanced audit log system, so you can control, analyze, and track every change.|Premium+|| +|**[Audit events](audit_events.md)**<br>To maintain the integrity of your code, GitLab Enterprise Edition Premium gives admins the ability to view any modifications made within the GitLab server in an advanced audit events system, so you can control, analyze, and track every change.|Premium+|| |**[Auditor users](auditor_users.md)**<br>Auditor users are users who are given read-only access to all projects, groups, and other resources on the GitLab instance.|Premium+|| |**[Credentials inventory](../user/admin_area/credentials_inventory.md)**<br>With a credentials inventory, GitLab administrators can keep track of the credentials used by all of the users in their GitLab instance. |Ultimate|| -|**Separation of Duties using [Protected branches](../user/project/protected_branches.md#protected-branches-approval-by-code-owners) and [custom CI Configuration Paths](../ci/pipelines/settings.md#custom-ci-configuration-path)**<br> GitLab Silver and Premium users can leverage GitLab's cross-project YAML configuration's to define deployers of code and developers of code. View the [Separation of Duties Deploy Project](https://gitlab.com/guided-explorations/separation-of-duties-deploy/blob/master/README.md) and [Separation of Duties Project](https://gitlab.com/guided-explorations/separation-of-duties/blob/master/README.md) to see how to use this set up to define these roles.|Premium+|| +|**Separation of Duties using [Protected branches](../user/project/protected_branches.md#protected-branches-approval-by-code-owners) and [custom CI Configuration Paths](../ci/pipelines/settings.md#custom-ci-configuration-path)**<br> GitLab Silver and Premium users can leverage the GitLab cross-project YAML configurations to define deployers of code and developers of code. View the [Separation of Duties Deploy Project](https://gitlab.com/guided-explorations/separation-of-duties-deploy/blob/master/README.md) and [Separation of Duties Project](https://gitlab.com/guided-explorations/separation-of-duties/blob/master/README.md) to see how to use this set up to define these roles.|Premium+|| diff --git a/doc/administration/consul.md b/doc/administration/consul.md index 491251bc842..5b577443c7c 100644 --- a/doc/administration/consul.md +++ b/doc/administration/consul.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -77,7 +77,7 @@ Identify any existing health issues in the cluster by running the following comm within each node. The command will return an empty array if the cluster is healthy: ```shell -curl http://127.0.0.1:8500/v1/health/state/critical +curl "http://127.0.0.1:8500/v1/health/state/critical" ``` Consul nodes communicate using the raft protocol. If the current leader goes @@ -133,7 +133,7 @@ you will need to follow the Consul [outage recovery](#outage-recovery) process. To be safe, it's recommended that you only restart Consul in one node at a time to ensure the cluster remains intact. For larger clusters, it is possible to restart multiple nodes at a time. See the -[Consul consensus document](https://www.consul.io/docs/internals/consensus.html#deployment-table) +[Consul consensus document](https://www.consul.io/docs/architecture/consensus#deployment-table) for how many failures it can tolerate. This will be the number of simultaneous restarts it can sustain. diff --git a/doc/administration/container_registry.md b/doc/administration/container_registry.md index 7a533aa9043..52941556237 100644 --- a/doc/administration/container_registry.md +++ b/doc/administration/container_registry.md @@ -3,3 +3,6 @@ redirect_to: 'packages/container_registry.md' --- This document was moved to [another location](packages/container_registry.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/custom_hooks.md b/doc/administration/custom_hooks.md index 4cb8b15e4d8..0580540eef9 100644 --- a/doc/administration/custom_hooks.md +++ b/doc/administration/custom_hooks.md @@ -3,3 +3,6 @@ redirect_to: 'server_hooks.md' --- This document was moved to [another location](server_hooks.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/database_load_balancing.md b/doc/administration/database_load_balancing.md index 2b25ee4bab8..45243afd8ac 100644 --- a/doc/administration/database_load_balancing.md +++ b/doc/administration/database_load_balancing.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Database Load Balancing **(PREMIUM ONLY)** @@ -167,13 +167,13 @@ When the list of hosts is updated, it might take a while for the old connections to be terminated. The `disconnect_timeout` setting can be used to enforce an upper limit on the time it will take to terminate all old database connections. -Some nameservers (like [Consul](https://www.consul.io/docs/agent/dns.html#udp-based-dns-queries)) can return a truncated list of hosts when +Some nameservers (like [Consul](https://www.consul.io/docs/discovery/dns#udp-based-dns-queries)) can return a truncated list of hosts when queried over UDP. To overcome this issue, you can use TCP for querying by setting `use_tcp` to `true`. ### Forking -NOTE: **Note:** +NOTE: Starting with GitLab 13.0, Puma is the default web server used in GitLab all-in-one package based installations as well as GitLab Helm chart deployments. diff --git a/doc/administration/dependency_proxy.md b/doc/administration/dependency_proxy.md index 4683565998a..c0e0643b5de 100644 --- a/doc/administration/dependency_proxy.md +++ b/doc/administration/dependency_proxy.md @@ -3,3 +3,6 @@ redirect_to: 'packages/dependency_proxy.md' --- This document was moved to [another location](packages/dependency_proxy.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/encrypted_configuration.md b/doc/administration/encrypted_configuration.md new file mode 100644 index 00000000000..01a1cf66bdc --- /dev/null +++ b/doc/administration/encrypted_configuration.md @@ -0,0 +1,37 @@ +--- +stage: Enablement +group: Distribution +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers" +type: reference +--- + +# Encrypted Configuration **(CORE ONLY)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/45712) in GitLab 13.7. + +GitLab can read settings for certain features from encrypted settings files. The supported features are: + +- [LDAP `user_bn` and `password`](auth/ldap/index.md#using-encrypted-credentials) + +In order to enable the encrypted configuration settings, a new base key needs to be generated for +`encrypted_settings_key_base`. The secret can be generated in the following ways: + +**Omnibus Installation** + +Starting with 13.7 the new secret is automatically generated for you, but you will need to ensure your +`/etc/gitlab/gitlab-secrets.json` contains the same values on all nodes. + +**GitLab Cloud Native Helm Chart** + +Starting with GitLab 13.7, the new secret is automatically generated if you have the `shared-secrets` chart enabled. Otherwise, you need +to follow the [secrets guide for adding the secret](https://docs.gitlab.com/charts/installation/secrets.html#gitlab-rails-secret). + +**Source Installation** + +The new secret can be generated by running: + +```shell +bundle exec rake gitlab:env:info RAILS_ENV=production GITLAB_GENERATE_ENCRYPTED_SETTINGS_KEY_BASE=true +``` + +This will print general info on the GitLab instance, but will also cause the key to be generated in `<path-to-gitlab-rails>/config/secrets.yml` diff --git a/doc/administration/environment_variables.md b/doc/administration/environment_variables.md index 25bfc3c132d..f886e19b13d 100644 --- a/doc/administration/environment_variables.md +++ b/doc/administration/environment_variables.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- diff --git a/doc/administration/external_pipeline_validation.md b/doc/administration/external_pipeline_validation.md index cb1d35f24d5..64426a60ab0 100644 --- a/doc/administration/external_pipeline_validation.md +++ b/doc/administration/external_pipeline_validation.md @@ -1,7 +1,7 @@ --- stage: Verify group: Continuous Integration -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference, howto --- @@ -9,14 +9,14 @@ type: reference, howto You can use an external service for validating a pipeline before it's created. -CAUTION: **Warning:** +WARNING: This is an experimental feature and subject to change without notice. ## Usage -GitLab will send a POST request to the external service URL with the pipeline -data as payload. GitLab will then invalidate the pipeline based on the response -code. If there's an error or the request times out, the pipeline will not be +GitLab sends a POST request to the external service URL with the pipeline +data as payload. GitLab then invalidates the pipeline based on the response +code. If there's an error or the request times out, the pipeline is not invalidated. Response Code Legend: diff --git a/doc/administration/feature_flags.md b/doc/administration/feature_flags.md index 2a5260a1342..272009c1f3a 100644 --- a/doc/administration/feature_flags.md +++ b/doc/administration/feature_flags.md @@ -13,12 +13,12 @@ to deploy features in an early stage of development so that they can be incrementally rolled out. Before making them permanently available, features can be deployed behind -flags for a [number of reasons](../development/feature_flags/process.md#when-to-use-feature-flags), such as: +flags for a [number of reasons](../development/feature_flags/index.md#when-to-use-feature-flags), such as: - To test the feature. - To get feedback from users and customers while in an early stage of the development of the feature. - To evaluate users adoption. -- To evaluate how it impacts GitLab's performance. +- To evaluate how it impacts the performance of GitLab. - To build it in smaller pieces throughout releases. Features behind flags can be gradually rolled out, typically: @@ -36,7 +36,7 @@ error, it's very important that you [**provide feedback**](https://gitlab.com/gi as possible so we can improve or fix it while behind a flag. When you upgrade GitLab to an earlier version, the feature flag status may change. -CAUTION: **Caution:** +WARNING: Features deployed behind feature flags may not be ready for production use. However, disabling features behind flags that were deployed enabled by default may also present a risk. If they're enabled, we recommend diff --git a/doc/administration/file_hooks.md b/doc/administration/file_hooks.md index a6610188381..5de2e4aec3f 100644 --- a/doc/administration/file_hooks.md +++ b/doc/administration/file_hooks.md @@ -1,7 +1,7 @@ --- stage: Create group: Source Code -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers" +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" type: reference --- @@ -11,14 +11,14 @@ type: reference > - Until GitLab 12.8, the feature name was Plugins. With custom file hooks, GitLab administrators can introduce custom integrations -without modifying GitLab's source code. +without modifying the GitLab source code. -NOTE: **Note:** +NOTE: Instead of writing and supporting your own file hook you can make changes directly to the GitLab source code and contribute back upstream. This way we can ensure functionality is preserved across versions and covered by tests. -NOTE: **Note:** +NOTE: File hooks must be configured on the filesystem of the GitLab server. Only GitLab server administrators will be able to complete these tasks. Explore [system hooks](../system_hooks/system_hooks.md) or [webhooks](../user/project/integrations/webhooks.md) as an option if you do not have filesystem access. diff --git a/doc/administration/geo/disaster_recovery/background_verification.md b/doc/administration/geo/disaster_recovery/background_verification.md index 89a93e45b1d..f2854d0538b 100644 --- a/doc/administration/geo/disaster_recovery/background_verification.md +++ b/doc/administration/geo/disaster_recovery/background_verification.md @@ -1,13 +1,13 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- # Automatic background verification **(PREMIUM ONLY)** -NOTE: **Note:** +NOTE: Automatic background verification of repositories and wikis was added in GitLab EE 10.6 but is enabled by default only on GitLab EE 11.1. You can disable or enable this feature manually by following diff --git a/doc/administration/geo/disaster_recovery/bring_primary_back.md b/doc/administration/geo/disaster_recovery/bring_primary_back.md index b73d3ba52a2..3f86e03bd7f 100644 --- a/doc/administration/geo/disaster_recovery/bring_primary_back.md +++ b/doc/administration/geo/disaster_recovery/bring_primary_back.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -13,7 +13,7 @@ restore your original configuration. This process consists of two steps: 1. Making the old **primary** node a **secondary** node. 1. Promoting a **secondary** node to a **primary** node. -CAUTION: **Caution:** +WARNING: If you have any doubts about the consistency of the data on this node, we recommend setting it up from scratch. ## Configure the former **primary** node to be a **secondary** node @@ -32,14 +32,14 @@ To bring the former **primary** node up to date: sudo gitlab-ctl start ``` - NOTE: **Note:** + NOTE: If you [disabled the **primary** node permanently](index.md#step-2-permanently-disable-the-primary-node), you need to undo those steps now. For Debian/Ubuntu you just need to run `sudo systemctl enable gitlab-runsvdir`. For CentOS 6, you need to install the GitLab instance from scratch and set it up as a **secondary** node by following [Setup instructions](../setup/index.md). In this case, you don't need to follow the next step. - NOTE: **Note:** + NOTE: If you [changed the DNS records](index.md#step-4-optional-updating-the-primary-domain-dns-record) for this node during disaster recovery procedure you may need to [block all the writes to this node](planned_failover.md#prevent-updates-to-the-primary-node) diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index 1e9b430834c..be84b260127 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -13,7 +13,7 @@ failover with minimal effort, in a disaster situation. See [Geo limitations](../index.md#limitations) for more information. -CAUTION: **Warning:** +WARNING: Disaster recovery for multi-secondary configurations is in **Alpha**. For the latest updates, check the [Disaster Recovery epic for complete maturity](https://gitlab.com/groups/gitlab-org/-/epics/590). Multi-secondary configurations require the complete re-synchronization and re-configuration of all non-promoted secondaries and @@ -36,7 +36,7 @@ order to avoid unnecessary data loss. ### Step 2. Permanently disable the **primary** node -CAUTION: **Warning:** +WARNING: If the **primary** node goes offline, there may be data saved on the **primary** node that has not been replicated to the **secondary** node. This data should be treated as lost if you proceed. @@ -58,7 +58,7 @@ must disable the **primary** node. sudo systemctl disable gitlab-runsvdir ``` - NOTE: **Note:** + NOTE: (**CentOS only**) In CentOS 6 or older, there is no easy way to prevent GitLab from being started if the machine reboots isn't available (see [Omnibus GitLab issue #3058](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3058)). It may be safest to uninstall the GitLab package completely: @@ -67,7 +67,7 @@ must disable the **primary** node. yum remove gitlab-ee ``` - NOTE: **Note:** + NOTE: (**Ubuntu 14.04 LTS**) If you are using an older version of Ubuntu or any other distribution based on the Upstart init system, you can prevent GitLab from starting if the machine reboots by doing the following: @@ -133,13 +133,14 @@ Note the following when promoting a secondary: ``` 1. Promote the **secondary** node to the **primary** node. - DANGER: **Warning:** + + WARNING: In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the secondary is paused fails. Do not pause replication before promoting a secondary. If the node is paused, be sure to resume before promoting. This issue has been fixed in GitLab 13.4 and later. - CAUTION: **Caution:** + WARNING: If the secondary node [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs a point-in-time recovery to the last known state. Data that was created on the primary while the secondary was paused will be lost. @@ -173,13 +174,13 @@ conjunction with multiple servers, as it can only perform changes on a **secondary** with only a single machine. Instead, you must do this manually. -DANGER: **Warning:** +WARNING: In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the secondary is paused fails. Do not pause replication before promoting a secondary. If the node is paused, be sure to resume before promoting. This issue has been fixed in GitLab 13.4 and later. -CAUTION: **Caution:** +WARNING: If the secondary node [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs a point-in-time recovery to the last known state. Data that was created on the primary while the secondary was paused will be lost. @@ -300,7 +301,7 @@ secondary domain, like changing Git remotes and API URLs. external_url 'https://<new_external_url>' ``` - NOTE: **Note:** + NOTE: Changing `external_url` won't prevent access via the old secondary URL, as long as the secondary DNS records are still intact. diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index 1238c4d8e2a..1468c5cd39d 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- diff --git a/doc/administration/geo/disaster_recovery/promotion_runbook.md b/doc/administration/geo/disaster_recovery/promotion_runbook.md index 7eb6ef01aee..eef1e7ae9dd 100644 --- a/doc/administration/geo/disaster_recovery/promotion_runbook.md +++ b/doc/administration/geo/disaster_recovery/promotion_runbook.md @@ -3,3 +3,6 @@ redirect_to: runbooks/planned_failover_single_node.md --- This document was moved to [another location](runbooks/planned_failover_single_node.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md index 3864445bbf4..f17990ce5f8 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md @@ -1,11 +1,11 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- -CAUTION: **Caution:** +WARNING: This runbook is in **alpha**. For complete, production-ready documentation, see the [disaster recovery documentation](../index.md). @@ -58,7 +58,7 @@ What is not covered: ### Preparation -NOTE: **Note:** +NOTE: Before following any of those steps, make sure you have `root` access to the **secondary** to promote it, since there isn't provided an automated way to promote a Geo replica and perform a failover. @@ -134,7 +134,7 @@ follow these steps to avoid unnecessary data loss: 1. Finish replicating and verifying all data: - CAUTION: **Caution:** + WARNING: Not all data is automatically replicated. Read more about [what is excluded](../planned_failover.md#not-all-data-is-automatically-replicated). @@ -163,12 +163,12 @@ follow these steps to avoid unnecessary data loss: 1. In this final step, you need to permanently disable the **primary** node. - CAUTION: **Caution:** + WARNING: When the **primary** node goes offline, there may be data saved on the **primary** node that has not been replicated to the **secondary** node. This data should be treated as lost if you proceed. - TIP: **Tip:** + NOTE: If you plan to [update the **primary** domain DNS record](../index.md#step-4-optional-updating-the-primary-domain-dns-record), you may wish to lower the TTL now to speed up propagation. @@ -188,12 +188,12 @@ follow these steps to avoid unnecessary data loss: sudo systemctl disable gitlab-runsvdir ``` - NOTE: **Note:** + NOTE: (**CentOS only**) In CentOS 6 or older, there is no easy way to prevent GitLab from being started if the machine reboots isn't available (see [Omnibus GitLab issue #3058](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3058)). It may be safest to uninstall the GitLab package completely with `sudo yum remove gitlab-ee`. - NOTE: **Note:** + NOTE: (**Ubuntu 14.04 LTS**) If you are using an older version of Ubuntu or any other distribution based on the Upstart init system, you can prevent GitLab from starting if the machine reboots as `root` with @@ -213,12 +213,12 @@ follow these steps to avoid unnecessary data loss: ### Promoting the **secondary** node -NOTE: **Note:** +NOTE: A new **secondary** should not be added at this time. If you want to add a new **secondary**, do this after you have completed the entire process of promoting the **secondary** to the **primary**. -CAUTION: **Caution:** +WARNING: If you encounter an `ActiveRecord::RecordInvalid: Validation failed: Name has already been taken` error during this process, read [the troubleshooting advice](../../replication/troubleshooting.md#fixing-errors-during-a-failover-or-when-promoting-a-secondary-to-a-primary-node). @@ -227,13 +227,13 @@ conjunction with multiple servers, as it can only perform changes on a **secondary** with only a single machine. Instead, you must do this manually. -DANGER: **Warning:** +WARNING: In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the secondary is paused fails. Do not pause replication before promoting a secondary. If the node is paused, be sure to resume before promoting. This issue has been fixed in GitLab 13.4 and later. -CAUTION: **Caution:** +WARNING: If the secondary node [has been paused](../../../geo/index.md#pausing-and-resuming-replication), this performs a point-in-time recovery to the last known state. Data that was created on the primary while the secondary was paused will be lost. diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md index 5e847030077..a2a9350e411 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md @@ -1,11 +1,11 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- -CAUTION: **Caution:** +WARNING: This runbook is in **alpha**. For complete, production-ready documentation, see the [disaster recovery documentation](../index.md). @@ -46,7 +46,7 @@ What is not covered: ### Preparation -NOTE: **Note:** +NOTE: Before following any of those steps, make sure you have `root` access to the **secondary** to promote it, since there isn't provided an automated way to promote a Geo replica and perform a failover. @@ -122,7 +122,7 @@ follow these steps to avoid unnecessary data loss: 1. Finish replicating and verifying all data: - CAUTION: **Caution:** + WARNING: Not all data is automatically replicated. Read more about [what is excluded](../planned_failover.md#not-all-data-is-automatically-replicated). @@ -151,12 +151,12 @@ follow these steps to avoid unnecessary data loss: 1. In this final step, you need to permanently disable the **primary** node. - CAUTION: **Caution:** + WARNING: When the **primary** node goes offline, there may be data saved on the **primary** node that has not been replicated to the **secondary** node. This data should be treated as lost if you proceed. - TIP: **Tip:** + NOTE: If you plan to [update the **primary** domain DNS record](../index.md#step-4-optional-updating-the-primary-domain-dns-record), you may wish to lower the TTL now to speed up propagation. @@ -176,12 +176,12 @@ follow these steps to avoid unnecessary data loss: sudo systemctl disable gitlab-runsvdir ``` - NOTE: **Note:** + NOTE: (**CentOS only**) In CentOS 6 or older, there is no easy way to prevent GitLab from being started if the machine reboots isn't available (see [Omnibus GitLab issue #3058](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3058)). It may be safest to uninstall the GitLab package completely with `sudo yum remove gitlab-ee`. - NOTE: **Note:** + NOTE: (**Ubuntu 14.04 LTS**) If you are using an older version of Ubuntu or any other distribution based on the Upstart init system, you can prevent GitLab from starting if the machine reboots as `root` with diff --git a/doc/administration/geo/glossary.md b/doc/administration/geo/glossary.md new file mode 100644 index 00000000000..e9d57284dd2 --- /dev/null +++ b/doc/administration/geo/glossary.md @@ -0,0 +1,112 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: howto +--- + + +# Geo Glossary + +NOTE: +We are updating the Geo documentation, user interface and commands to reflect these changes. Not all pages comply with +these definitions yet. + + These are the defined terms to describe all aspects of Geo. Using a set of clearly + defined terms helps us to communicate efficiently and avoids confusion. The language + on this page aims to be [ubiquitous](https://about.gitlab.com/handbook/communication/#ubiquitous-language) + and [as simple as possible](https://about.gitlab.com/handbook/communication/#simple-language). + + We provide example diagrams and statements to demonstrate correct usage of terms. + +| Term | Definition | Scope | Discouraged synonyms | +|---------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------|-------------------------------------------------| +| Node | An individual server that runs GitLab either with a specific role or as a whole (e.g. a Rails application node). In a cloud context this can be a specific machine type. | GitLab | instance, server | +| Site | One or a collection of nodes running a single GitLab application. A site can be single-node or multi-node. | GitLab | deployment, installation instance | +| Single-node site | A specific configuration of GitLab that uses exactly one node. | GitLab | single-server, single-instance +| Multi-node site | A specific configuration of GitLab that uses more than one node. | GitLab | multi-server, multi-instance, high availability | +| Primary site | A GitLab site that is configured to be read and writable. There can only be a single primary site. | Geo-specific | Geo deployment, Primary node | +| Secondary site(s) | GitLab site that is configured to be read-only. There can be one or more secondary sites. | Geo-specific | Geo deployment, Secondary node | +| Geo deployment | A collection of two or more GitLab sites with exactly one primary site being replicated by one or more secondary sites. | Geo-specific | | +| Reference architecture(s) | A [specified configuration of GitLab for a number of users](../reference_architectures/index.md), possibly including multiple nodes and multiple sites. | GitLab | | +| Promoting | Changing the role of a site from secondary to primary. | Geo-specific | | +| Demoting | Changing the role of a site from primary to secondary. | Geo-specific | | +| Failover | The entire process that shifts users from a primary Site to a secondary site. This includes promoting a secondary, but contains other parts as well e.g. scheduling maintenance. | Geo-specific | | + +## Examples + +### Single-node site + +```mermaid + graph TD + subgraph S-Site[Single-node site] + Node_3[GitLab node] + end +``` + +### Multi-node site + +```mermaid + graph TD + subgraph MN-Site[Multi-node site] + Node_1[Application node] + Node_2[Database node] + Node_3[Gitaly node] + end +``` + +### Geo deployment - Single-node sites + +This Geo deployment has a single-node primary site, a single-node secondary site: + +```mermaid + graph TD + subgraph Geo deployment + subgraph Primary[Primary site, single-node] + Node_1[GitLab node] + end + subgraph Secondary1[Secondary site 1, single-node] + Node_2[GitLab node] + end + end +``` + +### Geo deployment - Multi-node sites + +This Geo deployment has a multi-node primary site, a multi-node secondary site: + +```mermaid + graph TD + subgraph Geo deployment + subgraph Primary[Primary site, multi-node] + Node_1[Application node] + Node_2[Database node] + end + subgraph Secondary1[Secondary site 1, multi-node] + Node_5[Application node] + Node_6[Database node] + end + end +``` + +### Geo deployment - Mixed sites + +This Geo deployment has a multi-node primary site, a multi-node secondary site and another single-node secondary site: + +```mermaid + graph TD + subgraph Geo deployment + subgraph Primary[Primary site, multi-node] + Node_1[Application node] + Node_2[Database node] + Node_3[Gitaly node] + end + subgraph Secondary1[Secondary site 1, multi-node] + Node_5[Application node] + Node_6[Database node] + end + subgraph Secondary2[Secondary site 2, single-node] + Node_7[Single GitLab node] + end + end +``` diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index 02b907ae237..334f05ef3ce 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -17,7 +17,7 @@ Geo is the solution for widely distributed development teams and for providing a ## Overview -CAUTION: **Caution:** +WARNING: Geo undergoes significant changes from release to release. Upgrades **are** supported and [documented](#updating-geo), but you should ensure that you're using the right version of the documentation for your installation. Fetching large repositories can take a long time for teams located far from a single GitLab instance. @@ -51,6 +51,11 @@ Geo provides: - Authentication system hooks: **Secondary** nodes receives all authentication data (like user accounts and logins) from the **primary** instance. - An intuitive UI: **Secondary** nodes use the same web interface your team has grown accustomed to. In addition, there are visual notifications that block write operations and make it clear that a user is on a **secondary** node. +### Gitaly Cluster + +Geo should not be confused with [Gitaly Cluster](../gitaly/praefect.md). For more information about +the difference between Geo and Gitaly Cluster, see [Gitaly Cluster compared to Geo](../gitaly/praefect.md#gitaly-cluster-compared-to-geo). + ## How it works Your Geo instance can be used for cloning and fetching projects, in addition to reading any data. This will make working with large repositories over large distances much faster. @@ -119,7 +124,7 @@ The following are required to run Geo: - Git-lfs 2.4.2+ on the user side when using LFS - All nodes must run the same GitLab version. -Additionally, check GitLab's [minimum requirements](../../install/requirements.md), +Additionally, check the GitLab [minimum requirements](../../install/requirements.md), and we recommend you use: - At least GitLab Enterprise Edition 10.0 for basic Geo features. @@ -138,11 +143,11 @@ The following table lists basic ports that must be open between the **primary** See the full list of ports used by GitLab in [Package defaults](https://docs.gitlab.com/omnibus/package-information/defaults.html) -NOTE: **Note:** +NOTE: [Web terminal](../../ci/environments/index.md#web-terminals) support requires your load balancer to correctly handle WebSocket connections. When using HTTP or HTTPS proxying, your load balancer must be configured to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the [web terminal](../integration/terminal.md) integration guide for more details. -NOTE: **Note:** +NOTE: When using HTTPS protocol for port 443, you will need to add an SSL certificate to the load balancers. If you wish to terminate SSL at the GitLab application server instead, use TCP protocol. @@ -150,7 +155,7 @@ If you wish to terminate SSL at the GitLab application server instead, use TCP p We recommend that if you use LDAP on your **primary** node, you also set up secondary LDAP servers on each **secondary** node. Otherwise, users will not be able to perform Git operations over HTTP(s) on the **secondary** node using HTTP Basic Authentication. However, Git via SSH and personal access tokens will still work. -NOTE: **Note:** +NOTE: It is possible for all **secondary** nodes to share an LDAP server, but additional latency can be an issue. Also, consider what LDAP server will be available in a [disaster recovery](disaster_recovery/index.md) scenario if a **secondary** node is promoted to be a **primary** node. Check for instructions on how to set up replication in your LDAP service. Instructions will be different depending on the software or service used. For example, OpenLDAP provides [these instructions](https://www.openldap.org/doc/admin24/replication.html). @@ -196,19 +201,21 @@ For information on how to update your Geo nodes to the latest GitLab version, se > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35913) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.2. -DANGER: **Warning:** +WARNING: In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the secondary is paused fails. Do not pause replication before promoting a secondary. If the node is paused, be sure to resume before promoting. This issue has been fixed in GitLab 13.4 and later. -CAUTION: **Caution:** +WARNING: Pausing and resuming of replication is currently only supported for Geo installations using an Omnibus GitLab-managed database. External databases are currently not supported. In some circumstances, like during [upgrades](replication/updating_the_geo_nodes.md) or a [planned failover](disaster_recovery/planned_failover.md), it is desirable to pause replication between the primary and secondary. -Pausing and resuming replication is done via a command line tool from the secondary node. +Pausing and resuming replication is done via a command line tool from the secondary node where the `postgresql` service is enabled. + +If `postgresql` is on a standalone database node, ensure that `gitlab.rb` on that node contains the configuration line `gitlab_rails['geo_node_name'] = 'node_name'`, where `node_name` is the same as the `geo_name_name` on the application node. **To Pause: (from secondary)** @@ -250,6 +257,16 @@ For more information on tuning Geo, see [Tuning Geo](replication/tuning.md). For an example of how to set up a location-aware Git remote URL with AWS Route53, see [Location-aware Git remote URL with AWS Route53](replication/location_aware_git_url.md). +### Backfill + +Once a **secondary** node is set up, it will start replicating missing data from +the **primary** node in a process known as **backfill**. You can monitor the +synchronization process on each Geo node from the **primary** node's **Geo Nodes** +dashboard in your browser. + +Failures that happen during a backfill are scheduled to be retried at the end +of the backfill. + ## Remove Geo node For more information on removing a Geo node, see [Removing **secondary** Geo nodes](replication/remove_geo_node.md). @@ -260,7 +277,7 @@ To find out how to disable Geo, see [Disabling Geo](replication/disable_geo.md). ## Limitations -CAUTION: **Caution:** +WARNING: This list of limitations only reflects the latest version of GitLab. If you are using an older version, extra limitations may be in place. - Pushing directly to a **secondary** node redirects (for HTTP) or proxies (for SSH) the request to the **primary** node instead of [handling it directly](https://gitlab.com/gitlab-org/gitlab/-/issues/1381), except when using Git over HTTP with credentials embedded within the URI. For example, `https://user:password@secondary.tld`. @@ -270,7 +287,6 @@ This list of limitations only reflects the latest version of GitLab. If you are - Real-time updates of issues/merge requests (for example, via long polling) doesn't work on the **secondary** node. - [Selective synchronization](replication/configuration.md#selective-synchronization) applies only to files and repositories. Other datasets are replicated to the **secondary** node in full, making it inappropriate for use as an access control mechanism. - Object pools for forked project deduplication work only on the **primary** node, and are duplicated on the **secondary** node. -- [External merge request diffs](../merge_request_diffs.md) will not be replicated if they are on-disk, and viewing merge requests will fail. However, external MR diffs in object storage **are** supported. The default configuration (in-database) does work. - GitLab Runners cannot register with a **secondary** node. Support for this is [planned for the future](https://gitlab.com/gitlab-org/gitlab/-/issues/3294). - Geo **secondary** nodes can not be configured to [use high-availability configurations of PostgreSQL](https://gitlab.com/groups/gitlab-org/-/epics/2536). diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index 43a422c4bc3..42501e16f5f 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -9,7 +9,7 @@ type: howto ## Configuring a new **secondary** node -NOTE: **Note:** +NOTE: This is the final step in setting up a **secondary** Geo node. Stages of the setup process must be completed in the documented order. Before attempting the steps in this stage, [complete all prior stages](../setup/index.md#using-omnibus-gitlab). @@ -23,7 +23,7 @@ The basic steps of configuring a **secondary** node are to: You are encouraged to first read through all the steps before executing them in your testing/production environment. -NOTE: **Note:** +NOTE: **Do not** set up any custom authentication for the **secondary** nodes. This will be handled by the **primary** node. Any change that requires access to the **Admin Area** needs to be done in the **primary** node because the **secondary** node is a read-only replica. @@ -157,7 +157,7 @@ keys must be manually replicated to the **secondary** node. for file in /etc/ssh/ssh_host_*_key.pub; do ssh-keygen -lf $file; done ``` - NOTE: **Note:** + NOTE: The output for private keys and public keys command should generate the same fingerprint. 1. Restart `sshd` on your **secondary** node: diff --git a/doc/administration/geo/replication/database.md b/doc/administration/geo/replication/database.md index da5376190b9..099a73e93d8 100644 --- a/doc/administration/geo/replication/database.md +++ b/doc/administration/geo/replication/database.md @@ -3,3 +3,6 @@ redirect_to: '../setup/database.md' --- This document was moved to [another location](../setup/index.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index 9e896f2c07c..0d2922c3347 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -28,13 +28,13 @@ verification methods: | Database | Application data in PostgreSQL | Native | Native | | Database | Redis | _N/A_ (*1*) | _N/A_ | | Database | Elasticsearch | Native | Native | -| Database | Personal snippets | PostgreSQL Replication | PostgreSQL Replication | -| Database | Project snippets | PostgreSQL Replication | PostgreSQL Replication | | Database | SSH public keys | PostgreSQL Replication | PostgreSQL Replication | | Git | Project repository | Geo with Gitaly | Gitaly Checksum | | Git | Project wiki repository | Geo with Gitaly | Gitaly Checksum | | Git | Project designs repository | Geo with Gitaly | Gitaly Checksum | | Git | Object pools for forked project deduplication | Geo with Gitaly | _Not implemented_ | +| Git | Project Snippets | Geo with Gitaly | _Not implemented_ | +| Git | Personal Snippets | Geo with Gitaly | _Not implemented_ | | Blobs | User uploads _(filesystem)_ | Geo with API | _Not implemented_ | | Blobs | User uploads _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | LFS objects _(filesystem)_ | Geo with API | _Not implemented_ | @@ -81,6 +81,9 @@ Each project can have at most 3 different repositories: They all live in the same shard and share the same base name with a `-wiki` and `-design` suffix for Wiki and Design Repository cases. +Besides that, there are snippet repositories. They can be connected to a project or to some specific user. +Both types will be synced to a secondary node. + ### Blobs GitLab stores files and blobs such as Issue attachments or LFS objects into either: @@ -160,7 +163,7 @@ To enable, such as for package file replication: Feature.enable(:geo_package_file_replication) ``` -DANGER: **Warning:** +WARNING: Features not on this list, or with **No** in the **Replicated** column, are not replicated on the **secondary** node. Failing over without manually replicating data from those features will cause the data to be **lost**. @@ -192,7 +195,7 @@ successfully, you must replicate their data using some other means. | [Generic packages](../../../user/packages/generic_packages/index.md) | **Yes** (13.5) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | | [Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_terraform_state_version_replication`, enabled by default | | [External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_merge_request_diff_replication`, enabled by default | -| [Versioned snippets](../../../user/snippets.md#versioned-snippets) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2810) | No | | +| [Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2810) | No | | | [Server-side Git hooks](../../server_hooks.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1867) | No | No | | | [Elasticsearch integration](../../../integration/elasticsearch.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | No | | | [GitLab Pages](../../pages/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | No | | diff --git a/doc/administration/geo/replication/disable_geo.md b/doc/administration/geo/replication/disable_geo.md index dabf4499f9d..e8767a56266 100644 --- a/doc/administration/geo/replication/disable_geo.md +++ b/doc/administration/geo/replication/disable_geo.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- diff --git a/doc/administration/geo/replication/docker_registry.md b/doc/administration/geo/replication/docker_registry.md index 5403edc69da..f92a878662e 100644 --- a/doc/administration/geo/replication/docker_registry.md +++ b/doc/administration/geo/replication/docker_registry.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -17,7 +17,7 @@ distributed storage (`azure`, `gcs`, `s3`, `swift`, or `oss`) for your Docker Registry on the **primary** node, you can use the same storage for a **secondary** Docker Registry as well. For more information, read the [Load balancing considerations](https://docs.docker.com/registry/deploying/#load-balancing-considerations) -when deploying the Registry, and how to set up the storage driver for GitLab's +when deploying the Registry, and how to set up the storage driver for the GitLab integrated [Container Registry](../../packages/container_registry.md#use-object-storage). ## Replicating Docker Registry @@ -64,17 +64,17 @@ We need to make Docker Registry send notification events to the ] ``` - NOTE: **Note:** + NOTE: Replace `<replace_with_a_secret_token>` with a case sensitive alphanumeric string that starts with a letter. You can generate one with `< /dev/urandom tr -dc _A-Z-a-z-0-9 | head -c 32 | sed "s/^[0-9]*//"; echo` - NOTE: **Note:** + NOTE: If you use an external Registry (not the one integrated with GitLab), you must add these settings to its configuration yourself. In this case, you will also have to specify notification secret in `registry.notification_secret` section of `/etc/gitlab/gitlab.rb` file. - NOTE: **Note:** + NOTE: If you use GitLab HA, you will also have to specify the notification secret in `registry.notification_secret` section of `/etc/gitlab/gitlab.rb` file for every web node. diff --git a/doc/administration/geo/replication/external_database.md b/doc/administration/geo/replication/external_database.md index 0d0cd8a37d5..dfaf6819fa0 100644 --- a/doc/administration/geo/replication/external_database.md +++ b/doc/administration/geo/replication/external_database.md @@ -3,3 +3,6 @@ redirect_to: '../setup/external_database.md' --- This document was moved to [another location](../setup/external_database.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/geo/replication/faq.md b/doc/administration/geo/replication/faq.md index f7f391b360e..ab8199c3717 100644 --- a/doc/administration/geo/replication/faq.md +++ b/doc/administration/geo/replication/faq.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- diff --git a/doc/administration/geo/replication/geo_validation_tests.md b/doc/administration/geo/replication/geo_validation_tests.md index c28688930b5..2d701458e66 100644 --- a/doc/administration/geo/replication/geo_validation_tests.md +++ b/doc/administration/geo/replication/geo_validation_tests.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- diff --git a/doc/administration/geo/replication/high_availability.md b/doc/administration/geo/replication/high_availability.md index 214f15b7565..1da4246db1f 100644 --- a/doc/administration/geo/replication/high_availability.md +++ b/doc/administration/geo/replication/high_availability.md @@ -3,3 +3,6 @@ redirect_to: 'multiple_servers.md' --- This document was moved to [another location](multiple_servers.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/geo/replication/img/geo_node_dashboard.png b/doc/administration/geo/replication/img/geo_node_dashboard.png Binary files differindex 53ca4767c5b..8b9aceba825 100644 --- a/doc/administration/geo/replication/img/geo_node_dashboard.png +++ b/doc/administration/geo/replication/img/geo_node_dashboard.png diff --git a/doc/administration/geo/replication/img/geo_node_healthcheck.png b/doc/administration/geo/replication/img/geo_node_healthcheck.png Binary files differdeleted file mode 100644 index 33a31f7ab49..00000000000 --- a/doc/administration/geo/replication/img/geo_node_healthcheck.png +++ /dev/null diff --git a/doc/administration/geo/replication/index.md b/doc/administration/geo/replication/index.md index 5e9dd73ecc5..955e05491d2 100644 --- a/doc/administration/geo/replication/index.md +++ b/doc/administration/geo/replication/index.md @@ -3,3 +3,6 @@ redirect_to: '../index.md' --- This document was moved to [another location](../index.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/geo/replication/location_aware_git_url.md b/doc/administration/geo/replication/location_aware_git_url.md index 82fda8b0fc2..f2cf8c9bda0 100644 --- a/doc/administration/geo/replication/location_aware_git_url.md +++ b/doc/administration/geo/replication/location_aware_git_url.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -18,7 +18,7 @@ Though these instructions use [AWS Route53](https://aws.amazon.com/route53/), other services such as [Cloudflare](https://www.cloudflare.com/) could be used as well. -NOTE: **Note:** +NOTE: You can also use a load balancer to distribute web UI or API traffic to [multiple Geo **secondary** nodes](../../../user/admin_area/geo_nodes.md#multiple-secondary-nodes-behind-a-load-balancer). Importantly, the **primary** node cannot yet be included. See the feature request diff --git a/doc/administration/geo/replication/multiple_servers.md b/doc/administration/geo/replication/multiple_servers.md index 7d65d2165c5..0ab972c9077 100644 --- a/doc/administration/geo/replication/multiple_servers.md +++ b/doc/administration/geo/replication/multiple_servers.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -27,7 +27,7 @@ network topology of your deployment. The only external way to access the two Geo deployments is by HTTPS at `gitlab.us.example.com` and `gitlab.eu.example.com` in the example above. -NOTE: **Note:** +NOTE: The **primary** and **secondary** Geo deployments must be able to communicate to each other over HTTPS. ## Redis and PostgreSQL for multiple nodes @@ -37,7 +37,7 @@ Geo supports: - Redis and PostgreSQL on the **primary** node configured for multiple nodes. - Redis on **secondary** nodes configured for multiple nodes. -NOTE: **Note:** +NOTE: Support for PostgreSQL on **secondary** nodes in multi-node configuration [is planned](https://gitlab.com/groups/gitlab-org/-/epics/2536). @@ -48,7 +48,7 @@ For more information about setting up a multi-node PostgreSQL cluster and Redis [PostgreSQL](../../postgresql/replication_and_failover.md) and [Redis](../../redis/replication_and_failover.md), respectively. -NOTE: **Note:** +NOTE: It is possible to use cloud hosted services for PostgreSQL and Redis, but this is beyond the scope of this document. ## Prerequisites: Two working GitLab multi-node clusters @@ -90,7 +90,7 @@ The following steps enable a GitLab cluster to serve as the **primary** node. After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. -NOTE: **Note:** +NOTE: PostgreSQL and Redis should have already been disabled on the application servers, and connections from the application servers to those services on the backend servers configured, during normal GitLab multi-node set up. See @@ -136,13 +136,13 @@ documentation: - [Gitaly](../../gitaly/index.md), which will store data that is synchronized from the **primary** node. -NOTE: **Note:** +NOTE: [NFS](../../nfs.md) can be used in place of Gitaly but is not recommended. ### Step 2: Configure the main read-only replica PostgreSQL database on the **secondary** node -NOTE: **Note:** +NOTE: The following documentation assumes the database will be run on a single node only. Multi-node PostgreSQL on **secondary** nodes is [not currently supported](https://gitlab.com/groups/gitlab-org/-/epics/2536). @@ -174,6 +174,12 @@ the **primary** database. Use the following as a guide. roles ['geo_secondary_role', 'postgres_role'] ## + ## The unique identifier for the Geo node. + ## This should match the secondary's application node. + ## + gitlab_rails['geo_node_name'] = '<node_name_here>' + + ## ## Secondary address ## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node ## - replace '<tracking_database_ip>' with the public or VPC address of your Geo tracking database node @@ -226,7 +232,7 @@ If using an external PostgreSQL instance, refer also to ### Step 3: Configure the tracking database on the **secondary** node -NOTE: **Note:** +NOTE: This documentation assumes the tracking database will be run on only a single machine, rather than as a PostgreSQL cluster. @@ -359,14 +365,14 @@ then make the following modifications: registry['gid'] = 9002 ``` -NOTE: **Note:** +NOTE: If you had set up PostgreSQL cluster using the omnibus package and you had set up `postgresql['sql_user_password'] = 'md5 digest of secret'` setting, keep in mind that `gitlab_rails['db_password']` and `geo_secondary['db_password']` mentioned above contains the plaintext passwords. This is used to let the Rails servers connect to the databases. -NOTE: **Note:** +NOTE: Make sure that current node IP is listed in `postgresql['md5_auth_cidr_addresses']` setting of your remote database. After making these changes [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. diff --git a/doc/administration/geo/replication/object_storage.md b/doc/administration/geo/replication/object_storage.md index 3f5ba1939b8..ce791d66b34 100644 --- a/doc/administration/geo/replication/object_storage.md +++ b/doc/administration/geo/replication/object_storage.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -25,7 +25,7 @@ To have: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10586) in GitLab 12.4. -CAUTION: **Caution:** +WARNING: This is a [**beta** feature](https://about.gitlab.com/handbook/product/#beta) and is not ready yet for production use at any scale. The main limitations are a lack of testing at scale and no verification of any replicated data. **Secondary** nodes can replicate files stored on the **primary** node regardless of @@ -35,8 +35,8 @@ To enable GitLab replication, you must: 1. Go to **Admin Area > Geo**. 1. Press **Edit** on the **secondary** node. -1. Enable the **Allow this secondary node to replicate content on Object Storage** - checkbox. +1. In the **Synchronization Settings** section, find the **Allow this secondary node to replicate content on Object Storage** + checkbox to enable it. For LFS, follow the documentation to [set up LFS object storage](../../lfs/index.md#storing-lfs-objects-in-remote-object-storage). diff --git a/doc/administration/geo/replication/remove_geo_node.md b/doc/administration/geo/replication/remove_geo_node.md index 2b01231241c..519b93b447b 100644 --- a/doc/administration/geo/replication/remove_geo_node.md +++ b/doc/administration/geo/replication/remove_geo_node.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -42,7 +42,7 @@ Once GitLab has been uninstalled from the **secondary** node, the replication sl sudo gitlab-psql ``` - NOTE: **Note:** + NOTE: Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. 1. Find the name of the relevant replication slot. This is the slot that is specified with `--slot-name` when running the replicate command: `gitlab-ctl replicate-geo-database`. diff --git a/doc/administration/geo/replication/security_review.md b/doc/administration/geo/replication/security_review.md index 28b8c8a9d51..7c15662340c 100644 --- a/doc/administration/geo/replication/security_review.md +++ b/doc/administration/geo/replication/security_review.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -33,7 +33,7 @@ from [owasp.org](https://owasp.org/). ### How can the data be classified into categories according to its sensitivity? -- GitLabâs model of sensitivity is centered around public vs. internal vs. +- The GitLab model of sensitivity is centered around public vs. internal vs. private projects. Geo replicates them all indiscriminately. âSelective syncâ exists for files and repositories (but not database content), which would permit only less-sensitive projects to be replicated to a **secondary** node if desired. diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 15e3d3ff0a8..2c0160ed02a 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -35,7 +35,7 @@ to help identify if something is wrong: - Is the node's secondary tracking database connected? - Is the node's secondary tracking database up-to-date? - + For information on how to resolve common errors reported from the UI, see [Fixing Common Errors](#fixing-common-errors). @@ -308,7 +308,8 @@ log data to build up in `pg_xlog`. Removing the unused slots can reduce the amou sudo gitlab-psql ``` - Note: **Note:** Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. + NOTE: + Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. 1. View your replication slots with: @@ -425,6 +426,11 @@ GitLab you are running. GitLab versions 11.11.x or 12.0.x are affected by To resolve the issue, upgrade to GitLab 12.1 or newer. +### Failures during backfill + +During a [backfill](../index.md#backfill), failures are scheduled to be retried at the end +of the backfill queue, therefore these failures only clear up **after** the backfill completes. + ### Resetting Geo **secondary** node replication If you get a **secondary** node in a broken state and want to reset the replication state, @@ -461,13 +467,13 @@ to start again from scratch, there are a few steps that can help you: chown git:git /var/opt/gitlab/git-data/repositories ``` - TIP: **Tip:** + NOTE: You may want to remove the `/var/opt/gitlab/git-data/repositories.old` in the future as soon as you confirmed that you don't need it anymore, to save disk space. 1. _(Optional)_ Rename other data folders and create new ones - CAUTION: **Caution:** + WARNING: You may still have files on the **secondary** node that have been removed from **primary** node but removal have not been reflected. If you skip this step, they will never be removed from this Geo node. diff --git a/doc/administration/geo/replication/tuning.md b/doc/administration/geo/replication/tuning.md index fe0b189863c..183180e5ade 100644 --- a/doc/administration/geo/replication/tuning.md +++ b/doc/administration/geo/replication/tuning.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- diff --git a/doc/administration/geo/replication/updating_the_geo_nodes.md b/doc/administration/geo/replication/updating_the_geo_nodes.md index af59e45250c..ff3a1e8689b 100644 --- a/doc/administration/geo/replication/updating_the_geo_nodes.md +++ b/doc/administration/geo/replication/updating_the_geo_nodes.md @@ -1,13 +1,13 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- # Updating the Geo nodes **(PREMIUM ONLY)** -CAUTION: **Warning:** +WARNING: Read these sections carefully before updating your Geo nodes. Not following version-specific update steps may result in unexpected downtime. If you have any specific questions, [contact Support](https://about.gitlab.com/support/#contact-support). @@ -20,7 +20,7 @@ Updating Geo nodes involves performing: ## General update steps -NOTE: **Note:** +NOTE: These general update steps are not intended for [high-availability deployments](https://docs.gitlab.com/omnibus/update/README.html#multi-node--ha-deployment), and will cause downtime. If you want to avoid downtime, consider using [zero downtime updates](https://docs.gitlab.com/omnibus/update/README.html#zero-downtime-updates). To update the Geo nodes when a new GitLab version is released, update **primary** diff --git a/doc/administration/geo/replication/using_a_geo_server.md b/doc/administration/geo/replication/using_a_geo_server.md index aeed5a44b30..743b1b384db 100644 --- a/doc/administration/geo/replication/using_a_geo_server.md +++ b/doc/administration/geo/replication/using_a_geo_server.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- diff --git a/doc/administration/geo/replication/version_specific_updates.md b/doc/administration/geo/replication/version_specific_updates.md index 9ea5283812e..0d8eba555ab 100644 --- a/doc/administration/geo/replication/version_specific_updates.md +++ b/doc/administration/geo/replication/version_specific_updates.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -11,6 +11,10 @@ Check this document if it includes instructions for the version you are updating These steps go together with the [general steps](updating_the_geo_nodes.md#general-update-steps) for updating Geo nodes. +## Updating to GitLab 13.5 + +In GitLab 13.5, there is a [regression that prevents viewing a list of container repositories and registries](https://gitlab.com/gitlab-org/gitlab/-/issues/285475) on Geo secondaries. This issue is fixed in GitLab 13.6.1 and later. + ## Updating to GitLab 13.3 In GitLab 13.3, Geo removed the PostgreSQL [Foreign Data Wrapper](https://www.postgresql.org/docs/11/postgres-fdw.html) dependency for the tracking database. @@ -24,7 +28,7 @@ DROP SERVER gitlab_secondary CASCADE; DROP EXTENSION IF EXISTS postgres_fdw; ``` -DANGER: **Warning:** +WARNING: In GitLab 13.3, promoting a secondary node to a primary while the secondary is paused fails. Do not pause replication before promoting a secondary. If the node is paused, be sure to resume before promoting. To avoid this issue, @@ -50,7 +54,7 @@ the recommended procedure, see the ## Updating to GitLab 12.9 -CAUTION: **Warning:** +WARNING: GitLab 12.9.0 through GitLab 12.9.3 are affected by [a bug that stops repository verification](https://gitlab.com/gitlab-org/gitlab/-/issues/213523). The issue is fixed in GitLab 12.9.4. Upgrade to GitLab 12.9.4 or later. @@ -81,7 +85,7 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade ## Updating to GitLab 12.7 -DANGER: **Warning:** +WARNING: Only upgrade to GitLab 12.7.5 or later. Do not upgrade to versions 12.7.0 through 12.7.4 because there is [an initialization order bug](https://gitlab.com/gitlab-org/gitlab/-/issues/199672) that causes Geo @@ -141,7 +145,7 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade ## Updating to GitLab 12.3 -DANGER: **Warning:** +WARNING: If the existing PostgreSQL server version is 9.6.x, it is recommended to upgrade to GitLab 12.4 or later. By default, GitLab 12.3 attempts to update the embedded PostgreSQL server from 9.6 to 10.9. In certain circumstances, it will @@ -155,7 +159,7 @@ For the recommended procedure, see the ## Updating to GitLab 12.2 -DANGER: **Warning:** +WARNING: If the existing PostgreSQL server version is 9.6.x, it is recommended to upgrade to GitLab 12.4 or later. By default, GitLab 12.2 attempts to update the embedded PostgreSQL server from 9.6 to 10.9. In certain circumstances, it will @@ -185,7 +189,7 @@ The restart avoids a version mismatch when PostgreSQL tries to load the FDW exte ## Updating to GitLab 12.1 -DANGER: **Warning:** +WARNING: If the existing PostgreSQL server version is 9.6.x, it is recommended to upgrade to GitLab 12.4 or later. By default, GitLab 12.1 attempts to update the embedded PostgreSQL server from 9.6 to 10.9. In certain circumstances, it will @@ -199,14 +203,14 @@ For the recommended procedure, see the ## Updating to GitLab 12.0 -CAUTION: **Warning:** +WARNING: This version is affected by a [bug that results in new LFS objects not being replicated to Geo secondary nodes](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). The issue is fixed in GitLab 12.1; be sure to upgrade to GitLab 12.1 or later. ## Updating to GitLab 11.11 -CAUTION: **Warning:** +WARNING: This version is affected by a [bug that results in new LFS objects not being replicated to Geo secondary nodes](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). The issue is fixed in GitLab 12.1; be sure to upgrade to GitLab 12.1 or later. @@ -372,7 +376,7 @@ the now-unused SSH keys from your secondaries, as they may cause problems if the ### Hashed Storage -CAUTION: **Warning:** +WARNING: Hashed storage is in **Alpha**. It is considered experimental and not production-ready. See [Hashed Storage](../../repository_storage_types.md) for more detail. @@ -383,7 +387,7 @@ migrated we recommend leaving Hashed Storage enabled. ## Updating to GitLab 10.1 -CAUTION: **Warning:** +WARNING: Hashed storage is in **Alpha**. It is considered experimental and not production-ready. See [Hashed Storage](../../repository_storage_types.md) for more detail. diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index 24e55d26997..9778e08a30b 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -1,19 +1,19 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- # Geo database replication **(PREMIUM ONLY)** -NOTE: **Note:** +NOTE: If your GitLab installation uses external (not managed by Omnibus) PostgreSQL instances, the Omnibus roles will not be able to perform all necessary configuration steps. In this case, [follow the Geo with external PostgreSQL instances document instead](external_database.md). -NOTE: **Note:** +NOTE: The stages of the setup process must be completed in the documented order. Before attempting the steps in this stage, [complete all prior stages](../setup/index.md#using-omnibus-gitlab). @@ -44,7 +44,7 @@ The following guide assumes that: you have a new **secondary** server set up with the same versions of the OS, PostgreSQL, and GitLab on all nodes. -CAUTION: **Warning:** +WARNING: Geo works with streaming replication. Logical replication is not supported at this time. There is an [issue where support is being discussed](https://gitlab.com/gitlab-org/gitlab/-/issues/7420). @@ -79,7 +79,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o 1. GitLab 10.4 and up only: Do the following to make sure the `gitlab` database user has a password defined: - NOTE: **Note:** + NOTE: Until FDW settings are removed in GitLab version 14.0, avoid using single or double quotes in the password for PostgreSQL as that will lead to errors when reconfiguring. @@ -134,7 +134,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o connect to the **primary** node's database. For this reason, we need the address of each node. - NOTE: **Note:** + NOTE: For external PostgreSQL instances, see [additional instructions](external_database.md). If you are using a cloud provider, you can lookup the addresses for each @@ -151,7 +151,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ## ## Public address ## - echo "External address: $(curl --silent ipinfo.io/ip)" + echo "External address: $(curl --silent "ipinfo.io/ip")" ``` In most cases, the following addresses will be used to configure GitLab @@ -171,7 +171,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o corresponding to the given address. See [the PostgreSQL documentation](https://www.postgresql.org/docs/11/runtime-config-connection.html) for more details. - NOTE: **Note:** + NOTE: If you need to use `0.0.0.0` or `*` as the listen_address, you will also need to add `127.0.0.1/32` to the `postgresql['md5_auth_cidr_addresses']` setting, to allow Rails to connect through `127.0.0.1`. For more information, see [omnibus-5258](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5258). @@ -290,7 +290,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o gitlab-ctl stop sidekiq ``` - NOTE: **Note:** + NOTE: This step is important so we don't try to execute anything before the node is fully configured. 1. [Check TCP connectivity](../../raketasks/maintenance.md) to the **primary** node's PostgreSQL server: @@ -299,7 +299,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o gitlab-rake gitlab:tcp_check[<primary_node_ip>,5432] ``` - NOTE: **Note:** + NOTE: If this step fails, you may be using the wrong IP address, or a firewall may be preventing access to the server. Check the IP address, paying close attention to the difference between public and private addresses and ensure @@ -404,7 +404,7 @@ needed files for streaming replication. The directories used are the defaults that are set up in Omnibus. If you have changed any defaults, configure it as you see fit replacing the directories and paths. -CAUTION: **Warning:** +WARNING: Make sure to run this on the **secondary** server as it removes all PostgreSQL's data before running `pg_basebackup`. @@ -421,7 +421,7 @@ data before running `pg_basebackup`. 1. Execute the command below to start a backup/restore and begin the replication - CAUTION: **Warning:** + WARNING: Each Geo **secondary** node must have its own unique replication slot name. Using the same slot name between two secondaries will break PostgreSQL replication. @@ -431,14 +431,9 @@ data before running `pg_basebackup`. --host=<primary_node_ip> ``` - NOTE: **Note:** + NOTE: Replication slot names must only contain lowercase letters, numbers, and the underscore character. - NOTE: **Note:** - In GitLab 13.4, a seed project is added when GitLab is first installed. This makes it necessary to pass `--force` even - on a new Geo secondary node. There is an [issue to account for seed projects](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5618) - when checking the database. - When prompted, enter the _plaintext_ password you set up for the `gitlab_replicator` user in the first step. @@ -499,6 +494,8 @@ This experimental implementation has the following limitations: For instructions about how to set up Patroni on the primary node, see the [PostgreSQL replication and failover with Omnibus GitLab](../../postgresql/replication_and_failover.md#patroni) page. +If you are currently using `repmgr` on your Geo primary, see [these instructions](#migrating-from-repmgr-to-patroni) for migrating from `repmgr` to Patroni. + A production-ready and secure setup requires at least three Patroni instances on the primary, and a similar configuration on the secondary nodes. Be sure to use password credentials and other database best practices. @@ -549,6 +546,13 @@ patroni['standby_cluster']['primary_slot_name'] = 'geo_secondary' # or the uniqu patroni['replication_password'] = 'PLAIN_TEXT_POSTGRESQL_REPLICATION_PASSWORD' ``` +## Migrating from repmgr to Patroni + +1. Before migrating, it is recommended that there is no replication lag between the primary and secondary sites and that replication is paused. In GitLab 13.2 and later, you can pause and resume replication with `gitlab-ctl geo-replication-pause` and `gitlab-ctl geo-replication-resume` on a Geo secondary database node. +1. Follow the [instructions to migrate repmgr to Patroni](../../postgresql/replication_and_failover.md#switching-from-repmgr-to-patroni). When configuring Patroni on each primary site database node, add `patroni['replicaton_slots'] = { '<slot_name>' => 'physical' }` +to `gitlab.rb` where `<slot_name>` is the name of the replication slot for your Geo secondary. This will ensure that Patroni recognizes the replication slot as permanent and will not drop it upon restarting. +1. If database replication to the secondary was paused before migration, resume replication once Patroni is confirmed working on the primary. + ## Troubleshooting Read the [troubleshooting document](../replication/troubleshooting.md). diff --git a/doc/administration/geo/setup/external_database.md b/doc/administration/geo/setup/external_database.md index 33a7bc38b6e..1fb41fbb53a 100644 --- a/doc/administration/geo/setup/external_database.md +++ b/doc/administration/geo/setup/external_database.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -11,7 +11,7 @@ This document is relevant if you are using a PostgreSQL instance that is *not managed by Omnibus*. This includes cloud-managed instances like AWS RDS, or manually installed and configured PostgreSQL instances. -NOTE: **Note:** +NOTE: We strongly recommend running Omnibus-managed instances as they are actively developed and tested. We aim to be compatible with most external (not managed by Omnibus) databases but we do not guarantee compatibility. @@ -186,7 +186,7 @@ to grant additional roles to your tracking database user (by default, this is If you have an external database ready to be used as the tracking database, follow the instructions below to use it: -NOTE: **Note:** +NOTE: If you want to use AWS RDS as a tracking database, make sure it has access to the secondary database. Unfortunately, just assigning the same security group is not enough as outbound rules do not apply to RDS PostgreSQL databases. Therefore, you need to explicitly add an inbound diff --git a/doc/administration/geo/setup/index.md b/doc/administration/geo/setup/index.md index a4d9fcba14d..8308cbfa82e 100644 --- a/doc/administration/geo/setup/index.md +++ b/doc/administration/geo/setup/index.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -12,7 +12,7 @@ These instructions assume you have a working instance of GitLab. They guide you 1. Making your existing instance the **primary** node. 1. Adding **secondary** nodes. -CAUTION: **Caution:** +WARNING: The steps below should be followed in the order they appear. **Make sure the GitLab version is the same on all nodes.** ## Using Omnibus GitLab diff --git a/doc/administration/git_annex.md b/doc/administration/git_annex.md index 59c1e621e46..e21a8ac7e84 100644 --- a/doc/administration/git_annex.md +++ b/doc/administration/git_annex.md @@ -1,14 +1,14 @@ --- stage: Create group: Source Code -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers" +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" type: reference, howto disqus_identifier: 'https://docs.gitlab.com/ee/workflow/git_annex.html' --- # Git annex -CAUTION: **Warning:** +WARNING: [Git Annex support was removed](https://gitlab.com/gitlab-org/gitlab/-/issues/1648) in GitLab 9.0. Read through the [migration guide from git-annex to Git LFS](../topics/git/lfs/migrate_from_git_annex_to_git_lfs.md). @@ -94,7 +94,7 @@ one is located in `config.yml` of GitLab Shell. ## Using GitLab git-annex -NOTE: **Note:** +NOTE: Your Git remotes must be using the SSH protocol, not HTTP(S). Here is an example workflow of uploading a very large file and then checking it diff --git a/doc/administration/git_protocol.md b/doc/administration/git_protocol.md index 841e636bd0a..ca4fa0549d0 100644 --- a/doc/administration/git_protocol.md +++ b/doc/administration/git_protocol.md @@ -1,7 +1,7 @@ --- stage: Create group: Source Code -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers" +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" type: reference description: "Set and configure Git protocol v2" --- diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 0d9c761e078..e0b0d52fe3f 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -1,7 +1,7 @@ --- stage: Create group: Gitaly -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -22,10 +22,15 @@ In the Gitaly documentation: GitLab end users do not have direct access to Gitaly. Gitaly only manages Git repository access for GitLab. Other types of GitLab data aren't accessed using Gitaly. -CAUTION: **Caution:** -From GitLab 13.0, Gitaly support for NFS is deprecated. In GitLab 14.0, Gitaly support -for NFS is scheduled to be removed. Upgrade to [Gitaly Cluster](praefect.md) as soon as -possible. +<!-- vale gitlab.FutureTense = NO --> + +WARNING: +From GitLab 13.0, Gitaly support for NFS is deprecated. As of GitLab 14.0, NFS-related issues +with Gitaly will no longer be addressed. Upgrade to [Gitaly Cluster](praefect.md) as soon as +possible. Tools to [enable bulk moves](https://gitlab.com/groups/gitlab-org/-/epics/4916) +of projects to Gitaly Cluster are planned. + +<!-- vale gitlab.FutureTense = YES --> ## Architecture @@ -68,7 +73,7 @@ this default configuration used by: However, Gitaly can be deployed to its own server, which can benefit GitLab installations that span multiple machines. -NOTE: **Note:** +NOTE: When configured to run on their own servers, Gitaly servers [must be upgraded](https://docs.gitlab.com/omnibus/update/#upgrading-gitaly-servers) before Gitaly clients in your cluster. @@ -121,7 +126,7 @@ The following list depicts the network architecture of Gitaly: - Authentication is done through a static token which is shared among the Gitaly and GitLab Rails nodes. -DANGER: **Warning:** +WARNING: Gitaly servers must not be exposed to the public internet as Gitaly's network traffic is unencrypted by default. The use of firewall is highly recommended to restrict access to the Gitaly server. Another option is to [use TLS](#enable-tls-support). @@ -140,7 +145,7 @@ We assume your GitLab installation has three repository storages: You can use as few as one server with one repository storage if desired. -NOTE: **Note:** +NOTE: The token referred to throughout the Gitaly documentation is just an arbitrary password selected by the administrator. It is unrelated to tokens created for the GitLab API or other similar web API tokens. @@ -433,9 +438,9 @@ server (with `gitaly_address`) unless you setup with special path: /some/local/path ``` - NOTE: **Note:** - `/some/local/path` should be set to a local folder that exists, however no data will be stored in - this folder. This will no longer be necessary after + NOTE: + `/some/local/path` should be set to a local folder that exists, however no data is stored in + this folder. This requirement is scheduled to be removed when [this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/1282) is resolved. 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). @@ -450,7 +455,7 @@ server (with `gitaly_address`) unless you setup with special When you tail the Gitaly logs on your Gitaly server, you should see requests coming in. One sure way to trigger a Gitaly request is to clone a repository from GitLab over HTTP or HTTPS. -DANGER: **Warning:** +WARNING: If you have [server hooks](../server_hooks.md) configured, either per repository or globally, you must move these to the Gitaly servers. If you have multiple Gitaly servers, copy your server hooks to all Gitaly servers. @@ -461,7 +466,7 @@ GitLab can reside on the same server as one of many Gitaly servers, but doesn't configuration that mixes local and remote configuration. The following setup is incorrect, because: - All addresses must be reachable from the other Gitaly servers. -- `storage1` will be assigned a Unix socket for `gitaly_address` which is +- `storage1` is assigned a Unix socket for `gitaly_address` which is invalid for some of the Gitaly servers. ```ruby @@ -493,7 +498,7 @@ gitaly['key_path'] = "/etc/gitlab/ssl/key.pem" ``` `path` can only be included for storage shards on the local Gitaly server. -If it's excluded, default Git storage directory will be used for that storage shard. +If it's excluded, default Git storage directory is used for that storage shard. ### Disable Gitaly where not required (optional) @@ -529,12 +534,18 @@ To disable Gitaly on a GitLab server: ## Enable TLS support -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/22602) in GitLab 11.8. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/22602) in GitLab 11.8. +> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/3160) in GitLab 13.6, outgoing TLS connections to GitLab provide client certificates if configured. Gitaly supports TLS encryption. To communicate with a Gitaly instance that listens for secure connections, you must use `tls://` URL scheme in the `gitaly_address` of the corresponding storage entry in the GitLab configuration. +Gitaly provides the same server certificates as client certificates in TLS +connections to GitLab. This can be used as part of a mutual TLS authentication strategy +when combined with reverse proxies (for example, NGINX) that validate client certificate +to grant access to GitLab. + You must supply your own certificates as this isn't provided automatically. The certificate corresponding to each Gitaly server must be installed on that Gitaly server. @@ -584,7 +595,7 @@ To configure Gitaly with TLS: ``` 1. Copy all Gitaly server certificates (or their certificate authority) to - `/etc/gitlab/trusted-certs` so that Gitaly servers will trust the certificate when calling into themselves + `/etc/gitlab/trusted-certs` so that Gitaly servers trust the certificate when calling into themselves or other Gitaly servers: ```shell @@ -640,9 +651,9 @@ To configure Gitaly with TLS: path: /some/local/path ``` - NOTE: **Note:** - `/some/local/path` should be set to a local folder that exists, however no data will be stored - in this folder. This will no longer be necessary after + NOTE: + `/some/local/path` should be set to a local folder that exists, however no data is stored + in this folder. This requirement is scheduled to be removed when [Gitaly issue #1282](https://gitlab.com/gitlab-org/gitaly/-/issues/1282) is resolved. 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). @@ -662,7 +673,7 @@ To configure Gitaly with TLS: ``` 1. Copy all Gitaly server certificates (or their certificate authority) to the system trusted - certificates folder so Gitaly server will trust the certificate when calling into itself or other Gitaly + certificates folder so Gitaly server trusts the certificate when calling into itself or other Gitaly servers. ```shell @@ -716,7 +727,7 @@ We recommend: - At least 300MB memory per worker. - No more than one worker per core. -NOTE: **Note:** +NOTE: `gitaly-ruby` is planned to be eventually removed. To track progress, see the [Remove the Gitaly-Ruby sidecar](https://gitlab.com/groups/gitlab-org/-/epics/2862) epic. @@ -798,9 +809,9 @@ You can observe the behavior of this queue using the Gitaly logs and Prometheus: - `gitaly_rate_limiting_queued`. - `gitaly_rate_limiting_seconds`. -NOTE: **Note:** +NOTE: Though the name of the Prometheus metric contains `rate_limiting`, it is a concurrency limiter, not -a rate limiter. If a Gitaly client makes 1000 requests in a row very quickly, concurrency will not +a rate limiter. If a Gitaly client makes 1000 requests in a row very quickly, concurrency does not exceed 1 and the concurrency limiter has no effect. ## Rotate Gitaly authentication token @@ -829,7 +840,7 @@ behavior of your GitLab installation using Prometheus. Use the following Prometh sum(rate(gitaly_authentications_total[5m])) by (enforced, status) ``` -In a system where authentication is configured correctly and where you have live traffic, you will +In a system where authentication is configured correctly and where you have live traffic, you see something like this: ```prometheus @@ -885,7 +896,7 @@ To update to a new Gitaly authentication token, on each Gitaly client **and** Gi ``` If you run your [Prometheus query](#verify-authentication-monitoring) while this change is -being rolled out, you will see non-zero values for the `enforced="false",status="denied"` counter. +being rolled out, you see non-zero values for the `enforced="false",status="denied"` counter. ### Ensure there are no authentication failures @@ -908,7 +919,7 @@ your Gitaly servers as follows: gitaly['auth_transitioning'] = false ``` -CAUTION: **Caution:** +WARNING: Without completing this step, you have **no Gitaly authentication**. ### Verify authentication is enforced @@ -959,7 +970,7 @@ not an external process, there was very little overhead between: - GitLab application code that tried to look up data in Git repositories. - The Git implementation itself. -Because the combination of Rugged and Unicorn was so efficient, GitLab's application code ended up with lots of +Because the combination of Rugged and Unicorn was so efficient, the GitLab application code ended up with lots of duplicate Git object lookups. For example, looking up the `master` commit a dozen times in one request. We could write inefficient code without poor performance. @@ -995,9 +1006,9 @@ enables direct Git access. When GitLab calls a function that has a "Rugged patch", it performs two checks: - Is the feature flag for this patch set in the database? If so, the feature flag setting controls - GitLab's use of "Rugged patch" code. + the GitLab use of "Rugged patch" code. - If the feature flag is not set, GitLab tries accessing the filesystem underneath the - Gitaly server directly. If it can, it will use the "Rugged patch": + Gitaly server directly. If it can, it uses the "Rugged patch": - If using Unicorn. - If using Puma and [thread count](../../install/requirements.md#puma-threads) is set to `1`. @@ -1070,7 +1081,7 @@ gitaly-debug -h remote: GitLab: 401 Unauthorized ``` -You will need to sync your `gitlab-secrets.json` file with your Gitaly clients (GitLab +You need to sync your `gitlab-secrets.json` file with your Gitaly clients (GitLab app nodes). ### Client side gRPC logs diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index d091ae5895a..6eaafae6015 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -1,7 +1,7 @@ --- stage: Create group: Gitaly -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -12,7 +12,7 @@ be run in a clustered configuration to increase fault tolerance. In this configuration, every Git repository is stored on every Gitaly node in the cluster. Multiple clusters (or shards), can be configured. -NOTE: **Note:** +NOTE: Gitaly Clusters can be created using [GitLab Core](https://about.gitlab.com/pricing/#self-managed) and higher tiers. However, technical support is limited to GitLab Premium and Ultimate customers only. Not available in GitLab.com. @@ -47,18 +47,40 @@ The availability objectives for Gitaly clusters are: [Faster outage detection](https://gitlab.com/gitlab-org/gitaly/-/issues/2608) is planned to improve this to less than 1 second. -The current version supports: +Gitaly Cluster supports: -- Eventual consistency of the secondary replicas. -- Automatic failover from the primary to the secondary. -- Reporting of possible data loss if replication queue is non empty. -- Marking the newly promoted primary read only if possible data loss is - detected. +- [Strong consistency](#strong-consistency) of the secondary replicas. +- [Automatic failover](#automatic-failover-and-leader-election) from the primary to the secondary. +- Reporting of possible data loss if replication queue is non-empty. +- Marking repositories as [read only](#read-only-mode) if data loss is detected to prevent data inconsistencies. Follow the [HA Gitaly epic](https://gitlab.com/groups/gitlab-org/-/epics/1489) for improvements including [horizontally distributing reads](https://gitlab.com/groups/gitlab-org/-/epics/2013). +## Gitaly Cluster compared to Geo + +Gitaly Cluster and [Geo](../geo/index.md) both provide redundancy. However the redundancy of: + +- Gitaly Cluster provides fault tolerance for data storage and is invisible to the user. Users are + not aware when Gitaly Cluster is used. +- Geo provides [replication](../geo/index.md) and [disaster recovery](../geo/disaster_recovery/index.md) for + an entire instance of GitLab. Users know when they are using Geo for + [replication](../geo/index.md). Geo [replicates multiple datatypes](../geo/replication/datatypes.md#limitations-on-replicationverification), + including Git data. + +The following table outlines the major differences between Gitaly Cluster and Geo: + +| Tool | Nodes | Locations | Latency tolerance | Failover | Consistency | Provides redundancy for | +|:---------------|:---------|:----------|:-------------------|:-----------------------------------------------------|:------------------------------|:------------------------| +| Gitaly Cluster | Multiple | Single | Approximately 1 ms | [Automatic](#automatic-failover-and-leader-election) | [Strong](#strong-consistency) | Data storage in Git | +| Geo | Multiple | Multiple | Up to one minute | [Manual](../geo/disaster_recovery/index.md) | Eventual | Entire GitLab instance | + +For more information, see: + +- [Gitaly architecture](index.md#architecture). +- Geo [use cases](../geo/index.md#use-cases) and [architecture](../geo/index.md#architecture). + ## Cluster or shard Gitaly supports multiple models of scaling: @@ -69,8 +91,8 @@ Gitaly supports multiple models of scaling: - Sharding using [repository storage paths](../repository_storage_paths.md), where each repository is stored on the assigned Gitaly node. All requests are routed to this node. -| Cluster | Shard | -|---|---| +| Cluster | Shard | +|:--------------------------------------------------|:----------------------------------------------| |  |  | Generally, Gitaly Cluster can replace sharded configurations, at the expense of additional storage @@ -122,9 +144,7 @@ package (highly recommended), follow the steps below: Before beginning, you should already have a working GitLab instance. [Learn how to install GitLab](https://about.gitlab.com/install/). -Provision a PostgreSQL server (PostgreSQL 11 or newer). Configuration through -the Omnibus GitLab distribution is not yet supported. Follow this -[issue](https://gitlab.com/gitlab-org/gitaly/-/issues/2476) for updates. +Provision a PostgreSQL server (PostgreSQL 11 or newer). Prepare all your new nodes by [installing GitLab](https://about.gitlab.com/install/). @@ -133,7 +153,7 @@ GitLab](https://about.gitlab.com/install/). - 3 Gitaly nodes (high CPU, high memory, fast storage) - 1 GitLab server -You will need the IP/host address for each node. +You need the IP/host address for each node. 1. `LOAD_BALANCER_SERVER_ADDRESS`: the IP/host address of the load balancer 1. `POSTGRESQL_SERVER_ADDRESS`: the IP/host address of the PostgreSQL server @@ -149,7 +169,7 @@ If you are using Google Cloud Platform, SoftLayer, or any other vendor that prov The communication between components is secured with different secrets, which are described below. Before you begin, generate a unique secret for each, and -make note of it. This will make it easy to replace these placeholder tokens +make note of it. This makes it easy to replace these placeholder tokens with secure tokens as you complete the setup process. 1. `GITLAB_SHELL_SECRET_TOKEN`: this is used by Git hooks to make callback HTTP @@ -164,11 +184,11 @@ with secure tokens as you complete the setup process. 1. `PRAEFECT_SQL_PASSWORD`: this password is used by Praefect to connect to PostgreSQL. -We will note in the instructions below where these secrets are required. +We note in the instructions below where these secrets are required. ### PostgreSQL -NOTE: **Note:** +NOTE: Do not store the GitLab application database and the Praefect database on the same PostgreSQL server if using [Geo](../geo/index.md). The replication state is internal to each instance @@ -184,13 +204,13 @@ failure. For greater fault tolerance, the following options are available: - Use a cloud-managed PostgreSQL service. AWS [Relational Database Service](https://aws.amazon.com/rds/) is recommended. -To complete this section you will need: +To complete this section you need: - 1 Praefect node - 1 PostgreSQL server (PostgreSQL 11 or newer) - An SQL user with permissions to create databases -During this section, we will configure the PostgreSQL server, from the Praefect +During this section, we configure the PostgreSQL server, from the Praefect node, using `psql` which is installed by Omnibus GitLab. 1. SSH into the **Praefect** node and login as root: @@ -207,7 +227,7 @@ node, using `psql` which is installed by Omnibus GitLab. /opt/gitlab/embedded/bin/psql -U postgres -d template1 -h POSTGRESQL_SERVER_ADDRESS ``` - Create a new user `praefect` which will be used by Praefect. Replace + Create a new user `praefect` to be used by Praefect. Replace `PRAEFECT_SQL_PASSWORD` with the strong password you generated in the preparation step. @@ -234,8 +254,23 @@ The database used by Praefect is now configured. To reduce PostgreSQL resource consumption, we recommend setting up and configuring [PgBouncer](https://www.pgbouncer.org/) in front of the PostgreSQL instance. To do -this, replace value of the `POSTGRESQL_SERVER_ADDRESS` with corresponding IP or host -address of the PgBouncer instance. +this, set the corresponding IP or host address of the PgBouncer instance in +`/etc/gitlab/gitlab.rb` by changing the following settings: + +- `praefect['database_host']`, for the address. +- `praefect['database_port']`, for the port. + +Because PgBouncer manages resources more efficiently, Praefect still requires a +direct connection to the PostgreSQL database because it uses +[LISTEN](https://www.postgresql.org/docs/11/sql-listen.html) +functionality that is [not supported](https://www.pgbouncer.org/features.html) by +PgBouncer with `pool_mode = transaction`. + +Therefore, `praefect['database_host_no_proxy']` and `praefect['database_port_no_proxy']` +should be set to a direct connection and not a PgBouncer connection. + +Save the changes to `/etc/gitlab/gitlab.rb` and +[reconfigure Praefect](../restart_gitlab.md#omnibus-gitlab-reconfigure). This documentation doesn't provide PgBouncer installation instructions, but you can: @@ -267,7 +302,7 @@ The `praefect` user and its password should be included in the file (default is `userlist.txt`) used by PgBouncer if the [`auth_file`](https://www.pgbouncer.org/config.html#auth_file) configuration option is set. -NOTE: **Note:** +NOTE: By default PgBouncer uses port `6432` to accept incoming connections. You can change it by setting the [`listen_port`](https://www.pgbouncer.org/config.html#listen_port) configuration option. We recommend setting it to the default port value (`5432`) used by @@ -278,14 +313,13 @@ PostgreSQL instances. Otherwise you should change the configuration parameter > [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/2634) in GitLab 13.4, Praefect nodes can no longer be designated as `primary`. -NOTE: **Note:** +NOTE: If there are multiple Praefect nodes, complete these steps for **each** node. -To complete this section you will need: +To complete this section you need a [configured PostgreSQL server](#postgresql), including: -- [Configured PostgreSQL server](#postgresql), including: - - IP/host address (`POSTGRESQL_SERVER_ADDRESS`) - - password (`PRAEFECT_SQL_PASSWORD`) +- IP/host address (`POSTGRESQL_SERVER_ADDRESS`) +- Password (`PRAEFECT_SQL_PASSWORD`) Praefect should be run on a dedicated node. Do not run Praefect on the application server, or a Gitaly node. @@ -331,8 +365,8 @@ application server, or a Gitaly node. ``` 1. Configure a strong `auth_token` for **Praefect** by editing - `/etc/gitlab/gitlab.rb`. This will be needed by clients outside the cluster - (like GitLab Shell) to communicate with the Praefect cluster : + `/etc/gitlab/gitlab.rb`. This is needed by clients outside the cluster + (like GitLab Shell) to communicate with the Praefect cluster: ```ruby praefect['auth_token'] = 'PRAEFECT_EXTERNAL_TOKEN' @@ -341,7 +375,7 @@ application server, or a Gitaly node. 1. Configure **Praefect** to connect to the PostgreSQL database by editing `/etc/gitlab/gitlab.rb`. - You will need to replace `POSTGRESQL_SERVER_ADDRESS` with the IP/host address + You need to replace `POSTGRESQL_SERVER_ADDRESS` with the IP/host address of the database, and `PRAEFECT_SQL_PASSWORD` with the strong password set above. @@ -351,6 +385,8 @@ application server, or a Gitaly node. praefect['database_user'] = 'praefect' praefect['database_password'] = 'PRAEFECT_SQL_PASSWORD' praefect['database_dbname'] = 'praefect_production' + praefect['database_host_no_proxy'] = 'POSTGRESQL_SERVER_ADDRESS' + praefect['database_port_no_proxy'] = 5432 ``` If you want to use a TLS client certificate, the options below can be used: @@ -364,7 +400,7 @@ application server, or a Gitaly node. # praefect['database_sslrootcert'] = '/path/to/rootcert' ``` - By default Praefect will refuse to make an unencrypted connection to + By default, Praefect refuses to make an unencrypted connection to PostgreSQL. You can override this by uncommenting the following line: ```ruby @@ -377,15 +413,15 @@ application server, or a Gitaly node. The virtual storage's name must match the configured storage name in GitLab configuration. In a later step, we configure the storage name as `default` so we use `default` here as well. This cluster has three Gitaly nodes `gitaly-1`, - `gitaly-2`, and `gitaly-3`, which will be replicas of each other. + `gitaly-2`, and `gitaly-3`, which are intended to be replicas of each other. - CAUTION: **Caution:** + WARNING: If you have data on an already existing storage called `default`, you should configure the virtual storage with another name and [migrate the data to the Gitaly Cluster storage](#migrate-existing-repositories-to-gitaly-cluster) afterwards. - Replace `PRAEFECT_INTERNAL_TOKEN` with a strong secret, which will be used by + Replace `PRAEFECT_INTERNAL_TOKEN` with a strong secret, which is used by Praefect when communicating with Gitaly nodes in the cluster. This token is distinct from the `PRAEFECT_EXTERNAL_TOKEN`. @@ -479,6 +515,7 @@ To configure Praefect with TLS: **For Omnibus GitLab** 1. Create certificates for Praefect servers. + 1. On the Praefect servers, create the `/etc/gitlab/ssl` directory and copy your key and certificate there: @@ -497,7 +534,8 @@ To configure Praefect with TLS: praefect['key_path'] = "/etc/gitlab/ssl/key.pem" ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). + 1. On the Praefect clients (including each Gitaly server), copy the certificates, or their certificate authority, into `/etc/gitlab/trusted-certs`: @@ -510,8 +548,10 @@ To configure Praefect with TLS: ```ruby git_data_dirs({ - 'default' => { 'gitaly_address' => 'tls://praefect1.internal:3305' }, - 'storage1' => { 'gitaly_address' => 'tls://praefect2.internal:3305' }, + "default" => { + "gitaly_address" => 'tls://LOAD_BALANCER_SERVER_ADDRESS:2305', + "gitaly_token" => 'PRAEFECT_EXTERNAL_TOKEN' + } }) ``` @@ -546,21 +586,18 @@ To configure Praefect with TLS: repositories: storages: default: - gitaly_address: tls://praefect1.internal:3305 - path: /some/local/path - storage1: - gitaly_address: tls://praefect2.internal:3305 + gitaly_address: tls://LOAD_BALANCER_SERVER_ADDRESS:3305 path: /some/local/path ``` - NOTE: **Note:** + NOTE: `/some/local/path` should be set to a local folder that exists, however no - data will be stored in this folder. This will no longer be necessary after + data is stored in this folder. This requirement is scheduled to be removed when [this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/1282) is resolved. 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). 1. Copy all Praefect server certificates, or their certificate authority, to the system - trusted certificates on each Gitaly server so the Praefect server will trust the + trusted certificates on each Gitaly server so the Praefect server trusts the certificate when called by Gitaly servers: ```shell @@ -582,10 +619,10 @@ To configure Praefect with TLS: ### Gitaly -NOTE: **Note:** +NOTE: Complete these steps for **each** Gitaly node. -To complete this section you will need: +To complete this section you need: - [Configured Praefect node](#praefect) - 3 (or more) servers, with GitLab installed, to be configured as Gitaly nodes. @@ -595,19 +632,19 @@ Every Gitaly server assigned to the Praefect cluster needs to be configured. The configuration is the same as a normal [standalone Gitaly server](index.md), except: -- the storage names are exposed to Praefect, not GitLab -- the secret token is shared with Praefect, not GitLab +- The storage names are exposed to Praefect, not GitLab +- The secret token is shared with Praefect, not GitLab The configuration of all Gitaly nodes in the Praefect cluster can be identical, because we rely on Praefect to route operations correctly. Particular attention should be shown to: -- the `gitaly['auth_token']` configured in this section must match the `token` +- The `gitaly['auth_token']` configured in this section must match the `token` value under `praefect['virtual_storages']` on the Praefect node. This was set in the [previous section](#praefect). This document uses the placeholder `PRAEFECT_INTERNAL_TOKEN` throughout. -- the storage names in `git_data_dirs` configured in this section must match the +- The storage names in `git_data_dirs` configured in this section must match the storage names under `praefect['virtual_storages']` on the Praefect node. This was set in the [previous section](#praefect). This document uses `gitaly-1`, `gitaly-2`, and `gitaly-3` as Gitaly storage names. @@ -659,8 +696,8 @@ documentation](index.md#configure-gitaly-servers). ``` 1. Configure a strong `auth_token` for **Gitaly** by editing - `/etc/gitlab/gitlab.rb`. This will be needed by clients to communicate with - this Gitaly nodes. Typically, this token will be the same for all Gitaly + `/etc/gitlab/gitlab.rb`. This is needed by clients to communicate with + this Gitaly nodes. Typically, this token is the same for all Gitaly nodes. ```ruby @@ -728,7 +765,7 @@ documentation](index.md#configure-gitaly-servers). After all Gitaly nodes are configured, you can run the Praefect connection checker to verify Praefect can connect to all Gitaly servers in the Praefect -config. +configuration. 1. SSH into each **Praefect** node and run the Praefect connection checker: @@ -743,7 +780,7 @@ internal traffic from the GitLab application to the Praefect nodes. The specifics on which load balancer to use or the exact configuration is beyond the scope of the GitLab documentation. -NOTE: **Note:** +NOTE: The load balancer must be configured to accept traffic from the Gitaly nodes in addition to the GitLab nodes. Some requests handled by [`gitaly-ruby`](index.md#gitaly-ruby) sidecar processes call into the main Gitaly @@ -754,16 +791,16 @@ We hope that if youâre managing HA systems like GitLab, you have a load balanc of choice already. Some examples include [HAProxy](https://www.haproxy.org/) (open-source), [Google Internal Load Balancer](https://cloud.google.com/load-balancing/docs/internal/), [AWS Elastic Load Balancer](https://aws.amazon.com/elasticloadbalancing/), F5 -Big-IP LTM, and Citrix Net Scaler. This documentation will outline what ports +Big-IP LTM, and Citrix Net Scaler. This documentation outlines what ports and protocols you need configure. | LB Port | Backend Port | Protocol | -|---------|--------------|----------| +|:--------|:-------------|:---------| | 2305 | 2305 | TCP | ### GitLab -To complete this section you will need: +To complete this section you need: - [Configured Praefect node](#praefect) - [Configured Gitaly nodes](#gitaly) @@ -787,17 +824,17 @@ Particular attention should be shown to: 1. Configure the `external_url` so that files could be served by GitLab by proper endpoint access by editing `/etc/gitlab/gitlab.rb`: - You will need to replace `GITLAB_SERVER_URL` with the real external facing + You need to replace `GITLAB_SERVER_URL` with the real external facing URL on which current GitLab instance is serving: ```ruby external_url 'GITLAB_SERVER_URL' ``` -1. Disable the default Gitaly service running on the GitLab host. It won't be needed - as GitLab will connect to the configured cluster. +1. Disable the default Gitaly service running on the GitLab host. It isn't needed + because GitLab connects to the configured cluster. - CAUTION: **Caution:** + WARNING: If you have existing data stored on the default Gitaly storage, you should [migrate the data your Gitaly Cluster storage](#migrate-existing-repositories-to-gitaly-cluster) first. @@ -809,12 +846,14 @@ Particular attention should be shown to: 1. Add the Praefect cluster as a storage location by editing `/etc/gitlab/gitlab.rb`. - You will need to replace: + You need to replace: - `LOAD_BALANCER_SERVER_ADDRESS` with the IP address or hostname of the load balancer. - `PRAEFECT_EXTERNAL_TOKEN` with the real secret + If you are using TLS, the `gitaly_address` should begin with `tls://`. + ```ruby git_data_dirs({ "default" => { @@ -828,7 +867,7 @@ Particular attention should be shown to: nodes during a `git push` are properly authenticated by editing `/etc/gitlab/gitlab.rb`: - You will need to replace `GITLAB_SHELL_SECRET_TOKEN` with the real secret. + You need to replace `GITLAB_SHELL_SECRET_TOKEN` with the real secret. ```ruby gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN' @@ -837,7 +876,7 @@ Particular attention should be shown to: 1. Add Prometheus monitoring settings by editing `/etc/gitlab/gitlab.rb`. If Prometheus is enabled on a different node, make edits on that node instead. - You will need to replace: + You need to replace: - `PRAEFECT_HOST` with the IP address or hostname of the Praefect node - `GITALY_HOST` with the IP address or hostname of each Gitaly node @@ -922,7 +961,7 @@ To get started quickly: gitlab-ctl reconfigure ``` -1. Set the Grafana admin password. This command will prompt you to enter a new +1. Set the Grafana admin password. This command prompts you to enter a new password: ```shell @@ -966,7 +1005,7 @@ _Up to date_ in this context means that: - The last replication operation is in _completed_ state. If there is no such nodes, or any other error occurs during node selection, the primary -node will be chosen to serve the request. +node is chosen to serve the request. To track distribution of read operations, you can use the `gitaly_praefect_read_distribution` Prometheus counter metric. It has two labels: @@ -1032,6 +1071,55 @@ To monitor strong consistency, you can use the following Prometheus metrics: - `gitaly_hook_transaction_voting_delay_seconds`: Client-side delay introduced by waiting for the transaction to be committed. +## Replication factor + +Replication factor is the number of copies Praefect maintains of a given repository. A higher +replication factor offers better redundancy and distribution of read workload, but also results +in a higher storage cost. By default, Praefect replicates repositories to every storage in a +virtual storage. + +### Variable replication factor + +WARNING: +The feature is not production ready yet. After you set a replication factor, you can't unset it +without manually modifying database state. Variable replication factor requires you to enable +repository-specific primaries by configuring the `per_repository` primary election strategy. The election +strategy is not production ready yet. + +Praefect supports configuring a replication factor on a per-repository basis, by assigning +specific storage nodes to host a repository. + +[In an upcoming release](https://gitlab.com/gitlab-org/gitaly/-/issues/3362), we intend to +support configuring a default replication factor for a virtual storage. The default replication factor +is applied to every newly-created repository. + +Prafect does not store the actual replication factor, but assigns enough storages to host the repository +so the desired replication factor is met. If a storage node is later removed from the virtual storage, +the replication factor of repositories assigned to the storage is decreased accordingly. + +The only way to configure a repository's replication factor is the `set-replication-factor` +sub-command. `set-replication-factor` automatically assigns or unassigns random storage nodes as necessary to +reach the desired replication factor. The repository's primary node is always assigned +first and is never unassigned. + +```shell +sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml set-replication-factor -virtual-storage <virtual-storage> -repository <relative-path> -replication-factor <replication-factor> +``` + +- `-virtual-storage` is the virtual storage the repository is located in. +- `-repository` is the repository's relative path in the storage. +- `-replication-factor` is the desired replication factor of the repository. The minimum value is + `1`, as the primary needs a copy of the repository. The maximum replication factor is the number of + storages in the virtual storage. + +On success, the assigned host storages are printed. For example: + +```shell +$ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml set-replication-factor -virtual-storage default -repository @hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.git -replication-factor 2 + +current assignments: gitaly-1, gitaly-2 +``` + ## Automatic failover and leader election Praefect regularly checks the health of each backend Gitaly node. This @@ -1040,9 +1128,9 @@ current primary node is found to be unhealthy. - **PostgreSQL (recommended):** Enabled by default, and equivalent to: `praefect['failover_election_strategy'] = sql`. This configuration - option will allow multiple Praefect nodes to coordinate via the + option allows multiple Praefect nodes to coordinate via the PostgreSQL database to elect a primary Gitaly node. This configuration - will cause Praefect nodes to elect a new primary, monitor its health, + causes Praefect nodes to elect a new primary, monitor its health, and elect a new primary if the current one has not been reachable in 10 seconds by a majority of the Praefect nodes. - **Memory:** Enabled by setting `praefect['failover_election_strategy'] = 'local'` @@ -1051,8 +1139,7 @@ current primary node is found to be unhealthy. be elected. **Do not use with multiple Praefect nodes!** Using with multiple Praefect nodes is likely to result in a split brain. -It is likely that we will implement support for Consul, and a cloud native -strategy in the future. +We are likely to implement support for Consul, and a cloud native, strategy in the future. ## Primary Node Failure @@ -1090,15 +1177,14 @@ useful for identifying potential data loss after a failover. The following param available: - `-virtual-storage` that specifies which virtual storage to check. The default behavior is to - display outdated replicas of read-only repositories as they generally require administrator - action. + display outdated replicas of read-only repositories as they might require administrator action. - In GitLab 13.3 and later, `-partially-replicated` that specifies whether to display a list of [outdated replicas of writable repositories](#outdated-replicas-of-writable-repositories). -NOTE: **Note:** +NOTE: `dataloss` is still in beta and the output format is subject to change. -To check for outdated replicas of read-only repositories, run: +To check for repositories with outdated primaries, run: ```shell sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss [-virtual-storage <virtual-storage>] @@ -1110,24 +1196,45 @@ Every configured virtual storage is checked if none is specified: sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss ``` -The number of potentially unapplied changes to repositories is listed for each replica. Listed -repositories might have the latest changes but it is not guaranteed. Only outdated replicas of -read-only repositories are listed by default. For example: +Repositories which have assigned storage nodes that contain an outdated copy of the repository are listed +in the output. A number of useful information is printed for each repository: + +- A repository's relative path to the storage directory identifies each repository and groups the related + information. +- The repository's current status is printed in parentheses next to the disk path. If the repository's primary + is outdated, the repository is in `read-only` mode and can't accept writes. Otherwise, the mode is `writable`. +- The primary field lists the repository's current primary. If the repository has no primary, the field shows + `No Primary`. +- The In-Sync Storages lists replicas which have replicated the latest successful write and all writes + preceding it. +- The Outdated Storages lists replicas which contain an outdated copy of the repository. Replicas which have no copy + of the repository but should contain it are also listed here. The maximum number of changes the replica is missing + is listed next to replica. It's important to notice that the outdated replicas may be fully up to date or contain + later changes but Praefect can't guarantee it. + +Whether a replica is assigned to host the repository is listed with each replica's status. `assigned host` is printed +next to replicas which are assigned to store the repository. The text is omitted if the replica contains a copy of +the repository but is not assigned to store the repository. Such replicas won't be kept in-sync by Praefect but may +act as replication sources to bring assigned replicas up to date. + +Example output: ```shell Virtual storage: default - Primary: gitaly-3 Outdated repositories: - @hashed/2c/62/2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3.git (read-only): - gitaly-2 is behind by 1 change or less - gitaly-3 is behind by 2 changes or less + @hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.git (read-only): + Primary: gitaly-1 + In-Sync Storages: + gitaly-2, assigned host + Outdated Storages: + gitaly-1 is behind by 3 changes or less, assigned host + gitaly-3 is behind by 3 changes or less ``` A confirmation is printed out when every repository is writable. For example: ```shell Virtual storage: default - Primary: gitaly-1 All repositories are writable! ``` @@ -1135,8 +1242,8 @@ Virtual storage: default > [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/3019) in GitLab 13.3. -To also list information for outdated replicas of writable repositories, use the -`-partially-replicated` parameter. +To also list information of repositories whose primary is up to date but one or more assigned +replicas are outdated, use the `-partially-replicated` flag. A repository is writable if the primary has the latest changes. Secondaries might be temporarily outdated while they are waiting to replicate the latest changes. @@ -1149,21 +1256,23 @@ Example output: ```shell Virtual storage: default - Primary: gitaly-3 Outdated repositories: - @hashed/2c/62/2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3.git (read-only): - gitaly-2 is behind by 1 change or less - gitaly-3 is behind by 2 changes or less - @hashed/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a.git (writable): - gitaly-2 is behind by 1 change or less + @hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.git (writable): + Primary: gitaly-1 + In-Sync Storages: + gitaly-1, assigned host + Outdated Storages: + gitaly-2 is behind by 3 changes or less, assigned host + gitaly-3 is behind by 3 changes or less ``` -With the `-partially-replicated` flag set, a confirmation is printed out if every replica is fully up to date. +With the `-partially-replicated` flag set, a confirmation is printed out if every assigned replica is fully up to +date. + For example: ```shell Virtual storage: default - Primary: gitaly-1 All repositories are up to date! ``` @@ -1193,7 +1302,7 @@ Praefect provides the following subcommands to re-enable writes: sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml accept-dataloss -virtual-storage <virtual-storage> -repository <relative-path> -authoritative-storage <storage-name> ``` -CAUTION: **Caution:** +WARNING: `accept-dataloss` causes permanent data loss by overwriting other versions of the repository. Data [recovery efforts](#data-recovery) must be performed before using it. @@ -1257,23 +1366,27 @@ Gitaly Cluster automatically. Repositories may be moved from one storage location using the [Project repository storage moves API](../../api/project_repository_storage_moves.md): +NOTE: +The Project repository storage moves API [cannot move all repository types](../../api/project_repository_storage_moves.md#limitations). + To move repositories to Gitaly Cluster: -1. [Schedule a move](../../api/project_repository_storage_moves.md#schedule-a-repository-storage-move-for-a-project) - for the first repository using the API. For example: +1. [Schedule repository storage moves for all projects on a storage shard](../../api/project_repository_storage_moves.md#schedule-repository-storage-moves-for-all-projects-on-a-storage-shard) using the API. For example: ```shell curl --request POST --header "Private-Token: <your_access_token>" --header "Content-Type: application/json" \ - --data '{"destination_storage_name":"praefect"}' "https://gitlab.example.com/api/v4/projects/123/repository_storage_moves" + --data '{"source_storage_name":"gitaly","destination_storage_name":"praefect"}' "https://gitlab.example.com/api/v4/project_repository_storage_moves" ``` -1. Using the ID that is returned, [query the repository move](../../api/project_repository_storage_moves.md#get-a-single-repository-storage-move-for-a-project) +1. [Query the most recent repository moves](../../api/project_repository_storage_moves.md#retrieve-all-project-repository-storage-moves) using the API. The query indicates either: - - The move has completed successfully. The `state` field is `finished`. - - The move is in progress. Re-query the repository move until it completes successfully. - - The move has failed. Most failures are temporary and are solved by rescheduling the move. + - The moves have completed successfully. The `state` field is `finished`. + - The moves are in progress. Re-query the repository move until it completes successfully. + - The moves have failed. Most failures are temporary and are solved by rescheduling the move. -1. Once the move is successful, repeat these steps for all repositories for your projects. +1. Once the moves are complete, [query projects](../../api/projects.md#list-all-projects) + using the API to confirm that all projects have moved. No projects should be returned + with `repository_storage` field set to the old storage. ## Debugging Praefect diff --git a/doc/administration/gitaly/reference.md b/doc/administration/gitaly/reference.md index 53001b946d8..5a004d97220 100644 --- a/doc/administration/gitaly/reference.md +++ b/doc/administration/gitaly/reference.md @@ -1,7 +1,7 @@ --- stage: Create group: Gitaly -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -66,7 +66,7 @@ token = "the secret token" transitioning = true ``` -CAUTION: **Warning:** +WARNING: Remember to disable `transitioning` when you are done changing your token settings. @@ -75,7 +75,7 @@ the `gitaly_authentications_total` metric. ### TLS -Gitaly supports TLS encryption. You will need to bring your own certificates as +Gitaly supports TLS encryption. You need to bring your own certificates as this isn't provided automatically. | Name | Type | Required | Description | @@ -128,7 +128,7 @@ The following values can be set in the `[git]` section of the configuration file | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | -| `bin_path` | string | no | Path to Git binary. If not set, will be resolved using `PATH`. | +| `bin_path` | string | no | Path to Git binary. If not set, is resolved using `PATH`. | | `catfile_cache_size` | integer | no | Maximum number of cached [cat-file processes](#cat-file-cache). Default is `100`. | #### `cat-file` cache @@ -192,8 +192,8 @@ For historical reasons [GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell) contains the Git hooks that allow GitLab to validate and react to Git pushes. Because Gitaly "owns" Git pushes, GitLab Shell must therefore be -installed alongside Gitaly. This will be [simplified in the -future](https://gitlab.com/gitlab-org/gitaly/-/issues/1226). +installed alongside Gitaly. We plan to +[simplify this](https://gitlab.com/gitlab-org/gitaly/-/issues/1226). | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | @@ -238,9 +238,11 @@ The following values configure logging in Gitaly under the `[logging]` section. While the main Gitaly application logs go to `stdout`, there are some extra log files that go to a configured directory, like the GitLab Shell logs. -GitLab Shell does not support `panic` or `trace` level logs. `panic` will fall -back to `error`, while `trace` will fall back to `debug`. Any other invalid log -levels will default to `info`. +GitLab Shell does not support `panic` or `trace` level logs: + +- `panic` falls back to `error`. +- `trace` falls back to `debug`. +- Any other invalid log levels default to `info`. Example: diff --git a/doc/administration/housekeeping.md b/doc/administration/housekeeping.md index c784c788b31..90d2f0d916d 100644 --- a/doc/administration/housekeeping.md +++ b/doc/administration/housekeeping.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Housekeeping diff --git a/doc/administration/img/audit_log.png b/doc/administration/img/audit_log.png Binary files differdeleted file mode 100644 index d4f4c2abf38..00000000000 --- a/doc/administration/img/audit_log.png +++ /dev/null diff --git a/doc/administration/img/audit_log_v13_6.png b/doc/administration/img/audit_log_v13_6.png Binary files differnew file mode 100644 index 00000000000..82ff3e9c87b --- /dev/null +++ b/doc/administration/img/audit_log_v13_6.png diff --git a/doc/administration/img/export_audit_log_v13_4.png b/doc/administration/img/export_audit_log_v13_4.png Binary files differdeleted file mode 100644 index e4ba330b8a9..00000000000 --- a/doc/administration/img/export_audit_log_v13_4.png +++ /dev/null diff --git a/doc/administration/img/kroki_c4_diagram.png b/doc/administration/img/kroki_c4_diagram.png Binary files differnew file mode 100644 index 00000000000..55520e9e129 --- /dev/null +++ b/doc/administration/img/kroki_c4_diagram.png diff --git a/doc/administration/img/kroki_graphviz_diagram.png b/doc/administration/img/kroki_graphviz_diagram.png Binary files differnew file mode 100644 index 00000000000..77b30b54e07 --- /dev/null +++ b/doc/administration/img/kroki_graphviz_diagram.png diff --git a/doc/administration/img/kroki_nomnoml_diagram.png b/doc/administration/img/kroki_nomnoml_diagram.png Binary files differnew file mode 100644 index 00000000000..9e54a245e5a --- /dev/null +++ b/doc/administration/img/kroki_nomnoml_diagram.png diff --git a/doc/administration/img/kroki_plantuml_diagram.png b/doc/administration/img/kroki_plantuml_diagram.png Binary files differnew file mode 100644 index 00000000000..5d3f0628dfc --- /dev/null +++ b/doc/administration/img/kroki_plantuml_diagram.png diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index 20dce6d68df..7a49542f8bb 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -1,7 +1,7 @@ --- stage: Plan group: Project Management -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Incoming email @@ -21,10 +21,9 @@ GitLab has several features based on receiving incoming emails: ## Requirements -NOTE: **Note:** -It is **not** recommended to use an email address that receives or will receive any +It is **not** recommended to use an email address that receives any messages not intended for the GitLab instance. Any incoming emails not intended -for GitLab will receive a reject notice. +for GitLab receive a reject notice. Handling incoming emails requires an [IMAP](https://en.wikipedia.org/wiki/Internet_Message_Access_Protocol)-enabled email account. GitLab requires one of the following three strategies: @@ -38,14 +37,14 @@ Let's walk through each of these options. ### Email sub-addressing [Sub-addressing](https://en.wikipedia.org/wiki/Email_address#Sub-addressing) is -a mail server feature where any email to `user+arbitrary_tag@example.com` will end up +a mail server feature where any email to `user+arbitrary_tag@example.com` ends up in the mailbox for `user@example.com` . It is supported by providers such as Gmail, Google Apps, Yahoo! Mail, Outlook.com, and iCloud, as well as the [Postfix mail server](reply_by_email_postfix_setup.md), which you can run on-premises. Microsoft Exchange Server [does not support sub-addressing](#microsoft-exchange-server), and Microsoft Office 365 [does not support sub-addressing by default](#microsoft-office-365) -TIP: **Tip:** +NOTE: If your provider or server supports email sub-addressing, we recommend using it. A dedicated email address only supports Reply by Email functionality. A catch-all mailbox supports the same features as sub-addressing as of GitLab 11.7, @@ -86,7 +85,7 @@ To set up a basic Postfix mail server with IMAP access on Ubuntu, follow the ### Security concerns -CAUTION: **Caution:** +WARNING: Be careful when choosing the domain used for receiving incoming email. For example, suppose your top-level company domain is `hooli.com`. @@ -113,13 +112,13 @@ Alternatively, use a dedicated domain for GitLab email communications such as See GitLab issue [#30366](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/30366) for a real-world example of this exploit. -CAUTION: **Caution:** +WARNING: Use a mail server that has been configured to reduce spam. A Postfix mail server that is running on a default configuration, for example, -can result in abuse. All messages received on the configured mailbox will be processed -and messages that are not intended for the GitLab instance will receive a reject notice. -If the sender's address is spoofed, the reject notice will be delivered to the spoofed +can result in abuse. All messages received on the configured mailbox are processed +and messages that are not intended for the GitLab instance receive a reject notice. +If the sender's address is spoofed, the reject notice is delivered to the spoofed `FROM` address, which can cause the mail server's IP or domain to appear on a block list. @@ -254,7 +253,7 @@ incoming_email: Example configuration for Gmail/G Suite. Assumes mailbox `gitlab-incoming@gmail.com`. -NOTE: **Note:** +NOTE: `incoming_email_email` cannot be a Gmail alias account. Example for Omnibus installs: @@ -443,7 +442,7 @@ Example configurations for Microsoft Office 365 with IMAP enabled. ##### Sub-addressing mailbox -NOTE: **Note:** +NOTE: As of September 2020 sub-addressing support [has been added to Office 365](https://office365.uservoice.com/forums/273493-office-365-admin/suggestions/18612754-support-for-dynamic-email-aliases-in-office-36). This feature is not enabled by default, and must be enabled through PowerShell. @@ -452,9 +451,9 @@ This series of PowerShell commands enables [sub-addressing](#email-sub-addressin at the organization level in Office 365. This allows all mailboxes in the organization to receive sub-addressed mail: -NOTE: **Note:** -This series of commands will enable sub-addressing at the organization -level in Office 365. This will allow all mailboxes in the organization +NOTE: +This series of commands enables sub-addressing at the organization +level in Office 365. This allows all mailboxes in the organization to receive sub-addressed mail. ```powershell diff --git a/doc/administration/index.md b/doc/administration/index.md index 0aa94b86371..32c0b1b0712 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers" +stage: Enablement +group: Distribution +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" description: 'Learn how to install, configure, update, and maintain your GitLab instance.' --- @@ -51,26 +51,40 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Global user settings](user_settings.md): Configure instance-wide user permissions. - [Polling](polling.md): Configure how often the GitLab UI polls for updates. - [GitLab Pages configuration](pages/index.md): Enable and configure GitLab Pages. -- [GitLab Pages configuration for GitLab source installations](pages/source.md): Enable and configure GitLab Pages on [source installations](../install/installation.md#installation-from-source). +- [GitLab Pages configuration for GitLab source installations](pages/source.md): + Enable and configure GitLab Pages on [source installations](../install/installation.md#installation-from-source). - [Uploads administration](uploads.md): Configure GitLab uploads storage. - [Environment variables](environment_variables.md): Supported environment variables that can be used to override their default values to configure GitLab. -- [Plugins](file_hooks.md): With custom plugins, GitLab administrators can introduce custom integrations without modifying GitLab's source code. +- [Plugins](file_hooks.md): With custom plugins, GitLab administrators can + introduce custom integrations without modifying GitLab source code. - [Enforcing Terms of Service](../user/admin_area/settings/terms.md) - [Third party offers](../user/admin_area/settings/third_party_offers.md) -- [Compliance](compliance.md): A collection of features from across the application that you may configure to help ensure that your GitLab instance and DevOps workflow meet compliance standards. -- [Diff limits](../user/admin_area/diff_limits.md): Configure the diff rendering size limits of branch comparison pages. -- [Merge request diffs storage](merge_request_diffs.md): Configure merge requests diffs external storage. -- [Broadcast Messages](../user/admin_area/broadcast_messages.md): Send messages to GitLab users through the UI. -- [Elasticsearch](../integration/elasticsearch.md): Enable Elasticsearch to empower GitLab's Advanced Search. Useful when you deal with a huge amount of data. **(STARTER ONLY)** -- [External Classification Policy Authorization](../user/admin_area/settings/external_authorization.md) **(PREMIUM ONLY)** -- [Upload a license](../user/admin_area/license.md): Upload a license to unlock features that are in paid tiers of GitLab. **(STARTER ONLY)** -- [Admin Area](../user/admin_area/index.md): for self-managed instance-wide configuration and maintenance. -- [S/MIME Signing](smime_signing_email.md): how to sign all outgoing notification emails with S/MIME. -- [Enabling and disabling features flags](feature_flags.md): how to enable and disable GitLab features deployed behind feature flags. - -#### Customizing GitLab's appearance +- [Compliance](compliance.md): A collection of features from across the + application that you may configure to help ensure that your GitLab instance + and DevOps workflow meet compliance standards. +- [Diff limits](../user/admin_area/diff_limits.md): Configure the diff rendering + size limits of branch comparison pages. +- [Merge request diffs storage](merge_request_diffs.md): Configure merge + requests diffs external storage. +- [Broadcast Messages](../user/admin_area/broadcast_messages.md): Send messages + to GitLab users through the UI. +- [Elasticsearch](../integration/elasticsearch.md): Enable Elasticsearch to + empower Advanced Search. Useful when you deal with a huge amount of data. + **(STARTER ONLY)** +- [External Classification Policy Authorization](../user/admin_area/settings/external_authorization.md) + **(PREMIUM ONLY)** +- [Upload a license](../user/admin_area/license.md): Upload a license to unlock + features that are in paid tiers of GitLab. **(STARTER ONLY)** +- [Admin Area](../user/admin_area/index.md): for self-managed instance-wide + configuration and maintenance. +- [S/MIME Signing](smime_signing_email.md): how to sign all outgoing notification + emails with S/MIME. +- [Enabling and disabling features flags](feature_flags.md): how to enable and + disable GitLab features deployed behind feature flags. + +#### Customizing GitLab appearance - [Header logo](../user/admin_area/appearance.md#navigation-bar): Change the logo on all pages and email headers. - [Favicon](../user/admin_area/appearance.md#favicon): Change the default favicon to your own logo. @@ -104,7 +118,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Mattermost](https://docs.gitlab.com/omnibus/gitlab-mattermost/): Integrate with [Mattermost](https://mattermost.com), an open source, private cloud workplace for web messaging. - [PlantUML](integration/plantuml.md): Create simple diagrams in AsciiDoc and Markdown documents created in snippets, wikis, and repositories. -- [Web terminals](integration/terminal.md): Provide terminal access to your applications deployed to Kubernetes from within GitLab's CI/CD [environments](../ci/environments/index.md#web-terminals). +- [Web terminals](integration/terminal.md): Provide terminal access to your applications deployed to Kubernetes from within GitLab CI/CD [environments](../ci/environments/index.md#web-terminals). ## User settings and permissions @@ -118,7 +132,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - See also other [authentication](../topics/authentication/index.md#gitlab-administrators) topics (for example, enforcing 2FA). - [Email users](../tools/email.md): Email GitLab users from within GitLab. **(STARTER ONLY)** - [User Cohorts](../user/admin_area/analytics/user_cohorts.md): Display the monthly cohorts of new users and their activities over time. -- [Audit logs and events](audit_events.md): View the changes made within the GitLab server for: +- [Audit events](audit_events.md): View the changes made within the GitLab server for: - Groups and projects. **(STARTER)** - Instances. **(PREMIUM ONLY)** - [Auditor users](auditor_users.md): Users with read-only access to all projects, groups, and other resources on the GitLab instance. **(PREMIUM ONLY)** @@ -134,8 +148,8 @@ Learn how to install, configure, update, and maintain your GitLab instance. ## Project settings - [Issue closing pattern](issue_closing_pattern.md): Customize how to close an issue from commit messages. -- [Gitaly](gitaly/index.md): Configuring Gitaly, GitLab's Git repository storage service. -- [Default labels](../user/admin_area/labels.md): Create labels that will be automatically added to every new project. +- [Gitaly](gitaly/index.md): Configuring Gitaly, the Git repository storage service for GitLab. +- [Default labels](../user/admin_area/labels.md): Create labels that are automatically added to every new project. - [Restrict the use of public or internal projects](../public_access/public_access.md#restricting-the-use-of-public-or-internal-projects): Restrict the use of visibility levels for users when they create a project or a snippet. - [Custom project templates](../user/admin_area/custom_project_templates.md): Configure a set of projects to be used as custom templates when creating a new project. **(PREMIUM ONLY)** @@ -186,7 +200,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Monitoring GitLab](monitoring/index.md): - [Monitoring uptime](../user/admin_area/monitoring/health_check.md): Check the server status using the health check endpoint. - [IP whitelist](monitoring/ip_whitelist.md): Monitor endpoints that provide health check information when probed. - - [Monitoring GitHub imports](monitoring/github_imports.md): GitLab's GitHub Importer displays Prometheus metrics to monitor the health and progress of the importer. + - [Monitoring GitHub imports](monitoring/github_imports.md): The GitLab GitHub Importer displays Prometheus metrics to monitor the health and progress of the importer. ### Performance Monitoring @@ -199,7 +213,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. ## Analytics -- [Pseudonymizer](pseudonymizer.md): Export data from GitLab's database to CSV files in a secure way. **(ULTIMATE)** +- [Pseudonymizer](pseudonymizer.md): Export data from a GitLab database to CSV files in a secure way. **(ULTIMATE)** ## Troubleshooting @@ -219,7 +233,7 @@ information useful for troubleshooting, but if you are experiencing trouble with GitLab instance, you should check your [support options](https://about.gitlab.com/support/) before referring to these documents. -CAUTION: **Warning:** +WARNING: Using the commands listed in the documentation below could result in data loss or other damage to a GitLab instance, and should only be used by experienced administrators who are aware of the risks. diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index ec56a59fdc2..0eede5f3ca5 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -165,7 +165,7 @@ There is a limit when embedding metrics in GFM for performance reasons. On GitLab.com, the [maximum number of webhooks and their size](../user/gitlab_com/index.md#webhooks) per project, and per group, is limited. -To set this limit on a self-managed installation, run the following in the +To set this limit on a self-managed installation, where the default is `100` project webhooks and `50` group webhooks, run the following in the [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): ```ruby @@ -173,7 +173,7 @@ To set this limit on a self-managed installation, run the following in the # Plan.default.create_limits! # For project webhooks -Plan.default.actual_limits.update!(project_hooks: 100) +Plan.default.actual_limits.update!(project_hooks: 200) # For group webhooks Plan.default.actual_limits.update!(group_hooks: 100) @@ -181,6 +181,23 @@ Plan.default.actual_limits.update!(group_hooks: 100) Set the limit to `0` to disable it. +## Pull Mirroring Interval + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/237891) in GitLab 13.7. + +The [minimum time between pull refreshes](../user/project/repository/repository_mirroring.md) +defaults to 300 seconds (5 minutes). + +To change this limit on a self-managed installation, run the following in the +[GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): + +```ruby +# If limits don't exist for the default plan, you can create one with: +# Plan.default.create_limits! + +Plan.default.actual_limits.update!(pull_mirror_interval_seconds: 200) +``` + ## Incoming emails from auto-responders > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30327) in GitLab 12.4. @@ -235,8 +252,8 @@ If a new pipeline would cause the total number of jobs to exceed the limit, the will fail with a `job_activity_limit_exceeded` error. - On GitLab.com different [limits are defined per plan](../user/gitlab_com/index.md#gitlab-cicd) and they affect all projects under that plan. -- On [GitLab Starter](https://about.gitlab.com/pricing/#self-managed) tier or higher self-managed installations, this limit is defined for the `default` plan that affects all projects. - This limit is disabled by default. +- On [GitLab Starter](https://about.gitlab.com/pricing/#self-managed) tier or higher self-managed installations, this limit is defined under a `default` plan that affects all projects. + This limit is disabled (`0`) by default. To set this limit on a self-managed installation, run the following in the [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): @@ -250,6 +267,29 @@ Plan.default.actual_limits.update!(ci_active_jobs: 500) Set the limit to `0` to disable it. +### Maximum number of deployment jobs in a pipeline + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/46931) in GitLab 13.7. + +You can limit the maximum number of deployment jobs in a pipeline. A deployment is +any job with an [`environment`](../ci/environments/index.md) specified. The number +of deployments in a pipeline is checked at pipeline creation. Pipelines that have +too many deployments fail with a `deployments_limit_exceeded` error. + +The default limit is 500 for all [self-managed and GitLab.com plans](https://about.gitlab.com/pricing/). + +To change the limit on a self-managed installation, change the `default` plan's limit with the following +[GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session) command: + +```ruby +# If limits don't exist for the default plan, you can create one with: +# Plan.default.create_limits! + +Plan.default.actual_limits.update!(ci_pipeline_deployments: 500) +``` + +Set the limit to `0` to disable it. + ### Number of CI/CD subscriptions to a project > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9045) in GitLab 12.9. @@ -261,7 +301,7 @@ If a new subscription would cause the total number of subscription to exceed the limit, the subscription will be considered invalid. - On GitLab.com different [limits are defined per plan](../user/gitlab_com/index.md#gitlab-cicd) and they affect all projects under that plan. -- On [GitLab Starter](https://about.gitlab.com/pricing/#self-managed) tier or higher self-managed installations, this limit is defined for the `default` plan that affects all projects. +- On [GitLab Starter](https://about.gitlab.com/pricing/#self-managed) tier or higher self-managed installations, this limit is defined under a `default` plan that affects all projects. By default, there is a limit of `2` subscriptions. To set this limit on a self-managed installation, run the following in the [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): @@ -285,8 +325,8 @@ On GitLab.com, different limits are [defined per plan](../user/gitlab_com/index. and they affect all projects under that plan. On self-managed instances ([GitLab Starter](https://about.gitlab.com/pricing/#self-managed) -or higher tiers), this limit is defined for the `default` plan that affects all -projects. By default, there is no limit. +or higher tiers), this limit is defined under a `default` plan that affects all +projects. By default, there is a limit of `10` pipeline schedules. To set this limit on a self-managed installation, run the following in the [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): @@ -459,8 +499,8 @@ as the overall index size. This value defaults to `1024 KiB` (1 MiB) as any text files larger than this likely aren't meant to be read by humans. You must set a limit, as unlimited file sizes aren't supported. Setting this -value to be greater than the amount of memory on GitLab's Sidekiq nodes causes -GitLab's Sidekiq nodes to run out of memory, as they will pre-allocate this +value to be greater than the amount of memory on GitLab Sidekiq nodes causes +the GitLab Sidekiq nodes to run out of memory, as they will pre-allocate this amount of memory during indexing. ### Maximum field length @@ -527,12 +567,12 @@ More information can be found in the [Push event activities limit and bulk push On GitLab.com, the maximum file size for a package that's uploaded to the [GitLab Package Registry](../user/packages/package_registry/index.md) varies by format: -- Conan: 3GB +- Conan: 5GB - Generic: 5GB -- Maven: 3GB -- NPM: 500MB -- NuGet: 500MB -- PyPI: 3GB +- Maven: 5GB +- NPM: 5GB +- NuGet: 5GB +- PyPI: 5GB To set this limit on a self-managed installation, run the following in the [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): diff --git a/doc/administration/instance_review.md b/doc/administration/instance_review.md index b00586b281b..8f33ddf3ca5 100644 --- a/doc/administration/instance_review.md +++ b/doc/administration/instance_review.md @@ -1,7 +1,7 @@ --- stage: Growth group: Conversion -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Instance Review @@ -21,5 +21,9 @@ you qualify for a free Instance Review. 1. GitLab redirects you to a form with prefilled data obtained from your instance. 1. Click **Submit** to see the initial report. -A GitLab team member will contact you for further review, to provide suggestions -that will help you improve your usage of GitLab. +<!-- vale gitlab.FutureTense = NO --> + +You will be contacted by a GitLab team member for further review, to provide suggestions +intended to help you improve your usage of GitLab. + +<!-- vale gitlab.FutureTense = YES --> diff --git a/doc/administration/integration/kroki.md b/doc/administration/integration/kroki.md new file mode 100644 index 00000000000..dead5640873 --- /dev/null +++ b/doc/administration/integration/kroki.md @@ -0,0 +1,162 @@ +# Kroki diagrams **(CORE ONLY)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/241744) in GitLab 13.7. + +When [Kroki](https://kroki.io) integration is enabled and configured in +GitLab you can use it to create diagrams in AsciiDoc and Markdown documents. + +## Kroki Server + +When Kroki is enabled, GitLab sends diagrams to an instance of Kroki to display them as images. +You can use the free public cloud instance `https://kroki.io` or you can [install Kroki](https://docs.kroki.io/kroki/setup/install/) +on your own infrastructure. +After you've installed Kroki, make sure to update the server URL to point to your instance. + +### Docker + +With Docker, run a container like this: + +```shell +docker run -d --name kroki -p 8080:8000 yuzutech/kroki +``` + +The **Kroki URL** is the hostname of the server running the container. + +The [`yuzutech/kroki`](https://hub.docker.com/r/yuzutech/kroki) image contains the following diagrams libraries out-of-the-box: + +- [Bytefield](https://bytefield-svg.deepsymmetry.org/) +- [Ditaa](http://ditaa.sourceforge.net) +- [Erd](https://github.com/BurntSushi/erd) +- [GraphViz](https://www.graphviz.org/) +- [Nomnoml](https://github.com/skanaar/nomnoml) +- [PlantUML](https://github.com/plantuml/plantuml) + - [C4 model](https://github.com/RicardoNiepel/C4-PlantUML) (with PlantUML) +- [Svgbob](https://github.com/ivanceras/svgbob) +- [UMlet](https://github.com/umlet/umlet) +- [Vega](https://github.com/vega/vega) +- [Vega-Lite](https://github.com/vega/vega-lite) +- [WaveDrom](https://wavedrom.com/) + +If you want to use additional diagram libraries, +read the [Kroki installation](https://docs.kroki.io/kroki/setup/install/#_images) to learn how to start Kroki companion containers. + +## Enable Kroki in GitLab + +You need to enable Kroki integration from Settings under Admin Area. +To do that, log in with an administrator account and follow these steps: + +1. Select the Admin Area (**{admin}**) icon. +1. Navigate to **Settings > General**. +1. Expand the **Kroki** section. +1. Select **Enable Kroki** checkbox. +1. Enter the **Kroki URL**. + +## Create diagrams + +With Kroki integration enabled and configured, you can start adding diagrams to +your AsciiDoc or Markdown documentation using delimited blocks: + +- **Markdown** + + ````markdown + ```plantuml + Bob -> Alice : hello + Alice -> Bob : hi + ``` + ```` + +- **AsciiDoc** + + ```plaintext + [plantuml] + .... + Bob->Alice : hello + Alice -> Bob : hi + .... + ``` + +The above blocks are converted to an HTML image tag with source pointing to the +Kroki instance. If the Kroki server is correctly configured, this should +render a nice diagram instead of the block: + + + +Kroki supports more than a dozen diagram libraries. Here's a few examples: + +**GraphViz** + +```plaintext +[graphviz] +.... +digraph finite_state_machine { + rankdir=LR; + node [shape = doublecircle]; LR_0 LR_3 LR_4 LR_8; + node [shape = circle]; + LR_0 -> LR_2 [ label = "SS(B)" ]; + LR_0 -> LR_1 [ label = "SS(S)" ]; + LR_1 -> LR_3 [ label = "S($end)" ]; + LR_2 -> LR_6 [ label = "SS(b)" ]; + LR_2 -> LR_5 [ label = "SS(a)" ]; + LR_2 -> LR_4 [ label = "S(A)" ]; + LR_5 -> LR_7 [ label = "S(b)" ]; + LR_5 -> LR_5 [ label = "S(a)" ]; + LR_6 -> LR_6 [ label = "S(b)" ]; + LR_6 -> LR_5 [ label = "S(a)" ]; + LR_7 -> LR_8 [ label = "S(b)" ]; + LR_7 -> LR_5 [ label = "S(a)" ]; + LR_8 -> LR_6 [ label = "S(b)" ]; + LR_8 -> LR_5 [ label = "S(a)" ]; +} +.... +``` + + + +**C4 (based on PlantUML)** + +```plaintext +[c4plantuml] +.... +@startuml +!include C4_Context.puml + +title System Context diagram for Internet Banking System + +Person(customer, "Banking Customer", "A customer of the bank, with personal bank accounts.") +System(banking_system, "Internet Banking System", "Allows customers to check their accounts.") + +System_Ext(mail_system, "E-mail system", "The internal Microsoft Exchange e-mail system.") +System_Ext(mainframe, "Mainframe Banking System", "Stores all of the core banking information.") + +Rel(customer, banking_system, "Uses") +Rel_Back(customer, mail_system, "Sends e-mails to") +Rel_Neighbor(banking_system, mail_system, "Sends e-mails", "SMTP") +Rel(banking_system, mainframe, "Uses") +@enduml +.... +``` + + + +**Nomnoml** + +```plaintext +[nomnoml] +.... +[Pirate|eyeCount: Int|raid();pillage()| + [beard]--[parrot] + [beard]-:>[foul mouth] +] + +[<abstract>Marauder]<:--[Pirate] +[Pirate]- 0..7[mischief] +[jollyness]->[Pirate] +[jollyness]->[rum] +[jollyness]->[singing] +[Pirate]-> *[rum|tastiness: Int|swig()] +[Pirate]->[singing] +[singing]<->[rum] +.... +``` + + diff --git a/doc/administration/integration/plantuml.md b/doc/administration/integration/plantuml.md index 5bdea9d8843..a7458999aaa 100644 --- a/doc/administration/integration/plantuml.md +++ b/doc/administration/integration/plantuml.md @@ -1,7 +1,7 @@ --- stage: Create group: Source Code -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers" +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" type: reference, howto --- @@ -137,7 +137,7 @@ that, login with an Admin account and do following: - Check **Enable PlantUML** checkbox. - Set the PlantUML instance as `https://gitlab.example.com/-/plantuml/`. -NOTE: **Note:** +NOTE: If you are using a PlantUML server running v1.2020.9 and above (for example, [plantuml.com](https://plantuml.com)), set the `PLANTUML_ENCODING` environment variable to enable the `deflate` compression. On Omnibus, diff --git a/doc/administration/integration/terminal.md b/doc/administration/integration/terminal.md index 3c53535d856..f4c242b6e72 100644 --- a/doc/administration/integration/terminal.md +++ b/doc/administration/integration/terminal.md @@ -1,14 +1,14 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Create +group: Ecosystem +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Web terminals > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/7690) in GitLab 8.15. -NOTE: **Note:** +NOTE: Only project maintainers and owners can access web terminals. With the introduction of the [Kubernetes integration](../../user/project/clusters/index.md), @@ -27,7 +27,7 @@ In brief: - When a user navigates to the terminal page for an environment, they are served a JavaScript application that opens a WebSocket connection back to GitLab. - The WebSocket is handled in [Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse), - rather than the Rails application server. + rather than the Rails application server. - Workhorse queries Rails for connection details and user permissions. Rails queries Kubernetes for them in the background using [Sidekiq](../troubleshooting/sidekiq.md). - Workhorse acts as a proxy server between the user's browser and the Kubernetes @@ -44,14 +44,14 @@ everything protected with authorization guards. This is described in more detail below. - Interactive web terminals are completely disabled unless [`[session_server]`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-session_server-section) is configured. -- Every time the runner starts, it will generate an `x509` certificate that will be used for a `wss` (Web Socket Secure) connection. +- Every time the runner starts, it generates an `x509` certificate that is used for a `wss` (Web Socket Secure) connection. - For every created job, a random URL is generated which is discarded at the end of the job. This URL is used to establish a web socket connection. The URL for the session is in the format `(IP|HOST):PORT/session/$SOME_HASH`, where the `IP/HOST` and `PORT` are the configured [`listen_address`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-session_server-section). - Every session URL that is created has an authorization header that needs to be sent, to establish a `wss` connection. - The session URL is not exposed to the users in any way. GitLab holds all the state internally and proxies accordingly. ## Enabling and disabling terminal support -NOTE: **Note:** +NOTE: AWS Elastic Load Balancers (ELBs) do not support web sockets. AWS Application Load Balancers (ALBs) must be used if you want web terminals to work. See [AWS Elastic Load Balancing Product Comparison](https://aws.amazon.com/elasticloadbalancing/features/#compare) @@ -72,7 +72,7 @@ guides document the necessary steps for a selection of popular reverse proxies: - [HAProxy](https://www.haproxy.com/blog/websockets-load-balancing-with-haproxy/) - [Varnish](https://varnish-cache.org/docs/4.1/users-guide/vcl-example-websockets.html) -Workhorse won't let WebSocket requests through to non-WebSocket endpoints, so +Workhorse doesn't let WebSocket requests through to non-WebSocket endpoints, so it's safe to enable support for these headers globally. If you'd rather had a narrower set of rules, you can restrict it to URLs ending with `/terminal.ws` (although this may still have a few false positives). @@ -85,7 +85,7 @@ document for more details. If you'd like to disable web terminal support in GitLab, just stop passing the `Connection` and `Upgrade` hop-by-hop headers in the *first* HTTP reverse -proxy in the chain. For most users, this will be the NGINX server bundled with +proxy in the chain. For most users, this is the NGINX server bundled with Omnibus GitLab, in which case, you need to: - Find the `nginx['proxy_set_headers']` section of your `gitlab.rb` file @@ -95,9 +95,9 @@ Omnibus GitLab, in which case, you need to: For your own load balancer, just reverse the configuration changes recommended by the above guides. -When these headers are not passed through, Workhorse will return a +When these headers are not passed through, Workhorse returns a `400 Bad Request` response to users attempting to use a web terminal. In turn, -they will receive a `Connection failed` message. +they receive a `Connection failed` message. ## Limiting WebSocket connection time diff --git a/doc/administration/invalidate_markdown_cache.md b/doc/administration/invalidate_markdown_cache.md index ac43d0eae63..75bee6e0c9a 100644 --- a/doc/administration/invalidate_markdown_cache.md +++ b/doc/administration/invalidate_markdown_cache.md @@ -1,7 +1,7 @@ --- stage: Create group: Source Code -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers" +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" type: reference --- @@ -19,5 +19,5 @@ be done by [changing the application settings through the API](../api/settings.md#change-application-settings): ```shell -curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/application/settings?local_markdown_version=<increased_number> +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/application/settings?local_markdown_version=<increased_number>" ``` diff --git a/doc/administration/issue_closing_pattern.md b/doc/administration/issue_closing_pattern.md index 7abe0f725f2..6a5b90ae963 100644 --- a/doc/administration/issue_closing_pattern.md +++ b/doc/administration/issue_closing_pattern.md @@ -1,13 +1,13 @@ --- stage: Create group: Source Code -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers" +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" type: reference --- # Issue closing pattern **(CORE ONLY)** -NOTE: **Note:** +NOTE: This is the administration documentation. There is a separate [user documentation](../user/project/issues/managing_issues.md#closing-issues-automatically) on issue closing pattern. @@ -23,7 +23,7 @@ is installed on. The default pattern can be located in [`gitlab.yml.example`](https://gitlab.com/gitlab-org/gitlab/blob/master/config/gitlab.yml.example) under the "Automatic issue closing" section. -TIP: **Tip:** +NOTE: You are advised to use <https://rubular.com> to test the issue closing pattern. Because Rubular doesn't understand `%{issue_ref}`, you can replace this by `#\d+` when testing your patterns, which matches only local issue references like `#123`. diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index a010903013e..5d07e7d1b26 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -1,7 +1,7 @@ --- stage: Verify group: Continuous Integration -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference, howto --- @@ -103,7 +103,7 @@ If you configure GitLab to store artifacts on object storage, you may also want [eliminate local disk usage for job logs](job_logs.md#prevent-local-disk-usage). In both cases, job logs are archived and moved to object storage when the job completes. -DANGER: **Warning:** +WARNING: In a multi-server setup you must use one of the options to [eliminate local disk usage for job logs](job_logs.md#prevent-local-disk-usage), or job logs could be lost. @@ -111,21 +111,21 @@ In a multi-server setup you must use one of the options to #### Object Storage Settings -NOTE: **Note:** +NOTE: In GitLab 13.2 and later, we recommend using the [consolidated object storage settings](object_storage.md#consolidated-object-storage-configuration). This section describes the earlier configuration format. For source installations the following settings are nested under `artifacts:` and then `object_store:`. On Omnibus GitLab installs they are prefixed by `artifacts_object_store_`. -| Setting | Description | Default | -|---------|-------------|---------| -| `enabled` | Enable/disable object storage | `false` | -| `remote_directory` | The bucket name where Artifacts will be stored| | -| `direct_upload` | Set to `true` to enable direct upload of Artifacts without the need of local shared storage. Option may be removed once we decide to support only single storage for all files. | `false` | -| `background_upload` | Set to `false` to disable automatic upload. Option may be removed once upload is direct to S3 | `true` | -| `proxy_download` | Set to `true` to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` | -| `connection` | Various connection options described below | | +| Setting | Default | Description | +|---------------------|---------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `enabled` | `false` | Enable/disable object storage | +| `remote_directory` | | The bucket name where Artifacts are stored | +| `direct_upload` | `false` | Set to `true` to enable direct upload of Artifacts without the need of local shared storage. Option may be removed once we decide to support only single storage for all files. | +| `background_upload` | `true` | Set to `false` to disable automatic upload. Option may be removed once upload is direct to S3 | +| `proxy_download` | `false` | Set to `true` to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data. | +| `connection` | | Various connection options described below | #### Connection settings @@ -190,7 +190,7 @@ _The artifacts are stored by default in In some cases, you may need to run the [orphan artifact file cleanup Rake task](../raketasks/cleanup.md#remove-orphan-artifact-files) to clean up orphaned artifacts. -CAUTION: **Caution:** +WARNING: JUnit test report artifact (`junit.xml.gz`) migration [was not supported until GitLab 12.8](https://gitlab.com/gitlab-org/gitlab/-/issues/27698#note_317190991) by the `gitlab:artifacts:migrate` script. @@ -243,7 +243,7 @@ _The artifacts are stored by default in In some cases, you may need to run the [orphan artifact file cleanup Rake task](../raketasks/cleanup.md#remove-orphan-artifact-files) to clean up orphaned artifacts. -CAUTION: **Caution:** +WARNING: JUnit test report artifact (`junit.xml.gz`) migration [was not supported until GitLab 12.8](https://gitlab.com/gitlab-org/gitlab/-/issues/27698#note_317190991) by the `gitlab:artifacts:migrate` script. @@ -336,7 +336,7 @@ To migrate back to local storage: If [`artifacts:expire_in`](../ci/yaml/README.md#artifactsexpire_in) is used to set an expiry for the artifacts, they are marked for deletion right after that date passes. -Otherwise, they will expire per the [default artifacts expiration setting](../user/admin_area/settings/continuous_integration.md). +Otherwise, they expire per the [default artifacts expiration setting](../user/admin_area/settings/continuous_integration.md). Artifacts are cleaned up by the `expire_build_artifacts_worker` cron job which Sidekiq runs every hour at 50 minutes (`50 * * * *`). @@ -367,7 +367,7 @@ steps below. 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. -If the `expire` directive is not set explicitly in your pipeline, artifacts will expire per the +If the `expire` directive is not set explicitly in your pipeline, artifacts expire per the default artifacts expiration setting, which you can find in the [CI/CD Admin settings](../user/admin_area/settings/continuous_integration.md). ## Validation for dependencies @@ -444,7 +444,7 @@ reasons are: - The number of jobs run, and hence artifacts generated, is higher than expected. - Job logs are larger than expected, and have accumulated over time. -In these and other cases, you'll need to identify the projects most responsible +In these and other cases, identify the projects most responsible for disk space usage, figure out what types of artifacts are using the most space, and in some cases, manually delete job artifacts to reclaim disk space. @@ -481,7 +481,7 @@ the number you want. #### Delete job artifacts from jobs completed before a specific date -CAUTION: **Caution:** +WARNING: These commands remove data permanently from the database and from disk. We highly recommend running them only under the guidance of a Support Engineer, or running them in a test environment with a backup of the instance ready to be @@ -507,8 +507,8 @@ If you need to manually remove job artifacts associated with multiple jobs while 1. Delete job artifacts older than a specific date: - NOTE: **Note:** - This step will also erase artifacts that users have chosen to + NOTE: + This step also erases artifacts that users have chosen to ["keep"](../ci/pipelines/job_artifacts.md#browsing-artifacts). ```ruby @@ -528,7 +528,7 @@ If you need to manually remove job artifacts associated with multiple jobs while #### Delete job artifacts and logs from jobs completed before a specific date -CAUTION: **Caution:** +WARNING: These commands remove data permanently from the database and from disk. We highly recommend running them only under the guidance of a Support Engineer, or running them in a test environment with a backup of the instance ready to be @@ -552,7 +552,7 @@ If you need to manually remove **all** job artifacts associated with multiple jo builds_with_artifacts = Ci::Build.with_existing_job_artifacts(Ci::JobArtifact.trace) ``` -1. Select the user which will be mentioned in the web UI as erasing the job: +1. Select the user which is mentioned in the web UI as erasing the job: ```ruby admin_user = User.find_by(username: 'username') diff --git a/doc/administration/job_logs.md b/doc/administration/job_logs.md index 23343d309de..7af167de4ad 100644 --- a/doc/administration/job_logs.md +++ b/doc/administration/job_logs.md @@ -1,7 +1,7 @@ --- stage: Verify group: Runner -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -30,7 +30,7 @@ it would be `/home/git/gitlab`. ## Changing the job logs local location -To change the location where the job logs will be stored, follow the steps below. +To change the location where the job logs are stored, follow the steps below. **In Omnibus installations:** @@ -117,12 +117,12 @@ you can do so using one of the following options: There isn't a way to automatically expire old job logs, but it's safe to remove them if they're taking up too much space. If you remove the logs manually, the -job output in the UI will be empty. +job output in the UI is empty. For example, to delete all job logs older than 60 days, run the following from a shell in your GitLab instance: -DANGER: **Warning:** -This command will permanently delete the log files and is irreversible. +WARNING: +This command permanently deletes the log files and is irreversible. ```shell find /var/opt/gitlab/gitlab-rails/shared/artifacts -name "job.log" -mtime +60 -delete @@ -132,7 +132,7 @@ find /var/opt/gitlab/gitlab-rails/shared/artifacts -name "job.log" -mtime +60 -d > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/18169) in GitLab 10.4. -NOTE: **Note:** +NOTE: This beta feature is off by default. See below for how to [enable or disable](#enabling-incremental-logging) it. By combining the process with object storage settings, we can completely bypass @@ -144,7 +144,7 @@ with one change: _the stored path of the first two phases is different_. This in log architecture stores chunks of logs in Redis and a persistent store (object storage or database) instead of file storage. Redis is used as first-class storage, and it stores up-to 128KB of data. After the full chunk is sent, it is flushed to a persistent store, either object storage (temporary directory) or database. -After a while, the data in Redis and a persistent store will be archived to [object storage](#uploading-logs-to-object-storage). +After a while, the data in Redis and a persistent store is archived to [object storage](#uploading-logs-to-object-storage). The data are stored in the following Redis namespace: `Gitlab::Redis::SharedState`. @@ -184,10 +184,10 @@ Feature.enabled?(:ci_enable_live_trace) Feature.enable(:ci_enable_live_trace) ``` -NOTE: **Note:** -The transition period will be handled gracefully. Upcoming logs will be -generated with the incremental architecture, and on-going logs will stay with the -legacy architecture, which means that on-going logs won't be forcibly +NOTE: +The transition period is handled gracefully. Upcoming logs are +generated with the incremental architecture, and on-going logs stay with the +legacy architecture, which means that on-going logs aren't forcibly re-generated with the incremental architecture. **To disable incremental logging (trace):** @@ -196,10 +196,10 @@ re-generated with the incremental architecture. Feature.disable('ci_enable_live_trace') ``` -NOTE: **Note:** -The transition period will be handled gracefully. Upcoming logs will be generated -with the legacy architecture, and on-going incremental logs will stay with the incremental -architecture, which means that on-going incremental logs won't be forcibly re-generated +NOTE: +The transition period is handled gracefully. Upcoming logs are generated +with the legacy architecture, and on-going incremental logs stay with the incremental +architecture, which means that on-going incremental logs aren't forcibly re-generated with the legacy architecture. ### Potential implications @@ -209,13 +209,13 @@ In some cases, having data stored on Redis could incur data loss: 1. **Case 1: When all data in Redis are accidentally flushed** - On going incremental logs could be recovered by re-sending logs (this is supported by all versions of GitLab Runner). - - Finished jobs which have not archived incremental logs will lose the last part + - Finished jobs which have not archived incremental logs lose the last part (~128KB) of log data. 1. **Case 2: When Sidekiq workers fail to archive (e.g., there was a bug that prevents archiving process, Sidekiq inconsistency, etc.)** - - Currently all log data in Redis will be deleted after one week. If the - Sidekiq workers can't finish by the expiry date, the part of log data will be lost. + - All log data in Redis is deleted after one week. If the + Sidekiq workers can't finish by the expiry date, the part of log data is lost. Another issue that might arise is that it could consume all memory on the Redis instance. If the number of jobs is 1000, 128MB (128KB * 1000) is consumed. diff --git a/doc/administration/job_traces.md b/doc/administration/job_traces.md index d0b346a931e..2dbcf5d6705 100644 --- a/doc/administration/job_traces.md +++ b/doc/administration/job_traces.md @@ -3,3 +3,6 @@ redirect_to: 'job_logs.md' --- This document was moved to [another location](job_logs.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/lfs/index.md b/doc/administration/lfs/index.md index a360542ac44..bc349536a0c 100644 --- a/doc/administration/lfs/index.md +++ b/doc/administration/lfs/index.md @@ -1,7 +1,7 @@ --- stage: Create group: Source Code -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers" +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" type: reference, howto disqus_identifier: 'https://docs.gitlab.com/ee/workflow/lfs/lfs_administration.html' --- @@ -71,7 +71,7 @@ GitLab provides two different options for the uploading mechanism: "Direct uploa [Read more about using object storage with GitLab](../object_storage.md). -NOTE: **Note:** +NOTE: In GitLab 13.2 and later, we recommend using the [consolidated object storage settings](../object_storage.md#consolidated-object-storage-configuration). This section describes the earlier configuration format. diff --git a/doc/administration/lfs/lfs_administration.md b/doc/administration/lfs/lfs_administration.md index 7ace0ec5a93..a275efce5bb 100644 --- a/doc/administration/lfs/lfs_administration.md +++ b/doc/administration/lfs/lfs_administration.md @@ -3,3 +3,6 @@ redirect_to: 'index.md' --- This document was moved to [another location](index.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/lfs/manage_large_binaries_with_git_lfs.md b/doc/administration/lfs/manage_large_binaries_with_git_lfs.md index 4656bccf5e6..69c401acb86 100644 --- a/doc/administration/lfs/manage_large_binaries_with_git_lfs.md +++ b/doc/administration/lfs/manage_large_binaries_with_git_lfs.md @@ -3,3 +3,6 @@ redirect_to: '../../topics/git/lfs/index.md' --- This document was moved to [another location](../../topics/git/lfs/index.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/lfs/migrate_from_git_annex_to_git_lfs.md b/doc/administration/lfs/migrate_from_git_annex_to_git_lfs.md index dd98a50e353..16095c1dbf6 100644 --- a/doc/administration/lfs/migrate_from_git_annex_to_git_lfs.md +++ b/doc/administration/lfs/migrate_from_git_annex_to_git_lfs.md @@ -3,3 +3,6 @@ redirect_to: '../../topics/git/lfs/migrate_from_git_annex_to_git_lfs.md' --- This document was moved to [another location](../../topics/git/lfs/migrate_from_git_annex_to_git_lfs.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/libravatar.md b/doc/administration/libravatar.md index a92e6fade03..552dc7095e2 100644 --- a/doc/administration/libravatar.md +++ b/doc/administration/libravatar.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -41,7 +41,7 @@ the configuration options as follows: ### Your own Libravatar server If you are [running your own Libravatar service](https://wiki.libravatar.org/running_your_own/), -the URL will be different in the configuration, but you must provide the same +the URL is different in the configuration, but you must provide the same placeholders so GitLab can parse the URL correctly. For example, you host a service on `http://libravatar.example.com` and the diff --git a/doc/administration/load_balancer.md b/doc/administration/load_balancer.md index 410381ff2b0..ae96989a188 100644 --- a/doc/administration/load_balancer.md +++ b/doc/administration/load_balancer.md @@ -1,23 +1,23 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- # Load Balancer for multi-node GitLab -In an multi-node GitLab configuration, you will need a load balancer to route +In an multi-node GitLab configuration, you need a load balancer to route traffic to the application servers. The specifics on which load balancer to use or the exact configuration is beyond the scope of GitLab documentation. We hope that if you're managing HA systems like GitLab you have a load balancer of choice already. Some examples including HAProxy (open-source), F5 Big-IP LTM, -and Citrix Net Scaler. This documentation will outline what ports and protocols +and Citrix Net Scaler. This documentation outlines what ports and protocols you need to use with GitLab. ## SSL -How will you handle SSL in your multi-node environment? There are several different +How do you want to handle SSL in your multi-node environment? There are several different options: - Each application node terminates SSL @@ -29,8 +29,8 @@ options: ### Application nodes terminate SSL Configure your load balancer(s) to pass connections on port 443 as 'TCP' rather -than 'HTTP(S)' protocol. This will pass the connection to the application nodes -NGINX service untouched. NGINX will have the SSL certificate and listen on port 443. +than 'HTTP(S)' protocol. This passes the connection to the application nodes +NGINX service untouched. NGINX has the SSL certificate and listen on port 443. See [NGINX HTTPS documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) for details on managing SSL certificates and configuring NGINX. @@ -38,10 +38,10 @@ for details on managing SSL certificates and configuring NGINX. ### Load Balancer(s) terminate SSL without backend SSL Configure your load balancer(s) to use the 'HTTP(S)' protocol rather than 'TCP'. -The load balancer(s) will then be responsible for managing SSL certificates and +The load balancer(s) is be responsible for managing SSL certificates and terminating SSL. -Since communication between the load balancer(s) and GitLab will not be secure, +Since communication between the load balancer(s) and GitLab isn't secure, there is some additional configuration needed. See [NGINX Proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#supporting-proxied-ssl) for details. @@ -49,12 +49,12 @@ for details. ### Load Balancer(s) terminate SSL with backend SSL Configure your load balancer(s) to use the 'HTTP(S)' protocol rather than 'TCP'. -The load balancer(s) will be responsible for managing SSL certificates that -end users will see. +The load balancer(s) is responsible for managing SSL certificates that +end users see. -Traffic will also be secure between the load balancer(s) and NGINX in this +Traffic is secure between the load balancer(s) and NGINX in this scenario. There is no need to add configuration for proxied SSL since the -connection will be secure all the way. However, configuration will need to be +connection is secure all the way. However, configuration must be added to GitLab to configure SSL certificates. See [NGINX HTTPS documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) for details on managing SSL certificates and configuring NGINX. @@ -75,13 +75,13 @@ for details on managing SSL certificates and configuring NGINX. to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the [web terminal](integration/terminal.md) integration guide for more details. -- (*2*): When using HTTPS protocol for port 443, you will need to add an SSL +- (*2*): When using HTTPS protocol for port 443, you must add an SSL certificate to the load balancers. If you wish to terminate SSL at the GitLab application server instead, use TCP protocol. ### GitLab Pages Ports -If you're using GitLab Pages with custom domain support you will need some +If you're using GitLab Pages with custom domain support you need some additional port configurations. GitLab Pages requires a separate virtual IP address. Configure DNS to point the `pages_external_url` from `/etc/gitlab/gitlab.rb` at the new virtual IP address. See the @@ -103,7 +103,7 @@ GitLab Pages requires a separate virtual IP address. Configure DNS to point the Some organizations have policies against opening SSH port 22. In this case, it may be helpful to configure an alternate SSH hostname that allows users -to use SSH on port 443. An alternate SSH hostname will require a new virtual IP address +to use SSH on port 443. An alternate SSH hostname requires a new virtual IP address compared to the other GitLab HTTP configuration above. Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`. @@ -114,7 +114,7 @@ Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`. ## Readiness check -It is strongly recommend that multi-node deployments configure load balancers to use the [readiness check](../user/admin_area/monitoring/health_check.md#readiness) to ensure a node is ready to accept traffic, before routing traffic to it. This is especially important when utilizing Puma, as there is a brief period during a restart where Puma will not accept requests. +It is strongly recommend that multi-node deployments configure load balancers to use the [readiness check](../user/admin_area/monitoring/health_check.md#readiness) to ensure a node is ready to accept traffic, before routing traffic to it. This is especially important when utilizing Puma, as there is a brief period during a restart where Puma doesn't accept requests. <!-- ## Troubleshooting diff --git a/doc/administration/logs.md b/doc/administration/logs.md index e5523ba67aa..3d5ba903941 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -1,7 +1,7 @@ --- stage: Monitor group: Health -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Log system @@ -117,7 +117,7 @@ The ActionCable connection or channel class is used as the `controller`. } ``` -NOTE: **Note:** +NOTE: Starting with GitLab 12.5, if an error occurs, an `exception` field is included with `class`, `message`, and `backtrace`. Previous versions included an `error` field instead of @@ -383,7 +383,7 @@ only. For example: ## `audit_json.log` -NOTE: **Note:** +NOTE: Most log entries only exist in [GitLab Starter](https://about.gitlab.com/pricing/), however a few exist in GitLab Core. This file lives in `/var/log/gitlab/gitlab-rails/audit_json.log` for @@ -665,6 +665,31 @@ installations from source. It logs the progress of the export process. +## `features_json.log` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/59587) in GitLab 13.7. + +This file's location depends on how you installed GitLab: + +- For Omnibus GitLab packages: `/var/log/gitlab/gitlab-rails/features_json.log` +- For installations from source: `/home/git/gitlab/log/features_json.log` + +The modification events from [Feature flags in development of GitLab](../development/feature_flags/index.md) +are recorded in this file. For example: + +```json +{"severity":"INFO","time":"2020-11-24T02:30:59.860Z","correlation_id":null,"key":"cd_auto_rollback","action":"enable","extra.thing":"true"} +{"severity":"INFO","time":"2020-11-24T02:31:29.108Z","correlation_id":null,"key":"cd_auto_rollback","action":"enable","extra.thing":"true"} +{"severity":"INFO","time":"2020-11-24T02:31:29.129Z","correlation_id":null,"key":"cd_auto_rollback","action":"disable","extra.thing":"false"} +{"severity":"INFO","time":"2020-11-24T02:31:29.177Z","correlation_id":null,"key":"cd_auto_rollback","action":"enable","extra.thing":"Project:1"} +{"severity":"INFO","time":"2020-11-24T02:31:29.183Z","correlation_id":null,"key":"cd_auto_rollback","action":"disable","extra.thing":"Project:1"} +{"severity":"INFO","time":"2020-11-24T02:31:29.188Z","correlation_id":null,"key":"cd_auto_rollback","action":"enable_percentage_of_time","extra.percentage":"50"} +{"severity":"INFO","time":"2020-11-24T02:31:29.193Z","correlation_id":null,"key":"cd_auto_rollback","action":"disable_percentage_of_time"} +{"severity":"INFO","time":"2020-11-24T02:31:29.198Z","correlation_id":null,"key":"cd_auto_rollback","action":"enable_percentage_of_actors","extra.percentage":"50"} +{"severity":"INFO","time":"2020-11-24T02:31:29.203Z","correlation_id":null,"key":"cd_auto_rollback","action":"disable_percentage_of_actors"} +{"severity":"INFO","time":"2020-11-24T02:31:29.329Z","correlation_id":null,"key":"cd_auto_rollback","action":"remove"} +``` + ## `auth.log` > Introduced in GitLab 12.0. @@ -751,7 +776,7 @@ are generated: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/15442) in GitLab 12.3. -Contains details of GitLab's [Database Load Balancing](database_load_balancing.md). +Contains details of GitLab [Database Load Balancing](database_load_balancing.md). It's stored at: - `/var/log/gitlab/gitlab-rails/database_load_balancing.log` for Omnibus GitLab packages. @@ -973,12 +998,31 @@ For Omnibus GitLab installations, GitLab Exporter logs reside in `/var/log/gitla For Omnibus GitLab installations, GitLab Kubernetes Agent Server logs reside in `/var/log/gitlab/gitlab-kas/`. +## Performance bar stats + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/48149) in GitLab 13.7. + +This file lives in `/var/log/gitlab/gitlab-rails/performance_bar_json.log` for +Omnibus GitLab packages or in `/home/git/gitlab/log/performance_bar_json.log` for +installations from source. + +Performance bar statistics (currently only duration of SQL queries) are recorded in that file. For example: + +```json +{"severity":"INFO","time":"2020-12-04T09:29:44.592Z","correlation_id":"33680b1490ccd35981b03639c406a697","filename":"app/models/ci/pipeline.rb","filenum":"395","method":"each_with_object","request_id":"rYHomD0VJS4","duration_ms":26.889,"type": "sql"} +``` + +These statistics are logged on .com only, disabled on self-deployments. + ## Gathering logs When [troubleshooting](troubleshooting/index.md) issues that aren't localized to one of the previously listed components, it's helpful to simultaneously gather multiple logs and statistics from a GitLab instance. +NOTE: +GitLab Support often asks for one of these, and maintains the required tools. + ### Briefly tail the main logs If the bug or error is readily reproducible, save the main GitLab logs @@ -995,12 +1039,9 @@ Conclude the log gathering with <kbd>Ctrl</kbd> + <kbd>C</kbd>. If performance degradations or cascading errors occur that can't readily be attributed to one of the previously listed GitLab components, [GitLabSOS](https://gitlab.com/gitlab-com/support/toolbox/gitlabsos/) -can provide a perspective spanning all of Omnibus GitLab. For more details and instructions +can provide a broader perspective of the GitLab instance. For more details and instructions to run it, read [the GitLabSOS documentation](https://gitlab.com/gitlab-com/support/toolbox/gitlabsos/#gitlabsos). -NOTE: **Note:** -GitLab Support likes to use this custom-made tool. - ### Fast-stats [Fast-stats](https://gitlab.com/gitlab-com/support/toolbox/fast-stats) is a tool diff --git a/doc/administration/maven_packages.md b/doc/administration/maven_packages.md index cdcce79f8f9..690b6785941 100644 --- a/doc/administration/maven_packages.md +++ b/doc/administration/maven_packages.md @@ -3,3 +3,6 @@ redirect_to: 'packages/index.md' --- This document was moved to [another location](packages/index.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/maven_repository.md b/doc/administration/maven_repository.md index cdcce79f8f9..690b6785941 100644 --- a/doc/administration/maven_repository.md +++ b/doc/administration/maven_repository.md @@ -3,3 +3,6 @@ redirect_to: 'packages/index.md' --- This document was moved to [another location](packages/index.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/merge_request_diffs.md b/doc/administration/merge_request_diffs.md index 3c093d44780..92fdba6216c 100644 --- a/doc/administration/merge_request_diffs.md +++ b/doc/administration/merge_request_diffs.md @@ -1,7 +1,7 @@ --- stage: Create group: Editor -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers" +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" type: reference --- @@ -67,7 +67,7 @@ that only [stores outdated diffs](#alternative-in-database-storage) outside of d ## Using object storage -CAUTION: **Warning:** +WARNING: Currently migrating to object storage is **non-reversible** Instead of storing the external diffs on disk, we recommended the use of an object @@ -102,7 +102,7 @@ be configured already. ### Object Storage Settings -NOTE: **Note:** +NOTE: In GitLab 13.2 and later, we recommend using the [consolidated object storage settings](object_storage.md#consolidated-object-storage-configuration). This section describes the earlier configuration format. diff --git a/doc/administration/monitoring/github_imports.md b/doc/administration/monitoring/github_imports.md index da7796c05f0..cf7c105e8cf 100644 --- a/doc/administration/monitoring/github_imports.md +++ b/doc/administration/monitoring/github_imports.md @@ -1,7 +1,7 @@ --- stage: Monitor group: Health -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Monitoring GitHub imports diff --git a/doc/administration/monitoring/gitlab_instance_administration_project/index.md b/doc/administration/monitoring/gitlab_instance_administration_project/index.md index 1235eb2edec..59837dcdb20 100644 --- a/doc/administration/monitoring/gitlab_instance_administration_project/index.md +++ b/doc/administration/monitoring/gitlab_instance_administration_project/index.md @@ -3,3 +3,6 @@ redirect_to: '../gitlab_self_monitoring_project/index.md' --- This document was moved to [another location](../gitlab_self_monitoring_project/index.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md index fcaf9aaaaee..13d42ca2ee6 100644 --- a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md +++ b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md @@ -1,7 +1,7 @@ --- stage: Monitor group: Health -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # GitLab self monitoring project @@ -14,13 +14,13 @@ health of their GitLab instance. To surface this experience in a native way (similar to how you would interact with an application deployed using GitLab), a base project called "GitLab self monitoring" with [internal visibility](../../../public_access/public_access.md#internal-projects) -will be added under a group called "GitLab Instance Administrators" +is added under a group called "GitLab Instance Administrators" specifically created for visualizing and configuring the monitoring of your GitLab instance. -All administrators at the time of creation of the project and group will be -added as maintainers of the group and project, and as an admin, you'll be able -to add new members to the group to give them maintainer access to the project. +All administrators at the time of creation of the project and group are +added as maintainers of the group and project, and as an admin, you can +add new members to the group to give them maintainer access to the project. This project is used to self monitor your GitLab instance. The metrics dashboard of the project shows some basic resource usage charts, such as CPU and memory usage @@ -34,13 +34,13 @@ metrics exposed by the [GitLab exporter](../prometheus/gitlab_metrics.md#metrics 1. Navigate to **Admin Area > Settings > Metrics and profiling**, and expand the **Self monitoring** section. 1. Toggle the **Create Project** button on. -1. Once your GitLab instance creates the project, you'll see a link to the project in the text above the **Create Project** toggle. You can also find it under **Projects > Your projects**. +1. Once your GitLab instance creates the project, GitLab displays a link to the project in the text above the **Create Project** toggle. You can also find it under **Projects > Your projects**. ## Deleting the self monitoring project -CAUTION: **Warning:** -If you delete the self monitoring project, you will lose any changes made to the -project. If you create the project again, it will be created in its default state. +WARNING: +Deleting the self monitoring project removes any changes made to the project. If +you create the project again, it's created in its default state. 1. Navigate to **Admin Area > Settings > Metrics and profiling**, and expand the **Self monitoring** section. 1. Toggle the **Create Project** button off. @@ -63,7 +63,7 @@ You can also ## Connection to Prometheus -The project will be automatically configured to connect to the +The project is automatically configured to connect to the [internal Prometheus](../prometheus/index.md) instance if the Prometheus instance is present (should be the case if GitLab was installed via Omnibus and you haven't disabled it). diff --git a/doc/administration/monitoring/index.md b/doc/administration/monitoring/index.md index fbd4c1aa3dd..68dbe9f3cf9 100644 --- a/doc/administration/monitoring/index.md +++ b/doc/administration/monitoring/index.md @@ -1,7 +1,7 @@ --- stage: Monitor group: Health -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Monitoring GitLab @@ -13,7 +13,7 @@ Explore our features to monitor your GitLab instance: take action on alerts. - [Performance monitoring](performance/index.md): GitLab Performance Monitoring makes it possible to measure a wide variety of statistics of your instance. - [Prometheus](prometheus/index.md): Prometheus is a powerful time-series monitoring service, providing a flexible platform for monitoring GitLab and other software products. -- [GitHub imports](github_imports.md): Monitor the health and progress of GitLab's GitHub importer with various Prometheus metrics. +- [GitHub imports](github_imports.md): Monitor the health and progress of the GitHub importer with various Prometheus metrics. - [Monitoring uptime](../../user/admin_area/monitoring/health_check.md): Check the server status using the health check endpoint. - [IP whitelists](ip_whitelist.md): Configure GitLab for monitoring endpoints that provide health check information when probed. - [`nginx_status`](https://docs.gitlab.com/omnibus/settings/nginx.html#enablingdisabling-nginx_status): Monitor your NGINX server status diff --git a/doc/administration/monitoring/ip_whitelist.md b/doc/administration/monitoring/ip_whitelist.md index 480a4ea3598..83fbec86f57 100644 --- a/doc/administration/monitoring/ip_whitelist.md +++ b/doc/administration/monitoring/ip_whitelist.md @@ -1,14 +1,14 @@ --- stage: Monitor group: Health -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # IP whitelist > Introduced in GitLab 9.4. -NOTE: **Note:** +NOTE: We intend to [rename IP whitelist as `IP allowlist`](https://gitlab.com/gitlab-org/gitlab/-/issues/7554). GitLab provides some [monitoring endpoints](../../user/admin_area/monitoring/health_check.md) diff --git a/doc/administration/monitoring/performance/gitlab_configuration.md b/doc/administration/monitoring/performance/gitlab_configuration.md index eb9dab1af59..dd4481434a6 100644 --- a/doc/administration/monitoring/performance/gitlab_configuration.md +++ b/doc/administration/monitoring/performance/gitlab_configuration.md @@ -1,7 +1,7 @@ --- stage: Monitor group: Health -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # GitLab Configuration diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md index dc77a35e44b..e76e81c6499 100644 --- a/doc/administration/monitoring/performance/grafana_configuration.md +++ b/doc/administration/monitoring/performance/grafana_configuration.md @@ -1,7 +1,7 @@ --- stage: Monitor group: Health -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Grafana Configuration @@ -113,7 +113,7 @@ If you require access to your old Grafana data but don't meet one of these crite 1. [Exporting the dashboards](https://grafana.com/docs/grafana/latest/reference/export_import/#exporting-a-dashboard) you need. 1. Refreshing the data and [re-importing your dashboards](https://grafana.com/docs/grafana/latest/reference/export_import/#importing-a-dashboard). -DANGER: **Warning:** +WARNING: These actions pose a temporary vulnerability while your old Grafana data is in use. Deciding to take any of these actions should be weighed carefully with your need to access existing data and dashboards. diff --git a/doc/administration/monitoring/performance/index.md b/doc/administration/monitoring/performance/index.md index 6ea756619f5..5b82696ae4b 100644 --- a/doc/administration/monitoring/performance/index.md +++ b/doc/administration/monitoring/performance/index.md @@ -1,7 +1,7 @@ --- stage: Monitor group: Health -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # GitLab Performance Monitoring diff --git a/doc/administration/monitoring/performance/introduction.md b/doc/administration/monitoring/performance/introduction.md index 7ace0ec5a93..a275efce5bb 100644 --- a/doc/administration/monitoring/performance/introduction.md +++ b/doc/administration/monitoring/performance/introduction.md @@ -3,3 +3,6 @@ redirect_to: 'index.md' --- This document was moved to [another location](index.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index 45dc4730cab..a214660bd5c 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -1,7 +1,7 @@ --- stage: Monitor group: Health -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Performance Bar diff --git a/doc/administration/monitoring/performance/prometheus.md b/doc/administration/monitoring/performance/prometheus.md index f05a420fc19..2978de865e2 100644 --- a/doc/administration/monitoring/performance/prometheus.md +++ b/doc/administration/monitoring/performance/prometheus.md @@ -3,3 +3,6 @@ redirect_to: '../prometheus/index.md' --- This document was moved to [another location](../prometheus/index.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/monitoring/performance/request_profiling.md b/doc/administration/monitoring/performance/request_profiling.md index 6e03404fd26..23af50dec11 100644 --- a/doc/administration/monitoring/performance/request_profiling.md +++ b/doc/administration/monitoring/performance/request_profiling.md @@ -1,7 +1,7 @@ --- stage: Monitor group: Health -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Request Profiling diff --git a/doc/administration/monitoring/prometheus/gitlab_exporter.md b/doc/administration/monitoring/prometheus/gitlab_exporter.md index aa2eb9f3415..8e4d87cfa78 100644 --- a/doc/administration/monitoring/prometheus/gitlab_exporter.md +++ b/doc/administration/monitoring/prometheus/gitlab_exporter.md @@ -1,7 +1,7 @@ --- stage: Monitor group: Health -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # GitLab exporter diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index 5c9268b54e1..c5e0570515d 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -1,7 +1,7 @@ --- stage: Monitor group: Health -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # GitLab Prometheus metrics @@ -142,6 +142,7 @@ configuration option in `gitlab.yml`. These metrics are served from the | `sidekiq_jobs_queue_duration_seconds` | Histogram | 12.5 | Duration in seconds that a Sidekiq job was queued before being executed | `queue`, `boundary`, `external_dependencies`, `feature_category`, `urgency` | | `sidekiq_jobs_failed_total` | Counter | 12.2 | Sidekiq jobs failed | `queue`, `boundary`, `external_dependencies`, `feature_category`, `urgency` | | `sidekiq_jobs_retried_total` | Counter | 12.2 | Sidekiq jobs retried | `queue`, `boundary`, `external_dependencies`, `feature_category`, `urgency` | +| `sidekiq_jobs_dead_total` | Counter | 13.7 | Sidekiq dead jobs (jobs that have run out of retries) | `queue`, `boundary`, `external_dependencies`, `feature_category`, `urgency` | | `sidekiq_redis_requests_total` | Counter | 13.1 | Redis requests during a Sidekiq job execution | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency` | | `sidekiq_elasticsearch_requests_total` | Counter | 13.1 | Elasticsearch requests during a Sidekiq job execution | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency` | | `sidekiq_running_jobs` | Gauge | 12.2 | Number of Sidekiq jobs running | `queue`, `boundary`, `external_dependencies`, `feature_category`, `urgency` | @@ -309,7 +310,7 @@ instance (`cache`, `shared_state` etc.). ## Metrics shared directory -GitLab's Prometheus client requires a directory to store metrics data shared between multi-process services. +The GitLab Prometheus client requires a directory to store metrics data shared between multi-process services. Those files are shared among all instances running under Unicorn server. The directory must be accessible to all running Unicorn's processes, or metrics can't function correctly. diff --git a/doc/administration/monitoring/prometheus/gitlab_monitor_exporter.md b/doc/administration/monitoring/prometheus/gitlab_monitor_exporter.md index ae3a3d739b5..8fa2e16dcd0 100644 --- a/doc/administration/monitoring/prometheus/gitlab_monitor_exporter.md +++ b/doc/administration/monitoring/prometheus/gitlab_monitor_exporter.md @@ -3,3 +3,6 @@ redirect_to: 'gitlab_exporter.md' --- This document was moved to [another location](gitlab_exporter.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index 91f810dc681..80ddb87e727 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -1,7 +1,7 @@ --- stage: Monitor group: Health -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Monitoring GitLab with Prometheus @@ -10,10 +10,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w > > - Prometheus and the various exporters listed in this page are bundled in the > Omnibus GitLab package. Check each exporter's documentation for the timeline -> they got added. For installations from source you will have to install them -> yourself. Over subsequent releases additional GitLab metrics will be captured. +> they got added. For installations from source you must install them +> yourself. Over subsequent releases additional GitLab metrics are captured. > - Prometheus services are on by default with GitLab 9.0. -> - Prometheus and its exporters don't authenticate users, and will be available +> - Prometheus and its exporters don't authenticate users, and are available > to anyone who can access them. [Prometheus](https://prometheus.io) is a powerful time-series monitoring service, providing a flexible @@ -34,9 +34,9 @@ dashboard tool like [Grafana](https://grafana.com). For installations from source, you must install and configure it yourself. Prometheus and its exporters are on by default, starting with GitLab 9.0. -Prometheus will run as the `gitlab-prometheus` user and listen on +Prometheus runs as the `gitlab-prometheus` user and listen on `http://localhost:9090`. By default, Prometheus is only accessible from the GitLab server itself. -Each exporter will be automatically set up as a +Each exporter is automatically set up as a monitoring target for Prometheus, unless individually disabled. To disable Prometheus and all of its exporters, as well as any added in the future: @@ -53,7 +53,7 @@ To disable Prometheus and all of its exporters, as well as any added in the futu ### Changing the port and address Prometheus listens on -CAUTION: **Caution:** +WARNING: The following change was added in [Omnibus GitLab 8.17](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/1261). Although possible, it's not recommended to change the port Prometheus listens on, as this might affect or conflict with other services running on the GitLab @@ -179,12 +179,11 @@ The next step is to tell all the other nodes where the monitoring node is: After monitoring using Service Discovery is enabled with `consul['monitoring_service_discovery'] = true`, ensure that `prometheus['scrape_configs']` is not set in `/etc/gitlab/gitlab.rb`. Setting both -`consul['monitoring_service_discovery'] = true` and `prometheus['scrape_configs']` in `/etc/gitlab/gitlab.rb` -will result in errors. +`consul['monitoring_service_discovery'] = true` and `prometheus['scrape_configs']` in `/etc/gitlab/gitlab.rb` results in errors. ### Using an external Prometheus server -CAUTION: **Caution:** +WARNING: Prometheus and most exporters don't support authentication. We don't recommend exposing them outside the local network. A few configuration changes are required to allow GitLab to be monitored by an external Prometheus server. External servers are recommended for [GitLab deployments with multiple nodes](../../reference_architectures/index.md). @@ -234,7 +233,7 @@ To use an external Prometheus server: gitlab_rails['prometheus_address'] = '192.168.0.1:9090' ``` -1. To scrape NGINX metrics, you'll also need to configure NGINX to allow the Prometheus server +1. To scrape NGINX metrics, you must also configure NGINX to allow the Prometheus server IP. For example: ```ruby @@ -350,7 +349,7 @@ To add a Prometheus dashboard for a single server GitLab setup: GitLab monitors its own internal service metrics, and makes them available at the `/-/metrics` endpoint. Unlike other exporters, this endpoint requires authentication as it's available on the same URL and port as user traffic. -[â Read more about the GitLab Metrics.](gitlab_metrics.md) +Read more about the [GitLab Metrics](gitlab_metrics.md). ## Bundled software metrics @@ -399,7 +398,7 @@ The GitLab exporter allows you to measure various GitLab metrics, pulled from Re > - Introduced in GitLab 9.0. > - Pod monitoring introduced in GitLab 9.4. -If your GitLab server is running within Kubernetes, Prometheus will collect metrics from the Nodes and [annotated Pods](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config) in the cluster, including performance data on each container. This is particularly helpful if your CI/CD environments run in the same cluster, as you can use the [Prometheus project integration](../../../user/project/integrations/prometheus.md) to monitor them. +If your GitLab server is running within Kubernetes, Prometheus collects metrics from the Nodes and [annotated Pods](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config) in the cluster, including performance data on each container. This is particularly helpful if your CI/CD environments run in the same cluster, as you can use the [Prometheus project integration](../../../user/project/integrations/prometheus.md) to monitor them. To disable the monitoring of Kubernetes: diff --git a/doc/administration/monitoring/prometheus/node_exporter.md b/doc/administration/monitoring/prometheus/node_exporter.md index fea78a3685c..f27912d5806 100644 --- a/doc/administration/monitoring/prometheus/node_exporter.md +++ b/doc/administration/monitoring/prometheus/node_exporter.md @@ -1,7 +1,7 @@ --- stage: Monitor group: Health -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Node exporter @@ -24,5 +24,5 @@ To enable the node exporter: 1. Save the file, and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -Prometheus will now begin collecting performance data from the node exporter +Prometheus begins collecting performance data from the node exporter exposed at `localhost:9100`. diff --git a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md index ff0cfc65e10..0dffad59793 100644 --- a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md +++ b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md @@ -1,7 +1,7 @@ --- stage: Monitor group: Health -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # PgBouncer exporter @@ -26,9 +26,9 @@ To enable the PgBouncer exporter: 1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -Prometheus will now begin collecting performance data from the PgBouncer exporter +Prometheus begins collecting performance data from the PgBouncer exporter exposed at `localhost:9188`. -The PgBouncer exporter will also be enabled by default if the +The PgBouncer exporter is enabled by default if the [`pgbouncer_role`](https://docs.gitlab.com/omnibus/roles/#postgresql-roles) role is enabled. diff --git a/doc/administration/monitoring/prometheus/postgres_exporter.md b/doc/administration/monitoring/prometheus/postgres_exporter.md index f7368556235..8bc61ff53bb 100644 --- a/doc/administration/monitoring/prometheus/postgres_exporter.md +++ b/doc/administration/monitoring/prometheus/postgres_exporter.md @@ -1,7 +1,7 @@ --- stage: Monitor group: Health -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # PostgreSQL Server Exporter @@ -21,12 +21,12 @@ To enable the PostgreSQL Server Exporter: If PostgreSQL Server Exporter is configured on a separate node, make sure that the local address is [listed in `trust_auth_cidr_addresses`](../../postgresql/replication_and_failover.md#network-information) or the - exporter will not be able to connect to the database. + exporter can't connect to the database. 1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -Prometheus will now automatically begin collecting performance data from +Prometheus begins collecting performance data from the PostgreSQL Server Exporter exposed under `localhost:9187`. ## Advanced configuration diff --git a/doc/administration/monitoring/prometheus/redis_exporter.md b/doc/administration/monitoring/prometheus/redis_exporter.md index 41a84f1f3ed..03c215625ec 100644 --- a/doc/administration/monitoring/prometheus/redis_exporter.md +++ b/doc/administration/monitoring/prometheus/redis_exporter.md @@ -1,7 +1,7 @@ --- stage: Monitor group: Health -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Redis exporter @@ -25,5 +25,5 @@ To enable the Redis exporter: 1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -Prometheus will now begin collecting performance data from +Prometheus begins collecting performance data from the Redis exporter exposed at `localhost:9121`. diff --git a/doc/administration/monitoring/prometheus/registry_exporter.md b/doc/administration/monitoring/prometheus/registry_exporter.md index 3bf4db8a665..009d94491f3 100644 --- a/doc/administration/monitoring/prometheus/registry_exporter.md +++ b/doc/administration/monitoring/prometheus/registry_exporter.md @@ -1,7 +1,7 @@ --- stage: Monitor group: Health -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Registry exporter @@ -21,7 +21,7 @@ To enable it: 1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -Prometheus will now automatically begin collecting performance data from +Prometheus automatically begins collecting performance data from the registry exporter exposed under `localhost:5001/metrics`. [â Back to the main Prometheus page](index.md) diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md index 3e16b173003..1e2ac531329 100644 --- a/doc/administration/nfs.md +++ b/doc/administration/nfs.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -14,7 +14,7 @@ Pages](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/196). For data objects such as LFS, Uploads, Artifacts, etc., an [Object Storage service](object_storage.md) is recommended over NFS where possible, due to better performance. -CAUTION: **Caution:** +WARNING: From GitLab 13.0, using NFS for Git repositories is deprecated. In GitLab 14.0, support for NFS for Git repositories is scheduled to be removed. Upgrade to [Gitaly Cluster](gitaly/praefect.md) as soon as possible. @@ -118,7 +118,7 @@ To disable NFS server delegation, do the following: 1. Restart the NFS server process. For example, on CentOS run `service nfs restart`. -NOTE: **Note:** +NOTE: The kernel bug may be fixed in [more recent kernels with this commit](https://github.com/torvalds/linux/commit/95da1b3a5aded124dd1bda1e3cdb876184813140). Red Hat Enterprise 7 [shipped a kernel update](https://access.redhat.com/errata/RHSA-2019:2029) @@ -133,7 +133,7 @@ NFS performance with GitLab can in some cases be improved with [direct Git access](gitaly/index.md#direct-access-to-git-in-gitlab) using [Rugged](https://github.com/libgit2/rugged). -NOTE: **Note:** +NOTE: From GitLab 12.1, it will automatically be detected if Rugged can and should be used per storage. If you previously enabled Rugged using the feature flag, you will need to unset the feature flag by using: @@ -146,7 +146,7 @@ If the Rugged feature flag is explicitly set to either `true` or `false`, GitLab #### Improving NFS performance with Puma -NOTE: **Note:** +NOTE: From GitLab 12.7, Rugged is not automatically enabled if Puma thread count is greater than `1`. If you want to use Rugged with Puma, [set Puma thread count to `1`](https://docs.gitlab.com/omnibus/settings/puma.html#puma-settings). @@ -344,7 +344,7 @@ sudo ufw allow from <client_ip_address> to any port nfs ### Upgrade to Gitaly Cluster or disable caching if experiencing data loss -CAUTION: **Caution:** +WARNING: From GitLab 13.0, using NFS for Git repositories is deprecated. In GitLab 14.0, support for NFS for Git repositories is scheduled to be removed. Upgrade to [Gitaly Cluster](gitaly/praefect.md) as soon as possible. @@ -358,7 +358,7 @@ For example, we have seen [inconsistent updates after a push](https://gitlab.com | `actimeo=0` | Sets the time to zero that the NFS client caches files and directories before requesting fresh information from a server. | | `noac` | Tells the NFS client not to cache file attributes and forces application writes to become synchronous so that local changes to a file become visible on the server immediately. | -CAUTION: **Caution:** +WARNING: The `actimeo=0` and `noac` options both result in a significant reduction in performance, possibly leading to timeouts. You may be able to avoid timeouts and data loss using `actimeo=0` and `lookupcache=positive` _without_ `noac`, however we expect the performance reduction will still be significant. As noted above, we strongly recommend upgrading to @@ -370,7 +370,7 @@ GitLab strongly recommends against using AWS Elastic File System (EFS). Our support team will not be able to assist on performance issues related to file system access. -Customers and users have reported that AWS EFS does not perform well for GitLab's +Customers and users have reported that AWS EFS does not perform well for the GitLab use-case. Workloads where many small files are written in a serialized manner, like `git`, are not well-suited for EFS. EBS with an NFS server on top will perform much better. @@ -383,7 +383,7 @@ For more details on another person's experience with EFS, see this [Commit Brook ### Avoid using CephFS and GlusterFS GitLab strongly recommends against using CephFS and GlusterFS. -These distributed file systems are not well-suited for GitLab's input/output access patterns because Git uses many small files and access times and file locking times to propagate will make Git activity very slow. +These distributed file systems are not well-suited for the GitLab input/output access patterns because Git uses many small files and access times and file locking times to propagate will make Git activity very slow. ### Avoid using PostgreSQL with NFS @@ -402,14 +402,19 @@ Additionally, this configuration is specifically warned against in the For supported database architecture, see our documentation about [configuring a database for replication and failover](postgresql/replication_and_failover.md). -<!-- ## Troubleshooting +## Troubleshooting -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. +### Finding the requests that are being made to NFS -Each scenario can be a third-level heading, e.g. `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +In case of NFS-related problems, it can be helpful to trace +the filesystem requests that are being made by using `perf`: + +```shell +sudo perf trace -e 'nfs4:*' -p $(pgrep -fd ',' puma && pgrep -fd ',' unicorn) +``` + +On Ubuntu 16.04, use: + +```shell +sudo perf trace --no-syscalls --event 'nfs4:*' -p $(pgrep -fd ',' puma && pgrep -fd ',' unicorn) +``` diff --git a/doc/administration/npm_registry.md b/doc/administration/npm_registry.md index cdcce79f8f9..690b6785941 100644 --- a/doc/administration/npm_registry.md +++ b/doc/administration/npm_registry.md @@ -3,3 +3,6 @@ redirect_to: 'packages/index.md' --- This document was moved to [another location](packages/index.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index 2b714c36d93..a89c50a8412 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -114,6 +114,7 @@ See the section on [ETag mismatch errors](#etag-mismatch) for more details. gitlab_rails['object_store']['objects']['packages']['bucket'] = '<packages>' gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = '<dependency-proxy>' gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = '<terraform-state>' + gitlab_rails['object_store']['objects']['pages']['bucket'] = '<pages>' ``` For GitLab 9.4 or later, if you're using AWS IAM profiles, be sure to omit the @@ -160,6 +161,8 @@ See the section on [ETag mismatch errors](#etag-mismatch) for more details. bucket: <dependency_proxy> terraform_state: bucket: <terraform> + pages: + bucket: <pages> ``` 1. Edit `/home/git/gitlab-workhorse/config.toml` and add or amend the following lines: @@ -440,6 +443,8 @@ required parameter within each type: bucket: dependency_proxy terraform_state: bucket: terraform + pages: + bucket: pages ``` This maps to this Omnibus GitLab configuration: @@ -454,6 +459,7 @@ gitlab_rails['object_store']['objects']['packages']['bucket'] = 'packages' gitlab_rails['object_store']['objects']['dependency_proxy']['enabled'] = false gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = 'dependency-proxy' gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = 'terraform-state' +gitlab_rails['object_store']['objects']['pages']['bucket'] = 'pages' ``` This is the list of valid `objects` that can be used: @@ -467,6 +473,7 @@ This is the list of valid `objects` that can be used: | `packages` | [Project packages (e.g. PyPI, Maven, NuGet, etc.)](packages/index.md) | | `dependency_proxy` | [GitLab Dependency Proxy](packages/dependency_proxy.md) | | `terraform_state` | [Terraform state files](terraform_state.md) | +| `pages` | [GitLab Pages](pages/index.md) | Within each object type, three parameters can be defined: @@ -536,7 +543,7 @@ supported by consolidated configuration form, refer to the following guides: |Object storage type|Supported by consolidated configuration?| |-------------------|----------------------------------------| -| [Backups](../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage)|No| +| [Backups](../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage) | No | | [Job artifacts](job_artifacts.md#using-object-storage) including archived job logs | Yes | | [LFS objects](lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | | [Uploads](uploads.md#using-object-storage) | Yes | @@ -548,6 +555,7 @@ supported by consolidated configuration form, refer to the following guides: | [Pseudonymizer](pseudonymizer.md#configuration) (optional feature) **(ULTIMATE ONLY)** | No | | [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | | [Terraform state files](terraform_state.md#using-object-storage) | Yes | +| [GitLab Pages content](pages/index.md#using-object-storage) | Yes | ### Other alternatives to filesystem storage diff --git a/doc/administration/operations.md b/doc/administration/operations.md index 9cd78105bbb..cfabb5ec27d 100644 --- a/doc/administration/operations.md +++ b/doc/administration/operations.md @@ -3,3 +3,6 @@ redirect_to: 'operations/index.md' --- This document was moved to [another location](operations/index.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/operations/cleaning_up_redis_sessions.md b/doc/administration/operations/cleaning_up_redis_sessions.md index 8f90231a713..6513a4ed4c8 100644 --- a/doc/administration/operations/cleaning_up_redis_sessions.md +++ b/doc/administration/operations/cleaning_up_redis_sessions.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Cleaning up stale Redis sessions @@ -21,7 +21,7 @@ prefixed with `session:gitlab:`, so they would look like `session:gitlab:976aa289e2189b17d7ef525a6702ace9`. Below we describe how to remove the keys in the old format. -NOTE: **Note:** +NOTE: The instructions below must be modified in accordance with your configuration settings if you have used the advanced Redis settings outlined in diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md index 5de79882083..1f611a50a53 100644 --- a/doc/administration/operations/extra_sidekiq_processes.md +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Run multiple Sidekiq processes **(CORE ONLY)** @@ -11,7 +11,7 @@ These processes can be used to consume a dedicated set of queues. This can be used to ensure certain queues always have dedicated workers, no matter the number of jobs that need to be processed. -NOTE: **Note:** +NOTE: The information in this page applies only to Omnibus GitLab. ## Available Sidekiq queues @@ -209,7 +209,7 @@ sidekiq['queue_groups'] = [ ### Disable Sidekiq cluster -CAUTION: **Warning:** +WARNING: Sidekiq cluster is [scheduled](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/240) to be the only way to start Sidekiq in GitLab 14.0. @@ -341,7 +341,7 @@ being equal to `max_concurrency`. Running a single Sidekiq process is the default in GitLab 12.10 and earlier. -CAUTION: **Warning:** +WARNING: Running Sidekiq directly is scheduled to be removed in GitLab [14.0](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/240). @@ -376,7 +376,7 @@ This tells the additional processes how often to check for enqueued jobs. ## Troubleshoot using the CLI -CAUTION: **Warning:** +WARNING: It's recommended to use `/etc/gitlab/gitlab.rb` to configure the Sidekiq processes. If you experience a problem, you should contact GitLab support. Use the command line at your own risk. diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index c8830a45fb2..b93af074795 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Fast lookup of authorized SSH keys in the database @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1631) in [GitLab Starter](https://about.gitlab.com/pricing/) 9.3. > - [Available in](https://gitlab.com/gitlab-org/gitlab/-/issues/3953) GitLab Community Edition 10.4. -NOTE: **Note:** +NOTE: This document describes a drop-in replacement for the `authorized_keys` file. For normal (non-deploy key) users, consider using [SSH certificates](ssh_certificates.md). They are even faster, but are not a @@ -28,7 +28,8 @@ GitLab Shell solves this by providing a way to authorize SSH users via a fast, indexed lookup in the GitLab database. This page describes how to enable the fast lookup of authorized SSH keys. -> **Warning:** OpenSSH version 6.9+ is required because +WARNING: +OpenSSH version 6.9+ is required because `AuthorizedKeysCommand` must be able to accept a fingerprint. These instructions will break installations using older versions of OpenSSH, such as those included with CentOS 6 as of September 2017. If you want to use this @@ -80,18 +81,18 @@ Confirm that SSH is working by commenting out your user's key in the `authorized A successful pull would mean that GitLab was able to find the key in the database, since it is not present in the file anymore. -NOTE: **Note:** +NOTE: For Omnibus Docker, `AuthorizedKeysCommand` is setup by default in GitLab 11.11 and later. -NOTE: **Note:** +NOTE: For Installations from source, the command would be located at `/home/git/gitlab-shell/bin/gitlab-shell-authorized-keys-check` if [the install from source](../../install/installation.md#install-gitlab-shell) instructions were followed. You might want to consider creating a wrapper script somewhere else since this command needs to be owned by `root` and not be writable by group or others. You could also consider changing the ownership of this command as required, but that might require temporary ownership changes during `gitlab-shell` upgrades. -CAUTION: **Caution:** +WARNING: Do not disable writes until SSH is confirmed to be working perfectly, because the file will quickly become out-of-date. @@ -139,7 +140,7 @@ the database. The following instructions can be used to build OpenSSH 7.5: ```shell sudo su - cd /tmp - curl --remote-name https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-7.5p1.tar.gz + curl --remote-name "https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-7.5p1.tar.gz" tar xzvf openssh-7.5p1.tar.gz yum install rpm-build gcc make wget openssl-devel krb5-devel pam-devel libX11-devel xmkmf libXt-devel ``` diff --git a/doc/administration/operations/filesystem_benchmarking.md b/doc/administration/operations/filesystem_benchmarking.md index c55f253b772..9072214eb02 100644 --- a/doc/administration/operations/filesystem_benchmarking.md +++ b/doc/administration/operations/filesystem_benchmarking.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Filesystem Performance Benchmarking @@ -71,7 +71,7 @@ operations per second. ### Simple benchmarking -NOTE: **Note:** +NOTE: This test is naive but may be useful if `fio` is not available on the system. It's possible to receive good results on this test but still have poor performance due to read speed and various other diff --git a/doc/administration/operations/index.md b/doc/administration/operations/index.md index 864bbb3233e..b15417ea8d9 100644 --- a/doc/administration/operations/index.md +++ b/doc/administration/operations/index.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Performing Operations in GitLab diff --git a/doc/administration/operations/moving_repositories.md b/doc/administration/operations/moving_repositories.md index b311bee1a5b..7cc15f9cea4 100644 --- a/doc/administration/operations/moving_repositories.md +++ b/doc/administration/operations/moving_repositories.md @@ -1,7 +1,7 @@ --- stage: Create group: Gitaly -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -48,13 +48,13 @@ We look at three scenarios: - The target directory contains an outdated copy of the repositories. - How to deal with thousands of repositories. -DANGER: **Warning:** +WARNING: Each of the approaches we list can or does overwrite data in the target directory `/mnt/gitlab/repositories`. Do not mix up the source and the target. ### Recommended approach in all cases -GitLab's [backup and restore capability](../../raketasks/backup_restore.md) should be used. Git +The GitLab [backup and restore capability](../../raketasks/backup_restore.md) should be used. Git repositories are accessed, managed, and stored on GitLab servers by Gitaly as a database. Data loss can result from directly accessing and copying Gitaly's files using tools like `rsync`. @@ -94,7 +94,7 @@ If you want to compress the data before it goes over the network ### The target directory contains an outdated copy of the repositories: use `rsync` -DANGER: **Warning:** +WARNING: Using `rsync` to migrate Git data can cause data loss and repository corruption. [These instructions are being reviewed](https://gitlab.com/gitlab-org/gitlab/-/issues/270422). @@ -115,7 +115,7 @@ If you want to see progress, replace `-a` with `-av`. #### Single `rsync` to another server -DANGER: **Warning:** +WARNING: Using `rsync` to migrate Git data can cause data loss and repository corruption. [These instructions are being reviewed](https://gitlab.com/gitlab-org/gitlab/-/issues/270422). @@ -129,7 +129,7 @@ sudo -u git sh -c 'rsync -a --delete /var/opt/gitlab/git-data/repositories/. \ ### Thousands of Git repositories: use one `rsync` per repository -DANGER: **Warning:** +WARNING: Using `rsync` to migrate Git data can cause data loss and repository corruption. [These instructions are being reviewed](https://gitlab.com/gitlab-org/gitlab/-/issues/270422). @@ -150,7 +150,7 @@ longer exist at the source.** #### Parallel `rsync` for all repositories known to GitLab -DANGER: **Warning:** +WARNING: Using `rsync` to migrate Git data can cause data loss and repository corruption. [These instructions are being reviewed](https://gitlab.com/gitlab-org/gitlab/-/issues/270422). @@ -211,7 +211,7 @@ cat /home/git/transfer-logs/* | sort | uniq -u |\ #### Parallel `rsync` only for repositories with recent activity -DANGER: **Warning:** +WARNING: Using `rsync` to migrate Git data can cause data loss and repository corruption. [These instructions are being reviewed](https://gitlab.com/gitlab-org/gitlab/-/issues/270422). diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md index 5104b65c86d..44ac014650e 100644 --- a/doc/administration/operations/puma.md +++ b/doc/administration/operations/puma.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Switching to Puma diff --git a/doc/administration/operations/rails_console.md b/doc/administration/operations/rails_console.md index dac36135a8e..b40560bf6e5 100644 --- a/doc/administration/operations/rails_console.md +++ b/doc/administration/operations/rails_console.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # The Rails Console @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w The [Rails console](https://guides.rubyonrails.org/command_line.html#rails-console). provides a way to interact with your GitLab instance from the command line. -CAUTION: **Caution:** +WARNING: The Rails console interacts directly with GitLab. In many cases, there are no handrails to prevent you from permanently modifying, corrupting or destroying production data. If you would like to explore the Rails console diff --git a/doc/administration/operations/sidekiq_memory_killer.md b/doc/administration/operations/sidekiq_memory_killer.md index 0de8b681dd8..c7f00d05213 100644 --- a/doc/administration/operations/sidekiq_memory_killer.md +++ b/doc/administration/operations/sidekiq_memory_killer.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Memory -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Sidekiq MemoryKiller @@ -35,7 +35,9 @@ The MemoryKiller is controlled using environment variables. - `SIDEKIQ_DAEMON_MEMORY_KILLER`: defaults to 1. When set to 0, the MemoryKiller works in _legacy_ mode. Otherwise, the MemoryKiller works in _daemon_ mode. - In _legacy_ mode, the MemoryKiller checks the Sidekiq process RSS after each job. + In _legacy_ mode, the MemoryKiller checks the Sidekiq process RSS + ([Resident Set Size](https://github.com/mperham/sidekiq/wiki/Memory#rss)) + after each job. In _daemon_ mode, the MemoryKiller checks the Sidekiq process RSS every 3 seconds (defined by `SIDEKIQ_MEMORY_KILLER_CHECK_INTERVAL`). diff --git a/doc/administration/operations/speed_up_ssh.md b/doc/administration/operations/speed_up_ssh.md index 6dc83c42f53..2f3cf40a222 100644 --- a/doc/administration/operations/speed_up_ssh.md +++ b/doc/administration/operations/speed_up_ssh.md @@ -3,3 +3,6 @@ redirect_to: 'fast_ssh_key_lookup.md' --- This document was moved to [another location](fast_ssh_key_lookup.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/operations/ssh_certificates.md b/doc/administration/operations/ssh_certificates.md index 7cbd8c74f90..c0525cf6258 100644 --- a/doc/administration/operations/ssh_certificates.md +++ b/doc/administration/operations/ssh_certificates.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # User lookup via OpenSSH's AuthorizedPrincipalsCommand @@ -9,17 +9,18 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Available in](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/19911) GitLab > Community Edition 11.2. -GitLab's default SSH authentication requires users to upload their SSH +The default SSH authentication for GitLab requires users to upload their SSH public keys before they can use the SSH transport. -In centralized (e.g. corporate) environments this can be a hassle -operationally, particularly if the SSH keys are temporary keys issued -to the user, e.g. ones that expire 24 hours after issuing. +In centralized (for example, corporate) environments this can be a hassle +operationally, particularly if the SSH keys are temporary keys issued to the +user, including ones that expire 24 hours after issuing. In such setups some external automated process is needed to constantly upload the new keys to GitLab. -> **Warning:** OpenSSH version 6.9+ is required because that version +WARNING: +OpenSSH version 6.9+ is required because that version introduced the `AuthorizedPrincipalsCommand` configuration option. If using CentOS 6, you can [follow these instructions](fast_ssh_key_lookup.html#compiling-a-custom-version-of-openssh-for-centos-6) diff --git a/doc/administration/operations/unicorn.md b/doc/administration/operations/unicorn.md index 80dafde14aa..03995ee05ba 100644 --- a/doc/administration/operations/unicorn.md +++ b/doc/administration/operations/unicorn.md @@ -1,12 +1,12 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Understanding Unicorn and unicorn-worker-killer -NOTE: **Note:** +NOTE: Starting with GitLab 13.0, Puma is the default web server used in GitLab all-in-one package based installations as well as GitLab Helm chart deployments. @@ -51,7 +51,7 @@ master process has PID 56227 below. The main tunable options for Unicorn are the number of worker processes and the request timeout after which the Unicorn master terminates a worker process. See the [Omnibus GitLab Unicorn settings -documentation](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/unicorn.html) +documentation](https://docs.gitlab.com/omnibus/settings/unicorn.html) if you want to adjust these settings. ## unicorn-worker-killer diff --git a/doc/administration/packages.md b/doc/administration/packages.md index cdcce79f8f9..690b6785941 100644 --- a/doc/administration/packages.md +++ b/doc/administration/packages.md @@ -3,3 +3,6 @@ redirect_to: 'packages/index.md' --- This document was moved to [another location](packages/index.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index 541bd99084c..633129e98bd 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -1,7 +1,7 @@ --- stage: Package group: Package -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # GitLab Container Registry administration @@ -93,7 +93,7 @@ auth: rootcertbundle: /root/certs/certbundle ``` -CAUTION: **Caution:** +WARNING: If `auth` is not set up, users can pull Docker images without authentication. ## Container Registry domain configuration @@ -101,7 +101,7 @@ If `auth` is not set up, users can pull Docker images without authentication. There are two ways you can configure the Registry's external domain. Either: - [Use the existing GitLab domain](#configure-container-registry-under-an-existing-gitlab-domain). - The Registry listens on a port and reuses GitLab's TLS certificate. + The Registry listens on a port and reuses the TLS certificate from GitLab. - [Use a completely separate domain](#configure-container-registry-under-its-own-domain) with a new TLS certificate for that domain. @@ -374,7 +374,7 @@ driver for the Container Registry. [Read more about using object storage with GitLab](../object_storage.md). -CAUTION: **Warning:** +WARNING: GitLab does not back up Docker images that are not stored on the file system. Enable backups with your object storage provider if desired. @@ -468,7 +468,7 @@ you can pull from the Container Registry, but you cannot push. sudo aws --endpoint-url https://your-object-storage-backend.com s3 sync registry s3://mybucket ``` - TIP: **Tip:** + NOTE: If you have a lot of data, you may be able to improve performance by [running parallel sync operations](https://aws.amazon.com/premiumsupport/knowledge-center/s3-improve-transfer-sync-command/). @@ -485,7 +485,7 @@ you can pull from the Container Registry, but you cannot push. [`--dryrun`](https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html) flag and run the command. - DANGER: **Warning:** + WARNING: The [`--delete`](https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html) flag deletes files that exist in the destination but not in the source. If you swap the source and destination, all data in the Registry is deleted. @@ -612,8 +612,8 @@ You can use GitLab as an auth endpoint with an external container registry. gitlab_rails['registry_issuer'] = "omnibus-gitlab-issuer" ``` - `gitlab_rails['registry_enabled'] = true` is needed to enable GitLab's - Container Registry features and authentication endpoint. GitLab's bundled + `gitlab_rails['registry_enabled'] = true` is needed to enable GitLab + Container Registry features and authentication endpoint. The GitLab bundled Container Registry service does not start, even with this enabled. 1. A certificate-key pair is required for GitLab and the external container @@ -820,7 +820,7 @@ If you did not change the default location of the configuration file, run: sudo gitlab-ctl registry-garbage-collect ``` -This command will take some time to complete, depending on the amount of +This command takes some time to complete, depending on the amount of layers you have stored. If you changed the location of the Container Registry `config.yml`: @@ -837,7 +837,7 @@ understand the implications. > [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/3097) in Omnibus GitLab 11.10. -DANGER: **Warning:** +WARNING: This is a destructive operation. The GitLab Container Registry follows the same default workflow as Docker Distribution: @@ -867,7 +867,7 @@ You can perform garbage collection without stopping the Container Registry by pu it in read-only mode and by not using the built-in command. On large instances this could require Container Registry to be in read-only mode for a while. During this time, -you will be able to pull from the Container Registry, but you will not be able to +you are able to pull from the Container Registry, but you are not able to push. By default, the [registry storage path](#configure-storage-for-the-container-registry) @@ -896,7 +896,7 @@ To enable the read-only mode: sudo gitlab-ctl reconfigure ``` - This will set the Container Registry into the read only mode. + This command sets the Container Registry into the read only mode. 1. Next, trigger one of the garbage collect commands: @@ -908,7 +908,7 @@ To enable the read-only mode: sudo /opt/gitlab/embedded/bin/registry garbage-collect -m /var/opt/gitlab/registry/config.yml ``` - This will start the garbage collection, which might take some time to complete. + This command starts the garbage collection, which might take some time to complete. 1. Once done, in `/etc/gitlab/gitlab.rb` change it back to read-write mode: @@ -935,7 +935,7 @@ To enable the read-only mode: Ideally, you want to run the garbage collection of the registry regularly on a weekly basis at a time when the registry is not being in-use. -The simplest way is to add a new crontab job that it will run periodically +The simplest way is to add a new crontab job that it runs periodically once a week. Create a file under `/etc/cron.d/registry-garbage-collect`: @@ -1137,6 +1137,12 @@ and a simple solution would be to enable relative URLs in the Registry. ### Enable the Registry debug server +You can use the Container Registry debug server to diagnose problems. The debug endpoint can monitor metrics and health, as well as do profiling. + +WARNING: +Sensitive information may be available from the debug endpoint. +Access to the debug endpoint must be locked down in a production environment. + The optional debug server can be enabled by setting the registry debug address in your `gitlab.rb` configuration. @@ -1149,13 +1155,13 @@ After adding the setting, [reconfigure GitLab](../restart_gitlab.md#omnibus-gitl Use curl to request debug output from the debug server: ```shell -curl localhost:5001/debug/health -curl localhost:5001/debug/vars +curl "localhost:5001/debug/health" +curl "localhost:5001/debug/vars" ``` ### Advanced Troubleshooting -We will use a concrete example in the past to illustrate how to +We use a concrete example to illustrate how to diagnose a problem with the S3 setup. #### Unexpected 403 error during push @@ -1227,14 +1233,14 @@ To verify that the certificates are properly installed, run: mitmproxy --port 9000 ``` -This will run mitmproxy on port `9000`. In another window, run: +This command runs mitmproxy on port `9000`. In another window, run: ```shell -curl --proxy http://localhost:9000 https://httpbin.org/status/200 +curl --proxy "http://localhost:9000" "https://httpbin.org/status/200" ``` -If everything is set up correctly, you will see information on the mitmproxy window and -no errors from the curl commands. +If everything is set up correctly, information is displayed on the mitmproxy window and +no errors are generated by the curl commands. #### Running the Docker daemon with a proxy @@ -1248,7 +1254,7 @@ export HTTPS_PROXY="https://localhost:9000" docker daemon --debug ``` -This will launch the Docker daemon and proxy all connections through mitmproxy. +This command launches the Docker daemon and proxies all connections through mitmproxy. #### Running the Docker client @@ -1273,4 +1279,4 @@ The above image shows: What does this mean? This strongly suggests that the S3 user does not have the right [permissions to perform a HEAD request](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html). The solution: check the [IAM permissions again](https://docs.docker.com/registry/storage-drivers/s3/). -Once the right permissions were set, the error will go away. +Once the right permissions were set, the error goes away. diff --git a/doc/administration/packages/dependency_proxy.md b/doc/administration/packages/dependency_proxy.md index 56b39658dc2..9e315722c24 100644 --- a/doc/administration/packages/dependency_proxy.md +++ b/doc/administration/packages/dependency_proxy.md @@ -1,7 +1,7 @@ --- stage: Package group: Package -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # GitLab Dependency Proxy administration @@ -16,7 +16,7 @@ dependency proxies, see the [user guide](../../user/packages/dependency_proxy/in ## Enabling the Dependency Proxy feature -NOTE: **Note:** +NOTE: Dependency proxy requires the Puma web server to be enabled. To enable the dependency proxy feature: @@ -34,7 +34,7 @@ To enable the dependency proxy feature: **Installations from source** -1. After the installation is complete, you will have to configure the `dependency_proxy` +1. After the installation is complete, configure the `dependency_proxy` section in `config/gitlab.yml`. Set to `true` to enable it: ```yaml @@ -88,7 +88,7 @@ store the blobs of the dependency proxy. [Read more about using object storage with GitLab](../object_storage.md). -NOTE: **Note:** +NOTE: In GitLab 13.2 and later, we recommend using the [consolidated object storage settings](../object_storage.md#consolidated-object-storage-configuration). This section describes the earlier configuration format. @@ -161,3 +161,22 @@ This section describes the earlier configuration format. ``` 1. [Restart GitLab](../restart_gitlab.md#installations-from-source "How to restart GitLab") for the changes to take effect. + +## Disabling Authentication + +Authentication was introduced in 13.7 as part of [enabling private groups to use the +Dependency Proxy](https://gitlab.com/gitlab-org/gitlab/-/issues/11582). If you +previously used the Dependency Proxy without authentication and need to disable +this feature while you update your workflow to [authenticate with the Dependency +Proxy](../../user/packages/dependency_proxy/index.md#authenticate-with-the-dependency-proxy), +the following commands can be issued in a Rails console: + +```ruby +# Disable the authentication +Feature.disable(:dependency_proxy_for_private_groups) + +# Re-enable the authentication +Feature.enable(:dependency_proxy_for_private_groups) +``` + +The ability to disable this feature will be [removed in 13.9](https://gitlab.com/gitlab-org/gitlab/-/issues/276777). diff --git a/doc/administration/packages/index.md b/doc/administration/packages/index.md index 4af0de864ca..0f391371a6a 100644 --- a/doc/administration/packages/index.md +++ b/doc/administration/packages/index.md @@ -1,7 +1,7 @@ --- stage: Package group: Package -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # GitLab Package Registry administration @@ -32,8 +32,8 @@ The Package Registry supports the following formats: ## Accepting contributions -The below table lists formats that are not supported, but are accepting Community contributions for. Consider contributing to GitLab. This [development documentation](../../development/packages.md) will -guide you through the process. +The below table lists formats that are not supported, but are accepting Community contributions for. Consider contributing to GitLab. This [development documentation](../../development/packages.md) +guides you through the process. | Format | Status | | ------ | ------ | @@ -53,10 +53,10 @@ guide you through the process. ## Enabling the Packages feature -NOTE: **Note:** +NOTE: After the Packages feature is enabled, the repositories are available -for all new projects by default. To enable it for existing projects, users will -have to explicitly do so in the project's settings. +for all new projects by default. To enable it for existing projects, users +explicitly do so in the project's settings. To enable the Packages feature: @@ -72,7 +72,7 @@ To enable the Packages feature: **Installations from source** -1. After the installation is complete, you will have to configure the `packages` +1. After the installation is complete, you configure the `packages` section in `config/gitlab.yml`. Set to `true` to enable it: ```yaml @@ -84,7 +84,7 @@ To enable the Packages feature: **Helm Chart installations** -1. After the installation is complete, you will have to configure the `packages` +1. After the installation is complete, you configure the `packages` section in `global.appConfig.packages`. Set to `true` to enable it: ```yaml @@ -136,8 +136,8 @@ store packages. [Read more about using object storage with GitLab](../object_storage.md). -NOTE: **Note:** -We recommend using the [consolidated object storage settings](../object_storage.md#consolidated-object-storage-configuration). The following instructions apply to the original config format. +NOTE: +We recommend using the [consolidated object storage settings](../object_storage.md#consolidated-object-storage-configuration). The following instructions apply to the original configuration format. **Omnibus GitLab installations** @@ -214,7 +214,7 @@ We recommend using the [consolidated object storage settings](../object_storage. After [configuring the object storage](#using-object-storage), you may use the following task to migrate existing packages from the local storage to the remote one. -The processing will be done in a background worker and requires **no downtime**. +The processing is done in a background worker and requires **no downtime**. For Omnibus GitLab: diff --git a/doc/administration/pages/img/zip_cache_configuration.png b/doc/administration/pages/img/zip_cache_configuration.png Binary files differnew file mode 100644 index 00000000000..a332d8d0eb5 --- /dev/null +++ b/doc/administration/pages/img/zip_cache_configuration.png diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index 0ebeb96d247..af0476b72e9 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -1,7 +1,7 @@ --- stage: Release -group: Release Management -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +group: Release +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments description: 'Learn how to administer GitLab Pages.' --- @@ -16,7 +16,7 @@ description: 'Learn how to administer GitLab Pages.' GitLab Pages allows for hosting of static sites. It must be configured by an administrator. Separate [user documentation](../../user/project/pages/index.md) is available. -NOTE: **Note:** +NOTE: This guide is for Omnibus GitLab installations. If you have installed GitLab from source, see [GitLab Pages administration for source installations](source.md). @@ -53,7 +53,7 @@ supporting custom domains a secondary IP is not needed. Before proceeding with the Pages configuration, you will need to: -1. Have a domain for Pages that is not a subdomain of your GitLab's instance domain. +1. Have a domain for Pages that is not a subdomain of your GitLab instance domain. | GitLab domain | Pages domain | Does it work? | | :---: | :---: | :---: | @@ -68,7 +68,7 @@ Before proceeding with the Pages configuration, you will need to: so that your users don't have to bring their own. 1. (Only for custom domains) Have a **secondary IP**. -NOTE: **Note:** +NOTE: If your GitLab instance and the Pages daemon are deployed in a private network or behind a firewall, your GitLab Pages websites will only be accessible to devices/users that have access to the private network. ### Add the domain to the Public Suffix List @@ -101,7 +101,7 @@ where `example.io` is the domain under which GitLab Pages will be served and `192.0.2.1` is the IPv4 address of your GitLab instance and `2001::1` is the IPv6 address. If you don't have IPv6, you can omit the AAAA record. -NOTE: **Note:** +NOTE: You should not use the GitLab domain to serve user pages. For more information see the [security section](#security). ## Configuration @@ -181,7 +181,7 @@ behavior: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -NOTE: **Note:** +NOTE: `inplace_chroot` option might not work with the other features, such as [Pages Access Control](#access-control). The [GitLab Pages README](https://gitlab.com/gitlab-org/gitlab-pages#caveats) has more information about caveats and workarounds. @@ -204,13 +204,13 @@ control over how the Pages daemon runs and serves content in your environment. | `artifacts_server_url` | API URL to proxy artifact requests to. Defaults to GitLab `external URL` + `/api/v4`, for example `https://gitlab.com/api/v4`. | `auth_redirect_uri` | Callback URL for authenticating with GitLab. Defaults to project's subdomain of `pages_external_url` + `/auth`. | `auth_secret` | Secret key for signing authentication requests. Leave blank to pull automatically from GitLab during OAuth registration. -| `dir` | Working directory for config and secrets files. +| `dir` | Working directory for configuration and secrets files. | `enable` | Enable or disable GitLab Pages on the current system. | `external_http` | Configure Pages to bind to one or more secondary IP addresses, serving HTTP requests. Multiple addresses can be given as an array, along with exact ports, for example `['1.2.3.4', '1.2.3.5:8063']`. Sets value for `listen_http`. | `external_https` | Configure Pages to bind to one or more secondary IP addresses, serving HTTPS requests. Multiple addresses can be given as an array, along with exact ports, for example `['1.2.3.4', '1.2.3.5:8063']`. Sets value for `listen_https`. | `gitlab_client_http_timeout` | GitLab API HTTP client connection timeout in seconds (default: 10s). | `gitlab_client_jwt_expiry` | JWT Token expiry time in seconds (default: 30s). -| `domain_config_source` | Domain configuration source (default: `disk`) +| `domain_config_source` | Domain configuration source (default: `auto`) | `gitlab_id` | The OAuth application public ID. Leave blank to automatically fill when Pages authenticates with GitLab. | `gitlab_secret` | The OAuth application secret. Leave blank to automatically fill when Pages authenticates with GitLab. | `gitlab_server` | Server to use for authentication when access control is enabled; defaults to GitLab `external_url`. @@ -241,7 +241,7 @@ control over how the Pages daemon runs and serves content in your environment. | `pages_path` | The directory on disk where pages are stored, defaults to `GITLAB-RAILS/shared/pages`. | `pages_nginx[]` | | | `enable` | Include a virtual host `server{}` block for Pages inside NGINX. Needed for NGINX to proxy traffic back to the Pages daemon. Set to `false` if the Pages daemon should directly receive all requests, for example, when using [custom domains](index.md#custom-domains). -| `FF_ENABLE_REDIRECTS` | Feature flag to disable redirects (enabled by default). Read the [redirects documentation](../../user/project/pages/redirects.md#disable-redirects) for more info. | +| `FF_ENABLE_REDIRECTS` | Feature flag to disable redirects (enabled by default). Read the [redirects documentation](../../user/project/pages/redirects.md#disable-redirects) for more information. | --- @@ -375,7 +375,7 @@ Pages access control is disabled by default. To enable it: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). 1. Users can now configure it in their [projects' settings](../../user/project/pages/pages_access_control.md). -NOTE: **Important:** +NOTE: For this setting to be effective with multi-node setups, it has to be applied to all the App nodes and Sidekiq nodes. @@ -395,7 +395,7 @@ To do that: 1. Check the **Disable public access to Pages sites** checkbox. 1. Click **Save changes**. -CAUTION: **Warning:** +WARNING: For self-managed installations, all public websites remain private until they are redeployed. This issue will be resolved by [sourcing domain configuration from the GitLab API](https://gitlab.com/gitlab-org/gitlab/-/issues/218357). @@ -427,6 +427,42 @@ Authority (CA) in the system certificate store. For Omnibus, this is fixed by [installing a custom CA in Omnibus GitLab](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). +### Zip serving and cache configuration + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/392) in GitLab 13.7. + +WARNING: +These are advanced settings. The recommended default values are set inside GitLab Pages. You should +change these settings only if absolutely necessary. Use extreme caution. + +GitLab Pages can serve content from zip archives through object storage (an +[issue](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/485) exists for supporting disk storage +as well). It uses an in-memory cache to increase the performance when serving content from a zip +archive. You can modify the cache behavior by changing the following configuration flags. + +| Setting | Description | +| ------- | ----------- | +| `zip_cache_expiration` | The cache expiration interval of zip archives. Must be greater than zero to avoid serving stale content. Default is 60s. | +| `zip_cache_cleanup` | The interval at which archives are cleaned from memory if they have already expired. Default is 30s. | +| `zip_cache_refresh` | The time interval in which an archive is extended in memory if accessed before `zip_cache_expiration`. This works together with `zip_cache_expiration` to determine if an archive is extended in memory. See the [example below](#zip-cache-refresh-example) for important details. Default is 30s. | +| `zip_open_timeout` | The maximum time allowed to open a zip archive. Increase this time for big archives or slow network connections, as doing so may affect the latency of serving Pages. Default is 30s. | + +#### Zip cache refresh example + +Archives are refreshed in the cache (extending the time they are held in memory) if they're accessed +before `zip_cache_expiration`, and the time left before expiring is less than or equal to +`zip_cache_refresh`. For example, if `archive.zip` is accessed at time 0s, it expires in 60s (the +default for `zip_cache_expiration`). In the example below, if the archive is opened again after 15s +it is **not** refreshed because the time left for expiry (45s) is greater than `zip_cache_refresh` +(default 30s). However, if the archive is accessed again after 45s (from the first time it was +opened) it's refreshed. This extends the time the archive remains in memory from +`45s + zip_cache_expiration (60s)`, for a total of 105s. + +After an archive reaches `zip_cache_expiration`, it's marked as expired and removed on the next +`zip_cache_cleanup` interval. + + + ## Activate verbose logging for daemon Verbose logging was [introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/2533) in @@ -539,7 +575,7 @@ your main application server. To configure GitLab Pages on a separate server: -DANGER: **Warning:** +WARNING: The following procedure includes steps to back up and edit the `gitlab-secrets.json` file. This file contains secrets that control database encryption. Proceed with caution. @@ -632,13 +668,14 @@ Pages server. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217912) in GitLab 13.3. GitLab Pages can use different sources to get domain configuration. -The default value is `nil`; however, GitLab Pages will default to `disk`. +The default value is `nil`; however, GitLab Pages will default to `auto`. ```ruby gitlab_pages['domain_config_source'] = nil ``` -You can specify `gitlab` to enable [API-based configuration](#gitlab-api-based-configuration). +If left unchanged, GitLab Pages tries to use any available source (either `gitlab` or `disk`). The +preferred source is `gitlab`, which uses [API-based configuration](#gitlab-api-based-configuration). For more details see this [blog post](https://about.gitlab.com/blog/2020/08/03/how-gitlab-pages-uses-the-gitlab-api-to-serve-content/). @@ -655,15 +692,86 @@ was used prior to GitLab 13.0. Follow these steps to enable it: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -If you encounter an issue, you can disable it by choosing `disk` or `nil`: +If you encounter an issue, you can disable it by choosing `disk`: ```ruby -gitlab_pages['domain_config_source'] = nil +gitlab_pages['domain_config_source'] = "disk" ``` For other common issues, see the [troubleshooting section](#failed-to-connect-to-the-internal-gitlab-api) or report an issue. +## Using object storage + +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5577) in GitLab 13.6. + +[Read more about using object storage with GitLab](../object_storage.md). + +### Object storage settings + +The following settings are: + +- Nested under `pages:` and then `object_store:` on source installations. +- Prefixed by `pages_object_store_` on Omnibus GitLab installations. + +| Setting | Description | Default | +|---------|-------------|---------| +| `enabled` | Whether object storage is enabled. | `false` | +| `remote_directory` | The name of the bucket where Pages site content is stored. | | +| `connection` | Various connection options described below. | | + +#### S3-compatible connection settings + +See [the available connection settings for different providers](../object_storage.md#connection-settings). + +In Omnibus installations: + +1. Add the following lines to `/etc/gitlab/gitlab.rb` and replace the values with the ones you want: + + ```ruby + gitlab_rails['pages_object_store_enabled'] = true + gitlab_rails['pages_object_store_remote_directory'] = "pages" + gitlab_rails['pages_object_store_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-central-1', + 'aws_access_key_id' => 'AWS_ACCESS_KEY_ID', + 'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY' + } + ``` + + If you use AWS IAM profiles, be sure to omit the AWS access key and secret access key/value + pairs: + + ```ruby + gitlab_rails['pages_object_store_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-central-1', + 'use_iam_profile' => true + } + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect. + +In installations from source: + +1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: + + ```yaml + pages: + object_store: + enabled: true + remote_directory: "pages" # The bucket name + connection: + provider: AWS # Only AWS supported at the moment + aws_access_key_id: AWS_ACESS_KEY_ID + aws_secret_access_key: AWS_SECRET_ACCESS_KEY + region: eu-central-1 + ``` + +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) + for the changes to take effect. + ## Backup GitLab Pages are part of the [regular backup](../../raketasks/backup_restore.md), so there is no separate backup to configure. @@ -734,7 +842,7 @@ x509: certificate signed by unknown authority ``` The reason for those errors is that the files `resolv.conf` and `ca-bundle.pem` are missing inside the chroot. -The fix is to copy the host's `/etc/resolv.conf` and GitLab's certificate bundle inside the chroot: +The fix is to copy the host's `/etc/resolv.conf` and the GitLab certificate bundle inside the chroot: ```shell sudo mkdir -p /var/opt/gitlab/gitlab-rails/shared/pages/etc/ssl @@ -811,3 +919,11 @@ Upgrading to an [officially supported operating system](https://about.gitlab.com This problem comes from the permissions of the GitLab Pages OAuth application. To fix it, go to **Admin > Applications > GitLab Pages** and edit the application. Under **Scopes**, ensure that the `api` scope is selected and save your changes. + +### Workaround in case no wildcard DNS entry can be set + +If the wildcard DNS [prerequisite](#prerequisites) can't be met, you can still use GitLab Pages in a limited fashion: + +1. [Move](../../user/project/settings/index.md#transferring-an-existing-project-into-another-namespace) + all projects you need to use Pages with into a single group namespace, for example `pages`. +1. Configure a [DNS entry](#dns-configuration) without the `*.`-wildcard, for example `pages.example.io`. diff --git a/doc/administration/pages/source.md b/doc/administration/pages/source.md index 87217b141a4..c7c25f0f3a7 100644 --- a/doc/administration/pages/source.md +++ b/doc/administration/pages/source.md @@ -1,12 +1,12 @@ --- stage: Release -group: Release Management -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +group: Release +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # GitLab Pages administration for source installations -NOTE: **Note:** +NOTE: Before attempting to enable GitLab Pages, first make sure you have [installed GitLab](../../install/installation.md) successfully. @@ -77,7 +77,7 @@ host that GitLab runs. For example, an entry would look like this: where `example.io` is the domain under which GitLab Pages will be served and `192.0.2.1` is the IP address of your GitLab instance. -NOTE: **Note:** +NOTE: You should not use the GitLab domain to serve user pages. For more information see the [security section](#security). @@ -349,7 +349,7 @@ world. Custom domains and TLS are supported. ## NGINX caveats -NOTE: **Note:** +NOTE: The following information applies only for installations from source. Be extra careful when setting up the domain name in the NGINX configuration. You must @@ -395,7 +395,7 @@ API to check that the user is authorized to read that site. From [GitLab 12.8](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/3689) onward, Access Control parameters for Pages are set in a configuration file, which by convention is named `gitlab-pages-config`. The configuration file is passed to -pages using the `-config flag` or CONFIG environment variable. +pages using the `-config flag` or `CONFIG` environment variable. Pages access control is disabled by default. To enable it: diff --git a/doc/administration/plugins.md b/doc/administration/plugins.md index dbb733b9b19..f5d4b3aa2ff 100644 --- a/doc/administration/plugins.md +++ b/doc/administration/plugins.md @@ -3,3 +3,6 @@ redirect_to: 'file_hooks.md' --- This document was moved to [File Hooks](file_hooks.md), after the Plugins feature was renamed to File Hooks. + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/polling.md b/doc/administration/polling.md index 17b8045b51f..cd2e8ecad60 100644 --- a/doc/administration/polling.md +++ b/doc/administration/polling.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Polling configuration @@ -9,7 +9,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w The GitLab UI polls for updates for different resources (issue notes, issue titles, pipeline statuses, etc.) on a schedule appropriate to the resource. -In "Application settings -> Real-time features" you can configure "Polling +In **[Admin Area](../user/admin_area/index.md) > Settings > Preferences > Real-time features**, +you can configure "Polling interval multiplier". This multiplier is applied to all resources at once, and decimal values are supported. For the sake of the examples below, we will say that issue notes poll every 2 seconds, and issue titles poll every 5 diff --git a/doc/administration/postgresql/external.md b/doc/administration/postgresql/external.md index 4a164c66578..a9d0af952a0 100644 --- a/doc/administration/postgresql/external.md +++ b/doc/administration/postgresql/external.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Configure GitLab using an external PostgreSQL service diff --git a/doc/administration/postgresql/index.md b/doc/administration/postgresql/index.md index c7ec46db654..0e6fd757421 100644 --- a/doc/administration/postgresql/index.md +++ b/doc/administration/postgresql/index.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- diff --git a/doc/administration/postgresql/pgbouncer.md b/doc/administration/postgresql/pgbouncer.md index 7760b197267..f09ac3052f4 100644 --- a/doc/administration/postgresql/pgbouncer.md +++ b/doc/administration/postgresql/pgbouncer.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -36,7 +36,7 @@ This content has been moved to a [new location](replication_and_failover.md#conf 1. Run `gitlab-ctl reconfigure` - NOTE: **Note:** + NOTE: If the database was already running, it will need to be restarted after reconfigure by running `gitlab-ctl restart postgresql`. 1. On the node you are running PgBouncer on, make sure the following is set in `/etc/gitlab/gitlab.rb` diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index a29815672e5..303246c9c82 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # PostgreSQL replication and failover with Omnibus GitLab **(PREMIUM ONLY)** @@ -35,7 +35,7 @@ You also need to take into consideration the underlying network topology, making sure you have redundant connectivity between all Database and GitLab instances to avoid the network becoming a single point of failure. -NOTE: **Note:** +NOTE: As of GitLab 13.3, PostgreSQL 12 is shipped with Omnibus GitLab. Clustering for PostgreSQL 12 is only supported with Patroni. See the [Patroni](#patroni) section for further details. The support for repmgr will not be extended beyond PostgreSQL 11. @@ -460,7 +460,7 @@ Check the [Troubleshooting section](#troubleshooting) before proceeding. # END user configuration ``` - NOTE: **Note:** + NOTE: `pgbouncer_role` was introduced with GitLab 10.3. 1. Run `gitlab-ctl reconfigure` @@ -985,7 +985,7 @@ after it has been restored to service. gitlab-ctl restart repmgrd ``` - CAUTION: **Warning:** + WARNING: When the server is brought back online, and before you switch it to a standby node, repmgr will report that there are two masters. If there are any clients that are still attempting to write to the old master, @@ -1148,7 +1148,7 @@ If you're running into an issue with a component not outlined here, be sure to c ## Patroni -NOTE: **Note:** +NOTE: Starting from GitLab 13.1, Patroni is available for **experimental** use to replace repmgr. Due to its experimental nature, Patroni support is **subject to change without notice.** @@ -1326,7 +1326,7 @@ For further details, see [Patroni documentation on this subject](https://patroni ### Switching from repmgr to Patroni -CAUTION: **Warning:** +WARNING: Although switching from repmgr to Patroni is fairly straightforward the other way around is not. Rolling back from Patroni to repmgr can be complicated and may involve deletion of data directory. If you need to do that, please contact GitLab support. @@ -1345,7 +1345,7 @@ You can switch an exiting database cluster to use Patroni instead of repmgr with sudo gitlab-ctl stop postgresql ``` - NOTE: **Note:** + NOTE: Ensure that there is no `walsender` process running on the primary node. `ps aux | grep walsender` must not show any running process. @@ -1367,7 +1367,7 @@ As of GitLab 13.3, PostgreSQL 11.7 and 12.3 are both shipped with Omnibus GitLab uses PostgreSQL 11 by default. Therefore `gitlab-ctl pg-upgrade` does not automatically upgrade to PostgreSQL 12. If you want to upgrade to PostgreSQL 12, you must ask for it explicitly. -CAUTION: **Warning:** +WARNING: The procedure for upgrading PostgreSQL in a Patroni cluster is different than when upgrading using repmgr. The following outlines the key differences and important considerations that need to be accounted for when upgrading PostgreSQL. @@ -1401,7 +1401,7 @@ Considering these, you should carefully plan your PostgreSQL upgrade: gitlab-ctl patroni members ``` - NOTE: **Note:** + NOTE: `gitlab-ctl pg-upgrade` tries to detect the role of the node. If for any reason the auto-detection does not work or you believe it did not detect the role correctly, you can use the `--leader` or `--replica` arguments to manually override it. @@ -1446,7 +1446,7 @@ Considering these, you should carefully plan your PostgreSQL upgrade: sudo gitlab-ctl pg-upgrade -V 12 ``` -NOTE: **Note:** +NOTE: Reverting PostgreSQL upgrade with `gitlab-ctl revert-pg-upgrade` has the same considerations as `gitlab-ctl pg-upgrade`. You should follow the same procedure by first stopping the replicas, then reverting the leader, and finally reverting the replicas. diff --git a/doc/administration/postgresql/standalone.md b/doc/administration/postgresql/standalone.md index 2ac74e8a4a0..30495824cd2 100644 --- a/doc/administration/postgresql/standalone.md +++ b/doc/administration/postgresql/standalone.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Standalone PostgreSQL using Omnibus GitLab **(CORE ONLY)** @@ -59,7 +59,7 @@ together with Omnibus GitLab. This is recommended as part of our gitlab_rails['auto_migrate'] = false ``` - NOTE: **Note:** + NOTE: The role `postgres_role` was introduced with GitLab 10.3 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. diff --git a/doc/administration/pseudonymizer.md b/doc/administration/pseudonymizer.md index 41a7ec087ac..5f1272b1f4a 100644 --- a/doc/administration/pseudonymizer.md +++ b/doc/administration/pseudonymizer.md @@ -1,18 +1,18 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Pseudonymizer **(ULTIMATE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/5532) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.1. -As GitLab's database hosts sensitive information, using it unfiltered for analytics +As the GitLab database hosts sensitive information, using it unfiltered for analytics implies high security requirements. To help alleviate this constraint, the Pseudonymizer -service is used to export GitLab's data in a pseudonymized way. +service is used to export GitLab data in a pseudonymized way. -CAUTION: **Warning:** +WARNING: This process is not impervious. If the source data is available, it's possible for a user to correlate data to the pseudonymized version. @@ -28,7 +28,7 @@ To configure the pseudonymizer, you need to: - Provide a manifest file that describes which fields should be included or pseudonymized ([example `manifest.yml` file](https://gitlab.com/gitlab-org/gitlab/tree/master/config/pseudonymizer.yml)). - A default manifest is provided with the GitLab installation. Using a relative file path will be resolved from the Rails root. + A default manifest is provided with the GitLab installation, using a relative file path that resolves from the Rails root. Alternatively, you can use an absolute file path. - Use an object storage and specify the connection parameters in the `pseudonymizer.upload.connection` configuration option. @@ -100,7 +100,7 @@ sudo gitlab-rake gitlab:db:pseudonymizer sudo -u git -H bundle exec rake gitlab:db:pseudonymizer RAILS_ENV=production ``` -This will produce some CSV files that might be very large, so make sure the +This produces some CSV files that might be very large, so make sure the `PSEUDONYMIZER_OUTPUT_DIR` has sufficient space. As a rule of thumb, at least 10% of the database size is recommended. diff --git a/doc/administration/raketasks/check.md b/doc/administration/raketasks/check.md index f61a3107b3d..25869fd425d 100644 --- a/doc/administration/raketasks/check.md +++ b/doc/administration/raketasks/check.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Integrity check Rake task **(CORE ONLY)** @@ -56,7 +56,7 @@ sudo -u git -H bundle exec rake gitlab:git:fsck RAILS_ENV=production Various types of files can be uploaded to a GitLab installation by users. These integrity checks can detect missing files. Additionally, for locally stored files, checksums are generated and stored in the database upon upload, -and these checks will verify them against current files. +and these checks verify them against current files. Currently, integrity checks are supported for the following types of file: @@ -137,8 +137,8 @@ Done! ## LDAP check -The LDAP check Rake task will test the bind DN and password credentials -(if configured) and will list a sample of LDAP users. This task is also +The LDAP check Rake task tests the bind DN and password credentials +(if configured) and lists a sample of LDAP users. This task is also executed as part of the `gitlab:check` task, but can run independently. See [LDAP Rake Tasks - LDAP Check](ldap.md#check) for details. diff --git a/doc/administration/raketasks/doctor.md b/doc/administration/raketasks/doctor.md index e6a6751f199..c80f580cce6 100644 --- a/doc/administration/raketasks/doctor.md +++ b/doc/administration/raketasks/doctor.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Doctor Rake tasks **(CORE ONLY)** diff --git a/doc/administration/raketasks/geo.md b/doc/administration/raketasks/geo.md index 0ee6fd8fc75..492fe4b52ca 100644 --- a/doc/administration/raketasks/geo.md +++ b/doc/administration/raketasks/geo.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Geo Rake Tasks **(PREMIUM ONLY)** diff --git a/doc/administration/raketasks/github_import.md b/doc/administration/raketasks/github_import.md index d549110c32f..5c0bc721a9a 100644 --- a/doc/administration/raketasks/github_import.md +++ b/doc/administration/raketasks/github_import.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # GitHub import **(CORE ONLY)** @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w To retrieve and import GitHub repositories, you need a [GitHub personal access token](https://github.com/settings/tokens). A username should be passed as the second argument to the Rake task, -which will become the owner of the project. You can resume an import +which becomes the owner of the project. You can resume an import with the same command. Bear in mind that the syntax is very specific. Remove any spaces within the argument block and @@ -20,7 +20,7 @@ before/after the brackets. Also, some shells (for example, `zsh`) can interpret ## Caveats If the GitHub [rate limit](https://developer.github.com/v3/#rate-limiting) is reached while importing, -the importing process will wait (`sleep()`) until it can continue importing. +the importing process waits (`sleep()`) until it can continue importing. ## Importing multiple projects @@ -35,8 +35,8 @@ bundle exec rake "import:github[access_token,root,foo/bar]" RAILS_ENV=production ``` In this case, `access_token` is your GitHub personal access token, `root` -is your GitLab username, and `foo/bar` is the new GitLab namespace/project that -will get created from your GitHub project. Subgroups are also possible: `foo/foo/bar`. +is your GitLab username, and `foo/bar` is the new GitLab namespace/project +created from your GitHub project. Subgroups are also possible: `foo/foo/bar`. ## Importing a single project diff --git a/doc/administration/raketasks/ldap.md b/doc/administration/raketasks/ldap.md index 821a72291fd..7dafda89e3c 100644 --- a/doc/administration/raketasks/ldap.md +++ b/doc/administration/raketasks/ldap.md @@ -1,7 +1,7 @@ --- stage: Manage group: Access -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # LDAP Rake tasks **(CORE ONLY)** @@ -42,7 +42,7 @@ The following task will run a [group sync](../auth/ldap/index.md#group-sync) imm when you'd like to update all configured group memberships against LDAP without waiting for the next scheduled group sync to be run. -NOTE: **Note:** +NOTE: If you'd like to change the frequency at which a group sync is performed, [adjust the cron schedule](../auth/ldap/index.md#adjusting-ldap-group-sync-schedule) instead. @@ -147,3 +147,96 @@ confirmation dialog: ```shell sudo gitlab-rake gitlab:ldap:rename_provider[old_provider,new_provider] force=yes ``` + +## Secrets + +GitLab can use [LDAP configuration secrets](../auth/ldap/index.md#using-encrypted-credentials) to read from an encrypted file. The following Rake tasks are provided for updating the contents of the encrypted file. + +### Show secret + +Show the contents of the current LDAP secrets. + +**Omnibus Installation** + +```shell +sudo gitlab-rake gitlab:ldap:secret:show +``` + +**Source Installation** + +```shell +bundle exec rake gitlab:ldap:secret:show RAILS_ENV=production +``` + +**Example output:** + +```plaintext +main: + password: '123' + user_bn: 'gitlab-adm' +``` + +### Edit secret + +Opens the secret contents in your editor, and writes the resulting content to the encrypted secret file when you exit. + +**Omnibus Installation** + +```shell +sudo gitlab-rake gitlab:ldap:secret:edit EDITOR=vim +``` + +**Source Installation** + +```shell +bundle exec rake gitlab:ldap:secret:edit RAILS_ENV=production EDITOR=vim +``` + +### Write raw secret + +Write new secret content by providing it on STDIN. + +**Omnibus Installation** + +```shell +echo -e "main:\n password: '123'" | sudo gitlab-rake gitlab:ldap:secret:write +``` + +**Source Installation** + +```shell +echo -e "main:\n password: '123'" | bundle exec rake gitlab:ldap:secret:write RAILS_ENV=production +``` + +### Secrets examples + +**Editor example** + +The write task can be used in cases where the edit command does not work with your editor: + +```shell +# Write the existing secret to a plaintext file +sudo gitlab-rake gitlab:ldap:secret:show > ldap.yaml +# Edit the ldap file in your editor +... +# Re-encrypt the file +cat ldap.yaml | sudo gitlab-rake gitlab:ldap:secret:write +# Remove the plaintext file +rm ldap.yaml +``` + +**KMS integration example** + +It can also be used as a receiving application for content encrypted with a KMS: + +```shell +gcloud kms decrypt --key my-key --keyring my-test-kms --plaintext-file=- --ciphertext-file=my-file --location=us-west1 | sudo gitlab-rake gitlab:ldap:secret:write +``` + +**gcloud secret integration example** + +It can also be used as a receiving application for secrets out of gcloud: + +```shell +gcloud secrets versions access latest --secret="my-test-secret" > $1 | sudo gitlab-rake gitlab:ldap:secret:write +``` diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md index b93442be0a1..26381434ad4 100644 --- a/doc/administration/raketasks/maintenance.md +++ b/doc/administration/raketasks/maintenance.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Maintenance Rake tasks **(CORE ONLY)** @@ -107,8 +107,8 @@ The `gitlab:check` Rake task runs the following Rake tasks: - `gitlab:sidekiq:check` - `gitlab:app:check` -It will check that each component was set up according to the installation guide and suggest fixes -for issues found. This command must be run from your application server and will not work correctly on +It checks that each component was set up according to the installation guide and suggest fixes +for issues found. This command must be run from your application server and doesn't work correctly on component servers like [Gitaly](../gitaly/index.md#run-gitaly-on-its-own-server). You may also have a look at our troubleshooting guides for: @@ -272,7 +272,7 @@ clear it. To clear all exclusive leases: -DANGER: **Warning:** +WARNING: Don't run it while GitLab or Sidekiq is running ```shell @@ -300,7 +300,7 @@ To check the status of specific migrations, you can use the following Rake task: sudo gitlab-rake db:migrate:status ``` -This will output a table with a `Status` of `up` or `down` for +This outputs a table with a `Status` of `up` or `down` for each Migration ID. ```shell @@ -313,7 +313,7 @@ database: gitlabhq_production ## Run incomplete database migrations -Database migrations can be stuck in an incomplete state. That is, they'll have a `down` +Database migrations can be stuck in an incomplete state, with a `down` status in the output of the `sudo gitlab-rake db:migrate:status` command. To complete these migrations, use the following Rake task: diff --git a/doc/administration/raketasks/praefect.md b/doc/administration/raketasks/praefect.md index ca9c2065904..17c3e44bb7e 100644 --- a/doc/administration/raketasks/praefect.md +++ b/doc/administration/raketasks/praefect.md @@ -1,7 +1,7 @@ --- stage: Create group: Gitaly -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- diff --git a/doc/administration/raketasks/project_import_export.md b/doc/administration/raketasks/project_import_export.md index d83ab6e5cee..0ea0bb3c28f 100644 --- a/doc/administration/raketasks/project_import_export.md +++ b/doc/administration/raketasks/project_import_export.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Project import/export administration **(CORE ONLY)** @@ -36,7 +36,7 @@ sudo gitlab-rake gitlab:import_export:version bundle exec rake gitlab:import_export:version RAILS_ENV=production ``` -The current list of DB tables that will be exported can be listed by using the following command: +The current list of DB tables to export can be listed by using the following command: ```shell # Omnibus installations diff --git a/doc/administration/raketasks/storage.md b/doc/administration/raketasks/storage.md index 9b15f5ed4de..c7afebf15bd 100644 --- a/doc/administration/raketasks/storage.md +++ b/doc/administration/raketasks/storage.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Repository storage Rake tasks **(CORE ONLY)** @@ -74,7 +74,7 @@ To have a summary and then a list of projects and their attachments using hashed ## Migrate to hashed storage -DANGER: **Deprecated:** +WARNING: In GitLab 13.0, [hashed storage](../repository_storage_types.md#hashed-storage) is enabled by default and the legacy storage is deprecated. Support for legacy storage will be removed in GitLab 14.0. If you're on GitLab @@ -82,8 +82,9 @@ Support for legacy storage will be removed in GitLab 14.0. If you're on GitLab The option to choose between hashed and legacy storage in the admin area has been disabled. -This task will schedule all your existing projects and attachments associated -with it to be migrated to the **Hashed** storage type: +This task must be run on any machine that has Rails/Sidekiq configured and will +schedule all your existing projects and attachments associated with it to be +migrated to the **Hashed** storage type: - **Omnibus installation** @@ -123,7 +124,7 @@ commands below that helps you inspect projects and attachments in both legacy an ## Rollback from hashed storage to legacy storage -DANGER: **Deprecated:** +WARNING: In GitLab 13.0, [hashed storage](../repository_storage_types.md#hashed-storage) is enabled by default and the legacy storage is deprecated. Support for legacy storage will be removed in GitLab 14.0. If you're on GitLab diff --git a/doc/administration/raketasks/uploads/migrate.md b/doc/administration/raketasks/uploads/migrate.md index 075b27fb70d..2f51bf9357f 100644 --- a/doc/administration/raketasks/uploads/migrate.md +++ b/doc/administration/raketasks/uploads/migrate.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Uploads migrate Rake tasks **(CORE ONLY)** @@ -13,10 +13,10 @@ There is a Rake task for migrating uploads between different storage types. ## Migrate to object storage -After [configuring the object storage](../../uploads.md#using-object-storage) for GitLab's -uploads, use this task to migrate existing uploads from the local storage to the remote storage. +After [configuring the object storage](../../uploads.md#using-object-storage) for uploads +to GitLab, use this task to migrate existing uploads from the local storage to the remote storage. -All of the processing will be done in a background worker and requires **no downtime**. +All of the processing is done in a background worker and requires **no downtime**. Read more about using [object storage with GitLab](../../object_storage.md). @@ -55,8 +55,8 @@ The Rake task uses three parameters to find uploads to migrate: | `model_class` | string | Type of the model to migrate from. | | `mount_point` | string/symbol | Name of the model's column the uploader is mounted on. | -NOTE: **Note:** -These parameters are mainly internal to GitLab's structure, you may want to refer to the task list +NOTE: +These parameters are mainly internal to the structure of GitLab, you may want to refer to the task list instead below. This task also accepts an environment variable which you can use to override @@ -131,9 +131,9 @@ sudo -u git -H bundle exec rake "gitlab:uploads:migrate[DesignManagement::Design If you need to disable [object storage](../../object_storage.md) for any reason, you must first migrate your data out of object storage and back into your local storage. -CAUTION: **Warning:** +WARNING: **Extended downtime is required** so no new files are created in object storage during -the migration. A configuration setting will be added soon to allow migrating +the migration. A configuration setting is planned to allow migrating from object storage to local files with only a brief moment of downtime for configuration changes. To follow progress, see the [relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/30979). diff --git a/doc/administration/raketasks/uploads/sanitize.md b/doc/administration/raketasks/uploads/sanitize.md index 4f7c99babd8..637992d52ca 100644 --- a/doc/administration/raketasks/uploads/sanitize.md +++ b/doc/administration/raketasks/uploads/sanitize.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Uploads sanitize Rake tasks **(CORE ONLY)** @@ -41,8 +41,8 @@ The Rake task accepts following parameters. | Parameter | Type | Description | |:-------------|:--------|:----------------------------------------------------------------------------------------------------------------------------| -| `start_id` | integer | Only uploads with equal or greater ID will be processed | -| `stop_id` | integer | Only uploads with equal or smaller ID will be processed | +| `start_id` | integer | Only uploads with equal or greater ID are processed | +| `stop_id` | integer | Only uploads with equal or smaller ID are processed | | `dry_run` | boolean | Do not remove EXIF data, only check if EXIF data are present or not. Defaults to `true` | | `sleep_time` | float | Pause for number of seconds after processing each image. Defaults to 0.3 seconds | | `uploader` | string | Run sanitization only for uploads of the given uploader: `FileUploader`, `PersonalFileUploader`, or `NamespaceFileUploader` | @@ -66,7 +66,7 @@ To remove EXIF data on uploads with an ID between 100 and 5000 and pause for 0.1 sudo RAILS_ENV=production -u git -H bundle exec rake gitlab:uploads:sanitize:remove_exif[100,5000,false,0.1] 2>&1 | tee exif.log ``` -The output is written into an `exif.log` file because it will probably be long. +The output is written into an `exif.log` file because it is often long. If sanitization fails for an upload, an error message should be in the output of the Rake task. Typical reasons include that the file is missing in the storage or it's not a valid image. diff --git a/doc/administration/read_only_gitlab.md b/doc/administration/read_only_gitlab.md index 681102a8c39..d1cb53d0090 100644 --- a/doc/administration/read_only_gitlab.md +++ b/doc/administration/read_only_gitlab.md @@ -1,12 +1,12 @@ --- stage: Enablement group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Place GitLab into a read-only state **(CORE ONLY)** -CAUTION: **Warning:** +WARNING: This document should be used as a temporary solution. There's work in progress to make this [possible with Geo](https://gitlab.com/groups/gitlab-org/-/epics/2149). diff --git a/doc/administration/redis/index.md b/doc/administration/redis/index.md index 0bd56666ab8..30618075c8f 100644 --- a/doc/administration/redis/index.md +++ b/doc/administration/redis/index.md @@ -2,7 +2,7 @@ type: index stage: Enablement group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Configuring Redis for scaling diff --git a/doc/administration/redis/replication_and_failover.md b/doc/administration/redis/replication_and_failover.md index 72a52ba0a1f..1abc52b65cc 100644 --- a/doc/administration/redis/replication_and_failover.md +++ b/doc/administration/redis/replication_and_failover.md @@ -2,16 +2,16 @@ type: howto stage: Enablement group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Redis replication and failover with Omnibus GitLab **(PREMIUM ONLY)** -NOTE: **Note:** +NOTE: This is the documentation for the Omnibus GitLab packages. For using your own non-bundled Redis, follow the [relevant documentation](replication_and_failover_external.md). -NOTE: **Note:** +NOTE: In Redis lingo, primary is called master. In this document, primary is used instead of master, except the settings where `master` is required. @@ -162,7 +162,7 @@ is not achieved (see the odd number of nodes requirement above). In that case, a new attempt will be made after the amount of time defined in `sentinel['failover_timeout']` (in milliseconds). -NOTE: **Note:** +NOTE: We will see where `sentinel['failover_timeout']` is defined later. The `failover_timeout` variable has a lot of different use cases. According to @@ -194,7 +194,7 @@ It is assumed that you have installed GitLab and all its components from scratch If you already have Redis installed and running, read how to [switch from a single-machine installation](#switching-from-an-existing-single-machine-installation). -NOTE: **Note:** +NOTE: Redis nodes (both primary and replica) will need the same password defined in `redis['password']`. At any time during a failover the Sentinels can reconfigure a node and change its status from primary to replica and vice versa. @@ -277,7 +277,7 @@ If you fail to replicate first, you may loose data (unprocessed background jobs) 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -NOTE: **Note:** +NOTE: You can specify multiple roles like sentinel and Redis as: `roles ['redis_sentinel_role', 'redis_master_role']`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). @@ -327,7 +327,7 @@ Read more about [roles](https://docs.gitlab.com/omnibus/roles/). 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Go through the steps again for all the other replica nodes. -NOTE: **Note:** +NOTE: You can specify multiple roles like sentinel and Redis as: `roles ['redis_sentinel_role', 'redis_master_role']`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). @@ -339,7 +339,7 @@ the same Sentinels. ### Step 3. Configuring the Redis Sentinel instances -NOTE: **Note:** +NOTE: If you are using an external Redis Sentinel instance, be sure to exclude the `requirepass` parameter from the Sentinel configuration. This parameter will cause clients to report `NOAUTH @@ -465,7 +465,7 @@ the correct credentials for the Sentinel nodes. While it doesn't require a list of all Sentinel nodes, in case of a failure, it needs to access at least one of the listed. -NOTE: **Note:** +NOTE: The following steps should be performed in the GitLab application server which ideally should not have Redis or Sentinels on it for a HA setup. @@ -690,7 +690,7 @@ To make this work with Sentinel: sudo gitlab-ctl reconfigure ``` -NOTE: **Note:** +NOTE: For each persistence class, GitLab will default to using the configuration specified in `gitlab_rails['redis_sentinels']` unless overridden by the previously described settings. diff --git a/doc/administration/redis/replication_and_failover_external.md b/doc/administration/redis/replication_and_failover_external.md index 7561f1c0fbf..3bcefe2ef13 100644 --- a/doc/administration/redis/replication_and_failover_external.md +++ b/doc/administration/redis/replication_and_failover_external.md @@ -2,7 +2,7 @@ type: howto stage: Enablement group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Redis replication and failover providing your own instance **(CORE ONLY)** diff --git a/doc/administration/redis/standalone.md b/doc/administration/redis/standalone.md index ea5a7850244..0e7eca84d17 100644 --- a/doc/administration/redis/standalone.md +++ b/doc/administration/redis/standalone.md @@ -2,7 +2,7 @@ type: howto stage: Enablement group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Standalone Redis using Omnibus GitLab **(CORE ONLY)** diff --git a/doc/administration/redis/troubleshooting.md b/doc/administration/redis/troubleshooting.md index c9976ac791d..144660c50ea 100644 --- a/doc/administration/redis/troubleshooting.md +++ b/doc/administration/redis/troubleshooting.md @@ -2,7 +2,7 @@ type: reference stage: Enablement group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Troubleshooting Redis diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 9f522e0d599..216d3867d0a 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -2,7 +2,7 @@ reading_time: true stage: Enablement group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Reference architecture: up to 10,000 users **(PREMIUM ONLY)** @@ -24,8 +24,8 @@ full list of reference architectures, see | Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | | Redis - Cache | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | | Redis - Queues / Shared State | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | t2.small | B1MS | -| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | t2.small | B1MS | +| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | t3.small | B1MS | +| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | t3.small | B1MS | | Gitaly | 2 (minimum) | 16 vCPU, 60 GB memory | n1-standard-16 | m5.4xlarge | D16s v3 | | Sidekiq | 4 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | | GitLab Rails | 3 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | @@ -33,6 +33,81 @@ full list of reference architectures, see | Object storage | n/a | n/a | n/a | n/a | n/a | | NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +```mermaid +stateDiagram-v2 + [*] --> LoadBalancer + LoadBalancer --> ApplicationServer + + ApplicationServer --> BackgroundJobs + ApplicationServer --> Gitaly + ApplicationServer --> Redis_Cache + ApplicationServer --> Redis_Queues + ApplicationServer --> PgBouncer + PgBouncer --> Database + ApplicationServer --> ObjectStorage + BackgroundJobs --> ObjectStorage + + ApplicationMonitoring -->ApplicationServer + ApplicationMonitoring -->PgBouncer + ApplicationMonitoring -->Database + ApplicationMonitoring -->BackgroundJobs + + ApplicationServer --> Consul + + Consul --> Database + Consul --> PgBouncer + Redis_Cache --> Consul + Redis_Queues --> Consul + BackgroundJobs --> Consul + + state Consul { + "Consul_1..3" + } + + state Database { + "PG_Primary_Node" + "PG_Secondary_Node_1..2" + } + + state Redis_Cache { + "R_Cache_Primary_Node" + "R_Cache_Replica_Node_1..2" + "R_Cache_Sentinel_1..3" + } + + state Redis_Queues { + "R_Queues_Primary_Node" + "R_Queues_Replica_Node_1..2" + "R_Queues_Sentinel_1..3" + } + + state Gitaly { + "Gitaly_1..2" + } + + state BackgroundJobs { + "Sidekiq_1..4" + } + + state ApplicationServer { + "GitLab_Rails_1..3" + } + + state LoadBalancer { + "LoadBalancer_1" + } + + state ApplicationMonitoring { + "Prometheus" + "Grafana" + } + + state PgBouncer { + "Internal_Load_Balancer" + "PgBouncer_1..3" + } +``` + The Google Cloud Platform (GCP) architectures were built and tested using the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) CPU platform. On different hardware you may find that adjustments, either lower @@ -241,7 +316,7 @@ The following IPs will be used as an example: - `10.6.0.12`: Consul 2 - `10.6.0.13`: Consul 3 -NOTE: **Note:** +NOTE: The configuration processes for the other servers in your reference architecture will use the `/etc/gitlab/gitlab-secrets.json` file from your Consul server to connect with the other servers. @@ -962,7 +1037,7 @@ servers. The following IPs will be used as an example: - `10.6.0.72`: Sentinel - Cache 2 - `10.6.0.73`: Sentinel - Cache 3 -NOTE: **Note:** +NOTE: If you're using an external Redis Sentinel instance, be sure to exclude the `requirepass` parameter from the Sentinel configuration. This parameter causes clients to report `NOAUTH Authentication required.`. @@ -1226,7 +1301,7 @@ servers. The following IPs will be used as an example: - `10.6.0.82`: Sentinel - Queues 2 - `10.6.0.83`: Sentinel - Queues 3 -NOTE: **Note:** +NOTE: If you're using an external Redis Sentinel instance, be sure to exclude the `requirepass` parameter from the Sentinel configuration. This parameter causes clients to report `NOAUTH Authentication required.`. @@ -1347,7 +1422,7 @@ To configure the Sentinel Queues server: ## Configure Gitaly -NOTE: **Note:** +NOTE: [Gitaly Cluster](../gitaly/praefect.md) support for the Reference Architectures is being worked on as a [collaborative effort](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/1) between the Quality Engineering and Gitaly teams. When this component has been verified @@ -1356,19 +1431,17 @@ and improved designed. [Gitaly](../gitaly/index.md) server node requirements are dependent on data, specifically the number of projects and those projects' sizes. It's recommended -that a Gitaly server node stores no more than 5 TB of data. Although this -reference architecture includes a recommendation for the number of Gitaly server -nodes to use, depending on your storage requirements, you may require additional -Gitaly server nodes. +that a Gitaly server node stores no more than 5 TB of data. Depending on your +repository storage requirements, you may require additional Gitaly server nodes. Due to Gitaly having notable input and output requirements, we strongly -recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs should -have a throughput of at least 8,000 input/output operations per second (IOPS) -for read operations and 2,000 IOPS for write operations. These IOPS values are -initial recommendations, and may be adjusted to greater or lesser values -depending on the scale of your environment's workload. If you're running the -environment on a Cloud provider, refer to their documentation about how to -configure IOPS correctly. +recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs +should have a throughput of at least 8,000 +input/output operations per second (IOPS) for read operations and 2,000 IOPS for +write operations. These IOPS values are initial recommendations, and may be +adjusted to greater or lesser values depending on the scale of your +environment's workload. If you're running the environment on a Cloud provider, +refer to their documentation about how to configure IOPS correctly. Be sure to note the following items: @@ -1376,14 +1449,14 @@ Be sure to note the following items: [repository storage paths](../repository_storage_paths.md). - A Gitaly server can host one or more storage paths. - A GitLab server can use one or more Gitaly server nodes. -- Gitaly addresses must be specified to be correctly resolvable for _all_ - Gitaly clients. +- Gitaly addresses must be specified to be correctly resolvable for all Gitaly + clients. - Gitaly servers must not be exposed to the public internet, as Gitaly's network traffic is unencrypted by default. The use of a firewall is highly recommended to restrict access to the Gitaly server. Another option is to [use TLS](#gitaly-tls-support). -NOTE: **Note:** +NOTE: The token referred to throughout the Gitaly documentation is an arbitrary password selected by the administrator. This token is unrelated to tokens created for the GitLab API or other similar web API tokens. @@ -1406,8 +1479,8 @@ On each node: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page, and _do not_ provide the `EXTERNAL_URL` value. -1. Edit `/etc/gitlab/gitlab.rb` to configure the storage paths, enable - the network listener, and configure the token: +1. Edit the Gitaly server node's `/etc/gitlab/gitlab.rb` file to configure + storage paths, enable the network listener, and to configure the token: <!-- updates to following example must also be made at @@ -1501,7 +1574,7 @@ nodes (including the Gitaly node using the certificate) and on all client nodes that communicate with it following the procedure described in [GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). -NOTE: **Note:** +NOTE: The self-signed certificate must specify the address you use to access the Gitaly server. If you are addressing the Gitaly server by a hostname, you can either use the Common Name field for this, or add it as a Subject Alternative @@ -1677,7 +1750,7 @@ To configure the Sidekiq nodes, on each one: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -TIP: **Tip:** +NOTE: You can also run [multiple Sidekiq processes](../operations/extra_sidekiq_processes.md). <div align="right"> @@ -1690,12 +1763,6 @@ You can also run [multiple Sidekiq processes](../operations/extra_sidekiq_proces This section describes how to configure the GitLab application (Rails) component. -In our architecture, we run each GitLab Rails node using the Puma webserver, and -have its number of workers set to 90% of available CPUs, with four threads. For -nodes running Rails with other components, the worker value should be reduced -accordingly. We've determined that a worker value of 50% achieves a good balance, -but this is dependent on workload. - The following IPs will be used as an example: - `10.6.0.111`: GitLab application 1 @@ -1999,7 +2066,7 @@ on what features you intend to use: |Object storage type|Supported by consolidated configuration?| |-------------------|----------------------------------------| -| [Backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage)|No| +| [Backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage) | No | | [Job artifacts](../job_artifacts.md#using-object-storage) including archived job logs | Yes | | [LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | | [Uploads](../uploads.md#using-object-storage) | Yes | diff --git a/doc/administration/reference_architectures/1k_users.md b/doc/administration/reference_architectures/1k_users.md index 0cb7b8868c3..5b75c7ac9c4 100644 --- a/doc/administration/reference_architectures/1k_users.md +++ b/doc/administration/reference_architectures/1k_users.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Reference architecture: up to 1,000 users **(CORE ONLY)** diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index b106f7bced1..c5a8c3927c0 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -2,7 +2,7 @@ reading_time: true stage: Enablement group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Reference architecture: up to 25,000 users **(PREMIUM ONLY)** @@ -24,8 +24,8 @@ full list of reference architectures, see | Internal load balancing node | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.large | F2s v2 | | Redis - Cache | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | | Redis - Queues / Shared State | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | t2.small | B1MS | -| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | t2.small | B1MS | +| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | t3.small | B1MS | +| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | t3.small | B1MS | | Gitaly | 2 (minimum) | 32 vCPU, 120 GB memory | n1-standard-32 | m5.8xlarge | D32s v3 | | Sidekiq | 4 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | | GitLab Rails | 5 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | @@ -33,6 +33,81 @@ full list of reference architectures, see | Object storage | n/a | n/a | n/a | n/a | n/a | | NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +```mermaid +stateDiagram-v2 + [*] --> LoadBalancer + LoadBalancer --> ApplicationServer + + ApplicationServer --> BackgroundJobs + ApplicationServer --> Gitaly + ApplicationServer --> Redis_Cache + ApplicationServer --> Redis_Queues + ApplicationServer --> PgBouncer + PgBouncer --> Database + ApplicationServer --> ObjectStorage + BackgroundJobs --> ObjectStorage + + ApplicationMonitoring -->ApplicationServer + ApplicationMonitoring -->PgBouncer + ApplicationMonitoring -->Database + ApplicationMonitoring -->BackgroundJobs + + ApplicationServer --> Consul + + Consul --> Database + Consul --> PgBouncer + Redis_Cache --> Consul + Redis_Queues --> Consul + BackgroundJobs --> Consul + + state Consul { + "Consul_1..3" + } + + state Database { + "PG_Primary_Node" + "PG_Secondary_Node_1..2" + } + + state Redis_Cache { + "R_Cache_Primary_Node" + "R_Cache_Replica_Node_1..2" + "R_Cache_Sentinel_1..3" + } + + state Redis_Queues { + "R_Queues_Primary_Node" + "R_Queues_Replica_Node_1..2" + "R_Queues_Sentinel_1..3" + } + + state Gitaly { + "Gitaly_1..2" + } + + state BackgroundJobs { + "Sidekiq_1..4" + } + + state ApplicationServer { + "GitLab_Rails_1..5" + } + + state LoadBalancer { + "LoadBalancer_1" + } + + state ApplicationMonitoring { + "Prometheus" + "Grafana" + } + + state PgBouncer { + "Internal_Load_Balancer" + "PgBouncer_1..3" + } +``` + The Google Cloud Platform (GCP) architectures were built and tested using the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) CPU platform. On different hardware you may find that adjustments, either lower @@ -241,7 +316,7 @@ The following IPs will be used as an example: - `10.6.0.12`: Consul 2 - `10.6.0.13`: Consul 3 -NOTE: **Note:** +NOTE: The configuration processes for the other servers in your reference architecture will use the `/etc/gitlab/gitlab-secrets.json` file from your Consul server to connect with the other servers. @@ -962,7 +1037,7 @@ servers. The following IPs will be used as an example: - `10.6.0.72`: Sentinel - Cache 2 - `10.6.0.73`: Sentinel - Cache 3 -NOTE: **Note:** +NOTE: If you're using an external Redis Sentinel instance, be sure to exclude the `requirepass` parameter from the Sentinel configuration. This parameter causes clients to report `NOAUTH Authentication required.`. @@ -1226,7 +1301,7 @@ servers. The following IPs will be used as an example: - `10.6.0.82`: Sentinel - Queues 2 - `10.6.0.83`: Sentinel - Queues 3 -NOTE: **Note:** +NOTE: If you're using an external Redis Sentinel instance, be sure to exclude the `requirepass` parameter from the Sentinel configuration. This parameter causes clients to report `NOAUTH Authentication required.`. @@ -1347,7 +1422,7 @@ To configure the Sentinel Queues server: ## Configure Gitaly -NOTE: **Note:** +NOTE: [Gitaly Cluster](../gitaly/praefect.md) support for the Reference Architectures is being worked on as a [collaborative effort](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/1) between the Quality Engineering and Gitaly teams. When this component has been verified @@ -1356,19 +1431,17 @@ and improved designed. [Gitaly](../gitaly/index.md) server node requirements are dependent on data, specifically the number of projects and those projects' sizes. It's recommended -that a Gitaly server node stores no more than 5 TB of data. Although this -reference architecture includes a recommendation for the number of Gitaly server -nodes to use, depending on your storage requirements, you may require additional -Gitaly server nodes. +that a Gitaly server node stores no more than 5 TB of data. Depending on your +repository storage requirements, you may require additional Gitaly server nodes. Due to Gitaly having notable input and output requirements, we strongly -recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs should -have a throughput of at least 8,000 input/output operations per second (IOPS) -for read operations and 2,000 IOPS for write operations. These IOPS values are -initial recommendations, and may be adjusted to greater or lesser values -depending on the scale of your environment's workload. If you're running the -environment on a Cloud provider, refer to their documentation about how to -configure IOPS correctly. +recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs +should have a throughput of at least 8,000 +input/output operations per second (IOPS) for read operations and 2,000 IOPS for +write operations. These IOPS values are initial recommendations, and may be +adjusted to greater or lesser values depending on the scale of your +environment's workload. If you're running the environment on a Cloud provider, +refer to their documentation about how to configure IOPS correctly. Be sure to note the following items: @@ -1376,14 +1449,14 @@ Be sure to note the following items: [repository storage paths](../repository_storage_paths.md). - A Gitaly server can host one or more storage paths. - A GitLab server can use one or more Gitaly server nodes. -- Gitaly addresses must be specified to be correctly resolvable for _all_ - Gitaly clients. +- Gitaly addresses must be specified to be correctly resolvable for all Gitaly + clients. - Gitaly servers must not be exposed to the public internet, as Gitaly's network traffic is unencrypted by default. The use of a firewall is highly recommended to restrict access to the Gitaly server. Another option is to [use TLS](#gitaly-tls-support). -NOTE: **Note:** +NOTE: The token referred to throughout the Gitaly documentation is an arbitrary password selected by the administrator. This token is unrelated to tokens created for the GitLab API or other similar web API tokens. @@ -1406,8 +1479,8 @@ On each node: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page, and _do not_ provide the `EXTERNAL_URL` value. -1. Edit `/etc/gitlab/gitlab.rb` to configure the storage paths, enable - the network listener, and configure the token: +1. Edit the Gitaly server node's `/etc/gitlab/gitlab.rb` file to configure + storage paths, enable the network listener, and to configure the token: <!-- updates to following example must also be made at @@ -1501,7 +1574,7 @@ nodes (including the Gitaly node using the certificate) and on all client nodes that communicate with it following the procedure described in [GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). -NOTE: **Note:** +NOTE: The self-signed certificate must specify the address you use to access the Gitaly server. If you are addressing the Gitaly server by a hostname, you can either use the Common Name field for this, or add it as a Subject Alternative @@ -1677,7 +1750,7 @@ To configure the Sidekiq nodes, on each one: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -TIP: **Tip:** +NOTE: You can also run [multiple Sidekiq processes](../operations/extra_sidekiq_processes.md). <div align="right"> @@ -1690,12 +1763,6 @@ You can also run [multiple Sidekiq processes](../operations/extra_sidekiq_proces This section describes how to configure the GitLab application (Rails) component. -In our architecture, we run each GitLab Rails node using the Puma webserver, and -have its number of workers set to 90% of available CPUs, with four threads. For -nodes running Rails with other components, the worker value should be reduced -accordingly. We've determined that a worker value of 50% achieves a good balance, -but this is dependent on workload. - The following IPs will be used as an example: - `10.6.0.111`: GitLab application 1 @@ -1999,7 +2066,7 @@ on what features you intend to use: |Object storage type|Supported by consolidated configuration?| |-------------------|----------------------------------------| -| [Backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage)|No| +| [Backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage) | No | | [Job artifacts](../job_artifacts.md#using-object-storage) including archived job logs | Yes | | [LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | | [Uploads](../uploads.md#using-object-storage) | Yes | diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index f4842a8568b..c341ff5baec 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -2,7 +2,7 @@ reading_time: true stage: Enablement group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Reference architecture: up to 2,000 users **(CORE ONLY)** @@ -26,6 +26,46 @@ For a full list of reference architectures, see | Object storage | n/a | n/a | n/a | n/a | n/a | | NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +```mermaid +stateDiagram-v2 + [*] --> LoadBalancer + LoadBalancer --> ApplicationServer + + ApplicationServer --> Gitaly + ApplicationServer --> Redis + ApplicationServer --> Database + ApplicationServer --> ObjectStorage + + ApplicationMonitoring -->ApplicationServer + ApplicationMonitoring -->Redis + ApplicationMonitoring -->Database + + + state Database { + "PG_Node" + } + state Redis { + "Redis_Node" + } + + state Gitaly { + "Gitaly" + } + + state ApplicationServer { + "AppServ_1..2" + } + + state LoadBalancer { + "LoadBalancer" + } + + state ApplicationMonitoring { + "Prometheus" + "Grafana" + } +``` + The Google Cloud Platform (GCP) architectures were built and tested using the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) CPU platform. On different hardware you may find that adjustments, either lower @@ -356,7 +396,7 @@ are supported and can be added if needed. ## Configure Gitaly -NOTE: **Note:** +NOTE: [Gitaly Cluster](../gitaly/praefect.md) support for the Reference Architectures is being worked on as a [collaborative effort](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/1) between the Quality Engineering and Gitaly teams. When this component has been verified @@ -391,7 +431,7 @@ Be sure to note the following items: to restrict access to the Gitaly server. Another option is to [use TLS](#gitaly-tls-support). -NOTE: **Note:** +NOTE: The token referred to throughout the Gitaly documentation is an arbitrary password selected by the administrator. This token is unrelated to tokens created for the GitLab API or other similar web API tokens. @@ -485,7 +525,7 @@ nodes (including the Gitaly node using the certificate) and on all client nodes that communicate with it following the procedure described in [GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). -NOTE: **Note:** +NOTE: The self-signed certificate must specify the address you use to access the Gitaly server. If you are addressing the Gitaly server by a hostname, you can either use the Common Name field for this, or add it as a Subject Alternative @@ -858,7 +898,7 @@ on what features you intend to use: |Object storage type|Supported by consolidated configuration?| |-------------------|----------------------------------------| -| [Backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage)|No| +| [Backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage) | No | | [Job artifacts](../job_artifacts.md#using-object-storage) including archived job logs | Yes | | [LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | | [Uploads](../uploads.md#using-object-storage) | Yes | diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index b5b3e4e0300..370247ed856 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -2,7 +2,7 @@ reading_time: true stage: Enablement group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Reference architecture: up to 3,000 users **(PREMIUM ONLY)** @@ -11,7 +11,7 @@ This page describes GitLab reference architecture for up to 3,000 users. For a full list of reference architectures, see [Available reference architectures](index.md#available-reference-architectures). -NOTE: **Note:** +NOTE: This reference architecture is designed to help your organization achieve a highly-available GitLab deployment. If you do not have the expertise or need to maintain a highly-available environment, you can have a simpler and less @@ -37,6 +37,62 @@ costly-to-operate environment by using the | Object storage | n/a | n/a | n/a | n/a | n/a | | NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +```mermaid +stateDiagram-v2 + [*] --> LoadBalancer + LoadBalancer --> ApplicationServer + + ApplicationServer --> BackgroundJobs + ApplicationServer --> Gitaly + ApplicationServer --> Redis + ApplicationServer --> PgBouncer + PgBouncer --> Database + ApplicationServer --> ObjectStorage + BackgroundJobs --> ObjectStorage + + ApplicationMonitoring -->ApplicationServer + ApplicationMonitoring -->Redis + ApplicationMonitoring -->PgBouncer + ApplicationMonitoring -->Database + ApplicationMonitoring -->BackgroundJobs + + state Database { + "PG_Primary_Node" + "PG_Secondary_Node_1..2" + } + state Redis { + "R_Primary_Node" + "R_Replica_Node_1..2" + "R_Consul/Sentinel_1..3" + } + + state Gitaly { + "Gitaly_1..2" + } + + state BackgroundJobs { + "Sidekiq_1..4" + } + + state ApplicationServer { + "GitLab_Rails_1..3" + } + + state LoadBalancer { + "LoadBalancer_1" + } + + state ApplicationMonitoring { + "Prometheus" + "Grafana" + } + + state PgBouncer { + "Internal_Load_Balancer" + "PgBouncer_1..3" + } +``` + The Google Cloud Platform (GCP) architectures were built and tested using the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) CPU platform. On different hardware you may find that adjustments, either lower @@ -444,7 +500,7 @@ servers. The following IPs will be used as an example: - `10.6.0.12`: Consul/Sentinel 2 - `10.6.0.13`: Consul/Sentinel 3 -NOTE: **Note:** +NOTE: If you're using an external Redis Sentinel instance, be sure to exclude the `requirepass` parameter from the Sentinel configuration. This parameter causes clients to report `NOAUTH Authentication required.`. @@ -1058,7 +1114,7 @@ Refer to your preferred Load Balancer's documentation for further guidance. ## Configure Gitaly -NOTE: **Note:** +NOTE: [Gitaly Cluster](../gitaly/praefect.md) support for the Reference Architectures is being worked on as a [collaborative effort](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/1) between the Quality Engineering and Gitaly teams. When this component has been verified @@ -1067,19 +1123,17 @@ and improved designed. [Gitaly](../gitaly/index.md) server node requirements are dependent on data, specifically the number of projects and those projects' sizes. It's recommended -that a Gitaly server node stores no more than 5 TB of data. Although this -reference architecture includes a recommendation for the number of Gitaly server -nodes to use, depending on your storage requirements, you may require additional -Gitaly server nodes. +that a Gitaly server node stores no more than 5 TB of data. Depending on your +repository storage requirements, you may require additional Gitaly server nodes. Due to Gitaly having notable input and output requirements, we strongly -recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs should -have a throughput of at least 8,000 input/output operations per second (IOPS) -for read operations and 2,000 IOPS for write operations. These IOPS values are -initial recommendations, and may be adjusted to greater or lesser values -depending on the scale of your environment's workload. If you're running the -environment on a Cloud provider, refer to their documentation about how to -configure IOPS correctly. +recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs +should have a throughput of at least 8,000 +input/output operations per second (IOPS) for read operations and 2,000 IOPS for +write operations. These IOPS values are initial recommendations, and may be +adjusted to greater or lesser values depending on the scale of your +environment's workload. If you're running the environment on a Cloud provider, +refer to their documentation about how to configure IOPS correctly. Be sure to note the following items: @@ -1087,14 +1141,14 @@ Be sure to note the following items: [repository storage paths](../repository_storage_paths.md). - A Gitaly server can host one or more storage paths. - A GitLab server can use one or more Gitaly server nodes. -- Gitaly addresses must be specified to be correctly resolvable for _all_ - Gitaly clients. +- Gitaly addresses must be specified to be correctly resolvable for all Gitaly + clients. - Gitaly servers must not be exposed to the public internet, as Gitaly's network traffic is unencrypted by default. The use of a firewall is highly recommended to restrict access to the Gitaly server. Another option is to [use TLS](#gitaly-tls-support). -NOTE: **Note:** +NOTE: The token referred to throughout the Gitaly documentation is an arbitrary password selected by the administrator. This token is unrelated to tokens created for the GitLab API or other similar web API tokens. @@ -1117,8 +1171,8 @@ On each node: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page, and _do not_ provide the `EXTERNAL_URL` value. -1. Edit `/etc/gitlab/gitlab.rb` to configure the storage paths, enable - the network listener, and configure the token: +1. Edit the Gitaly server node's `/etc/gitlab/gitlab.rb` file to configure + storage paths, enable the network listener, and to configure the token: <!-- updates to following example must also be made at @@ -1244,7 +1298,7 @@ nodes (including the Gitaly node using the certificate) and on all client nodes that communicate with it following the procedure described in [GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). -NOTE: **Note:** +NOTE: The self-signed certificate must specify the address you use to access the Gitaly server. If you are addressing the Gitaly server by a hostname, you can either use the Common Name field for this, or add it as a Subject Alternative @@ -1412,7 +1466,7 @@ To configure the Sidekiq nodes, one each one: run: sidekiq: (pid 30142) 77351s; run: log: (pid 29638) 77386s ``` -TIP: **Tip:** +NOTE: You can also run [multiple Sidekiq processes](../operations/extra_sidekiq_processes.md). <div align="right"> @@ -1425,12 +1479,6 @@ You can also run [multiple Sidekiq processes](../operations/extra_sidekiq_proces This section describes how to configure the GitLab application (Rails) component. -In our architecture, we run each GitLab Rails node using the Puma webserver, and -have its number of workers set to 90% of available CPUs, with four threads. For -nodes running Rails with other components, the worker value should be reduced -accordingly. We've determined that a worker value of 50% achieves a good balance, -but this is dependent on workload. - On each node perform the following: 1. If you're [using NFS](#configure-nfs-optional): @@ -1734,7 +1782,7 @@ on what features you intend to use: |Object storage type|Supported by consolidated configuration?| |-------------------|----------------------------------------| -| [Backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage)|No| +| [Backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage) | No | | [Job artifacts](../job_artifacts.md#using-object-storage) including archived job logs | Yes | | [LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | | [Uploads](../uploads.md#using-object-storage) | Yes | diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 152eb9cb90d..f76c8a8a877 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -2,7 +2,7 @@ reading_time: true stage: Enablement group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Reference architecture: up to 50,000 users **(PREMIUM ONLY)** @@ -24,8 +24,8 @@ full list of reference architectures, see | Internal load balancing node | 1 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | | Redis - Cache | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | | Redis - Queues / Shared State | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | t2.small | B1MS | -| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | t2.small | B1MS | +| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | t3.small | B1MS | +| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | t3.small | B1MS | | Gitaly | 2 (minimum) | 64 vCPU, 240 GB memory | n1-standard-64 | m5.16xlarge | D64s v3 | | Sidekiq | 4 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | | GitLab Rails | 12 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | @@ -33,6 +33,81 @@ full list of reference architectures, see | Object storage | n/a | n/a | n/a | n/a | n/a | | NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +```mermaid +stateDiagram-v2 + [*] --> LoadBalancer + LoadBalancer --> ApplicationServer + + ApplicationServer --> BackgroundJobs + ApplicationServer --> Gitaly + ApplicationServer --> Redis_Cache + ApplicationServer --> Redis_Queues + ApplicationServer --> PgBouncer + PgBouncer --> Database + ApplicationServer --> ObjectStorage + BackgroundJobs --> ObjectStorage + + ApplicationMonitoring -->ApplicationServer + ApplicationMonitoring -->PgBouncer + ApplicationMonitoring -->Database + ApplicationMonitoring -->BackgroundJobs + + ApplicationServer --> Consul + + Consul --> Database + Consul --> PgBouncer + Redis_Cache --> Consul + Redis_Queues --> Consul + BackgroundJobs --> Consul + + state Consul { + "Consul_1..3" + } + + state Database { + "PG_Primary_Node" + "PG_Secondary_Node_1..2" + } + + state Redis_Cache { + "R_Cache_Primary_Node" + "R_Cache_Replica_Node_1..2" + "R_Cache_Sentinel_1..3" + } + + state Redis_Queues { + "R_Queues_Primary_Node" + "R_Queues_Replica_Node_1..2" + "R_Queues_Sentinel_1..3" + } + + state Gitaly { + "Gitaly_1..2" + } + + state BackgroundJobs { + "Sidekiq_1..4" + } + + state ApplicationServer { + "GitLab_Rails_1..12" + } + + state LoadBalancer { + "LoadBalancer_1" + } + + state ApplicationMonitoring { + "Prometheus" + "Grafana" + } + + state PgBouncer { + "Internal_Load_Balancer" + "PgBouncer_1..3" + } +``` + The Google Cloud Platform (GCP) architectures were built and tested using the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) CPU platform. On different hardware you may find that adjustments, either lower @@ -241,7 +316,7 @@ The following IPs will be used as an example: - `10.6.0.12`: Consul 2 - `10.6.0.13`: Consul 3 -NOTE: **Note:** +NOTE: The configuration processes for the other servers in your reference architecture will use the `/etc/gitlab/gitlab-secrets.json` file from your Consul server to connect with the other servers. @@ -962,7 +1037,7 @@ servers. The following IPs will be used as an example: - `10.6.0.72`: Sentinel - Cache 2 - `10.6.0.73`: Sentinel - Cache 3 -NOTE: **Note:** +NOTE: If you're using an external Redis Sentinel instance, be sure to exclude the `requirepass` parameter from the Sentinel configuration. This parameter causes clients to report `NOAUTH Authentication required.`. @@ -1226,7 +1301,7 @@ servers. The following IPs will be used as an example: - `10.6.0.82`: Sentinel - Queues 2 - `10.6.0.83`: Sentinel - Queues 3 -NOTE: **Note:** +NOTE: If you're using an external Redis Sentinel instance, be sure to exclude the `requirepass` parameter from the Sentinel configuration. This parameter causes clients to report `NOAUTH Authentication required.`. @@ -1347,7 +1422,7 @@ To configure the Sentinel Queues server: ## Configure Gitaly -NOTE: **Note:** +NOTE: [Gitaly Cluster](../gitaly/praefect.md) support for the Reference Architectures is being worked on as a [collaborative effort](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/1) between the Quality Engineering and Gitaly teams. When this component has been verified @@ -1356,19 +1431,17 @@ and improved designed. [Gitaly](../gitaly/index.md) server node requirements are dependent on data, specifically the number of projects and those projects' sizes. It's recommended -that a Gitaly server node stores no more than 5 TB of data. Although this -reference architecture includes a recommendation for the number of Gitaly server -nodes to use, depending on your storage requirements, you may require additional -Gitaly server nodes. +that a Gitaly server node stores no more than 5 TB of data. Depending on your +repository storage requirements, you may require additional Gitaly server nodes. Due to Gitaly having notable input and output requirements, we strongly -recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs should -have a throughput of at least 8,000 input/output operations per second (IOPS) -for read operations and 2,000 IOPS for write operations. These IOPS values are -initial recommendations, and may be adjusted to greater or lesser values -depending on the scale of your environment's workload. If you're running the -environment on a Cloud provider, refer to their documentation about how to -configure IOPS correctly. +recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs +should have a throughput of at least 8,000 +input/output operations per second (IOPS) for read operations and 2,000 IOPS for +write operations. These IOPS values are initial recommendations, and may be +adjusted to greater or lesser values depending on the scale of your +environment's workload. If you're running the environment on a Cloud provider, +refer to their documentation about how to configure IOPS correctly. Be sure to note the following items: @@ -1376,14 +1449,14 @@ Be sure to note the following items: [repository storage paths](../repository_storage_paths.md). - A Gitaly server can host one or more storage paths. - A GitLab server can use one or more Gitaly server nodes. -- Gitaly addresses must be specified to be correctly resolvable for _all_ - Gitaly clients. +- Gitaly addresses must be specified to be correctly resolvable for all Gitaly + clients. - Gitaly servers must not be exposed to the public internet, as Gitaly's network traffic is unencrypted by default. The use of a firewall is highly recommended to restrict access to the Gitaly server. Another option is to [use TLS](#gitaly-tls-support). -NOTE: **Note:** +NOTE: The token referred to throughout the Gitaly documentation is an arbitrary password selected by the administrator. This token is unrelated to tokens created for the GitLab API or other similar web API tokens. @@ -1406,8 +1479,8 @@ On each node: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page, and _do not_ provide the `EXTERNAL_URL` value. -1. Edit `/etc/gitlab/gitlab.rb` to configure the storage paths, enable - the network listener, and configure the token: +1. Edit the Gitaly server node's `/etc/gitlab/gitlab.rb` file to configure + storage paths, enable the network listener, and to configure the token: <!-- updates to following example must also be made at @@ -1501,7 +1574,7 @@ nodes (including the Gitaly node using the certificate) and on all client nodes that communicate with it following the procedure described in [GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). -NOTE: **Note:** +NOTE: The self-signed certificate must specify the address you use to access the Gitaly server. If you are addressing the Gitaly server by a hostname, you can either use the Common Name field for this, or add it as a Subject Alternative @@ -1677,7 +1750,7 @@ To configure the Sidekiq nodes, on each one: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -TIP: **Tip:** +NOTE: You can also run [multiple Sidekiq processes](../operations/extra_sidekiq_processes.md). <div align="right"> @@ -1690,12 +1763,6 @@ You can also run [multiple Sidekiq processes](../operations/extra_sidekiq_proces This section describes how to configure the GitLab application (Rails) component. -In our architecture, we run each GitLab Rails node using the Puma webserver, and -have its number of workers set to 90% of available CPUs, with four threads. For -nodes running Rails with other components, the worker value should be reduced -accordingly. We've determined that a worker value of 50% achieves a good balance, -but this is dependent on workload. - The following IPs will be used as an example: - `10.6.0.111`: GitLab application 1 @@ -1999,7 +2066,7 @@ on what features you intend to use: |Object storage type|Supported by consolidated configuration?| |-------------------|----------------------------------------| -| [Backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage)|No| +| [Backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage) | No | | [Job artifacts](../job_artifacts.md#using-object-storage) including archived job logs | Yes | | [LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | | [Uploads](../uploads.md#using-object-storage) | Yes | diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index f023971bdc0..6a0547aeaf9 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -2,7 +2,7 @@ reading_time: true stage: Enablement group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Reference architecture: up to 5,000 users **(PREMIUM ONLY)** @@ -11,7 +11,7 @@ This page describes GitLab reference architecture for up to 5,000 users. For a full list of reference architectures, see [Available reference architectures](index.md#available-reference-architectures). -NOTE: **Note:** +NOTE: This reference architecture is designed to help your organization achieve a highly-available GitLab deployment. If you do not have the expertise or need to maintain a highly-available environment, you can have a simpler and less @@ -37,6 +37,63 @@ costly-to-operate environment by using the | Object storage | n/a | n/a | n/a | n/a | n/a | | NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +```mermaid +stateDiagram-v2 + [*] --> LoadBalancer + LoadBalancer --> ApplicationServer + + ApplicationServer --> BackgroundJobs + ApplicationServer --> Gitaly + ApplicationServer --> Redis + ApplicationServer --> PgBouncer + PgBouncer --> Database + ApplicationServer --> ObjectStorage + BackgroundJobs --> ObjectStorage + + ApplicationMonitoring -->ApplicationServer + ApplicationMonitoring -->Redis + ApplicationMonitoring -->PgBouncer + ApplicationMonitoring -->Database + ApplicationMonitoring -->BackgroundJobs + + state Database { + "PG_Primary_Node" + "PG_Secondary_Node_1..2" + } + + state Redis { + "R_Primary_Node" + "R_Replica_Node_1..2" + "R_Consul/Sentinel_1..3" + } + + state Gitaly { + "Gitaly_1..2" + } + + state BackgroundJobs { + "Sidekiq_1..4" + } + + state ApplicationServer { + "GitLab_Rails_1..3" + } + + state LoadBalancer { + "LoadBalancer_1" + } + + state ApplicationMonitoring { + "Prometheus" + "Grafana" + } + + state PgBouncer { + "Internal_Load_Balancer" + "PgBouncer_1..3" + } +``` + The Google Cloud Platform (GCP) architectures were built and tested using the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) CPU platform. On different hardware you may find that adjustments, either lower @@ -444,7 +501,7 @@ servers. The following IPs will be used as an example: - `10.6.0.12`: Consul/Sentinel 2 - `10.6.0.13`: Consul/Sentinel 3 -NOTE: **Note:** +NOTE: If you're using an external Redis Sentinel instance, be sure to exclude the `requirepass` parameter from the Sentinel configuration. This parameter causes clients to report `NOAUTH Authentication required.`. @@ -1057,7 +1114,7 @@ Refer to your preferred Load Balancer's documentation for further guidance. ## Configure Gitaly -NOTE: **Note:** +NOTE: [Gitaly Cluster](../gitaly/praefect.md) support for the Reference Architectures is being worked on as a [collaborative effort](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/1) between the Quality Engineering and Gitaly teams. When this component has been verified @@ -1066,19 +1123,17 @@ and improved designed. [Gitaly](../gitaly/index.md) server node requirements are dependent on data, specifically the number of projects and those projects' sizes. It's recommended -that a Gitaly server node stores no more than 5 TB of data. Although this -reference architecture includes a recommendation for the number of Gitaly server -nodes to use, depending on your storage requirements, you may require additional -Gitaly server nodes. +that a Gitaly server node stores no more than 5 TB of data. Depending on your +repository storage requirements, you may require additional Gitaly server nodes. Due to Gitaly having notable input and output requirements, we strongly -recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs should -have a throughput of at least 8,000 input/output operations per second (IOPS) -for read operations and 2,000 IOPS for write operations. These IOPS values are -initial recommendations, and may be adjusted to greater or lesser values -depending on the scale of your environment's workload. If you're running the -environment on a Cloud provider, refer to their documentation about how to -configure IOPS correctly. +recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs +should have a throughput of at least 8,000 +input/output operations per second (IOPS) for read operations and 2,000 IOPS for +write operations. These IOPS values are initial recommendations, and may be +adjusted to greater or lesser values depending on the scale of your +environment's workload. If you're running the environment on a Cloud provider, +refer to their documentation about how to configure IOPS correctly. Be sure to note the following items: @@ -1086,14 +1141,14 @@ Be sure to note the following items: [repository storage paths](../repository_storage_paths.md). - A Gitaly server can host one or more storage paths. - A GitLab server can use one or more Gitaly server nodes. -- Gitaly addresses must be specified to be correctly resolvable for _all_ - Gitaly clients. +- Gitaly addresses must be specified to be correctly resolvable for all Gitaly + clients. - Gitaly servers must not be exposed to the public internet, as Gitaly's network traffic is unencrypted by default. The use of a firewall is highly recommended to restrict access to the Gitaly server. Another option is to [use TLS](#gitaly-tls-support). -NOTE: **Note:** +NOTE: The token referred to throughout the Gitaly documentation is an arbitrary password selected by the administrator. This token is unrelated to tokens created for the GitLab API or other similar web API tokens. @@ -1116,8 +1171,8 @@ On each node: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page, and _do not_ provide the `EXTERNAL_URL` value. -1. Edit `/etc/gitlab/gitlab.rb` to configure the storage paths, enable - the network listener, and configure the token: +1. Edit the Gitaly server node's `/etc/gitlab/gitlab.rb` file to configure + storage paths, enable the network listener, and to configure the token: <!-- updates to following example must also be made at @@ -1243,7 +1298,7 @@ nodes (including the Gitaly node using the certificate) and on all client nodes that communicate with it following the procedure described in [GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). -NOTE: **Note:** +NOTE: The self-signed certificate must specify the address you use to access the Gitaly server. If you are addressing the Gitaly server by a hostname, you can either use the Common Name field for this, or add it as a Subject Alternative @@ -1411,7 +1466,7 @@ To configure the Sidekiq nodes, one each one: run: sidekiq: (pid 30142) 77351s; run: log: (pid 29638) 77386s ``` -TIP: **Tip:** +NOTE: You can also run [multiple Sidekiq processes](../operations/extra_sidekiq_processes.md). <div align="right"> @@ -1424,12 +1479,6 @@ You can also run [multiple Sidekiq processes](../operations/extra_sidekiq_proces This section describes how to configure the GitLab application (Rails) component. -In our architecture, we run each GitLab Rails node using the Puma webserver, and -have its number of workers set to 90% of available CPUs, with four threads. For -nodes running Rails with other components, the worker value should be reduced -accordingly. We've determined that a worker value of 50% achieves a good balance, -but this is dependent on workload. - On each node perform the following: 1. If you're [using NFS](#configure-nfs-optional): @@ -1733,7 +1782,7 @@ on what features you intend to use: |Object storage type|Supported by consolidated configuration?| |-------------------|----------------------------------------| -| [Backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage)|No| +| [Backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage) | No | | [Job artifacts](../job_artifacts.md#using-object-storage) including archived job logs | Yes | | [LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | | [Uploads](../uploads.md#using-object-storage) | Yes | diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index 8816d0eecf4..6b04dad9baf 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -2,14 +2,14 @@ type: reference, concepts stage: Enablement group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Reference architectures You can set up GitLab on a single server or scale it up to serve many users. This page details the recommended Reference Architectures that were built and -verified by GitLab's Quality and Support teams. +verified by the GitLab Quality and Support teams. Below is a chart representing each architecture tier and the number of users they can handle. As your number of users grow with time, itâs recommended that @@ -18,8 +18,8 @@ you scale GitLab accordingly.  <!-- Internal link: https://docs.google.com/spreadsheets/d/1obYP4fLKkVVDOljaI3-ozhmCiPtEeMblbBKkf2OADKs/edit#gid=1403207183 --> -Testing on these reference architectures were performed with -[GitLab's Performance Tool](https://gitlab.com/gitlab-org/quality/performance) +Testing on these reference architectures was performed with the +[GitLab Performance Tool](https://gitlab.com/gitlab-org/quality/performance) at specific coded workloads, and the throughputs used for testing were calculated based on sample customer data. Select the [reference architecture](#available-reference-architectures) that matches your scale. @@ -36,8 +36,8 @@ the [default setup](#automated-backups) by [installing GitLab](../../install/README.md) on a single machine to minimize maintenance and resource costs. -If your organization has more than 2,000 users, the recommendation is to scale -GitLab's components to multiple machine nodes. The machine nodes are grouped by +If your organization has more than 2,000 users, the recommendation is to scale the +GitLab components to multiple machine nodes. The machine nodes are grouped by components. The addition of these nodes increases the performance and scalability of to your GitLab instance. @@ -47,7 +47,7 @@ When scaling GitLab, there are several factors to consider: - A load balancer is added in front to distribute traffic across the application nodes. - The application nodes connects to a shared file server and PostgreSQL and Redis services on the backend. -NOTE: **Note:** +NOTE: Depending on your workflow, the following recommended reference architectures may need to be adapted accordingly. Your workload is influenced by factors including how active your users are, how much automation you use, mirroring, @@ -68,6 +68,11 @@ The following reference architectures are available: - [Up to 25,000 users](25k_users.md) - [Up to 50,000 users](50k_users.md) +A GitLab [Premium or Ultimate](https://about.gitlab.com/pricing/#self-managed) license is required +to get assistance from Support with troubleshooting the [2,000 users](2k_users.md) +and higher reference architectures. +[Read more about our definition of scaled architectures](https://about.gitlab.com/support/#definition-of-scaled-architecture). + ## Availability Components GitLab comes with the following components for your use, listed from least to @@ -151,7 +156,27 @@ is recommended. instance to other geographical locations as a read-only fully operational instance that can also be promoted in case of disaster. -## Configuring select components with Cloud Native Helm +## Deviating from the suggested reference architectures + +As a general rule of thumb, the further away you move from the Reference Architectures, +the harder it will be get support for it. With any deviation, you're introducing +a layer of complexity that will add challenges to finding out where potential +issues might lie. + +The reference architectures use the official GitLab Linux packages (Omnibus +GitLab) to install and configure the various components (with one notable exception being the suggested select Cloud Native installation method described below). The components are +installed on separate machines (virtualized or bare metal), with machine hardware +requirements listed in the "Configuration" column and equivalent VM standard sizes listed +in GCP/AWS/Azure columns of each [available reference architecture](#available-reference-architectures). + +Running components on Docker (including Compose) with the same specs should be fine, as Docker is well known in terms of support. +However, it is still an additional layer and may still add some support complexities, such as not being able to run `strace` easily in containers. + +Other technologies, like [Docker swarm](https://docs.docker.com/engine/swarm/) +are not officially supported, but can be implemented at your own risk. In that +case, GitLab Support will not be able to help you. + +### Configuring select components with Cloud Native Helm We also provide [Helm charts](https://docs.gitlab.com/charts/) as a Cloud Native installation method for GitLab. For the reference architectures, select components can be set up in this diff --git a/doc/administration/reference_architectures/troubleshooting.md b/doc/administration/reference_architectures/troubleshooting.md index db2e9b89ba2..406ff57c66d 100644 --- a/doc/administration/reference_architectures/troubleshooting.md +++ b/doc/administration/reference_architectures/troubleshooting.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Troubleshooting a reference architecture setup @@ -434,13 +434,13 @@ If the monitoring node is not receiving any data, check that the exporters are capturing data. ```shell -curl http[s]://localhost:<EXPORTER LISTENING PORT>/metric +curl "http[s]://localhost:<EXPORTER LISTENING PORT>/metric" ``` or ```shell -curl http[s]://localhost:<EXPORTER LISTENING PORT>/-/metric +curl "http[s]://localhost:<EXPORTER LISTENING PORT>/-/metric" ``` ## Troubleshooting PgBouncer diff --git a/doc/administration/reply_by_email.md b/doc/administration/reply_by_email.md index 4108635ba2c..ebb9e086cb7 100644 --- a/doc/administration/reply_by_email.md +++ b/doc/administration/reply_by_email.md @@ -1,7 +1,7 @@ --- stage: Plan group: Certify -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Reply by email @@ -15,34 +15,40 @@ replying to notification emails. Make sure [incoming email](incoming_email.md) is set up. -## How it works? +## How it works -### 1. GitLab sends a notification email +Replying by email happens in three steps: + +1. GitLab sends a notification email. +1. You reply to the notification email. +1. GitLab receives your reply to the notification email. + +### GitLab sends a notification email When GitLab sends a notification and Reply by email is enabled, the `Reply-To` header is set to the address defined in your GitLab configuration, with the `%{key}` placeholder (if present) replaced by a specific "reply key". In addition, this "reply key" is also added to the `References` header. -### 2. You reply to the notification email +### You reply to the notification email -When you reply to the notification email, your email client will: +When you reply to the notification email, your email client: -- send the email to the `Reply-To` address it got from the notification email -- set the `In-Reply-To` header to the value of the `Message-ID` header from the +- sends the email to the `Reply-To` address it got from the notification email +- sets the `In-Reply-To` header to the value of the `Message-ID` header from the notification email -- set the `References` header to the value of the `Message-ID` plus the value of +- sets the `References` header to the value of the `Message-ID` plus the value of the notification email's `References` header. -### 3. GitLab receives your reply to the notification email +### GitLab receives your reply to the notification email -When GitLab receives your reply, it will look for the "reply key" in the +When GitLab receives your reply, it looks for the "reply key" in the following headers, in this order: 1. the `To` header 1. the `References` header -If it finds a reply key, it will be able to leave your reply as a comment on +If it finds a reply key, it leaves your reply as a comment on the entity the notification was about (issue, merge request, commit...). For more details about the `Message-ID`, `In-Reply-To`, and `References headers`, diff --git a/doc/administration/reply_by_email_postfix_setup.md b/doc/administration/reply_by_email_postfix_setup.md index aaadeef8bf5..f310ef04295 100644 --- a/doc/administration/reply_by_email_postfix_setup.md +++ b/doc/administration/reply_by_email_postfix_setup.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Set up Postfix for incoming email @@ -91,7 +91,7 @@ The instructions make the assumption that you will be using the email address `i quit ``` - NOTE: **Note:** + NOTE: The `.` is a literal period on its own line. If you receive an error after entering `rcpt to: incoming@localhost` @@ -272,7 +272,7 @@ Courier, which we will install later to add IMAP authentication, requires mailbo quit ``` - NOTE: **Note:** + NOTE: The `.` is a literal period on its own line. 1. Check if the `incoming` user received the email: diff --git a/doc/administration/repository_checks.md b/doc/administration/repository_checks.md index 7d79840b56d..6f7a72a0ef2 100644 --- a/doc/administration/repository_checks.md +++ b/doc/administration/repository_checks.md @@ -1,7 +1,7 @@ --- stage: Create group: Editor -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers" +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" type: reference --- @@ -17,7 +17,7 @@ before the check result is visible on the project admin page. If the checks failed you can see their output on in the [`repocheck.log` file.](logs.md#repochecklog) -NOTE: **Note:** +NOTE: It is OFF by default because it still causes too many false alarms. ## Periodic checks diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md index 16e17458e10..090f95eca12 100644 --- a/doc/administration/repository_storage_paths.md +++ b/doc/administration/repository_storage_paths.md @@ -1,7 +1,7 @@ --- stage: Create group: Gitaly -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference, howto --- @@ -40,35 +40,34 @@ storage2: ## Configure GitLab -> **Warning:** -> In order for [backups](../raketasks/backup_restore.md) to work correctly, the storage path must **not** be a -> mount point and the GitLab user should have correct permissions for the parent -> directory of the path. In Omnibus GitLab this is taken care of automatically, -> but for source installations you should be extra careful. -> -> The thing is that for compatibility reasons `gitlab.yml` has a different -> structure than Omnibus. In `gitlab.yml` you indicate the path for the -> repositories, for example `/home/git/repositories`, while in Omnibus you -> indicate `git_data_dirs`, which for the example above would be `/home/git`. -> Then, Omnibus will create a `repositories` directory under that path to use with -> `gitlab.yml`. -> -> This little detail matters because while restoring a backup, the current -> contents of `/home/git/repositories` [are moved to](https://gitlab.com/gitlab-org/gitlab/blob/033e5423a2594e08a7ebcd2379bd2331f4c39032/lib/backup/repository.rb#L54-56) `/home/git/repositories.old`, -> so if `/home/git/repositories` is the mount point, then `mv` would be moving -> things between mount points, and bad things could happen. Ideally, -> `/home/git` would be the mount point, so then things would be moving within the -> same mount point. This is guaranteed with Omnibus installations (because they -> don't specify the full repository path but the parent path), but not for source -> installations. +In order for [backups](../raketasks/backup_restore.md) to work correctly, the storage path must **not** be a +mount point and the GitLab user should have correct permissions for the parent +directory of the path. In Omnibus GitLab this is taken care of automatically, +but for source installations you should be extra careful. + +The thing is that for compatibility reasons `gitlab.yml` has a different +structure than Omnibus. In `gitlab.yml` you indicate the path for the +repositories, for example `/home/git/repositories`, while in Omnibus you +indicate `git_data_dirs`, which for the example above would be `/home/git`. +Then, Omnibus creates a `repositories` directory under that path to use with +`gitlab.yml`. + +This little detail matters because while restoring a backup, the current +contents of `/home/git/repositories` [are moved to](https://gitlab.com/gitlab-org/gitlab/blob/033e5423a2594e08a7ebcd2379bd2331f4c39032/lib/backup/repository.rb#L54-56) `/home/git/repositories.old`, +so if `/home/git/repositories` is the mount point, then `mv` would be moving +things between mount points, and bad things could happen. Ideally, +`/home/git` would be the mount point, so then things would be moving within the +same mount point. This is guaranteed with Omnibus installations (because they +don't specify the full repository path but the parent path), but not for source +installations. Now that you've read that big fat warning above, let's edit the configuration files and add the full paths of the alternative repository storage paths. In the example below, we add two more mount points that are named `nfs_1` and `nfs_2` respectively. -NOTE: **Note:** -This example uses NFS. We do not recommend using EFS for storage as it may impact GitLab's performance. See the [relevant documentation](nfs.md#avoid-using-awss-elastic-file-system-efs) for more details. +NOTE: +This example uses NFS. We do not recommend using EFS for storage as it may impact GitLab performance. See the [relevant documentation](nfs.md#avoid-using-awss-elastic-file-system-efs) for more details. **For installations from source** @@ -89,9 +88,8 @@ This example uses NFS. We do not recommend using EFS for storage as it may impac 1. [Restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. -NOTE: **Note:** -The [`gitlab_shell: repos_path` entry](https://gitlab.com/gitlab-org/gitlab-foss/-/blob/8-9-stable/config/gitlab.yml.example#L457) in `gitlab.yml` will be -deprecated and replaced by `repositories: storages` in the future, so if you +NOTE: +We plan to replace [`gitlab_shell: repos_path` entry](https://gitlab.com/gitlab-org/gitlab-foss/-/blob/8-9-stable/config/gitlab.yml.example#L457) in `gitlab.yml` with `repositories: storages`. If you are upgrading from a version prior to 8.10, make sure to add the configuration as described in the step above. After you make the changes and confirm they are working, you can remove the `repos_path` line. @@ -112,16 +110,15 @@ working, you can remove the `repos_path` line. Note that Omnibus stores the repositories in a `repositories` subdirectory of the `git-data` directory. -## Choose where new repositories will be stored +## Choose where new repositories are stored Once you set the multiple storage paths, you can choose where new repositories -will be stored under **Admin Area > Settings > Repository > -Repository storage > Storage nodes for new repositories**. +are stored in the Admin Area under **Settings > Repository > Repository storage > Storage nodes for new repositories**. Each storage can be assigned a weight from 0-100. When a new project is created, these -weights are used to determine the storage location the repository will be created on. +weights are used to determine the storage location the repository is created on.  Beginning with GitLab 8.13.4, multiple paths can be chosen. New repositories -will be randomly placed on one of the selected paths. +are randomly placed on one of the selected paths. diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md index d2d492c8f1a..6ec43a8ce06 100644 --- a/doc/administration/repository_storage_types.md +++ b/doc/administration/repository_storage_types.md @@ -1,7 +1,7 @@ --- stage: Create group: Gitaly -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference, howto --- @@ -19,7 +19,7 @@ locations that can be: - Accessed via [Gitaly](gitaly/index.md) on its own machine. In GitLab, this is configured in `/etc/gitlab/gitlab.rb` by the `git_data_dirs({})` -configuration hash. The storage layouts discussed here will apply to any shard +configuration hash. The storage layouts discussed here apply to any shard defined in it. The `default` repository shard that is available in any installations @@ -28,24 +28,24 @@ Anything discussed below is expected to be part of that folder. ## Hashed storage -NOTE: **Note:** +NOTE: In GitLab 13.0, hashed storage is enabled by default and the legacy storage is -deprecated. Support for legacy storage will be removed in GitLab 14.0. +deprecated. Support for legacy storage is scheduled to be removed in GitLab 14.0. If you haven't migrated yet, check the [migration instructions](raketasks/storage.md#migrate-to-hashed-storage). The option to choose between hashed and legacy storage in the admin area has been disabled. Hashed storage is the storage behavior we rolled out with 10.0. Instead -of coupling project URL and the folder structure where the repository will be +of coupling project URL and the folder structure where the repository is stored on disk, we are coupling a hash, based on the project's ID. This makes the folder structure immutable, and therefore eliminates any requirement to synchronize state from URLs to disk structure. This means that renaming a group, -user, or project will cost only the database transaction, and will take effect +user, or project costs only the database transaction, and takes effect immediately. The hash also helps to spread the repositories more evenly on the disk, so the -top-level directory will contain less folders than the total amount of top-level +top-level directory contains fewer folders than the total number of top-level namespaces. The hash format is based on the hexadecimal representation of SHA256: @@ -64,7 +64,7 @@ by another folder with the next 2 characters. They are both stored in a special ### Translating hashed storage paths Troubleshooting problems with the Git repositories, adding hooks, and other -tasks will require you translate between the human readable project name +tasks requires you translate between the human readable project name and the hashed storage path. #### From project name to hashed path @@ -102,7 +102,7 @@ To translate from a hashed storage path to a project name: ProjectRepository.find_by(disk_path: '@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9').project ``` -The quoted string in that command is the directory tree you'll find on your +The quoted string in that command is the directory tree you can find on your GitLab server. For example, on a default Omnibus installation this would be `/var/opt/gitlab/git-data/repositories/@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git` with `.git` from the end of the directory name removed. @@ -117,7 +117,7 @@ The output includes the project ID and the project name: > [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/1606) in GitLab 12.1. -DANGER: **Warning:** +WARNING: Do not run `git prune` or `git gc` in pool repositories! This can cause data loss in "real" repositories that depend on the pool in question. @@ -135,7 +135,7 @@ when housekeeping is run on the source project. ### Hashed storage coverage migration -Files stored in an S3 compatible endpoint will not have the downsides +Files stored in an S3-compatible endpoint do not have the downsides mentioned earlier, if they are not prefixed with `#{namespace}/#{project_name}`, which is true for CI Cache and LFS Objects. @@ -179,11 +179,11 @@ LFS objects are also [S3 compatible](lfs/index.md#storing-lfs-objects-in-remote- ## Legacy storage -NOTE: **Deprecated:** +WARNING: In GitLab 13.0, hashed storage is enabled by default and the legacy storage is deprecated. If you haven't migrated yet, check the [migration instructions](raketasks/storage.md#migrate-to-hashed-storage). -Support for legacy storage will be removed in GitLab 14.0. If you're on GitLab +Support for legacy storage is scheduled to be removed in GitLab 14.0. If you're on GitLab 13.0 and later, switching new projects to legacy storage is not possible. The option to choose between hashed and legacy storage in the admin area has been disabled. @@ -199,7 +199,7 @@ easy for Administrators to find where the repository is stored. On the other hand this has some drawbacks: -Storage location will concentrate huge amount of top-level namespaces. The +Storage location concentrates a huge number of top-level namespaces. The impact can be reduced by the introduction of [multiple storage paths](repository_storage_paths.md). @@ -209,6 +209,6 @@ an old removed or renamed project sharing the same URL. This means that `mygroup/myproject` from your backup may not be the same original project that is at that same URL today. -Any change in the URL will need to be reflected on disk (when groups / users or +Any change in the URL needs to be reflected on disk (when groups / users or projects are renamed). This can add a lot of load in big installations, especially if using any type of network based filesystem. diff --git a/doc/administration/repository_storages.md b/doc/administration/repository_storages.md index af7a385e5a0..93d0ee3cac9 100644 --- a/doc/administration/repository_storages.md +++ b/doc/administration/repository_storages.md @@ -3,3 +3,6 @@ redirect_to: 'repository_storage_paths.md' --- This document was moved to [another location](repository_storage_paths.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/restart_gitlab.md b/doc/administration/restart_gitlab.md index f6a4a39ad60..4f104c6a63f 100644 --- a/doc/administration/restart_gitlab.md +++ b/doc/administration/restart_gitlab.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # How to restart GitLab diff --git a/doc/administration/scaling/index.md b/doc/administration/scaling/index.md index 748373c8941..4f934646ed6 100644 --- a/doc/administration/scaling/index.md +++ b/doc/administration/scaling/index.md @@ -3,3 +3,6 @@ redirect_to: ../reference_architectures/index.md --- This document was moved to [another location](../reference_architectures/index.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/server_hooks.md b/doc/administration/server_hooks.md index b5e975d397f..645d3bc4e9f 100644 --- a/doc/administration/server_hooks.md +++ b/doc/administration/server_hooks.md @@ -1,7 +1,7 @@ --- stage: Create group: Gitaly -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference, howto disqus_identifier: 'https://docs.gitlab.com/ee/administration/custom_hooks.html' --- @@ -85,7 +85,7 @@ configuration: - GitLab 13.0 and earlier, this is set in `gitlab-shell/config.yml`. - GitLab 13.1 and later, this is set in `gitaly/config.toml` under the `[hooks]` section. -NOTE: **Note:** +NOTE: The `custom_hooks_dir` value in `gitlab-shell/config.yml` is still honored in GitLab 13.1 and later if the value in `gitaly/config.toml` is blank or non-existent. @@ -123,13 +123,13 @@ Within a directory, server hooks: - Are executed in alphabetical order. - Stop executing when a hook exits with a non-zero value. -Note: +`<hook_name>.d` must be either `pre-receive.d`, `post-receive.d`, or `update.d` to work properly. +Any other names are ignored. -- `<hook_name>.d` must be either `pre-receive.d`, `post-receive.d`, or `update.d` to work properly. - Any other names are ignored. -- Files in `.d` directories must be executable and not match the backup file pattern (`*~`). -- For `<project>.git` you need to [translate](repository_storage_types.md#translating-hashed-storage-paths) - your project name into the hashed storage format that GitLab uses. +Files in `.d` directories must be executable and not match the backup file pattern (`*~`). + +For `<project>.git` you need to [translate](repository_storage_types.md#translating-hashed-storage-paths) +your project name into the hashed storage format that GitLab uses. ## Environment Variables @@ -152,7 +152,7 @@ Pre-receive and post-receive server hooks can also access the following Git envi | `GIT_PUSH_OPTION_COUNT` | Number of push options. See [Git `pre-receive` documentation](https://git-scm.com/docs/githooks#pre-receive). | | `GIT_PUSH_OPTION_<i>` | Value of push options where `i` is from `0` to `GIT_PUSH_OPTION_COUNT - 1`. See [Git `pre-receive` documentation](https://git-scm.com/docs/githooks#pre-receive). | -NOTE: **Note:** +NOTE: While other environment variables can be passed to server hooks, your application should not rely on them as they can change. @@ -160,7 +160,7 @@ them as they can change. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/5073) in GitLab 8.10. -To have custom error messages appear in GitLab's UI when a commit is declined or an error occurs +To have custom error messages appear in the GitLab UI when a commit is declined or an error occurs during the Git hook, your script should: - Send the custom error messages to either the script's `stdout` or `stderr`. @@ -168,7 +168,7 @@ during the Git hook, your script should: ### Example custom error message -This hook script written in Bash generates the following message in GitLab's UI: +This hook script written in Bash generates the following message in the GitLab UI: ```shell #!/bin/sh diff --git a/doc/administration/sidekiq.md b/doc/administration/sidekiq.md index 2cc6acc8c87..a97eff23c2f 100644 --- a/doc/administration/sidekiq.md +++ b/doc/administration/sidekiq.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -25,8 +25,9 @@ you want using steps 1 and 2 from the GitLab downloads page. ## Optional: Enable extra Sidekiq processes sidekiq_cluster['enable'] = true - sidekiq_cluster['enable'] = true - "elastic_indexer" + sidekiq['queue_groups'] = [ + "elastic_indexer", + "*" ] ``` diff --git a/doc/administration/smime_signing_email.md b/doc/administration/smime_signing_email.md index f5b40210e62..7492bf4457a 100644 --- a/doc/administration/smime_signing_email.md +++ b/doc/administration/smime_signing_email.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Signing outgoing email with S/MIME @@ -26,7 +26,7 @@ files must be provided: Optionally, you can also provide a bundle of CA certs (PEM-encoded) to be included on each signature. This will typically be an intermediate CA. -CAUTION: **Caution:** +WARNING: Be mindful of the access levels for your private keys and visibility to third parties. diff --git a/doc/administration/snippets/index.md b/doc/administration/snippets/index.md index 5e61d20c683..0bf0eb3b1ed 100644 --- a/doc/administration/snippets/index.md +++ b/doc/administration/snippets/index.md @@ -2,7 +2,7 @@ type: reference, howto stage: Create group: Editor -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers" +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" --- # Snippets settings **(CORE ONLY)** @@ -29,7 +29,7 @@ This setting is not available through the [Admin Area settings](../../user/admin In order to configure this setting, use either the Rails console or the [Application settings API](../../api/settings.md). -NOTE: **IMPORTANT:** +NOTE: The value of the limit **must** be in bytes. #### Through the Rails console @@ -64,11 +64,11 @@ The process to set the snippets size limit through the Application Settings API exactly the same as you would do to [update any other setting](../../api/settings.md#change-application-settings). ```shell -curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/application/settings?snippet_size_limit=52428800 +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/application/settings?snippet_size_limit=52428800" ``` You can also use the API to [retrieve the current value](../../api/settings.md#get-current-application-settings). ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/application/settings +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/application/settings" ``` diff --git a/doc/administration/static_objects_external_storage.md b/doc/administration/static_objects_external_storage.md index 34c7c8947fc..b10af12de67 100644 --- a/doc/administration/static_objects_external_storage.md +++ b/doc/administration/static_objects_external_storage.md @@ -1,7 +1,7 @@ --- stage: Create group: Editor -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers" +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" type: reference --- diff --git a/doc/administration/terraform_state.md b/doc/administration/terraform_state.md index 55e166d2bf7..edd44380f30 100644 --- a/doc/administration/terraform_state.md +++ b/doc/administration/terraform_state.md @@ -1,7 +1,7 @@ --- stage: Configure group: Configure -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Terraform state administration (alpha) @@ -20,7 +20,7 @@ These locations can be configured using the options described below. ## Using local storage -NOTE: **Note:** +NOTE: This is the default configuration To change the location where Terraform state files are stored locally, follow the steps @@ -52,8 +52,8 @@ below. ## Using object storage **(CORE ONLY)** -Instead of storing Terraform state files on disk, we recommend the use of an object -store that is S3-compatible instead. This configuration relies on valid credentials to +Instead of storing Terraform state files on disk, we recommend the use of [one of the supported object +storage options](object_storage.md#options). This configuration relies on valid credentials to be configured already. [Read more about using object storage with GitLab](object_storage.md). @@ -68,7 +68,7 @@ The following settings are: | Setting | Description | Default | |---------|-------------|---------| | `enabled` | Enable/disable object storage | `false` | -| `remote_directory` | The bucket name where Terraform state files will be stored | | +| `remote_directory` | The bucket name where Terraform state files are stored | | | `connection` | Various connection options described below | | ### S3-compatible connection settings @@ -91,7 +91,7 @@ See [the available connection settings for different providers](object_storage.m } ``` - NOTE: **Note:** + NOTE: If you are using AWS IAM profiles, be sure to omit the AWS access key and secret access key/value pairs. ```ruby diff --git a/doc/administration/timezone.md b/doc/administration/timezone.md index bec82f66948..798e7f5050c 100644 --- a/doc/administration/timezone.md +++ b/doc/administration/timezone.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Changing your time zone @@ -20,7 +20,7 @@ To see all available time zones, run `bundle exec rake time:zones:all`. For Omnibus installations, run `gitlab-rake time:zones:all`. -NOTE: **Note:** +NOTE: This Rake task does not list timezones in TZInfo format required by Omnibus GitLab during a reconfigure: [#27209](https://gitlab.com/gitlab-org/gitlab/-/issues/27209). ## Changing time zone in Omnibus installations diff --git a/doc/administration/troubleshooting/debug.md b/doc/administration/troubleshooting/debug.md index d67c9963092..8c8fa25aa5e 100644 --- a/doc/administration/troubleshooting/debug.md +++ b/doc/administration/troubleshooting/debug.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Debugging Tips diff --git a/doc/administration/troubleshooting/diagnostics_tools.md b/doc/administration/troubleshooting/diagnostics_tools.md index 132f8524ca1..27a7493b318 100644 --- a/doc/administration/troubleshooting/diagnostics_tools.md +++ b/doc/administration/troubleshooting/diagnostics_tools.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- diff --git a/doc/administration/troubleshooting/elasticsearch.md b/doc/administration/troubleshooting/elasticsearch.md index 12aa91e6f14..755273eb06e 100644 --- a/doc/administration/troubleshooting/elasticsearch.md +++ b/doc/administration/troubleshooting/elasticsearch.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Global Search -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Troubleshooting Elasticsearch diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index cae4aa96f75..2482a4fe7ad 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -14,13 +14,13 @@ having an issue with GitLab, it is highly recommended that you check your [support options](https://about.gitlab.com/support/) first, before attempting to use this information. -CAUTION: **Caution:** +WARNING: Please note that some of these scripts could be damaging if not run correctly, or under the right conditions. We highly recommend running them under the guidance of a Support Engineer, or running them in a test environment with a backup of the instance ready to be restored, just in case. -CAUTION: **Caution:** +WARNING: Please also note that as GitLab changes, changes to the code are inevitable, and so some scripts may not work as they once used to. These are not kept up-to-date as these scripts/commands were added as they were found/needed. As @@ -308,7 +308,7 @@ pp p.statistics # compare with earlier values ### Recreate -CAUTION: **Caution:** +WARNING: This is a destructive operation, the Wiki will be empty. A Projects Wiki can be recreated by this command: @@ -374,6 +374,26 @@ Clear the cache: sudo gitlab-rake cache:clear ``` +### Export a repository + +It's typically recommended to export a project through [the web interface](../../user/project/settings/import_export.md#exporting-a-project-and-its-data) or through [the API](../../api/project_import_export.md). In situations where this is not working as expected, it may be preferable to export a project directly via the Rails console: + +```ruby +user = User.find_by_username('USERNAME') +project = Project.find_by_full_path('PROJECT_PATH') +Projects::ImportExport::ExportService.new(project, user).execute +``` + +If the project you wish to export is available at `https://gitlab.example.com/baltig/pipeline-templates`, the value to use for `PROJECT_PATH` would be `baltig/pipeline-templates`. + +If this all runs successfully, you will see output like the following before being returned to the Rails console prompt: + +```ruby +=> nil +``` + +The exported project will be located within a `.tar.gz` file in `/var/opt/gitlab/gitlab-rails/uploads/-/system/import_export_upload/export_file/`. + ## Repository ### Search sequence of pushes to a repository @@ -464,8 +484,9 @@ User.billable.count ::HistoricalData.max_historical_user_count ``` +Using cURL and jq (up to a max 100, see the [pagination docs](../../api/README.md#pagination)): + ```shell -# Using curl and jq (up to a max 100, see pagination docs https://docs.gitlab.com/ee/api/#pagination curl --silent --header "Private-Token: ********************" "https://gitlab.example.com/api/v4/users?per_page=100&active" | jq --compact-output '.[] | [.id,.name,.username]' ``` @@ -491,10 +512,22 @@ users.each do |user| end ``` +### Deactivate Users that have no recent activity + +```ruby +days_inactive = 90 +inactive_users = User.active.where("last_activity_on <= ?", days_inactive.days.ago) + +inactive_users.each do |user| + puts "user '#{user.username}': #{user.last_activity_on}" + user.deactivate! +end +``` + ### Block Users that have no recent activity ```ruby -days_inactive = 60 +days_inactive = 90 inactive_users = User.active.where("last_activity_on <= ?", days_inactive.days.ago) inactive_users.each do |user| @@ -565,6 +598,17 @@ group = Group.find_by_path_or_name('group-name') group.project_creation_level=0 ``` +### Modify group - disable 2FA requirement + +WARNING: +When disabling the 2FA Requirement on a subgroup, the whole parent group (including all subgroups) is affected by this change. + +```ruby +group = Group.find_by_path_or_name('group-name') +group.require_two_factor_authentication=false +group.save +``` + ## SCIM ### Fixing bad SCIM identities @@ -1015,3 +1059,55 @@ This will also refresh the cached usage ping displayed in the admin area ```ruby Gitlab::UsageData.to_json(force_refresh: true) ``` + +#### Generate and print + +Generates usage ping data in JSON format. + +```shell +rake gitlab:usage_data:generate +``` + +#### Generate and send usage ping + +Prints the metrics saved in `conversational_development_index_metrics`. + +```shell +rake gitlab:usage_data:generate_and_send +``` + +## Elasticsearch + +### Configuration attributes + +Open the rails console (`gitlab rails c`) and run the following command to see all the available attributes: + +```ruby +ApplicationSetting.last.attributes +``` + +Among other attributes, in the output you will notice that all the settings available in the [Elasticsearch Integration page](../../integration/elasticsearch.md), like: `elasticsearch_indexing`, `elasticsearch_url`, `elasticsearch_replicas`, `elasticsearch_pause_indexing`, etc. + +#### Setting attributes + +You can then set anyone of Elasticsearch integration settings by issuing a command similar to: + +```ruby +ApplicationSetting.last.update_attributes(elasticsearch_url: '<your ES URL and port>') + +#or + +ApplicationSetting.last.update_attributes(elasticsearch_indexing: false) +``` + +#### Getting attributes + +You can then check if the settings have been set in the [Elasticsearch Integration page](../../integration/elasticsearch.md) or in the rails console by issuing: + +```ruby +Gitlab::CurrentSettings.elasticsearch_url + +#or + +Gitlab::CurrentSettings.elasticsearch_indexing +``` diff --git a/doc/administration/troubleshooting/group_saml_scim.md b/doc/administration/troubleshooting/group_saml_scim.md index efc8eaab198..e9dbbfbde12 100644 --- a/doc/administration/troubleshooting/group_saml_scim.md +++ b/doc/administration/troubleshooting/group_saml_scim.md @@ -1,7 +1,7 @@ --- stage: Manage group: Access -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -9,7 +9,7 @@ type: reference These are notes and screenshots regarding Group SAML and SCIM that the GitLab Support Team sometimes uses while troubleshooting, but which do not fit into the official documentation. GitLab is making this public, so that anyone can make use of the Support teamâs collected knowledge. -Please refer to GitLab's [Group SAML](../../user/group/saml_sso/index.md) docs for information on the feature and how to set it up. +Please refer to the GitLab [Group SAML](../../user/group/saml_sso/index.md) docs for information on the feature and how to set it up. When troubleshooting a SAML configuration, GitLab team members will frequently start with the [SAML troubleshooting section](../../user/group/saml_sso/index.md#troubleshooting). @@ -22,7 +22,7 @@ This section includes relevant screenshots of the following example configuratio - [Azure Active Directory](#azure-active-directory) - [OneLogin](#onelogin) -CAUTION: **Caution:** +WARNING: These screenshots are updated only as needed by GitLab Support. They are **not** official documentation. If you are currently having an issue with GitLab, you may want to check your [support options](https://about.gitlab.com/support/). diff --git a/doc/administration/troubleshooting/img/network_monitor_xid.png b/doc/administration/troubleshooting/img/network_monitor_xid.png Binary files differindex 7fc2cf47ea0..fa64fd1509c 100644 --- a/doc/administration/troubleshooting/img/network_monitor_xid.png +++ b/doc/administration/troubleshooting/img/network_monitor_xid.png diff --git a/doc/administration/troubleshooting/index.md b/doc/administration/troubleshooting/index.md index 8a4a0a4caac..67115ce31c0 100644 --- a/doc/administration/troubleshooting/index.md +++ b/doc/administration/troubleshooting/index.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Troubleshooting a GitLab installation diff --git a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md index 21fd183dfd0..07a7baf338b 100644 --- a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md +++ b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -11,7 +11,7 @@ This is a list of useful information regarding Kubernetes that the GitLab Suppor Team sometimes uses while troubleshooting. GitLab is making this public, so that anyone can make use of the Support team's collected knowledge -CAUTION: **Caution:** +WARNING: These commands **can alter or break** your Kubernetes components so use these at your own risk. If you are on a [paid tier](https://about.gitlab.com/pricing/) and are not sure how @@ -126,7 +126,7 @@ and they will assist you with any issues you are having. kubectl get pods | grep task-runner # enter it - kubectl exec -it <task-runner-pod-name> bash + kubectl exec -it <task-runner-pod-name> -- bash # open rails console # rails console can be also called from other GitLab pods @@ -139,10 +139,10 @@ and they will assist you with any issues you are having. /usr/local/bin/gitlab-rake gitlab:check # open console without entering pod - kubectl exec -it <task-runner-pod-name> /srv/gitlab/bin/rails console + kubectl exec -it <task-runner-pod-name> -- /srv/gitlab/bin/rails console # check the status of DB migrations - kubectl exec -it <task-runner-pod-name> /usr/local/bin/gitlab-rake db:migrate:status + kubectl exec -it <task-runner-pod-name> -- /usr/local/bin/gitlab-rake db:migrate:status ``` You can also use `gitlab-rake`, instead of `/usr/local/bin/gitlab-rake`. @@ -207,7 +207,7 @@ all Kubernetes resources and dependent charts: helm get manifest <release name> ``` -## Installation of minimal GitLab config via Minikube on macOS +## Installation of minimal GitLab configuration via Minikube on macOS This section is based on [Developing for Kubernetes with Minikube](https://docs.gitlab.com/charts/development/minikube/index.html) and [Helm](https://docs.gitlab.com/charts/installation/tools.html#helm). Refer diff --git a/doc/administration/troubleshooting/linux_cheat_sheet.md b/doc/administration/troubleshooting/linux_cheat_sheet.md index b1042a9402b..c61a78624c3 100644 --- a/doc/administration/troubleshooting/linux_cheat_sheet.md +++ b/doc/administration/troubleshooting/linux_cheat_sheet.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -13,7 +13,7 @@ and it may be useful for users with experience with Linux. If you are currently having an issue with GitLab, you may want to check your [support options](https://about.gitlab.com/support/) first, before attempting to use this information. -CAUTION: **Caution:** +WARNING: If you are administering GitLab you are expected to know these commands for your distribution of choice. If you are a GitLab Support Engineer, consider this a cross-reference to translate `yum` -> `apt-get` and the like. @@ -23,7 +23,7 @@ on. Contributions are welcome to help add them. ## System Commands -### Distro Information +### Distribution Information ```shell # Debian/Ubuntu @@ -200,7 +200,7 @@ or you can build it from source if you have the Rust compiler. #### How to use the tool -First run the tool with no arguments other than the strace output file name to get +First run the tool with no arguments other than the strace output filename to get a summary of the top processes sorted by time spent actively performing tasks. You can also sort based on total time, # of syscalls made, PID #, and # of child processes using the `-S` or `--sort` flag. The number of results defaults to 25 processes, but @@ -303,7 +303,7 @@ nslookup example.com 1.1.1.1 whois <ip_address> | grep -i "orgname\|netname" # Curl headers with redirect -curl --head --location https://example.com +curl --head --location "https://example.com" ``` ## Package Management diff --git a/doc/administration/troubleshooting/log_parsing.md b/doc/administration/troubleshooting/log_parsing.md index 4e9e8cd591f..144aa0f6d3b 100644 --- a/doc/administration/troubleshooting/log_parsing.md +++ b/doc/administration/troubleshooting/log_parsing.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Parsing GitLab logs with `jq` @@ -143,16 +143,39 @@ jq 'select(."grpc.code" != null and ."grpc.code" != "OK")' current jq 'select(."grpc.time_ms" > 30000)' current ``` -#### Print top three projects by request volume and their three longest durations - -```shell -jq -s -r 'map(select(."grpc.request.glProjectPath" != null and ."grpc.request.glProjectPath" != "" and ."grpc.time_ms" != null)) | group_by(."grpc.request.glProjectPath") | sort_by(-length) | limit(3; .[]) | sort_by(-."grpc.time_ms") | "CT: \(length)\tPROJECT: \(.[0]."grpc.request.glProjectPath")\tDURS: \(.[0]."grpc.time_ms"), \(.[1]."grpc.time_ms"), \(.[2]."grpc.time_ms")"' current +#### Print top ten projects by request volume and their three longest durations + +```shell +jq --raw-output --slurp ' + map( + select( + ."grpc.request.glProjectPath" != null + and ."grpc.request.glProjectPath" != "" + and ."grpc.time_ms" != null + ) + ) + | group_by(."grpc.request.glProjectPath") + | sort_by(-length) + | limit(10; .[]) + | sort_by(-."grpc.time_ms") + | [ + length, + .[0]."grpc.time_ms", + .[1]."grpc.time_ms", + .[2]."grpc.time_ms", + .[0]."grpc.request.glProjectPath" + ] + | @sh' /var/log/gitlab/gitaly/current \ +| awk 'BEGIN { printf "%7s %10s %10s %10s\t%s\n", "CT", "MAX DURS", "", "", "PROJECT" } + { printf "%7u %7u ms, %7u ms, %7u ms\t%s\n", $1, $2, $3, $4, $5 }' ``` **Example output** ```plaintext -CT: 635 PROJECT: groupA/project1 DURS: 4292.269, 4228.853, 2885.548 -CT: 462 PROJECT: groupB/project5 DURS: 4368.981, 3623.553, 361.399 -CT: 455 PROJECT: groupC/project7 DURS: 387.295, 381.874, 373.988 + CT MAX DURS PROJECT + 206 4898 ms, 1101 ms, 1032 ms 'groupD/project4' + 109 1420 ms, 962 ms, 875 ms 'groupEF/project56' + 663 106 ms, 96 ms, 94 ms 'groupABC/project123' + ... ``` diff --git a/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md b/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md index 475f3d56836..68c12117222 100644 --- a/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md +++ b/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Navigating GitLab via Rails console @@ -12,7 +12,7 @@ Thanks to this, we also get access to the amazing tools built right into Rails. In this guide, we'll introduce the [Rails console](../operations/rails_console.md#starting-a-rails-console-session) and the basics of interacting with your GitLab instance from the command line. -CAUTION: **Caution:** +WARNING: The Rails console interacts directly with your GitLab instance. In many cases, there are no handrails to prevent you from permanently modifying, corrupting or destroying production data. If you would like to explore the Rails console diff --git a/doc/administration/troubleshooting/postgresql.md b/doc/administration/troubleshooting/postgresql.md index d22e76a505a..7052b68370c 100644 --- a/doc/administration/troubleshooting/postgresql.md +++ b/doc/administration/troubleshooting/postgresql.md @@ -1,22 +1,23 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- # PostgreSQL -This page is useful information about PostgreSQL that the GitLab Support -Team sometimes uses while troubleshooting. GitLab is making this public, so that anyone -can make use of the Support team's collected knowledge. +This page contains information about PostgreSQL the GitLab Support team uses +when troubleshooting. GitLab makes this information public, so that anyone can +make use of the Support team's collected knowledge. -CAUTION: **Caution:** -Some procedures documented here may break your GitLab instance. Use at your own risk. +WARNING: +Some procedures documented here may break your GitLab instance. Use at your +own risk. -If you are on a [paid tier](https://about.gitlab.com/pricing/) and are not sure how -to use these commands, it is best to [contact Support](https://about.gitlab.com/support/) -and they will assist you with any issues you are having. +If you're on a [paid tier](https://about.gitlab.com/pricing/) and aren't sure +how to use these commands, [contact Support](https://about.gitlab.com/support/) +for assistance with any issues you're having. ## Other GitLab PostgreSQL documentation @@ -24,51 +25,57 @@ This section is for links to information elsewhere in the GitLab documentation. ### Procedures -- [Connect to the PostgreSQL console.](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database) +- [Connect to the PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database). -- [Omnibus database procedures](https://docs.gitlab.com/omnibus/settings/database.html) including +- [Omnibus database procedures](https://docs.gitlab.com/omnibus/settings/database.html) including: - SSL: enabling, disabling, and verifying. - Enabling Write Ahead Log (WAL) archiving. - Using an external (non-Omnibus) PostgreSQL installation; and backing it up. - Listening on TCP/IP as well as or instead of sockets. - Storing data in another location. - Destructively reseeding the GitLab database. - - Guidance around updating packaged PostgreSQL, including how to stop it happening automatically. + - Guidance around updating packaged PostgreSQL, including how to stop it + happening automatically. -- [More about external PostgreSQL](../postgresql/external.md) +- [Information about external PostgreSQL](../postgresql/external.md). -- [Running Geo with external PostgreSQL](../geo/setup/external_database.md) +- [Running Geo with external PostgreSQL](../geo/setup/external_database.md). -- [Upgrades when running PostgreSQL configured for HA.](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-gitlab-ha-cluster) +- [Upgrades when running PostgreSQL configured for HA](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-gitlab-ha-cluster). -- Consuming PostgreSQL from [within CI runners](../../ci/services/postgres.md) +- Consuming PostgreSQL from [within CI runners](../../ci/services/postgres.md). -- [Using Slony to update PostgreSQL](../../update/upgrading_postgresql_using_slony.md) - - Uses replication to handle PostgreSQL upgrades - providing the schemas are the same. - - Reduces downtime to a short window for swinging over to the newer version. +- [Using Slony to update PostgreSQL](../../update/upgrading_postgresql_using_slony.md). + - Uses replication to handle PostgreSQL upgrades if the schemas are the same. + - Reduces downtime to a short window for switching to the newer version. -- Managing Omnibus PostgreSQL versions [from the development docs](https://docs.gitlab.com/omnibus/development/managing-postgresql-versions.html) +- Managing Omnibus PostgreSQL versions [from the development docs](https://docs.gitlab.com/omnibus/development/managing-postgresql-versions.html). - [PostgreSQL scaling](../postgresql/replication_and_failover.md) - - including [troubleshooting](../postgresql/replication_and_failover.md#troubleshooting) `gitlab-ctl repmgr-check-master` (or `gitlab-ctl patroni check-leader` if you are using Patroni) and PgBouncer errors + - Including [troubleshooting](../postgresql/replication_and_failover.md#troubleshooting) + `gitlab-ctl repmgr-check-master` (or `gitlab-ctl patroni check-leader` if + you're using Patroni) and PgBouncer errors. -- [Developer database documentation](../../development/README.md#database-guides) - some of which is absolutely not for production use. Including: - - understanding EXPLAIN plans +- [Developer database documentation](../../development/README.md#database-guides), + some of which is absolutely not for production use. Including: + - Understanding EXPLAIN plans. ### Troubleshooting/Fixes -- [GitLab database requirements](../../install/requirements.md#database) including - - Support for MySQL was removed in GitLab 12.1; [migrate to PostgreSQL](../../update/mysql_to_postgresql.md) - - required extension `pg_trgm` - - required extension `btree_gist` +- [GitLab database requirements](../../install/requirements.md#database), + including + - Support for MySQL was removed in GitLab 12.1; [migrate to PostgreSQL](../../update/mysql_to_postgresql.md). + - Required extension: `pg_trgm` + - Required extension: `btree_gist` -- Errors like this in the `production/sidekiq` log; see: [Set default_transaction_isolation into read committed](https://docs.gitlab.com/omnibus/settings/database.html#set-default_transaction_isolation-into-read-committed): +- Errors like this in the `production/sidekiq` log; see: + [Set default_transaction_isolation into read committed](https://docs.gitlab.com/omnibus/settings/database.html#set-default_transaction_isolation-into-read-committed): ```plaintext ActiveRecord::StatementInvalid PG::TRSerializationFailure: ERROR: could not serialize access due to concurrent update ``` -- PostgreSQL HA - [replication slot errors](https://docs.gitlab.com/omnibus/settings/database.html#troubleshooting-upgrades-in-an-ha-cluster): +- PostgreSQL HA [replication slot errors](https://docs.gitlab.com/omnibus/settings/database.html#troubleshooting-upgrades-in-an-ha-cluster): ```plaintext pg_basebackup: could not create temporary replication slot "pg_basebackup_12345": ERROR: all replication slots are in use @@ -87,11 +94,11 @@ This section is for links to information elsewhere in the GitLab documentation. PANIC: could not write to file âpg_xlog/xlogtemp.123â: No space left on device ``` -- [Checking Geo configuration](../geo/replication/troubleshooting.md) including - - reconfiguring hosts/ports - - checking and fixing user/password mappings +- [Checking Geo configuration](../geo/replication/troubleshooting.md), including: + - Reconfiguring hosts/ports. + - Checking and fixing user/password mappings. -- [Common Geo errors](../geo/replication/troubleshooting.md#fixing-common-errors) +- [Common Geo errors](../geo/replication/troubleshooting.md#fixing-common-errors). ## Support topics @@ -99,9 +106,12 @@ This section is for links to information elsewhere in the GitLab documentation. References: -- [Issue #1 Deadlocks with GitLab 12.1, PostgreSQL 10.7](https://gitlab.com/gitlab-org/gitlab/-/issues/30528) -- [Customer ticket (internal) GitLab 12.1.6](https://gitlab.zendesk.com/agent/tickets/134307) and [Google doc (internal)](https://docs.google.com/document/d/19xw2d_D1ChLiU-MO1QzWab-4-QXgsIUcN5e_04WTKy4) -- [Issue #2 deadlocks can occur if an instance is flooded with pushes](https://gitlab.com/gitlab-org/gitlab/-/issues/33650). Provided for context about how GitLab code can have this sort of unanticipated effect in unusual situations. +- [Issue #1 Deadlocks with GitLab 12.1, PostgreSQL 10.7](https://gitlab.com/gitlab-org/gitlab/-/issues/30528). +- [Customer ticket (internal) GitLab 12.1.6](https://gitlab.zendesk.com/agent/tickets/134307) + and [Google doc (internal)](https://docs.google.com/document/d/19xw2d_D1ChLiU-MO1QzWab-4-QXgsIUcN5e_04WTKy4). +- [Issue #2 deadlocks can occur if an instance is flooded with pushes](https://gitlab.com/gitlab-org/gitlab/-/issues/33650). + Provided for context about how GitLab code can have this sort of + unanticipated effect in unusual situations. ```plaintext ERROR: deadlock detected @@ -119,17 +129,29 @@ Quoting from issue [#1](https://gitlab.com/gitlab-org/gitlab/-/issues/30528): > "If a deadlock is hit, and we resolve it through aborting the transaction after a short period, then the retry mechanisms we already have will make the deadlocked piece of work try again, and it's unlikely we'll deadlock multiple times in a row." -TIP: **Tip:** -In support, our general approach to reconfiguring timeouts (applies also to the HTTP stack as well) is that it's acceptable to do it temporarily as a workaround. If it makes GitLab usable for the customer, then it buys time to understand the problem more completely, implement a hot fix, or make some other change that addresses the root cause. Generally, the timeouts should be put back to reasonable defaults once the root cause is resolved. +NOTE: +In Support, our general approach to reconfiguring timeouts (applies also to the +HTTP stack) is that it's acceptable to do it temporarily as a workaround. If it +makes GitLab usable for the customer, then it buys time to understand the +problem more completely, implement a hot fix, or make some other change that +addresses the root cause. Generally, the timeouts should be put back to +reasonable defaults after the root cause is resolved. -In this case, the guidance we had from development was to drop deadlock_timeout and/or statement_timeout but to leave the third setting at 60s. Setting idle_in_transaction protects the database from sessions potentially hanging for days. There's more discussion in [the issue relating to introducing this timeout on GitLab.com](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1053). +In this case, the guidance we had from development was to drop deadlock_timeout +or statement_timeout, but to leave the third setting at 60s. Setting +idle_in_transaction protects the database from sessions potentially hanging for +days. There's more discussion in [the issue relating to introducing this timeout on GitLab.com](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1053). PostgresSQL defaults: - `statement_timeout = 0` (never) - `idle_in_transaction_session_timeout = 0` (never) -Comments in issue [#1](https://gitlab.com/gitlab-org/gitlab/-/issues/30528) indicate that these should both be set to at least a number of minutes for all Omnibus installations (so they don't hang indefinitely). However, 15s for statement_timeout is very short, and will only be effective if the underlying infrastructure is very performant. +Comments in issue [#1](https://gitlab.com/gitlab-org/gitlab/-/issues/30528) +indicate that these should both be set to at least a number of minutes for all +Omnibus GitLab installations (so they don't hang indefinitely). However, 15s +for statement_timeout is very short, and will only be effective if the +underlying infrastructure is very performant. See current settings with: @@ -147,5 +169,5 @@ It may take a little while to respond. {"idle_in_transaction_session_timeout"=>"1min"} ``` -NOTE: **Note:** +NOTE: These are Omnibus GitLab settings. If an external database, such as a customer's PostgreSQL installation or Amazon RDS is being used, these values don't get set, and would have to be set externally. diff --git a/doc/administration/troubleshooting/sidekiq.md b/doc/administration/troubleshooting/sidekiq.md index d415aa0d980..e4082f87c7d 100644 --- a/doc/administration/troubleshooting/sidekiq.md +++ b/doc/administration/troubleshooting/sidekiq.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Troubleshooting Sidekiq diff --git a/doc/administration/troubleshooting/ssl.md b/doc/administration/troubleshooting/ssl.md index 996856521b7..d7bfd537eca 100644 --- a/doc/administration/troubleshooting/ssl.md +++ b/doc/administration/troubleshooting/ssl.md @@ -1,32 +1,36 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- # Troubleshooting SSL -This page contains a list of common SSL-related errors and scenarios that you may face while working with GitLab. -It should serve as an addition to the main SSL docs available here: +This page contains a list of common SSL-related errors and scenarios that you +may encounter while working with GitLab. It should serve as an addition to the +main SSL docs available here: -- [Omnibus SSL Configuration](https://docs.gitlab.com/omnibus/settings/ssl.html) -- [Self-signed certificates or custom Certification Authorities for GitLab Runner](https://docs.gitlab.com/runner/configuration/tls-self-signed.html) -- [Manually configuring HTTPS](https://docs.gitlab.com/omnibus/settings/nginx.html#manually-configuring-https) +- [Omnibus SSL Configuration](https://docs.gitlab.com/omnibus/settings/ssl.html). +- [Self-signed certificates or custom Certification Authorities for GitLab Runner](https://docs.gitlab.com/runner/configuration/tls-self-signed.html). +- [Manually configuring HTTPS](https://docs.gitlab.com/omnibus/settings/nginx.html#manually-configuring-https). ## Using an internal CA certificate with GitLab -After configuring a GitLab instance with an internal CA certificate, you might not be able to access it via various CLI tools. You may see the following symptoms: +After configuring a GitLab instance with an internal CA certificate, you might +not be able to access it by using various CLI tools. You may see experience the +following issues: - `curl` fails: ```shell - curl https://gitlab.domain.tld + curl "https://gitlab.domain.tld" curl: (60) SSL certificate problem: unable to get local issuer certificate More details here: https://curl.haxx.se/docs/sslcerts.html ``` -- Testing via the [rails console](../operations/rails_console.md#starting-a-rails-console-session) also fails: +- Testing by using the [rails console](../operations/rails_console.md#starting-a-rails-console-session) + also fails: ```ruby uri = URI.parse("https://gitlab.domain.tld") @@ -40,33 +44,36 @@ After configuring a GitLab instance with an internal CA certificate, you might n OpenSSL::SSL::SSLError (SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)) ``` -- The error `SSL certificate problem: unable to get local issuer certificate` is shown when setting up a [mirror](../../user/project/repository/repository_mirroring.md#repository-mirroring) from this GitLab instance. +- The error `SSL certificate problem: unable to get local issuer certificate` + is displayed when setting up a [mirror](../../user/project/repository/repository_mirroring.md#repository-mirroring) + from this GitLab instance. - `openssl` works when specifying the path to the certificate: ```shell /opt/gitlab/embedded/bin/openssl s_client -CAfile /root/my-cert.crt -connect gitlab.domain.tld:443 ``` -If you have the problems listed above, add your certificate to `/etc/gitlab/trusted-certs` and run `sudo gitlab-ctl reconfigure`. +If you have the previously described issues, add your certificate to +`/etc/gitlab/trusted-certs`, and then run `sudo gitlab-ctl reconfigure`. ## X.509 key values mismatch error -After configuring your instance with a certificate bundle, NGINX may throw the -following error: +After configuring your instance with a certificate bundle, NGINX may display +the following error message: `SSL: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch` -This error means that the server certificate and key you have provided do not -match. You can confirm this by running the following command and comparing the -output: +This error message means that the server certificate and key you have provided +don't match. You can confirm this by running the following command and then +comparing the output: ```shell openssl rsa -noout -modulus -in path/to/your/.key | openssl md5 openssl x509 -noout -modulus -in path/to/your/.crt | openssl md5 ``` -The following is an example of an md5 output between a matching key and certificate. Note the -matching md5 hashes: +The following is an example of an md5 output between a matching key and +certificate. Note the matching md5 hashes: ```shell $ openssl rsa -noout -modulus -in private.key | openssl md5 @@ -75,7 +82,8 @@ $ openssl x509 -noout -modulus -in public.crt | openssl md5 4f49b61b25225abeb7542b29ae20e98c ``` -This is an opposing output with a non-matching key and certificate which shows different md5 hashes: +This is an opposing output with a non-matching key and certificate which shows +different md5 hashes: ```shell $ openssl rsa -noout -modulus -in private.key | openssl md5 @@ -84,14 +92,16 @@ $ openssl x509 -noout -modulus -in public.crt | openssl md5 4f49b61b25225abeb7542b29ae20e98c ``` -If the two outputs differ like the above example, there is a mismatch between the certificate -and key. You should contact the provider of the SSL certificate for further support. +If the two outputs differ like the previous example, there's a mismatch between +the certificate and key. Contact the provider of the SSL certificate for +further support. ## Using GitLab Runner with a GitLab instance configured with internal CA certificate or self-signed certificate Besides getting the errors mentioned in [Using an internal CA certificate with GitLab](ssl.md#using-an-internal-ca-certificate-with-gitlab), -your CI pipelines may get stuck in `Pending` status. In the runner logs you may see the below error: +your CI pipelines may get stuck in `Pending` status. In the runner logs you may +see the following error message: ```shell Dec 6 02:43:17 runner-host01 gitlab-runner[15131]: #033[0;33mWARNING: Checking for jobs... failed @@ -100,23 +110,27 @@ https://gitlab.domain.tld/api/v4/jobs/request: Post https://gitlab.domain.tld/ap x509: certificate signed by unknown authority ``` -If you face similar problem, add your certificate to `/etc/gitlab-runner/certs` and restart the runner via `gitlab-runner restart`. +If you encounter a similar problem, add your certificate to `/etc/gitlab-runner/certs`, +and the restart the runner by running `gitlab-runner restart`. ## Mirroring a remote GitLab repository that uses a self-signed SSL certificate -**Scenario:** When configuring a local GitLab instance to [mirror a repository](../../user/project/repository/repository_mirroring.md) from a remote GitLab instance that uses a self-signed certificate, you may see the `SSL certificate problem: self signed certificate` error in the UI. +When configuring a local GitLab instance to [mirror a repository](../../user/project/repository/repository_mirroring.md) +from a remote GitLab instance that uses a self-signed certificate, you may see +the `SSL certificate problem: self signed certificate` error message in the +user interface. The cause of the issue can be confirmed by checking if: - `curl` fails: ```shell - $ curl https://gitlab.domain.tld + $ curl "https://gitlab.domain.tld" curl: (60) SSL certificate problem: self signed certificate More details here: https://curl.haxx.se/docs/sslcerts.html ``` -- Testing via the Rails console also fails: +- Testing by using the Rails console also fails: ```ruby uri = URI.parse("https://gitlab.domain.tld") @@ -132,10 +146,15 @@ The cause of the issue can be confirmed by checking if: To fix this problem: -- Add the self-signed certificate from the remote GitLab instance to the `/etc/gitlab/trusted-certs` directory on the local GitLab instance and run `sudo gitlab-ctl reconfigure` as per the instructions for [installing custom public certificates](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). -- If your local GitLab instance was installed using the Helm Charts, you can [add your self-signed certificate to your GitLab instance](https://docs.gitlab.com/runner/install/kubernetes.html#providing-a-custom-certificate-for-accessing-gitlab). +- Add the self-signed certificate from the remote GitLab instance to the + `/etc/gitlab/trusted-certs` directory on the local GitLab instance, and then + run `sudo gitlab-ctl reconfigure` as per the instructions for + [installing custom public certificates](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). +- If your local GitLab instance was installed using the Helm Charts, you can + [add your self-signed certificate to your GitLab instance](https://docs.gitlab.com/runner/install/kubernetes.html#providing-a-custom-certificate-for-accessing-gitlab). -You may also get another error when trying to mirror a repository from a remote GitLab instance that uses a self-signed certificate: +You may also get another error message when trying to mirror a repository from +a remote GitLab instance that uses a self-signed certificate: ```shell 2:Fetching remote upstream failed: fatal: unable to access &#39;https://gitlab.domain.tld/root/test-repo/&#39;: @@ -144,12 +163,16 @@ SSL: unable to obtain common name from peer certificate In this case, the problem can be related to the certificate itself: -- Double check that your self-signed certificate is not missing a common name. If it is then regenerate a valid certificate -- add it to `/etc/gitlab/trusted-certs` and run `sudo gitlab-ctl reconfigure` +1. Validate that your self-signed certificate isn't missing a common name. If it + is, regenerate a valid certificate +1. Add the certificate to `/etc/gitlab/trusted-certs`. +1. Run `sudo gitlab-ctl reconfigure`. ## Unable to perform Git operations due to an internal or self-signed certificate -If your GitLab instance is using a self-signed certificate, or the certificate is signed by an internal certificate authority (CA), you might run into the following errors when attempting to perform Git operations: +If your GitLab instance is using a self-signed certificate, or if the +certificate is signed by an internal certificate authority (CA), you might +experience the following errors when attempting to perform Git operations: ```shell $ git clone https://gitlab.domain.tld/group/project.git @@ -165,15 +188,19 @@ fatal: unable to access 'https://gitlab.domain.tld/group/project.git/': server c To fix this problem: -- If possible, use SSH remotes for all Git operations. This is considered more secure and convenient to use. +- If possible, use SSH remotes for all Git operations. This is considered more + secure and convenient to use. - If you must use HTTPS remotes, you can try the following: - - Copy the self signed certificate or the internal root CA certificate to a local directory (for example, `~/.ssl`) and configure Git to trust your certificate: + - Copy the self-signed certificate or the internal root CA certificate to a + local directory (for example, `~/.ssl`) and configure Git to trust your + certificate: ```shell git config --global http.sslCAInfo ~/.ssl/gitlab.domain.tld.crt ``` - - Disable SSL verification in your Git client. Note that this intended as a temporary measure as it could be considered a **security risk**. + - Disable SSL verification in your Git client. Note that this intended as a + temporary measure, as it could be considered a security risk. ```shell git config --global http.sslVerify false @@ -204,10 +231,10 @@ A misconfiguration may result in: message: SSL_connect returned=1 errno=0 state=error: wrong version number (OpenSSL::SSL::SSLError) ``` -Some of these errors come from the Excon Ruby gem, and could be generated in circumstances -where GitLab is configured to initiate an HTTPS session to a remote server -that is serving just HTTP. +Some of these errors come from the Excon Ruby gem, and could be generated in +circumstances where GitLab is configured to initiate an HTTPS session to a +remote server that is serving only HTTP. -One scenario is that you're using [object storage](../object_storage.md) -which is not served under HTTPS. GitLab is misconfigured and attempts a TLS handshake, +One scenario is that you're using [object storage](../object_storage.md), which +isn't served under HTTPS. GitLab is misconfigured and attempts a TLS handshake, but the object storage will respond with plain HTTP. diff --git a/doc/administration/troubleshooting/test_environments.md b/doc/administration/troubleshooting/test_environments.md index 9855a27ca30..53e51bbfe80 100644 --- a/doc/administration/troubleshooting/test_environments.md +++ b/doc/administration/troubleshooting/test_environments.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -13,7 +13,7 @@ for users with experience with these tools. If you are currently having an issue GitLab, you may want to check your [support options](https://about.gitlab.com/support/) first, before attempting to use this information. -NOTE: **Note:** +NOTE: This page was initially written for Support Engineers, so some of the links are only available internally at GitLab. @@ -103,9 +103,14 @@ docker run -d --name elasticsearch \ docker.elastic.co/elasticsearch/elasticsearch:5.5.1 ``` -Then confirm it works in the browser at `curl http://<IP_ADDRESS>:9200/_cat/health`. +Then confirm it works in the browser at `curl "http://<IP_ADDRESS>:9200/_cat/health"`. Elasticsearch's default username is `elastic` and password is `changeme`. +### Kroki + +See [our Kroki docs](../integration/kroki.md#docker) +on running Kroki in Docker. + ### PlantUML See [our PlantUML docs](../integration/plantuml.md#docker) diff --git a/doc/administration/troubleshooting/tracing_correlation_id.md b/doc/administration/troubleshooting/tracing_correlation_id.md index 8840c2706d6..2981b9e1368 100644 --- a/doc/administration/troubleshooting/tracing_correlation_id.md +++ b/doc/administration/troubleshooting/tracing_correlation_id.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -56,10 +56,10 @@ interested in. ### Getting the correlation ID from curl -If you're using `curl` then you can use the verbose option to show request and response headers, as well as other debug info. +If you're using `curl` then you can use the verbose option to show request and response headers, as well as other debug information. ```shell -â ~ curl --verbose https://gitlab.example.com/api/v4/projects +â ~ curl --verbose "https://gitlab.example.com/api/v4/projects" # look for a line that looks like this < x-request-id: 4rAMkV3gof4 ``` diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index cd15301edf6..94e7bbe2cff 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Uploads administration **(CORE ONLY)** @@ -44,7 +44,7 @@ stored locally, use the steps in this section based on your installation method: **In Omnibus GitLab installations:** -NOTE: **Note:** +NOTE: For historical reasons, uploads are stored into a base directory, which by default is `uploads/-/system`. It's strongly discouraged to change this configuration option for an existing GitLab installation. diff --git a/doc/administration/user_settings.md b/doc/administration/user_settings.md index 94ce1debfea..9892d2a0764 100644 --- a/doc/administration/user_settings.md +++ b/doc/administration/user_settings.md @@ -1,7 +1,7 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Modifying global user settings diff --git a/doc/administration/wikis/index.md b/doc/administration/wikis/index.md index 077b4f064dc..026f9b6f471 100644 --- a/doc/administration/wikis/index.md +++ b/doc/administration/wikis/index.md @@ -2,7 +2,7 @@ type: reference, howto stage: Create group: Knowledge -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Wiki settings **(CORE ONLY)** @@ -30,7 +30,7 @@ This setting is not available through the [Admin Area settings](../../user/admin In order to configure this setting, use either the Rails console or the [Application settings API](../../api/settings.md). -NOTE: **Note:** +NOTE: The value of the limit **must** be in bytes. The minimum value is 1024 bytes. #### Through the Rails console @@ -65,11 +65,11 @@ The process to set the wiki page size limit through the Application Settings API exactly the same as you would do to [update any other setting](../../api/settings.md#change-application-settings). ```shell -curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/application/settings?wiki_page_max_content_bytes=52428800 +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/application/settings?wiki_page_max_content_bytes=52428800" ``` You can also use the API to [retrieve the current value](../../api/settings.md#get-current-application-settings). ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/application/settings +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/application/settings" ``` |
