diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2022-06-20 14:10:13 +0300 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2022-06-20 14:10:13 +0300 |
| commit | 0ea3fcec397b69815975647f5e2aa5fe944a8486 (patch) | |
| tree | 7979381b89d26011bcf9bdc989a40fcc2f1ed4ff /doc/administration | |
| parent | 72123183a20411a36d607d70b12d57c484394c8e (diff) | |
Add latest changes from gitlab-org/gitlab@15-1-stable-eev15.1.0-rc42
Diffstat (limited to 'doc/administration')
159 files changed, 1439 insertions, 1155 deletions
diff --git a/doc/administration/application_settings_cache.md b/doc/administration/application_settings_cache.md index b3aac932fa8..6c58e6886c4 100644 --- a/doc/administration/application_settings_cache.md +++ b/doc/administration/application_settings_cache.md @@ -1,10 +1,10 @@ --- -stage: Enablement +stage: Data Stores group: Memory info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Change the expiration interval for application cache **(FREE SELF)** +# Application cache interval **(FREE SELF)** By default, GitLab caches application settings for 60 seconds. Occasionally, you may need to increase that interval to have more delay between application @@ -14,6 +14,8 @@ We recommend you set this value to greater than `0` seconds. Setting it to `0` causes the `application_settings` table to load for every request. This causes extra load for Redis and PostgreSQL. +## Change the expiration interval for application cache + To change the expiry value: **For Omnibus installations** @@ -32,8 +34,6 @@ To change the expiry value: gitlab-ctl restart ``` ---- - **For installations from source** 1. Edit `config/gitlab.yml`: diff --git a/doc/administration/audit_event_streaming.md b/doc/administration/audit_event_streaming.md index 4678e23dfc9..ad235ead992 100644 --- a/doc/administration/audit_event_streaming.md +++ b/doc/administration/audit_event_streaming.md @@ -10,11 +10,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Enabled on GitLab.com and by default on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/338939) in GitLab 14.7. > - [Feature flag `ff_external_audit_events_namespace`](https://gitlab.com/gitlab-org/gitlab/-/issues/349588) removed in GitLab 14.8. -Event streaming allows owners of top-level groups to set an HTTP endpoint to receive **all** audit events about the group, and its -subgroups and projects as structured JSON. +Users can set an HTTP endpoint for a top-level group to receive all audit events about the group, its subgroups, and +projects as structured JSON. -Top-level group owners can manage their audit logs in third-party systems such as Splunk, using the Splunk -[HTTP Event Collector](https://docs.splunk.com/Documentation/Splunk/8.2.2/Data/UsetheHTTPEventCollector). Any service that can receive +Top-level group owners can manage their audit logs in third-party systems. Any service that can receive structured JSON data can be used as the endpoint. NOTE: @@ -132,7 +131,7 @@ Delete an event streaming destination by specifying an ID. Get the required ID b streaming destinations. ```graphql -mutation { +mutation { externalAuditEventDestinationDestroy(input: { id: destination }) { errors } @@ -144,6 +143,43 @@ Destination is deleted if: - The returned `errors` object is empty. - The API responds with `200 OK`. +## Custom HTTP header values + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/361216) in GitLab 15.1 [with a flag](feature_flags.md) named `streaming_audit_event_headers`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available per group, ask an administrator to [enable the feature flag](../administration/feature_flags.md) named `streaming_audit_event_headers`. +On GitLab.com, this feature is not available. +The feature is not ready for production use. + +Each streaming destination can have up to 20 custom HTTP headers included with each streamed event. + +### Add with the API + +Group owners can add a HTTP header using the GraphQL `auditEventsStreamingHeadersCreate` mutation. + +```graphql +mutation { + auditEventsStreamingHeadersCreate(input: { destinationId: "gid://gitlab/AuditEvents::ExternalAuditEventDestination/24601", key: "foo", value: "bar" }) { + errors + } +} +``` + +### Delete with the API + +Group owners can remove a HTTP header using the GraphQL `auditEventsStreamingHeadersDestroy` mutation. + +```graphql +mutation { + auditEventsStreamingHeadersDestroy(input: { headerId: "gid://gitlab/AuditEvents::ExternalAuditEventDestination/24601" }) { + errors + } +} +``` + +The header is created if the returned `errors` object is empty. + ## Verify event authenticity > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345424) in GitLab 14.8. @@ -158,9 +194,11 @@ the destination's value when [listing streaming destinations](#list-streaming-de > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332747) in GitLab 14.9 [with a flag](../administration/feature_flags.md) named `audit_event_streaming_git_operations`. Disabled by default. > - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/357211) in GitLab 15.0. +> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/357211) in GitLab 15.1 by default. FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](feature_flags.md) named `audit_event_streaming_git_operations`. On GitLab.com, this feature is available. +On self-managed GitLab, by default this feature is available. To hide the +feature, ask an administrator to [disable the feature flag](feature_flags.md) named `audit_event_streaming_git_operations`. Streaming audit events can be sent when signed-in users push or pull a project's remote Git repositories: diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index 6fac21ef889..a329adbed22 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -115,8 +115,14 @@ From there, you can see the following actions: - Instance administrator started or stopped impersonation of a group member. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/300961) in GitLab 14.8. - Group deploy token was successfully created, revoked, or deleted. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/353452) in GitLab 14.9. - Failed attempt to create a group deploy token. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/353452) in GitLab 14.9. -- [IP restrictions](../user/group/index.md#restrict-group-access-by-ip-address) changed. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/358986) in GitLab 15.0 +- [IP restrictions](../user/group/index.md#group-access-restriction-by-ip-address) changed. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/358986) in GitLab 15.0. - Changes to push rules. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/227629) in GitLab 15.0. +- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/356152) in GitLab 15.1, changes to the following merge request approvals settings: + - Prevent approval by author. + - Prevent approvals by users who add commits. + - Prevent editing approval rules in projects and merge requests. + - Require user password to approve. + - Remove all approvals when commits are added to the source branch. Group events can also be accessed via the [Group Audit Events API](../api/audit_events.md#group-audit-events) @@ -228,6 +234,7 @@ The following user actions are recorded: - Administrator added or removed ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/323905) in GitLab 14.1) - Removed SSH key ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/220127) in GitLab 14.1) - Added or removed GPG key ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/220127) in GitLab 14.1) +- A user's two-factor authentication was disabled ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/238177) in GitLab 15.1) Instance events can also be accessed via the [Instance Audit Events API](../api/audit_events.md#instance-audit-events). diff --git a/doc/administration/auditor_users.md b/doc/administration/auditor_users.md index 1d0aff51a04..d82683e1778 100644 --- a/doc/administration/auditor_users.md +++ b/doc/administration/auditor_users.md @@ -14,7 +14,7 @@ Users with auditor access have read-only access to all groups, projects, and oth For more information, see [Auditor user permissions and restrictions](#auditor-user-permissions-and-restrictions) section. -Situations where auditor access for users could be helpful include: +Situations where auditor access for users could be helpful include: - Your compliance department wants to run tests against the entire GitLab base to ensure users are complying with password, credit card, and other sensitive diff --git a/doc/administration/auth/authentiq.md b/doc/administration/auth/authentiq.md index a4a085a494c..61ce2b5ab25 100644 --- a/doc/administration/auth/authentiq.md +++ b/doc/administration/auth/authentiq.md @@ -69,7 +69,7 @@ Authentiq generates a Client ID and the accompanying Client Secret for you to us 1. [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) or [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect if you installed GitLab via Omnibus or from source respectively. -On the sign in page there should now be an Authentiq icon below the regular sign in form. Click the +On the sign in page there should now be an Authentiq icon below the regular sign in form. Select the icon to begin the authentication process. If the user: - Has the Authentiq ID app installed in their iOS or Android device, they can: diff --git a/doc/administration/auth/cognito.md b/doc/administration/auth/cognito.md index 66f8407894c..db8cdd3e7bb 100644 --- a/doc/administration/auth/cognito.md +++ b/doc/administration/auth/cognito.md @@ -18,13 +18,13 @@ The following steps enable AWS Cognito as an authentication provider: 1. Sign in to the [AWS console](https://console.aws.amazon.com/console/home). 1. Select **Cognito** from the **Services** menu. -1. Select **Manage User Pools**, and click the **Create a user pool** button in the top right corner. -1. Enter the pool name and then click the **Step through settings** button. +1. Select **Manage User Pools**, and select the **Create a user pool** button in the top right corner. +1. Enter the pool name and then select the **Step through settings** button. 1. Under **How do you want your end users to sign in?**, select **Email address or phone number** and **Allow email addresses**. 1. Under **Which standard attributes do you want to require?**, select **email**. 1. Go to the next steps of configuration and set the rest of the settings to suit your needs - in the basic setup they are not related to GitLab configuration. -1. In the **App clients** settings, click **Add an app client**, add **App client name** and select the **Enable username password based authentication** checkbox. -1. Click **Create app client**. +1. In the **App clients** settings, select **Add an app client**, add **App client name** and select the **Enable username password based authentication** checkbox. +1. Select **Create app client**. 1. In the next step, you can set up AWS Lambda functions for sending emails. You can then finish creating the pool. 1. After creating the user pool, go to **App client settings** and provide the required information: @@ -85,7 +85,7 @@ Include the code block in the `/etc/gitlab/gitlab.rb` file: 1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. Your sign-in page should now display a Cognito button below the regular sign-in form. -To begin the authentication process, click the icon, and AWS Cognito asks the user to sign in and authorize the GitLab application. +To begin the authentication process, select the icon, and AWS Cognito asks the user to sign in and authorize the GitLab application. If successful, the user is redirected and signed in to your GitLab instance. For more information, see [Configure initial settings](../../integration/omniauth.md#configure-initial-settings). diff --git a/doc/administration/auth/index.md b/doc/administration/auth/index.md index fca4f163075..ad8d78fdfd6 100644 --- a/doc/administration/auth/index.md +++ b/doc/administration/auth/index.md @@ -8,31 +8,14 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GitLab authentication and authorization **(FREE SELF)** -GitLab integrates with the following external authentication and authorization -providers: - -- [Atlassian](atlassian.md) -- [Auth0](../../integration/auth0.md) -- [Authentiq](authentiq.md) -- [AWS Cognito](cognito.md) -- [Azure](../../integration/azure.md) -- [Bitbucket Cloud](../../integration/bitbucket.md) -- [CAS](../../integration/cas.md) -- [Crowd](crowd.md) -- [Facebook](../../integration/facebook.md) -- [GitHub](../../integration/github.md) -- [GitLab.com](../../integration/gitlab.md) -- [Google OAuth](../../integration/google.md) -- [JWT](jwt.md) -- [Kerberos](../../integration/kerberos.md) +GitLab integrates with a number of [OmniAuth providers](../../integration/omniauth.md#supported-providers), +and the following external authentication and authorization providers: + - [LDAP](ldap/index.md): Includes Active Directory, Apple Open Directory, Open LDAP, and 389 Server. - [Google Secure LDAP](ldap/google_secure_ldap.md) -- [Salesforce](../../integration/salesforce.md) -- [SAML](../../integration/saml.md) - [SAML for GitLab.com groups](../../user/group/saml_sso/index.md) **(PREMIUM SAAS)** - [Smartcard](smartcard.md) **(PREMIUM SELF)** -- [Twitter](../../integration/twitter.md) NOTE: UltraAuth has removed their software which supports OmniAuth integration. We have therefore removed all references to UltraAuth integration. @@ -47,7 +30,7 @@ For more information, see the links shown on this page for each external provide | **User Provisioning** | SCIM<br>Just-In-Time (JIT) Provisioning | LDAP Sync | | **User Detail Updating** (not group management) | Not Available | LDAP Sync | | **Authentication** | SAML at top-level group (1 provider) | LDAP (multiple providers)<br>Generic OAuth2<br>SAML (only 1 permitted per unique provider)<br>Kerberos<br>JWT<br>Smartcard<br>OmniAuth Providers (only 1 permitted per unique provider) | -| **Provider-to-GitLab Role Sync** | SAML Group Sync | LDAP Group Sync | +| **Provider-to-GitLab Role Sync** | SAML Group Sync | LDAP Group Sync<br>SAML Group Sync ([GitLab 15.1](https://gitlab.com/gitlab-org/gitlab/-/issues/285150) and later) | | **User Removal** | SCIM (remove user from top-level group) | LDAP (Blocking User from Instance) | ## Change apps or configuration diff --git a/doc/administration/auth/jwt.md b/doc/administration/auth/jwt.md index 98f928fd7ee..99cba3f220d 100644 --- a/doc/administration/auth/jwt.md +++ b/doc/administration/auth/jwt.md @@ -74,7 +74,7 @@ JWT provides you with a secret key for you to use. installed GitLab via Omnibus or from source respectively. On the sign in page there should now be a JWT icon below the regular sign in form. -Click the icon to begin the authentication process. JWT asks the user to +Select the icon to begin the authentication process. JWT asks the user to sign in and authorize the GitLab application. If everything goes well, the user is redirected to GitLab and signed in. diff --git a/doc/administration/auth/ldap/google_secure_ldap.md b/doc/administration/auth/ldap/google_secure_ldap.md index 10745c5e2bf..7f399f7e730 100644 --- a/doc/administration/auth/ldap/google_secure_ldap.md +++ b/doc/administration/auth/ldap/google_secure_ldap.md @@ -26,7 +26,7 @@ The steps below cover: 1. Provide an `LDAP client name` and an optional `Description`. Any descriptive values are acceptable. For example, the name could be 'GitLab' and the - description could be 'GitLab LDAP Client'. Click the **Continue** button. + description could be 'GitLab LDAP Client'. Select the **Continue** button.  @@ -42,15 +42,15 @@ The steps below cover: 1. Download the generated certificate. This is required for GitLab to communicate with the Google Secure LDAP service. Save the downloaded certificates - for later use. After downloading, click the **Continue to Client Details** button. + for later use. After downloading, select the **Continue to Client Details** button. 1. Expand the **Service Status** section and turn the LDAP client 'ON for everyone'. - After selecting 'Save', click on the 'Service Status' bar again to collapse + After selecting 'Save', select the 'Service Status' bar again to collapse and return to the rest of the settings. 1. Expand the **Authentication** section and choose 'Generate New Credentials'. - Copy/note these credentials for later use. After selecting 'Close', click - on the 'Authentication' bar again to collapse and return to the rest of the settings. + Copy/note these credentials for later use. After selecting 'Close', select + the 'Authentication' bar again to collapse and return to the rest of the settings. Now the Google Secure LDAP Client configuration is finished. The screenshot below shows an example of the final settings. Continue on to configure GitLab. diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index 9232e79b800..55b57193bf3 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -169,7 +169,7 @@ These configuration settings are available: | `host` | IP address or domain name of your LDAP server. Ignored when `hosts` is defined. | **{check-circle}** Yes | `'ldap.mydomain.com'` | | `port` | The port to connect with on your LDAP server. Always an integer, not a string. Ignored when `hosts` is defined. | **{check-circle}** Yes | `389` or `636` (for SSL) | | `hosts` (GitLab 14.7 and later) | An array of host and port pairs to open connections. | **{dotted-circle}** No | `[['ldap1.mydomain.com', 636], ['ldap2.mydomain.com', 636]]` | -| `uid` | LDAP attribute for username. Should be the attribute, not the value that maps to the `uid`. | **{check-circle}** Yes | `'sAMAccountName'` or `'uid'` or `'userPrincipalName'` | +| `uid` | The LDAP attribute that maps to the username that users use to sign in. Should be the attribute, not the value that maps to the `uid`. Does not affect the GitLab username (see [attributes section](#attribute-configuration-settings)). | **{check-circle}** Yes | `'sAMAccountName'` or `'uid'` or `'userPrincipalName'` | | `bind_dn` | The full DN of the user you bind with. | **{dotted-circle}** No | `'america\momo'` or `'CN=Gitlab,OU=Users,DC=domain,DC=com'` | | `password` | The password of the bind user. | **{dotted-circle}** No | `'your_great_password'` | | `encryption` | Encryption method. The `method` key is deprecated in favor of `encryption`. | **{check-circle}** Yes | `'start_tls'` or `'simple_tls'` or `'plain'` | diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md index 5c5d5aaffe8..c7572ec0a18 100644 --- a/doc/administration/auth/ldap/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap/ldap-troubleshooting.md @@ -88,7 +88,7 @@ options = { # :filter is optional # This filter includes OID 1.2.840.113556.1.4.1941 - # It will search for all direct and nested members of the group gitlab_grp in the LDAP directory + # It will search for all direct and nested members of the group gitlab_grp in the LDAP directory filter: Net::LDAP::Filter.construct("(memberOf:1.2.840.113556.1.4.1941:=CN=gitlab_grp,DC=example,DC=com)"), # :attributes is optional @@ -707,7 +707,7 @@ For more information, see the [official `ldapsearch` documentation](https://linu ### Using **AdFind** (Windows) -You can use the [`AdFind`](https://social.technet.microsoft.com/wiki/contents/articles/7535.adfind-command-examples.aspx) utility (on Windows based systems) to test that your LDAP server is accessible and authentication is working correctly. AdFind is a freeware utility built by [Joe Richards](http://www.joeware.net/freetools/tools/adfind/index.htm). +You can use the [`AdFind`](https://social.technet.microsoft.com/wiki/contents/articles/7535.adfind-command-examples.aspx) utility (on Windows based systems) to test that your LDAP server is accessible and authentication is working correctly. AdFind is a freeware utility built by [Joe Richards](https://www.joeware.net/freetools/tools/adfind/index.htm). **Return all objects** diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md index 561cbd1b3ae..deb47160d98 100644 --- a/doc/administration/auth/oidc.md +++ b/doc/administration/auth/oidc.md @@ -120,7 +120,7 @@ The OpenID Connect provides you with a client's details and secret for you to us for the changes to take effect if you installed GitLab via Omnibus or from source respectively. On the sign in page, there should now be an OpenID Connect icon below the regular sign in form. -Click the icon to begin the authentication process. The OpenID Connect provider asks the user to +Select the icon to begin the authentication process. The OpenID Connect provider asks the user to sign in and authorize the GitLab application (if confirmation required by the client). If everything goes well, the user is redirected to GitLab and signed in. diff --git a/doc/administration/auth/smartcard.md b/doc/administration/auth/smartcard.md index 3f0d95967bf..0590410bf31 100644 --- a/doc/administration/auth/smartcard.md +++ b/doc/administration/auth/smartcard.md @@ -120,8 +120,14 @@ more information, see [the relevant issue](https://gitlab.com/gitlab-org/gitlab/ 1. Edit `/etc/gitlab/gitlab.rb`: ```ruby + # Allow smartcard authentication gitlab_rails['smartcard_enabled'] = true + + # Path to a file containing a CA certificate gitlab_rails['smartcard_ca_file'] = "/etc/ssl/certs/CA.pem" + + # Host and port where the client side certificate is requested by the + # webserver (NGINX/Apache) gitlab_rails['smartcard_client_certificate_required_host'] = "smartcard.example.com" gitlab_rails['smartcard_client_certificate_required_port'] = 3444 ``` diff --git a/doc/administration/compliance.md b/doc/administration/compliance.md index 99c291379f1..869de929eb8 100644 --- a/doc/administration/compliance.md +++ b/doc/administration/compliance.md @@ -6,12 +6,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Compliance features **(FREE)** -These GitLab features can help ensure that your GitLab instance meets common -compliance standards. For more information about compliance management, see the -compliance management [solutions page](https://about.gitlab.com/solutions/compliance/). +GitLab compliance features ensure your GitLab instance meets common compliance standards, and are available at various pricing tiers. For more information about compliance management, see the compliance +management [solutions page](https://about.gitlab.com/solutions/compliance/). -The [security features](../security/index.md) in GitLab may also help you meet -relevant compliance standards. +The [security features](../security/index.md) in GitLab may also help you meet relevant compliance standards. ## Policy management diff --git a/doc/administration/configure.md b/doc/administration/configure.md index cba4a23d95a..4335c6d972f 100644 --- a/doc/administration/configure.md +++ b/doc/administration/configure.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference @@ -46,5 +46,5 @@ The following table outlines failure modes and mitigation paths for the product | Single Gitaly Node + Geo Secondary | Downtime - Must restore from backup, can perform a manual failover to secondary | Downtime - Must restore from Backup, errors could have propagated to secondary | Manual intervention - failover to Geo secondary | | | Sharded Gitaly Install | Partial Downtime - Only repositories on impacted node affected, must restore from backup | Partial Downtime - Only repositories on impacted node affected, must restore from backup | Downtime - Must wait for outage to end | | | Sharded Gitaly Install + Geo Secondary | Partial Downtime - Only repositories on impacted node affected, must restore from backup, could perform manual failover to secondary for impacted repositories | Partial Downtime - Only repositories on impacted node affected, must restore from backup, errors could have propagated to secondary | Manual intervention - failover to Geo secondary | | -| Gitaly Cluster Install* | No Downtime - will swap repository primary to another node after 10 seconds | N/A - All writes are voted on by multiple Gitaly Cluster nodes | Downtime - Must wait for outage to end | Snapshot backups for Gitaly Cluster nodes not supported at this time | -| Gitaly Cluster Install* + Geo Secondary | No Downtime - will swap repository primary to another node after 10 seconds | N/A - All writes are voted on by multiple Gitaly Cluster nodes | Manual intervention - failover to Geo secondary | Snapshot backups for Gitaly Cluster nodes not supported at this time | +| Gitaly Cluster Install* | No Downtime - will swap repository primary to another node after 10 seconds | Not applicable; All writes are voted on by multiple Gitaly Cluster nodes | Downtime - Must wait for outage to end | Snapshot backups for Gitaly Cluster nodes not supported at this time | +| Gitaly Cluster Install* + Geo Secondary | No Downtime - will swap repository primary to another node after 10 seconds | Not applicable; All writes are voted on by multiple Gitaly Cluster nodes | Manual intervention - failover to Geo secondary | Snapshot backups for Gitaly Cluster nodes not supported at this time | diff --git a/doc/administration/consul.md b/doc/administration/consul.md index 72b3487a549..e1f2bd90e05 100644 --- a/doc/administration/consul.md +++ b/doc/administration/consul.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference @@ -161,7 +161,7 @@ sudo gitlab-ctl restart consul ### Consul nodes unable to communicate By default, Consul attempts to -[bind](https://www.consul.io/docs/agent/options#_bind) to `0.0.0.0`, but +[bind](https://www.consul.io/docs/agent/config/config-files#bind_addr) to `0.0.0.0`, but it advertises the first private IP address on the node for other Consul nodes to communicate with it. If the other nodes cannot communicate with a node on this address, then the cluster has a failed status. diff --git a/doc/administration/docs_self_host.md b/doc/administration/docs_self_host.md index 8464c35c3bb..338601a054f 100644 --- a/doc/administration/docs_self_host.md +++ b/doc/administration/docs_self_host.md @@ -1,14 +1,19 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# How to host the GitLab product documentation **(FREE SELF)** +# Host the GitLab product documentation **(FREE SELF)** If you are not able to access the GitLab product documentation at `docs.gitlab.com`, you can host the documentation yourself instead. +Prerequisites: + +- The version of the product documentation site must be the same as the version of + your GitLab installation. + ## Documentation self-hosting options To host the GitLab product documentation, you can use: @@ -30,28 +35,48 @@ The following examples use GitLab 14.5. ### Self-host the product documentation with Docker -You can run the GitLab product documentation website in a Docker container: +The documentation website is served under the port `4000` inside the container. +In the following example, we expose this on the host under the same port. + +Make sure you either: + +- Allow port `4000` in your firewall. +- Use a different port. In following examples, replace the leftmost `4000` with the port different port. + +To run the GitLab product documentation website in a Docker container: -1. Expose port `4000`. The Docker image uses this port for the web server. 1. On the server where you host GitLab, or on any other server that your GitLab instance - can communicate with, pull the docs site: + can communicate with: - ```shell - docker run -it --rm -p 4000:4000 registry.gitlab.com/gitlab-org/gitlab-docs:14.5 - ``` + - If you use plain Docker, run: - If you host your GitLab instance using [Docker compose](../install/docker.md#install-gitlab-using-docker-compose), - add the following to `docker-compose.yaml`: + ```shell + docker run --detach --name gitlab_docs -it --rm -p 4000:4000 registry.gitlab.com/gitlab-org/gitlab-docs:14.5 + ``` - ```yaml - version: '3.6' - services: - docs: - image: registry.gitlab.com/gitlab-org/gitlab-docs:14.5 - hostname: 'https://gitlab.example.com' - ports: - - '4000:4000' - ``` + - If you host your GitLab instance using + [Docker compose](../install/docker.md#install-gitlab-using-docker-compose), + add the following to your existing `docker-compose.yaml`: + + ```yaml + version: '3.6' + services: + gitlab_docs: + image: registry.gitlab.com/gitlab-org/gitlab-docs:14.5 + hostname: 'https://docs.gitlab.example.com:4000' + ports: + - '4000:4000' + ``` + + Then, pull the changes: + + ```shell + docker-compose up -d + ``` + +1. Visit `http://0.0.0.0:4000` to view the documentation website and verify + it works. +1. [Redirect the help links to the new Docs site](#redirect-the-help-links-to-the-new-docs-site). ### Self-host the product documentation with GitLab Pages @@ -89,29 +114,59 @@ To host the product documentation site with GitLab Pages: | [Project website](../user/project/pages/getting_started_part_one.md#project-website-examples) | Not supported | Supported | | [User or group website](../user/project/pages/getting_started_part_one.md#user-and-group-website-examples) | Supported | Supported | +1. [Redirect the help links to the new Docs site](#redirect-the-help-links-to-the-new-docs-site). + ### Self-host the product documentation on your own web server -Because the product documentation site is static, from the container, you can take the contents -of `/usr/share/nginx/html` and use your own web server to host +Because the product documentation site is static, you can take the contents of +`/usr/share/nginx/html` from inside the container, and use your own web server to host the docs wherever you want. -Run the following commands, replacing `<destination>` with the directory where the -documentation files will be copied to: +The `html` directory should be served as is and it has the following structure: -```shell -docker create -it --name gitlab-docs registry.gitlab.com/gitlab-org/gitlab-docs:14.5 -docker cp gitlab-docs:/usr/share/nginx/html <destination> -docker rm -f gitlab-docs +```plaintext +├── 14.5/ +├── index.html ``` -## Redirect the `/help` links to the new docs page +In this example: + +- `14.5/` is the directory where the documentation is hosted. +- `index.html` is a simple HTML file that redirects to the directory containing the documentation. In this + case, `14.5/`. + +To extract the HTML files of the Docs site: + +1. Create the container that holds the HTML files of the documentation website: + + ```shell + docker create -it --name gitlab_docs registry.gitlab.com/gitlab-org/gitlab-docs:14.5 + ``` + +1. Copy the website under `/srv/gitlab/`: + + ```shell + docker cp gitlab-docs:/usr/share/nginx/html /srv/gitlab/ + ``` + + You will end up with a `/srv/gitlab/html/` directory that holds the documentation website. + +1. Remove the container: + + ```shell + docker rm -f gitlab_docs + ``` + +1. Point your web server to serve the contents of `/srv/gitlab/html/`. +1. [Redirect the help links to the new Docs site](#redirect-the-help-links-to-the-new-docs-site). + +## Redirect the `/help` links to the new Docs site After your local product documentation site is running, [redirect the help links](../user/admin_area/settings/help_page.md#redirect-help-pages) -in the GitLab application to your local site. - -Be sure to use the fully qualified domain name as the docs URL. For example, if you -used the [Docker method](#self-host-the-product-documentation-with-docker), enter `http://0.0.0.0:4000`. +in the GitLab application to your local site, by using the fully qualified domain +name as the docs URL. For example, if you used the +[Docker method](#self-host-the-product-documentation-with-docker), enter `http://0.0.0.0:4000`. You don't need to append the version. GitLab detects it and appends it to documentation URL requests as needed. For example, if your GitLab version is @@ -124,6 +179,84 @@ documentation URL requests as needed. For example, if your GitLab version is To test the setting, select a **Learn more** link within the GitLab application. +## Upgrade the product documentation to a later version + +Upgrading the Docs site to a later version requires downloading the newer Docker image tag. + +### Upgrade using Docker + +To upgrade to a later version [using Docker](#self-host-the-product-documentation-with-docker): + +- If you use plain Docker: + + 1. Stop the running container: + + ```shell + sudo docker stop gitlab_docs + ``` + + 1. Remove the existing container: + + ```shell + sudo docker rm gitlab_docs + ``` + + 1. Pull the new image. For example, 14.6: + + ```shell + docker run --detach --name gitlab_docs -it --rm -p 4000:4000 registry.gitlab.com/gitlab-org/gitlab-docs:14.6 + ``` + +- If you use Docker compose: + + 1. Change the version in `docker-compose.yaml`, for example 14.6: + + ```yaml + version: '3.6' + services: + gitlab_docs: + image: registry.gitlab.com/gitlab-org/gitlab-docs:14.6 + hostname: 'https://docs.gitlab.example.com:4000' + ports: + - '4000:4000' + ``` + + 1. Pull the changes: + + ```shell + docker-compose up -d + ``` + +### Upgrade using GitLab Pages + +To upgrade to a later version [using GitLab Pages](#self-host-the-product-documentation-with-gitlab-pages): + +1. Edit your existing `.gitlab-ci.yml` file, and replace the `image`'s version number: + + ```yaml + image: registry.gitlab.com/gitlab-org/gitlab-docs:14.5 + ``` + +1. Commit the changes, push, and GitLab Pages pulls the new Docs site version. + +### Upgrade using your own web-server + +To upgrade to a later version [using your own web-server](#self-host-the-product-documentation-on-your-own-web-server): + +1. Copy the HTML files of the Docs site: + + ```shell + docker create -it --name gitlab_docs registry.gitlab.com/gitlab-org/gitlab-docs:14.6 + docker cp gitlab_docs:/usr/share/nginx/html /srv/gitlab/ + docker rm -f gitlab_docs + ``` + +1. Optional. Remove the old site: + + ```shell + rm -r /srv/gitlab/html/14.5/ + ``` + ## Known issues If you self-host the product documentation: diff --git a/doc/administration/encrypted_configuration.md b/doc/administration/encrypted_configuration.md index 5c943e56b98..38f033e260a 100644 --- a/doc/administration/encrypted_configuration.md +++ b/doc/administration/encrypted_configuration.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" type: reference diff --git a/doc/administration/environment_variables.md b/doc/administration/environment_variables.md index 1fe5b223f8b..3adb057daa4 100644 --- a/doc/administration/environment_variables.md +++ b/doc/administration/environment_variables.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference @@ -25,7 +25,7 @@ You can use the following environment variables to override certain values: | `EXTERNAL_VALIDATION_SERVICE_TIMEOUT` | integer | Timeout, in seconds, for an [external CI/CD pipeline validation service](external_pipeline_validation.md). Default is `5`. | | `EXTERNAL_VALIDATION_SERVICE_URL` | string | URL to an [external CI/CD pipeline validation service](external_pipeline_validation.md). | | `EXTERNAL_VALIDATION_SERVICE_TOKEN` | string | The `X-Gitlab-Token` for authentication with an [external CI/CD pipeline validation service](external_pipeline_validation.md). | -| `GITLAB_CDN_HOST` | string | Sets the base URL for a CDN to serve static assets (for example, `//mycdnsubdomain.fictional-cdn.com`). | +| `GITLAB_CDN_HOST` | string | Sets the base URL for a CDN to serve static assets (for example, `https://mycdnsubdomain.fictional-cdn.com`). | | `GITLAB_EMAIL_DISPLAY_NAME` | string | The name used in the **From** field in emails sent by GitLab. | | `GITLAB_EMAIL_FROM` | string | The email address used in the **From** field in emails sent by GitLab. | | `GITLAB_EMAIL_REPLY_TO` | string | The email address used in the **Reply-To** field in emails sent by GitLab. | diff --git a/doc/administration/external_pipeline_validation.md b/doc/administration/external_pipeline_validation.md index a4ed287cc3b..06eb9ffc84e 100644 --- a/doc/administration/external_pipeline_validation.md +++ b/doc/administration/external_pipeline_validation.md @@ -54,7 +54,9 @@ required number of seconds. "required": [ "id", "path", - "created_at" + "created_at", + "shared_runners_enabled", + "group_runners_enabled" ], "properties": { "id": { "type": "integer" }, diff --git a/doc/administration/feature_flags.md b/doc/administration/feature_flags.md index aebd6eb1d4c..8a021c6d588 100644 --- a/doc/administration/feature_flags.md +++ b/doc/administration/feature_flags.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments description: "GitLab administrator: enable and disable GitLab features deployed behind feature flags" diff --git a/doc/administration/file_hooks.md b/doc/administration/file_hooks.md index 4ddca58c774..ed5dbd83a8e 100644 --- a/doc/administration/file_hooks.md +++ b/doc/administration/file_hooks.md @@ -51,7 +51,7 @@ Follow the steps below to set up a custom hook: in any language, and ensure the 'shebang' at the top properly reflects the language type. For example, if the script is in Ruby the shebang will probably be `#!/usr/bin/env ruby`. -1. The data to the file hook is provided as JSON on STDIN. It is exactly the +1. The data to the file hook is provided as JSON on `STDIN`. It is exactly the same as for [system hooks](system_hooks.md). That's it! Assuming the file hook code is properly implemented, the hook fires diff --git a/doc/administration/geo/disaster_recovery/background_verification.md b/doc/administration/geo/disaster_recovery/background_verification.md index d7db48bb6cf..57f8664ba11 100644 --- a/doc/administration/geo/disaster_recovery/background_verification.md +++ b/doc/administration/geo/disaster_recovery/background_verification.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/geo/disaster_recovery/bring_primary_back.md b/doc/administration/geo/disaster_recovery/bring_primary_back.md index 4e68cc39580..833b9a877e9 100644 --- a/doc/administration/geo/disaster_recovery/bring_primary_back.md +++ b/doc/administration/geo/disaster_recovery/bring_primary_back.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto @@ -18,8 +18,8 @@ If you have any doubts about the consistency of the data on this site, we recomm ## Configure the former **primary** site to be a **secondary** site -Since the former **primary** site will be out of sync with the current **primary** site, the first step is to bring the former **primary** site up to date. Note, deletion of data stored on disk like -repositories and uploads will not be replayed when bringing the former **primary** site back +Since the former **primary** site is out of sync with the current **primary** site, the first step is to bring the former **primary** site up to date. Note, deletion of data stored on disk like +repositories and uploads is not replayed when bringing the former **primary** site back into sync, which may result in increased disk usage. Alternatively, you can [set up a new **secondary** GitLab instance](../setup/index.md) to avoid this. @@ -34,8 +34,8 @@ To bring the former **primary** site up to date: NOTE: If you [disabled the **primary** site permanently](index.md#step-2-permanently-disable-the-primary-site), - you need to undo those steps now. For Debian/Ubuntu you just need to run - `sudo systemctl enable gitlab-runsvdir`. For CentOS 6, you need to install + you need to undo those steps now. For distributions with systemd, such as Debian/Ubuntu/CentOS7+, you must run + `sudo systemctl enable gitlab-runsvdir`. For distributions without systemd, such as CentOS 6, you need to install the GitLab instance from scratch and set it up as a **secondary** site by following [Setup instructions](../setup/index.md). In this case, you don't need to follow the next step. diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index baece830318..e5b9a14cfbd 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -335,16 +335,14 @@ a **secondary** with only a single node. Instead, you must do this manually. #### Promoting a **secondary** site with an external PostgreSQL database running GitLab 14.5 and later -The `gitlab-ctl geo promote` command can be used in conjunction with -an external PostgreSQL database, but it can only perform changes on -a **secondary** PostgreSQL database managed by Omnibus. -You must promote the replica database associated with the **secondary** -site first. +The `gitlab-ctl geo promote` command can be used in conjunction with an external PostgreSQL database. +In this case, you must first manually promote the replica database associated +with the **secondary** site: 1. Promote the replica database associated with the **secondary** site. This sets the database to read-write. The instructions vary depending on where your database is hosted: - [Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.Promote) - - [Azure PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/howto-read-replicas-portal#stop-replication) + - [Azure PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/single-server/how-to-read-replicas-portal#stop-replication) - [Google Cloud SQL](https://cloud.google.com/sql/docs/mysql/replication/manage-replicas#promote-replica) - For other external PostgreSQL databases, save the following script in your secondary site, for example `/tmp/geo_promote.sh`, and modify the connection diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index 57bad6177d9..7d8dd7d5d2a 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto @@ -143,7 +143,7 @@ If the **primary** site uses custom or self-signed TLS certificates to secure in ### Ensure Geo replication is up-to-date -The maintenance window won't end until Geo replication and verification is +The maintenance window does not end until Geo replication and verification is completely finished. To keep the window as short as possible, you should ensure these processes are close to 100% as possible during active use. @@ -201,7 +201,7 @@ be disabled on the **primary** site: ## Finish replicating and verifying all data NOTE: -GitLab 13.9 through GitLab 14.3 are affected by a bug in which the Geo secondary site statuses will appear to stop updating and become unhealthy. For more information, see [Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](../replication/troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). +GitLab 13.9 through GitLab 14.3 are affected by a bug in which the Geo secondary site statuses appears to stop updating and become unhealthy. For more information, see [Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](../replication/troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). 1. If you are manually replicating any data not managed by Geo, trigger the final replication process now. diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md index 59134df8e7f..46b2ccddefd 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md index 25f6999ce43..da26023e4f9 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto diff --git a/doc/administration/geo/glossary.md b/doc/administration/geo/glossary.md index c6b3f26dc67..c9d4ccacae3 100644 --- a/doc/administration/geo/glossary.md +++ b/doc/administration/geo/glossary.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index dc25be825ea..c5472e412d3 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -13,7 +13,7 @@ a warm-standby as part of a disaster recovery strategy. WARNING: Geo undergoes significant changes from release to release. Upgrades are -supported and [documented](#updating-geo), but you should ensure that you're +supported and [documented](#upgrading-geo), but you should ensure that you're using the right version of the documentation for your installation. Fetching large repositories can take a long time for teams located far from a single GitLab instance. @@ -53,8 +53,7 @@ Geo provides: ### Gitaly Cluster Geo should not be confused with [Gitaly Cluster](../gitaly/praefect.md). For more information about -the difference between Geo and Gitaly Cluster, see -[How does Gitaly Cluster compare to Geo?](../gitaly/faq.md#how-does-gitaly-cluster-compare-to-geo). +the difference between Geo and Gitaly Cluster, see [Comparison to Geo](../gitaly/index.md#comparison-to-geo). ## How it works @@ -234,9 +233,9 @@ After installing GitLab on the **secondary** sites and performing the initial co For information on configuring Geo, see [Geo configuration](replication/configuration.md). -### Updating Geo +### Upgrading Geo -For information on how to update your Geo sites to the latest GitLab version, see [Updating the Geo sites](replication/updating_the_geo_sites.md). +For information on how to update your Geo sites to the latest GitLab version, see [Upgrading the Geo sites](replication/upgrading_the_geo_sites.md). ### Pausing and resuming replication @@ -252,7 +251,7 @@ WARNING: Pausing and resuming of replication is only supported for Geo installations using an Omnibus GitLab-managed database. External databases are not supported. -In some circumstances, like during [upgrades](replication/updating_the_geo_sites.md) or a [planned failover](disaster_recovery/planned_failover.md), it is desirable to pause replication between the primary and secondary. +In some circumstances, like during [upgrades](replication/upgrading_the_geo_sites.md) or a [planned failover](disaster_recovery/planned_failover.md), it is desirable to pause replication between the primary and secondary. Pausing and resuming replication is done via a command line tool from the node in the secondary site where the `postgresql` service is enabled. diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index adf89c24a4e..043b96478c9 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto @@ -384,7 +384,7 @@ when: ## Upgrading Geo -See the [updating the Geo sites document](updating_the_geo_sites.md). +See the [upgrading the Geo sites document](upgrading_the_geo_sites.md). ## Troubleshooting diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index 83e207e7a8f..8a5aa935d11 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto @@ -26,7 +26,7 @@ verification methods: | Type | Feature / component | Replication method | Verification method | |:---------|:------------------------------------------------|:--------------------------------------|:-----------------------| | Database | Application data in PostgreSQL | Native | Native | -| Database | Redis | _N/A_ (*1*) | _N/A_ | +| Database | Redis | Not applicable (*1*) | Not applicable | | Database | Elasticsearch | Native | Native | | Database | SSH public keys | PostgreSQL Replication | PostgreSQL Replication | | Git | Project repository | Geo with Gitaly | Gitaly Checksum | @@ -203,8 +203,10 @@ successfully, you must replicate their data using some other means. |[External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | **Yes** (14.6) | Via Object Storage provider if supported. Native Geo support (Beta). | Replication is behind the feature flag `geo_merge_request_diff_replication`, enabled by default. Verification was behind the feature flag `geo_merge_request_diff_verification`, removed in 14.7.| |[Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [**Yes** (14.2)](https://gitlab.com/groups/gitlab-org/-/epics/2810) | No | Verification was implemented behind the feature flag `geo_snippet_repository_verification` in 13.11, and the feature flag was removed in 14.2. | |[GitLab Pages](../../pages/index.md) | [**Yes** (14.3)](https://gitlab.com/groups/gitlab-org/-/epics/589) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (14.6) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_pages_deployment_replication`, enabled by default. Verification was behind the feature flag `geo_pages_deployment_verification`, removed in 14.7. | +|[Incident Metric Images](../../../operations/incident_management/incidents.md#metrics) | [Planned](https://gitlab.com/gitlab-org/gitlab/-/issues/352326) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/362561) | No | | +|[Alert Metric Images](../../../operations/incident_management/alerts.md#metrics-tab) | [Planned](https://gitlab.com/gitlab-org/gitlab/-/issues/352326) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/362561) | No | | |[Server-side Git hooks](../../server_hooks.md) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/1867) | No | No | Not planned because of current implementation complexity, low customer interest, and availability of alternatives to hooks. | -|[Elasticsearch integration](../../../integration/elasticsearch.md) | [Not planned](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | No | Not planned because further product discovery is required and Elasticsearch (ES) clusters can be rebuilt. Secondaries use the same ES cluster as the primary. | +|[Elasticsearch integration](../../../integration/advanced_search/elasticsearch.md) | [Not planned](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | No | Not planned because further product discovery is required and Elasticsearch (ES) clusters can be rebuilt. Secondaries use the same ES cluster as the primary. | |[Dependency proxy images](../../../user/packages/dependency_proxy/index.md) | [Not planned](https://gitlab.com/gitlab-org/gitlab/-/issues/259694) | No | No | Blocked by [Geo: Secondary Mimicry](https://gitlab.com/groups/gitlab-org/-/epics/1528). Replication of this cache is not needed for disaster recovery purposes because it can be recreated from external sources. | |[Vulnerability Export](../../../user/application_security/vulnerability_report/#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | No | Not planned because they are ephemeral and sensitive information. They can be regenerated on demand. | diff --git a/doc/administration/geo/replication/disable_geo.md b/doc/administration/geo/replication/disable_geo.md index 0fa469e57cd..5d4ae12c990 100644 --- a/doc/administration/geo/replication/disable_geo.md +++ b/doc/administration/geo/replication/disable_geo.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto diff --git a/doc/administration/geo/replication/docker_registry.md b/doc/administration/geo/replication/docker_registry.md index 4004ef3c17f..855e33d9a51 100644 --- a/doc/administration/geo/replication/docker_registry.md +++ b/doc/administration/geo/replication/docker_registry.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto @@ -114,10 +114,10 @@ For each application and Sidekiq node on the **secondary** site: ```ruby gitlab_rails['geo_registry_replication_enabled'] = true - + # Primary registry's hostname and port, it will be used by # the secondary node to directly communicate to primary registry - gitlab_rails['geo_registry_replication_primary_api_url'] = 'https://primary.example.com:5050/' + gitlab_rails['geo_registry_replication_primary_api_url'] = 'https://primary.example.com:5050/' ``` 1. Reconfigure the node for the change to take effect: diff --git a/doc/administration/geo/replication/faq.md b/doc/administration/geo/replication/faq.md index 12b3b382bf7..bdf1771e8a8 100644 --- a/doc/administration/geo/replication/faq.md +++ b/doc/administration/geo/replication/faq.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -32,10 +32,10 @@ To ensure that problems with pipelines (for example, syncs failing too many time the number of concurrent syncs falls below `repos_max_capacity` and there are no new projects waiting to be synced. Geo also has a checksum feature which runs a SHA256 sum across all the Git references to the SHA values. -If the refs don't match between the **primary** site and the **secondary** site, then the **secondary** site will mark that project as dirty and try to resync it. +If the refs don't match between the **primary** site and the **secondary** site, then the **secondary** site marks that project as dirty and try to resync it. So even if we have an outdated tracking database, the validation should activate and find discrepancies in the repository state and resync. -## Can I use Geo in a disaster recovery situation? +## Can you use Geo in a disaster recovery situation? Yes, but there are limitations to what we replicate (see [What data is replicated to a **secondary** site?](#what-data-is-replicated-to-a-secondary-site)). @@ -46,7 +46,7 @@ Read the documentation for [Disaster Recovery](../disaster_recovery/index.md). We currently replicate project repositories, LFS objects, generated attachments and avatars, and the whole database. This means user accounts, -issues, merge requests, groups, project data, and so on, will be available for +issues, merge requests, groups, project data, and so on, are available for query. For more details, see the [supported Geo data types](datatypes.md). @@ -69,6 +69,6 @@ That's totally fine. We use HTTP(s) to fetch repository changes from the **prima Yes. See [Docker Registry for a **secondary** site](docker_registry.md). -## Can I login to a secondary site? +## Can you login to a secondary site? Yes, but secondary sites receive all authentication data (like user accounts and logins) from the primary instance. This means you are re-directed to the primary for authentication and then routed back. diff --git a/doc/administration/geo/replication/geo_validation_tests.md b/doc/administration/geo/replication/geo_validation_tests.md index e4432c41ff1..6540366bf7f 100644 --- a/doc/administration/geo/replication/geo_validation_tests.md +++ b/doc/administration/geo/replication/geo_validation_tests.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/geo/replication/location_aware_git_url.md b/doc/administration/geo/replication/location_aware_git_url.md index 8689fac987f..18103211580 100644 --- a/doc/administration/geo/replication/location_aware_git_url.md +++ b/doc/administration/geo/replication/location_aware_git_url.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto @@ -57,34 +57,34 @@ routing configurations.  -1. Click the **Create traffic policy** button. +1. Select the **Create traffic policy** button.  -1. Fill in the **Policy Name** field with `Single Git Host` and click **Next**. +1. Fill in the **Policy Name** field with `Single Git Host` and select **Next**.  1. Leave **DNS type** as `A: IP Address in IPv4 format`. -1. Click **Connect to...** and select **Geolocation rule**. +1. Select **Connect to...** and select **Geolocation rule**.  1. For the first **Location**, leave it as `Default`. -1. Click **Connect to...** and select **New endpoint**. +1. Select **Connect to...** and select **New endpoint**. 1. Choose **Type** `value` and fill it in with `<your **primary** IP address>`. 1. For the second **Location**, choose `Europe`. -1. Click **Connect to...** and select **New endpoint**. +1. Select **Connect to...** and select **New endpoint**. 1. Choose **Type** `value` and fill it in with `<your **secondary** IP address>`.  -1. Click **Create traffic policy**. +1. Select **Create traffic policy**.  1. Fill in **Policy record DNS name** with `git`. -1. Click **Create policy records**. +1. Select **Create policy records**.  diff --git a/doc/administration/geo/replication/multiple_servers.md b/doc/administration/geo/replication/multiple_servers.md index 7b800817461..afa4f4eb552 100644 --- a/doc/administration/geo/replication/multiple_servers.md +++ b/doc/administration/geo/replication/multiple_servers.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto diff --git a/doc/administration/geo/replication/object_storage.md b/doc/administration/geo/replication/object_storage.md index 06f25fa15af..9ea226fd93b 100644 --- a/doc/administration/geo/replication/object_storage.md +++ b/doc/administration/geo/replication/object_storage.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto diff --git a/doc/administration/geo/replication/remove_geo_site.md b/doc/administration/geo/replication/remove_geo_site.md index b69dfe2e723..0d6715a93b7 100644 --- a/doc/administration/geo/replication/remove_geo_site.md +++ b/doc/administration/geo/replication/remove_geo_site.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto @@ -44,7 +44,7 @@ Once GitLab has been uninstalled from each node on the **secondary** site, the r ``` NOTE: - Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. + Using `gitlab-rails dbconsole` does not work, because managing replication slots requires superuser permissions. 1. Find the name of the relevant replication slot. This is the slot that is specified with `--slot-name` when running the replicate command: `gitlab-ctl replicate-geo-database`. diff --git a/doc/administration/geo/replication/security_review.md b/doc/administration/geo/replication/security_review.md index df893298f85..bc08b7b71e7 100644 --- a/doc/administration/geo/replication/security_review.md +++ b/doc/administration/geo/replication/security_review.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 5a29c5a3c54..bb7dbef214b 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -398,7 +398,7 @@ where some queries never complete due to being canceled on every replication. These long-running queries are [planned to be removed in the future](https://gitlab.com/gitlab-org/gitlab/-/issues/34269), but as a workaround, we recommend enabling -[hot_standby_feedback](https://www.postgresql.org/docs/10/hot-standby.html#HOT-STANDBY-CONFLICT). +[`hot_standby_feedback`](https://www.postgresql.org/docs/10/hot-standby.html#HOT-STANDBY-CONFLICT). This increases the likelihood of bloat on the **primary** node as it prevents `VACUUM` from removing recently-dead rows. However, it has been used successfully in production on GitLab.com. @@ -767,7 +767,7 @@ The appropriate action sometimes depends on the cause. For example, you can remo In some cases, a file may be determined to be of low value, and so it may be worth deleting the record. -Geo itself is an excellent mitigation for files missing on the primary. If a file disappears on the primary but it was already synced to the secondary, you can grab the secondary's file. In cases like this, the `File is not checksummable` error message will not occur on Geo secondary sites, and only the primary will log this error message. +Geo itself is an excellent mitigation for files missing on the primary. If a file disappears on the primary but it was already synced to the secondary, you can grab the secondary's file. In cases like this, the `File is not checksummable` error message does not occur on Geo secondary sites, and only the primary logs this error message. This problem is more likely to show up in Geo secondary sites which were set up long after the original GitLab site. In this case, Geo is only surfacing an existing problem. @@ -815,6 +815,20 @@ Gitlab::Geo.verification_enabled_replicator_classes.each do |klass| end ``` +### Message: curl 18 transfer closed with outstanding read data remaining & fetch-pack: unexpected disconnect while reading sideband packet + +Unstable networking conditions can cause Gitaly to fail when trying to fetch large repository +data from the primary site. This is more likely to happen if a repository has to be +replicated from scratch between sites. + +Geo retries several times, but if the transmission is consistently interrupted +by network hiccups, an alternative method such as `rsync` can be used to circumvent `git` and +create the initial copy of any repository that fails to be replicated by Geo. + +We recommend transferring each failing repository individually and checking for consistency +after each transfer. Follow the [single target `rsync` instructions](../../operations/moving_repositories.md#single-rsync-to-another-server) +to transfer each affected repository from the primary to the secondary site. + ## Fixing errors during a failover or when promoting a secondary to a primary node The following are possible error messages that might be encountered during failover or @@ -1104,9 +1118,9 @@ If using a load balancer, ensure that the load balancer's URL is set as the `ext ### Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode -In GitLab 13.9 through GitLab 14.3, when [GitLab Maintenance Mode](../../maintenance_mode/index.md) is enabled, the status of Geo secondary sites will stop getting updated. After 10 minutes, the status changes to `Unhealthy`. +In GitLab 13.9 through GitLab 14.3, when [GitLab Maintenance Mode](../../maintenance_mode/index.md) is enabled, the status of Geo secondary sites stops getting updated. After 10 minutes, the status changes to `Unhealthy`. -Geo secondary sites will continue to replicate and verify data, and the secondary sites should still be usable. You can use the [Sync status Rake task](#sync-status-rake-task) to determine the actual status of a secondary site during Maintenance Mode. +Geo secondary sites continue to replicate and verify data, and the secondary sites should still be usable. You can use the [Sync status Rake task](#sync-status-rake-task) to determine the actual status of a secondary site during Maintenance Mode. This bug was [fixed in GitLab 14.4](https://gitlab.com/gitlab-org/gitlab/-/issues/292983). @@ -1129,7 +1143,7 @@ Geo::TrackingBase::SecondaryNotConfigured: Geo secondary database is not configu On a Geo primary site this error can be ignored. -This happens because GitLab is attempting to display registries from the [Geo tracking database](../../../administration/geo/#geo-tracking-database) which doesn't exist on the primary site (only the original projects exist on the primary; no replicated projects are present, therefore no tracking database exists). +This happens because GitLab is attempting to display registries from the [Geo tracking database](../../../administration/geo/#geo-tracking-database) which doesn't exist on the primary site (only the original projects exist on the primary; no replicated projects are present, therefore no tracking database exists). ## Fixing client errors @@ -1148,7 +1162,7 @@ The partial failover to a secondary Geo *site* may be the result of a temporary/ 1. SSH into every Sidekiq, PostgresSQL, Gitaly, and Rails node in the **secondary** site and run one of the following commands: - To promote the secondary node to primary: - + ```shell sudo gitlab-ctl geo promote ``` @@ -1158,8 +1172,8 @@ The partial failover to a secondary Geo *site* may be the result of a temporary/ ```shell sudo gitlab-ctl geo promote --force ``` - -1. Verify you can connect to the newly-promoted **primary** site using the URL used previously for the **secondary** site. + +1. Verify you can connect to the newly-promoted **primary** site using the URL used previously for the **secondary** site. 1. If **successful**, the **secondary** site is now promoted to the **primary** site. If the above steps are **not successful**, proceed through the next steps: diff --git a/doc/administration/geo/replication/tuning.md b/doc/administration/geo/replication/tuning.md index bcff6181296..670459624f3 100644 --- a/doc/administration/geo/replication/tuning.md +++ b/doc/administration/geo/replication/tuning.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto @@ -7,7 +7,7 @@ type: howto # Tuning Geo **(PREMIUM SELF)** -You can limit the number of concurrent operations the nodes can run +You can limit the number of concurrent operations the sites can run in the background. ## Changing the sync/verification concurrency values @@ -15,8 +15,8 @@ in the background. On the **primary** site: 1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **Geo > Nodes**. -1. Select **Edit** of the secondary node you want to tune. +1. On the left sidebar, select **Geo > Sites**. +1. Select **Edit** of the secondary site you want to tune. 1. Under **Tuning settings**, there are several variables that can be tuned to improve the performance of Geo: @@ -25,7 +25,7 @@ On the **primary** site: - Container repositories synchronization concurrency limit - Verification concurrency limit -Increasing the concurrency values will increase the number of jobs that are scheduled. +Increasing the concurrency values increases the number of jobs that are scheduled. However, this may not lead to more downloads in parallel unless the number of available Sidekiq threads is also increased. For example, if repository synchronization concurrency is increased from 25 to 50, you may also want to increase the number diff --git a/doc/administration/geo/replication/updating_the_geo_sites.md b/doc/administration/geo/replication/upgrading_the_geo_sites.md index e82afe5f0d4..30961de0381 100644 --- a/doc/administration/geo/replication/updating_the_geo_sites.md +++ b/doc/administration/geo/replication/upgrading_the_geo_sites.md @@ -1,31 +1,31 @@ --- -stage: Enablement +stage: Systems group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- -# Updating the Geo sites **(PREMIUM SELF)** +# Upgrading the Geo sites **(PREMIUM SELF)** WARNING: Read these sections carefully before updating your Geo sites. Not following -version-specific update steps may result in unexpected downtime. If you have +version-specific upgrade steps may result in unexpected downtime. If you have any specific questions, [contact Support](https://about.gitlab.com/support/#contact-support). -Updating Geo sites involves performing: +Upgrading Geo sites involves performing: -1. [Version-specific update steps](version_specific_updates.md), depending on the - version being updated to or from. -1. [General update steps](#general-update-steps), for all updates. +1. [Version-specific upgrade steps](version_specific_upgrades.md), depending on the + version being upgraded to or from. +1. [General upgrade steps](#general-upgrade-steps), for all upgrades. -## General update steps +## General upgrade steps NOTE: -These general update steps are not intended for multi-site deployments, -and will cause downtime. If you want to avoid downtime, consider using +These general upgrade steps are not intended for multi-site deployments, +and cause downtime. If you want to avoid downtime, consider using [zero downtime upgrades](../../../update/zero_downtime.md#multi-node--ha-deployment-with-geo). -To update the Geo sites when a new GitLab version is released, update **primary** +To upgrade the Geo sites when a new GitLab version is released, upgrade **primary** and all **secondary** sites: 1. **Optional:** [Pause replication on each **secondary** sites.](../index.md#pausing-and-resuming-replication) @@ -34,11 +34,11 @@ and all **secondary** sites: 1. SSH into each node of **secondary** sites. 1. [Upgrade GitLab on each **secondary** site](../../../update/package/index.md#upgrade-using-the-official-repositories). 1. If you paused replication in step 1, [resume replication on each **secondary**](../index.md#pausing-and-resuming-replication) -1. [Test](#check-status-after-updating) **primary** and **secondary** sites, and check version in each. +1. [Test](#check-status-after-upgrading) **primary** and **secondary** sites, and check version in each. -### Check status after updating +### Check status after upgrading -Now that the update process is complete, you may want to check whether +Now that the upgrade process is complete, you may want to check whether everything is working correctly: 1. Run the Geo Rake task on an application node for the primary and secondary sites. Everything should be green: diff --git a/doc/administration/geo/replication/usage.md b/doc/administration/geo/replication/usage.md index d96bfae78a1..fe0e06e7ea4 100644 --- a/doc/administration/geo/replication/usage.md +++ b/doc/administration/geo/replication/usage.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -11,9 +11,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w After you set up the [database replication and configure the Geo nodes](../index.md#setup-instructions), use your closest GitLab site as you would do with the primary one. You can push directly to a **secondary** site (for both HTTP, SSH including -Git LFS), and the request will be proxied to the primary site instead. +Git LFS), and the request is proxied to the primary site instead. -Example of the output you will see when pushing to a **secondary** site: +Example of the output you see when pushing to a **secondary** site: ```shell $ git push @@ -31,7 +31,7 @@ If you're using HTTPS instead of [SSH](../../../user/ssh.md) to push to the seco you can't store credentials in the URL like `user:password@URL`. Instead, you can use a [`.netrc` file](https://www.gnu.org/software/inetutils/manual/html_node/The-_002enetrc-file.html) for Unix-like operating systems or `_netrc` for Windows. In that case, the credentials -will be stored as a plain text. If you're looking for a more secure way to store credentials, +are stored as a plain text. If you're looking for a more secure way to store credentials, you can use [Git Credential Storage](https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage). ## Fetch Go modules from Geo secondary sites diff --git a/doc/administration/geo/replication/version_specific_updates.md b/doc/administration/geo/replication/version_specific_upgrades.md index 6b617a21be8..f0925bdf87e 100644 --- a/doc/administration/geo/replication/version_specific_updates.md +++ b/doc/administration/geo/replication/version_specific_upgrades.md @@ -1,24 +1,24 @@ --- -stage: Enablement +stage: Systems group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Version-specific update instructions **(PREMIUM SELF)** +# Version-specific upgrade instructions **(PREMIUM SELF)** -Review this page for update instructions for your version. These steps -accompany the [general steps](updating_the_geo_sites.md#general-update-steps) -for updating Geo sites. +Review this page for upgrade instructions for your version. These steps +accompany the [general steps](upgrading_the_geo_sites.md#general-upgrade-steps) +for upgrading Geo sites. -## Updating to 14.9 +## Upgrading to 14.9 -**DO NOT** update to GitLab 14.9.0. Instead, use 14.9.1 or later. +**Do not** upgrade to GitLab 14.9.0. Instead, use 14.9.1 or later. We've discovered an issue with Geo's CI verification feature that may [cause job traces to be lost](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6664). This issue was fixed in [the GitLab 14.9.1 patch release](https://about.gitlab.com/releases/2022/03/23/gitlab-14-9-1-released/). -If you have already updated to GitLab 14.9.0, you can disable the feature causing the issue by [disabling the `geo_job_artifact_replication` feature flag](../../feature_flags.md#how-to-enable-and-disable-features-behind-flags). +If you have already upgraded to GitLab 14.9.0, you can disable the feature causing the issue by [disabling the `geo_job_artifact_replication` feature flag](../../feature_flags.md#how-to-enable-and-disable-features-behind-flags). -## Updating to 14.2 through 14.7 +## Upgrading to 14.2 through 14.7 There is [an issue in GitLab 14.2 through 14.7](https://gitlab.com/gitlab-org/gitlab/-/issues/299819#note_822629467) that affects Geo when the GitLab-managed object storage replication is used, causing blob object types to fail synchronization. @@ -33,11 +33,11 @@ results in a loop that consistently fails for all objects stored in object stora For information on how to fix this, see [Troubleshooting - Failed syncs with GitLab-managed object storage replication](troubleshooting.md#failed-syncs-with-gitlab-managed-object-storage-replication). -## Updating to 14.4 +## Upgrading to 14.4 There is [an issue in GitLab 14.4.0 through 14.4.2](../../../update/index.md#1440) that can affect Geo and other features that rely on cronjobs. We recommend upgrading to GitLab 14.4.3 or later. -## Updating to 14.1, 14.2, 14.3 +## Upgrading to 14.1, 14.2, 14.3 ### Multi-arch images @@ -72,13 +72,13 @@ Otherwise, for each **secondary** site, on a Rails application node, open a [Rai end ``` -If you are running a version prior to 14.1 and are using Geo and multi-arch containers in your Container Registry, we recommend [upgrading](updating_the_geo_sites.md) to at least GitLab 14.1. +If you are running a version prior to 14.1 and are using Geo and multi-arch containers in your Container Registry, we recommend [upgrading](upgrading_the_geo_sites.md) to at least GitLab 14.1. ### Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode -GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) causes Geo secondary site statuses to appear to stop updating and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). +GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) causes Geo secondary site statuses to appear to stop upgrading and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). -## Updating to GitLab 14.0/14.1 +## Upgrading to GitLab 14.0/14.1 ### Primary sites can not be removed from the UI @@ -90,13 +90,13 @@ If you are running an affected version and need to remove your Primary site, you ### Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode -GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) causes Geo secondary site statuses to appear to stop updating and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). +GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) causes Geo secondary site statuses to appear to stop upgrading and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). -## Updating to GitLab 13.12 +## Upgrading to GitLab 13.12 -### Secondary sites re-download all LFS files upon update +### Secondary sites re-download all LFS files upon upgrade -We found an issue where [secondary sites re-download all LFS files](https://gitlab.com/gitlab-org/gitlab/-/issues/334550) upon update. This bug: +We found an issue where [secondary sites re-download all LFS files](https://gitlab.com/gitlab-org/gitlab/-/issues/334550) upon upgrade. This bug: - Only applies to Geo secondary sites that have replicated LFS objects. - Is _not_ a data loss risk. @@ -104,9 +104,9 @@ We found an issue where [secondary sites re-download all LFS files](https://gitl - May impact performance for GitLab installations with a large number of LFS files. If you don't have many LFS objects or can stand a bit of churn, then it is safe to let the secondary sites re-download LFS objects. -If you do have many LFS objects, or many Geo secondary sites, or limited bandwidth, or a combination of them all, then we recommend you skip GitLab 13.12.0 through 13.12.6 and update to GitLab 13.12.7 or newer. +If you do have many LFS objects, or many Geo secondary sites, or limited bandwidth, or a combination of them all, then we recommend you skip GitLab 13.12.0 through 13.12.6 and upgrade to GitLab 13.12.7 or newer. -#### If you have already updated to an affected version, and the re-sync is ongoing +#### If you have already upgraded to an affected version, and the re-sync is ongoing You can manually migrate the legacy sync state to the new state column by running the following command in a [Rails console](../../operations/rails_console.md). It should take under a minute: @@ -116,29 +116,29 @@ Geo::LfsObjectRegistry.where(state: 0, success: true).update_all(state: 2) ### Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode -GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) causes Geo secondary site statuses to appear to stop updating and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). +GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) causes Geo secondary site statuses to appear to stop upgrading and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). -## Updating to GitLab 13.11 +## Upgrading to GitLab 13.11 We found an [issue with Git clone/pull through HTTP(s)](https://gitlab.com/gitlab-org/gitlab/-/issues/330787) on Geo secondaries and on any GitLab instance if maintenance mode is enabled. This was caused by a regression in GitLab Workhorse. This is fixed in the [GitLab 13.11.4 patch release](https://about.gitlab.com/releases/2021/05/14/gitlab-13-11-4-released/). To avoid this issue, upgrade to GitLab 13.11.4 or later. ### Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode -GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) causes Geo secondary site statuses to appear to stop updating and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). +GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) causes Geo secondary site statuses to appear to stop upgrading and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). -## Updating to GitLab 13.10 +## Upgrading to GitLab 13.10 ### Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode -GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) causes Geo secondary site statuses to appear to stop updating and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). +GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) causes Geo secondary site statuses to appear to stop upgrading and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). -## Updating to GitLab 13.9 +## Upgrading to GitLab 13.9 -### Error during zero-downtime update: "cannot drop column asset_proxy_whitelist" +### Error during zero-downtime upgrade: "cannot drop column asset_proxy_whitelist" We've detected an issue [with a column rename](https://gitlab.com/gitlab-org/gitlab/-/issues/324160) that prevents upgrades to GitLab 13.9.0, 13.9.1, 13.9.2 and 13.9.3 when following the zero-downtime steps. It is necessary -to perform the following additional steps for the zero-downtime update: +to perform the following additional steps for the zero-downtime upgrade: 1. Before running the final `sudo gitlab-rake db:migrate` command on the deploy node, execute the following queries using the PostgreSQL console (or `sudo gitlab-psql`) @@ -169,27 +169,27 @@ PG::DependentObjectsStillExist: ERROR: cannot drop column asset_proxy_whitelist DETAIL: trigger trigger_0d588df444c8 on table application_settings depends on column asset_proxy_whitelist of table application_settings ``` -To work around this bug, follow the previous steps to complete the update. +To work around this bug, follow the previous steps to complete the upgrade. More details are available [in this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/324160). ### Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode -GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) causes Geo secondary site statuses to appear to stop updating and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). +GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) causes Geo secondary site statuses to appear to stop upgrading and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). -## Updating to GitLab 13.7 +## Upgrading to GitLab 13.7 We've detected an issue with the `FetchRemove` call used by Geo secondaries. This causes performance issues as we execute reference transaction hooks for -each updated reference. Delay any upgrade attempts until this is in the +each upgraded reference. Delay any upgrade attempts until this is in the [13.7.5 patch release.](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3002). More details are available [in this issue](https://gitlab.com/gitlab-org/git/-/issues/79). -## Updating to GitLab 13.5 +## Upgrading to GitLab 13.5 GitLab 13.5 has a [regression that prevents viewing a list of container repositories and registries](https://gitlab.com/gitlab-org/gitlab/-/issues/285475) on Geo secondaries. This issue is fixed in GitLab 13.6.1 and later. -## Updating to GitLab 13.3 +## Upgrading to GitLab 13.3 In GitLab 13.3, Geo removed the PostgreSQL [Foreign Data Wrapper](https://www.postgresql.org/docs/11/postgres-fdw.html) dependency for the tracking database. @@ -219,61 +219,61 @@ when using `--force` or `--skip-preflight-checks`, due to [an issue](https://git The [troubleshooting steps](troubleshooting.md#errors-when-using---skip-preflight-checks-or---force) contain a workaround if you run into errors during the failover. -## Updating to GitLab 13.2 +## Upgrading to GitLab 13.2 In GitLab 13.2, promoting a secondary site to a primary while the secondary is paused fails. Do not pause replication before promoting a secondary. If the site is paused, be sure to resume before promoting. To avoid this issue, upgrade to GitLab 13.4 or later. -## Updating to GitLab 13.0 +## Upgrading to GitLab 13.0 Upgrading to GitLab 13.0 requires GitLab 12.10 to already be using PostgreSQL version 11. For the recommended procedure, see the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). -## Updating to GitLab 12.10 +## Upgrading to GitLab 12.10 -GitLab 12.10 doesn't attempt to update the embedded PostgreSQL server when +GitLab 12.10 doesn't attempt to upgrade the embedded PostgreSQL server when using Geo, because the PostgreSQL upgrade requires downtime for secondaries while reinitializing streaming replication. It must be upgraded manually. For the recommended procedure, see the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). -## Updating to GitLab 12.9 +## Upgrading to GitLab 12.9 WARNING: GitLab 12.9.0 through GitLab 12.9.3 are affected by [a bug that stops repository verification](https://gitlab.com/gitlab-org/gitlab/-/issues/213523). The issue is fixed in GitLab 12.9.4. Upgrade to GitLab 12.9.4 or later. -By default, GitLab 12.9 attempts to update the embedded PostgreSQL server +By default, GitLab 12.9 attempts to upgrade the embedded PostgreSQL server version from 9.6 to 10.12, which requires downtime on secondaries while reinitializing streaming replication. For the recommended procedure, see the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). You can temporarily disable this behavior by running the following before -updating: +upgrading: ```shell sudo touch /etc/gitlab/disable-postgresql-upgrade ``` -## Updating to GitLab 12.8 +## Upgrading to GitLab 12.8 -By default, GitLab 12.8 attempts to update the embedded PostgreSQL server +By default, GitLab 12.8 attempts to upgrade the embedded PostgreSQL server version from 9.6 to 10.12, which requires downtime on secondaries while reinitializing streaming replication. For the recommended procedure, see the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). You can temporarily disable this behavior by running the following before -updating: +upgrading: ```shell sudo touch /etc/gitlab/disable-postgresql-upgrade ``` -## Updating to GitLab 12.7 +## Upgrading to GitLab 12.7 WARNING: Only upgrade to GitLab 12.7.5 or later. Do not upgrade to versions 12.7.0 @@ -281,65 +281,65 @@ through 12.7.4 because there is [an initialization order bug](https://gitlab.com [The fix](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/24021) was shipped in 12.7.5. -By default, GitLab 12.7 attempts to update the embedded PostgreSQL server +By default, GitLab 12.7 attempts to upgrade the embedded PostgreSQL server version from 9.6 to 10.9, which requires downtime on secondaries while reinitializing streaming replication. For the recommended procedure, see the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). You can temporarily disable this behavior by running the following before -updating: +upgrading: ```shell sudo touch /etc/gitlab/disable-postgresql-upgrade ``` -## Updating to GitLab 12.6 +## Upgrading to GitLab 12.6 -By default, GitLab 12.6 attempts to update the embedded PostgreSQL server +By default, GitLab 12.6 attempts to upgrade the embedded PostgreSQL server version from 9.6 to 10.9, which requires downtime on secondaries while reinitializing streaming replication. For the recommended procedure, see the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). You can temporarily disable this behavior by running the following before -updating: +upgrading: ```shell sudo touch /etc/gitlab/disable-postgresql-upgrade ``` -## Updating to GitLab 12.5 +## Upgrading to GitLab 12.5 -By default, GitLab 12.5 attempts to update the embedded PostgreSQL server +By default, GitLab 12.5 attempts to upgrade the embedded PostgreSQL server version from 9.6 to 10.9, which requires downtime on secondaries while reinitializing streaming replication. For the recommended procedure, see the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). You can temporarily disable this behavior by running the following before -updating: +upgrading: ```shell sudo touch /etc/gitlab/disable-postgresql-upgrade ``` -## Updating to GitLab 12.4 +## Upgrading to GitLab 12.4 -By default, GitLab 12.4 attempts to update the embedded PostgreSQL server +By default, GitLab 12.4 attempts to upgrade the embedded PostgreSQL server version from 9.6 to 10.9, which requires downtime on secondaries while reinitializing streaming replication. For the recommended procedure, see the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). You can temporarily disable this behavior by running the following before -updating: +upgrading: ```shell sudo touch /etc/gitlab/disable-postgresql-upgrade ``` -## Updating to GitLab 12.3 +## Upgrading to GitLab 12.3 WARNING: If the existing PostgreSQL server version is 9.6.x, we recommend upgrading to -GitLab 12.4 or later. By default, GitLab 12.3 attempts to update the embedded +GitLab 12.4 or later. By default, GitLab 12.3 attempts to upgrade the embedded PostgreSQL server version from 9.6 to 10.9. In certain circumstances, it can fail. For more information, see the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). @@ -349,11 +349,11 @@ requires downtime for secondaries while reinitializing streaming replication. For the recommended procedure, see the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). -## Updating to GitLab 12.2 +## Upgrading to GitLab 12.2 WARNING: If the existing PostgreSQL server version is 9.6.x, we recommend upgrading to -GitLab 12.4 or later. By default, GitLab 12.2 attempts to update the embedded +GitLab 12.4 or later. By default, GitLab 12.2 attempts to upgrade the embedded PostgreSQL server version from 9.6 to 10.9. In certain circumstances, it can fail. For more information, see the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). @@ -363,12 +363,12 @@ requires downtime for secondaries while reinitializing streaming replication. For the recommended procedure, see the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). -GitLab 12.2 includes the following minor PostgreSQL updates: +GitLab 12.2 includes the following minor PostgreSQL upgrades: - To version `9.6.14`, if you run PostgreSQL 9.6. - To version `10.9`, if you run PostgreSQL 10. -This update occurs even if major PostgreSQL updates are disabled. +This upgrade occurs even if major PostgreSQL upgrades are disabled. Before [refreshing Foreign Data Wrapper during a Geo upgrade](../../../update/zero_downtime.md#step-4-run-post-deployment-migrations-and-checks), restart the Geo tracking database: @@ -380,11 +380,11 @@ sudo gitlab-ctl restart geo-postgresql The restart avoids a version mismatch when PostgreSQL tries to load the FDW extension. -## Updating to GitLab 12.1 +## Upgrading to GitLab 12.1 WARNING: If the existing PostgreSQL server version is 9.6.x, we recommend upgrading to -GitLab 12.4 or later. By default, GitLab 12.1 attempts to update the embedded +GitLab 12.4 or later. By default, GitLab 12.1 attempts to upgrade the embedded PostgreSQL server version from 9.6 to 10.9. In certain circumstances, it can fail. For more information, see the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). @@ -394,7 +394,7 @@ requires downtime for secondaries while reinitializing streaming replication. For the recommended procedure, see the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). -## Updating to GitLab 12.0 +## Upgrading to GitLab 12.0 WARNING: This version is affected by a [bug that results in new LFS objects not being diff --git a/doc/administration/geo/secondary_proxy/index.md b/doc/administration/geo/secondary_proxy/index.md index 768adab9101..9a1aab8c238 100644 --- a/doc/administration/geo/secondary_proxy/index.md +++ b/doc/administration/geo/secondary_proxy/index.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto @@ -10,6 +10,7 @@ type: howto > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5914) in GitLab 14.4 [with a flag](../../feature_flags.md) named `geo_secondary_proxy`. Disabled by default. > - [Enabled by default for unified URLs](https://gitlab.com/gitlab-org/gitlab/-/issues/325732) in GitLab 14.6. > - [Disabled by default for different URLs](https://gitlab.com/gitlab-org/gitlab/-/issues/325732) in GitLab 14.6 [with a flag](../../feature_flags.md) named `geo_secondary_proxy_separate_urls`. +> - [Enabled by default for different URLs](https://gitlab.com/gitlab-org/gitlab/-/issues/346112) in GitLab 15.1. FLAG: On self-managed GitLab, this feature is only available by default for Geo sites using a unified URL. See below to @@ -36,6 +37,9 @@ Use secondary proxying for use-cases including: - Having all Geo sites behind a single URL. - Geographically load-balancing traffic without worrying about write access. +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For an overview, see: [Secondary proxying using geographic load-balancer and AWS Route53](https://www.youtube.com/watch?v=TALLy7__Na8). + ## Set up a unified URL for Geo sites Secondary sites can transparently serve read-write traffic. You can @@ -104,17 +108,19 @@ gitlab: GEO_SECONDARY_PROXY: "0" ``` -## Enable Geo proxying with Separate URLs +## Geo proxying with Separate URLs + +Since GitLab 15.1, Geo secondary proxying is enabled by default for separate URLs also. -The ability to use proxying with separate URLs is still in development. You can follow the -["Geo secondary proxying with separate URLs" epic](https://gitlab.com/groups/gitlab-org/-/epics/6865) -for progress. +There are minor known issues linked in the ["Geo secondary proxying with separate URLs" +epic](https://gitlab.com/groups/gitlab-org/-/epics/6865). You can also add feedback in the epic about any use-cases that +are not possible anymore with proxying enabled. -To try out this feature, enable the `geo_secondary_proxy_separate_urls` feature flag. +If you run into issues, to disable this feature, disable the `geo_secondary_proxy_separate_urls` feature flag. SSH into one node running Rails on your primary Geo site and run: ```shell -sudo gitlab-rails runner "Feature.enable(:geo_secondary_proxy_separate_urls)" +sudo gitlab-rails runner "Feature.disable(:geo_secondary_proxy_separate_urls)" ``` In Kubernetes, you can run the same command in the toolbox pod. Refer to the diff --git a/doc/administration/geo/secondary_proxy/location_aware_external_url.md b/doc/administration/geo/secondary_proxy/location_aware_external_url.md index b5a8d4baa1f..1430e99557f 100644 --- a/doc/administration/geo/secondary_proxy/location_aware_external_url.md +++ b/doc/administration/geo/secondary_proxy/location_aware_external_url.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index 52f4adc4e03..7dfca4c8f73 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto @@ -177,10 +177,10 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o In most cases, the following addresses are used to configure GitLab Geo: - | Configuration | Address | - |:----------------------------------------|:------------------------------------------------------| - | `postgresql['listen_address']` | **Primary** node's public or VPC private address. | - | `postgresql['md5_auth_cidr_addresses']` | **Secondary** node's public or VPC private addresses. | + | Configuration | Address | + |:----------------------------------------|:----------------------------------------------------------------------| + | `postgresql['listen_address']` | **Primary** node's public or VPC private address. | + | `postgresql['md5_auth_cidr_addresses']` | **Primary** and **Secondary** nodes' public or VPC private addresses. | If you are using Google Cloud Platform, SoftLayer, or any other vendor that provides a virtual private cloud (VPC) you can use the **primary** and **secondary** nodes diff --git a/doc/administration/geo/setup/external_database.md b/doc/administration/geo/setup/external_database.md index 58fd6b28797..7e8ffa829f9 100644 --- a/doc/administration/geo/setup/external_database.md +++ b/doc/administration/geo/setup/external_database.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -51,7 +51,7 @@ developed and tested. We aim to be compatible with most external gitlab-ctl set-geo-primary-node ``` - This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`. + This command uses your defined `external_url` in `/etc/gitlab/gitlab.rb`. ### Configure the external database to be replicated @@ -64,14 +64,14 @@ To set up an external database, you can either: Given you have a primary node set up on AWS EC2 that uses RDS. You can now just create a read-only replica in a different region and the -replication process will be managed by AWS. Make sure you've set Network ACL, Subnet, and +replication process is managed by AWS. Make sure you've set Network ACL (Access Control List), Subnet, and Security Group according to your needs, so the secondary application node can access the database. The following instructions detail how to create a read-only replica for common cloud providers: - Amazon RDS - [Creating a Read Replica](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.Create) -- Azure Database for PostgreSQL - [Create and manage read replicas in Azure Database for PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/howto-read-replicas-portal) +- Azure Database for PostgreSQL - [Create and manage read replicas in Azure Database for PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/single-server/how-to-read-replicas-portal) - Google Cloud SQL - [Creating read replicas](https://cloud.google.com/sql/docs/postgres/replication/create-replica) Once your read-only replica is set up, you can skip to [configure your secondary application node](#configure-secondary-application-nodes-to-use-the-external-read-replica). @@ -190,7 +190,7 @@ to grant additional roles to your tracking database user (by default, this is `gitlab_geo`): - Amazon RDS requires the [`rds_superuser`](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.html#Appendix.PostgreSQL.CommonDBATasks.Roles) role. -- Azure Database for PostgreSQL requires the [`azure_pg_admin`](https://docs.microsoft.com/en-us/azure/postgresql/howto-create-users#how-to-create-additional-admin-users-in-azure-database-for-postgresql) role. +- Azure Database for PostgreSQL requires the [`azure_pg_admin`](https://docs.microsoft.com/en-us/azure/postgresql/single-server/how-to-create-users#how-to-create-additional-admin-users-in-azure-database-for-postgresql) role. - Google Cloud SQL requires the [`cloudsqlsuperuser`](https://cloud.google.com/sql/docs/postgres/users#default-users) role. This is for the installation of extensions during installation and upgrades. As an alternative, diff --git a/doc/administration/geo/setup/index.md b/doc/administration/geo/setup/index.md index 04a341aa822..f4c21293782 100644 --- a/doc/administration/geo/setup/index.md +++ b/doc/administration/geo/setup/index.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto @@ -19,7 +19,7 @@ The steps below should be followed in the order they appear. **Make sure the Git If you installed GitLab using the Omnibus packages (highly recommended): -1. [Install GitLab Enterprise Edition](https://about.gitlab.com/install/) on the nodes that will serve as the **secondary** site. Do not create an account or log in to the new **secondary** site. +1. [Install GitLab Enterprise Edition](https://about.gitlab.com/install/) on the nodes that serve as the **secondary** site. Do not create an account or log in to the new **secondary** site. The **GitLab version must match** across primary and secondary sites. 1. [Add the GitLab License](../../../user/admin_area/license.md) on the **primary** site to unlock Geo. The license must be for [GitLab Premium](https://about.gitlab.com/pricing/) or higher. 1. [Set up the database replication](database.md) (`primary (read-write) <-> secondary (read-only)` topology). 1. [Configure fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md). This step is required and needs to be done on **both** the **primary** and **secondary** sites. diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md index 6888a2abe9a..26aa750900b 100644 --- a/doc/administration/gitaly/configure_gitaly.md +++ b/doc/administration/gitaly/configure_gitaly.md @@ -67,7 +67,7 @@ When running Gitaly on its own server, note the following regarding GitLab versi servers. - From GitLab 11.8 to 12.2, it is possible to use Elasticsearch in a Gitaly setup that doesn't use NFS. To use Elasticsearch in these versions, the - [repository indexer](../../integration/elasticsearch.md#elasticsearch-repository-indexer) + [repository indexer](../../integration/advanced_search/elasticsearch.md#elasticsearch-repository-indexer) must be enabled in your GitLab configuration. - [In GitLab 12.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/6481), the new indexer is the default and no configuration is required. @@ -825,14 +825,14 @@ information, see the [relevant documentation](monitoring.md#monitor-gitaly-concu ## Control groups -> - Introduced in GitLab 13.10. +> Introduced in GitLab 13.10. Gitaly shells out to Git for many of its operations. Git can consume a lot of resources for certain operations, especially for large repositories. Control groups (cgroups) in Linux allow limits to be imposed on how much memory and CPU can be consumed. See the [`cgroups` Linux man page](https://man7.org/linux/man-pages/man7/cgroups.7.html) for more information. -cgroups can be useful for protecting the system against resource exhaustion because of overconsumption of memory and CPU. +cgroups can be useful for protecting the system against resource exhaustion because of overcomsumption of memory and CPU. Gitaly has built-in cgroups control. When configured, Gitaly assigns Git processes to a cgroup based on the repository the Git command is operating in. @@ -856,7 +856,59 @@ Using cgroups allows the kernel to kill these operations before they hog up all ### Configure cgroups in Gitaly -To configure cgroups in Gitaly, add `gitaly['cgroups']` to `/etc/gitlab/gitlab.rb`. For +Two ways of configuring cgroups are available. + +#### Configure cgroups (new method) + +> This method of configuring cgroups introduced in GitLab 15.1. + +FLAG: +On self-managed GitLab, by default this method of configuring cgroups is not available. To make it available, ask an administrator to +[enable the feature flag](../feature_flags.md) named `gitaly_run_cmds_in_cgroup`. + +Gitaly creates a pool of cgroups that are isolated based on the repository used in the Git command to be placed under one of these cgroups. + +To configure cgroups in Gitaly, add `gitaly['cgroups']` to `/etc/gitlab/gitlab.rb`. + +For example: + +```ruby +# in /etc/gitlab/gitlab.rb +gitaly['cgroups_mountpoint'] = "/sys/fs/cgroup" +gitaly['cgroups_hierarchy_root'] =>"gitaly" +gitaly['cgroups_memory_bytes'] = 64424509440, # 60gb +gitaly['cgroups_cpu_shares'] = 1024 +gitaly['cgroups_repositories_count'] => 1000, +gitaly['cgroups_repositories_memory_bytes'] => 32212254720 # 20gb +gitaly['cgroups_repositories_cpu_shares'] => 512 +``` + +- `cgroups_mountpoint` is where the parent cgroup directory is mounted. Defaults to `/sys/fs/cgroup`. +- `cgroups_hierarchy_root` is the parent cgroup under which Gitaly creates groups, and + is expected to be owned by the user and group Gitaly runs as. Omnibus GitLab + creates the set of directories `mountpoint/<cpu|memory>/hierarchy_root` + when Gitaly starts. +- `cgroups_memory_bytes` is the total memory limit that is imposed collectively on all + Git processes that Gitaly spawns. 0 implies no limit. +- `cgroups_cpu_shares` is the cpu limit that is imposed collectively on all Git + processes that Gitaly spawns. 0 implies no limit. The maximum is 1024 shares, + which represents 100% of CPU. +- `cgroups_repositories_count` is the number of cgroups in the cgroups pool. Each time a new Git + command is spawned, Gitaly assigns it to one of these cgroups based + on the repository the command is for. A circular hashing algorithm assigns + Git commands to these cgroups, so a Git command for a repository is + always assigned to the same cgroup. +- `cgroups_repositories_memory_bytes` is the total memory limit that is imposed collectively on all + Git processes that Gitaly spawns. 0 implies no limit. This value cannot exceed + that of the top level `cgroups_memory_bytes`. +- `cgroups_repositories_cpu_shares` is the CPU limit that is imposed collectively on all Git + processes Gitaly spawns. 0 implies no limit. The maximum is 1024 shares, + which represents 100% of CPU. This value cannot exceed that of the top + level`cgroups_cpu_shares`. + +#### Configure cgroups (legacy method) + +To configure cgroups in Gitaly for GitLab versions using the legacy method, add `gitaly['cgroups']` to `/etc/gitlab/gitlab.rb`. For example: ```ruby @@ -868,7 +920,6 @@ gitaly['cgroups_memory_limit'] = 32212254720 gitaly['cgroups_memory_enabled'] = true gitaly['cgroups_cpu_shares'] = 1024 gitaly['cgroups_cpu_enabled'] = true - ``` - `cgroups_count` is the number of cgroups created. Each time a new @@ -883,9 +934,28 @@ gitaly['cgroups_cpu_enabled'] = true - `cgroups_memory_enabled` enables or disables the memory limit on cgroups. - `cgroups_memory_bytes` is the total memory limit each cgroup imposes on the processes added to it. - `cgroups_cpu_enabled` enables or disables the CPU limit on cgroups. -- `cgroups_cpu_shares` is the CPU limit each cgroup imposes on the processes added to it. The maximum is 1024 shares, - which represents 100% of CPU. - which represents 100% of CPU. +- `cgroups_cpu_shares` is the CPU limit each cgroup imposes on the processes added to it. The maximum is 1024 shares, which represents 100% of CPU. + +### Configuring oversubscription + +In the previous example using the new configuration method: + +- The top level memory limit is capped at 60gb. +- Each of the 1000 cgroups in the repositories pool is capped at 20gb. + +This is called "oversubscription". Each cgroup in the pool has a much larger capacity than 1/1000th +of the top-level memory limit. + +This strategy has two main benefits: + +- It gives the host protection from overall memory starvation (OOM), because the top-level + cgroup's memory limit can be set to a threshold smaller than the host's + capacity. Processes outside of that cgroup are not at risk of OOM. +- It allows each individual cgroup in the pool to burst up to a generous upper + bound (in this example 20 GB) that is smaller than the parent cgroup's limit, + but substantially larger than 1/N of the parent's limit. In this example, up + to 3 child cgroups can concurrently burst up to their max. In general, all + 1000 cgroups would use much less than the 20 GB. ## Background Repository Optimization diff --git a/doc/administration/gitaly/faq.md b/doc/administration/gitaly/faq.md index a5c2c7d1469..f6571295200 100644 --- a/doc/administration/gitaly/faq.md +++ b/doc/administration/gitaly/faq.md @@ -1,90 +1,6 @@ --- -stage: Create -group: Gitaly -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'index.md' +remove_date: '2022-08-24' --- -# Frequently asked questions **(FREE SELF)** - -The following are answers to frequently asked questions about Gitaly and Gitaly Cluster. For -troubleshooting information, see [Troubleshooting Gitaly and Gitaly Cluster](troubleshooting.md). - -## How does Gitaly Cluster compare to Geo? - -Gitaly Cluster and [Geo](../geo/index.md) both provide redundancy. However the redundancy of: - -- Gitaly Cluster provides fault tolerance for data storage and is invisible to the user. Users are - not aware when Gitaly Cluster is used. -- Geo provides [replication](../geo/index.md) and [disaster recovery](../geo/disaster_recovery/index.md) for - an entire instance of GitLab. Users know when they are using Geo for - [replication](../geo/index.md). Geo [replicates multiple data types](../geo/replication/datatypes.md#limitations-on-replicationverification), - including Git data. - -The following table outlines the major differences between Gitaly Cluster and Geo: - -| Tool | Nodes | Locations | Latency tolerance | Failover | Consistency | Provides redundancy for | -|:---------------|:---------|:----------|:------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------|:--------------------------------------|:------------------------| -| Gitaly Cluster | Multiple | Single | [Less than 1 second, ideally single-digit milliseconds](praefect.md#network-latency-and-connectivity) | [Automatic](praefect.md#automatic-failover-and-primary-election-strategies) | [Strong](index.md#strong-consistency) | Data storage in Git | -| Geo | Multiple | Multiple | Up to one minute | [Manual](../geo/disaster_recovery/index.md) | Eventual | Entire GitLab instance | - -For more information, see: - -- Geo [use cases](../geo/index.md#use-cases). -- Geo [architecture](../geo/index.md#architecture). - -## Are there instructions for migrating to Gitaly Cluster? - -Yes! For more information, see [Migrating to Gitaly Cluster](index.md#migrating-to-gitaly-cluster). - -## What are some repository storage recommendations? - -The size of the required storage can vary between instances and depends on the set -[replication factor](index.md#replication-factor). You might want to include implementing -repository storage redundancy. - -For a replication factor: - -- Of `1`: NFS, Gitaly, and Gitaly Cluster have roughly the same storage requirements. -- More than `1`: The amount of required storage is `used space * replication factor`. `used space` - should include any planned future growth. - -## What are some Praefect database storage requirements? - -The requirements are relatively low because the database contains only metadata of: - -- Where repositories are located. -- Some queued work. - -It depends on the number of repositories, but a useful minimum is 5-10 GB, similar to the main -GitLab application database. - -## Can the GitLab application database and the Praefect database be on the same servers? - -Yes, however Praefect should have it's own database server when using Omnibus GitLab PostgreSQL. If -there is a failover, Praefect isn't aware and starts to fail as the database it's trying to use would -either: - -- Be unavailable. -- In read-only mode. - -A future solution may allow for Praefect and Omnibus GitLab databases on the same PostgreSQL server. -For more information, see the relevant: - -- [Omnibus GitLab issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919). -- [Gitaly issue](https://gitlab.com/gitlab-org/gitaly/-/issues/3398). - -## Is PgBouncer required for the Praefect database? - -No, because the number of connections Praefect makes is low. You can use the same PgBouncer instance -for both the GitLab application database and the Praefect database if you wish. - -## Are there any special considerations for Gitaly Cluster when PostgreSQL is upgraded? - -There are no special requirements. Gitaly Cluster requires PostgreSQL version 11 or later. - -## Praefect database tables are empty? - -These tables are created per the [specific configuration section](praefect.md#postgresql). - -If you find you have an empty Praefect database table, see the -[relevant troubleshooting section](troubleshooting.md#relation-does-not-exist-errors). +This document was moved to [another location](index.md). diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index f43f9f5d6c1..160313073a5 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -71,7 +71,7 @@ the current status of these issues, please refer to the referenced issues and ep | Issue | Summary | How to avoid | |:--------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------| | Gitaly Cluster + Geo - Issues retrying failed syncs | If Gitaly Cluster is used on a Geo secondary site, repositories that have failed to sync could continue to fail when Geo tries to resync them. Recovering from this state requires assistance from support to run manual steps. Work is in-progress to update Gitaly Cluster to [identify repositories with a unique and persistent identifier](https://gitlab.com/gitlab-org/gitaly/-/issues/3485), which is expected to resolve the issue. | No known solution at this time. | -| Praefect unable to insert data into the database due to migrations not being applied after an upgrade | If the database is not kept up to date with completed migrations, then the Praefect node is unable to perform normal operation. | Make sure the Praefect database is up and running with all migrations completed (For example: `/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migrate-status` should show a list of all applied migrations). Consider [requesting live upgrade assistance](https://about.gitlab.com/support/scheduling-upgrade-assistance.html) so your upgrade plan can be reviewed by support. | +| Praefect unable to insert data into the database due to migrations not being applied after an upgrade | If the database is not kept up to date with completed migrations, then the Praefect node is unable to perform normal operation. | Make sure the Praefect database is up and running with all migrations completed (For example: `/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migrate-status` should show a list of all applied migrations). Consider [requesting live upgrade assistance](https://about.gitlab.com/support/scheduling-upgrade-assistance/) so your upgrade plan can be reviewed by support. | | Restoring a Gitaly Cluster node from a snapshot in a running cluster | Because the Gitaly Cluster runs with consistent state, introducing a single node that is behind will result in the cluster not being able to reconcile the nodes data and other nodes data | Don't restore a single Gitaly Cluster node from a backup snapshot. If you must restore from backup, it's best to snapshot all Gitaly Cluster nodes at the same time and take a database dump of the Praefect database. | ### Snapshot backup and recovery limitations @@ -206,6 +206,29 @@ WARNING: If complete cluster failure occurs, disaster recovery plans should be executed. These can affect the RPO and RTO discussed above. +### Comparison to Geo + +Gitaly Cluster and [Geo](../geo/index.md) both provide redundancy. However the redundancy of: + +- Gitaly Cluster provides fault tolerance for data storage and is invisible to the user. Users are + not aware when Gitaly Cluster is used. +- Geo provides [replication](../geo/index.md) and [disaster recovery](../geo/disaster_recovery/index.md) for + an entire instance of GitLab. Users know when they are using Geo for + [replication](../geo/index.md). Geo [replicates multiple data types](../geo/replication/datatypes.md#limitations-on-replicationverification), + including Git data. + +The following table outlines the major differences between Gitaly Cluster and Geo: + +| Tool | Nodes | Locations | Latency tolerance | Failover | Consistency | Provides redundancy for | +|:---------------|:---------|:----------|:------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------|:--------------------------------------|:------------------------| +| Gitaly Cluster | Multiple | Single | [Less than 1 second, ideally single-digit milliseconds](praefect.md#network-latency-and-connectivity) | [Automatic](praefect.md#automatic-failover-and-primary-election-strategies) | [Strong](index.md#strong-consistency) | Data storage in Git | +| Geo | Multiple | Multiple | Up to one minute | [Manual](../geo/disaster_recovery/index.md) | Eventual | Entire GitLab instance | + +For more information, see: + +- Geo [use cases](../geo/index.md#use-cases). +- Geo [architecture](../geo/index.md#architecture). + ### Virtual storage Virtual storage makes it viable to have a single repository storage in GitLab to simplify repository @@ -320,9 +343,9 @@ Use the [`praefect metadata`](troubleshooting.md#view-repository-metadata) subco - The virtual storage and relative path. - The repository ID. -The repository on disk also contains the project path in the Git configuration file. The configuration file can be used to determine -the project's location even if the repository's metadata has been deleted. Follow the -[instructions in hashed storage's documentation](../repository_storage_types.md#from-hashed-path-to-project-name). +The repository on disk also contains the project path in the Git configuration file. The configuration +file can be used to determine the project path even if the repository's metadata has been deleted. +Follow the [instructions in hashed storage's documentation](../repository_storage_types.md#from-hashed-path-to-project-name). #### Atomicity of operations @@ -330,7 +353,7 @@ Gitaly Cluster uses the PostgreSQL metadata store with the storage layout to ens deletion, and move operations. The disk operations can't be atomically applied across multiple storages. However, PostgreSQL guarantees the atomicity of the metadata operations. Gitaly Cluster models the operations in a manner that the failing operations always leave the metadata consistent. The disks may contain stale state even after successful operations. This is expected and the leftover state -won't intefere with future operations but may use up disk space unnecessarily until a clean up is performed. +won't interfere with future operations but may use up disk space unnecessarily until a clean up is performed. There is on-going work on a [background crawler](https://gitlab.com/gitlab-org/gitaly/-/issues/3719) that cleans up the leftover repositories from the storages. @@ -366,7 +389,7 @@ relative path of the repository in the metadata store. ### Moving beyond NFS Engineering support for NFS for Git repositories is deprecated. Technical support is planned to be unavailable starting -November 22, 2022. Please see our [statement of support](https://about.gitlab.com/support/statement-of-support.html#gitaly-and-nfs) +November 22, 2022. Please see our [statement of support](https://about.gitlab.com/support/statement-of-support/#gitaly-and-nfs) for more details. [Network File System (NFS)](https://en.wikipedia.org/wiki/Network_File_System) @@ -495,24 +518,31 @@ For configuration information, see [Configure replication factor](praefect.md#co For more information on configuring Gitaly Cluster, see [Configure Gitaly Cluster](praefect.md). -## Migrating to Gitaly Cluster +### Upgrade Gitaly Cluster + +To upgrade a Gitaly Cluster, follow the documentation for +[zero downtime upgrades](../../update/zero_downtime.md#gitaly-or-gitaly-cluster). -See the [Before deploying Gitaly Cluster](#before-deploying-gitaly-cluster) section before continuing. The basic process -for migrating to Gitaly Cluster involves: +## Migrate to Gitaly Cluster + +WARNING: +Some [known issues](#known-issues) exist in Gitaly Cluster. Review the following information before you continue. + +Before migrating to Gitaly Cluster: + +- Review [Before deploying Gitaly Cluster](#before-deploying-gitaly-cluster). +- Upgrade to the latest possible version of GitLab, to take advantage of improvements and bug fixes. + +To migrate to Gitaly Cluster: 1. Create the required storage. Refer to - [repository storage recommendations](faq.md#what-are-some-repository-storage-recommendations). + [repository storage recommendations](praefect.md#repository-storage-recommendations). 1. Create and configure [Gitaly Cluster](praefect.md). 1. [Move the repositories](../operations/moving_repositories.md#move-repositories). To migrate to Gitaly Cluster, existing repositories stored outside Gitaly Cluster must be moved. There is no automatic migration but the moves can be scheduled with the GitLab API. -WARNING: -Some [known database inconsistency issues](#known-issues) exist in Gitaly Cluster. We recommend you -remain on your current service for now. - -NOTE: -GitLab requires a `default` repository storage to be configured. +Even if you don't use the `default` repository storage, you must ensure it is configured. [Read more about this limitation](configure_gitaly.md#gitlab-requires-a-default-repository-storage). ## Migrate off Gitaly Cluster diff --git a/doc/administration/gitaly/monitoring.md b/doc/administration/gitaly/monitoring.md index 17f94f912ee..a0a2e43569c 100644 --- a/doc/administration/gitaly/monitoring.md +++ b/doc/administration/gitaly/monitoring.md @@ -155,6 +155,7 @@ The following metrics are available from the `/metrics` endpoint: - `gitaly_praefect_node_latency_bucket`, a histogram measuring the latency in Gitaly returning health check information to Praefect. This indicates Praefect connection saturation. Available in GitLab 12.10 and later. +- `gitaly_praefect_connections_total`, the total number of connections to Praefect. [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4220) in GitLab 14.7. To monitor [strong consistency](index.md#strong-consistency), you can use the following Prometheus metrics: diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index fb9f3e9c361..f6166d713b1 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -19,9 +19,6 @@ Configure Gitaly Cluster using either: Smaller GitLab installations may need only [Gitaly itself](index.md). -To upgrade a Gitaly Cluster, follow the documentation for -[zero downtime upgrades](../../update/zero_downtime.md#gitaly-cluster). - ## Requirements The minimum recommended configuration for a Gitaly Cluster requires: @@ -54,7 +51,7 @@ Achieving acceptable latency between Gitaly nodes: are designed for this type of synchronization. Latency of less than 2ms should be sufficient for Gitaly Cluster. If you can't provide low network latencies for replication (for example, between distant locations), consider Geo. For -more information, see [How does Gitaly Cluster compare to Geo](faq.md#how-does-gitaly-cluster-compare-to-geo). +more information, see [Comparison to Geo](index.md#comparison-to-geo). Gitaly Cluster [components](index.md#components) communicate with each other over many routes. Your firewall rules must allow the following for Gitaly Cluster to function properly: @@ -64,6 +61,7 @@ allow the following for Gitaly Cluster to function properly: | GitLab | Praefect load balancer | `2305` | `3305` | | Praefect load balancer | Praefect | `2305` | `3305` | | Praefect | Gitaly | `8075` | `9999` | +| Praefect | GitLab (internal API) | `80` | `443` | | Gitaly | GitLab (internal API) | `80` | `443` | | Gitaly | Praefect load balancer | `2305` | `3305` | | Gitaly | Praefect | `2305` | `3305` | @@ -74,6 +72,16 @@ Gitaly does not directly connect to Praefect. However, requests from Gitaly to t load balancer may still be blocked unless firewalls on the Praefect nodes allow traffic from the Gitaly nodes. +### Praefect database storage + +The requirements are relatively low because the database contains only metadata of: + +- Where repositories are located. +- Some queued work. + +It depends on the number of repositories, but a useful minimum is 5-10 GB, similar to the main +GitLab application database. + ## Setup Instructions If you [installed](https://about.gitlab.com/install/) GitLab using the Omnibus GitLab package @@ -168,6 +176,18 @@ The following options are available: - Use a cloud-managed PostgreSQL service. AWS [Relational Database Service](https://aws.amazon.com/rds/) is recommended. +Setting up PostgreSQL creates empty Praefect tables. For more information, see the +[relevant troubleshooting section](troubleshooting.md#relation-does-not-exist-errors). + +#### Running GitLab and Praefect databases on the same server + +The GitLab application database and the Praefect database can be run on the same server. However, Praefect should have +its own database server when using Omnibus GitLab PostgreSQL. If there is a failover, Praefect isn't aware and starts to +fail as the database it's trying to use would either: + +- Be unavailable. +- In read-only mode. + #### Manual database setup To complete this section you need: @@ -278,8 +298,12 @@ reads distribution caching is enabled by configuration #### Use PgBouncer To reduce PostgreSQL resource consumption, we recommend setting up and configuring -[PgBouncer](https://www.pgbouncer.org/) in front of the PostgreSQL instance. To do -this, you must point Praefect to PgBouncer by setting database parameters on Praefect configuration: +[PgBouncer](https://www.pgbouncer.org/) in front of the PostgreSQL instance. However, PgBouncer isn't required because +Praefect makes a low number of connections. If you choose to use PgBouncer, you can use the same PgBouncer instance for +both the GitLab application database and the Praefect database. + +To configure PgBouncer in front of the PostgreSQL instance, you must point Praefect to PgBouncer by setting database +parameters on Praefect configuration: ```ruby praefect['database_host'] = PGBOUNCER_HOST @@ -475,7 +499,7 @@ Updates to example must be made at: # Some metrics run queries against the database. Enabling separate database metrics allows # these metrics to be collected when the metrics are - # scraped on a separate /db_metrics endpoint. + # scraped on a separate /db_metrics endpoint. praefect['separate_database_metrics'] = true ``` @@ -519,7 +543,7 @@ Updates to example must be made at: WARNING: If you have data on an already existing storage called `default`, you should configure the virtual storage with another name and - [migrate the data to the Gitaly Cluster storage](index.md#migrating-to-gitaly-cluster) + [migrate the data to the Gitaly Cluster storage](index.md#migrate-to-gitaly-cluster) afterwards. Replace `PRAEFECT_INTERNAL_TOKEN` with a strong secret, which is used by @@ -632,12 +656,8 @@ and on all Praefect clients that communicate with it following the procedure des Note the following: -- The certificate must specify the address you use to access the Praefect server. If - addressing the Praefect server by: - - - Hostname, you can either use the Common Name field for this, or add it as a Subject - Alternative Name. - - IP address, you must add it as a Subject Alternative Name to the certificate. +- The certificate must specify the address you use to access the Praefect server. You must add the hostname or IP + address as a Subject Alternative Name to the certificate. - When running Praefect sub-commands such as `dial-nodes` and `list-untracked-repositories` from the command line with [Gitaly TLS enabled](configure_gitaly.md#enable-tls-support), you must set the `SSL_CERT_DIR` or `SSL_CERT_FILE` environment variable so that the Gitaly certificate is trusted. For example: @@ -651,6 +671,8 @@ Note the following: This allows you to do a gradual transition from unencrypted to encrypted traffic, if necessary. + To disable the unencrypted listener, set `praefect['listen_addr'] = nil`. + To configure Praefect with TLS: **For Omnibus GitLab** @@ -992,7 +1014,7 @@ Particular attention should be shown to: WARNING: If you have existing data stored on the default Gitaly storage, - you should [migrate the data your Gitaly Cluster storage](index.md#migrating-to-gitaly-cluster) + you should [migrate the data your Gitaly Cluster storage](index.md#migrate-to-gitaly-cluster) first. ```ruby @@ -1107,7 +1129,7 @@ Particular attention should be shown to: #### Use TCP for existing GitLab instances When adding Gitaly Cluster to an existing Gitaly instance, the existing Gitaly storage -must use a TCP address. If `gitaly_address` is not specified, then a Unix socket is used, +must be listening on TCP/TLS. If `gitaly_address` is not specified, then a Unix socket is used, which prevents the communication with the cluster. For example: @@ -1116,7 +1138,7 @@ For example: git_data_dirs({ 'default' => { 'gitaly_address' => 'tcp://old-gitaly.internal:8075' }, 'cluster' => { - 'gitaly_address' => 'tcp://<PRAEFECT_LOADBALANCER_HOST>:2305', + 'gitaly_address' => 'tls://<PRAEFECT_LOADBALANCER_HOST>:3305', 'gitaly_token' => '<praefect_external_token>' } }) @@ -1223,6 +1245,18 @@ You can configure: If `default_replication_factor` is unset, the repositories are always replicated on every node defined in `virtual_storages`. If a new node is introduced to the virtual storage, both new and existing repositories are replicated to the node automatically. +### Repository storage recommendations + +The size of the required storage can vary between instances and depends on the set +[replication factor](index.md#replication-factor). You might want to include implementing +repository storage redundancy. + +For a replication factor: + +- Of `1`: NFS, Gitaly, and Gitaly Cluster have roughly the same storage requirements. +- More than `1`: The amount of required storage is `used space * replication factor`. `used space` + should include any planned future growth. + ## Repository verification > [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4080) in GitLab 15.0. diff --git a/doc/administration/gitaly/recovery.md b/doc/administration/gitaly/recovery.md index 6de2acf1792..ecd68234b01 100644 --- a/doc/administration/gitaly/recovery.md +++ b/doc/administration/gitaly/recovery.md @@ -359,7 +359,7 @@ Add the `-older-than` option to avoid showing repositories that are the process Praefect database. Replace `<duration>` with a time duration (for example, `5s`, `10m`, or `1h`). Defaults to `6h`. ```shell -sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml list-untracked-repositories -older-than <duration> +sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml list-untracked-repositories -older-than <duration> ``` Only repositories with a creation time before the specified duration are considered. diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index c79ed1d1707..a1f542ddac8 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -8,9 +8,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w Refer to the information below when troubleshooting Gitaly and Gitaly Cluster. -Before troubleshooting, see the Gitaly and Gitaly Cluster -[frequently asked questions](faq.md). - ## Troubleshoot Gitaly The following sections provide possible solutions to Gitaly errors. @@ -349,6 +346,14 @@ that do not exist in a repository. "error":"not found: .gitlab-ci.yml" ``` +### Git pushes are slow when Dynatrace is enabled + +Dynatrace can cause the `/opt/gitlab/embedded/bin/gitaly-hooks` reference transaction hook, +to take several seconds to start up and shut down. `gitaly-hooks` is executed twice when users +push, which causes a significant delay. + +If Git pushes are too slow when Dynatrace is enabled, disable Dynatrace. + ## Troubleshoot Praefect (Gitaly Cluster) The following sections provide possible solutions to Gitaly Cluster errors. diff --git a/doc/administration/housekeeping.md b/doc/administration/housekeeping.md index 71b8439f70a..93409b54c0d 100644 --- a/doc/administration/housekeeping.md +++ b/doc/administration/housekeeping.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/inactive_project_deletion.md b/doc/administration/inactive_project_deletion.md index a2d2093c57b..224b52d420e 100644 --- a/doc/administration/inactive_project_deletion.md +++ b/doc/administration/inactive_project_deletion.md @@ -9,32 +9,35 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/85689) in GitLab 15.0 [with a flag](../administration/feature_flags.md) named `inactive_projects_deletion`. Disabled by default. FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../administration/feature_flags.md) named `inactive_projects_deletion`. On GitLab.com, this feature is not available. This feature is not ready for production use. Administrators of large GitLab instances can find that over time, projects become inactive and are no longer used. -These projects take up unnecessary disk space. With inactive project deletion, you can identify these projects, warn -the maintainers ahead of time, and then delete the projects if they remain inactive. When an inactive project is +These projects take up unnecessary disk space. With inactive project deletion, you can identify these projects, warn +the maintainers ahead of time, and then delete the projects if they remain inactive. When an inactive project is deleted, the action generates an audit event that it was performed by the first active administrator. ## Configure inactive project deletion -You can configure inactive projects deletion or turn it off using the -[Application settings API](../api/settings.md#change-application-settings). +You can configure inactive projects deletion or turn it off using either: + +- [The GitLab API](#using-the-api) (GitLab 15.0 and later). +- [The GitLab UI](#using-the-gitlab-ui) (GitLab 15.1 and later). The following options are available: -- `delete_inactive_projects`: Enable or disable inactive project deletion. -- `inactive_projects_min_size_mb`: Minimum size (MB) of inactive projects to be considered for deletion. - Projects smaller in size than this threshold aren't considered inactive. -- `inactive_projects_delete_after_months`: Minimum duration (months) after which a project is scheduled for deletion if - it continues be inactive. -- `inactive_projects_send_warning_email_after_months`: Minimum duration (months) after which a deletion warning email is - sent if a project continues to be inactive. The warning email is sent to users with the Owner and Maintainer roles of - the inactive project. This duration should be less than the `inactive_projects_delete_after_months` duration. +- **Delete inactive projects** (`delete_inactive_projects`): Enable or disable inactive project deletion. +- **Delete inactive projects that exceed** (`inactive_projects_min_size_mb`): Minimum size (MB) of inactive projects to + be considered for deletion. Projects smaller in size than this threshold aren't considered inactive. +- **Delete project after** (`inactive_projects_delete_after_months`): Minimum duration (months) after which a project is + scheduled for deletion if it continues be inactive. +- **Send warning email** (`inactive_projects_send_warning_email_after_months`): Minimum duration (months) after which a + deletion warning email is sent if a project continues to be inactive. The warning email is sent to users with the + Owner and Maintainer roles of the inactive project. This duration must be less than the + **Delete project after** (`inactive_projects_delete_after_months`) duration. -For example: +For example (using the API): - `delete_inactive_projects` enabled. - `inactive_projects_min_size_mb` set to `50`. @@ -49,6 +52,22 @@ In this scenario, when a project's size is: with the scheduled date of deletion. - More than 12 months, the project is scheduled for deletion. +### Using the API + +You can use the [Application settings API](../api/settings.md#change-application-settings) to configure inactive projects. + +### Using the GitLab UI + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/85575) in GitLab 15.1. + +To configure inactive projects with the GitLab UI: + +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Settings > Repository**. +1. Expand **Repository maintenance**. +1. In the **Inactive project deletion** section, configure the necessary options. +1. Select **Save changes**. + ## Determine when a project was last active You can view a project's activities and determine when the project was last active in the following ways: diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index 2fd57036169..95301ec026a 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -68,7 +68,8 @@ this method only supports replies, and not the other features of [incoming email ## Accepted headers -> Accepting `Received` headers [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/81489) in GitLab 14.9 [with a flag](feature_flags.md) named `use_received_header_for_incoming_emails`. Enabled by default. +> - Accepting `Received` headers [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/81489) in GitLab 14.9 [with a flag](feature_flags.md) named `use_received_header_for_incoming_emails`. Enabled by default. +> - Accepting `Received` headers: [feature flag](feature_flags.md) named `use_received_header_for_incoming_emails` [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/362596) in GitLab 14.1. Email is processed correctly when a configured email address is present in one of the following headers (sorted in the order they are checked): @@ -132,7 +133,7 @@ issue by email" or "[Create new merge request by email](../user/project/merge_requests/creating_merge_requests.md#by-sending-an-email)" features by using a project's unique address as the email when signing up for Slack. This would send a confirmation email, which would create a new issue or -merge request on the project owned by the attacker, allowing them to click the +merge request on the project owned by the attacker, allowing them to select the confirmation link and validate their account on your company's private Slack instance. diff --git a/doc/administration/index.md b/doc/administration/index.md index 1d8dcd34d68..29fb65e2deb 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" description: 'Learn how to install, configure, update, and maintain your GitLab instance.' @@ -65,7 +65,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. requests diffs external storage. - [Broadcast Messages](../user/admin_area/broadcast_messages.md): Send messages to GitLab users through the UI. -- [Elasticsearch](../integration/elasticsearch.md): Enable Elasticsearch to +- [Elasticsearch](../integration/advanced_search/elasticsearch.md): Enable Elasticsearch to empower Advanced Search. Use when you deal with a huge amount of data. - [External Classification Policy Authorization](../user/admin_area/settings/external_authorization.md) - [Add a license](../user/admin_area/license.md): Add a license to unlock @@ -120,7 +120,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Creating users](../user/profile/account/create_accounts.md): Create users manually or through authentication integrations. - [Libravatar](libravatar.md): Use Libravatar instead of Gravatar for user avatars. -- [Sign-up restrictions](../user/admin_area/settings/sign_up_restrictions.md): block email addresses of specific domains, or whitelist only specific domains. +- [Sign-up restrictions](../user/admin_area/settings/sign_up_restrictions.md): block email addresses of specific domains, or allow only specific domains. - [Access restrictions](../user/admin_area/settings/visibility_and_access_controls.md#configure-enabled-git-access-protocols): Define which Git access protocols can be used to talk to GitLab (SSH, HTTP, HTTPS). - [Authentication and Authorization](auth/index.md): Configure external authentication with LDAP, SAML, CAS, and additional providers. - [Sync LDAP](auth/ldap/index.md) @@ -194,7 +194,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Monitoring GitLab](monitoring/index.md): - [Monitoring uptime](../user/admin_area/monitoring/health_check.md): Check the server status using the health check endpoint. - - [IP whitelist](monitoring/ip_whitelist.md): Monitor endpoints that provide health check information when probed. + - [IP allowlist](monitoring/ip_whitelist.md): Monitor endpoints that provide health check information when probed. - [Monitoring GitHub imports](monitoring/github_imports.md): The GitLab GitHub Importer displays Prometheus metrics to monitor the health and progress of the importer. ### Performance Monitoring @@ -239,7 +239,7 @@ who are aware of the risks. - Related links: - [GitLab Developer Documentation](../development/index.md) - [Repairing and recovering broken Git repositories](https://git.seveas.net/repairing-and-recovering-broken-git-repositories.html) - - [Testing with OpenSSL](https://www.feistyduck.com/library/openssl-cookbook/online/ch-testing-with-openssl.html) + - [Testing with OpenSSL](https://www.feistyduck.com/library/openssl-cookbook/online/testing-with-openssl/index.html) - [`strace` zine](https://wizardzines.com/zines/strace/) - GitLab.com-specific resources: - [Group SAML/SCIM setup](troubleshooting/group_saml_scim.md) diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index 8d3fec3b19e..d86414ae285 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference @@ -133,8 +133,9 @@ Limit the maximum daily member invitations allowed per group hierarchy. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/61151) in GitLab 13.12. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/330133) in GitLab 14.1. +> - [Limit changed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/89591) from per-hook to per-top-level namespace in GitLab 15.1. -Limit the number of times any given webhook can be called per minute. +Limit the number of times a webhook can be called per minute, per top-level namespace. This only applies to project and group webhooks. Calls over the rate limit are logged into `auth.log`. @@ -164,6 +165,14 @@ This setting limits global search requests. | Authenticated user | 30 | | Unauthenticated user | 10 | +### Pipeline creation rate limit + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/362475) in GitLab 15.0. + +This setting limits the request rate to the pipeline creation endpoints. + +Read more about [pipeline creation rate limits](../user/admin_area/settings/rate_limit_on_pipelines_creation.md). + ## Gitaly concurrency limit Clone traffic can put a large strain on your Gitaly service. To prevent such workloads from overwhelming your Gitaly server, you can set concurrency limits in Gitaly's configuration file. @@ -383,6 +392,38 @@ Plan.default.actual_limits.update!(ci_active_jobs: 500) Set the limit to `0` to disable it. +### Number of pipelines running concurrently + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32823) in GitLab 12.5. + +The total number of pipelines running concurrently can be limited per project. +When enabled, the limit is checked each time a new pipeline is created. +Without a concurrent pipelines limit, a sudden flood of triggered pipelines could +overwhelm the instance resources. + +If a new pipeline would cause the total number of pipelines to exceed the limit, +the pipeline fails with a `The pipeline activity limit was exceeded.` error. + +On [GitLab Premium](https://about.gitlab.com/pricing/) self-managed or +higher installations, this limit is defined under a `default` plan that affects all +projects. This limit is disabled (`0`) by default. GitLab SaaS subscribers have different +limits [defined per plan](../user/gitlab_com/index.md#gitlab-cicd), and they affect +all projects under that plan. + +To set this limit for a self-managed installation, enable the **Maximum number of active pipelines per project** +[setting in the Admin Area](../user/admin_area/settings/continuous_integration.md#set-cicd-limits). + +Alternatively, you can run the following in the [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): + +```ruby +# If limits don't exist for the default plan, you can create one with: +# Plan.default.create_limits! + +Plan.default.actual_limits.update!(ci_active_pipelines: 100) +``` + +Set the limit to `0` to disable it. + ### Maximum time jobs can run > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/16777) in GitLab 12.3. @@ -514,6 +555,26 @@ Plan.default.actual_limits.update!(ci_daily_pipeline_schedule_triggers: 1440) This limit is [enabled on GitLab.com](../user/gitlab_com/index.md#gitlab-cicd). +### Limit the number of schedule rules defined for security policy project + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/335659) in GitLab 15.1. + +You can limit the total number of schedule rules per security policy project. This limit is +checked each time policies with schedule rules are updated. If a new schedule rule would +cause the total number of schedule rules to exceed the limit, the new schedule rule is +not processed. + +By default, self-managed instances do not limit the number of processable schedule rules. + +To set this limit for a self-managed installation, run the following in the +[GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): + +```ruby +Plan.default.actual_limits.update!(security_policy_scan_execution_schedules: 100) +``` + +This limit is [enabled on GitLab.com](../user/gitlab_com/index.md#gitlab-cicd). + ### Number of instance level variables > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216097) in GitLab 13.1. @@ -646,7 +707,7 @@ Update `ci_jobs_trace_size_limit` with the new value in megabytes: Plan.default.actual_limits.update!(ci_jobs_trace_size_limit: 125) ``` -GitLab Runner also has an +GitLab Runner also has an [`output_limit` setting](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) that configures the maximum log size in a runner. Jobs that exceed the runner limit continue to run, but the log is truncated when it hits the limit. @@ -825,9 +886,9 @@ prevent any more changes from rendering. For more information about these limits Reports that go over the 20 MB limit aren't loaded. Affected reports: -- [Merge request security reports](../user/project/merge_requests/testing_and_reports_in_merge_requests.md#security-reports) +- [Merge request security reports](../ci/testing/index.md#security-reports) - [CI/CD parameter `artifacts:expose_as`](../ci/yaml/index.md#artifactsexpose_as) -- [Unit test reports](../ci/unit_test_reports.md) +- [Unit test reports](../ci/testing/unit_test_reports.md) ## Advanced Search limits @@ -864,7 +925,7 @@ indexed, which have a separate limit. For more information, read - For self-managed installations, the field length is unlimited by default. You can configure this limit for self-managed installations when you -[enable Elasticsearch](../integration/elasticsearch.md#enable-advanced-search). +[enable Elasticsearch](../integration/advanced_search/elasticsearch.md#enable-advanced-search). Set the limit to `0` to disable it. ## Wiki limits @@ -981,7 +1042,7 @@ varies by file type: If a branch is merged while open merge requests still point to it, GitLab can retarget merge requests pointing to the now-merged branch. To learn more, read -[Branch retargeting on merge](../user/project/merge_requests/getting_started.md#branch-retargeting-on-merge). +[Update merge requests when target branch merges](../user/project/merge_requests/index.md#update-merge-requests-when-target-branch-merges). ## CDN-based limits on GitLab.com @@ -999,3 +1060,11 @@ The [secure files API](../api/secure_files.md) enforces the following limits: - Files must be smaller than 5 MB. - Projects cannot have more than 100 secure files. + +## Changelog API limits + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/89032) in GitLab 15.1 [with a flag](../administration/feature_flags.md) named `changelog_commits_limitation`. Disabled by default. + +The [changelog API](../api/repositories.md#add-changelog-data-to-a-changelog-file) enforces the following limits: + +- The commit range between `from` and `to` cannot exceed 15000 commits. diff --git a/doc/administration/integration/kroki.md b/doc/administration/integration/kroki.md index 0f02e3783a5..a6fcfe6c80f 100644 --- a/doc/administration/integration/kroki.md +++ b/doc/administration/integration/kroki.md @@ -1,6 +1,6 @@ --- -stage: Ecosystem -group: Integrations +stage: Plan +group: Project Management info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/integration/mailgun.md b/doc/administration/integration/mailgun.md index 5a56aed4427..fbad201be7d 100644 --- a/doc/administration/integration/mailgun.md +++ b/doc/administration/integration/mailgun.md @@ -1,6 +1,6 @@ --- -stage: Growth -group: Expansion +stage: Plan +group: Project Management info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference, howto --- @@ -9,26 +9,37 @@ type: reference, howto When you use Mailgun to send emails for your GitLab instance and [Mailgun](https://www.mailgun.com/) integration is enabled and configured in GitLab, you can receive their webhook for -permanent invite email failures. To set up the integration, you must: +tracking delivery failures. To set up the integration, you must: 1. [Configure your Mailgun domain](#configure-your-mailgun-domain). 1. [Enable Mailgun integration](#enable-mailgun-integration). -After completing the integration, Mailgun `permanent_failure` webhooks are sent to your GitLab instance. +After completing the integration, Mailgun `temporary_failure` and `permanent_failure` webhooks are sent to your GitLab instance. ## Configure your Mailgun domain -Before you can enable Mailgun in GitLab, set up your own Mailgun permanent failure endpoint to receive the webhooks. +> [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/359113) the `/-/members/mailgun/permanent_failures` URL in GitLab 15.0. +> [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/359113) the URL to handle both temporary and permanent failures in GitLab 15.0. + +Before you can enable Mailgun in GitLab, set up your own Mailgun endpoints to receive the webhooks. Using the [Mailgun webhook guide](https://www.mailgun.com/blog/a-guide-to-using-mailguns-webhooks/): 1. Add a webhook with the **Event type** set to **Permanent Failure**. -1. Fill in the URL of your instance and include the `/-/members/mailgun/permanent_failures` path. - - Example: `https://myinstance.gitlab.com/-/members/mailgun/permanent_failures` +1. Enter the URL of your instance and include the `/-/mailgun/webhooks` path. + + For example: + + ```plaintext + https://myinstance.gitlab.com/-/mailgun/webhooks + ``` + +1. Add another webhook with the **Event type** set to **Temporary Failure**. +1. Enter the URL of your instance and use the same `/-/mailgun/webhooks` path. ## Enable Mailgun integration -After configuring your Mailgun domain for the permanent failures endpoint, +After configuring your Mailgun domain for the webhook endpoints, you're ready to enable the Mailgun integration: 1. Sign in to GitLab as an [Administrator](../../user/permissions.md) user. diff --git a/doc/administration/lfs/index.md b/doc/administration/lfs/index.md index 0d75880bdd1..ebf76cc6aec 100644 --- a/doc/administration/lfs/index.md +++ b/doc/administration/lfs/index.md @@ -24,7 +24,7 @@ There are various configuration options to help GitLab server administrators: - Enabling/disabling Git LFS support. - Changing the location of LFS object storage. -- Setting up object storage supported by [Fog](http://fog.io/about/provider_documentation.html). +- Setting up object storage supported by [Fog](https://fog.io/about/provider_documentation.html). ### Configuration for Omnibus installations @@ -57,7 +57,7 @@ In `config/gitlab.yml`: You can store LFS objects in remote object storage. This allows you to reduce reads and writes to the local disk, and free up disk space significantly. -GitLab is tightly integrated with `Fog`, so you can refer to its [documentation](http://fog.io/about/provider_documentation.html) +GitLab is tightly integrated with `Fog`, so you can refer to its [documentation](https://fog.io/about/provider_documentation.html) to check which storage services can be integrated with GitLab. You can also use external object storage in a private local network. For example, [MinIO](https://min.io/) is a standalone object storage service that works with GitLab instances. diff --git a/doc/administration/libravatar.md b/doc/administration/libravatar.md index 142cf2f3685..442c233a536 100644 --- a/doc/administration/libravatar.md +++ b/doc/administration/libravatar.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto diff --git a/doc/administration/load_balancer.md b/doc/administration/load_balancer.md index 45fd6e65078..e6a1c0c25d5 100644 --- a/doc/administration/load_balancer.md +++ b/doc/administration/load_balancer.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/logs.md b/doc/administration/logs.md index c89c485e7d3..5a2c1190896 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -61,7 +61,7 @@ Some of these services have their own environment variables to override the log | GitLab Export | `INFO` | `EXPORT_DEBUG` | | GitLab Geo | `INFO` | | | GitLab Import | `INFO` | `IMPORT_DEBUG` | -| GitLab QA Runtime | `ERROR` | `QA_DEBUG` | +| GitLab QA Runtime | `INFO` | `QA_LOG_LEVEL` | | Google APIs | `INFO` | | | Rack Timeout | `ERROR` | | | Sidekiq (server) | `INFO` | | @@ -144,7 +144,8 @@ Line breaks were added to examples for legibility: "db_duration_s":0.08, "view_duration_s":2.39, "duration_s":20.54, - "pid": 81836 + "pid": 81836, + "worker_id":"puma_0" } ``` @@ -167,7 +168,8 @@ seconds: - `redis_<instance>_duration_s`: Total time to retrieve data from a Redis instance - `redis_<instance>_read_bytes`: Total bytes read from a Redis instance - `redis_<instance>_write_bytes`: Total bytes written to a Redis instance -- `pid`: Process ID of the Puma worker +- `pid`: The worker's Linux process ID (changes when workers restart) +- `worker_id`: The worker's logical ID (does not change when workers restart) User clone and fetch activity using HTTP transport appears in the log as `action: git_upload_pack`. @@ -240,6 +242,7 @@ Starting with GitLab 12.5, if an error occurs, an "view_duration_s":2.39, "duration_s":20.54, "pid": 81836, + "worker_id": "puma_0", "exception.class": "NameError", "exception.message": "undefined local variable or method `adsf' for #<Admin::DashboardController:0x00007ff3c9648588>", "exception.backtrace": [ @@ -319,7 +322,10 @@ It helps you see requests made directly to the API. For example: "username":"root", "queue_duration":100.31, "gitaly_calls":30, - "gitaly_duration":5.36 + "gitaly_duration":5.36, + "pid": 81836, + "worker_id": "puma_0", + ... } ``` @@ -546,6 +552,7 @@ Sidekiq. For example: "created_at":"2018-04-03T22:57:21.930Z", "enqueued_at":"2018-04-03T22:57:21.931Z", "pid":10077, + "worker_id":"sidekiq_0", "message":"UpdateAllMirrorsWorker JID-06aeaa3b0aadacf9981f368e: done: 0.139 sec", "job_status":"done", "duration":0.139, diff --git a/doc/administration/maintenance_mode/index.md b/doc/administration/maintenance_mode/index.md index 41eed500594..9128f978dd5 100644 --- a/doc/administration/maintenance_mode/index.md +++ b/doc/administration/maintenance_mode/index.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -14,7 +14,7 @@ Once Maintenance Mode is enabled, in-progress actions finish relatively quickly In that state, various maintenance tasks are easier, and services can be stopped completely or be further degraded for a much shorter period of time than might otherwise be needed. For example, stopping cron jobs and draining queues should be fairly quick. -Maintenance Mode allows most external actions that do not change internal state. On a high-level, HTTP POST, PUT, PATCH, and DELETE requests are blocked and a detailed overview of [how special cases are handled](#rest-api) is available. +Maintenance Mode allows most external actions that do not change internal state. On a high-level, HTTP `POST`, `PUT`, `PATCH`, and `DELETE` requests are blocked and a detailed overview of [how special cases are handled](#rest-api) is available. ## Enable Maintenance Mode @@ -107,22 +107,22 @@ Notification emails continue to arrive, but emails that require database writes, ### REST API -For most JSON requests, POST, PUT, PATCH, and DELETE are blocked, and the API returns a 403 response with the error message: `You cannot perform write operations on a read-only instance`. Only the following requests are allowed: +For most JSON requests, `POST`, `PUT`, `PATCH`, and `DELETE` are blocked, and the API returns a 403 response with the error message: `You cannot perform write operations on a read-only instance`. Only the following requests are allowed: |HTTP request | Allowed routes | Notes | |:----:|:--------------------------------------:|:----:| -| POST | `/admin/application_settings/general` | To allow updating application settings in the administrator UI | -| PUT | `/api/v4/application/settings` | To allow updating application settings with the API | -| POST | `/users/sign_in` | To allow users to log in. | -| POST | `/users/sign_out`| To allow users to log out. | -| POST | `/oauth/token` | To allow users to log in to a Geo secondary for the first time. | -| POST | `/admin/session`, `/admin/session/destroy` | To allow [Admin Mode for GitLab administrators](https://gitlab.com/groups/gitlab-org/-/epics/2158) | -| POST | Paths ending with `/compare`| Git revision routes. | -| POST | `.git/git-upload-pack` | To allow Git pull/clone. | -| POST | `/api/v4/internal` | [internal API routes](../../development/internal_api/index.md) | -| POST | `/admin/sidekiq` | To allow management of background jobs in the Admin Area | -| POST | `/admin/geo` | To allow updating Geo Nodes in the administrator UI | -| POST | `/api/v4/geo_replication`| To allow certain Geo-specific administrator UI actions on secondary sites | +| `POST` | `/admin/application_settings/general` | To allow updating application settings in the administrator UI | +| `PUT` | `/api/v4/application/settings` | To allow updating application settings with the API | +| `POST` | `/users/sign_in` | To allow users to log in. | +| `POST` | `/users/sign_out`| To allow users to log out. | +| `POST` | `/oauth/token` | To allow users to log in to a Geo secondary for the first time. | +| `POST` | `/admin/session`, `/admin/session/destroy` | To allow [Admin Mode for GitLab administrators](https://gitlab.com/groups/gitlab-org/-/epics/2158) | +| `POST` | Paths ending with `/compare`| Git revision routes. | +| `POST` | `.git/git-upload-pack` | To allow Git pull/clone. | +| `POST` | `/api/v4/internal` | [internal API routes](../../development/internal_api/index.md) | +| `POST` | `/admin/sidekiq` | To allow management of background jobs in the Admin Area | +| `POST` | `/admin/geo` | To allow updating Geo Nodes in the administrator UI | +| `POST` | `/api/v4/geo_replication`| To allow certain Geo-specific administrator UI actions on secondary sites | ### GraphQL API diff --git a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md index 6f2e3cce0fa..2553638291d 100644 --- a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md +++ b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md @@ -54,7 +54,7 @@ you create the project again, it's created in its default state. 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, go to **Settings > Metrics and profiling** and expand **Self monitoring**. 1. Toggle **Self monitoring** off. -1. In the confirmation dialog that opens, click **Delete self monitoring project**. +1. In the confirmation dialog that opens, select **Delete self monitoring project**. It can take a few seconds for it to be deleted. 1. After the project is deleted, GitLab displays a message confirming your action. @@ -82,10 +82,10 @@ you [configure it manually](../../../user/project/integrations/prometheus.md#man ## Take action on Prometheus alerts **(ULTIMATE)** -You can [add a webhook](../../../operations/metrics/alerts.md#external-prometheus-instances) -to the Prometheus configuration for GitLab to receive notifications of any alerts. +You can [add a Prometheus integration](../../../operations/incident_management/integrations.md) +to GitLab to receive notifications of any alerts. -Once the webhook is setup, you can +Once the integration is setup, you can [take action on incoming alerts](../../../operations/metrics/alerts.md#trigger-actions-from-alerts). ## Add custom metrics to the self monitoring project diff --git a/doc/administration/monitoring/index.md b/doc/administration/monitoring/index.md index 381c807dbc5..82a6da1d56a 100644 --- a/doc/administration/monitoring/index.md +++ b/doc/administration/monitoring/index.md @@ -20,7 +20,7 @@ Explore our features to monitor your GitLab instance: importer with various Prometheus metrics. - [Monitoring uptime](../../user/admin_area/monitoring/health_check.md): Check the server status using the health check endpoint. - - [IP whitelists](ip_whitelist.md): Configure GitLab for monitoring endpoints that + - [IP allowlists](ip_whitelist.md): Configure GitLab for monitoring endpoints that provide health check information when probed. - [`nginx_status`](https://docs.gitlab.com/omnibus/settings/nginx.html#enablingdisabling-nginx_status): Monitor your NGINX server status. diff --git a/doc/administration/monitoring/ip_allowlist.md b/doc/administration/monitoring/ip_allowlist.md new file mode 100644 index 00000000000..adf9516733a --- /dev/null +++ b/doc/administration/monitoring/ip_allowlist.md @@ -0,0 +1,57 @@ +--- +stage: Data Stores +group: Memory +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# IP whitelist **(FREE SELF)** + +NOTE: +We intend to [rename IP whitelist as `IP allowlist`](https://gitlab.com/groups/gitlab-org/-/epics/3478). + +GitLab provides some [monitoring endpoints](../../user/admin_area/monitoring/health_check.md) +that provide health check information when probed. + +To control access to those endpoints via IP whitelisting, you can add single +hosts or use IP ranges: + +**For Omnibus installations** + +1. Open `/etc/gitlab/gitlab.rb` and add or uncomment the following: + + ```ruby + gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '192.168.0.1'] + ``` + +1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. + +--- + +**For installations using cloud native Helm charts** + +You can set the required IPs under the `gitlab.webservice.monitoring.ipWhitelist` key. For example: + +```yaml +gitlab: + webservice: + monitoring: + # Monitoring IP whitelist + ipWhitelist: + - 0.0.0.0/0 # Default +``` + +--- + +**For installations from source** + +1. Edit `config/gitlab.yml`: + + ```yaml + monitoring: + # by default only local IPs are allowed to access monitoring resources + ip_whitelist: + - 127.0.0.0/8 + - 192.168.0.1 + ``` + +1. Save the file and [restart](../restart_gitlab.md#installations-from-source) GitLab for the changes to take effect. diff --git a/doc/administration/monitoring/ip_whitelist.md b/doc/administration/monitoring/ip_whitelist.md index b8347ba3f0d..9fb4ffe3089 100644 --- a/doc/administration/monitoring/ip_whitelist.md +++ b/doc/administration/monitoring/ip_whitelist.md @@ -1,57 +1,11 @@ --- -stage: Monitor -group: Respond -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'ip_allowlist.md' +remove_date: '2022-08-31' --- -# IP whitelist **(FREE SELF)** +This document was moved to [another location](ip_allowlist.md). -NOTE: -We intend to [rename IP whitelist as `IP allowlist`](https://gitlab.com/groups/gitlab-org/-/epics/3478). - -GitLab provides some [monitoring endpoints](../../user/admin_area/monitoring/health_check.md) -that provide health check information when probed. - -To control access to those endpoints via IP whitelisting, you can add single -hosts or use IP ranges: - -**For Omnibus installations** - -1. Open `/etc/gitlab/gitlab.rb` and add or uncomment the following: - - ```ruby - gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '192.168.0.1'] - ``` - -1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. - ---- - -**For installations using cloud native Helm charts** - -You can set the required IPs under the `gitlab.webservice.monitoring.ipWhitelist` key. For example: - -```yaml -gitlab: - webservice: - monitoring: - # Monitoring IP whitelist - ipWhitelist: - - 0.0.0.0/0 # Default -``` - ---- - -**For installations from source** - -1. Edit `config/gitlab.yml`: - - ```yaml - monitoring: - # by default only local IPs are allowed to access monitoring resources - ip_whitelist: - - 127.0.0.0/8 - - 192.168.0.1 - ``` - -1. Save the file and [restart](../restart_gitlab.md#installations-from-source) GitLab for the changes to take effect. +<!-- This redirect file can be deleted after <2022-08-31>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index bc311f73dfe..aefd7a49b8b 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -21,7 +21,7 @@ From left to right, the performance bar displays: - **Current Host**: the current host serving the page. - **Database queries**: the time taken (in milliseconds) and the total number - of database queries, displayed in the format `00ms / 00 (00 cached) pg`. Click to display + of database queries, displayed in the format `00ms / 00 (00 cached) pg`. Select to display a modal window with more details. You can use this to see the following details for each query: - **In a transaction**: shows up below the query if it was executed in @@ -35,21 +35,21 @@ From left to right, the performance bar displays: GitLab features. The name shown is the same name used to configure database connections in GitLab. - **Gitaly calls**: the time taken (in milliseconds) and the total number of - [Gitaly](../../gitaly/index.md) calls. Click to display a modal window with more + [Gitaly](../../gitaly/index.md) calls. Select to display a modal window with more details. - **Rugged calls**: the time taken (in milliseconds) and the total number of [Rugged](../../nfs.md#improving-nfs-performance-with-gitlab) calls. - Click to display a modal window with more details. + Select to display a modal window with more details. - **Redis calls**: the time taken (in milliseconds) and the total number of - Redis calls. Click to display a modal window with more details. + Redis calls. Select to display a modal window with more details. - **Elasticsearch calls**: the time taken (in milliseconds) and the total number of - Elasticsearch calls. Click to display a modal window with more details. + Elasticsearch calls. Select to display a modal window with more details. - **External HTTP calls**: the time taken (in milliseconds) and the total - number of external calls to other systems. Click to display a modal window + number of external calls to other systems. Select to display a modal window with more details. - **Load timings** of the page: if your browser supports load timings, several values in milliseconds, separated by slashes. - Click to display a modal window with more details. The values, from left to right: + Select to display a modal window with more details. The values, from left to right: - **Backend**: time needed for the base page to load. - [**First Contentful Paint**](https://web.dev/first-contentful-paint/): Time until something was visible to the user. Displays `NaN` if your browser does not @@ -112,7 +112,7 @@ for a given group: 1. On the left sidebar, select **Settings > Metrics and profiling** (`admin/application_settings/metrics_and_profiling`), and expand **Profiling - Performance bar**. -1. Click **Allow non-administrators access to the performance bar**. +1. Select **Allow non-administrators access to the performance bar**. 1. In the **Allow access to members of the following group** field, provide the full path of the group allowed to access the performance. -1. Click **Save changes**. +1. Select **Save changes**. diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index 22f7419be9a..51360800d66 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -137,8 +137,8 @@ The steps below are the minimum necessary to configure a Monitoring node running # Enable service discovery for Prometheus consul['enable'] = true - consul['monitoring_service_discovery'] = true - consul['configuration'] = { + consul['monitoring_service_discovery'] = true + consul['configuration'] = { retry_join: %w(10.0.0.1 10.0.0.2 10.0.0.3), # The addresses can be IPs or FQDNs } @@ -205,7 +205,7 @@ To use an external Prometheus server: ``` 1. Install and set up a dedicated Prometheus instance, if necessary, using the [official installation instructions](https://prometheus.io/docs/prometheus/latest/installation/). -1. Add the Prometheus server IP address to the [monitoring IP whitelist](../ip_whitelist.md). For example: +1. Add the Prometheus server IP address to the [monitoring IP allowlist](../ip_whitelist.md). For example: ```ruby gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '192.168.0.1'] @@ -381,11 +381,12 @@ memory, disk, and CPU utilization. [Read more about the node exporter](node_exporter.md). -### Puma exporter +### Web exporter -The Puma exporter allows you to measure various Puma metrics. +The web exporter is a dedicated metrics server that allows splitting end-user and Prometheus traffic +into two separate applications to improve performance and availability. -[Read more about the Puma exporter](puma_exporter.md). +[Read more about the web exporter](puma_exporter.md). ### Redis exporter diff --git a/doc/administration/monitoring/prometheus/puma_exporter.md b/doc/administration/monitoring/prometheus/puma_exporter.md index 794e2c10b25..a3e095f5f09 100644 --- a/doc/administration/monitoring/prometheus/puma_exporter.md +++ b/doc/administration/monitoring/prometheus/puma_exporter.md @@ -1,29 +1,11 @@ --- -stage: Monitor -group: Respond -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'web_exporter.md' +remove_date: '2022-09-01' --- -# Puma exporter **(FREE SELF)** +This document was moved to [another location](web_exporter.md). -You can use the [Puma exporter](https://github.com/sapcc/puma-exporter) -to measure various Puma metrics. - -To enable the Puma exporter: - -1. [Enable Prometheus](index.md#configuring-prometheus). -1. Edit `/etc/gitlab/gitlab.rb` to add (or find and uncomment) the following lines. Make sure - `puma['exporter_enabled']` is set to `true`: - - ```ruby - puma['exporter_enabled'] = true - puma['exporter_address'] = "127.0.0.1" - puma['exporter_port'] = 8083 - ``` - -1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) - for the changes to take effect. - -Prometheus begins collecting performance data from the Puma exporter exposed at `localhost:8083`. - -For more information on using Puma with GitLab, see [Puma](../../operations/puma.md). +<!-- This redirect file can be deleted after <2022-09-01>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
\ No newline at end of file diff --git a/doc/administration/monitoring/prometheus/web_exporter.md b/doc/administration/monitoring/prometheus/web_exporter.md new file mode 100644 index 00000000000..4b449a1d74e --- /dev/null +++ b/doc/administration/monitoring/prometheus/web_exporter.md @@ -0,0 +1,53 @@ +--- +stage: Data Stores +group: Memory +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Web exporter (dedicated metrics server) **(FREE SELF)** + +When [monitoring GitLab with Prometheus](index.md), GitLab runs various collectors that +sample the application for data related to usage, load and performance. GitLab can then make +this data available to a Prometheus scraper by running one or more Prometheus exporters. +A Prometheus exporter is an HTTP server that serializes metric data into a format the +Prometheus scraper understands. + +NOTE: +This page is about web application metrics. +To export background job metrics, learn how to [configure the Sidekiq metrics server](../../sidekiq.md#configure-the-sidekiq-metrics-server). + +We provide two mechanisms by which web application metrics can be exported: + +- Through the main Rails application. This means [Puma](../../operations/puma.md), the application server we use, + makes metric data available via its own `/-/metrics` endpoint. This is the default, + and is described in [GitLab Metrics](index.md#gitlab-metrics). We recommend this + default for small GitLab installations where the amount of metrics collected is small. +- Through a dedicated metrics server. Enabling this server will cause Puma to launch an + additional process whose sole responsibility is to serve metrics. This approach leads + to better fault isolation and performance for very large GitLab installations, but + comes with additional memory use. We recommend this approach for medium to large + GitLab installations that seek high performance and availability. + +Both the dedicated server and the Rails `/-/metrics` endpoint serve the same data, so +they are functionally equivalent and differ merely in their performance characteristics. + +To enable the dedicated server: + +1. [Enable Prometheus](index.md#configuring-prometheus). +1. Edit `/etc/gitlab/gitlab.rb` to add (or find and uncomment) the following lines. Make sure + `puma['exporter_enabled']` is set to `true`: + + ```ruby + puma['exporter_enabled'] = true + puma['exporter_address'] = "127.0.0.1" + puma['exporter_port'] = 8083 + ``` + +1. When using the GitLab-bundled Prometheus, make sure that its `scrape_config` is pointing + to `localhost:8083/metrics`. Refer to the [Adding custom scrape configurations](index.md#adding-custom-scrape-configurations) page + for how to configure scraper targets. For external Prometheus setup, refer to + [Using an external Prometheus server](index.md#using-an-external-prometheus-server) instead. +1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect. + +Metrics can now be served and scraped from `localhost:8083/metrics`. diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md index b0c50d63e78..d9866a6c09f 100644 --- a/doc/administration/nfs.md +++ b/doc/administration/nfs.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index 0560a8813df..7e416ed560a 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -239,18 +239,19 @@ in the `connection` setting. The connection settings match those provided by [fog-aws](https://github.com/fog/fog-aws): -| Setting | Description | Default | -|---------------------------------|------------------------------------|---------| -| `provider` | Always `AWS` for compatible hosts. | `AWS` | -| `aws_access_key_id` | AWS credentials, or compatible. | | -| `aws_secret_access_key` | AWS credentials, or compatible. | | -| `aws_signature_version` | AWS signature version to use. `2` or `4` are valid options. Digital Ocean Spaces and other providers may need `2`. | `4` | -| `enable_signature_v4_streaming` | Set to `true` to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be `false`. | `true` | -| `region` | AWS region. | | -| `host` | S3 compatible host for when not using AWS. For example, `localhost` or `storage.example.com`. HTTPS and port 443 is assumed. | `s3.amazonaws.com` | -| `endpoint` | Can be used when configuring an S3 compatible service such as [MinIO](https://min.io), by entering a URL such as `http://127.0.0.1:9000`. This takes precedence over `host`. | (optional) | -| `path_style` | Set to `true` to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Leave as `false` for AWS S3. | `false`. | -| `use_iam_profile` | Set to `true` to use IAM profile instead of access keys. | `false` | +| Setting | Description | Default | +|---------------------------------------------|------------------------------------|---------| +| `provider` | Always `AWS` for compatible hosts. | `AWS` | +| `aws_access_key_id` | AWS credentials, or compatible. | | +| `aws_secret_access_key` | AWS credentials, or compatible. | | +| `aws_signature_version` | AWS signature version to use. `2` or `4` are valid options. Digital Ocean Spaces and other providers may need `2`. | `4` | +| `enable_signature_v4_streaming` | Set to `true` to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be `false`. | `true` | +| `region` | AWS region. | | +| `host` | S3 compatible host for when not using AWS. For example, `localhost` or `storage.example.com`. HTTPS and port 443 is assumed. | `s3.amazonaws.com` | +| `endpoint` | Can be used when configuring an S3 compatible service such as [MinIO](https://min.io), by entering a URL such as `http://127.0.0.1:9000`. This takes precedence over `host`. | (optional) | +| `path_style` | Set to `true` to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Leave as `false` for AWS S3. | `false`. | +| `use_iam_profile` | Set to `true` to use IAM profile instead of access keys. | `false` | +| `aws_credentials_refresh_threshold_seconds` | Sets the [automatic refresh threshold](https://github.com/fog/fog-aws#controlling-credential-refresh-time-with-iam-authentication) when using temporary credentials in IAM. | `15` | #### Oracle Cloud S3 connection settings @@ -540,6 +541,10 @@ supported by consolidated configuration form, refer to the following guides: | [Terraform state files](terraform_state.md#using-object-storage) | **{check-circle}** Yes | | [Pages content](pages/index.md#using-object-storage) | **{check-circle}** Yes | +WARNING: +The use of [encrypted S3 buckets](#encrypted-s3-buckets) with non-consolidated configuration is not supported. +You may start getting [ETag mismatch errors](#etag-mismatch) if you use it. + ### Other alternatives to file system storage If you're working to [scale out](reference_architectures/index.md) your GitLab implementation, @@ -765,7 +770,7 @@ Prerequisites: 1. [Install](https://rclone.org/downloads/) Rclone. 1. Configure Rclone by running the following: - + ```shell rclone config ``` @@ -778,7 +783,7 @@ Prerequisites: rclone ls old:uploads | head ``` - This should print a partial list of the objects currently stored in your `uploads` bucket. If you get an error, or if + This should print a partial list of the objects currently stored in your `uploads` bucket. If you get an error, or if the list is empty, go back and update your Rclone configuration using `rclone config`. 1. Perform an initial copy. You do not need to take your GitLab server offline for this step. @@ -788,7 +793,7 @@ Prerequisites: ``` 1. After the first sync completes, use the web UI or command-line interface of your new object storage provider to - verify that there are objects in the new bucket. If there are none, or if you encounter an error while running `rclone + verify that there are objects in the new bucket. If there are none, or if you encounter an error while running `rclone sync`, check your Rclone configuration and try again. After you have done at least one successful Rclone copy from the old location to the new location, schedule maintenance and take your GitLab server offline. During your maintenance window you must do two things: diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md index 75f2ad5ed26..cffe682a80b 100644 --- a/doc/administration/operations/extra_sidekiq_processes.md +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/operations/extra_sidekiq_routing.md b/doc/administration/operations/extra_sidekiq_routing.md index cd3a53b7c63..2a14fa1d312 100644 --- a/doc/administration/operations/extra_sidekiq_routing.md +++ b/doc/administration/operations/extra_sidekiq_routing.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -136,15 +136,15 @@ GitLab Enterprise Edition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee `queue_selector` supports the following operators, listed from highest to lowest precedence: -- `|` - the logical OR operator. For example, `query_a|query_b` (where `query_a` +- `|` - the logical `OR` operator. For example, `query_a|query_b` (where `query_a` and `query_b` are queries made up of the other operators here) will include queues that match either query. -- `&` - the logical AND operator. For example, `query_a&query_b` (where +- `&` - the logical `AND` operator. For example, `query_a&query_b` (where `query_a` and `query_b` are queries made up of the other operators here) will only include queues that match both queries. -- `!=` - the NOT IN operator. For example, `feature_category!=issue_tracking` +- `!=` - the `NOT IN` operator. For example, `feature_category!=issue_tracking` excludes all queues from the `issue_tracking` feature category. -- `=` - the IN operator. For example, `resource_boundary=cpu` includes all +- `=` - the `IN` operator. For example, `resource_boundary=cpu` includes all queues that are CPU bound. - `,` - the concatenate set operator. For example, `feature_category=continuous_integration,pages` includes all queues from @@ -152,8 +152,8 @@ to lowest precedence: example is also possible using the OR operator, but allows greater brevity, as well as being lower precedence. -The operator precedence for this syntax is fixed: it's not possible to make AND -have higher precedence than OR. +The operator precedence for this syntax is fixed: it's not possible to make `AND` +have higher precedence than `OR`. [In GitLab 12.9](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26594) and later, as with the standard queue group syntax above, a single `*` as the diff --git a/doc/administration/operations/filesystem_benchmarking.md b/doc/administration/operations/filesystem_benchmarking.md index a0ad2e24a4c..9403634d905 100644 --- a/doc/administration/operations/filesystem_benchmarking.md +++ b/doc/administration/operations/filesystem_benchmarking.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/operations/index.md b/doc/administration/operations/index.md index 7ccfa2739bb..fe531407eb2 100644 --- a/doc/administration/operations/index.md +++ b/doc/administration/operations/index.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/operations/moving_repositories.md b/doc/administration/operations/moving_repositories.md index f0eb5792a96..1459b707e5c 100644 --- a/doc/administration/operations/moving_repositories.md +++ b/doc/administration/operations/moving_repositories.md @@ -27,7 +27,7 @@ For more information, see: querying and scheduling snippet repository moves. - [The API documentation](../../api/group_repository_storage_moves.md) details the endpoints for querying and scheduling group repository moves **(PREMIUM SELF)**. -- [Migrating to Gitaly Cluster](../gitaly/index.md#migrating-to-gitaly-cluster). +- [Migrate to Gitaly Cluster](../gitaly/index.md#migrate-to-gitaly-cluster). ### Move Repositories @@ -276,7 +276,7 @@ If the source or target directory has many contents, this startup phase of `rsyn server. You can reduce the workload of `rsync` by dividing its work into smaller pieces, and sync one repository at a time. -In addition to `rsync` we use [GNU Parallel](http://www.gnu.org/software/parallel/). +In addition to `rsync` we use [GNU Parallel](https://www.gnu.org/software/parallel/). This utility is not included in GitLab, so you must install it yourself with `apt` or `yum`. @@ -377,3 +377,14 @@ sudo -u git -H bundle exec rake gitlab:list_repos SINCE='2015-10-1 12:00 UTC' |\ /home/git/repositories \ /mnt/gitlab/repositories ``` + +## Troubleshooting + +See the following for information on troubleshooting repository moves. + +### Repository move fails for archived projects + +Because of a [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/363670), +[archived projects](../../user/project/settings/index.md#advanced-settings) fail to move even though the data is cloned +by Gitaly. Make sure archived projects are +[unarchived](../../user/project/settings/index.md#unarchiving-a-project) before initiating a move. diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md index 12a8b2faadc..0025ca109e5 100644 --- a/doc/administration/operations/puma.md +++ b/doc/administration/operations/puma.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -252,19 +252,19 @@ To switch from Unicorn to Puma: | `unicorn['enable']` | `puma['enable']` | | `unicorn['worker_timeout']` | `puma['worker_timeout']` | | `unicorn['worker_processes']` | `puma['worker_processes']` | - | n/a | `puma['ha']` | - | n/a | `puma['min_threads']` | - | n/a | `puma['max_threads']` | + | Not applicable | `puma['ha']` | + | Not applicable | `puma['min_threads']` | + | Not applicable | `puma['max_threads']` | | `unicorn['listen']` | `puma['listen']` | | `unicorn['port']` | `puma['port']` | | `unicorn['socket']` | `puma['socket']` | | `unicorn['pidfile']` | `puma['pidfile']` | - | `unicorn['tcp_nopush']` | n/a | - | `unicorn['backlog_socket']` | n/a | + | `unicorn['tcp_nopush']` | Not applicable | + | `unicorn['backlog_socket']` | Not applicable | | `unicorn['somaxconn']` | `puma['somaxconn']` | - | n/a | `puma['state_path']` | + | Not applicable | `puma['state_path']` | | `unicorn['log_directory']` | `puma['log_directory']` | - | `unicorn['worker_memory_limit_min']` | n/a | + | `unicorn['worker_memory_limit_min']` | Not applicable | | `unicorn['worker_memory_limit_max']` | `puma['per_worker_max_memory_mb']` | | `unicorn['exporter_enabled']` | `puma['exporter_enabled']` | | `unicorn['exporter_address']` | `puma['exporter_address']` | @@ -281,4 +281,4 @@ To switch from Unicorn to Puma: ## Related topics -- [Use the Puma exporter to measure various Puma metrics](../monitoring/prometheus/puma_exporter.md) +- [Use a dedicated metrics server to export web metrics](../monitoring/prometheus/puma_exporter.md) diff --git a/doc/administration/operations/rails_console.md b/doc/administration/operations/rails_console.md index a93365c08b2..df039ee734f 100644 --- a/doc/administration/operations/rails_console.md +++ b/doc/administration/operations/rails_console.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/operations/sidekiq_memory_killer.md b/doc/administration/operations/sidekiq_memory_killer.md index e48ac6e65eb..d9558c3d7a6 100644 --- a/doc/administration/operations/sidekiq_memory_killer.md +++ b/doc/administration/operations/sidekiq_memory_killer.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Memory info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/operations/ssh_certificates.md b/doc/administration/operations/ssh_certificates.md index 4fcc5d7c916..3141312612e 100644 --- a/doc/administration/operations/ssh_certificates.md +++ b/doc/administration/operations/ssh_certificates.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -53,7 +53,7 @@ the GitLab server itself, but your setup may vary. If the CA is only used for GitLab consider putting this in the `Match User git` section (described below). -The SSH certificates being issued by that CA **MUST** have a "key ID" +The SSH certificates being issued by that CA **must** have a "key ID" corresponding to that user's username on GitLab, for example (some output omitted for brevity): diff --git a/doc/administration/package_information/defaults.md b/doc/administration/package_information/defaults.md index 54e68392aa5..65cb8ef1e6c 100644 --- a/doc/administration/package_information/defaults.md +++ b/doc/administration/package_information/defaults.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -31,7 +31,7 @@ by default: | GitLab Exporter | Yes | Port | X | 9168 | | Sidekiq exporter | Yes | Port | X | 8082 | | Sidekiq health check | No | Port | X | 8092[^Sidekiq-health] | -| Puma exporter | No | Port | X | 8083 | +| Web exporter | No | Port | X | 8083 | | Geo PostgreSQL | No | Socket | Port (5431) | X | | Redis Sentinel | No | Port | X | 26379 | | Incoming email | No | Port | X | 143 | diff --git a/doc/administration/package_information/deprecation_policy.md b/doc/administration/package_information/deprecation_policy.md index 7298bce6c95..420690866e5 100644 --- a/doc/administration/package_information/deprecation_policy.md +++ b/doc/administration/package_information/deprecation_policy.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/package_information/index.md b/doc/administration/package_information/index.md index c908f526569..746961e8613 100644 --- a/doc/administration/package_information/index.md +++ b/doc/administration/package_information/index.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -14,11 +14,11 @@ at [bundling dependencies document](omnibus_packages.md). The released package versions are in the format `MAJOR.MINOR.PATCH-EDITION.OMNIBUS_RELEASE` -| Component | Meaning | Example | -|-------------------|---------|---------| -| MAJOR.MINOR.PATCH | The GitLab version this corresponds to. | 13.3.0 | -| EDITION | The edition of GitLab this corresponds to. | ee | -| OMNIBUS_RELEASE | The Omnibus GitLab release. Usually, this is 0. This is incremented if we need to build a new package without changing the GitLab version. | 0 | +| Component | Meaning | Example | +|---------------------|---------|---------| +| `MAJOR.MINOR.PATCH` | The GitLab version this corresponds to. | 13.3.0 | +| `EDITION` | The edition of GitLab this corresponds to. | ee | +| `OMNIBUS_RELEASE` | The Omnibus GitLab release. Usually, this is 0. This is incremented if we need to build a new package without changing the GitLab version. | 0 | ## Licenses @@ -87,7 +87,7 @@ Depending on the init system, this `WARNING` can be one of: /sbin/init: unrecognized option '--version' ``` -when the underlying init system *IS NOT* upstart. +when the underlying init system *is not* upstart. ```plaintext -.mount loaded active mounted / diff --git a/doc/administration/package_information/licensing.md b/doc/administration/package_information/licensing.md index d27c1df0ccf..e5afe6b85dd 100644 --- a/doc/administration/package_information/licensing.md +++ b/doc/administration/package_information/licensing.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/package_information/omnibus_packages.md b/doc/administration/package_information/omnibus_packages.md index 21e3f5711ba..d19f5f67b74 100644 --- a/doc/administration/package_information/omnibus_packages.md +++ b/doc/administration/package_information/omnibus_packages.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/package_information/postgresql_versions.md b/doc/administration/package_information/postgresql_versions.md index 1b3e49cfb82..8aab4121a0b 100644 --- a/doc/administration/package_information/postgresql_versions.md +++ b/doc/administration/package_information/postgresql_versions.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -21,6 +21,9 @@ For example: [Find out which versions of PostgreSQL (and other components) ship with each Omnibus GitLab release](https://gitlab-org.gitlab.io/omnibus-gitlab/licenses.html). +The lowest supported PostgreSQL versions are listed in the +[installation requirements](../../install/requirements.md#postgresql-requirements). + Read more about update policies and warnings in the PostgreSQL [upgrade docs](https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server). diff --git a/doc/administration/package_information/signed_packages.md b/doc/administration/package_information/signed_packages.md index 24857dcfc27..351c2ac5d5f 100644 --- a/doc/administration/package_information/signed_packages.md +++ b/doc/administration/package_information/signed_packages.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/package_information/supported_os.md b/doc/administration/package_information/supported_os.md index 647620858d9..f2838cd779b 100644 --- a/doc/administration/package_information/supported_os.md +++ b/doc/administration/package_information/supported_os.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -15,21 +15,23 @@ page](https://about.gitlab.com/install/). The following lists the currently supported OSs and their possible EOL dates. -| OS Version | First supported GitLab version | Arch | OS EOL | Details | -| ---------------- | ------------------------------ | --------------- | ------------- | ------------------------------------------------------------ | -| AlmaLinux 8 | GitLab CE / GitLab EE 14.5.0 | x86_64, aarch64 | 2029 | <https://almalinux.org/> | -| CentOS 7 | GitLab CE / GitLab EE 7.10.0 | x86_64 | June 2024 | <https://wiki.centos.org/About/Product> | -| CentOS 8 | GitLab CE / GitLab EE 12.8.1 | x86_64, aarch64 | Dec 2021 | <https://wiki.centos.org/About/Product> | -| Debian 9 | GitLab CE / GitLab EE 9.3.0 | amd64 | 2022 | <https://wiki.debian.org/LTS> | -| Debian 10 | GitLab CE / GitLab EE 12.2.0 | amd64, arm64 | 2024 | <https://wiki.debian.org/LTS> | -| Debian 11 | GitLab CE / GitLab EE 14.6.0 | amd64, arm64 | 2026 | <https://wiki.debian.org/LTS> | -| OpenSUSE 15.3 | GitLab CE / GitLab EE 14.5.0 | x86_64, aarch64 | Nov 2022 | <https://en.opensuse.org/Lifetime> | -| RHEL 8 | GitLab CE / GitLab EE 12.8.1 | x86_64, arm64 | May 2024 | <https://access.redhat.com/support/policy/updates/errata/#Life_Cycle_Dates> | -| SLES 12 | GitLab EE 9.0.0 | x86_64 | Oct 2027 | <https://www.suse.com/lifecycle/> | -| Ubuntu 18.04 | GitLab CE / GitLab EE 10.7.0 | amd64 | April 2023 | <https://wiki.ubuntu.com/Releases> | -| Ubuntu 20.04 | GitLab CE / GitLab EE 13.2.0 | amd64, arm64 | April 2025 | <https://wiki.ubuntu.com/Releases> | -| Amazon Linux 2 | GitLab CE / GitLab EE 14.9.0 | amd64, arm64 | June 2023 | <https://aws.amazon.com/amazon-linux-2/faqs/> | -| Raspberry Pi OS (Buster) (formerly known as Raspbian Buster) | GitLab CE 12.2.0 | armhf | 2024 | <https://www.raspberrypi.com/news/new-old-functionality-with-raspberry-pi-os-legacy/> | +| OS Version | First supported GitLab version | Arch | Install Docs | OS EOL | Details | +| ------------------------------------------------------------ | ------------------------------ | --------------- | :----------------------------------------------------------: | ---------- | ------------------------------------------------------------ | +| AlmaLinux 8 | GitLab CE / GitLab EE 14.5.0 | x86_64, aarch64 | [AlmaLinux Install Docs](https://about.gitlab.com/install/#almalinux-8) | 2029 | <https://almalinux.org/> | +| CentOS 7 | GitLab CE / GitLab EE 7.10.0 | x86_64 | [CentOS Install Docs](https://about.gitlab.com/install/#centos-7) | June 2024 | <https://wiki.centos.org/About/Product> | +| Debian 9 | GitLab CE / GitLab EE 9.3.0 | amd64 | [Debian Install Docs](https://about.gitlab.com/install/#debian) | 2022 | <https://wiki.debian.org/LTS> | +| Debian 10 | GitLab CE / GitLab EE 12.2.0 | amd64, arm64 | [Debian Install Docs](https://about.gitlab.com/install/#debian) | 2024 | <https://wiki.debian.org/LTS> | +| Debian 11 | GitLab CE / GitLab EE 14.6.0 | amd64, arm64 | [Debian Install Docs](https://about.gitlab.com/install/#debian) | 2026 | <https://wiki.debian.org/LTS> | +| OpenSUSE 15.3 | GitLab CE / GitLab EE 14.5.0 | x86_64, aarch64 | [OpenSUSE Install Docs](https://about.gitlab.com/install/#opensuse-leap-15-3) | Nov 2022 | <https://en.opensuse.org/Lifetime> | +| RHEL 8 | GitLab CE / GitLab EE 12.8.1 | x86_64, arm64 | [Use CentOS Install Docs](https://about.gitlab.com/install/#centos-7) | May 2024 | [RHEL Details](https://access.redhat.com/support/policy/updates/errata/#Life_Cycle_Dates) | +| SLES 12 | GitLab EE 9.0.0 | x86_64 | [Use OpenSUSE Install Docs](https://about.gitlab.com/install/#opensuse-leap-15-3) | Oct 2027 | <https://www.suse.com/lifecycle/> | + +| Oracle Linux | GitLab CE / GitLab EE 8.14.0 | x86_64 | [Use CentOS Install Docs](https://about.gitlab.com/install/#centos-7) | Jul 2024 | <https://www.oracle.com/a/ocom/docs/elsp-lifetime-069338.pdf> | +| Scientific Linux | GitLab CE / GitLab EE 8.14.0 | x86_64 | [Use CentOS Install Docs](https://about.gitlab.com/install/#centos-7) | June 2024 | <https://scientificlinux.org/downloads/sl-versions/sl7/> | +| Ubuntu 18.04 | GitLab CE / GitLab EE 10.7.0 | amd64 | [Ubuntu Install Docs](https://about.gitlab.com/install/#ubuntu) | April 2023 | <https://wiki.ubuntu.com/Releases> | +| Ubuntu 20.04 | GitLab CE / GitLab EE 13.2.0 | amd64, arm64 | [Ubuntu Install Docs](https://about.gitlab.com/install/#ubuntu) | April 2025 | <https://wiki.ubuntu.com/Releases> | +| Amazon Linux 2 | GitLab CE / GitLab EE 14.9.0 | amd64, arm64 | [Amazon Linux 2 Instal Docsl](https://about.gitlab.com/install/#amazonlinux-2) | June 2023 | <https://aws.amazon.com/amazon-linux-2/faqs/> | +| Raspberry Pi OS (Buster) (formerly known as Raspbian Buster) | GitLab CE 12.2.0 | armhf | [Raspberry Pi Install Docs](https://about.gitlab.com/install/#raspberry-pi-os) | 2024 | [Raspberry Pi Details](https://www.raspberrypi.com/news/new-old-functionality-with-raspberry-pi-os-legacy/) | NOTE: CentOS 8 was EOL on December 31, 2021. In GitLab 14.5 and later, @@ -82,6 +84,7 @@ release for them can be found below: | Raspbian Stretch | [June 2020](https://downloads.raspberrypi.org/raspbian/images/raspbian-2019-04-09/) | [GitLab CE](https://packages.gitlab.com/app/gitlab/raspberry-pi2/search?q=gitlab-ce_13.3&dist=raspbian%2Fstretch) 13.3 | | Debian Jessie | [June 2020](https://www.debian.org/News/2020/20200709) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_13.2&dist=debian%2Fjessie) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_13.2&dist=debian%2Fjessie) 13.3 | | CentOS 6 | [November 2020](https://wiki.centos.org/About/Product) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=13.6&filter=all&filter=all&dist=el%2F6) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=13.6&filter=all&filter=all&dist=el%2F6) 13.6 | +| CentOS 8 | [December 2021](https://wiki.centos.org/About/Product) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=14.6&filter=all&filter=all&dist=el%2F8) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=14.6&filter=all&filter=all&dist=el%2F8) 14.6 | | OpenSUSE 15.1 | [November 2020](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-13.12&dist=opensuse%2F15.1) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-13.12&dist=opensuse%2F15.1) 13.12 | | Ubuntu 16.04 | [April 2021](https://ubuntu.com/info/release-end-of-life) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_13.12&dist=ubuntu%2Fxenial) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_13.12&dist=ubuntu%2Fxenial) 13.12 | | OpenSUSE 15.2 | [December 2021](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-14.7&dist=opensuse%2F15.2) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-14.7&dist=opensuse%2F15.2) 14.7 | diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index 0edad83cc18..2fa3eea54bd 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -317,6 +317,17 @@ the Container Registry by themselves, follow the steps below. 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. +### Increase token duration + +In GitLab, tokens for the Container Registry expire every five minutes. +To increase the token duration: + +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **Container Registry**. +1. For the **Authorization token duration (minutes)**, update the value. +1. Select **Save changes**. + ## Configure storage for the Container Registry NOTE: @@ -477,7 +488,7 @@ To configure the `s3` storage driver in Omnibus: **Installations from source** -Configuring the storage driver is done in the registry configuration YML file created +Configuring the storage driver is done in the registry configuration YAML file created when you [deployed your Docker registry](https://docs.docker.com/registry/deploying/). `s3` storage driver example: @@ -600,7 +611,7 @@ However, this behavior is undesirable for registries used by internal hosts that **Installations from source** -1. Add the `redirect` flag to your registry configuration YML file: +1. Add the `redirect` flag to your registry configuration YAML file: ```yaml storage: @@ -652,7 +663,7 @@ For Omnibus GitLab installations: For installations from source: -1. Edit your registry configuration YML file: +1. Edit your registry configuration YAML file: ```yaml storage: @@ -705,7 +716,7 @@ In the examples below we set the Registry's port to `5010`. ## Disable Container Registry per project If Registry is enabled in your GitLab instance, but you don't need it for your -project, you can [disable it from your project's settings](../../user/project/settings/index.md#sharing-and-permissions). +project, you can [disable it from your project's settings](../../user/project/settings/index.md#configure-project-visibility-features-and-permissions). ## Use an external container registry with GitLab as an auth endpoint @@ -845,7 +856,7 @@ To configure a notification endpoint in Omnibus: **Installations from source** -Configuring the notification endpoint is done in your registry configuration YML file created +Configuring the notification endpoint is done in your registry configuration YAML file created when you [deployed your Docker registry](https://docs.docker.com/registry/deploying/). Example: @@ -1140,7 +1151,7 @@ for GitLab to run separately from Registry: - `gitlab_rails['registry_enabled']`, must be set to `true`. This setting signals to GitLab that it should allow Registry API requests. - `gitlab_rails['registry_api_url']`, default [set programmatically](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/gitlab/libraries/registry.rb#L52). This is the Registry URL used internally that users do not need to interact with, `registry['registry_http_addr']` with scheme. -- `gitlab_rails['registry_host']`, eg. `registry.gitlab.example`. Registry endpoint without the scheme, the address that gets shown to the end user. +- `gitlab_rails['registry_host']`, for example, `registry.gitlab.example`. Registry endpoint without the scheme, the address that gets shown to the end user. - `gitlab_rails['registry_port']`. Registry endpoint port, visible to the end user. - `gitlab_rails['registry_issuer']` must match the issuer in the Registry configuration. - `gitlab_rails['registry_key_path']`, path to the key that matches the certificate on the @@ -1159,7 +1170,7 @@ The flow described by the diagram above: 1. A user runs `docker login registry.gitlab.example` on their client. This reaches the web server (or LB) on port 443. 1. Web server connects to the Registry backend pool (by default, using port 5000). Since the user - didn’t provide a valid token, the Registry returns a 401 HTTP code and the URL (`token_realm` from + didn't provide a valid token, the Registry returns a 401 HTTP code and the URL (`token_realm` from Registry configuration) where to get one. This points to the GitLab API. 1. The Docker client then connects to the GitLab API and obtains a token. 1. The API signs the token with the registry key and hands it to the Docker client @@ -1169,7 +1180,7 @@ Reference: <https://docs.docker.com/registry/spec/auth/token/> ### Communication between GitLab and Registry -Registry doesn’t have a way to authenticate users internally so it relies on +Registry doesn't have a way to authenticate users internally so it relies on GitLab to validate credentials. The connection between Registry and GitLab is TLS encrypted. The key is used by GitLab to sign the tokens while the certificate is used by Registry to validate the signature. By default, a self-signed certificate key pair is generated @@ -1226,29 +1237,6 @@ mounting the Docker daemon and setting `privileged = false` in the GitLab Runner Additional information about this: [issue 18239](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/18239). -### `unauthorized: authentication required` when pushing large images - -Example error: - -```shell -docker push gitlab.example.com/myproject/docs:latest -The push refers to a repository [gitlab.example.com/myproject/docs] -630816f32edb: Preparing -530d5553aec8: Preparing -... -4b0bab9ff599: Waiting -d1c800db26c7: Waiting -42755cf4ee95: Waiting -unauthorized: authentication required -``` - -GitLab has a default token expiration of 5 minutes for the registry. When pushing -larger images, or images that take longer than 5 minutes to push, users may -encounter this error. On GitLab.com, the expiration time is 15 minutes. - -Administrators can increase the token duration in **Admin Area > Settings > -CI/CD > Container Registry > Authorization token duration (minutes)**. - ### Docker login attempt fails with: 'token signed by untrusted key' [Registry relies on GitLab to validate credentials](#architecture-of-gitlab-container-registry) @@ -1309,7 +1297,10 @@ openssl rsa -noout -modulus -in /var/opt/gitlab/gitlab-rails/etc/gitlab-registry ``` If the two pieces of the certificate do not align, remove the files and run `gitlab-ctl reconfigure` -to regenerate the pair. If you have overridden the automatically generated self-signed pair with +to regenerate the pair. The pair is recreated using the existing values in `/etc/gitlab/gitlab-secrets.json` if they exist. To generate a new pair, +delete the `registry` section in your `/etc/gitlab/gitlab-secrets.json` before running `gitlab-ctl reconfigure`. + +If you have overridden the automatically generated self-signed pair with your own certificates and have made sure that their contents align, you can delete the 'registry' section in your `/etc/gitlab/gitlab-secrets.json` and run `gitlab-ctl reconfigure`. @@ -1378,7 +1369,7 @@ You can add a configuration option for backwards compatibility. **For installations from source** -1. Edit the YML configuration file you created when you [deployed the registry](https://docs.docker.com/registry/deploying/). Add the following snippet: +1. Edit the YAML configuration file you created when you [deployed the registry](https://docs.docker.com/registry/deploying/). Add the following snippet: ```yaml compatibility: @@ -1426,7 +1417,7 @@ and a simple solution would be to enable relative URLs in the Registry. **For installations from source** -1. Edit the YML configuration file you created when you [deployed the registry](https://docs.docker.com/registry/deploying/). Add the following snippet: +1. Edit the YAML configuration file you created when you [deployed the registry](https://docs.docker.com/registry/deploying/). Add the following snippet: ```yaml http: @@ -1741,7 +1732,7 @@ In this case, follow these steps: ```ruby gitlab_rails['registry_enabled'] = true ``` - + 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Try the removal again. diff --git a/doc/administration/packages/index.md b/doc/administration/packages/index.md index ef00127a70e..56786f334b0 100644 --- a/doc/administration/packages/index.md +++ b/doc/administration/packages/index.md @@ -26,7 +26,7 @@ The Packages feature allows GitLab to act as a repository and supports the follo ## Accepting contributions -The below table lists formats that are not supported, but are accepting Community contributions for. Consider contributing to GitLab. This [development documentation](../../development/packages.md) +The below table lists formats that are not supported, but are accepting Community contributions for. Consider contributing to GitLab. This [development documentation](../../development/packages/index.md) guides you through the process. <!-- vale gitlab.Spelling = NO --> diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index f144c60fcfe..1535de0c2c2 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -83,7 +83,7 @@ added `gitlab.io` [in 2016](https://gitlab.com/gitlab-com/infrastructure/-/issue ### DNS configuration GitLab Pages expect to run on their own virtual host. In your DNS server/provider -add a [wildcard DNS A record](https://en.wikipedia.org/wiki/Wildcard_DNS_record) pointing to the +add a [wildcard DNS `A` record](https://en.wikipedia.org/wiki/Wildcard_DNS_record) pointing to the host that GitLab runs. For example, an entry would look like this: ```plaintext @@ -257,7 +257,6 @@ control over how the Pages daemon runs and serves content in your environment. | `auth_scope` | The OAuth application scope to use for authentication. Must match GitLab Pages OAuth application settings. Leave blank to use `api` scope by default. | | `gitlab_server` | Server to use for authentication when access control is enabled; defaults to GitLab `external_url`. | | `headers` | Specify any additional http headers that should be sent to the client with each response. Multiple headers can be given as an array, header and value as one string, for example `['my-header: myvalue', 'my-other-header: my-other-value']` | -| `inplace_chroot` | [REMOVED in GitLab 14.3.](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/561) On [systems that don't support bind-mounts](index.md#gitlab-pages-fails-to-start-in-docker-container), this instructs GitLab Pages to `chroot` into its `pages_path` directory. Some caveats exist when using in-place `chroot`; refer to the GitLab Pages [README](https://gitlab.com/gitlab-org/gitlab-pages/blob/master/README.md#caveats) for more information. | | `enable_disk` | Allows the GitLab Pages daemon to serve content from disk. Shall be disabled if shared disk storage isn't available. | | `insecure_ciphers` | Use default list of cipher suites, may contain insecure ones like 3DES and RC4. | | `internal_gitlab_server` | Internal GitLab server address used exclusively for API requests. Useful if you want to send that traffic over an internal load balancer. Defaults to GitLab `external_url`. | @@ -681,35 +680,44 @@ Follow the steps below to configure the proxy listener of GitLab Pages. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -## Set global maximum pages size per project **(FREE SELF)** +## Set global maximum size of each GitLab Pages site **(FREE SELF)** + +Prerequisites: + +- Only GitLab administrators can edit this setting. To set the global maximum pages size for a project: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Preferences**. 1. Expand **Pages**. -1. Edit the **Maximum size of pages**. +1. Enter a value under **Maximum size of pages**. 1. Select **Save changes**. -## Override maximum pages size per project or group **(PREMIUM SELF)** +## Set maximum size of each GitLab Pages site in a group **(PREMIUM SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16610) in GitLab 12.7. - -NOTE: -Only GitLab administrators are able to view and override the **Maximum size of Pages** setting. +Prerequisites: -To override the global maximum pages size for a specific project: +- You must have at least the Maintainer role for the group. -1. On the top bar, select **Menu > Projects** and find your project. -1. On the left sidebar, select **Settings > Pages**. -1. Enter a value under **Maximum size of pages** in MB. -1. Select **Save changes**. - -To override the global maximum pages size for a specific group: +To set the maximum size of each GitLab Pages site in a group, overriding the inherited setting: 1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **Settings > General**. 1. Expand **Pages**. +1. Enter a value under **Maximum size** in MB. +1. Select **Save changes**. + +## Set maximum size of GitLab Pages site in a project **(PREMIUM SELF)** + +Prerequisites: + +- You must have at least the Maintainer role for the project. + +To set the maximum size of GitLab Pages site in a project, overriding the inherited setting: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Pages**. 1. Enter a value under **Maximum size of pages** in MB. 1. Select **Save changes**. @@ -1187,7 +1195,8 @@ Rate limits are enforced using the following: Include any troubleshooting steps that you can foresee. If you know beforehand what issues one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. +important to describe those, too. Think of things that may go wrong and include them in +the section below. This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. @@ -1207,72 +1216,6 @@ sudo gitlab-ctl tail gitlab-pages You can also find the log file in `/var/log/gitlab/gitlab-pages/current`. -### `open /etc/ssl/ca-bundle.pem: permission denied` - -WARNING: -This issue is fixed in GitLab 14.3 and above, try upgrading GitLab first. - -GitLab Pages runs inside a `chroot` jail, usually in a uniquely numbered directory like -`/tmp/gitlab-pages-*`. - -In the jail, a bundle of trusted certificates is -provided at `/etc/ssl/ca-bundle.pem`. It's -[copied there](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/51) -from `/opt/gitlab/embedded/ssl/certs/cacert.pem` -as part of starting up Pages. - -If the permissions on the source file are incorrect (they should be `0644`), then -the file inside the `chroot` jail is also wrong. - -Pages logs errors in `/var/log/gitlab/gitlab-pages/current` like: - -```plaintext -x509: failed to load system roots and no roots provided -open /etc/ssl/ca-bundle.pem: permission denied -``` - -The use of a `chroot` jail makes this error misleading, as it is not -referring to `/etc/ssl` on the root file system. - -The fix is to correct the source file permissions and restart Pages: - -```shell -sudo chmod 644 /opt/gitlab/embedded/ssl/certs/cacert.pem -sudo gitlab-ctl restart gitlab-pages -``` - -### `dial tcp: lookup gitlab.example.com` and `x509: certificate signed by unknown authority` - -WARNING: -This issue is fixed in GitLab 14.3 and above, try upgrading GitLab first. - -When setting both `inplace_chroot` and `access_control` to `true`, you might encounter errors like: - -```plaintext -dial tcp: lookup gitlab.example.com on [::1]:53: dial udp [::1]:53: connect: cannot assign requested address -``` - -Or: - -```plaintext -open /opt/gitlab/embedded/ssl/certs/cacert.pem: no such file or directory -x509: certificate signed by unknown authority -``` - -The reason for those errors is that the files `resolv.conf`, `/etc/hosts/`, `/etc/nsswitch.conf` and `ca-bundle.pem` are missing inside the `chroot`. -The fix is to copy these files inside the `chroot`: - -```shell -sudo mkdir -p /var/opt/gitlab/gitlab-rails/shared/pages/etc/ssl -sudo mkdir -p /var/opt/gitlab/gitlab-rails/shared/pages/opt/gitlab/embedded/ssl/certs/ - -sudo cp /etc/resolv.conf /var/opt/gitlab/gitlab-rails/shared/pages/etc/ -sudo cp /etc/hosts /var/opt/gitlab/gitlab-rails/shared/pages/etc/ -sudo cp /etc/nsswitch.conf /var/opt/gitlab/gitlab-rails/shared/pages/etc/ -sudo cp /opt/gitlab/embedded/ssl/certs/cacert.pem /var/opt/gitlab/gitlab-rails/shared/pages/opt/gitlab/embedded/ssl/certs/ -sudo cp /opt/gitlab/embedded/ssl/certs/cacert.pem /var/opt/gitlab/gitlab-rails/shared/pages/etc/ssl/ca-bundle.pem -``` - ### `unsupported protocol scheme \"\""` If you see the following error: @@ -1541,7 +1484,7 @@ In GitLab 14.0-14.2 you can temporarily enable legacy storage and configuration To do that: -1. Please describe the issue you're seeing in [here](https://gitlab.com/gitlab-org/gitlab/-/issues/331699). +1. Please describe the issue you're seeing in the [migration feedback issue](https://gitlab.com/gitlab-org/gitlab/-/issues/331699). 1. Edit `/etc/gitlab/gitlab.rb`: @@ -1551,28 +1494,6 @@ To do that: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -### GitLab Pages fails to start in Docker container - -WARNING: -This issue is fixed in GitLab 14.3 and above, try upgrading GitLab first. - -The GitLab Pages daemon doesn't have permissions to bind mounts when it runs -in a Docker container. To overcome this issue, you must change the `chroot` -behavior: - -1. Edit `/etc/gitlab/gitlab.rb`. -1. Set the `inplace_chroot` to `true` for GitLab Pages: - - ```ruby - gitlab_pages['inplace_chroot'] = true - ``` - -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - -NOTE: -`inplace_chroot` option might not work with the other features, such as [Pages Access Control](#access-control). -The [GitLab Pages README](https://gitlab.com/gitlab-org/gitlab-pages#caveats) has more information about caveats and workarounds. - ### GitLab Pages deploy job fails with error "is not a recognized provider" If the **pages** job succeeds but the **deploy** job gives the error "is not a recognized provider": diff --git a/doc/administration/pages/source.md b/doc/administration/pages/source.md index 6c148387d7d..4bb9fc56b6f 100644 --- a/doc/administration/pages/source.md +++ b/doc/administration/pages/source.md @@ -68,7 +68,7 @@ Before proceeding with the Pages configuration, make sure that: ### DNS configuration GitLab Pages expect to run on their own virtual host. In your DNS server/provider -you need to add a [wildcard DNS A record](https://en.wikipedia.org/wiki/Wildcard_DNS_record) pointing to the +you need to add a [wildcard DNS `A` record](https://en.wikipedia.org/wiki/Wildcard_DNS_record) pointing to the host that GitLab runs. For example, an entry would look like this: ```plaintext diff --git a/doc/administration/polling.md b/doc/administration/polling.md index 8bd28824f83..5699bf78c04 100644 --- a/doc/administration/polling.md +++ b/doc/administration/polling.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/postgresql/database_load_balancing.md b/doc/administration/postgresql/database_load_balancing.md index 1324666c32b..b03abf9881c 100644 --- a/doc/administration/postgresql/database_load_balancing.md +++ b/doc/administration/postgresql/database_load_balancing.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -97,9 +97,9 @@ For example, on an environment that has PostgreSQL running on the hosts `host1.e Service discovery allows GitLab to automatically retrieve a list of PostgreSQL hosts to use. It periodically -checks a DNS A record, using the IPs returned by this record as the addresses +checks a DNS `A` record, using the IPs returned by this record as the addresses for the secondaries. For service discovery to work, all you need is a DNS server -and an A record containing the IP addresses of your secondaries. +and an `A` record containing the IP addresses of your secondaries. When using Omnibus GitLab the provided [Consul](../consul.md) service works as a DNS server and returns PostgreSQL addresses via the `postgresql-ha.service.consul` @@ -125,23 +125,23 @@ record. For example: |----------------------|---------------------------------------------------------------------------------------------------|-----------| | `nameserver` | The nameserver to use for looking up the DNS record. | localhost | | `record` | The record to look up. This option is required for service discovery to work. | | -| `record_type` | Optional record type to look up, this can be either A or SRV (GitLab 12.3 and later) | A | +| `record_type` | Optional record type to look up, this can be either `A` or `SRV` (GitLab 12.3 and later) | `A` | | `port` | The port of the nameserver. | 8600 | | `interval` | The minimum time in seconds between checking the DNS record. | 60 | | `disconnect_timeout` | The time in seconds after which an old connection is closed, after the list of hosts was updated. | 120 | | `use_tcp` | Lookup DNS resources using TCP instead of UDP | false | If `record_type` is set to `SRV`, then GitLab continues to use round-robin algorithm -and ignores the `weight` and `priority` in the record. Since SRV records usually +and ignores the `weight` and `priority` in the record. Since `SRV` records usually return hostnames instead of IPs, GitLab needs to look for the IPs of returned hostnames -in the additional section of the SRV response. If no IP is found for a hostname, GitLab -needs to query the configured `nameserver` for ANY record for each such hostname looking for A or AAAA +in the additional section of the `SRV` response. If no IP is found for a hostname, GitLab +needs to query the configured `nameserver` for `ANY` record for each such hostname looking for `A` or `AAAA` records, eventually dropping this hostname from rotation if it can't resolve its IP. -The `interval` value specifies the _minimum_ time between checks. If the A +The `interval` value specifies the _minimum_ time between checks. If the `A` record has a TTL greater than this value, then service discovery honors said -TTL. For example, if the TTL of the A record is 90 seconds, then service -discovery waits at least 90 seconds before checking the A record again. +TTL. For example, if the TTL of the `A` record is 90 seconds, then service +discovery waits at least 90 seconds before checking the `A` record again. When the list of hosts is updated, it might take a while for the old connections to be terminated. The `disconnect_timeout` setting can be used to enforce an diff --git a/doc/administration/postgresql/external.md b/doc/administration/postgresql/external.md index f4e4c7f8bef..5d693793a92 100644 --- a/doc/administration/postgresql/external.md +++ b/doc/administration/postgresql/external.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/postgresql/index.md b/doc/administration/postgresql/index.md index 15425a6d9f2..8b811deee16 100644 --- a/doc/administration/postgresql/index.md +++ b/doc/administration/postgresql/index.md @@ -1,12 +1,12 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Configuring PostgreSQL for scaling **(FREE SELF)** -In this section, you'll be guided through configuring a PostgreSQL database to +In this section, you are guided through configuring a PostgreSQL database to be used with GitLab in one of our [reference architectures](../reference_architectures/index.md). There are essentially three setups to choose from. diff --git a/doc/administration/postgresql/pgbouncer.md b/doc/administration/postgresql/pgbouncer.md index ed3c662eba3..8ae2b6497f8 100644 --- a/doc/administration/postgresql/pgbouncer.md +++ b/doc/administration/postgresql/pgbouncer.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference @@ -7,7 +7,7 @@ type: reference # Working with the bundled PgBouncer service **(PREMIUM SELF)** -[PgBouncer](http://www.pgbouncer.org/) is used to seamlessly migrate database +[PgBouncer](https://www.pgbouncer.org/) is used to seamlessly migrate database connections between servers in a failover scenario. Additionally, it can be used in a non-fault-tolerant setup to pool connections, speeding up response time while reducing resource usage. @@ -227,12 +227,12 @@ the database. Each of the listed services below use the following formula to def - `headroom` can be configured via `DB_POOL_HEADROOM` environment variable (default to `10`) To calculate the `default_pool_size`, multiply the number of instances of `puma`, `sidekiq` and `geo-logcursor` by the -number of connections each can consume as per listed above. The total will be the suggested `default_pool_size`. +number of connections each can consume as per listed above. The total is the suggested `default_pool_size`. If you are using more than one PgBouncer with an internal Load Balancer, you may be able to divide the `default_pool_size` by the number of instances to guarantee an evenly distributed load between them. -The `pgbouncer['max_client_conn']` is the hard-limit of connections PgBouncer can accept. It's unlikely you will need +The `pgbouncer['max_client_conn']` is the hard limit of connections PgBouncer can accept. It's unlikely you need to change this. If you are hitting that limit, you may want to consider adding additional PgBouncers with an internal Load Balancer. diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index 84122149cb8..e9b607ad5d4 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -42,7 +42,7 @@ collections "**Consul** x3" as consul #e76a9b card "Database" as database { collections "**PGBouncer x3**\n//Consul//" as pgbouncer #4EA7FF - + card "**PostgreSQL** //Primary//\n//Patroni//\n//PgBouncer//\n//Consul//" as postgres_primary #4EA7FF collections "**PostgreSQL** //Secondary// **x2**\n//Patroni//\n//PgBouncer//\n//Consul//" as postgres_secondary #4EA7FF diff --git a/doc/administration/postgresql/standalone.md b/doc/administration/postgresql/standalone.md index b21625acb56..5428e44ccc0 100644 --- a/doc/administration/postgresql/standalone.md +++ b/doc/administration/postgresql/standalone.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/pseudonymizer.md b/doc/administration/pseudonymizer.md index ad4cfd11474..b14c1f64efe 100644 --- a/doc/administration/pseudonymizer.md +++ b/doc/administration/pseudonymizer.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments remove_date: '2022-08-22' diff --git a/doc/administration/raketasks/check.md b/doc/administration/raketasks/check.md index fba151fefe1..d66ff9afb94 100644 --- a/doc/administration/raketasks/check.md +++ b/doc/administration/raketasks/check.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/raketasks/geo.md b/doc/administration/raketasks/geo.md index 6306e54bdc6..5c6c99d099b 100644 --- a/doc/administration/raketasks/geo.md +++ b/doc/administration/raketasks/geo.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -33,7 +33,7 @@ sudo -u git -H bundle exec rake geo:git:housekeeping:incremental_repack RAILS_EN ### Full Repack This is equivalent of running `git repack -d -A --pack-kept-objects` on a -_bare_ repository which will optionally, write a reachability bitmap index +_bare_ repository which optionally, writes a reachability bitmap index when this is enabled in GitLab. **Omnibus Installation** diff --git a/doc/administration/raketasks/github_import.md b/doc/administration/raketasks/github_import.md index 0cdfd1c28ff..30c8518c209 100644 --- a/doc/administration/raketasks/github_import.md +++ b/doc/administration/raketasks/github_import.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -12,7 +12,7 @@ which becomes the owner of the project. You can resume an import with the same command. Bear in mind that the syntax is very specific. Remove any spaces within the argument block and -before/after the brackets. Also, some shells (for example, `zsh`) can interpret the open/close brackets +before/after the brackets. Also, some shells (for example, Zsh) can interpret the open/close brackets (`[]`) separately. You may need to either escape the brackets or use double quotes. ## Caveats diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md index 82ebdcaab42..f4bd6f255e5 100644 --- a/doc/administration/raketasks/maintenance.md +++ b/doc/administration/raketasks/maintenance.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/raketasks/project_import_export.md b/doc/administration/raketasks/project_import_export.md index 13f8dfdccc2..f623d4e4526 100644 --- a/doc/administration/raketasks/project_import_export.md +++ b/doc/administration/raketasks/project_import_export.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/raketasks/smtp.md b/doc/administration/raketasks/smtp.md index c738f1dcd00..bc78d54aa1b 100644 --- a/doc/administration/raketasks/smtp.md +++ b/doc/administration/raketasks/smtp.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/raketasks/storage.md b/doc/administration/raketasks/storage.md index 689ed5c5824..27b899dd1b1 100644 --- a/doc/administration/raketasks/storage.md +++ b/doc/administration/raketasks/storage.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/raketasks/uploads/migrate.md b/doc/administration/raketasks/uploads/migrate.md index aec75f0b302..c73840cb9ff 100644 --- a/doc/administration/raketasks/uploads/migrate.md +++ b/doc/administration/raketasks/uploads/migrate.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/raketasks/uploads/sanitize.md b/doc/administration/raketasks/uploads/sanitize.md index 7dc813de14e..bf6dc4fd776 100644 --- a/doc/administration/raketasks/uploads/sanitize.md +++ b/doc/administration/raketasks/uploads/sanitize.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/read_only_gitlab.md b/doc/administration/read_only_gitlab.md index b7e8397dd95..a601ba4cd7a 100644 --- a/doc/administration/read_only_gitlab.md +++ b/doc/administration/read_only_gitlab.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/redis/index.md b/doc/administration/redis/index.md index 23d13491c75..3581b27a4c8 100644 --- a/doc/administration/redis/index.md +++ b/doc/administration/redis/index.md @@ -1,6 +1,6 @@ --- type: index -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/redis/replication_and_failover.md b/doc/administration/redis/replication_and_failover.md index 086057d80b4..9b1a456835a 100644 --- a/doc/administration/redis/replication_and_failover.md +++ b/doc/administration/redis/replication_and_failover.md @@ -1,6 +1,6 @@ --- type: howto -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -12,8 +12,8 @@ This is the documentation for the Omnibus GitLab packages. For using your own non-bundled Redis, follow the [relevant documentation](replication_and_failover_external.md). NOTE: -In Redis lingo, primary is called master. In this document, primary is used -instead of master, except the settings where `master` is required. +In Redis lingo, `primary` is called `master`. In this document, `primary` is used +instead of `master`, except the settings where `master` is required. Using [Redis](https://redis.io/) in scalable environment is possible using a **Primary** x **Replica** topology with a [Redis Sentinel](https://redis.io/topics/sentinel) service to watch and automatically diff --git a/doc/administration/redis/replication_and_failover_external.md b/doc/administration/redis/replication_and_failover_external.md index c19e42a5f14..998455e5621 100644 --- a/doc/administration/redis/replication_and_failover_external.md +++ b/doc/administration/redis/replication_and_failover_external.md @@ -1,6 +1,6 @@ --- type: howto -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -218,7 +218,7 @@ The following steps should be performed in the GitLab application server which ideally should not have Redis or Sentinels in the same machine: 1. Edit `/home/git/gitlab/config/resque.yml` following the example in - [resque.yml.example](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/resque.yml.example), and uncomment the Sentinel lines, pointing to + [`resque.yml.example`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/resque.yml.example), and uncomment the Sentinel lines, pointing to the correct server credentials: ```yaml diff --git a/doc/administration/redis/standalone.md b/doc/administration/redis/standalone.md index 8a3b88780a1..6e0e1eb886e 100644 --- a/doc/administration/redis/standalone.md +++ b/doc/administration/redis/standalone.md @@ -1,6 +1,6 @@ --- type: howto -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/redis/troubleshooting.md b/doc/administration/redis/troubleshooting.md index f4aab9d7b7f..4426adc5d51 100644 --- a/doc/administration/redis/troubleshooting.md +++ b/doc/administration/redis/troubleshooting.md @@ -1,6 +1,6 @@ --- type: reference -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index f70912dbecb..99272cdd226 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -18,23 +18,23 @@ full list of reference architectures, see > - **Test requests per second (RPS) rates:** API: 200 RPS, Web: 20 RPS, Git (Pull): 20 RPS, Git (Push): 4 RPS > - **[Latest Results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k)** -| Service | Nodes | Configuration | GCP | AWS | Azure | -|-----------------------------------------------------|-------------|-------------------------|------------------|--------------|-----------| -| External load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| PostgreSQL<sup>1</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | `D8s v3` | -| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Gitaly<sup>5</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | `D16s v3` | -| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| GitLab Rails | 3 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | -| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | -| Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | n/a | -| NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Service | Nodes | Configuration | GCP | AWS | Azure | +|-----------------------------------------------------|----------------|-------------------------|------------------|----------------|----------------| +| External load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| PostgreSQL<sup>1</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | `D8s v3` | +| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Gitaly<sup>5</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | `D16s v3` | +| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| GitLab Rails | 3 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | +| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Object storage<sup>4</sup> | Not applicable | Not applicable | Not applicable | Not applicable | Not applicable | +| NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> @@ -86,7 +86,7 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 - + redis_cache -[hidden]-> redis_persistent } @@ -1620,18 +1620,12 @@ and on all Praefect clients that communicate with it following the procedure des Note the following: -- The certificate must specify the address you use to access the Praefect server. If - addressing the Praefect server by: - - - Hostname, you can either use the Common Name field for this, or add it as a Subject - Alternative Name. - - IP address, you must add it as a Subject Alternative Name to the certificate. - +- The certificate must specify the address you use to access the Praefect server. You must add the hostname or IP + address as a Subject Alternative Name to the certificate. - You can configure Praefect servers with both an unencrypted listening address `listen_addr` and an encrypted listening address `tls_listen_addr` at the same time. This allows you to do a gradual transition from unencrypted to encrypted traffic, if - necessary. - + necessary. To disable the unencrypted listener, set `praefect['listen_addr'] = nil`. - The Internal Load Balancer will also access to the certificates and need to be configured to allow for TLS pass-through. Refer to the load balancers documentation on how to configure this. @@ -2186,7 +2180,7 @@ GitLab has been tested on a number of object storage providers: - [Amazon S3](https://aws.amazon.com/s3/) - [Google Cloud Storage](https://cloud.google.com/storage) -- [Digital Ocean Spaces](http://www.digitalocean.com/products/spaces) +- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift (S3 compatibility mode)](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) @@ -2202,10 +2196,6 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. -GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. - -In GitLab 13.6 and later, it's also recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. This is required when no NFS node has been deployed. - For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based on what features you intend to use: @@ -2235,15 +2225,21 @@ in the future. </a> </div> +## Enable incremental logging + +GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared through NFS on any GitLab Rails and Sidekiq nodes. + +While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs. + ## Configure Advanced Search -You can leverage Elasticsearch and [enable Advanced Search](../../integration/elasticsearch.md) +You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md) for faster, more advanced code search across your entire GitLab instance. Elasticsearch cluster design and requirements are dependent on your specific data. For recommended best practices about how to set up your Elasticsearch cluster alongside your instance, read how to -[choose the optimal cluster configuration](../../integration/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). +[choose the optimal cluster configuration](../../integration/advanced_search/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -2317,18 +2313,18 @@ future with further specific cloud provider details. Next are the backend components that run on static compute VMs via Omnibus (or External PaaS services where applicable): -| Service | Nodes | Configuration | GCP | AWS | -|------------------------------------------|-------|-----------------------|------------------|--------------| -| Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| PostgreSQL<sup>1</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | -| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | -| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | -| Gitaly<sup>5</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | -| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | +| Service | Nodes | Configuration | GCP | AWS | +|------------------------------------------|----------------|-----------------------|------------------|----------------| +| Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| PostgreSQL<sup>1</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | +| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| Gitaly<sup>5</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | +| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Object storage<sup>4</sup> | Not applicable | Not applicable | Not applicable | Not applicable | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> @@ -2381,7 +2377,7 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 - + redis_cache -[hidden]-> redis_persistent } diff --git a/doc/administration/reference_architectures/1k_users.md b/doc/administration/reference_architectures/1k_users.md index 8d5afd732d9..c356c7224cb 100644 --- a/doc/administration/reference_architectures/1k_users.md +++ b/doc/administration/reference_architectures/1k_users.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -62,7 +62,7 @@ monitor .[#7FFFD4,norank]--> redis @enduml ``` -The diagram above shows that while GitLab can be installed on a single server, it is internally composed of multiple services. As a GitLab instance is scaled, each of these services are broken out and independently scaled according to the demands placed on them. In some cases PaaS can be leveraged for some services (e.g. Cloud Object Storage for some file systems). For the sake of redundancy some of the services become clusters of nodes storing the same data. In a horizontal configuration of GitLab there are various ancillary services required to coordinate clusters or discover of resources (e.g. PgBouncer for PostgreSQL connection management, Consul for Prometheus end point discovery). +The diagram above shows that while GitLab can be installed on a single server, it is internally composed of multiple services. As a GitLab instance is scaled, each of these services are broken out and independently scaled according to the demands placed on them. In some cases PaaS can be leveraged for some services (for example, Cloud Object Storage for some file systems). For the sake of redundancy some of the services become clusters of nodes storing the same data. In a horizontal configuration of GitLab there are various ancillary services required to coordinate clusters or discover of resources (for example, PgBouncer for PostgreSQL connection management, Consul for Prometheus end point discovery). ## Requirements @@ -106,13 +106,13 @@ performance and reliability at an increased complexity cost. ## Configure Advanced Search **(PREMIUM SELF)** -You can leverage Elasticsearch and [enable Advanced Search](../../integration/elasticsearch.md) +You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md) for faster, more advanced code search across your entire GitLab instance. Elasticsearch cluster design and requirements are dependent on your specific data. For recommended best practices about how to set up your Elasticsearch cluster alongside your instance, read how to -[choose the optimal cluster configuration](../../integration/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). +[choose the optimal cluster configuration](../../integration/advanced_search/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). ## Cloud Native Hybrid reference architecture with Helm Charts diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index d9f349b59c7..367654b8e59 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -18,22 +18,22 @@ full list of reference architectures, see > - **Test requests per second (RPS) rates:** API: 500 RPS, Web: 50 RPS, Git (Pull): 50 RPS, Git (Push): 10 RPS > - **[Latest Results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/25k)** -| Service | Nodes | Configuration | GCP | AWS | Azure | -|---------------------------------------------------|-------------|-------------------------|------------------|--------------|-----------| -| External load balancing node<sup>3</sup> | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | -| Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| PostgreSQL<sup>1</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | `D16s v3` | -| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Internal load balancing node<sup>3</sup> | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | -| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Gitaly<sup>5</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` | -| Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | -| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| GitLab Rails | 5 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | -| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | -| Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | n/a | +| Service | Nodes | Configuration | GCP | AWS | Azure | +|---------------------------------------------------|----------------|-------------------------|------------------|----------------|----------------| +| External load balancing node<sup>3</sup> | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| PostgreSQL<sup>1</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | `D16s v3` | +| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Internal load balancing node<sup>3</sup> | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Gitaly<sup>5</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` | +| Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| GitLab Rails | 5 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | +| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Object storage<sup>4</sup> | Not applicable | Not applicable | Not applicable | Not applicable | Not applicable | | NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> @@ -1624,18 +1624,12 @@ and on all Praefect clients that communicate with it following the procedure des Note the following: -- The certificate must specify the address you use to access the Praefect server. If - addressing the Praefect server by: - - - Hostname, you can either use the Common Name field for this, or add it as a Subject - Alternative Name. - - IP address, you must add it as a Subject Alternative Name to the certificate. - +- The certificate must specify the address you use to access the Praefect server. You must add the hostname or IP + address as a Subject Alternative Name to the certificate. - You can configure Praefect servers with both an unencrypted listening address `listen_addr` and an encrypted listening address `tls_listen_addr` at the same time. This allows you to do a gradual transition from unencrypted to encrypted traffic, if - necessary. - + necessary. To disable the unencrypted listener, set `praefect['listen_addr'] = nil`. - The Internal Load Balancer will also access to the certificates and need to be configured to allow for TLS passthrough. Refer to the load balancers documentation on how to configure this. @@ -2190,7 +2184,7 @@ GitLab has been tested on a number of object storage providers: - [Amazon S3](https://aws.amazon.com/s3/) - [Google Cloud Storage](https://cloud.google.com/storage) -- [Digital Ocean Spaces](http://www.digitalocean.com/products/spaces) +- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift (S3 compatibility mode)](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) @@ -2206,10 +2200,6 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. -GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. - -In GitLab 13.6 and later, it's also recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. This is required when no NFS node has been deployed. - For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based on what features you intend to use: @@ -2239,15 +2229,21 @@ in the future. </a> </div> +## Enable incremental logging + +GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared through NFS on any GitLab Rails and Sidekiq nodes. + +While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs. + ## Configure Advanced Search -You can leverage Elasticsearch and [enable Advanced Search](../../integration/elasticsearch.md) +You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md) for faster, more advanced code search across your entire GitLab instance. Elasticsearch cluster design and requirements are dependent on your specific data. For recommended best practices about how to set up your Elasticsearch cluster alongside your instance, read how to -[choose the optimal cluster configuration](../../integration/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). +[choose the optimal cluster configuration](../../integration/advanced_search/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -2315,18 +2311,18 @@ future with further specific cloud provider details. Next are the backend components that run on static compute VMs via Omnibus (or External PaaS services where applicable): -| Service | Nodes | Configuration | GCP | AWS | -|------------------------------------------|-------|------------------------|------------------|--------------| -| Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| PostgreSQL<sup>1</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | -| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Internal load balancing node<sup>3</sup> | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | `c5.xlarge` | -| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | -| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | -| Gitaly<sup>5</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | -| Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | -| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | +| Service | Nodes | Configuration | GCP | AWS | +|------------------------------------------|----------------|------------------------|------------------|----------------| +| Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| PostgreSQL<sup>1</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | +| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Internal load balancing node<sup>3</sup> | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | `c5.xlarge` | +| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| Gitaly<sup>5</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | +| Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | +| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Object storage<sup>4</sup> | Not applicable | Not applicable | Not applicable | Not applicable | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index d029f356612..3b9a8f966c8 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -19,16 +19,16 @@ For a full list of reference architectures, see > - **Test requests per second (RPS) rates:** API: 40 RPS, Web: 4 RPS, Git (Pull): 4 RPS, Git (Push): 1 RPS > - **[Latest Results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/2k)** -| Service | Nodes | Configuration | GCP | AWS | Azure | -|------------------------------------------|--------|-------------------------|-----------------|--------------|----------| -| Load balancer<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| PostgreSQL<sup>1</sup> | 1 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | -| Redis<sup>2</sup> | 1 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `m5.large` | `D2s v3` | -| Gitaly | 1 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| GitLab Rails | 2 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | -| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | n/a | -| NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Service | Nodes | Configuration | GCP | AWS | Azure | +|------------------------------------------|----------------|-------------------------|-----------------|----------------|----------------| +| Load balancer<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| PostgreSQL<sup>1</sup> | 1 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | +| Redis<sup>2</sup> | 1 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `m5.large` | `D2s v3` | +| Gitaly | 1 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| GitLab Rails | 2 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | +| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Object storage<sup>4</sup> | Not applicable | Not applicable | Not applicable | Not applicable | Not applicable | +| NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | <!-- markdownlint-disable MD029 --> 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. @@ -527,11 +527,9 @@ that communicate with it following the procedure described in NOTE: The self-signed certificate must specify the address you use to access the -Gitaly server. If you are addressing the Gitaly server by a hostname, you can -either use the Common Name field for this, or add it as a Subject Alternative +Gitaly server. If you are addressing the Gitaly server by a hostname, add it as a Subject Alternative Name. If you are addressing the Gitaly server by its IP address, you must add it as a Subject Alternative Name to the certificate. -[gRPC does not support using an IP address as Common Name in a certificate](https://github.com/grpc/grpc/issues/2691). It's possible to configure Gitaly servers with both an unencrypted listening address (`listen_addr`) and an encrypted listening address (`tls_listen_addr`) @@ -892,7 +890,7 @@ GitLab has been tested on a number of object storage providers: - [Amazon S3](https://aws.amazon.com/s3/) - [Google Cloud Storage](https://cloud.google.com/storage) -- [Digital Ocean Spaces](http://www.digitalocean.com/products/spaces) +- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift (S3 compatibility mode)](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) @@ -943,13 +941,13 @@ in the future. ## Configure Advanced Search **(PREMIUM SELF)** -You can leverage Elasticsearch and [enable Advanced Search](../../integration/elasticsearch.md) +You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md) for faster, more advanced code search across your entire GitLab instance. Elasticsearch cluster design and requirements are dependent on your specific data. For recommended best practices about how to set up your Elasticsearch cluster alongside your instance, read how to -[choose the optimal cluster configuration](../../integration/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). +[choose the optimal cluster configuration](../../integration/advanced_search/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -1020,12 +1018,12 @@ future with further specific cloud provider details. Next are the backend components that run on static compute VMs via Omnibus (or External PaaS services where applicable): -| Service | Nodes | Configuration | GCP | AWS | -|----------------------------|-------|------------------------|-----------------|-------------| -| PostgreSQL<sup>1</sup> | 1 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | -| Redis<sup>2</sup> | 1 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `m5.large` | -| Gitaly | 1 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | -| Object storage<sup>3</sup> | n/a | n/a | n/a | n/a | +| Service | Nodes | Configuration | GCP | AWS | +|----------------------------|----------------|------------------------|-----------------|----------------| +| PostgreSQL<sup>1</sup> | 1 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | +| Redis<sup>2</sup> | 1 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `m5.large` | +| Gitaly | 1 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| Object storage<sup>3</sup> | Not applicable | Not applicable | Not applicable | Not applicable | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index 14b8982766c..a43d33a4d4a 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -28,21 +28,21 @@ For a full list of reference architectures, see > - **Test requests per second (RPS) rates:** API: 60 RPS, Web: 6 RPS, Git (Pull): 6 RPS, Git (Push): 1 RPS > - **[Latest Results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/3k)** -| Service | Nodes | Configuration | GCP | AWS | Azure | -|--------------------------------------------|-------------|-----------------------|-----------------|--------------|----------| -| External load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Redis<sup>2</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | -| Consul<sup>1</sup> + Sentinel<sup>2</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| PostgreSQL<sup>1</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | -| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Gitaly<sup>5</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | -| GitLab Rails | 3 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | -| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | n/a | +| Service | Nodes | Configuration | GCP | AWS | Azure | +|--------------------------------------------|----------------|-----------------------|-----------------|----------------|----------------| +| External load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Redis<sup>2</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | +| Consul<sup>1</sup> + Sentinel<sup>2</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| PostgreSQL<sup>1</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | +| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Gitaly<sup>5</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | +| GitLab Rails | 3 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | +| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Object storage<sup>4</sup> | Not applicable | Not applicable | Not applicable | Not applicable | Not applicable | | NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> @@ -1564,18 +1564,12 @@ and on all Praefect clients that communicate with it following the procedure des Note the following: -- The certificate must specify the address you use to access the Praefect server. If - addressing the Praefect server by: - - - Hostname, you can either use the Common Name field for this, or add it as a Subject - Alternative Name. - - IP address, you must add it as a Subject Alternative Name to the certificate. - +- The certificate must specify the address you use to access the Praefect server. You must add the hostname or IP + address as a Subject Alternative Name to the certificate. - You can configure Praefect servers with both an unencrypted listening address `listen_addr` and an encrypted listening address `tls_listen_addr` at the same time. This allows you to do a gradual transition from unencrypted to encrypted traffic, if - necessary. - + necessary. To disable the unencrypted listener, set `praefect['listen_addr'] = nil`. - The Internal Load Balancer will also access to the certificates and need to be configured to allow for TLS passthrough. Refer to the load balancers documentation on how to configure this. @@ -2125,7 +2119,7 @@ GitLab has been tested on a number of object storage providers: - [Amazon S3](https://aws.amazon.com/s3/) - [Google Cloud Storage](https://cloud.google.com/storage) -- [Digital Ocean Spaces](http://www.digitalocean.com/products/spaces) +- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift (S3 compatibility mode)](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) @@ -2141,10 +2135,6 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. -GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. - -In GitLab 13.6 and later, it's also recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. This is required when no NFS node has been deployed. - For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based on what features you intend to use: @@ -2174,15 +2164,21 @@ in the future. </a> </div> +## Enable incremental logging + +GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared through NFS on any GitLab Rails and Sidekiq nodes. + +While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs. + ## Configure Advanced Search -You can leverage Elasticsearch and [enable Advanced Search](../../integration/elasticsearch.md) +You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md) for faster, more advanced code search across your entire GitLab instance. Elasticsearch cluster design and requirements are dependent on your specific data. For recommended best practices about how to set up your Elasticsearch cluster alongside your instance, read how to -[choose the optimal cluster configuration](../../integration/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). +[choose the optimal cluster configuration](../../integration/advanced_search/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -2275,17 +2271,17 @@ future with further specific cloud provider details. Next are the backend components that run on static compute VMs via Omnibus (or External PaaS services where applicable): -| Service | Nodes | Configuration | GCP | AWS | -|-------------------------------------------|-------|-----------------------|-----------------|-------------| -| Redis<sup>2</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | -| Consul<sup>1</sup> + Sentinel<sup>2</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| PostgreSQL<sup>1</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | -| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Gitaly<sup>5</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | -| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | +| Service | Nodes | Configuration | GCP | AWS | +|-------------------------------------------|----------------|-----------------------|-----------------|----------------| +| Redis<sup>2</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | +| Consul<sup>1</sup> + Sentinel<sup>2</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| PostgreSQL<sup>1</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | +| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Gitaly<sup>5</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Object storage<sup>4</sup> | Not applicable | Not applicable | Not applicable | Not applicable | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 078e3a289cc..316969dfacc 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -18,23 +18,23 @@ full list of reference architectures, see > - **Test requests per second (RPS) rates:** API: 1000 RPS, Web: 100 RPS, Git (Pull): 100 RPS, Git (Push): 20 RPS > - **[Latest Results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/50k)** -| Service | Nodes | Configuration | GCP | AWS | Azure | -|---------------------------------------------------|-------------|-------------------------|------------------|---------------|-----------| -| External load balancing node<sup>3</sup> | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | -| Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| PostgreSQL<sup>1</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` | -| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Internal load balancing node<sup>3</sup> | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | -| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Gitaly<sup>5</sup> | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | `m5.16xlarge` | `D64s v3` | -| Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | -| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| GitLab Rails | 12 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | -| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | -| Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | n/a | -| NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Service | Nodes | Configuration | GCP | AWS | Azure | +|---------------------------------------------------|----------------|-------------------------|------------------|----------------|----------------| +| External load balancing node<sup>3</sup> | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | +| Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| PostgreSQL<sup>1</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` | +| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Internal load balancing node<sup>3</sup> | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | +| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Gitaly<sup>5</sup> | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | `m5.16xlarge` | `D64s v3` | +| Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| GitLab Rails | 12 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | +| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Object storage<sup>4</sup> | Not applicable | Not applicable | Not applicable | Not applicable | Not applicable | +| NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> @@ -1633,18 +1633,12 @@ and on all Praefect clients that communicate with it following the procedure des Note the following: -- The certificate must specify the address you use to access the Praefect server. If - addressing the Praefect server by: - - - Hostname, you can either use the Common Name field for this, or add it as a Subject - Alternative Name. - - IP address, you must add it as a Subject Alternative Name to the certificate. - +- The certificate must specify the address you use to access the Praefect server. You must add the hostname or IP + address as a Subject Alternative Name to the certificate. - You can configure Praefect servers with both an unencrypted listening address `listen_addr` and an encrypted listening address `tls_listen_addr` at the same time. This allows you to do a gradual transition from unencrypted to encrypted traffic, if - necessary. - + necessary. To disable the unencrypted listener, set `praefect['listen_addr'] = nil`. - The Internal Load Balancer will also access to the certificates and need to be configured to allow for TLS passthrough. Refer to the load balancers documentation on how to configure this. @@ -2206,7 +2200,7 @@ GitLab has been tested on a number of object storage providers: - [Amazon S3](https://aws.amazon.com/s3/) - [Google Cloud Storage](https://cloud.google.com/storage) -- [Digital Ocean Spaces](http://www.digitalocean.com/products/spaces) +- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift (S3 compatibility mode)](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) @@ -2222,10 +2216,6 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. -GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. - -In GitLab 13.6 and later, it's also recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. This is required when no NFS node has been deployed. - For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based on what features you intend to use: @@ -2255,15 +2245,21 @@ in the future. </a> </div> +## Enable incremental logging + +GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared through NFS on any GitLab Rails and Sidekiq nodes. + +While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs. + ## Configure Advanced Search -You can leverage Elasticsearch and [enable Advanced Search](../../integration/elasticsearch.md) +You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md) for faster, more advanced code search across your entire GitLab instance. Elasticsearch cluster design and requirements are dependent on your specific data. For recommended best practices about how to set up your Elasticsearch cluster alongside your instance, read how to -[choose the optimal cluster configuration](../../integration/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). +[choose the optimal cluster configuration](../../integration/advanced_search/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -2331,18 +2327,18 @@ future with further specific cloud provider details. Next are the backend components that run on static compute VMs via Omnibus (or External PaaS services where applicable): -| Service | Nodes | Configuration | GCP | AWS | -|------------------------------------------|-------|------------------------|------------------|---------------| -| Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| PostgreSQL<sup>1</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | -| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Internal load balancing node<sup>3</sup> | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | -| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | -| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | -| Gitaly<sup>5</sup> | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | `m5.16xlarge` | -| Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | -| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | +| Service | Nodes | Configuration | GCP | AWS | +|------------------------------------------|----------------|------------------------|------------------|----------------| +| Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| PostgreSQL<sup>1</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | +| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Internal load balancing node<sup>3</sup> | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | +| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| Gitaly<sup>5</sup> | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | `m5.16xlarge` | +| Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | +| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Object storage<sup>4</sup> | Not applicable | Not applicable | Not applicable | Not applicable | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index d18c77902f6..ef44d688f31 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -25,22 +25,22 @@ costly-to-operate environment by using the > - **Test requests per second (RPS) rates:** API: 100 RPS, Web: 10 RPS, Git (Pull): 10 RPS, Git (Push): 2 RPS > - **[Latest Results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/5k)** -| Service | Nodes | Configuration | GCP | AWS | Azure | -|--------------------------------------------|-------------|-------------------------|-----------------|--------------|----------| -| External load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Redis<sup>2</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | -| Consul<sup>1</sup> + Sentinel<sup>2</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| PostgreSQL<sup>1</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Gitaly<sup>5</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | `D8s v3` | -| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | -| GitLab Rails | 3 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | `c5.4xlarge` | `F16s v2`| -| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | n/a | -| NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Service | Nodes | Configuration | GCP | AWS | Azure | +|--------------------------------------------|----------------|-------------------------|-----------------|----------------|----------------| +| External load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Redis<sup>2</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | +| Consul<sup>1</sup> + Sentinel<sup>2</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| PostgreSQL<sup>1</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Gitaly<sup>5</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | `D8s v3` | +| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | +| GitLab Rails | 3 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | `c5.4xlarge` | `F16s v2` | +| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Object storage<sup>4</sup> | Not applicable | Not applicable | Not applicable | Not applicable | Not applicable | +| NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> @@ -1562,18 +1562,12 @@ and on all Praefect clients that communicate with it following the procedure des Note the following: -- The certificate must specify the address you use to access the Praefect server. If - addressing the Praefect server by: - - - Hostname, you can either use the Common Name field for this, or add it as a Subject - Alternative Name. - - IP address, you must add it as a Subject Alternative Name to the certificate. - +- The certificate must specify the address you use to access the Praefect server. You must add the hostname or IP + address as a Subject Alternative Name to the certificate. - You can configure Praefect servers with both an unencrypted listening address `listen_addr` and an encrypted listening address `tls_listen_addr` at the same time. This allows you to do a gradual transition from unencrypted to encrypted traffic, if - necessary. - + necessary. To disable the unencrypted listener, set `praefect['listen_addr'] = nil`. - The Internal Load Balancer will also access to the certificates and need to be configured to allow for TLS passthrough. Refer to the load balancers documentation on how to configure this. @@ -2125,7 +2119,7 @@ GitLab has been tested on a number of object storage providers: - [Amazon S3](https://aws.amazon.com/s3/) - [Google Cloud Storage](https://cloud.google.com/storage) -- [Digital Ocean Spaces](http://www.digitalocean.com/products/spaces) +- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift (S3 compatibility mode)](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) @@ -2141,10 +2135,6 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. -GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. - -In GitLab 13.6 and later, it's also recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. This is required when no NFS node has been deployed. - For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based on what features you intend to use: @@ -2174,15 +2164,21 @@ in the future. </a> </div> +## Enable incremental logging + +GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared through NFS on any GitLab Rails and Sidekiq nodes. + +While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs. + ## Configure Advanced Search -You can leverage Elasticsearch and [enable Advanced Search](../../integration/elasticsearch.md) +You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md) for faster, more advanced code search across your entire GitLab instance. Elasticsearch cluster design and requirements are dependent on your specific data. For recommended best practices about how to set up your Elasticsearch cluster alongside your instance, read how to -[choose the optimal cluster configuration](../../integration/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). +[choose the optimal cluster configuration](../../integration/advanced_search/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -2250,17 +2246,17 @@ future with further specific cloud provider details. Next are the backend components that run on static compute VMs via Omnibus (or External PaaS services where applicable): -| Service | Nodes | Configuration | GCP | AWS | -|-------------------------------------------|-------|-----------------------|-----------------|--------------| -| Redis<sup>2</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | -| Consul<sup>1</sup> + Sentinel<sup>2</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| PostgreSQL<sup>1</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | -| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Gitaly<sup>5</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | -| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | +| Service | Nodes | Configuration | GCP | AWS | +|-------------------------------------------|----------------|-----------------------|-----------------|----------------| +| Redis<sup>2</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | +| Consul<sup>1</sup> + Sentinel<sup>2</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| PostgreSQL<sup>1</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Gitaly<sup>5</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | +| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Object storage<sup>4</sup> | Not applicable | Not applicable | Not applicable | Not applicable | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index 089e8881d7d..d9fa6c7bdd9 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -1,6 +1,6 @@ --- type: reference, concepts -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -116,7 +116,7 @@ per 1,000 users: ### How to interpret the results NOTE: -Read our blog post on [how our QA team leverages GitLab’s performance testing tool](https://about.gitlab.com/blog/2020/02/18/how-were-building-up-performance-testing-of-gitlab/). +Read our blog post on [how our QA team leverages GitLab performance testing tool](https://about.gitlab.com/blog/2020/02/18/how-were-building-up-performance-testing-of-gitlab/). Testing is done publicly and all results are shared. @@ -215,7 +215,7 @@ table.test-coverage th { ## Cost to run -The following table details the cost to run the different reference architectures across GCP, AWS, and Azure. Bare-metal costs are not included here as it varies widely depending on each customer configuration. +The following table details the cost to run the different reference architectures across GCP, AWS, and Azure. Bare-metal costs are not included here as it varies widely depending on each customer configuration. <table class="test-coverage"> <col> diff --git a/doc/administration/reference_architectures/troubleshooting.md b/doc/administration/reference_architectures/troubleshooting.md index c8c13fca59d..2d10d05da37 100644 --- a/doc/administration/reference_architectures/troubleshooting.md +++ b/doc/administration/reference_architectures/troubleshooting.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/reply_by_email_postfix_setup.md b/doc/administration/reply_by_email_postfix_setup.md index aae5475312f..2652ad20329 100644 --- a/doc/administration/reply_by_email_postfix_setup.md +++ b/doc/administration/reply_by_email_postfix_setup.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md index 00aa8c214c5..3acaad8231e 100644 --- a/doc/administration/repository_storage_paths.md +++ b/doc/administration/repository_storage_paths.md @@ -174,4 +174,4 @@ information. ## Move repositories To move a repository to a different repository storage (for example, from `default` to `storage2`), use the -same process as [migrating to Gitaly Cluster](gitaly/index.md#migrating-to-gitaly-cluster). +same process as [migrating to Gitaly Cluster](gitaly/index.md#migrate-to-gitaly-cluster). diff --git a/doc/administration/restart_gitlab.md b/doc/administration/restart_gitlab.md index 44bfa153123..06b6cbd4271 100644 --- a/doc/administration/restart_gitlab.md +++ b/doc/administration/restart_gitlab.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -97,7 +97,7 @@ It also [restarts GitLab components](#how-to-restart-gitlab) where needed, if any of their configuration files have changed. If you manually edit any files in `/var/opt/gitlab` that are managed by Chef, -running reconfigure reverts the changes AND restarts the services that +running reconfigure reverts the changes and restarts the services that depend on those files. ## Installations from source diff --git a/doc/administration/server_hooks.md b/doc/administration/server_hooks.md index 9d8bc03e5e9..a83a17d6d52 100644 --- a/doc/administration/server_hooks.md +++ b/doc/administration/server_hooks.md @@ -23,7 +23,7 @@ alternatives to server hooks include: - [Webhooks](../user/project/integrations/webhooks.md). - [GitLab CI/CD](../ci/index.md). -- [Push rules](../push_rules/push_rules.md), for a user-configurable Git hook interface. +- [Push rules](../user/project/repository/push_rules.md), for a user-configurable Git hook interface. [Geo](geo/index.md) doesn't replicate server hooks to secondary nodes. diff --git a/doc/administration/sidekiq.md b/doc/administration/sidekiq.md index d9031e2221b..528ecf12df9 100644 --- a/doc/administration/sidekiq.md +++ b/doc/administration/sidekiq.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -233,7 +233,7 @@ To enable LDAP with the synchronization worker for Sidekiq: gitlab_rails['ldap_servers'] = { 'main' => { 'label' => 'LDAP', - 'host' => 'ldap.mydomain.com', + 'host' => 'ldap.mydomain.com', 'port' => 389, 'uid' => 'sAMAccountName', 'encryption' => 'simple_tls', @@ -269,7 +269,7 @@ To enable LDAP with the synchronization worker for Sidekiq: 'external_groups' => [], 'sync_ssh_keys' => false } - } + } gitlab_rails['ldap_sync_worker_cron'] = "0 */12 * * *" ``` diff --git a/doc/administration/sidekiq_health_check.md b/doc/administration/sidekiq_health_check.md index 3ff17d3682b..aaab2b7e086 100644 --- a/doc/administration/sidekiq_health_check.md +++ b/doc/administration/sidekiq_health_check.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/smime_signing_email.md b/doc/administration/smime_signing_email.md index 8fdd87a5b2d..943fd91c885 100644 --- a/doc/administration/smime_signing_email.md +++ b/doc/administration/smime_signing_email.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/system_hooks.md b/doc/administration/system_hooks.md index 71d7e7f1426..cb141a18ae6 100644 --- a/doc/administration/system_hooks.md +++ b/doc/administration/system_hooks.md @@ -61,7 +61,7 @@ To create a system hook: 1. Provide the **URL** and **Secret Token**. 1. Select the checkbox next to each optional **Trigger** you want to enable. 1. Select **Enable SSL verification**, if desired. -1. Click **Add system hook**. +1. Select **Add system hook**. ## Hooks request example diff --git a/doc/administration/timezone.md b/doc/administration/timezone.md index f4339263d34..93419751251 100644 --- a/doc/administration/timezone.md +++ b/doc/administration/timezone.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/troubleshooting/debug.md b/doc/administration/troubleshooting/debug.md index 0520ce470cb..81ca1bda5d0 100644 --- a/doc/administration/troubleshooting/debug.md +++ b/doc/administration/troubleshooting/debug.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -174,7 +174,7 @@ hearing back from the Unicorn worker. If the CPU spins to 100% while this in progress, there may be something taking longer than it should. To fix this issue, we first need to figure out what is happening. The -following tips are only recommended if you do NOT mind users being affected by +following tips are only recommended if you do not mind users being affected by downtime. Otherwise skip to the next section. 1. Load the problematic URL diff --git a/doc/administration/troubleshooting/defcon.md b/doc/administration/troubleshooting/defcon.md index a7cc47f8547..292b4b13967 100644 --- a/doc/administration/troubleshooting/defcon.md +++ b/doc/administration/troubleshooting/defcon.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference diff --git a/doc/administration/troubleshooting/diagnostics_tools.md b/doc/administration/troubleshooting/diagnostics_tools.md index 479fdb963cb..6055746238f 100644 --- a/doc/administration/troubleshooting/diagnostics_tools.md +++ b/doc/administration/troubleshooting/diagnostics_tools.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference diff --git a/doc/administration/troubleshooting/elasticsearch.md b/doc/administration/troubleshooting/elasticsearch.md index 97e625eb0a3..7ce09252680 100644 --- a/doc/administration/troubleshooting/elasticsearch.md +++ b/doc/administration/troubleshooting/elasticsearch.md @@ -1,13 +1,16 @@ --- -stage: Enablement +stage: Data Stores group: Global Search info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Troubleshooting Elasticsearch **(PREMIUM SELF)** -To install and configure Elasticsearch, and for common and known issues, -visit the [administrator documentation](../../integration/elasticsearch.md). +To install and configure Elasticsearch, +visit the [administrator documentation](../../integration/advanced_search/elasticsearch.md). + +For troubleshooting, visit the +[administrator troubleshooting documentation](../../integration/advanced_search/elasticsearch_troubleshooting.md). Troubleshooting Elasticsearch requires: @@ -219,7 +222,7 @@ The output from the last command is the key here. If it shows: If all the settings look correct and it is still not using Elasticsearch for the search function, it is best to escalate to GitLab support. This could be a bug/issue. -Moving past that, it is best to attempt the same [search via the Rails console](../../integration/elasticsearch.md#i-indexed-all-the-repositories-but-i-cant-get-any-hits-for-my-search-term-in-the-ui) +Moving past that, it is best to attempt the same [search via the Rails console](../../integration/advanced_search/elasticsearch_troubleshooting.md#i-indexed-all-the-repositories-but-i-cant-get-any-hits-for-my-search-term-in-the-ui) or the [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html), and compare the results from what you see in GitLab. @@ -242,7 +245,7 @@ The best place to start is to determine if the issue is with creating an empty i If it is, check on the Elasticsearch side to determine if the `gitlab-production` (the name for the GitLab index) exists. If it exists, manually delete it on the Elasticsearch side and attempt to recreate it from the -[`recreate_index`](../../integration/elasticsearch.md#gitlab-advanced-search-rake-tasks) +[`recreate_index`](../../integration/advanced_search/elasticsearch.md#gitlab-advanced-search-rake-tasks) Rake task. If you still encounter issues, try creating an index manually on the Elasticsearch @@ -257,12 +260,12 @@ during the indexing of projects. If errors do occur, they stem from either the i - On the GitLab side. You need to rectify those. If they are not something you are familiar with, contact GitLab support for guidance. -- Within the Elasticsearch instance itself. See if the error is [documented and has a fix](../../integration/elasticsearch.md#troubleshooting). If not, speak with your Elasticsearch administrator. +- Within the Elasticsearch instance itself. See if the error is [documented and has a fix](../../integration/advanced_search/elasticsearch_troubleshooting.md). If not, speak with your Elasticsearch administrator. If the indexing process does not present errors, check the status of the indexed projects. You can do this via the following Rake tasks: -- [`sudo gitlab-rake gitlab:elastic:index_projects_status`](../../integration/elasticsearch.md#gitlab-advanced-search-rake-tasks) (shows the overall status) -- [`sudo gitlab-rake gitlab:elastic:projects_not_indexed`](../../integration/elasticsearch.md#gitlab-advanced-search-rake-tasks) (shows specific projects that are not indexed) +- [`sudo gitlab-rake gitlab:elastic:index_projects_status`](../../integration/advanced_search/elasticsearch.md#gitlab-advanced-search-rake-tasks) (shows the overall status) +- [`sudo gitlab-rake gitlab:elastic:projects_not_indexed`](../../integration/advanced_search/elasticsearch.md#gitlab-advanced-search-rake-tasks) (shows specific projects that are not indexed) If: @@ -375,7 +378,7 @@ If you still encounter issues after retrying the migration, reach out to GitLab ## Common issues -All common issues [should be documented](../../integration/elasticsearch.md#troubleshooting). If not, +All common issues [should be documented](../../integration/advanced_search/elasticsearch_troubleshooting.md). If not, feel free to update that page with issues you encounter and solutions. ## Replication diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 34481582d8b..b57bc0a1119 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -296,18 +296,6 @@ namespace = Namespace.find_by_full_path("<new_namespace>") ::Projects::TransferService.new(p, current_user).execute(namespace) ``` -### For Removing webhooks that is getting timeout due to large webhook logs - -```ruby -# ID is the webhook_id -hook=WebHook.find(ID) - -WebHooks::DestroyService.new(current_user).execute(hook) - -#In case the service gets timeout consider removing webhook_logs -hook.web_hook_logs.limit(BATCH_SIZE).delete_all -``` - ### Bulk update service integration password for _all_ projects For example, change the Jira user's password for all projects that have the Jira @@ -760,7 +748,7 @@ parent.members_with_descendants.count # This section lists all the groups which are pending deletion # Group.all.each do |g| - if g.marked_for_deletion? + if g.marked_for_deletion? puts "Group ID: #{g.id}" puts "Group name: #{g.name}" puts "Group path: #{g.full_path}" @@ -1458,7 +1446,7 @@ Open the rails console (`gitlab rails c`) and run the following command to see a ApplicationSetting.last.attributes ``` -Among other attributes, the output contains all the settings available in the [Elasticsearch Integration page](../../integration/elasticsearch.md), such as `elasticsearch_indexing`, `elasticsearch_url`, `elasticsearch_replicas`, and `elasticsearch_pause_indexing`. +Among other attributes, the output contains all the settings available in the [Elasticsearch Integration page](../../integration/advanced_search/elasticsearch.md), such as `elasticsearch_indexing`, `elasticsearch_url`, `elasticsearch_replicas`, and `elasticsearch_pause_indexing`. #### Setting attributes @@ -1474,7 +1462,7 @@ ApplicationSetting.last.update(elasticsearch_indexing: false) #### Getting attributes -You can then check if the settings have been set in the [Elasticsearch Integration page](../../integration/elasticsearch.md) or in the rails console by issuing: +You can then check if the settings have been set in the [Elasticsearch Integration page](../../integration/advanced_search/elasticsearch.md) or in the rails console by issuing: ```ruby Gitlab::CurrentSettings.elasticsearch_url diff --git a/doc/administration/troubleshooting/index.md b/doc/administration/troubleshooting/index.md index bc9c39d57ea..7d40a9e9683 100644 --- a/doc/administration/troubleshooting/index.md +++ b/doc/administration/troubleshooting/index.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md index 290d6d9f21d..0c93d1ab3ee 100644 --- a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md +++ b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference diff --git a/doc/administration/troubleshooting/linux_cheat_sheet.md b/doc/administration/troubleshooting/linux_cheat_sheet.md index 913437c2d77..0245af39e45 100644 --- a/doc/administration/troubleshooting/linux_cheat_sheet.md +++ b/doc/administration/troubleshooting/linux_cheat_sheet.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference diff --git a/doc/administration/troubleshooting/log_parsing.md b/doc/administration/troubleshooting/log_parsing.md index 9aa490f73ef..4cc62c08f4f 100644 --- a/doc/administration/troubleshooting/log_parsing.md +++ b/doc/administration/troubleshooting/log_parsing.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md b/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md index 91db321295d..51ef3d95a4e 100644 --- a/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md +++ b/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/troubleshooting/postgresql.md b/doc/administration/troubleshooting/postgresql.md index d0ed3c5c12a..cdbf786bdb2 100644 --- a/doc/administration/troubleshooting/postgresql.md +++ b/doc/administration/troubleshooting/postgresql.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Data Stores group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -67,7 +67,7 @@ This section is for links to information elsewhere in the GitLab documentation. - Required extension: `btree_gist` - Errors like this in the `production/sidekiq` log; see: - [Set default_transaction_isolation into read committed](https://docs.gitlab.com/omnibus/settings/database.html#set-default_transaction_isolation-into-read-committed): + [Set `default_transaction_isolation` into read committed](https://docs.gitlab.com/omnibus/settings/database.html#set-default_transaction_isolation-into-read-committed): ```plaintext ActiveRecord::StatementInvalid PG::TRSerializationFailure: ERROR: could not serialize access due to concurrent update @@ -138,8 +138,12 @@ idle_in_transaction_session_timeout = 60s Quoting from issue [#30528](https://gitlab.com/gitlab-org/gitlab/-/issues/30528): +<!-- vale gitlab.FutureTense = NO --> + > "If a deadlock is hit, and we resolve it through aborting the transaction after a short period, then the retry mechanisms we already have will make the deadlocked piece of work try again, and it's unlikely we'll deadlock multiple times in a row." +<!-- vale gitlab.FutureTense = YES --> + NOTE: In Support, our general approach to reconfiguring timeouts (applies also to the HTTP stack) is that it's acceptable to do it temporarily as a workaround. If it @@ -148,9 +152,9 @@ problem more completely, implement a hot fix, or make some other change that addresses the root cause. Generally, the timeouts should be put back to reasonable defaults after the root cause is resolved. -In this case, the guidance we had from development was to drop deadlock_timeout -or statement_timeout, but to leave the third setting at 60s. Setting -idle_in_transaction protects the database from sessions potentially hanging for +In this case, the guidance we had from development was to drop `deadlock_timeout` +or `statement_timeout`, but to leave the third setting at 60 seconds. Setting +`idle_in_transaction` protects the database from sessions potentially hanging for days. There's more discussion in [the issue relating to introducing this timeout on GitLab.com](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1053). PostgresSQL defaults: @@ -161,7 +165,7 @@ PostgresSQL defaults: Comments in issue [#30528](https://gitlab.com/gitlab-org/gitlab/-/issues/30528) indicate that these should both be set to at least a number of minutes for all Omnibus GitLab installations (so they don't hang indefinitely). However, 15s -for statement_timeout is very short, and will only be effective if the +for `statement_timeout` is very short, and is only effective if the underlying infrastructure is very performant. See current settings with: diff --git a/doc/administration/troubleshooting/sidekiq.md b/doc/administration/troubleshooting/sidekiq.md index 7a64bcc9b87..40bfe22ac63 100644 --- a/doc/administration/troubleshooting/sidekiq.md +++ b/doc/administration/troubleshooting/sidekiq.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/troubleshooting/ssl.md b/doc/administration/troubleshooting/ssl.md index 83df9ba19ff..d5d50127ad5 100644 --- a/doc/administration/troubleshooting/ssl.md +++ b/doc/administration/troubleshooting/ssl.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference diff --git a/doc/administration/troubleshooting/test_environments.md b/doc/administration/troubleshooting/test_environments.md index 0821f952d61..94d17ba714e 100644 --- a/doc/administration/troubleshooting/test_environments.md +++ b/doc/administration/troubleshooting/test_environments.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference @@ -26,27 +26,8 @@ but contributions are welcome. ### GitLab -Please see [our Docker test environment docs](../../install/digitaloceandocker.md#create-new-gitlab-container) -for how to run GitLab on Docker. When spinning this up with `docker-machine`, ensure -you change a few things: - -1. Update the name of the `docker-machine` host. You can see a list of hosts - with `docker-machine ls`. -1. Expose the necessary ports using the `-p` flag. Docker normally doesn't - allow access to any ports it uses outside of the container, so they must be - explicitly exposed. -1. Add any necessary `gitlab.rb` configuration to the - `GITLAB_OMNIBUS_CONFIG` variable. - -For example, when the `docker-machine` host we want to use is `do-docker`: - -```shell -docker run --detach --name gitlab \ ---env GITLAB_OMNIBUS_CONFIG="external_url 'http://$(docker-machine ip do-docker)'; gitlab_rails['gitlab_shell_ssh_port'] = 2222;" \ ---hostname $(docker-machine ip do-docker) \ --p 80:80 -p 2222:22 \ -gitlab/gitlab-ee:11.5.3-ee.0 -``` +Please see [our official Docker installation method](../../install/docker.md) +for how to run GitLab on Docker. ### SAML diff --git a/doc/administration/troubleshooting/tracing_correlation_id.md b/doc/administration/troubleshooting/tracing_correlation_id.md index 3a0c6a30cde..fed3604057b 100644 --- a/doc/administration/troubleshooting/tracing_correlation_id.md +++ b/doc/administration/troubleshooting/tracing_correlation_id.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -34,7 +34,7 @@ To locate a relevant request and view its correlation ID: 1. Enable persistent logging in your network monitor. Some actions in GitLab will redirect you quickly after you submit a form, so this will help capture all relevant activity. 1. To help isolate the requests you are looking for, you can filter for `document` requests. -1. Click the request of interest to view further detail. +1. Select the request of interest to view further detail. 1. Go to the **Headers** section and look for **Response Headers**. There you should find an `x-request-id` header with a value that was randomly generated by GitLab for the request. diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index d66aa5945b6..e618c4787ab 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -1,5 +1,5 @@ --- -stage: Enablement +stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/whats-new.md b/doc/administration/whats-new.md index d3768907989..beb47254573 100644 --- a/doc/administration/whats-new.md +++ b/doc/administration/whats-new.md @@ -21,7 +21,7 @@ feature list, the feature list is tailored to your subscription type: ## Self-managed installations -Due to our release post process, the content for **What's new** is not yet finalized +Due to our release post process, the content for **What's new** is not finalized when a new version (`.0` release) is cut. The updated **What's new** is included in the first patch release, such as `13.10.1`. |