diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2020-09-19 04:45:44 +0300 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2020-09-19 04:45:44 +0300 |
| commit | 85dc423f7090da0a52c73eb66faf22ddb20efff9 (patch) | |
| tree | 9160f299afd8c80c038f08e1545be119f5e3f1e1 /doc/administration | |
| parent | 15c2c8c66dbe422588e5411eee7e68f1fa440bb8 (diff) | |
Add latest changes from gitlab-org/gitlab@13-4-stable-ee
Diffstat (limited to 'doc/administration')
79 files changed, 2311 insertions, 1472 deletions
diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index e7eab5a291e..099346b2b0b 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -68,7 +68,7 @@ From there, you can see the following actions: - Roles allowed to create project changed. - Group CI/CD variable added, removed, or protected status changed. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30857) in GitLab 13.3. -Group events can also be accessed via the [Group Audit Events API](../api/audit_events.md#group-audit-events-starter) +Group events can also be accessed via the [Group Audit Events API](../api/audit_events.md#group-audit-events) ### Project events **(STARTER)** @@ -96,8 +96,9 @@ From there, you can see the following actions: - Permission to approve merge requests by authors was updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7531) in GitLab 12.9) - Number of required approvals was updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7531) in GitLab 12.9) - Added or removed users and groups from project approval groups ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213603) in GitLab 13.2) +- Project CI/CD variable added, removed, or protected status changed. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30857) in GitLab 13.4. -Project events can also be accessed via the [Project Audit Events API](../api/audit_events.md#project-audit-events-starter) +Project events can also be accessed via the [Project Audit Events API](../api/audit_events.md#project-audit-events) ### Instance events **(PREMIUM ONLY)** @@ -132,7 +133,7 @@ the filter dropdown box. You can further filter by specific group, project, or u  -Instance events can also be accessed via the [Instance Audit Events API](../api/audit_events.md#instance-audit-events-premium-only) +Instance events can also be accessed via the [Instance Audit Events API](../api/audit_events.md#instance-audit-events) ### Missing events @@ -171,4 +172,77 @@ the steps bellow. ```ruby Feature.enable(:repository_push_audit_event) - ``` + +## Export to CSV **(PREMIUM ONLY)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1449) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.4. +> - It's [deployed behind a feature flag](../user/feature_flags.md), disabled by default. +> - It's disabled on GitLab.com. +> - It's not recommended for production use. +> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-audit-log-export-to-csv). **(PREMIUM ONLY)** + +CAUTION: **Warning:** +This feature might not be available to you. Check the **version history** note above for details. + +Export to CSV allows customers to export the current filter view of your audit log as a +CSV file, +which stores tabular data in plain text. The data provides a comprehensive view with respect to +audit events. + +To export the Audit Log to CSV, navigate to +**{monitor}** **Admin Area > Monitoring > Audit Log** + +1. Click in the field **Search**. +1. In the dropdown menu that appears, select the event type that you want to filter by. +1. Select the preferred date range. +1. Click **Export as CSV**. + + + +### Sort + +Exported events are always sorted by `ID` in ascending order. + +### Format + +Data is encoded with a comma as the column delimiter, with `"` used to quote fields if needed, and newlines to separate rows. +The first row contains the headers, which are listed in the following table along with a description of the values: + +| Column | Description | +|---------|-------------| +| ID | Audit event `id` | +| Author ID | ID of the author | +| Author Name | Full name of the author | +| Entity ID | ID of the scope | +| Entity Type | Type of the entity (`Project`/`Group`/`User`) | +| Entity Path | Path of the entity | +| Target ID | ID of the target | +| Target Type | Type of the target | +| Target Details | Details of the target | +| Action | Description of the action | +| IP Address | IP address of the author who performed the action | +| Created At (UTC) | Formatted as `YYYY-MM-DD HH:MM:SS` | + +### Limitation + +The Audit Log CSV file size is limited to a maximum of `15 MB`. +The remaining records are truncated when this limit is reached. + +### Enable or disable Audit Log Export to CSV + +The Audit Log Export to CSV is under development and not ready for production use. It is +deployed behind a feature flag that is **disabled by default**. +[GitLab administrators with access to the GitLab Rails console](../administration/feature_flags.md) +can enable it. + +To enable it: + +```ruby +Feature.enable(:audit_log_export_csv) +``` + +To disable it: + +```ruby +Feature.disable(:audit_log_export_csv) +``` diff --git a/doc/administration/audit_reports.md b/doc/administration/audit_reports.md index d5a08b711be..83fbeda26aa 100644 --- a/doc/administration/audit_reports.md +++ b/doc/administration/audit_reports.md @@ -1,6 +1,7 @@ --- stage: Manage group: Compliance +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers description: 'Learn how to create evidence artifacts typically requested by a 3rd party auditor.' --- @@ -8,7 +9,7 @@ description: 'Learn how to create evidence artifacts typically requested by a 3r GitLab can help owners and administrators respond to auditors by generating comprehensive reports. These **Audit Reports** vary in scope, depending on the -need: +needs. ## Use cases @@ -26,6 +27,3 @@ need: - `https://docs.gitlab.com/ee/administration/audit_events.html` - `https://docs.gitlab.com/ee/administration/logs.html` - -We plan on making Audit Events [downloadable as a CSV](https://gitlab.com/gitlab-org/gitlab/-/issues/1449) -in the near future. diff --git a/doc/administration/auth/README.md b/doc/administration/auth/README.md index 60e1dfb4637..926a4abab7d 100644 --- a/doc/administration/auth/README.md +++ b/doc/administration/auth/README.md @@ -11,6 +11,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w GitLab integrates with the following external authentication and authorization providers: +- [Atlassian](atlassian.md) - [Auth0](../../integration/auth0.md) - [Authentiq](authentiq.md) - [AWS Cognito](cognito.md) diff --git a/doc/administration/auth/atlassian.md b/doc/administration/auth/atlassian.md new file mode 100644 index 00000000000..3a1f5eeb0c2 --- /dev/null +++ b/doc/administration/auth/atlassian.md @@ -0,0 +1,86 @@ +--- +type: reference +stage: Manage +group: Access +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# Atlassian OmniAuth Provider + +To enable the Atlassian OmniAuth provider for passwordless authentication you must register an application with Atlassian. + +## Atlassian application registration + +1. Go to <https://developer.atlassian.com/apps/> and sign-in with the Atlassian + account that will administer the application. + +1. Click **Create a new app**. + +1. Choose an App Name, such as 'GitLab', and click **Create**. + +1. Note the `Client ID` and `Secret` for the [GitLab configuration](#gitlab-configuration) steps. + +1. In the left sidebar under **APIS AND FEATURES**, click **OAuth 2.0 (3LO)**. + +1. Enter the GitLab callback URL using the format `https://gitlab.example.com/users/auth/atlassian_oauth2/callback` and click **Save changes**. + +1. Click **+ Add** in the left sidebar under **APIS AND FEATURES**. + +1. Click **Add** for **Jira platform REST API** and then **Configure**. + +1. Click **Add** next to the following scopes: + - **View Jira issue data** + - **View user profiles** + - **Create and manage issues** + +## GitLab configuration + +1. On your GitLab server, open the configuration file: + + For Omnibus GitLab installations: + + ```shell + sudo editor /etc/gitlab/gitlab.rb + ``` + + For installations from source: + + ```shell + sudo -u git -H editor /home/git/gitlab/config/gitlab.yml + ``` + +1. See [Initial OmniAuth Configuration](../../integration/omniauth.md#initial-omniauth-configuration) for initial settings to enable single sign-on and add `atlassian_oauth2` as an OAuth provider. + +1. Add the provider configuration for Atlassian: + + For Omnibus GitLab installations: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + name: "atlassian_oauth2", + app_id: "YOUR_CLIENT_ID", + app_secret: "YOUR_CLIENT_SECRET", + args: { scope: 'offline_access read:jira-user read:jira-work', prompt: 'consent' } + } + ] + ``` + + For installations from source: + + ```yaml + - name: "atlassian_oauth2", + app_id: "YOUR_CLIENT_ID", + app_secret: "YOUR_CLIENT_SECRET", + args: { scope: 'offline_access read:jira-user read:jira-work', prompt: 'consent' } + ``` + +1. Change `YOUR_CLIENT_ID` and `YOUR_CLIENT_SECRET` to the Client credentials you received in [application registration](#atlassian-application-registration) steps. + +1. Save the configuration file. + +1. [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) or [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect if you installed GitLab via Omnibus or from source respectively. + +On the sign-in page there should now be an Atlassian icon below the regular sign in form. Click the icon to begin the authentication process. + +If everything goes right, the user is signed in to GitLab using their Atlassian credentials. diff --git a/doc/administration/auth/cognito.md b/doc/administration/auth/cognito.md index b4df6446835..e777acd0d32 100644 --- a/doc/administration/auth/cognito.md +++ b/doc/administration/auth/cognito.md @@ -37,7 +37,7 @@ The following steps enable AWS Cognito as an authentication provider: 1. Save changes for the app client settings. 1. Under **Domain name** include the AWS domain name for your AWS Cognito application. -1. Under **App Clients**, find your **App client id** and **App client secret**. These values correspond to the OAuth2 Client ID and Client Secret. Save these values. +1. Under **App Clients**, find your app client ID and app client secret. These values correspond to the OAuth2 Client ID and Client Secret. Save these values. ## Configure GitLab diff --git a/doc/administration/auth/ldap/google_secure_ldap.md b/doc/administration/auth/ldap/google_secure_ldap.md index 1f8fca33811..90f0e681dd1 100644 --- a/doc/administration/auth/ldap/google_secure_ldap.md +++ b/doc/administration/auth/ldap/google_secure_ldap.md @@ -35,7 +35,7 @@ The steps below cover: credentials' and 'Read user information'. Select 'Add LDAP Client' TIP: **Tip:** - If you plan to use GitLab [LDAP Group Sync](index.md#group-sync-starter-only) + If you plan to use GitLab [LDAP Group Sync](index.md#group-sync) , turn on 'Read group information'.  diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index 548e734c931..1dac098ec0c 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -16,6 +16,8 @@ This integration works with most LDAP-compliant directory servers, including: - Open LDAP - 389 Server +Users added through LDAP take a [licensed seat](../../../subscriptions/self_managed/index.md#choose-the-number-of-users). + GitLab Enterprise Editions (EE) include enhanced integration, including group membership syncing as well as multiple LDAP servers support. @@ -35,7 +37,7 @@ GitLab assumes that LDAP users: - Are not able to change their LDAP `mail`, `email`, or `userPrincipalName` attributes. An LDAP user who is allowed to change their email on the LDAP server can potentially - [take over any account](#enabling-ldap-sign-in-for-existing-gitlab-users-core-only) + [take over any account](#enabling-ldap-sign-in-for-existing-gitlab-users) on your GitLab server. - Have unique email addresses, otherwise it is possible for LDAP users with the same email address to share the same GitLab account. @@ -55,7 +57,7 @@ immediately block all access. NOTE: **Note:** GitLab Enterprise Edition Starter supports a -[configurable sync time](#adjusting-ldap-user-sync-schedule-starter-only). +[configurable sync time](#adjusting-ldap-user-sync-schedule). ## Git password authentication **(CORE ONLY)** @@ -338,7 +340,7 @@ sync, while also allowing your SAML identity provider to handle additional checks like custom 2FA. When LDAP web sign in is disabled, users will not see a **LDAP** tab on the sign in page. -This does not disable [using LDAP credentials for Git access](#git-password-authentication-core-only). +This does not disable [using LDAP credentials for Git access](#git-password-authentication). **Omnibus configuration** @@ -389,7 +391,7 @@ that your GitLab instance will connect to. To add another LDAP server: -1. Duplicate the settings under [the main configuration](#configuration-core-only). +1. Duplicate the settings under [the main configuration](#configuration). 1. Edit them to match the additional LDAP server. Be sure to choose a different provider ID made of letters a-z and numbers 0-9. @@ -544,11 +546,11 @@ following. 1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. To take advantage of group sync, group owners or maintainers will need to [create one -or more LDAP group links](#adding-group-links-starter-only). +or more LDAP group links](#adding-group-links). ### Adding group links **(STARTER ONLY)** -For information on adding group links via CNs and filters, refer to [the GitLab groups documentation](../../../user/group/index.md#manage-group-memberships-via-ldap-starter-only). +For information on adding group links via CNs and filters, refer to [the GitLab groups documentation](../../../user/group/index.md#manage-group-memberships-via-ldap). ### Administrator sync **(STARTER ONLY)** @@ -609,7 +611,7 @@ When enabled, the following applies: To enable it you need to: -1. [Enable LDAP](#configuration-core-only) +1. [Enable LDAP](#configuration) 1. Navigate to **(admin)** **Admin Area > Settings -> Visibility and access controls**. 1. Make sure the "Lock memberships to LDAP synchronization" checkbox is enabled. @@ -657,7 +659,7 @@ sync to run once every 2 hours at the top of the hour. ### External groups **(STARTER ONLY)** Using the `external_groups` setting will allow you to mark all users belonging -to these groups as [external users](../../../user/permissions.md#external-users-core-only). +to these groups as [external users](../../../user/permissions.md#external-users). Group membership is checked periodically through the `LdapGroupSync` background task. diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md index 75183f54990..3d3ac124ac4 100644 --- a/doc/administration/auth/ldap/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap/ldap-troubleshooting.md @@ -56,7 +56,7 @@ main: # 'main' is the GitLab 'provider ID' of this LDAP server The following allows you to perform a search in LDAP using the rails console. Depending on what you're trying to do, it may make more sense to query [a -user](#query-a-user-in-ldap) or [a group](#query-a-group-in-ldap-starter-only) directly, or +user](#query-a-user-in-ldap) or [a group](#query-a-group-in-ldap) directly, or even [use `ldapsearch`](#ldapsearch) instead. ```ruby @@ -90,8 +90,8 @@ established but GitLab doesn't show you LDAP users in the output, one of the following is most likely true: - The `bind_dn` user doesn't have enough permissions to traverse the user tree. -- The user(s) don't fall under the [configured `base`](index.md#configuration-core-only). -- The [configured `user_filter`](index.md#set-up-ldap-user-filter-core-only) blocks access to the user(s). +- The user(s) don't fall under the [configured `base`](index.md#configuration). +- The [configured `user_filter`](index.md#set-up-ldap-user-filter) blocks access to the user(s). In this case, you con confirm which of the above is true using [ldapsearch](#ldapsearch) with the existing LDAP configuration in your @@ -102,9 +102,9 @@ In this case, you con confirm which of the above is true using A user can have trouble signing in for any number of reasons. To get started, here are some questions to ask yourself: -- Does the user fall under the [configured `base`](index.md#configuration-core-only) in +- Does the user fall under the [configured `base`](index.md#configuration) in LDAP? The user must fall under this `base` to sign-in. -- Does the user pass through the [configured `user_filter`](index.md#set-up-ldap-user-filter-core-only)? +- Does the user pass through the [configured `user_filter`](index.md#set-up-ldap-user-filter)? If one is not configured, this question can be ignored. If it is, then the user must also pass through this filter to be allowed to sign-in. - Refer to our docs on [debugging the `user_filter`](#debug-ldap-user-filter). @@ -122,7 +122,7 @@ If the logs don't lead to the root of the problem, use the to see if GitLab can read this user on the LDAP server. It can also be helpful to -[debug a user sync](#sync-all-users-starter-only) to +[debug a user sync](#sync-all-users) to investigate further. #### Invalid credentials on sign-in @@ -136,6 +136,27 @@ are true for the user in question: - Run [an LDAP check command](#ldap-check) to make sure that the LDAP settings are correct and [GitLab can see your users](#no-users-are-found). +#### Access denied for your LDAP account + +There is [a bug](https://gitlab.com/gitlab-org/gitlab/-/issues/235930) that +may affect users with [Auditor level access](../../auditor_users.md). When +downgrading from Premium/Ultimate, Auditor users who try to sign in +may see the following message: `Access denied for your LDAP account`. + +We have a workaround, based on toggling the access level of affected users: + +1. As an administrator, go to **Admin Area > Overview > Users**. +1. Select the name of the affected user. +1. In the user's administrative page, press **Edit** on the top right of the page. +1. Change the user's access level from **Regular** to **Admin** (or vice versa), + and press **Save changes** at the bottom of the page. +1. Press **Edit** on the top right of the user's profile page + again. +1. Restore the user's original access level (**Regular** or **Admin**) + and press **Save changes** again. + +The user should now be able to sign in. + #### Email has already been taken A user tries to sign-in with the correct LDAP credentials, is denied access, @@ -175,7 +196,7 @@ profile](../../../user/profile/index.md#user-profile) or an admin can do it. #### Debug LDAP user filter [`ldapsearch`](#ldapsearch) allows you to test your configured -[user filter](index.md#set-up-ldap-user-filter-core-only) +[user filter](index.md#set-up-ldap-user-filter) to confirm that it returns the users you expect it to return. ```shell @@ -191,7 +212,7 @@ ldapsearch -H ldaps://$host:$port -D "$bind_dn" -y bind_dn_password.txt -b "$ba #### Sync all users **(STARTER ONLY)** -The output from a manual [user sync](index.md#user-sync-starter-only) can show you what happens when +The output from a manual [user sync](index.md#user-sync) can show you what happens when GitLab tries to sync its users against LDAP. Enter the [rails console](#rails-console) and then run: @@ -202,11 +223,11 @@ LdapSyncWorker.new.perform ``` Next, [learn how to read the -output](#example-console-output-after-a-user-sync-starter-only). +output](#example-console-output-after-a-user-sync). ##### Example console output after a user sync **(STARTER ONLY)** -The output from a [manual user sync](#sync-all-users-starter-only) will be very verbose, and a +The output from a [manual user sync](#sync-all-users) will be very verbose, and a single user's successful sync can look like this: ```shell @@ -304,9 +325,9 @@ LDAP group sync, but for some reason it's not happening. There are several things to check to debug the situation. - Ensure LDAP configuration has a `group_base` specified. - [This configuration](index.md#group-sync-starter-only) is required for group sync to work properly. + [This configuration](index.md#group-sync) is required for group sync to work properly. - Ensure the correct [LDAP group link is added to the GitLab - group](index.md#adding-group-links-starter-only). + group](index.md#adding-group-links). - Check that the user has an LDAP identity: 1. Sign in to GitLab as an administrator user. 1. Navigate to **Admin area -> Users**. @@ -316,10 +337,10 @@ things to check to debug the situation. an LDAP DN as the 'Identifier'. If not, this user hasn't signed in with LDAP yet and must do so first. - You've waited an hour or [the configured - interval](index.md#adjusting-ldap-group-sync-schedule-starter-only) for the group to + interval](index.md#adjusting-ldap-group-sync-schedule) for the group to sync. To speed up the process, either go to the GitLab group **Settings -> Members** and press **Sync now** (sync one group) or [run the group sync Rake - task](../../raketasks/ldap.md#run-a-group-sync-starter-only) (sync all groups). + task](../../raketasks/ldap.md#run-a-group-sync) (sync all groups). If all of the above looks good, jump in to a little more advanced debugging in the rails console. @@ -327,23 +348,23 @@ the rails console. 1. Enter the [rails console](#rails-console). 1. Choose a GitLab group to test with. This group should have an LDAP group link already configured. -1. [Enable debug logging, find the above GitLab group, and sync it with LDAP](#sync-one-group-starter-only). +1. [Enable debug logging, find the above GitLab group, and sync it with LDAP](#sync-one-group). 1. Look through the output of the sync. See [example log - output](#example-console-output-after-a-group-sync-starter-only) + output](#example-console-output-after-a-group-sync) for how to read the output. 1. If you still aren't able to see why the user isn't being added, [query the - LDAP group directly](#query-a-group-in-ldap-starter-only) to see what members are listed. + LDAP group directly](#query-a-group-in-ldap) to see what members are listed. 1. Is the user's DN or UID in one of the lists from the above output? One of the DNs or UIDs here should match the 'Identifier' from the LDAP identity checked earlier. If it doesn't, the user does not appear to be in the LDAP group. #### Admin privileges not granted -When [Administrator sync](index.md#administrator-sync-starter-only) has been configured +When [Administrator sync](index.md#administrator-sync) has been configured but the configured users aren't granted the correct admin privileges, confirm the following are true: -- A [`group_base` is also configured](index.md#group-sync-starter-only). +- A [`group_base` is also configured](index.md#group-sync). - The configured `admin_group` in the `gitlab.rb` is a CN, rather than a DN or an array. - This CN falls under the scope of the configured `group_base`. - The members of the `admin_group` have already signed into GitLab with their LDAP @@ -351,17 +372,17 @@ the following are true: accounts are already connected to LDAP. If all the above are true and the users are still not getting access, [run a manual -group sync](#sync-all-groups-starter-only) in the rails console and [look through the -output](#example-console-output-after-a-group-sync-starter-only) to see what happens when +group sync](#sync-all-groups) in the rails console and [look through the +output](#example-console-output-after-a-group-sync) to see what happens when GitLab syncs the `admin_group`. #### Sync all groups **(STARTER ONLY)** NOTE: **Note:** To sync all groups manually when debugging is unnecessary, [use the Rake -task](../../raketasks/ldap.md#run-a-group-sync-starter-only) instead. +task](../../raketasks/ldap.md#run-a-group-sync) instead. -The output from a manual [group sync](index.md#group-sync-starter-only) can show you what happens +The output from a manual [group sync](index.md#group-sync) can show you what happens when GitLab syncs its LDAP group memberships against LDAP. ```ruby @@ -371,12 +392,12 @@ LdapAllGroupsSyncWorker.new.perform ``` Next, [learn how to read the -output](#example-console-output-after-a-group-sync-starter-only). +output](#example-console-output-after-a-group-sync). ##### Example console output after a group sync **(STARTER ONLY)** Like the output from the user sync, the output from the [manual group -sync](#sync-all-groups-starter-only) will also be very verbose. However, it contains lots +sync](#sync-all-groups) will also be very verbose. However, it contains lots of helpful information. Indicates the point where syncing actually begins: @@ -456,7 +477,7 @@ this line will indicate the sync is finished: Finished syncing admin users for 'ldapmain' provider ``` -If [admin sync](index.md#administrator-sync-starter-only) is not configured, you'll see a message +If [admin sync](index.md#administrator-sync) is not configured, you'll see a message stating as such: ```shell @@ -465,7 +486,7 @@ No `admin_group` configured for 'ldapmain' provider. Skipping #### Sync one group **(STARTER ONLY)** -[Syncing all groups](#sync-all-groups-starter-only) can produce a lot of noise in the output, which can be +[Syncing all groups](#sync-all-groups) can produce a lot of noise in the output, which can be distracting when you're only interested in troubleshooting the memberships of a single GitLab group. In that case, here's how you can just sync this group and see its debug output: @@ -483,7 +504,7 @@ EE::Gitlab::Auth::Ldap::Sync::Group.execute_all_providers(group) ``` The output will be similar to -[that you'd get from syncing all groups](#example-console-output-after-a-group-sync-starter-only). +[that you'd get from syncing all groups](#example-console-output-after-a-group-sync). #### Query a group in LDAP **(STARTER ONLY)** @@ -541,7 +562,7 @@ emails.each do |username, email| end ``` -You can then [run a UserSync](#sync-all-users-starter-only) **(STARTER ONLY)** to sync the latest DN +You can then [run a UserSync](#sync-all-users) **(STARTER ONLY)** to sync the latest DN for each of these users. ## Debugging Tools diff --git a/doc/administration/compliance.md b/doc/administration/compliance.md index 47e4b6c068c..237261f2567 100644 --- a/doc/administration/compliance.md +++ b/doc/administration/compliance.md @@ -11,10 +11,10 @@ GitLab’s [security features](../security/README.md) may also help you meet rel |**[Enforce TOS acceptance](../user/admin_area/settings/terms.md)**<br>Enforce your users accepting new terms of service by blocking GitLab traffic.|Core+|| |**[Email all users of a project, group, or entire server](../user/admin_area/settings/terms.md)**<br>An admin can email groups of users based on project or group membership, or email everyone using the GitLab instance. This is great for scheduled maintenance or upgrades.|Starter+|| |**[Omnibus package supports log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-forwarding)**<br>Forward your logs to a central system.|Starter+|| -|**[Lock project membership to group](../user/group/index.md#member-lock-starter)**<br>Group owners can prevent new members from being added to projects within a group.|Starter+|✓| -|**[LDAP group sync](auth/ldap/index.md#group-sync-starter-only)**<br>GitLab Enterprise Edition gives admins the ability to automatically sync groups and manage SSH keys, permissions, and authentication, so you can focus on building your product, not configuring your tools.|Starter+|| -|**[LDAP group sync filters](auth/ldap/index.md#group-sync-starter-only)**<br>GitLab Enterprise Edition Premium gives more flexibility to synchronize with LDAP based on filters, meaning you can leverage LDAP attributes to map GitLab permissions.|Premium+|| +|**[Lock project membership to group](../user/group/index.md#member-lock)**<br>Group owners can prevent new members from being added to projects within a group.|Starter+|✓| +|**[LDAP group sync](auth/ldap/index.md#group-sync)**<br>GitLab Enterprise Edition gives admins the ability to automatically sync groups and manage SSH keys, permissions, and authentication, so you can focus on building your product, not configuring your tools.|Starter+|| +|**[LDAP group sync filters](auth/ldap/index.md#group-sync)**<br>GitLab Enterprise Edition Premium gives more flexibility to synchronize with LDAP based on filters, meaning you can leverage LDAP attributes to map GitLab permissions.|Premium+|| |**[Audit logs](audit_events.md)**<br>To maintain the integrity of your code, GitLab Enterprise Edition Premium gives admins the ability to view any modifications made within the GitLab server in an advanced audit log system, so you can control, analyze, and track every change.|Premium+|| |**[Auditor users](auditor_users.md)**<br>Auditor users are users who are given read-only access to all projects, groups, and other resources on the GitLab instance.|Premium+|| |**[Credentials inventory](../user/admin_area/credentials_inventory.md)**<br>With a credentials inventory, GitLab administrators can keep track of the credentials used by all of the users in their GitLab instance. |Ultimate|| -|**Separation of Duties using [Protected branches](../user/project/protected_branches.md#protected-branches-approval-by-code-owners-premium) and [custom CI Configuration Paths](../ci/pipelines/settings.md#custom-ci-configuration-path)**<br> GitLab Silver and Premium users can leverage GitLab's cross-project YAML configuration's to define deployers of code and developers of code. View the [Separation of Duties Deploy Project](https://gitlab.com/guided-explorations/separation-of-duties-deploy/blob/master/README.md) and [Separation of Duties Project](https://gitlab.com/guided-explorations/separation-of-duties/blob/master/README.md) to see how to use this set up to define these roles.|Premium+|| +|**Separation of Duties using [Protected branches](../user/project/protected_branches.md#protected-branches-approval-by-code-owners) and [custom CI Configuration Paths](../ci/pipelines/settings.md#custom-ci-configuration-path)**<br> GitLab Silver and Premium users can leverage GitLab's cross-project YAML configuration's to define deployers of code and developers of code. View the [Separation of Duties Deploy Project](https://gitlab.com/guided-explorations/separation-of-duties-deploy/blob/master/README.md) and [Separation of Duties Project](https://gitlab.com/guided-explorations/separation-of-duties/blob/master/README.md) to see how to use this set up to define these roles.|Premium+|| diff --git a/doc/administration/database_load_balancing.md b/doc/administration/database_load_balancing.md index 0f566fcc114..36ef905fd90 100644 --- a/doc/administration/database_load_balancing.md +++ b/doc/administration/database_load_balancing.md @@ -221,7 +221,7 @@ without it immediately leading to errors being presented to the users. ## Logging The load balancer logs various events in -[`database_load_balancing.log`](logs.md#database_load_balancinglog-premium-only), such as +[`database_load_balancing.log`](logs.md#database_load_balancinglog), such as - When a host is marked as offline - When a host comes back online diff --git a/doc/administration/environment_variables.md b/doc/administration/environment_variables.md index ebc3848017f..d48a47e9645 100644 --- a/doc/administration/environment_variables.md +++ b/doc/administration/environment_variables.md @@ -32,7 +32,7 @@ Variable | Type | Description `GITLAB_EMAIL_SUBJECT_SUFFIX` | string | The e-mail subject suffix used in e-mails sent by GitLab `GITLAB_UNICORN_MEMORY_MIN` | integer | The minimum memory threshold (in bytes) for the Unicorn worker killer `GITLAB_UNICORN_MEMORY_MAX` | integer | The maximum memory threshold (in bytes) for the Unicorn worker killer -`GITLAB_SHARED_RUNNERS_REGISTRATION_TOKEN` | string | Sets the initial registration token used for GitLab Runners +`GITLAB_SHARED_RUNNERS_REGISTRATION_TOKEN` | string | Sets the initial registration token used for runners `UNSTRUCTURED_RAILS_LOG` | string | Enables the unstructured log in addition to JSON logs (defaults to `true`) ## Complete database variables diff --git a/doc/administration/feature_flags.md b/doc/administration/feature_flags.md index 7cc7692975a..a8a14063f26 100644 --- a/doc/administration/feature_flags.md +++ b/doc/administration/feature_flags.md @@ -1,7 +1,7 @@ --- -stage: Release -group: Progressive Delivery -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +stage: none +group: Development +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" type: reference description: "GitLab administrator: enable and disable GitLab features deployed behind feature flags" --- diff --git a/doc/administration/file_hooks.md b/doc/administration/file_hooks.md index c0b31769e7f..a6610188381 100644 --- a/doc/administration/file_hooks.md +++ b/doc/administration/file_hooks.md @@ -71,9 +71,9 @@ Below is an example that will only response on the event `project_create` and will inform the admins from the GitLab instance that a new project has been created. ```ruby +#!/opt/gitlab/embedded/bin/ruby # By using the embedded ruby version we eliminate the possibility that our chosen language # would be unavailable from -#!/opt/gitlab/embedded/bin/ruby require 'json' require 'mail' diff --git a/doc/administration/geo/disaster_recovery/bring_primary_back.md b/doc/administration/geo/disaster_recovery/bring_primary_back.md index 3b7c7fd549c..83081e2cef6 100644 --- a/doc/administration/geo/disaster_recovery/bring_primary_back.md +++ b/doc/administration/geo/disaster_recovery/bring_primary_back.md @@ -21,7 +21,7 @@ If you have any doubts about the consistency of the data on this node, we recomm Since the former **primary** node will be out of sync with the current **primary** node, the first step is to bring the former **primary** node up to date. Note, deletion of data stored on disk like repositories and uploads will not be replayed when bringing the former **primary** node back into sync, which may result in increased disk usage. -Alternatively, you can [set up a new **secondary** GitLab instance](../replication/index.md#setup-instructions) to avoid this. +Alternatively, you can [set up a new **secondary** GitLab instance](../setup/index.md) to avoid this. To bring the former **primary** node up to date: @@ -37,7 +37,7 @@ To bring the former **primary** node up to date: you need to undo those steps now. For Debian/Ubuntu you just need to run `sudo systemctl enable gitlab-runsvdir`. For CentOS 6, you need to install the GitLab instance from scratch and set it up as a **secondary** node by - following [Setup instructions](../replication/index.md#setup-instructions). In this case, you don't need to follow the next step. + following [Setup instructions](../setup/index.md). In this case, you don't need to follow the next step. NOTE: **Note:** If you [changed the DNS records](index.md#step-4-optional-updating-the-primary-domain-dns-record) @@ -45,12 +45,12 @@ To bring the former **primary** node up to date: all the writes to this node](planned_failover.md#prevent-updates-to-the-primary-node) during this procedure. -1. [Setup database replication](../replication/database.md). Note that in this +1. [Setup database replication](../setup/database.md). Note that in this case, **primary** node refers to the current **primary** node, and **secondary** node refers to the former **primary** node. If you have lost your original **primary** node, follow the -[setup instructions](../replication/index.md#setup-instructions) to set up a new **secondary** node. +[setup instructions](../setup/index.md) to set up a new **secondary** node. ## Promote the **secondary** node to **primary** node diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index 2d837ebb369..8862776ee1b 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -11,11 +11,13 @@ Geo replicates your database, your Git repositories, and few other assets. We will support and replicate more data in the future, that will enable you to failover with minimal effort, in a disaster situation. -See [Geo current limitations](../replication/index.md#current-limitations) for more information. +See [Geo current limitations](../index.md#current-limitations) for more information. CAUTION: **Warning:** Disaster recovery for multi-secondary configurations is in **Alpha**. -For the latest updates, check the multi-secondary [Disaster Recovery epic](https://gitlab.com/groups/gitlab-org/-/epics/65). +For the latest updates, check the [Disaster Recovery epic for complete maturity](https://gitlab.com/groups/gitlab-org/-/epics/590). +Multi-secondary configurations require the complete re-synchronization and re-configuration of all non-promoted secondaries and +will cause downtime. ## Promoting a **secondary** Geo node in single-secondary configurations @@ -96,6 +98,10 @@ must disable the **primary** node. Note the following when promoting a secondary: +- If replication was paused on the secondary node, for example as a part of upgrading, + while you were running a version of GitLab lower than 13.4, you _must_ + [enable the node via the database](../replication/troubleshooting.md#while-promoting-the-secondary-i-got-an-error-activerecordrecordinvalid) + before proceeding. - A new **secondary** should not be added at this time. If you want to add a new **secondary**, do this after you have completed the entire process of promoting the **secondary** to the **primary**. @@ -123,28 +129,23 @@ Note the following when promoting a secondary: ``` 1. Promote the **secondary** node to the **primary** node. - - Before promoting a secondary node to primary, preflight checks should be run. They can be run separately or along with the promotion script. - + To promote the secondary node to primary along with preflight checks: ```shell gitlab-ctl promote-to-primary-node ``` - CAUTION: **Warning:** - Skipping preflight checks will promote the secondary to a primary without any further confirmation! - - If you have already run the [preflight checks](planned_failover.md#preflight-checks) or don't want to run them, you can skip preflight checks with: + If you have already run the [preflight checks](planned_failover.md#preflight-checks) separately or don't want to run them, you can skip preflight checks with: ```shell gitlab-ctl promote-to-primary-node --skip-preflight-check ``` - You can also run preflight checks separately: + You can also promote the secondary node to primary **without any further confirmation**, even when preflight checks fail: ```shell - gitlab-ctl promotion-preflight-checks + gitlab-ctl promote-to-primary-node --force ``` 1. Verify you can connect to the newly promoted **primary** node using the URL used @@ -321,7 +322,7 @@ secondary domain, like changing Git remotes and API URLs. Promoting a **secondary** node to **primary** node using the process above does not enable Geo on the new **primary** node. -To bring a new **secondary** node online, follow the [Geo setup instructions](../replication/index.md#setup-instructions). +To bring a new **secondary** node online, follow the [Geo setup instructions](../index.md#setup-instructions). ### Step 6. (Optional) Removing the secondary's tracking database @@ -374,7 +375,7 @@ and after that you also need two extra steps. gitlab_rails['auto_migrate'] = false ``` - (For more details about these settings you can read [Configure the primary server](../replication/database.md#step-1-configure-the-primary-server)) + (For more details about these settings you can read [Configure the primary server](../setup/database.md#step-1-configure-the-primary-server)) 1. Save the file and reconfigure GitLab for the database listen changes and the replication slot changes to be applied. @@ -407,21 +408,9 @@ and after that you also need two extra steps. ### Step 2. Initiate the replication process Now we need to make each **secondary** node listen to changes on the new **primary** node. To do that you need -to [initiate the replication process](../replication/database.md#step-3-initiate-the-replication-process) again but this time +to [initiate the replication process](../setup/database.md#step-3-initiate-the-replication-process) again but this time for another **primary** node. All the old replication settings will be overwritten. ## Troubleshooting -### I followed the disaster recovery instructions and now two-factor auth is broken - -The setup instructions for Geo prior to 10.5 failed to replicate the -`otp_key_base` secret, which is used to encrypt the two-factor authentication -secrets stored in the database. If it differs between **primary** and **secondary** -nodes, users with two-factor authentication enabled won't be able to log in -after a failover. - -If you still have access to the old **primary** node, you can follow the -instructions in the -[Upgrading to GitLab 10.5](../replication/version_specific_updates.md#updating-to-gitlab-105) -section to resolve the error. Otherwise, the secret is lost and you'll need to -[reset two-factor authentication for all users](../../../security/two_factor_authentication.md#disabling-2fa-for-everyone). +This section was moved to [another location](../replication/troubleshooting.md#fixing-errors-during-a-failover-or-when-promoting-a-secondary-to-a-primary-node). diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index a0cf263a762..9b9c386652c 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -27,7 +27,7 @@ have a high degree of confidence in being able to perform them accurately. ## Not all data is automatically replicated -If you are using any GitLab features that Geo [doesn't support](../replication/index.md#current-limitations), +If you are using any GitLab features that Geo [doesn't support](../index.md#current-limitations), you must make separate provisions to ensure that the **secondary** node has an up-to-date copy of any data associated with that feature. This may extend the required scheduled maintenance period significantly. @@ -51,12 +51,6 @@ Run this command to list out all preflight checks and automatically check if rep gitlab-ctl promotion-preflight-checks ``` -You can run this command in `force` mode to promote to primary even if preflight checks fail: - -```shell -sudo gitlab-ctl promotion-preflight-checks --force -``` - Each step is described in more detail below. ### Object storage diff --git a/doc/administration/geo/disaster_recovery/promotion_runbook.md b/doc/administration/geo/disaster_recovery/promotion_runbook.md new file mode 100644 index 00000000000..fb2353513df --- /dev/null +++ b/doc/administration/geo/disaster_recovery/promotion_runbook.md @@ -0,0 +1,269 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: howto +--- + +CAUTION: **Caution:** +This runbook is in **alpha**. For complete, production-ready documentation, see the +[disaster recovery documentation](index.md). + +# Disaster Recovery (Geo) promotion runbooks **(PREMIUM ONLY)** + +## Geo planned failover runbook 1 + +| Component | Configuration | +| ----------- | --------------- | +| PostgreSQL | Omnibus-managed | +| Geo site | Single-node | +| Secondaries | One | + +This runbook will guide you through a planned failover of a single-node Geo site +with one secondary. The following general architecture is assumed: + +```mermaid +graph TD + subgraph main[Geo deployment] + subgraph Primary[Primary site] + Node_1[(GitLab node)] + end + subgraph Secondary1[Secondary site] + Node_2[(GitLab node)] + end + end +``` + +This guide will result in the following: + +1. An offline primary. +1. A promoted secondary that is now the new primary. + +What is not covered: + +1. Re-adding the old **primary** as a secondary. +1. Adding a new secondary. + +### Preparation + +NOTE: **Note:** +Before following any of those steps, make sure you have `root` access to the +**secondary** to promote it, since there isn't provided an automated way to +promote a Geo replica and perform a failover. + +On the **secondary** node, navigate to the **Admin Area > Geo** dashboard to +review its status. Replicated objects (shown in green) should be close to 100%, +and there should be no failures (shown in red). If a large proportion of +objects aren't yet replicated (shown in gray), consider giving the node more +time to complete. + + + +If any objects are failing to replicate, this should be investigated before +scheduling the maintenance window. After a planned failover, anything that +failed to replicate will be **lost**. + +You can use the +[Geo status API](../../../api/geo_nodes.md#retrieve-project-sync-or-verification-failures-that-occurred-on-the-current-node) +to review failed objects and the reasons for failure. +A common cause of replication failures is the data being missing on the +**primary** node - you can resolve these failures by restoring the data from backup, +or removing references to the missing data. + +The maintenance window won't end until Geo replication and verification is +completely finished. To keep the window as short as possible, you should +ensure these processes are close to 100% as possible during active use. + +If the **secondary** node is still replicating data from the **primary** node, +follow these steps to avoid unnecessary data loss: + +1. Until a [read-only mode](https://gitlab.com/gitlab-org/gitlab/-/issues/14609) + is implemented, updates must be prevented from happening manually to the + **primary**. Note that your **secondary** node still needs read-only + access to the **primary** node during the maintenance window: + + 1. At the scheduled time, using your cloud provider or your node's firewall, block + all HTTP, HTTPS and SSH traffic to/from the **primary** node, **except** for your IP and + the **secondary** node's IP. + + For instance, you can run the following commands on the **primary** node: + + ```shell + sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 22 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 22 -j ACCEPT + sudo iptables -A INPUT --destination-port 22 -j REJECT + + sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 80 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 80 -j ACCEPT + sudo iptables -A INPUT --tcp-dport 80 -j REJECT + + sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 443 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 443 -j ACCEPT + sudo iptables -A INPUT --tcp-dport 443 -j REJECT + ``` + + From this point, users will be unable to view their data or make changes on the + **primary** node. They will also be unable to log in to the **secondary** node. + However, existing sessions will work for the remainder of the maintenance period, and + public data will be accessible throughout. + + 1. Verify the **primary** node is blocked to HTTP traffic by visiting it in browser via + another IP. The server should refuse connection. + + 1. Verify the **primary** node is blocked to Git over SSH traffic by attempting to pull an + existing Git repository with an SSH remote URL. The server should refuse + connection. + + 1. On the **primary** node, disable non-Geo periodic background jobs by navigating + to **Admin Area > Monitoring > Background Jobs > Cron**, clicking `Disable All`, + and then clicking `Enable` for the `geo_sidekiq_cron_config_worker` cron job. + This job will re-enable several other cron jobs that are essential for planned + failover to complete successfully. + +1. Finish replicating and verifying all data: + + CAUTION: **Caution:** + Not all data is automatically replicated. Read more about + [what is excluded](planned_failover.md#not-all-data-is-automatically-replicated). + + 1. If you are manually replicating any + [data not managed by Geo](../replication/datatypes.md#limitations-on-replicationverification), + trigger the final replication process now. + 1. On the **primary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues** + and wait for all queues except those with `geo` in the name to drop to 0. + These queues contain work that has been submitted by your users; failing over + before it is completed will cause the work to be lost. + 1. On the **primary** node, navigate to **Admin Area > Geo** and wait for the + following conditions to be true of the **secondary** node you are failing over to: + - All replication meters to each 100% replicated, 0% failures. + - All verification meters reach 100% verified, 0% failures. + - Database replication lag is 0ms. + - The Geo log cursor is up to date (0 events behind). + + 1. On the **secondary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues** + and wait for all the `geo` queues to drop to 0 queued and 0 running jobs. + 1. On the **secondary** node, use [these instructions](../../raketasks/check.md) + to verify the integrity of CI artifacts, LFS objects, and uploads in file + storage. + + At this point, your **secondary** node will contain an up-to-date copy of everything the + **primary** node has, meaning nothing will be lost when you fail over. + +1. In this final step, you need to permanently disable the **primary** node. + + CAUTION: **Caution:** + When the **primary** node goes offline, there may be data saved on the **primary** node + that has not been replicated to the **secondary** node. This data should be treated + as lost if you proceed. + + TIP: **Tip:** + If you plan to [update the **primary** domain DNS record](index.md#step-4-optional-updating-the-primary-domain-dns-record), + you may wish to lower the TTL now to speed up propagation. + + When performing a failover, we want to avoid a split-brain situation where + writes can occur in two different GitLab instances. So to prepare for the + failover, you must disable the **primary** node: + + - If you have SSH access to the **primary** node, stop and disable GitLab: + + ```shell + sudo gitlab-ctl stop + ``` + + Prevent GitLab from starting up again if the server unexpectedly reboots: + + ```shell + sudo systemctl disable gitlab-runsvdir + ``` + + NOTE: **Note:** + (**CentOS only**) In CentOS 6 or older, there is no easy way to prevent GitLab from being + started if the machine reboots isn't available (see [Omnibus GitLab issue #3058](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3058)). + It may be safest to uninstall the GitLab package completely with `sudo yum remove gitlab-ee`. + + NOTE: **Note:** + (**Ubuntu 14.04 LTS**) If you are using an older version of Ubuntu + or any other distribution based on the Upstart init system, you can prevent GitLab + from starting if the machine reboots as `root` with + `initctl stop gitlab-runsvvdir && echo 'manual' > /etc/init/gitlab-runsvdir.override && initctl reload-configuration`. + + - If you do not have SSH access to the **primary** node, take the machine offline and + prevent it from rebooting. Since there are many ways you may prefer to accomplish + this, we will avoid a single recommendation. You may need to: + + - Reconfigure the load balancers. + - Change DNS records (for example, point the **primary** DNS record to the **secondary** + node in order to stop usage of the **primary** node). + - Stop the virtual servers. + - Block traffic through a firewall. + - Revoke object storage permissions from the **primary** node. + - Physically disconnect a machine. + +### Promoting the **secondary** node + +Note the following when promoting a secondary: + +- A new **secondary** should not be added at this time. If you want to add a new + **secondary**, do this after you have completed the entire process of promoting + the **secondary** to the **primary**. +- If you encounter an `ActiveRecord::RecordInvalid: Validation failed: Name has already been taken` + error during this process, read + [the troubleshooting advice](../replication/troubleshooting.md#fixing-errors-during-a-failover-or-when-promoting-a-secondary-to-a-primary-node). + +To promote the secondary node: + +1. SSH in to your **secondary** node and login as root: + + ```shell + sudo -i + ``` + +1. Edit `/etc/gitlab/gitlab.rb` to reflect its new status as **primary** by + removing any lines that enabled the `geo_secondary_role`: + + ```ruby + ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. + geo_secondary_role['enable'] = true + + ## In 11.5+ documentation, the role was enabled as follows. Remove this line. + roles ['geo_secondary_role'] + ``` + +1. Run the following command to list out all preflight checks and automatically + check if replication and verification are complete before scheduling a planned + failover to ensure the process will go smoothly: + + ```shell + gitlab-ctl promotion-preflight-checks + ``` + +1. Promote the **secondary**: + + ```shell + gitlab-ctl promote-to-primary-node + ``` + + If you have already run the [preflight checks](planned_failover.md#preflight-checks) + or don't want to run them, you can skip them: + + ```shell + gitlab-ctl promote-to-primary-node --skip-preflight-check + ``` + + You can also promote the secondary node to primary **without any further confirmation**, even when preflight checks fail: + + ```shell + sudo gitlab-ctl promote-to-primary-node --force + ``` + +1. Verify you can connect to the newly promoted **primary** node using the URL used + previously for the **secondary** node. + + If successful, the **secondary** node has now been promoted to the **primary** node. + +### Next steps + +To regain geographic redundancy as quickly as possible, you should +[add a new **secondary** node](../setup/index.md). To +do that, you can re-add the old **primary** as a new secondary and bring it back +online. diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md new file mode 100644 index 00000000000..6fdf213ac78 --- /dev/null +++ b/doc/administration/geo/index.md @@ -0,0 +1,295 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: howto +--- + +# Geo **(PREMIUM ONLY)** + +> - Introduced in GitLab Enterprise Edition 8.9. +> - Using Geo in combination with +> [multi-node architectures](../reference_architectures/index.md) +> is considered **Generally Available** (GA) in +> [GitLab Premium](https://about.gitlab.com/pricing/) 10.4. + +Geo is the solution for widely distributed development teams and for providing a warm-standby as part of a disaster recovery strategy. + +## Overview + +CAUTION: **Caution:** +Geo undergoes significant changes from release to release. Upgrades **are** supported and [documented](#updating-geo), but you should ensure that you're using the right version of the documentation for your installation. + +Fetching large repositories can take a long time for teams located far from a single GitLab instance. + +Geo provides local, read-only instances of your GitLab instances. This can reduce the time it takes +to clone and fetch large repositories, speeding up development. + +For a video introduction to Geo, see [Introduction to GitLab Geo - GitLab Features](https://www.youtube.com/watch?v=-HDLxSjEh6w). + +To make sure you're using the right version of the documentation, navigate to [the source version of this page on GitLab.com](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/administration/geo/index.md) and choose the appropriate release from the **Switch branch/tag** dropdown. For example, [`v11.2.3-ee`](https://gitlab.com/gitlab-org/gitlab/blob/v11.2.3-ee/doc/administration/geo/index.md). + +## Use cases + +Implementing Geo provides the following benefits: + +- Reduce from minutes to seconds the time taken for your distributed developers to clone and fetch large repositories and projects. +- Enable all of your developers to contribute ideas and work in parallel, no matter where they are. +- Balance the read-only load between your **primary** and **secondary** nodes. + +In addition, it: + +- Can be used for cloning and fetching projects, in addition to reading any data available in the GitLab web interface (see [current limitations](#current-limitations)). +- Overcomes slow connections between distant offices, saving time by improving speed for distributed teams. +- Helps reducing the loading time for automated tasks, custom integrations, and internal workflows. +- Can quickly fail over to a **secondary** node in a [disaster recovery](disaster_recovery/index.md) scenario. +- Allows [planned failover](disaster_recovery/planned_failover.md) to a **secondary** node. + +Geo provides: + +- Read-only **secondary** nodes: Maintain one **primary** GitLab node while still enabling read-only **secondary** nodes for each of your distributed teams. +- Authentication system hooks: **Secondary** nodes receives all authentication data (like user accounts and logins) from the **primary** instance. +- An intuitive UI: **Secondary** nodes utilize the same web interface your team has grown accustomed to. In addition, there are visual notifications that block write operations and make it clear that a user is on a **secondary** node. + +## How it works + +Your Geo instance can be used for cloning and fetching projects, in addition to reading any data. This will make working with large repositories over large distances much faster. + + + +When Geo is enabled, the: + +- Original instance is known as the **primary** node. +- Replicated read-only nodes are known as **secondary** nodes. + +Keep in mind that: + +- **Secondary** nodes talk to the **primary** node to: + - Get user data for logins (API). + - Replicate repositories, LFS Objects, and Attachments (HTTPS + JWT). +- Since GitLab Premium 10.0, the **primary** node no longer talks to **secondary** nodes to notify for changes (API). +- Pushing directly to a **secondary** node (for both HTTP and SSH, including Git LFS) was [introduced](https://about.gitlab.com/releases/2018/09/22/gitlab-11-3-released/) in [GitLab Premium](https://about.gitlab.com/pricing/#self-managed) 11.3. +- There are [limitations](#current-limitations) in the current implementation. + +### Architecture + +The following diagram illustrates the underlying architecture of Geo. + + + +In this diagram: + +- There is the **primary** node and the details of one **secondary** node. +- Writes to the database can only be performed on the **primary** node. A **secondary** node receives database + updates via PostgreSQL streaming replication. +- If present, the [LDAP server](#ldap) should be configured to replicate for [Disaster Recovery](disaster_recovery/index.md) scenarios. +- A **secondary** node performs different type of synchronizations against the **primary** node, using a special + authorization protected by JWT: + - Repositories are cloned/updated via Git over HTTPS. + - Attachments, LFS objects, and other files are downloaded via HTTPS using a private API endpoint. + +From the perspective of a user performing Git operations: + +- The **primary** node behaves as a full read-write GitLab instance. +- **Secondary** nodes are read-only but proxy Git push operations to the **primary** node. This makes **secondary** nodes appear to support push operations themselves. + +To simplify the diagram, some necessary components are omitted. Note that: + +- Git over SSH requires [`gitlab-shell`](https://gitlab.com/gitlab-org/gitlab-shell) and OpenSSH. +- Git over HTTPS required [`gitlab-workhorse`](https://gitlab.com/gitlab-org/gitlab-workhorse). + +Note that a **secondary** node needs two different PostgreSQL databases: + +- A read-only database instance that streams data from the main GitLab database. +- [Another database instance](#geo-tracking-database) used internally by the **secondary** node to record what data has been replicated. + +In **secondary** nodes, there is an additional daemon: [Geo Log Cursor](#geo-log-cursor). + +## Requirements for running Geo + +The following are required to run Geo: + +- An operating system that supports OpenSSH 6.9+ (needed for + [fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md)) + The following operating systems are known to ship with a current version of OpenSSH: + - [CentOS](https://www.centos.org) 7.4+ + - [Ubuntu](https://ubuntu.com) 16.04+ +- PostgreSQL 11+ with [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication) +- Git 2.9+ +- All nodes must run the same GitLab version. + +Additionally, check GitLab's [minimum requirements](../../install/requirements.md), +and we recommend you use: + +- At least GitLab Enterprise Edition 10.0 for basic Geo features. +- The latest version for a better experience. + +### Firewall rules + +The following table lists basic ports that must be open between the **primary** and **secondary** nodes for Geo. + +| **Primary** node | **Secondary** node | Protocol | +|:-----------------|:-------------------|:-------------| +| 80 | 80 | HTTP | +| 443 | 443 | TCP or HTTPS | +| 22 | 22 | TCP | +| 5432 | | PostgreSQL | + +See the full list of ports used by GitLab in [Package defaults](https://docs.gitlab.com/omnibus/package-information/defaults.html) + +NOTE: **Note:** +[Web terminal](../../ci/environments/index.md#web-terminals) support requires your load balancer to correctly handle WebSocket connections. +When using HTTP or HTTPS proxying, your load balancer must be configured to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the [web terminal](../integration/terminal.md) integration guide for more details. + +NOTE: **Note:** +When using HTTPS protocol for port 443, you will need to add an SSL certificate to the load balancers. +If you wish to terminate SSL at the GitLab application server instead, use TCP protocol. + +### LDAP + +We recommend that if you use LDAP on your **primary** node, you also set up secondary LDAP servers on each **secondary** node. Otherwise, users will not be able to perform Git operations over HTTP(s) on the **secondary** node using HTTP Basic Authentication. However, Git via SSH and personal access tokens will still work. + +NOTE: **Note:** +It is possible for all **secondary** nodes to share an LDAP server, but additional latency can be an issue. Also, consider what LDAP server will be available in a [disaster recovery](disaster_recovery/index.md) scenario if a **secondary** node is promoted to be a **primary** node. + +Check for instructions on how to set up replication in your LDAP service. Instructions will be different depending on the software or service used. For example, OpenLDAP provides [these instructions](https://www.openldap.org/doc/admin24/replication.html). + +### Geo Tracking Database + +The tracking database instance is used as metadata to control what needs to be updated on the disk of the local instance. For example: + +- Download new assets. +- Fetch new LFS Objects. +- Fetch changes from a repository that has recently been updated. + +Because the replicated database instance is read-only, we need this additional database instance for each **secondary** node. + +### Geo Log Cursor + +This daemon: + +- Reads a log of events replicated by the **primary** node to the **secondary** database instance. +- Updates the Geo Tracking Database instance with changes that need to be executed. + +When something is marked to be updated in the tracking database instance, asynchronous jobs running on the **secondary** node will execute the required operations and update the state. + +This new architecture allows GitLab to be resilient to connectivity issues between the nodes. It doesn't matter how long the **secondary** node is disconnected from the **primary** node as it will be able to replay all the events in the correct order and become synchronized with the **primary** node again. + +## Setup instructions + +For setup instructions, see [Setting up Geo](setup/index.md). + +## Post-installation documentation + +After installing GitLab on the **secondary** nodes and performing the initial configuration, see the following documentation for post-installation information. + +### Configuring Geo + +For information on configuring Geo, see [Geo configuration](replication/configuration.md). + +### Updating Geo + +For information on how to update your Geo nodes to the latest GitLab version, see [Updating the Geo nodes](replication/updating_the_geo_nodes.md). + +### Pausing and resuming replication + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35913) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.2. + +In some circumstances, like during [upgrades](replication/updating_the_geo_nodes.md) or a [planned failover](disaster_recovery/planned_failover.md), it is desirable to pause replication between the primary and secondary. + +Pausing and resuming replication is done via a command line tool from the secondary node. + +**To Pause: (from secondary)** + +```shell +gitlab-ctl geo-replication-pause +``` + +**To Resume: (from secondary)** + +```shell +gitlab-ctl geo-replication-resume +``` + +### Configuring Geo for multiple nodes + +For information on configuring Geo for multiple nodes, see [Geo for multiple servers](replication/multiple_servers.md). + +### Configuring Geo with Object Storage + +For information on configuring Geo with object storage, see [Geo with Object storage](replication/object_storage.md). + +### Disaster Recovery + +For information on using Geo in disaster recovery situations to mitigate data-loss and restore services, see [Disaster Recovery](disaster_recovery/index.md). + +### Replicating the Container Registry + +For more information on how to replicate the Container Registry, see [Docker Registry for a **secondary** node](replication/docker_registry.md). + +### Security Review + +For more information on Geo security, see [Geo security review](replication/security_review.md). + +### Tuning Geo + +For more information on tuning Geo, see [Tuning Geo](replication/tuning.md). + +### Set up a location-aware Git URL + +For an example of how to set up a location-aware Git remote URL with AWS Route53, see [Location-aware Git remote URL with AWS Route53](replication/location_aware_git_url.md). + +## Remove Geo node + +For more information on removing a Geo node, see [Removing **secondary** Geo nodes](replication/remove_geo_node.md). + +## Disable Geo + +To find out how to disable Geo, see [Disabling Geo](replication/disable_geo.md). + +## Current limitations + +CAUTION: **Caution:** +This list of limitations only reflects the latest version of GitLab. If you are using an older version, extra limitations may be in place. + +- Pushing directly to a **secondary** node redirects (for HTTP) or proxies (for SSH) the request to the **primary** node instead of [handling it directly](https://gitlab.com/gitlab-org/gitlab/-/issues/1381), except when using Git over HTTP with credentials embedded within the URI. For example, `https://user:password@secondary.tld`. +- Cloning, pulling, or pushing repositories that exist on the **primary** node but not on the **secondary** nodes where [selective synchronization](replication/configuration.md#selective-synchronization) does not include the project is not supported over SSH [but support is planned](https://gitlab.com/groups/gitlab-org/-/epics/2562). HTTP(S) is supported. +- The **primary** node has to be online for OAuth login to happen. Existing sessions and Git are not affected. Support for the **secondary** node to use an OAuth provider independent from the primary is [being planned](https://gitlab.com/gitlab-org/gitlab/-/issues/208465). +- The installation takes multiple manual steps that together can take about an hour depending on circumstances. We are working on improving this experience. See [Omnibus GitLab issue #2978](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/2978) for details. +- Real-time updates of issues/merge requests (for example, via long polling) doesn't work on the **secondary** node. +- [Selective synchronization](replication/configuration.md#selective-synchronization) applies only to files and repositories. Other datasets are replicated to the **secondary** node in full, making it inappropriate for use as an access control mechanism. +- Object pools for forked project deduplication work only on the **primary** node, and are duplicated on the **secondary** node. +- [External merge request diffs](../merge_request_diffs.md) will not be replicated if they are on-disk, and viewing merge requests will fail. However, external MR diffs in object storage **are** supported. The default configuration (in-database) does work. +- GitLab Runners cannot register with a **secondary** node. Support for this is [planned for the future](https://gitlab.com/gitlab-org/gitlab/-/issues/3294). + +### Limitations on replication/verification + +You can keep track of the progress to implement the missing items in +these epics/issues: + +- [Unreplicated Data Types](https://gitlab.com/groups/gitlab-org/-/epics/893) +- [Verify all replicated data](https://gitlab.com/groups/gitlab-org/-/epics/1430) + +There is a complete list of all GitLab [data types](replication/datatypes.md) and [existing support for replication and verification](replication/datatypes.md#limitations-on-replicationverification). + +## Frequently Asked Questions + +For answers to common questions, see the [Geo FAQ](replication/faq.md). + +## Log files + +Since GitLab 9.5, Geo stores structured log messages in a `geo.log` file. For Omnibus installations, this file is at `/var/log/gitlab/gitlab-rails/geo.log`. + +This file contains information about when Geo attempts to sync repositories and files. Each line in the file contains a separate JSON entry that can be ingested into. For example, Elasticsearch or Splunk. + +For example: + +```json +{"severity":"INFO","time":"2017-08-06T05:40:16.104Z","message":"Repository update","project_id":1,"source":"repository","resync_repository":true,"resync_wiki":true,"class":"Gitlab::Geo::LogCursor::Daemon","cursor_delay_s":0.038} +``` + +This message shows that Geo detected that a repository update was needed for project `1`. + +## Troubleshooting + +For troubleshooting steps, see [Geo Troubleshooting](replication/troubleshooting.md). diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index 74fa8e3b8f2..8c818bcfbe2 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -12,7 +12,7 @@ type: howto NOTE: **Note:** This is the final step in setting up a **secondary** Geo node. Stages of the setup process must be completed in the documented order. -Before attempting the steps in this stage, [complete all prior stages](index.md#using-omnibus-gitlab). +Before attempting the steps in this stage, [complete all prior stages](../setup/index.md#using-omnibus-gitlab). The basic steps of configuring a **secondary** node are to: diff --git a/doc/administration/geo/replication/database.md b/doc/administration/geo/replication/database.md index 0bc37ce6438..da5376190b9 100644 --- a/doc/administration/geo/replication/database.md +++ b/doc/administration/geo/replication/database.md @@ -1,469 +1,5 @@ --- -stage: Enablement -group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers -type: howto +redirect_to: '../setup/database.md' --- -# Geo database replication **(PREMIUM ONLY)** - -NOTE: **Note:** -If your GitLab installation uses external (not managed by Omnibus) PostgreSQL -instances, the Omnibus roles will not be able to perform all necessary -configuration steps. In this case, -[follow the Geo with external PostgreSQL instances document instead](external_database.md). - -NOTE: **Note:** -The stages of the setup process must be completed in the documented order. -Before attempting the steps in this stage, [complete all prior stages](index.md#using-omnibus-gitlab). - -This document describes the minimal steps you have to take in order to -replicate your **primary** GitLab database to a **secondary** node's database. You may -have to change some values according to your database setup, how big it is, etc. - -You are encouraged to first read through all the steps before executing them -in your testing/production environment. - -## PostgreSQL replication - -The GitLab **primary** node where the write operations happen will connect to -the **primary** database server, and **secondary** nodes will -connect to their own database servers (which are also read-only). - -We recommend using [PostgreSQL replication slots](https://medium.com/@tk512/replication-slots-in-postgresql-b4b03d277c75) -to ensure that the **primary** node retains all the data necessary for the **secondary** nodes to -recover. See below for more details. - -The following guide assumes that: - -- You are using Omnibus and therefore you are using PostgreSQL 11 or later - which includes the [`pg_basebackup` tool](https://www.postgresql.org/docs/11/app-pgbasebackup.html). -- You have a **primary** node already set up (the GitLab server you are - replicating from), running Omnibus' PostgreSQL (or equivalent version), and - you have a new **secondary** server set up with the same versions of the OS, - PostgreSQL, and GitLab on all nodes. - -CAUTION: **Warning:** -Geo works with streaming replication. Logical replication is not supported at this time. -There is an [issue where support is being discussed](https://gitlab.com/gitlab-org/gitlab/-/issues/7420). - -### Step 1. Configure the **primary** server - -1. SSH into your GitLab **primary** server and login as root: - - ```shell - sudo -i - ``` - -1. Edit `/etc/gitlab/gitlab.rb` and add a **unique** name for your node: - - ```ruby - # The unique identifier for the Geo node. - gitlab_rails['geo_node_name'] = '<node_name_here>' - ``` - -1. Reconfigure the **primary** node for the change to take effect: - - ```shell - gitlab-ctl reconfigure - ``` - -1. Execute the command below to define the node as **primary** node: - - ```shell - gitlab-ctl set-geo-primary-node - ``` - - This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`. - -1. GitLab 10.4 and up only: Do the following to make sure the `gitlab` database user has a password defined: - - Generate a MD5 hash of the desired password: - - ```shell - gitlab-ctl pg-password-md5 gitlab - # Enter password: <your_password_here> - # Confirm password: <your_password_here> - # fca0b89a972d69f00eb3ec98a5838484 - ``` - - Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` - postgresql['sql_user_password'] = '<md5_hash_of_your_password>' - - # Every node that runs Puma or Sidekiq needs to have the database - # password specified as below. If you have a high-availability setup, this - # must be present in all application nodes. - gitlab_rails['db_password'] = '<your_password_here>' - ``` - -1. Omnibus GitLab already has a [replication user](https://wiki.postgresql.org/wiki/Streaming_Replication) - called `gitlab_replicator`. You must set the password for this user manually. - You will be prompted to enter a password: - - ```shell - gitlab-ctl set-replication-password - ``` - - This command will also read the `postgresql['sql_replication_user']` Omnibus - setting in case you have changed `gitlab_replicator` username to something - else. - - If you are using an external database not managed by Omnibus GitLab, you need - to create the replicator user and define a password to it manually: - - ```sql - --- Create a new user 'replicator' - CREATE USER gitlab_replicator; - - --- Set/change a password and grants replication privilege - ALTER USER gitlab_replicator WITH REPLICATION ENCRYPTED PASSWORD '<replication_password>'; - ``` - -1. Configure PostgreSQL to listen on network interfaces: - - For security reasons, PostgreSQL does not listen on any network interfaces - by default. However, Geo requires the **secondary** node to be able to - connect to the **primary** node's database. For this reason, we need the address of - each node. - - NOTE: **Note:** - For external PostgreSQL instances, see [additional instructions](external_database.md). - - If you are using a cloud provider, you can lookup the addresses for each - Geo node through your cloud provider's management console. - - To lookup the address of a Geo node, SSH in to the Geo node and execute: - - ```shell - ## - ## Private address - ## - ip route get 255.255.255.255 | awk '{print "Private address:", $NF; exit}' - - ## - ## Public address - ## - echo "External address: $(curl --silent ipinfo.io/ip)" - ``` - - In most cases, the following addresses will be used to configure GitLab - Geo: - - | Configuration | Address | - |:----------------------------------------|:------------------------------------------------------| - | `postgresql['listen_address']` | **Primary** node's public or VPC private address. | - | `postgresql['md5_auth_cidr_addresses']` | **Secondary** node's public or VPC private addresses. | - - If you are using Google Cloud Platform, SoftLayer, or any other vendor that - provides a virtual private cloud (VPC) you can use the **primary** and **secondary** nodes - private addresses (corresponds to "internal address" for Google Cloud Platform) for - `postgresql['md5_auth_cidr_addresses']` and `postgresql['listen_address']`. - - The `listen_address` option opens PostgreSQL up to network connections with the interface - corresponding to the given address. See [the PostgreSQL documentation](https://www.postgresql.org/docs/11/runtime-config-connection.html) - for more details. - - NOTE: **Note:** - If you need to use `0.0.0.0` or `*` as the listen_address, you will also need to add - `127.0.0.1/32` to the `postgresql['md5_auth_cidr_addresses']` setting, to allow Rails to connect through - `127.0.0.1`. For more information, see [omnibus-5258](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5258). - - Depending on your network configuration, the suggested addresses may not - be correct. If your **primary** node and **secondary** nodes connect over a local - area network, or a virtual network connecting availability zones like - [Amazon's VPC](https://aws.amazon.com/vpc/) or [Google's VPC](https://cloud.google.com/vpc/) - you should use the **secondary** node's private address for `postgresql['md5_auth_cidr_addresses']`. - - Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP - addresses with addresses appropriate to your network configuration: - - ```ruby - ## - ## Geo Primary role - ## - configure dependent flags automatically to enable Geo - ## - roles ['geo_primary_role'] - - ## - ## Primary address - ## - replace '<primary_node_ip>' with the public or VPC address of your Geo primary node - ## - postgresql['listen_address'] = '<primary_node_ip>' - - ## - # Allow PostgreSQL client authentication from the primary and secondary IPs. These IPs may be - # public or VPC addresses in CIDR format, for example ['198.51.100.1/32', '198.51.100.2/32'] - ## - postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32'] - - ## - ## Replication settings - ## - set this to be the number of Geo secondary nodes you have - ## - postgresql['max_replication_slots'] = 1 - # postgresql['max_wal_senders'] = 10 - # postgresql['wal_keep_segments'] = 10 - - ## - ## Disable automatic database migrations temporarily - ## (until PostgreSQL is restarted and listening on the private address). - ## - gitlab_rails['auto_migrate'] = false - ``` - -1. Optional: If you want to add another **secondary** node, the relevant setting would look like: - - ```ruby - postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32', '<another_secondary_node_ip>/32'] - ``` - - You may also want to edit the `wal_keep_segments` and `max_wal_senders` to match your - database replication requirements. Consult the [PostgreSQL - Replication documentation](https://www.postgresql.org/docs/11/runtime-config-replication.html) - for more information. - -1. Save the file and reconfigure GitLab for the database listen changes and - the replication slot changes to be applied: - - ```shell - gitlab-ctl reconfigure - ``` - - Restart PostgreSQL for its changes to take effect: - - ```shell - gitlab-ctl restart postgresql - ``` - -1. Re-enable migrations now that PostgreSQL is restarted and listening on the - private address. - - Edit `/etc/gitlab/gitlab.rb` and **change** the configuration to `true`: - - ```ruby - gitlab_rails['auto_migrate'] = true - ``` - - Save the file and reconfigure GitLab: - - ```shell - gitlab-ctl reconfigure - ``` - -1. Now that the PostgreSQL server is set up to accept remote connections, run - `netstat -plnt | grep 5432` to make sure that PostgreSQL is listening on port - `5432` to the **primary** server's private address. - -1. A certificate was automatically generated when GitLab was reconfigured. This - will be used automatically to protect your PostgreSQL traffic from - eavesdroppers, but to protect against active ("man-in-the-middle") attackers, - the **secondary** node needs a copy of the certificate. Make a copy of the PostgreSQL - `server.crt` file on the **primary** node by running this command: - - ```shell - cat ~gitlab-psql/data/server.crt - ``` - - Copy the output into a clipboard or into a local file. You - will need it when setting up the **secondary** node! The certificate is not sensitive - data. - -### Step 2. Configure the **secondary** server - -1. SSH into your GitLab **secondary** server and login as root: - - ```shell - sudo -i - ``` - -1. Stop application server and Sidekiq - - ```shell - gitlab-ctl stop puma - gitlab-ctl stop sidekiq - ``` - - NOTE: **Note:** - This step is important so we don't try to execute anything before the node is fully configured. - -1. [Check TCP connectivity](../../raketasks/maintenance.md) to the **primary** node's PostgreSQL server: - - ```shell - gitlab-rake gitlab:tcp_check[<primary_node_ip>,5432] - ``` - - NOTE: **Note:** - If this step fails, you may be using the wrong IP address, or a firewall may - be preventing access to the server. Check the IP address, paying close - attention to the difference between public and private addresses and ensure - that, if a firewall is present, the **secondary** node is permitted to connect to the - **primary** node on port 5432. - -1. Create a file `server.crt` in the **secondary** server, with the content you got on the last step of the **primary** node's setup: - - ```shell - editor server.crt - ``` - -1. Set up PostgreSQL TLS verification on the **secondary** node: - - Install the `server.crt` file: - - ```shell - install \ - -D \ - -o gitlab-psql \ - -g gitlab-psql \ - -m 0400 \ - -T server.crt ~gitlab-psql/.postgresql/root.crt - ``` - - PostgreSQL will now only recognize that exact certificate when verifying TLS - connections. The certificate can only be replicated by someone with access - to the private key, which is **only** present on the **primary** node. - -1. Test that the `gitlab-psql` user can connect to the **primary** node's database - (the default Omnibus database name is `gitlabhq_production`): - - ```shell - sudo \ - -u gitlab-psql /opt/gitlab/embedded/bin/psql \ - --list \ - -U gitlab_replicator \ - -d "dbname=gitlabhq_production sslmode=verify-ca" \ - -W \ - -h <primary_node_ip> - ``` - - When prompted enter the password you set in the first step for the - `gitlab_replicator` user. If all worked correctly, you should see - the list of **primary** node's databases. - - A failure to connect here indicates that the TLS configuration is incorrect. - Ensure that the contents of `~gitlab-psql/data/server.crt` on the **primary** node - match the contents of `~gitlab-psql/.postgresql/root.crt` on the **secondary** node. - -1. Configure PostgreSQL: - - This step is similar to how we configured the **primary** instance. - We need to enable this, even if using a single node. - - Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP - addresses with addresses appropriate to your network configuration: - - ```ruby - ## - ## Geo Secondary role - ## - configure dependent flags automatically to enable Geo - ## - roles ['geo_secondary_role'] - - ## - ## Secondary address - ## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node - ## - postgresql['listen_address'] = '<secondary_node_ip>' - postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32'] - - ## - ## Database credentials password (defined previously in primary node) - ## - replicate same values here as defined in primary node - ## - postgresql['sql_user_password'] = '<md5_hash_of_your_password>' - gitlab_rails['db_password'] = '<your_password_here>' - ``` - - For external PostgreSQL instances, see [additional instructions](external_database.md). - If you bring a former **primary** node back online to serve as a **secondary** node, then you also need to remove `roles ['geo_primary_role']` or `geo_primary_role['enable'] = true`. - -1. Reconfigure GitLab for the changes to take effect: - - ```shell - gitlab-ctl reconfigure - ``` - -1. Restart PostgreSQL for the IP change to take effect: - - ```shell - gitlab-ctl restart postgresql - ``` - -### Step 3. Initiate the replication process - -Below we provide a script that connects the database on the **secondary** node to -the database on the **primary** node, replicates the database, and creates the -needed files for streaming replication. - -The directories used are the defaults that are set up in Omnibus. If you have -changed any defaults, configure it as you see fit replacing the directories and paths. - -CAUTION: **Warning:** -Make sure to run this on the **secondary** server as it removes all PostgreSQL's -data before running `pg_basebackup`. - -1. SSH into your GitLab **secondary** server and login as root: - - ```shell - sudo -i - ``` - -1. Choose a database-friendly name to use for your **secondary** node to - use as the replication slot name. For example, if your domain is - `secondary.geo.example.com`, you may use `secondary_example` as the slot - name as shown in the commands below. - -1. Execute the command below to start a backup/restore and begin the replication - - CAUTION: **Warning:** - Each Geo **secondary** node must have its own unique replication slot name. - Using the same slot name between two secondaries will break PostgreSQL replication. - - ```shell - gitlab-ctl replicate-geo-database \ - --slot-name=<secondary_node_name> \ - --host=<primary_node_ip> - ``` - - NOTE: **Note:** - Replication slot names must only contain lowercase letters, numbers, and the underscore character. - - When prompted, enter the _plaintext_ password you set up for the `gitlab_replicator` - user in the first step. - - This command also takes a number of additional options. You can use `--help` - to list them all, but here are a couple of tips: - - - If PostgreSQL is listening on a non-standard port, add `--port=` as well. - - If your database is too large to be transferred in 30 minutes, you will need - to increase the timeout, e.g., `--backup-timeout=3600` if you expect the - initial replication to take under an hour. - - Pass `--sslmode=disable` to skip PostgreSQL TLS authentication altogether - (e.g., you know the network path is secure, or you are using a site-to-site - VPN). This is **not** safe over the public Internet! - - You can read more details about each `sslmode` in the - [PostgreSQL documentation](https://www.postgresql.org/docs/11/libpq-ssl.html#LIBPQ-SSL-PROTECTION); - the instructions above are carefully written to ensure protection against - both passive eavesdroppers and active "man-in-the-middle" attackers. - - Change the `--slot-name` to the name of the replication slot - to be used on the **primary** database. The script will attempt to create the - replication slot automatically if it does not exist. - - If you're repurposing an old server into a Geo **secondary** node, you'll need to - add `--force` to the command line. - - When not in a production machine you can disable backup step if you - really sure this is what you want by adding `--skip-backup` - -The replication process is now complete. - -## PgBouncer support (optional) - -[PgBouncer](https://www.pgbouncer.org/) may be used with GitLab Geo to pool -PostgreSQL connections. We recommend using PgBouncer if you use GitLab in a -high-availability configuration with a cluster of nodes supporting a Geo -**primary** node and another cluster of nodes supporting a Geo **secondary** node. For more -information, see [High Availability with Omnibus GitLab](../../postgresql/replication_and_failover.md). - -## Troubleshooting - -Read the [troubleshooting document](troubleshooting.md). +This document was moved to [another location](../setup/index.md). diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index b8d01e80371..166a724f9c1 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -45,8 +45,8 @@ verification methods: | Blobs | Archived CI build traces _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | Container registry _(filesystem)_ | Geo with API/Docker API | _Not implemented_ | | Blobs | Container registry _(object storage)_ | Geo with API/Managed/Docker API (*2*) | _Not implemented_ | -| Blobs | Package registry _(filesystem)_ | Geo with API | _Not implemented_ | -| Blobs | Package registry _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | Package registry _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | Package registry _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | - (*1*): Redis replication can be used as part of HA with Redis sentinel. It's not used between Geo nodes. - (*2*): Object storage replication can be performed by Geo or by your object storage provider/appliance @@ -134,7 +134,7 @@ The replication for some data types is behind a corresponding feature flag: > - They're enabled on GitLab.com. > - They can't be enabled or disabled per-project. > - They are recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable them](#enable-or-disable-replication-for-some-data-types-core-only). **(CORE ONLY)** +> - For GitLab self-managed instances, GitLab administrators can opt to [disable them](#enable-or-disable-replication-for-some-data-types). **(CORE ONLY)** #### Enable or disable replication (for some data types) **(CORE ONLY)** @@ -160,34 +160,34 @@ replicating data from those features will cause the data to be **lost**. If you wish to use those features on a **secondary** node, or to execute a failover successfully, you must replicate their data using some other means. -| Feature | Replicated (added in GitLab version) | Verified (added in GitLab version) | Notes | -|:---------------------------------------------------------------------|:---------------------------------------------------------|:--------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------| -| Application data in PostgreSQL | **Yes** (10.2) | **Yes** (10.2) | | -| Project repository | **Yes** (10.2) | **Yes** (10.7) | | -| Project wiki repository | **Yes** (10.2) | **Yes** (10.7) | | -| Project designs repository | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | | -| Uploads | **Yes** (10.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Verified only on transfer, or manually (*1*) | -| LFS objects | **Yes** (10.2) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8922) | Verified only on transfer, or manually (*1*). Unavailable for new LFS objects in 11.11.x and 12.0.x (*2*). | -| CI job artifacts (other than traces) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Verified only manually (*1*) | -| Archived traces | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Verified only on transfer, or manually (*1*) | -| Personal snippets | **Yes** (10.2) | **Yes** (10.2) | | -| [Versioned snippets](../../../user/snippets.md#versioned-snippets) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2810) | | -| Project snippets | **Yes** (10.2) | **Yes** (10.2) | | -| Object pools for forked project deduplication | **Yes** | No | | -| [Server-side Git hooks](../../server_hooks.md) | No | No | | -| [Elasticsearch integration](../../../integration/elasticsearch.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | | -| [GitLab Pages](../../pages/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | | -| [Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | | -| [NPM Registry](../../../user/packages/npm_registry/index.md) | **Yes** (13.2) | No | Behind feature flag `geo_package_file_replication`, enabled by default | | -| [Maven Repository](../../../user/packages/maven_repository/index.md) | **Yes** (13.2) | No | Behind feature flag `geo_package_file_replication`, enabled by default | | -| [Conan Repository](../../../user/packages/conan_repository/index.md) | **Yes** (13.2) | No | Behind feature flag `geo_package_file_replication`, enabled by default | | -| [NuGet Repository](../../../user/packages/nuget_repository/index.md) | **Yes** (13.2) | No | Behind feature flag `geo_package_file_replication`, enabled by default | | -| [PyPi Repository](../../../user/packages/pypi_repository/index.md) | **Yes** (13.2) | No | Behind feature flag `geo_package_file_replication`, enabled by default | | -| [Composer Repository](../../../user/packages/composer_repository/index.md) | **Yes** (13.2) | No | Behind feature flag `geo_package_file_replication`, enabled by default | | -| [External merge request diffs](../../merge_request_diffs.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/33817) | No | | -| [Terraform State](../../terraform_state.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/3112)(*3*) | No | | -| [Vulnerability Export](../../../user/application_security/security_dashboard/#export-vulnerabilities) | [No](https://gitlab.com/groups/gitlab-org/-/epics/3111)(*3*) | No | | | -| Content in object storage | **Yes** (12.4) | No | | +| Feature | Replicated (added in GitLab version) | Verified (added in GitLab version) | Notes | +|:------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------|:----------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------| +| Application data in PostgreSQL | **Yes** (10.2) | **Yes** (10.2) | | +| Project repository | **Yes** (10.2) | **Yes** (10.7) | | +| Project wiki repository | **Yes** (10.2) | **Yes** (10.7) | | +| Project designs repository | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | | +| Uploads | **Yes** (10.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Verified only on transfer, or manually (*1*) | +| LFS objects | **Yes** (10.2) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8922) | Verified only on transfer, or manually (*1*). Unavailable for new LFS objects in 11.11.x and 12.0.x (*2*). | +| CI job artifacts (other than traces) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Verified only manually (*1*) | +| Archived traces | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Verified only on transfer, or manually (*1*) | +| Personal snippets | **Yes** (10.2) | **Yes** (10.2) | | +| [Versioned snippets](../../../user/snippets.md#versioned-snippets) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2810) | | +| Project snippets | **Yes** (10.2) | **Yes** (10.2) | | +| Object pools for forked project deduplication | **Yes** | No | | +| [Server-side Git hooks](../../server_hooks.md) | No | No | | +| [Elasticsearch integration](../../../integration/elasticsearch.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | | +| [GitLab Pages](../../pages/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | | +| [Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | | +| [NPM Registry](../../../user/packages/npm_registry/index.md) | **Yes** (13.2) | No | Behind feature flag `geo_package_file_replication`, enabled by default | +| [Maven Repository](../../../user/packages/maven_repository/index.md) | **Yes** (13.2) | No | Behind feature flag `geo_package_file_replication`, enabled by default | +| [Conan Repository](../../../user/packages/conan_repository/index.md) | **Yes** (13.2) | No | Behind feature flag `geo_package_file_replication`, enabled by default | +| [NuGet Repository](../../../user/packages/nuget_repository/index.md) | **Yes** (13.2) | No | Behind feature flag `geo_package_file_replication`, enabled by default | +| [PyPi Repository](../../../user/packages/pypi_repository/index.md) | **Yes** (13.2) | No | Behind feature flag `geo_package_file_replication`, enabled by default | +| [Composer Repository](../../../user/packages/composer_repository/index.md) | **Yes** (13.2) | No | Behind feature flag `geo_package_file_replication`, enabled by default | +| [External merge request diffs](../../merge_request_diffs.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/33817) | No | | +| [Terraform State](../../terraform_state.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/3112)(*3*) | No | | +| [Vulnerability Export](../../../user/application_security/security_dashboard/#export-vulnerabilities) | [No](https://gitlab.com/groups/gitlab-org/-/epics/3111)(*3*) | No | | +| Content in object storage | **Yes** (12.4) | No | | - (*1*): The integrity can be verified manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing diff --git a/doc/administration/geo/replication/external_database.md b/doc/administration/geo/replication/external_database.md index d860a3dd368..0d0cd8a37d5 100644 --- a/doc/administration/geo/replication/external_database.md +++ b/doc/administration/geo/replication/external_database.md @@ -1,234 +1,5 @@ --- -stage: Enablement -group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers -type: howto +redirect_to: '../setup/external_database.md' --- -# Geo with external PostgreSQL instances **(PREMIUM ONLY)** - -This document is relevant if you are using a PostgreSQL instance that is *not -managed by Omnibus*. This includes cloud-managed instances like AWS RDS, or -manually installed and configured PostgreSQL instances. - -NOTE: **Note:** -We strongly recommend running Omnibus-managed instances as they are actively -developed and tested. We aim to be compatible with most external -(not managed by Omnibus) databases but we do not guarantee compatibility. - -## **Primary** node - -1. SSH into a GitLab **primary** application server and login as root: - - ```shell - sudo -i - ``` - -1. Edit `/etc/gitlab/gitlab.rb` and add a **unique** ID for your node (arbitrary value): - - ```ruby - # The unique identifier for the Geo node. - gitlab_rails['geo_node_name'] = '<node_name_here>' - ``` - -1. Reconfigure the **primary** node for the change to take effect: - - ```shell - gitlab-ctl reconfigure - ``` - -1. Execute the command below to define the node as **primary** node: - - ```shell - gitlab-ctl set-geo-primary-node - ``` - - This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`. - -### Configure the external database to be replicated - -To set up an external database, you can either: - -- Set up streaming replication yourself (for example, in AWS RDS). -- Perform the Omnibus configuration manually as follows. - -#### Leverage your cloud provider's tools to replicate the primary database - -Given you have a primary node set up on AWS EC2 that uses RDS. -You can now just create a read-only replica in a different region and the -replication process will be managed by AWS. Make sure you've set Network ACL, Subnet, and -Security Group according to your needs, so the secondary application node can access the database. - -The following instructions detail how to create a read-only replica for common -cloud providers: - -- Amazon RDS - [Creating a Read Replica](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.Create) -- Azure Database for PostgreSQL - [Create and manage read replicas in Azure Database for PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/howto-read-replicas-portal) - -Once your read-only replica is set up, you can skip to [configure you secondary application node](#configure-secondary-application-nodes-to-use-the-external-read-replica). - -#### Manually configure the primary database for replication - -The [`geo_primary_role`](https://docs.gitlab.com/omnibus/roles/#gitlab-geo-roles) -configures the **primary** node's database to be replicated by making changes to -`pg_hba.conf` and `postgresql.conf`. Make the following configuration changes -manually to your external database configuration and ensure that you restart PostgreSQL -afterwards for the changes to take effect: - -```plaintext -## -## Geo Primary Role -## - pg_hba.conf -## -host all all <trusted primary IP>/32 md5 -host replication gitlab_replicator <trusted primary IP>/32 md5 -host all all <trusted secondary IP>/32 md5 -host replication gitlab_replicator <trusted secondary IP>/32 md5 -``` - -```plaintext -## -## Geo Primary Role -## - postgresql.conf -## -wal_level = hot_standby -max_wal_senders = 10 -wal_keep_segments = 50 -max_replication_slots = 1 # number of secondary instances -hot_standby = on -``` - -## **Secondary** nodes - -### Manually configure the replica database - -Make the following configuration changes manually to your `pg_hba.conf` and `postgresql.conf` -of your external replica database and ensure that you restart PostgreSQL afterwards -for the changes to take effect: - -```plaintext -## -## Geo Secondary Role -## - pg_hba.conf -## -host all all <trusted secondary IP>/32 md5 -host replication gitlab_replicator <trusted secondary IP>/32 md5 -host all all <trusted primary IP>/24 md5 -``` - -```plaintext -## -## Geo Secondary Role -## - postgresql.conf -## -wal_level = hot_standby -max_wal_senders = 10 -wal_keep_segments = 10 -hot_standby = on -``` - -### Configure **secondary** application nodes to use the external read-replica - -With Omnibus, the -[`geo_secondary_role`](https://docs.gitlab.com/omnibus/roles/#gitlab-geo-roles) -has three main functions: - -1. Configure the replica database. -1. Configure the tracking database. -1. Enable the [Geo Log Cursor](index.md#geo-log-cursor) (not covered in this section). - -To configure the connection to the external read-replica database and enable Log Cursor: - -1. SSH into a GitLab **secondary** application server and login as root: - - ```shell - sudo -i - ``` - -1. Edit `/etc/gitlab/gitlab.rb` and add the following - - ```ruby - ## - ## Geo Secondary role - ## - configure dependent flags automatically to enable Geo - ## - roles ['geo_secondary_role'] - - # note this is shared between both databases, - # make sure you define the same password in both - gitlab_rails['db_password'] = '<your_password_here>' - - gitlab_rails['db_username'] = 'gitlab' - gitlab_rails['db_host'] = '<database_read_replica_host>' - - # Disable the bundled Omnibus PostgreSQL, since we are - # using an external PostgreSQL - postgresql['enable'] = false - ``` - -1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) - -### Configure the tracking database - -**Secondary** nodes use a separate PostgreSQL installation as a tracking -database to keep track of replication status and automatically recover from -potential replication issues. Omnibus automatically configures a tracking database -when `roles ['geo_secondary_role']` is set. -If you want to run this database external to Omnibus, please follow the instructions below. - -If you are using a cloud-managed service for the tracking database, you may need -to grant additional roles to your tracking database user (by default, this is -`gitlab_geo`): - -- Amazon RDS requires the [`rds_superuser`](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.html#Appendix.PostgreSQL.CommonDBATasks.Roles) role. -- Azure Database for PostgreSQL requires the [`azure_pg_admin`](https://docs.microsoft.com/en-us/azure/postgresql/howto-create-users#how-to-create-additional-admin-users-in-azure-database-for-postgresql) role. - -If you have an external database ready to be used as the tracking database, -follow the instructions below to use it: - -NOTE: **Note:** -If you want to use AWS RDS as a tracking database, make sure it has access to -the secondary database. Unfortunately, just assigning the same security group is not enough as -outbound rules do not apply to RDS PostgreSQL databases. Therefore, you need to explicitly add an inbound -rule to the read-replica's security group allowing any TCP traffic from -the tracking database on port 5432. - -1. Ensure that your secondary node can communicate with your tracking database by - manually changing the `pg_hba.conf` that is associated with your tracking database. - Remember to restart PostgreSQL afterwards for the changes to take effect: - - ```plaintext - ## - ## Geo Tracking Database Role - ## - pg_hba.conf - ## - host all all <trusted tracking IP>/32 md5 - host all all <trusted secondary IP>/32 md5 - ``` - -1. SSH into a GitLab **secondary** server and login as root: - - ```shell - sudo -i - ``` - -1. Edit `/etc/gitlab/gitlab.rb` with the connection parameters and credentials for - the machine with the PostgreSQL instance: - - ```ruby - geo_secondary['db_username'] = 'gitlab_geo' - geo_secondary['db_password'] = '<your_password_here>' - - geo_secondary['db_host'] = '<tracking_database_host>' - geo_secondary['db_port'] = <tracking_database_port> # change to the correct port - geo_postgresql['enable'] = false # don't use internal managed instance - ``` - -1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) - -1. Run the tracking database migrations: - - ```shell - gitlab-rake geo:db:create - gitlab-rake geo:db:migrate - ``` +This document was moved to [another location](../setup/external_database.md). diff --git a/doc/administration/geo/replication/faq.md b/doc/administration/geo/replication/faq.md index 522ad32c352..3892d73b465 100644 --- a/doc/administration/geo/replication/faq.md +++ b/doc/administration/geo/replication/faq.md @@ -9,7 +9,7 @@ type: howto ## What are the minimum requirements to run Geo? -The requirements are listed [on the index page](index.md#requirements-for-running-geo) +The requirements are listed [on the index page](../index.md#requirements-for-running-geo) ## How does Geo know which projects to sync? diff --git a/doc/administration/geo/replication/geo_validation_tests.md b/doc/administration/geo/replication/geo_validation_tests.md index 323f6f367b1..8247b8c6336 100644 --- a/doc/administration/geo/replication/geo_validation_tests.md +++ b/doc/administration/geo/replication/geo_validation_tests.md @@ -158,3 +158,15 @@ The following are PostgreSQL upgrade validation tests we performed. - [`gitlab-ctl` reconfigure fails on Redis node in multi-node Geo setup](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4706). - [Geo multi-node upgrade from 12.0.9 to 12.1.9 does not upgrade PostgreSQL](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4705). - [Refresh foreign tables fails on app server in multi-node setup after upgrade to 12.1.9](https://gitlab.com/gitlab-org/gitlab/-/issues/32119). + +## Other tests + +The following are additional validation tests we performed. + +### August 2020 + +[Test Gitaly Cluster on a Geo Deployment](https://gitlab.com/gitlab-org/gitlab/-/issues/223210): + +- Description: Tested a Geo deployment with Gitaly clusters configured on both the primary and secondary Geo sites. Triggered automatic Gitaly cluster failover on the primary Geo site, and ran end-to-end Geo tests. Then triggered Gitaly cluster failover on the secondary Geo site, and re-ran the end-to-end Geo tests. + +- Outcome: Successful end-to-end tests before and after Gitaly cluster failover on the primary site, and before and after Gitaly cluster failover on the secondary site. diff --git a/doc/administration/geo/replication/index.md b/doc/administration/geo/replication/index.md index f1cc9f0df8b..5e9dd73ecc5 100644 --- a/doc/administration/geo/replication/index.md +++ b/doc/administration/geo/replication/index.md @@ -1,316 +1,5 @@ --- -stage: Enablement -group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers -type: howto +redirect_to: '../index.md' --- -# Replication (Geo) **(PREMIUM ONLY)** - -> - Introduced in GitLab Enterprise Edition 8.9. -> - Using Geo in combination with -> [multi-node architectures](../../reference_architectures/index.md) -> is considered **Generally Available** (GA) in -> [GitLab Premium](https://about.gitlab.com/pricing/) 10.4. - -Replication with Geo is the solution for widely distributed development teams. - -## Overview - -Fetching large repositories can take a long time for teams located far from a single GitLab instance. - -Geo provides local, read-only instances of your GitLab instances. This can reduce the time it takes -to clone and fetch large repositories, speeding up development. - -NOTE: **Note:** -Check the [requirements](#requirements-for-running-geo) carefully before setting up Geo. - -For a video introduction to Geo, see [Introduction to GitLab Geo - GitLab Features](https://www.youtube.com/watch?v=-HDLxSjEh6w). - -CAUTION: **Caution:** -Geo undergoes significant changes from release to release. Upgrades **are** supported and [documented](#updating-geo), but you should ensure that you're using the right version of the documentation for your installation. - -To make sure you're using the right version of the documentation, navigate to [the source version of this page on GitLab.com](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/administration/geo/replication/index.md) and choose the appropriate release from the **Switch branch/tag** dropdown. For example, [`v11.2.3-ee`](https://gitlab.com/gitlab-org/gitlab/blob/v11.2.3-ee/doc/administration/geo/replication/index.md). - -## Use cases - -Implementing Geo provides the following benefits: - -- Reduce from minutes to seconds the time taken for your distributed developers to clone and fetch large repositories and projects. -- Enable all of your developers to contribute ideas and work in parallel, no matter where they are. -- Balance the read-only load between your **primary** and **secondary** nodes. - -In addition, it: - -- Can be used for cloning and fetching projects, in addition to reading any data available in the GitLab web interface (see [current limitations](#current-limitations)). -- Overcomes slow connections between distant offices, saving time by improving speed for distributed teams. -- Helps reducing the loading time for automated tasks, custom integrations, and internal workflows. -- Can quickly fail over to a **secondary** node in a [disaster recovery](../disaster_recovery/index.md) scenario. -- Allows [planned failover](../disaster_recovery/planned_failover.md) to a **secondary** node. - -Geo provides: - -- Read-only **secondary** nodes: Maintain one **primary** GitLab node while still enabling read-only **secondary** nodes for each of your distributed teams. -- Authentication system hooks: **Secondary** nodes receives all authentication data (like user accounts and logins) from the **primary** instance. -- An intuitive UI: **Secondary** nodes utilize the same web interface your team has grown accustomed to. In addition, there are visual notifications that block write operations and make it clear that a user is on a **secondary** node. - -## How it works - -Your Geo instance can be used for cloning and fetching projects, in addition to reading any data. This will make working with large repositories over large distances much faster. - - - -When Geo is enabled, the: - -- Original instance is known as the **primary** node. -- Replicated read-only nodes are known as **secondary** nodes. - -Keep in mind that: - -- **Secondary** nodes talk to the **primary** node to: - - Get user data for logins (API). - - Replicate repositories, LFS Objects, and Attachments (HTTPS + JWT). -- Since GitLab Premium 10.0, the **primary** node no longer talks to **secondary** nodes to notify for changes (API). -- Pushing directly to a **secondary** node (for both HTTP and SSH, including Git LFS) was [introduced](https://about.gitlab.com/releases/2018/09/22/gitlab-11-3-released/) in [GitLab Premium](https://about.gitlab.com/pricing/#self-managed) 11.3. -- There are [limitations](#current-limitations) in the current implementation. - -### Architecture - -The following diagram illustrates the underlying architecture of Geo. - - - -In this diagram: - -- There is the **primary** node and the details of one **secondary** node. -- Writes to the database can only be performed on the **primary** node. A **secondary** node receives database - updates via PostgreSQL streaming replication. -- If present, the [LDAP server](#ldap) should be configured to replicate for [Disaster Recovery](../disaster_recovery/index.md) scenarios. -- A **secondary** node performs different type of synchronizations against the **primary** node, using a special - authorization protected by JWT: - - Repositories are cloned/updated via Git over HTTPS. - - Attachments, LFS objects, and other files are downloaded via HTTPS using a private API endpoint. - -From the perspective of a user performing Git operations: - -- The **primary** node behaves as a full read-write GitLab instance. -- **Secondary** nodes are read-only but proxy Git push operations to the **primary** node. This makes **secondary** nodes appear to support push operations themselves. - -To simplify the diagram, some necessary components are omitted. Note that: - -- Git over SSH requires [`gitlab-shell`](https://gitlab.com/gitlab-org/gitlab-shell) and OpenSSH. -- Git over HTTPS required [`gitlab-workhorse`](https://gitlab.com/gitlab-org/gitlab-workhorse). - -Note that a **secondary** node needs two different PostgreSQL databases: - -- A read-only database instance that streams data from the main GitLab database. -- [Another database instance](#geo-tracking-database) used internally by the **secondary** node to record what data has been replicated. - -In **secondary** nodes, there is an additional daemon: [Geo Log Cursor](#geo-log-cursor). - -## Requirements for running Geo - -The following are required to run Geo: - -- An operating system that supports OpenSSH 6.9+ (needed for - [fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md)) - The following operating systems are known to ship with a current version of OpenSSH: - - [CentOS](https://www.centos.org) 7.4+ - - [Ubuntu](https://ubuntu.com) 16.04+ -- PostgreSQL 11+ with [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication) -- Git 2.9+ -- All nodes must run the same GitLab version. - -Additionally, check GitLab's [minimum requirements](../../../install/requirements.md), -and we recommend you use: - -- At least GitLab Enterprise Edition 10.0 for basic Geo features. -- The latest version for a better experience. - -### Firewall rules - -The following table lists basic ports that must be open between the **primary** and **secondary** nodes for Geo. - -| **Primary** node | **Secondary** node | Protocol | -|:-----------------|:-------------------|:-------------| -| 80 | 80 | HTTP | -| 443 | 443 | TCP or HTTPS | -| 22 | 22 | TCP | -| 5432 | | PostgreSQL | - -See the full list of ports used by GitLab in [Package defaults](https://docs.gitlab.com/omnibus/package-information/defaults.html) - -NOTE: **Note:** -[Web terminal](../../../ci/environments/index.md#web-terminals) support requires your load balancer to correctly handle WebSocket connections. -When using HTTP or HTTPS proxying, your load balancer must be configured to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the [web terminal](../../integration/terminal.md) integration guide for more details. - -NOTE: **Note:** -When using HTTPS protocol for port 443, you will need to add an SSL certificate to the load balancers. -If you wish to terminate SSL at the GitLab application server instead, use TCP protocol. - -### LDAP - -We recommend that if you use LDAP on your **primary** node, you also set up secondary LDAP servers on each **secondary** node. Otherwise, users will not be able to perform Git operations over HTTP(s) on the **secondary** node using HTTP Basic Authentication. However, Git via SSH and personal access tokens will still work. - -NOTE: **Note:** -It is possible for all **secondary** nodes to share an LDAP server, but additional latency can be an issue. Also, consider what LDAP server will be available in a [disaster recovery](../disaster_recovery/index.md) scenario if a **secondary** node is promoted to be a **primary** node. - -Check for instructions on how to set up replication in your LDAP service. Instructions will be different depending on the software or service used. For example, OpenLDAP provides [these instructions](https://www.openldap.org/doc/admin24/replication.html). - -### Geo Tracking Database - -The tracking database instance is used as metadata to control what needs to be updated on the disk of the local instance. For example: - -- Download new assets. -- Fetch new LFS Objects. -- Fetch changes from a repository that has recently been updated. - -Because the replicated database instance is read-only, we need this additional database instance for each **secondary** node. - -### Geo Log Cursor - -This daemon: - -- Reads a log of events replicated by the **primary** node to the **secondary** database instance. -- Updates the Geo Tracking Database instance with changes that need to be executed. - -When something is marked to be updated in the tracking database instance, asynchronous jobs running on the **secondary** node will execute the required operations and update the state. - -This new architecture allows GitLab to be resilient to connectivity issues between the nodes. It doesn't matter how long the **secondary** node is disconnected from the **primary** node as it will be able to replay all the events in the correct order and become synchronized with the **primary** node again. - -## Setup instructions - -These instructions assume you have a working instance of GitLab. They guide you through: - -1. Making your existing instance the **primary** node. -1. Adding **secondary** nodes. - -CAUTION: **Caution:** -The steps below should be followed in the order they appear. **Make sure the GitLab version is the same on all nodes.** - -### Using Omnibus GitLab - -If you installed GitLab using the Omnibus packages (highly recommended): - -1. [Install GitLab Enterprise Edition](https://about.gitlab.com/install/) on the server that will serve as the **secondary** node. Do not create an account or log in to the new **secondary** node. -1. [Upload the GitLab License](../../../user/admin_area/license.md) on the **primary** node to unlock Geo. The license must be for [GitLab Premium](https://about.gitlab.com/pricing/) or higher. -1. [Set up the database replication](database.md) (`primary (read-write) <-> secondary (read-only)` topology). -1. [Configure fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md). This step is required and needs to be done on **both** the **primary** and **secondary** nodes. -1. [Configure GitLab](configuration.md) to set the **primary** and **secondary** nodes. -1. Optional: [Configure a secondary LDAP server](../../auth/ldap/index.md) for the **secondary** node. See [notes on LDAP](#ldap). -1. [Follow the "Using a Geo Server" guide](using_a_geo_server.md). - -## Post-installation documentation - -After installing GitLab on the **secondary** nodes and performing the initial configuration, see the following documentation for post-installation information. - -### Configuring Geo - -For information on configuring Geo, see [Geo configuration](configuration.md). - -### Updating Geo - -For information on how to update your Geo nodes to the latest GitLab version, see [Updating the Geo nodes](updating_the_geo_nodes.md). - -### Pausing and resuming replication - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35913) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.2. - -In some circumstances, like during [upgrades](updating_the_geo_nodes.md) or a [planned failover](../disaster_recovery/planned_failover.md), it is desirable to pause replication between the primary and secondary. - -Pausing and resuming replication is done via a command line tool from the secondary node. - -**To Pause: (from secondary)** - -```shell -gitlab-ctl geo-replication-pause -``` - -**To Resume: (from secondary)** - -```shell -gitlab-ctl geo-replication-resume -``` - -### Configuring Geo for multiple nodes - -For information on configuring Geo for multiple nodes, see [Geo for multiple servers](multiple_servers.md). - -### Configuring Geo with Object Storage - -For information on configuring Geo with object storage, see [Geo with Object storage](object_storage.md). - -### Disaster Recovery - -For information on using Geo in disaster recovery situations to mitigate data-loss and restore services, see [Disaster Recovery](../disaster_recovery/index.md). - -### Replicating the Container Registry - -For more information on how to replicate the Container Registry, see [Docker Registry for a **secondary** node](docker_registry.md). - -### Security Review - -For more information on Geo security, see [Geo security review](security_review.md). - -### Tuning Geo - -For more information on tuning Geo, see [Tuning Geo](tuning.md). - -### Set up a location-aware Git URL - -For an example of how to set up a location-aware Git remote URL with AWS Route53, see [Location-aware Git remote URL with AWS Route53](location_aware_git_url.md). - -## Remove Geo node - -For more information on removing a Geo node, see [Removing **secondary** Geo nodes](remove_geo_node.md). - -## Disable Geo - -To find out how to disable Geo, see [Disabling Geo](disable_geo.md). - -## Current limitations - -CAUTION: **Caution:** -This list of limitations only reflects the latest version of GitLab. If you are using an older version, extra limitations may be in place. - -- Pushing directly to a **secondary** node redirects (for HTTP) or proxies (for SSH) the request to the **primary** node instead of [handling it directly](https://gitlab.com/gitlab-org/gitlab/-/issues/1381), except when using Git over HTTP with credentials embedded within the URI. For example, `https://user:password@secondary.tld`. -- Cloning, pulling, or pushing repositories that exist on the **primary** node but not on the **secondary** nodes where [selective synchronization](configuration.md#selective-synchronization) does not include the project is not supported over SSH [but support is planned](https://gitlab.com/groups/gitlab-org/-/epics/2562). HTTP(S) is supported. -- The **primary** node has to be online for OAuth login to happen. Existing sessions and Git are not affected. Support for the **secondary** node to use an OAuth provider independent from the primary is [being planned](https://gitlab.com/gitlab-org/gitlab/-/issues/208465). -- The installation takes multiple manual steps that together can take about an hour depending on circumstances. We are working on improving this experience. See [Omnibus GitLab issue #2978](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/2978) for details. -- Real-time updates of issues/merge requests (for example, via long polling) doesn't work on the **secondary** node. -- [Selective synchronization](configuration.md#selective-synchronization) applies only to files and repositories. Other datasets are replicated to the **secondary** node in full, making it inappropriate for use as an access control mechanism. -- Object pools for forked project deduplication work only on the **primary** node, and are duplicated on the **secondary** node. -- [External merge request diffs](../../merge_request_diffs.md) will not be replicated if they are on-disk, and viewing merge requests will fail. However, external MR diffs in object storage **are** supported. The default configuration (in-database) does work. -- GitLab Runners cannot register with a **secondary** node. Support for this is [planned for the future](https://gitlab.com/gitlab-org/gitlab/-/issues/3294). - -### Limitations on replication/verification - -You can keep track of the progress to implement the missing items in -these epics/issues: - -- [Unreplicated Data Types](https://gitlab.com/groups/gitlab-org/-/epics/893) -- [Verify all replicated data](https://gitlab.com/groups/gitlab-org/-/epics/1430) - -There is a complete list of all GitLab [data types](datatypes.md) and [existing support for replication and verification](datatypes.md#limitations-on-replicationverification). - -## Frequently Asked Questions - -For answers to common questions, see the [Geo FAQ](faq.md). - -## Log files - -Since GitLab 9.5, Geo stores structured log messages in a `geo.log` file. For Omnibus installations, this file is at `/var/log/gitlab/gitlab-rails/geo.log`. - -This file contains information about when Geo attempts to sync repositories and files. Each line in the file contains a separate JSON entry that can be ingested into. For example, Elasticsearch or Splunk. - -For example: - -```json -{"severity":"INFO","time":"2017-08-06T05:40:16.104Z","message":"Repository update","project_id":1,"source":"repository","resync_repository":true,"resync_wiki":true,"class":"Gitlab::Geo::LogCursor::Daemon","cursor_delay_s":0.038} -``` - -This message shows that Geo detected that a repository update was needed for project `1`. - -## Troubleshooting - -For troubleshooting steps, see [Geo Troubleshooting](troubleshooting.md). +This document was moved to [another location](../index.md). diff --git a/doc/administration/geo/replication/location_aware_git_url.md b/doc/administration/geo/replication/location_aware_git_url.md index 8b086e3ff5f..ddcdea736e7 100644 --- a/doc/administration/geo/replication/location_aware_git_url.md +++ b/doc/administration/geo/replication/location_aware_git_url.md @@ -44,7 +44,7 @@ In any case, you require: - A Route53 Hosted Zone managing your domain. If you have not yet setup a Geo **primary** node and **secondary** node, please consult -[the Geo setup instructions](index.md#setup-instructions). +[the Geo setup instructions](../index.md#setup-instructions). ## Create a traffic policy diff --git a/doc/administration/geo/replication/multiple_servers.md b/doc/administration/geo/replication/multiple_servers.md index 31d1ea2cd2b..cba41c375a3 100644 --- a/doc/administration/geo/replication/multiple_servers.md +++ b/doc/administration/geo/replication/multiple_servers.md @@ -147,7 +147,7 @@ The following documentation assumes the database will be run on a single node only. Multi-node PostgreSQL on **secondary** nodes is [not currently supported](https://gitlab.com/groups/gitlab-org/-/epics/2536). -Configure the [**secondary** database](database.md) as a read-only replica of +Configure the [**secondary** database](../setup/database.md) as a read-only replica of the **primary** database. Use the following as a guide. 1. Generate an MD5 hash of the desired password for the database user that the @@ -222,7 +222,7 @@ the **primary** database. Use the following as a guide. After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. If using an external PostgreSQL instance, refer also to -[Geo with external PostgreSQL instances](external_database.md). +[Geo with external PostgreSQL instances](../setup/external_database.md). ### Step 3: Configure the tracking database on the **secondary** node @@ -294,7 +294,7 @@ Configure the tracking database. After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. If using an external PostgreSQL instance, refer also to -[Geo with external PostgreSQL instances](external_database.md). +[Geo with external PostgreSQL instances](../setup/external_database.md). ### Step 4: Configure the frontend application servers on the **secondary** node diff --git a/doc/administration/geo/replication/object_storage.md b/doc/administration/geo/replication/object_storage.md index 159e2524198..642d31f6298 100644 --- a/doc/administration/geo/replication/object_storage.md +++ b/doc/administration/geo/replication/object_storage.md @@ -26,7 +26,7 @@ To have: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10586) in GitLab 12.4. CAUTION: **Caution:** -This is a [**beta** feature](https://about.gitlab.com/handbook/product/#beta) and is not ready yet for production use at any scale. +This is a [**beta** feature](https://about.gitlab.com/handbook/product/#beta) and is not ready yet for production use at any scale. The main limitations are a lack of testing at scale and no verification of any replicated data. **Secondary** nodes can replicate files stored on the **primary** node regardless of whether they are stored on the local filesystem or in object storage. @@ -44,7 +44,7 @@ For LFS, follow the documentation to For CI job artifacts, there is similar documentation to configure [jobs artifact object storage](../../job_artifacts.md#using-object-storage) -For user uploads, there is similar documentation to configure [upload object storage](../../uploads.md#using-object-storage-core-only) +For user uploads, there is similar documentation to configure [upload object storage](../../uploads.md#using-object-storage) If you want to migrate the **primary** node's files to object storage, you can configure the **secondary** in a few ways: diff --git a/doc/administration/geo/replication/security_review.md b/doc/administration/geo/replication/security_review.md index f5edf79c6e4..28b8c8a9d51 100644 --- a/doc/administration/geo/replication/security_review.md +++ b/doc/administration/geo/replication/security_review.md @@ -37,7 +37,7 @@ from [owasp.org](https://owasp.org/). private projects. Geo replicates them all indiscriminately. “Selective sync” exists for files and repositories (but not database content), which would permit only less-sensitive projects to be replicated to a **secondary** node if desired. -- See also: [GitLab data classification policy](https://about.gitlab.com/handbook/engineering/security/data-classification-policy.html). +- See also: [GitLab data classification policy](https://about.gitlab.com/handbook/engineering/security/data-classification-standard.html). ### What data backup and retention requirements have been defined for the application? @@ -123,7 +123,7 @@ from [owasp.org](https://owasp.org/). - Geo imposes no additional restrictions on operating system (see the [GitLab installation](https://about.gitlab.com/install/) page for more - details), however we recommend using the operating systems listed in the [Geo documentation](index.md#requirements-for-running-geo). + details), however we recommend using the operating systems listed in the [Geo documentation](../index.md#requirements-for-running-geo). ### What details regarding required OS components and lock‐down needs have been defined? diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index b8172322c10..f6d6f39fb19 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -251,7 +251,7 @@ sudo gitlab-rake gitlab:geo:check When performing a PostgreSQL major version (9 > 10) update this is expected. Follow: - - [initiate-the-replication-process](database.md#step-3-initiate-the-replication-process) + - [initiate-the-replication-process](../setup/database.md#step-3-initiate-the-replication-process) ## Fixing replication errors @@ -268,7 +268,7 @@ default to 1. You may need to increase this value if you have more Be sure to restart PostgreSQL for this to take effect. See the [PostgreSQL replication -setup](database.md#postgresql-replication) guide for more details. +setup](../setup/database.md#postgresql-replication) guide for more details. ### Message: `FATAL: could not start WAL streaming: ERROR: replication slot "geo_secondary_my_domain_com" does not exist`? @@ -276,11 +276,11 @@ This occurs when PostgreSQL does not have a replication slot for the **secondary** node by that name. You may want to rerun the [replication -process](database.md) on the **secondary** node . +process](../setup/database.md) on the **secondary** node . ### Message: "Command exceeded allowed execution time" when setting up replication? -This may happen while [initiating the replication process](database.md#step-3-initiate-the-replication-process) on the **secondary** node, +This may happen while [initiating the replication process](../setup/database.md#step-3-initiate-the-replication-process) on the **secondary** node, and indicates that your initial dataset is too large to be replicated in the default timeout (30 minutes). Re-run `gitlab-ctl replicate-geo-database`, but include a larger value for @@ -603,9 +603,9 @@ or `gitlab-ctl promote-to-primary-node`, either: bug](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22021) was fixed. -If the above does not work, another possible reason is that you have paused replication -from the original primary node before attempting to promote this node. +### Message: ActiveRecord::RecordInvalid: Validation failed: Enabled Geo primary node cannot be disabled +This error may occur if you have paused replication from the original primary node before attempting to promote this node. To double check this, you can do the following: - Get the current secondary node's ID using: @@ -632,6 +632,23 @@ To double check this, you can do the following: UPDATE geo_nodes SET enabled = 't' WHERE id = ID_FROM_ABOVE; ``` +### While Promoting the secondary, I got an error `ActiveRecord::RecordInvalid` + +If you disabled a secondary node, either with the [replication pause task](../index.md#pausing-and-resuming-replication) +(13.2) or via the UI (13.1 and earlier), you must first re-enable the +node before you can continue. This is fixed in 13.4. + +From `gitlab-psql`, execute the following, replacing `<your secondary url>` +with the URL for your secondary server starting with `http` or `https` and ending with a `/`. + +```shell +SECONDARY_URL="https://<secondary url>/" +DATABASE_NAME="gitlabhq_production" +sudo gitlab-psql -d "$DATABASE_NAME" -c "UPDATE geo_nodes SET enabled = true WHERE url = '$SECONDARY_URL';" +``` + +This should update 1 row. + ### Message: ``NoMethodError: undefined method `secondary?' for nil:NilClass`` When [promoting a **secondary** node](../disaster_recovery/index.md#step-3-promoting-a-secondary-node), @@ -674,6 +691,20 @@ sudo /opt/gitlab/embedded/bin/gitlab-pg-ctl promote GitLab 12.9 and later are [unaffected by this error](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5147). +### Two-factor authentication is broken after a failover + +The setup instructions for Geo prior to 10.5 failed to replicate the +`otp_key_base` secret, which is used to encrypt the two-factor authentication +secrets stored in the database. If it differs between **primary** and **secondary** +nodes, users with two-factor authentication enabled won't be able to log in +after a failover. + +If you still have access to the old **primary** node, you can follow the +instructions in the +[Upgrading to GitLab 10.5](../replication/version_specific_updates.md#updating-to-gitlab-105) +section to resolve the error. Otherwise, the secret is lost and you'll need to +[reset two-factor authentication for all users](../../../security/two_factor_authentication.md#disabling-2fa-for-everyone). + ## Expired artifacts If you notice for some reason there are more artifacts on the Geo @@ -723,7 +754,7 @@ This error refers to a problem with the database replica on a **secondary** node which Geo expects to have access to. It usually means, either: - An unsupported replication method was used (for example, logical replication). -- The instructions to setup a [Geo database replication](database.md) were not followed correctly. +- The instructions to setup a [Geo database replication](../setup/database.md) were not followed correctly. - Your database connection details are incorrect, that is you have specified the wrong user in your `/etc/gitlab/gitlab.rb` file. @@ -743,7 +774,7 @@ The most common problems that prevent the database from replicating correctly ar - Database replication slot is misconfigured. - Database is not using a replication slot or another alternative and cannot catch-up because WAL files were purged. -Make sure you follow the [Geo database replication](database.md) instructions for supported configuration. +Make sure you follow the [Geo database replication](../setup/database.md) instructions for supported configuration. ### Geo database version (...) does not match latest migration (...) diff --git a/doc/administration/geo/replication/updating_the_geo_nodes.md b/doc/administration/geo/replication/updating_the_geo_nodes.md index c0b1bc6688c..b78aeb06ebf 100644 --- a/doc/administration/geo/replication/updating_the_geo_nodes.md +++ b/doc/administration/geo/replication/updating_the_geo_nodes.md @@ -7,33 +7,15 @@ type: howto # Updating the Geo nodes **(PREMIUM ONLY)** +CAUTION: **Warning:** +Please ensure you read these sections carefully before updating your Geo nodes! Not following version-specific update steps may result in unexpected downtime. Please [contact support](https://about.gitlab.com/support/#contact-support) if you have any specific questions. + Updating Geo nodes involves performing: -1. [Version-specific update steps](#version-specific-update-steps), depending on the +1. [Version-specific update steps](version_specific_updates.md), depending on the version being updated to or from. 1. [General update steps](#general-update-steps), for all updates. -## Version specific update steps - -Depending on which version of Geo you are updating to/from, there may be -different steps. - -- [Updating to GitLab 12.9](version_specific_updates.md#updating-to-gitlab-129) -- [Updating to GitLab 12.7](version_specific_updates.md#updating-to-gitlab-127) -- [Updating to GitLab 12.2](version_specific_updates.md#updating-to-gitlab-122) -- [Updating to GitLab 12.1](version_specific_updates.md#updating-to-gitlab-121) -- [Updating to GitLab 12.0](version_specific_updates.md#updating-to-gitlab-120) -- [Updating to GitLab 11.11](version_specific_updates.md#updating-to-gitlab-1111) -- [Updating to GitLab 10.8](version_specific_updates.md#updating-to-gitlab-108) -- [Updating to GitLab 10.6](version_specific_updates.md#updating-to-gitlab-106) -- [Updating to GitLab 10.5](version_specific_updates.md#updating-to-gitlab-105) -- [Updating to GitLab 10.3](version_specific_updates.md#updating-to-gitlab-103) -- [Updating to GitLab 10.2](version_specific_updates.md#updating-to-gitlab-102) -- [Updating to GitLab 10.1](version_specific_updates.md#updating-to-gitlab-101) -- [Updating to GitLab 10.0](version_specific_updates.md#updating-to-gitlab-100) -- [Updating from GitLab 9.3 or older](version_specific_updates.md#updating-from-gitlab-93-or-older) -- [Updating to GitLab 9.0](version_specific_updates.md#updating-to-gitlab-90) - ## General update steps NOTE: **Note:** @@ -42,12 +24,12 @@ These general update steps are not intended for [high-availability deployments]( To update the Geo nodes when a new GitLab version is released, update **primary** and all **secondary** nodes: -1. **Optional:** [Pause replication on each **secondary** node.](./index.md#pausing-and-resuming-replication) +1. **Optional:** [Pause replication on each **secondary** node.](../index.md#pausing-and-resuming-replication) 1. Log into the **primary** node. 1. [Update GitLab on the **primary** node using Omnibus](https://docs.gitlab.com/omnibus/update/README.html). 1. Log into each **secondary** node. 1. [Update GitLab on each **secondary** node using Omnibus](https://docs.gitlab.com/omnibus/update/README.html). -1. If you paused replication in step 1, [resume replication on each **secondary**](./index.md#pausing-and-resuming-replication) +1. If you paused replication in step 1, [resume replication on each **secondary**](../index.md#pausing-and-resuming-replication) 1. [Test](#check-status-after-updating) **primary** and **secondary** nodes, and check version in each. ### Check status after updating diff --git a/doc/administration/geo/replication/using_a_geo_server.md b/doc/administration/geo/replication/using_a_geo_server.md index 3f2895f1c71..aeed5a44b30 100644 --- a/doc/administration/geo/replication/using_a_geo_server.md +++ b/doc/administration/geo/replication/using_a_geo_server.md @@ -9,7 +9,7 @@ type: howto # Using a Geo Server **(PREMIUM ONLY)** -After you set up the [database replication and configure the Geo nodes](index.md#setup-instructions), use your closest GitLab node as you would a normal standalone GitLab instance. +After you set up the [database replication and configure the Geo nodes](../index.md#setup-instructions), use your closest GitLab node as you would a normal standalone GitLab instance. Pushing directly to a **secondary** node (for both HTTP, SSH including Git LFS) was [introduced](https://about.gitlab.com/releases/2018/09/22/gitlab-11-3-released/) in [GitLab Premium](https://about.gitlab.com/pricing/#self-managed) 11.3. diff --git a/doc/administration/geo/replication/version_specific_updates.md b/doc/administration/geo/replication/version_specific_updates.md index 900d09bdd34..1ae246e3e61 100644 --- a/doc/administration/geo/replication/version_specific_updates.md +++ b/doc/administration/geo/replication/version_specific_updates.md @@ -314,7 +314,7 @@ sudo gitlab-ctl reconfigure ``` If you do not perform this step, you may find that two-factor authentication -[is broken following DR](../disaster_recovery/index.md#i-followed-the-disaster-recovery-instructions-and-now-two-factor-auth-is-broken). +[is broken following DR](troubleshooting.md#two-factor-authentication-is-broken-after-a-failover). To prevent SSH requests to the newly promoted **primary** node from failing due to SSH host key mismatch when updating the **primary** node domain's DNS record @@ -343,7 +343,7 @@ Support for TLS-secured PostgreSQL replication has been added. If you are currently using PostgreSQL replication across the open internet without an external means of securing the connection (e.g., a site-to-site VPN), then you should immediately reconfigure your **primary** and **secondary** PostgreSQL instances -according to the [updated instructions](database.md). +according to the [updated instructions](../setup/database.md). If you *are* securing the connections externally and wish to continue doing so, ensure you include the new option `--sslmode=prefer` in future invocations of @@ -441,7 +441,7 @@ Omnibus is the following: 1. Check the steps about defining `postgresql['sql_user_password']`, `gitlab_rails['db_password']`. 1. Make sure `postgresql['max_replication_slots']` matches the number of **secondary** Geo nodes locations. 1. Install GitLab on the **secondary** server. -1. Re-run the [database replication process](database.md#step-3-initiate-the-replication-process). +1. Re-run the [database replication process](../setup/database.md#step-3-initiate-the-replication-process). ## Updating to GitLab 9.0 diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md new file mode 100644 index 00000000000..aefa8a0e399 --- /dev/null +++ b/doc/administration/geo/setup/database.md @@ -0,0 +1,473 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: howto +--- + +# Geo database replication **(PREMIUM ONLY)** + +NOTE: **Note:** +If your GitLab installation uses external (not managed by Omnibus) PostgreSQL +instances, the Omnibus roles will not be able to perform all necessary +configuration steps. In this case, +[follow the Geo with external PostgreSQL instances document instead](external_database.md). + +NOTE: **Note:** +The stages of the setup process must be completed in the documented order. +Before attempting the steps in this stage, [complete all prior stages](../setup/index.md#using-omnibus-gitlab). + +This document describes the minimal steps you have to take in order to +replicate your **primary** GitLab database to a **secondary** node's database. You may +have to change some values according to your database setup, how big it is, etc. + +You are encouraged to first read through all the steps before executing them +in your testing/production environment. + +## PostgreSQL replication + +The GitLab **primary** node where the write operations happen will connect to +the **primary** database server, and **secondary** nodes will +connect to their own database servers (which are also read-only). + +We recommend using [PostgreSQL replication slots](https://medium.com/@tk512/replication-slots-in-postgresql-b4b03d277c75) +to ensure that the **primary** node retains all the data necessary for the **secondary** nodes to +recover. See below for more details. + +The following guide assumes that: + +- You are using Omnibus and therefore you are using PostgreSQL 11 or later + which includes the [`pg_basebackup` tool](https://www.postgresql.org/docs/11/app-pgbasebackup.html). +- You have a **primary** node already set up (the GitLab server you are + replicating from), running Omnibus' PostgreSQL (or equivalent version), and + you have a new **secondary** server set up with the same versions of the OS, + PostgreSQL, and GitLab on all nodes. + +CAUTION: **Warning:** +Geo works with streaming replication. Logical replication is not supported at this time. +There is an [issue where support is being discussed](https://gitlab.com/gitlab-org/gitlab/-/issues/7420). + +### Step 1. Configure the **primary** server + +1. SSH into your GitLab **primary** server and login as root: + + ```shell + sudo -i + ``` + +1. Edit `/etc/gitlab/gitlab.rb` and add a **unique** name for your node: + + ```ruby + # The unique identifier for the Geo node. + gitlab_rails['geo_node_name'] = '<node_name_here>' + ``` + +1. Reconfigure the **primary** node for the change to take effect: + + ```shell + gitlab-ctl reconfigure + ``` + +1. Execute the command below to define the node as **primary** node: + + ```shell + gitlab-ctl set-geo-primary-node + ``` + + This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`. + +1. GitLab 10.4 and up only: Do the following to make sure the `gitlab` database user has a password defined: + + NOTE: **Note:** + Until FDW settings are removed in GitLab version 14.0, avoid using single or double quotes in the + password for PostgreSQL as that will lead to errors when reconfiguring. + + Generate a MD5 hash of the desired password: + + ```shell + gitlab-ctl pg-password-md5 gitlab + # Enter password: <your_password_here> + # Confirm password: <your_password_here> + # fca0b89a972d69f00eb3ec98a5838484 + ``` + + Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` + postgresql['sql_user_password'] = '<md5_hash_of_your_password>' + + # Every node that runs Puma or Sidekiq needs to have the database + # password specified as below. If you have a high-availability setup, this + # must be present in all application nodes. + gitlab_rails['db_password'] = '<your_password_here>' + ``` + +1. Omnibus GitLab already has a [replication user](https://wiki.postgresql.org/wiki/Streaming_Replication) + called `gitlab_replicator`. You must set the password for this user manually. + You will be prompted to enter a password: + + ```shell + gitlab-ctl set-replication-password + ``` + + This command will also read the `postgresql['sql_replication_user']` Omnibus + setting in case you have changed `gitlab_replicator` username to something + else. + + If you are using an external database not managed by Omnibus GitLab, you need + to create the replicator user and define a password to it manually: + + ```sql + --- Create a new user 'replicator' + CREATE USER gitlab_replicator; + + --- Set/change a password and grants replication privilege + ALTER USER gitlab_replicator WITH REPLICATION ENCRYPTED PASSWORD '<replication_password>'; + ``` + +1. Configure PostgreSQL to listen on network interfaces: + + For security reasons, PostgreSQL does not listen on any network interfaces + by default. However, Geo requires the **secondary** node to be able to + connect to the **primary** node's database. For this reason, we need the address of + each node. + + NOTE: **Note:** + For external PostgreSQL instances, see [additional instructions](external_database.md). + + If you are using a cloud provider, you can lookup the addresses for each + Geo node through your cloud provider's management console. + + To lookup the address of a Geo node, SSH in to the Geo node and execute: + + ```shell + ## + ## Private address + ## + ip route get 255.255.255.255 | awk '{print "Private address:", $NF; exit}' + + ## + ## Public address + ## + echo "External address: $(curl --silent ipinfo.io/ip)" + ``` + + In most cases, the following addresses will be used to configure GitLab + Geo: + + | Configuration | Address | + |:----------------------------------------|:------------------------------------------------------| + | `postgresql['listen_address']` | **Primary** node's public or VPC private address. | + | `postgresql['md5_auth_cidr_addresses']` | **Secondary** node's public or VPC private addresses. | + + If you are using Google Cloud Platform, SoftLayer, or any other vendor that + provides a virtual private cloud (VPC) you can use the **primary** and **secondary** nodes + private addresses (corresponds to "internal address" for Google Cloud Platform) for + `postgresql['md5_auth_cidr_addresses']` and `postgresql['listen_address']`. + + The `listen_address` option opens PostgreSQL up to network connections with the interface + corresponding to the given address. See [the PostgreSQL documentation](https://www.postgresql.org/docs/11/runtime-config-connection.html) + for more details. + + NOTE: **Note:** + If you need to use `0.0.0.0` or `*` as the listen_address, you will also need to add + `127.0.0.1/32` to the `postgresql['md5_auth_cidr_addresses']` setting, to allow Rails to connect through + `127.0.0.1`. For more information, see [omnibus-5258](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5258). + + Depending on your network configuration, the suggested addresses may not + be correct. If your **primary** node and **secondary** nodes connect over a local + area network, or a virtual network connecting availability zones like + [Amazon's VPC](https://aws.amazon.com/vpc/) or [Google's VPC](https://cloud.google.com/vpc/) + you should use the **secondary** node's private address for `postgresql['md5_auth_cidr_addresses']`. + + Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP + addresses with addresses appropriate to your network configuration: + + ```ruby + ## + ## Geo Primary role + ## - configure dependent flags automatically to enable Geo + ## + roles ['geo_primary_role'] + + ## + ## Primary address + ## - replace '<primary_node_ip>' with the public or VPC address of your Geo primary node + ## + postgresql['listen_address'] = '<primary_node_ip>' + + ## + # Allow PostgreSQL client authentication from the primary and secondary IPs. These IPs may be + # public or VPC addresses in CIDR format, for example ['198.51.100.1/32', '198.51.100.2/32'] + ## + postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32'] + + ## + ## Replication settings + ## - set this to be the number of Geo secondary nodes you have + ## + postgresql['max_replication_slots'] = 1 + # postgresql['max_wal_senders'] = 10 + # postgresql['wal_keep_segments'] = 10 + + ## + ## Disable automatic database migrations temporarily + ## (until PostgreSQL is restarted and listening on the private address). + ## + gitlab_rails['auto_migrate'] = false + ``` + +1. Optional: If you want to add another **secondary** node, the relevant setting would look like: + + ```ruby + postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32', '<another_secondary_node_ip>/32'] + ``` + + You may also want to edit the `wal_keep_segments` and `max_wal_senders` to match your + database replication requirements. Consult the [PostgreSQL - Replication documentation](https://www.postgresql.org/docs/11/runtime-config-replication.html) + for more information. + +1. Save the file and reconfigure GitLab for the database listen changes and + the replication slot changes to be applied: + + ```shell + gitlab-ctl reconfigure + ``` + + Restart PostgreSQL for its changes to take effect: + + ```shell + gitlab-ctl restart postgresql + ``` + +1. Re-enable migrations now that PostgreSQL is restarted and listening on the + private address. + + Edit `/etc/gitlab/gitlab.rb` and **change** the configuration to `true`: + + ```ruby + gitlab_rails['auto_migrate'] = true + ``` + + Save the file and reconfigure GitLab: + + ```shell + gitlab-ctl reconfigure + ``` + +1. Now that the PostgreSQL server is set up to accept remote connections, run + `netstat -plnt | grep 5432` to make sure that PostgreSQL is listening on port + `5432` to the **primary** server's private address. + +1. A certificate was automatically generated when GitLab was reconfigured. This + will be used automatically to protect your PostgreSQL traffic from + eavesdroppers, but to protect against active ("man-in-the-middle") attackers, + the **secondary** node needs a copy of the certificate. Make a copy of the PostgreSQL + `server.crt` file on the **primary** node by running this command: + + ```shell + cat ~gitlab-psql/data/server.crt + ``` + + Copy the output into a clipboard or into a local file. You + will need it when setting up the **secondary** node! The certificate is not sensitive + data. + +### Step 2. Configure the **secondary** server + +1. SSH into your GitLab **secondary** server and login as root: + + ```shell + sudo -i + ``` + +1. Stop application server and Sidekiq + + ```shell + gitlab-ctl stop puma + gitlab-ctl stop sidekiq + ``` + + NOTE: **Note:** + This step is important so we don't try to execute anything before the node is fully configured. + +1. [Check TCP connectivity](../../raketasks/maintenance.md) to the **primary** node's PostgreSQL server: + + ```shell + gitlab-rake gitlab:tcp_check[<primary_node_ip>,5432] + ``` + + NOTE: **Note:** + If this step fails, you may be using the wrong IP address, or a firewall may + be preventing access to the server. Check the IP address, paying close + attention to the difference between public and private addresses and ensure + that, if a firewall is present, the **secondary** node is permitted to connect to the + **primary** node on port 5432. + +1. Create a file `server.crt` in the **secondary** server, with the content you got on the last step of the **primary** node's setup: + + ```shell + editor server.crt + ``` + +1. Set up PostgreSQL TLS verification on the **secondary** node: + + Install the `server.crt` file: + + ```shell + install \ + -D \ + -o gitlab-psql \ + -g gitlab-psql \ + -m 0400 \ + -T server.crt ~gitlab-psql/.postgresql/root.crt + ``` + + PostgreSQL will now only recognize that exact certificate when verifying TLS + connections. The certificate can only be replicated by someone with access + to the private key, which is **only** present on the **primary** node. + +1. Test that the `gitlab-psql` user can connect to the **primary** node's database + (the default Omnibus database name is `gitlabhq_production`): + + ```shell + sudo \ + -u gitlab-psql /opt/gitlab/embedded/bin/psql \ + --list \ + -U gitlab_replicator \ + -d "dbname=gitlabhq_production sslmode=verify-ca" \ + -W \ + -h <primary_node_ip> + ``` + + When prompted enter the password you set in the first step for the + `gitlab_replicator` user. If all worked correctly, you should see + the list of **primary** node's databases. + + A failure to connect here indicates that the TLS configuration is incorrect. + Ensure that the contents of `~gitlab-psql/data/server.crt` on the **primary** node + match the contents of `~gitlab-psql/.postgresql/root.crt` on the **secondary** node. + +1. Configure PostgreSQL: + + This step is similar to how we configured the **primary** instance. + We need to enable this, even if using a single node. + + Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP + addresses with addresses appropriate to your network configuration: + + ```ruby + ## + ## Geo Secondary role + ## - configure dependent flags automatically to enable Geo + ## + roles ['geo_secondary_role'] + + ## + ## Secondary address + ## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node + ## + postgresql['listen_address'] = '<secondary_node_ip>' + postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32'] + + ## + ## Database credentials password (defined previously in primary node) + ## - replicate same values here as defined in primary node + ## + postgresql['sql_user_password'] = '<md5_hash_of_your_password>' + gitlab_rails['db_password'] = '<your_password_here>' + ``` + + For external PostgreSQL instances, see [additional instructions](external_database.md). + If you bring a former **primary** node back online to serve as a **secondary** node, then you also need to remove `roles ['geo_primary_role']` or `geo_primary_role['enable'] = true`. + +1. Reconfigure GitLab for the changes to take effect: + + ```shell + gitlab-ctl reconfigure + ``` + +1. Restart PostgreSQL for the IP change to take effect: + + ```shell + gitlab-ctl restart postgresql + ``` + +### Step 3. Initiate the replication process + +Below we provide a script that connects the database on the **secondary** node to +the database on the **primary** node, replicates the database, and creates the +needed files for streaming replication. + +The directories used are the defaults that are set up in Omnibus. If you have +changed any defaults, configure it as you see fit replacing the directories and paths. + +CAUTION: **Warning:** +Make sure to run this on the **secondary** server as it removes all PostgreSQL's +data before running `pg_basebackup`. + +1. SSH into your GitLab **secondary** server and login as root: + + ```shell + sudo -i + ``` + +1. Choose a database-friendly name to use for your **secondary** node to + use as the replication slot name. For example, if your domain is + `secondary.geo.example.com`, you may use `secondary_example` as the slot + name as shown in the commands below. + +1. Execute the command below to start a backup/restore and begin the replication + + CAUTION: **Warning:** + Each Geo **secondary** node must have its own unique replication slot name. + Using the same slot name between two secondaries will break PostgreSQL replication. + + ```shell + gitlab-ctl replicate-geo-database \ + --slot-name=<secondary_node_name> \ + --host=<primary_node_ip> + ``` + + NOTE: **Note:** + Replication slot names must only contain lowercase letters, numbers, and the underscore character. + + When prompted, enter the _plaintext_ password you set up for the `gitlab_replicator` + user in the first step. + + This command also takes a number of additional options. You can use `--help` + to list them all, but here are a couple of tips: + + - If PostgreSQL is listening on a non-standard port, add `--port=` as well. + - If your database is too large to be transferred in 30 minutes, you will need + to increase the timeout, e.g., `--backup-timeout=3600` if you expect the + initial replication to take under an hour. + - Pass `--sslmode=disable` to skip PostgreSQL TLS authentication altogether + (e.g., you know the network path is secure, or you are using a site-to-site + VPN). This is **not** safe over the public Internet! + - You can read more details about each `sslmode` in the + [PostgreSQL documentation](https://www.postgresql.org/docs/11/libpq-ssl.html#LIBPQ-SSL-PROTECTION); + the instructions above are carefully written to ensure protection against + both passive eavesdroppers and active "man-in-the-middle" attackers. + - Change the `--slot-name` to the name of the replication slot + to be used on the **primary** database. The script will attempt to create the + replication slot automatically if it does not exist. + - If you're repurposing an old server into a Geo **secondary** node, you'll need to + add `--force` to the command line. + - When not in a production machine you can disable backup step if you + really sure this is what you want by adding `--skip-backup` + +The replication process is now complete. + +## PgBouncer support (optional) + +[PgBouncer](https://www.pgbouncer.org/) may be used with GitLab Geo to pool +PostgreSQL connections. We recommend using PgBouncer if you use GitLab in a +high-availability configuration with a cluster of nodes supporting a Geo +**primary** node and another cluster of nodes supporting a Geo **secondary** node. For more +information, see [High Availability with Omnibus GitLab](../../postgresql/replication_and_failover.md). + +## Troubleshooting + +Read the [troubleshooting document](../replication/troubleshooting.md). diff --git a/doc/administration/geo/setup/external_database.md b/doc/administration/geo/setup/external_database.md new file mode 100644 index 00000000000..33a7bc38b6e --- /dev/null +++ b/doc/administration/geo/setup/external_database.md @@ -0,0 +1,234 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: howto +--- + +# Geo with external PostgreSQL instances **(PREMIUM ONLY)** + +This document is relevant if you are using a PostgreSQL instance that is *not +managed by Omnibus*. This includes cloud-managed instances like AWS RDS, or +manually installed and configured PostgreSQL instances. + +NOTE: **Note:** +We strongly recommend running Omnibus-managed instances as they are actively +developed and tested. We aim to be compatible with most external +(not managed by Omnibus) databases but we do not guarantee compatibility. + +## **Primary** node + +1. SSH into a GitLab **primary** application server and login as root: + + ```shell + sudo -i + ``` + +1. Edit `/etc/gitlab/gitlab.rb` and add a **unique** ID for your node (arbitrary value): + + ```ruby + # The unique identifier for the Geo node. + gitlab_rails['geo_node_name'] = '<node_name_here>' + ``` + +1. Reconfigure the **primary** node for the change to take effect: + + ```shell + gitlab-ctl reconfigure + ``` + +1. Execute the command below to define the node as **primary** node: + + ```shell + gitlab-ctl set-geo-primary-node + ``` + + This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`. + +### Configure the external database to be replicated + +To set up an external database, you can either: + +- Set up streaming replication yourself (for example, in AWS RDS). +- Perform the Omnibus configuration manually as follows. + +#### Leverage your cloud provider's tools to replicate the primary database + +Given you have a primary node set up on AWS EC2 that uses RDS. +You can now just create a read-only replica in a different region and the +replication process will be managed by AWS. Make sure you've set Network ACL, Subnet, and +Security Group according to your needs, so the secondary application node can access the database. + +The following instructions detail how to create a read-only replica for common +cloud providers: + +- Amazon RDS - [Creating a Read Replica](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.Create) +- Azure Database for PostgreSQL - [Create and manage read replicas in Azure Database for PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/howto-read-replicas-portal) + +Once your read-only replica is set up, you can skip to [configure you secondary application node](#configure-secondary-application-nodes-to-use-the-external-read-replica). + +#### Manually configure the primary database for replication + +The [`geo_primary_role`](https://docs.gitlab.com/omnibus/roles/#gitlab-geo-roles) +configures the **primary** node's database to be replicated by making changes to +`pg_hba.conf` and `postgresql.conf`. Make the following configuration changes +manually to your external database configuration and ensure that you restart PostgreSQL +afterwards for the changes to take effect: + +```plaintext +## +## Geo Primary Role +## - pg_hba.conf +## +host all all <trusted primary IP>/32 md5 +host replication gitlab_replicator <trusted primary IP>/32 md5 +host all all <trusted secondary IP>/32 md5 +host replication gitlab_replicator <trusted secondary IP>/32 md5 +``` + +```plaintext +## +## Geo Primary Role +## - postgresql.conf +## +wal_level = hot_standby +max_wal_senders = 10 +wal_keep_segments = 50 +max_replication_slots = 1 # number of secondary instances +hot_standby = on +``` + +## **Secondary** nodes + +### Manually configure the replica database + +Make the following configuration changes manually to your `pg_hba.conf` and `postgresql.conf` +of your external replica database and ensure that you restart PostgreSQL afterwards +for the changes to take effect: + +```plaintext +## +## Geo Secondary Role +## - pg_hba.conf +## +host all all <trusted secondary IP>/32 md5 +host replication gitlab_replicator <trusted secondary IP>/32 md5 +host all all <trusted primary IP>/24 md5 +``` + +```plaintext +## +## Geo Secondary Role +## - postgresql.conf +## +wal_level = hot_standby +max_wal_senders = 10 +wal_keep_segments = 10 +hot_standby = on +``` + +### Configure **secondary** application nodes to use the external read-replica + +With Omnibus, the +[`geo_secondary_role`](https://docs.gitlab.com/omnibus/roles/#gitlab-geo-roles) +has three main functions: + +1. Configure the replica database. +1. Configure the tracking database. +1. Enable the [Geo Log Cursor](../index.md#geo-log-cursor) (not covered in this section). + +To configure the connection to the external read-replica database and enable Log Cursor: + +1. SSH into a GitLab **secondary** application server and login as root: + + ```shell + sudo -i + ``` + +1. Edit `/etc/gitlab/gitlab.rb` and add the following + + ```ruby + ## + ## Geo Secondary role + ## - configure dependent flags automatically to enable Geo + ## + roles ['geo_secondary_role'] + + # note this is shared between both databases, + # make sure you define the same password in both + gitlab_rails['db_password'] = '<your_password_here>' + + gitlab_rails['db_username'] = 'gitlab' + gitlab_rails['db_host'] = '<database_read_replica_host>' + + # Disable the bundled Omnibus PostgreSQL, since we are + # using an external PostgreSQL + postgresql['enable'] = false + ``` + +1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) + +### Configure the tracking database + +**Secondary** nodes use a separate PostgreSQL installation as a tracking +database to keep track of replication status and automatically recover from +potential replication issues. Omnibus automatically configures a tracking database +when `roles ['geo_secondary_role']` is set. +If you want to run this database external to Omnibus, please follow the instructions below. + +If you are using a cloud-managed service for the tracking database, you may need +to grant additional roles to your tracking database user (by default, this is +`gitlab_geo`): + +- Amazon RDS requires the [`rds_superuser`](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.html#Appendix.PostgreSQL.CommonDBATasks.Roles) role. +- Azure Database for PostgreSQL requires the [`azure_pg_admin`](https://docs.microsoft.com/en-us/azure/postgresql/howto-create-users#how-to-create-additional-admin-users-in-azure-database-for-postgresql) role. + +If you have an external database ready to be used as the tracking database, +follow the instructions below to use it: + +NOTE: **Note:** +If you want to use AWS RDS as a tracking database, make sure it has access to +the secondary database. Unfortunately, just assigning the same security group is not enough as +outbound rules do not apply to RDS PostgreSQL databases. Therefore, you need to explicitly add an inbound +rule to the read-replica's security group allowing any TCP traffic from +the tracking database on port 5432. + +1. Ensure that your secondary node can communicate with your tracking database by + manually changing the `pg_hba.conf` that is associated with your tracking database. + Remember to restart PostgreSQL afterwards for the changes to take effect: + + ```plaintext + ## + ## Geo Tracking Database Role + ## - pg_hba.conf + ## + host all all <trusted tracking IP>/32 md5 + host all all <trusted secondary IP>/32 md5 + ``` + +1. SSH into a GitLab **secondary** server and login as root: + + ```shell + sudo -i + ``` + +1. Edit `/etc/gitlab/gitlab.rb` with the connection parameters and credentials for + the machine with the PostgreSQL instance: + + ```ruby + geo_secondary['db_username'] = 'gitlab_geo' + geo_secondary['db_password'] = '<your_password_here>' + + geo_secondary['db_host'] = '<tracking_database_host>' + geo_secondary['db_port'] = <tracking_database_port> # change to the correct port + geo_postgresql['enable'] = false # don't use internal managed instance + ``` + +1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) + +1. Run the tracking database migrations: + + ```shell + gitlab-rake geo:db:create + gitlab-rake geo:db:migrate + ``` diff --git a/doc/administration/geo/setup/index.md b/doc/administration/geo/setup/index.md new file mode 100644 index 00000000000..a4d9fcba14d --- /dev/null +++ b/doc/administration/geo/setup/index.md @@ -0,0 +1,32 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: howto +--- + +# Setting up Geo + +These instructions assume you have a working instance of GitLab. They guide you through: + +1. Making your existing instance the **primary** node. +1. Adding **secondary** nodes. + +CAUTION: **Caution:** +The steps below should be followed in the order they appear. **Make sure the GitLab version is the same on all nodes.** + +## Using Omnibus GitLab + +If you installed GitLab using the Omnibus packages (highly recommended): + +1. [Install GitLab Enterprise Edition](https://about.gitlab.com/install/) on the server that will serve as the **secondary** node. Do not create an account or log in to the new **secondary** node. +1. [Upload the GitLab License](../../../user/admin_area/license.md) on the **primary** node to unlock Geo. The license must be for [GitLab Premium](https://about.gitlab.com/pricing/) or higher. +1. [Set up the database replication](database.md) (`primary (read-write) <-> secondary (read-only)` topology). +1. [Configure fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md). This step is required and needs to be done on **both** the **primary** and **secondary** nodes. +1. [Configure GitLab](../replication/configuration.md) to set the **primary** and **secondary** nodes. +1. Optional: [Configure a secondary LDAP server](../../auth/ldap/index.md) for the **secondary** node. See [notes on LDAP](../index.md#ldap). +1. [Follow the "Using a Geo Server" guide](../replication/using_a_geo_server.md). + +## Post-installation documentation + +After installing GitLab on the **secondary** nodes and performing the initial configuration, see the [following documentation for post-installation information](../index.md#post-installation-documentation). diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 9558488c89e..e6b137bac29 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -988,9 +988,12 @@ When GitLab calls a function that has a "Rugged patch", it performs two checks: - Is the feature flag for this patch set in the database? If so, the feature flag setting controls GitLab's use of "Rugged patch" code. - If the feature flag is not set, GitLab tries accessing the filesystem underneath the - Gitaly server directly. If it can, it will use the "Rugged patch". + Gitaly server directly. If it can, it will use the "Rugged patch": + - If using Unicorn. + - If using Puma and [thread count](../../install/requirements.md#puma-threads) is set + to `1`. -The result of both of these checks is cached. +The result of these checks is cached. To see if GitLab can access the repository filesystem directly, we use the following heuristic: @@ -1072,6 +1075,24 @@ You can run a gRPC trace with: sudo GRPC_TRACE=all GRPC_VERBOSITY=DEBUG gitlab-rake gitlab:gitaly:check ``` +### Correlating Git processes with RPCs + +Sometimes you need to find out which Gitaly RPC created a particular Git process. + +One method for doing this is via `DEBUG` logging. However, this needs to be enabled +ahead of time and the logs produced are quite verbose. + +A lightweight method for doing this correlation is by inspecting the environment +of the Git process (using its `PID`) and looking at the `CORRELATION_ID` variable: + +```shell +PID=<Git process ID> +sudo cat /proc/$PID/environ | tr '\0' '\n' | grep ^CORRELATION_ID= +``` + +Please note that this method is not reliable for `git cat-file` processes because Gitaly +internally pools and re-uses those across RPCs. + ### Observing `gitaly-ruby` traffic [`gitaly-ruby`](#gitaly-ruby) is an internal implementation detail of Gitaly, diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index 2e9e036c24e..876904a2093 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -129,7 +129,7 @@ the Omnibus GitLab distribution is not yet supported. Follow this Prepare all your new nodes by [installing GitLab](https://about.gitlab.com/install/). -- 1 Praefect node (minimal storage required) +- At least 1 Praefect node (minimal storage required) - 3 Gitaly nodes (high CPU, high memory, fast storage) - 1 GitLab server @@ -171,7 +171,7 @@ We will note in the instructions below where these secrets are required. NOTE: **Note:** Do not store the GitLab application database and the Praefect database on the same PostgreSQL server if using -[Geo](../geo/replication/index.md). The replication state is internal to each instance +[Geo](../geo/index.md). The replication state is internal to each instance of GitLab and should not be replicated. These instructions help set up a single PostgreSQL database, which creates a single point of @@ -232,18 +232,19 @@ The database used by Praefect is now configured. #### PgBouncer -To reduce PostgreSQL resource consumption, you should set up and configure +To reduce PostgreSQL resource consumption, we recommend setting up and configuring [PgBouncer](https://www.pgbouncer.org/) in front of the PostgreSQL instance. To do this, replace value of the `POSTGRESQL_SERVER_ADDRESS` with corresponding IP or host address of the PgBouncer instance. This documentation doesn't provide PgBouncer installation instructions, -you can: +but you can: - Find instructions on the [official website](https://www.pgbouncer.org/install.html). - Use a [Docker image](https://hub.docker.com/r/edoburu/pgbouncer/). -In addition to base PgBouncer configuration options, set the following values: +In addition to the base PgBouncer configuration options, set the following values in +your `pgbouncer.ini` file: - The [Praefect PostgreSQL database](#postgresql) in the `[databases]` section: @@ -275,6 +276,11 @@ PostgreSQL instances. Otherwise you should change the configuration parameter ### Praefect +> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/2634) in GitLab 13.4, Praefect nodes can no longer be designated as `primary`. + +NOTE: **Note:** +If there are multiple Praefect nodes, complete these steps for **each** node. + To complete this section you will need: - [Configured PostgreSQL server](#postgresql), including: @@ -376,7 +382,7 @@ application server, or a Gitaly node. CAUTION: **Caution:** If you have data on an already existing storage called `default`, you should configure the virtual storage with another name and - [migrate the data to the Praefect storage](#migrating-existing-repositories-to-praefect) + [migrate the data to the Gitaly Cluster storage](#migrate-existing-repositories-to-gitaly-cluster) afterwards. Replace `PRAEFECT_INTERNAL_TOKEN` with a strong secret, which will be used by @@ -388,11 +394,6 @@ application server, or a Gitaly node. More Gitaly nodes can be added to the cluster to increase the number of replicas. More clusters can also be added for very large GitLab instances. - NOTE: **Note:** - The `gitaly-1` node is currently denoted the primary. This - can be used to manually fail from one node to another. This will be removed - in the [future](https://gitlab.com/gitlab-org/gitaly/-/issues/2634). - ```ruby # Name of storage hash must match storage name in git_data_dirs on GitLab # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1') @@ -401,7 +402,6 @@ application server, or a Gitaly node. 'gitaly-1' => { 'address' => 'tcp://GITALY_HOST:8075', 'token' => 'PRAEFECT_INTERNAL_TOKEN', - 'primary' => true }, 'gitaly-2' => { 'address' => 'tcp://GITALY_HOST:8075', @@ -426,7 +426,7 @@ application server, or a Gitaly node. 1. To ensure that Praefect [has updated its Prometheus listen address](https://gitlab.com/gitlab-org/gitaly/-/issues/2734), [restart - Gitaly](../restart_gitlab.md#omnibus-gitlab-restart): + Praefect](../restart_gitlab.md#omnibus-gitlab-restart): ```shell gitlab-ctl restart praefect @@ -444,7 +444,7 @@ application server, or a Gitaly node. **The steps above must be completed for each Praefect node!** -## Enabling TLS support +#### Enabling TLS support > [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/1698) in GitLab 13.2. @@ -677,7 +677,7 @@ documentation](index.md#configure-gitaly-servers). # Configure the gitlab-shell API callback URL. Without this, `git push` will # fail. This can be your front door GitLab URL or an internal load balancer. - # Examples: 'https://example.gitlab.com', 'http://1.2.3.4' + # Examples: 'https://gitlab.example.com', 'http://1.2.3.4' gitlab_rails['internal_api_url'] = 'http://GITLAB_HOST' ``` @@ -730,7 +730,7 @@ After all Gitaly nodes are configured, you can run the Praefect connection checker to verify Praefect can connect to all Gitaly servers in the Praefect config. -1. SSH into the **Praefect** node and run the Praefect connection checker: +1. SSH into each **Praefect** node and run the Praefect connection checker: ```shell sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dial-nodes @@ -774,9 +774,9 @@ application. This is done by updating the `git_data_dirs`. Particular attention should be shown to: - the storage name added to `git_data_dirs` in this section must match the - storage name under `praefect['virtual_storages']` on the Praefect node. This + storage name under `praefect['virtual_storages']` on the Praefect node(s). This was set in the [Praefect](#praefect) section of this guide. This document uses - `storage-1` as the Praefect storage name. + `default` as the Praefect storage name. 1. SSH into the **GitLab** node and login as root: @@ -799,7 +799,8 @@ Particular attention should be shown to: CAUTION: **Caution:** If you have existing data stored on the default Gitaly storage, - you should [migrate the data your Praefect storage first](#migrating-existing-repositories-to-praefect). + you should [migrate the data your Gitaly Cluster storage](#migrate-existing-repositories-to-gitaly-cluster) + first. ```ruby gitaly['enable'] = false @@ -833,7 +834,8 @@ Particular attention should be shown to: gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN' ``` -1. Add Prometheus monitoring settings by editing `/etc/gitlab/gitlab.rb`. +1. Add Prometheus monitoring settings by editing `/etc/gitlab/gitlab.rb`. If Prometheus + is enabled on a different node, make edits on that node instead. You will need to replace: @@ -871,7 +873,7 @@ Particular attention should be shown to: gitlab-ctl reconfigure ``` -1. Verify each `gitlab-shell` on each Gitaly instance can reach GitLab. On each Gitaly instance run: +1. Verify each `gitlab-shell` on each Gitaly node can reach GitLab. On each Gitaly node run: ```shell /opt/gitlab/embedded/service/gitlab-shell/bin/check -config /opt/gitlab/embedded/service/gitlab-shell/config.yml @@ -901,7 +903,7 @@ for detailed documentation. To get started quickly: -1. SSH into the **GitLab** node and login as root: +1. SSH into the **GitLab** node (or whichever node has Grafana enabled) and login as root: ```shell sudo -i @@ -978,6 +980,7 @@ They reflect configuration defined for this instance of Praefect. > - Introduced in GitLab 13.1 in [alpha](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga), disabled by default. > - Entered [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga) in GitLab 13.2, disabled by default. > - From GitLab 13.3, disabled unless primary-wins reference transactions strategy is disabled. +> - From GitLab 13.4, enabled by default. Praefect guarantees eventual consistency by replicating all writes to secondary nodes after the write to the primary Gitaly node has happened. @@ -990,8 +993,13 @@ information, see the [strong consistency epic](https://gitlab.com/groups/gitlab- To enable strong consistency: -- In GitLab 13.3 and later, reference transactions are enabled by default with - a primary-wins strategy. This strategy causes all transactions to succeed for +- In GitLab 13.4 and later, the strong consistency voting strategy has been + improved. Instead of requiring all nodes to agree, only the primary and half + of the secondaries need to agree. This strategy is enabled by default. To + disable it and continue using the primary-wins strategy, enable the + `:gitaly_reference_transactions_primary_wins` feature flag. +- In GitLab 13.3, reference transactions are enabled by default with a + primary-wins strategy. This strategy causes all transactions to succeed for the primary and thus does not ensure strong consistency. To enable strong consistency, disable the `:gitaly_reference_transactions_primary_wins` feature flag. @@ -1034,11 +1042,6 @@ current primary node is found to be unhealthy. will cause Praefect nodes to elect a new primary, monitor its health, and elect a new primary if the current one has not been reachable in 10 seconds by a majority of the Praefect nodes. -- **Manual:** Automatic failover is disabled. The primary node can be - reconfigured in `/etc/gitlab/gitlab.rb` on the Praefect node. Modify the - `praefect['virtual_storages']` field by moving the `primary = true` to promote - a different Gitaly node to primary. In the steps above, `gitaly-1` was set to - the primary. Requires `praefect['failover_enabled'] = false` in the configuration. - **Memory:** Enabled by setting `praefect['failover_election_strategy'] = 'local'` in `/etc/gitlab/gitlab.rb` on the Praefect node. If a sufficient number of health checks fail for the current primary backend Gitaly node, and new primary will @@ -1072,7 +1075,7 @@ recovery efforts by preventing writes that may conflict with the unreplicated wr To enable writes again, an administrator can: 1. [Check](#check-for-data-loss) for data loss. -1. Attempt to [recover](#recover-missing-data) missing data. +1. Attempt to [recover](#data-recovery) missing data. 1. Either [enable writes](#enable-writes-or-accept-data-loss) in the virtual storage or [accept data loss](#enable-writes-or-accept-data-loss) if necessary, depending on the version of GitLab. @@ -1166,17 +1169,6 @@ Virtual storage: default To check a project's repository checksums across on all Gitaly nodes, run the [replicas Rake task](../raketasks/praefect.md#replica-checksums) on the main GitLab node. -### Recover missing data - -The Praefect `reconcile` sub-command can be used to recover unreplicated changes from another replica. -The source must be on a later version than the target storage. - -```shell -sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml reconcile -virtual <virtual-storage> -reference <up-to-date-storage> -target <outdated-storage> -f -``` - -Refer to [Gitaly node recovery](#gitaly-node-recovery) section for more details on the `reconcile` sub-command. - ### Enable writes or accept data loss Praefect provides the following subcommands to re-enable writes: @@ -1200,43 +1192,85 @@ Praefect provides the following subcommands to re-enable writes: CAUTION: **Caution:** `accept-dataloss` causes permanent data loss by overwriting other versions of the repository. Data -[recovery efforts](#recover-missing-data) must be performed before using it. +[recovery efforts](#data-recovery) must be performed before using it. + +## Data recovery + +If a Gitaly node fails replication jobs for any reason, it ends up hosting outdated versions of the +affected repositories. Praefect provides tools for: + +- [Automatic](#automatic-reconciliation) reconciliation, for GitLab 13.4 and later. +- [Manual](#manual-reconciliation) reconciliation, for: + - GitLab 13.3 and earlier. + - Repositories upgraded to GitLab 13.4 and later without entries in the `repositories` table. + A migration tool [is planned](https://gitlab.com/gitlab-org/gitaly/-/issues/3033). + +These tools reconcile the outdated repositories to bring them fully up to date again. + +### Automatic reconciliation + +> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/2717) in GitLab 13.4. -## Gitaly node recovery +Praefect automatically reconciles repositories that are not up to date. By default, this is done every +five minutes. For each outdated repository on a healthy Gitaly node, the Praefect picks a +random, fully up to date replica of the repository on another healthy Gitaly node to replicate from. A +replication job is scheduled only if there are no other replication jobs pending for the target +repository. -When a secondary Gitaly node fails and is no longer able to replicate changes, it starts -to drift from the primary Gitaly node. If the failed Gitaly node eventually recovers, -it needs to be reconciled with the primary Gitaly node. The primary Gitaly node is considered -the single source of truth for the state of a shard. +The reconciliation frequency can be changed via the configuration. The value can be any valid +[Go duration value](https://golang.org/pkg/time/#ParseDuration). Values below 0 disable the feature. -The Praefect `reconcile` sub-command allows for the manual reconciliation between a secondary -Gitaly node and the current primary Gitaly node. +Examples: -Run the following command on the Praefect server after all placeholders -(`<virtual-storage>` and `<target-storage>`) have been replaced: +```ruby +praefect['reconciliation_scheduling_interval'] = '5m' # the default value +``` + +```ruby +praefect['reconciliation_scheduling_interval'] = '30s' # reconcile every 30 seconds +``` + +```ruby +praefect['reconciliation_scheduling_interval'] = '0' # disable the feature +``` + +### Manual reconciliation + +The Praefect `reconcile` sub-command allows for the manual reconciliation between two Gitaly nodes. The +command replicates every repository on a later version on the reference storage to the target storage. ```shell -sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml reconcile -virtual <virtual-storage> -target <target-storage> +sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml reconcile -virtual <virtual-storage> -reference <up-to-date-storage> -target <outdated-storage> -f ``` - Replace the placeholder `<virtual-storage>` with the virtual storage containing the Gitaly node storage to be checked. -- Replace the placeholder `<target-storage>` with the Gitaly storage name. - -The command will return a list of repositories that were found to be -inconsistent against the current primary. Each of these inconsistencies will -also be logged with an accompanying replication job ID. +- Replace the placeholder `<up-to-date-storage>` with the Gitaly storage name containing up to date repositories. +- Replace the placeholder `<outdated-storage>` with the Gitaly storage name containing outdated repositories. -## Migrating existing repositories to Praefect +## Migrate existing repositories to Gitaly Cluster -If your GitLab instance already has repositories, these won't be migrated -automatically. +If your GitLab instance already has repositories on single Gitaly nodes, these aren't migrated to +Gitaly Cluster automatically. Repositories may be moved from one storage location using the [Project repository storage moves API](../../api/project_repository_storage_moves.md): -```shell -curl --request POST --header "Private-Token: <your_access_token>" --header "Content-Type: application/json" \ ---data '{"destination_storage_name":"praefect"}' "https://gitlab.example.com/api/v4/projects/123/repository_storage_moves" -``` +To move repositories to Gitaly Cluster: + +1. [Schedule a move](../../api/project_repository_storage_moves.md#schedule-a-repository-storage-move-for-a-project) + for the first repository using the API. For example: + + ```shell + curl --request POST --header "Private-Token: <your_access_token>" --header "Content-Type: application/json" \ + --data '{"destination_storage_name":"praefect"}' "https://gitlab.example.com/api/v4/projects/123/repository_storage_moves" + ``` + +1. Using the ID that is returned, [query the repository move](../../api/project_repository_storage_moves.md#get-a-single-repository-storage-move-for-a-project) + using the API. The query indicates either: + - The move has completed successfully. The `state` field is `finished`. + - The move is in progress. Re-query the repository move until it completes successfully. + - The move has failed. Most failures are temporary and are solved by rescheduling the move. + +1. Once the move is successful, repeat these steps for all repositories for your projects. ## Debugging Praefect diff --git a/doc/administration/img/export_audit_log_v13_4.png b/doc/administration/img/export_audit_log_v13_4.png Binary files differnew file mode 100644 index 00000000000..1b404b5742c --- /dev/null +++ b/doc/administration/img/export_audit_log_v13_4.png diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index 36156c4a580..c0c03044225 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -13,7 +13,7 @@ GitLab has several features based on receiving incoming emails: - [New issue by email](../user/project/issues/managing_issues.md#new-issue-via-email): allow GitLab users to create a new issue by sending an email to a user-specific email address. -- [New merge request by email](../user/project/merge_requests/creating_merge_requests.md#new-merge-request-by-email-core-only): +- [New merge request by email](../user/project/merge_requests/creating_merge_requests.md#new-merge-request-by-email): allow GitLab users to create a new merge request by sending an email to a user-specific email address. - [Service Desk](../user/project/service_desk.md): provide e-mail support to @@ -84,8 +84,8 @@ To set up a basic Postfix mail server with IMAP access on Ubuntu, follow the ### Security Concerns -**WARNING:** Be careful when choosing the domain used for receiving incoming -email. +WARNING: **WARNING:** +Be careful when choosing the domain used for receiving incoming email. For the sake of example, suppose your top-level company domain is `hooli.com`. All employees in your company have an email address at that domain via Google @@ -95,7 +95,7 @@ email address in order to sign up. If you also host a public-facing GitLab instance at `hooli.com` and set your incoming email domain to `hooli.com`, an attacker could abuse the "Create new issue by email" or -"[Create new merge request by email](../user/project/merge_requests/creating_merge_requests.md#new-merge-request-by-email-core-only)" +"[Create new merge request by email](../user/project/merge_requests/creating_merge_requests.md#new-merge-request-by-email)" features by using a project's unique address as the email when signing up for Slack. This would send a confirmation email, which would create a new issue or merge request on the project owned by the attacker, allowing them to click the @@ -111,7 +111,7 @@ Alternatively, use a dedicated domain for GitLab email communications such as See GitLab issue [#30366](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/30366) for a real-world example of this exploit. -CAUTION:**Caution:** +CAUTION: **Caution:** Be sure to use a mail server that has been configured to reduce spam. A Postfix mail server that is running on a default configuration, for example, diff --git a/doc/administration/index.md b/doc/administration/index.md index ed079abf708..a6448fcf64f 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -36,7 +36,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Omnibus support for log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-shipping-gitlab-enterprise-edition-only) **(STARTER ONLY)** - [Reference architectures](reference_architectures/index.md): Add additional resources to support more users. - [Installing GitLab on Amazon Web Services (AWS)](../install/aws/index.md): Set up GitLab on Amazon AWS. -- [Geo](geo/replication/index.md): Replicate your GitLab instance to other geographic locations as a read-only fully operational version. **(PREMIUM ONLY)** +- [Geo](geo/index.md): Replicate your GitLab instance to other geographic locations as a read-only fully operational version. **(PREMIUM ONLY)** - [Disaster Recovery](geo/disaster_recovery/index.md): Quickly fail-over to a different site with minimal effort in a disaster situation. **(PREMIUM ONLY)** - [Pivotal Tile](../install/pivotal/index.md): Deploy GitLab as a preconfigured appliance using Ops Manager (BOSH) for Pivotal Cloud Foundry. **(PREMIUM ONLY)** - [Add License](../user/admin_area/license.md): Upload a license at install time to unlock features that are in paid tiers of GitLab. **(STARTER ONLY)** @@ -60,7 +60,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Diff limits](../user/admin_area/diff_limits.md): Configure the diff rendering size limits of branch comparison pages. - [Merge request diffs storage](merge_request_diffs.md): Configure merge requests diffs external storage. - [Broadcast Messages](../user/admin_area/broadcast_messages.md): Send messages to GitLab users through the UI. -- [Elasticsearch](../integration/elasticsearch.md): Enable Elasticsearch to empower GitLab's Advanced Global Search. Useful when you deal with a huge amount of data. **(STARTER ONLY)** +- [Elasticsearch](../integration/elasticsearch.md): Enable Elasticsearch to empower GitLab's Advanced Search. Useful when you deal with a huge amount of data. **(STARTER ONLY)** - [External Classification Policy Authorization](../user/admin_area/settings/external_authorization.md) **(PREMIUM ONLY)** - [Upload a license](../user/admin_area/license.md): Upload a license to unlock features that are in paid tiers of GitLab. **(STARTER ONLY)** - [Admin Area](../user/admin_area/index.md): for self-managed instance-wide configuration and maintenance. @@ -73,7 +73,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Favicon](../user/admin_area/appearance.md#favicon): Change the default favicon to your own logo. - [Branded login page](../user/admin_area/appearance.md#sign-in--sign-up-pages): Customize the login page with your own logo, title, and description. - ["New Project" page](../user/admin_area/appearance.md#new-project-pages): Customize the text to be displayed on the page that opens whenever your users create a new project. -- [Additional custom email text](../user/admin_area/settings/email.md#custom-additional-text-premium-only): Add additional custom text to emails sent from GitLab. **(PREMIUM ONLY)** +- [Additional custom email text](../user/admin_area/settings/email.md#custom-additional-text): Add additional custom text to emails sent from GitLab. **(PREMIUM ONLY)** ### Maintaining GitLab @@ -120,7 +120,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Auditor users](auditor_users.md): Users with read-only access to all projects, groups, and other resources on the GitLab instance. **(PREMIUM ONLY)** - [Incoming email](incoming_email.md): Configure incoming emails to allow users to [reply by email](reply_by_email.md), create [issues by email](../user/project/issues/managing_issues.md#new-issue-via-email) and - [merge requests by email](../user/project/merge_requests/creating_merge_requests.md#new-merge-request-by-email-core-only), and to enable [Service Desk](../user/project/service_desk.md). + [merge requests by email](../user/project/merge_requests/creating_merge_requests.md#new-merge-request-by-email), and to enable [Service Desk](../user/project/service_desk.md). - [Postfix for incoming email](reply_by_email_postfix_setup.md): Set up a basic Postfix mail server with IMAP authentication on Ubuntu for incoming emails. @@ -157,8 +157,8 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [External Pipeline Validation](external_pipeline_validation.md): Enable, disable and configure external pipeline validation. - [Job artifacts](job_artifacts.md): Enable, disable, and configure job artifacts (a set of files and directories which are outputted by a job when it completes successfully). - [Job logs](job_logs.md): Information about the job logs. -- [Register Runners](../ci/runners/README.md#types-of-runners): Learn how to register and configure Runners. -- [Shared Runners pipelines quota](../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota-starter-only): Limit the usage of pipeline minutes for Shared Runners. **(STARTER ONLY)** +- [Register runners](../ci/runners/README.md#types-of-runners): Learn how to register and configure runners. +- [Shared runners pipelines quota](../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota): Limit the usage of pipeline minutes for shared runners. **(STARTER ONLY)** - [Enable/disable Auto DevOps](../topics/autodevops/index.md#enablingdisabling-auto-devops): Enable or disable Auto DevOps for your instance. ## Snippet settings @@ -231,6 +231,6 @@ who are aware of the risks. - [GitLab Developer Docs](../development/README.md) - [Repairing and recovering broken Git repositories](https://git.seveas.net/repairing-and-recovering-broken-git-repositories.html) - [Testing with OpenSSL](https://www.feistyduck.com/library/openssl-cookbook/online/ch-testing-with-openssl.html) - - [`Strace` zine](https://wizardzines.com/zines/strace/) + - [`strace` zine](https://wizardzines.com/zines/strace/) - GitLab.com-specific resources: - [Group SAML/SCIM setup](troubleshooting/group_saml_scim.md) diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index f30dba331b8..abd98002934 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -319,9 +319,9 @@ Plan.default.actual_limits.update!(ci_instance_level_variables: 30) > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37226) in GitLab 13.3. Job artifacts defined with [`artifacts:reports`](../ci/pipelines/job_artifacts.md#artifactsreports) -that are uploaded by the Runner are rejected if the file size exceeds the maximum +that are uploaded by the runner are rejected if the file size exceeds the maximum file size limit. The limit is determined by comparing the project's -[maximum artifact size setting](../user/admin_area/settings/continuous_integration.md#maximum-artifacts-size-core-only) +[maximum artifact size setting](../user/admin_area/settings/continuous_integration.md#maximum-artifacts-size) with the instance limit for the given artifact type, and choosing the smaller value. Limits are set in megabytes, so the smallest possible value that can be defined is `1 MB`. @@ -424,6 +424,12 @@ panel_groups: label: Legend Label ``` +## Environment Dashboard limits **(PREMIUM)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/33895) in GitLab 13.4. + +See [Environment Dashboard](../ci/environments/environments_dashboard.md#adding-a-project-to-the-dashboard) for the maximum number of displayed projects. + ## Environment data on Deploy Boards [Deploy Boards](../user/project/deploy_boards.md) load information from Kubernetes about @@ -434,11 +440,11 @@ Kubernetes won't be shown. Reports that go over the 20 MB limit won't be loaded. Affected reports: -- [Merge Request security reports](../user/project/merge_requests/testing_and_reports_in_merge_requests.md#security-reports-ultimate) +- [Merge Request security reports](../user/project/merge_requests/testing_and_reports_in_merge_requests.md#security-reports) - [CI/CD parameter `artifacts:expose_as`](../ci/yaml/README.md#artifactsexpose_as) -- [JUnit test reports](../ci/junit_test_reports.md) +- [Unit test reports](../ci/unit_test_reports.md) -## Advanced Global Search limits +## Advanced Search limits ### Maximum file size indexed @@ -514,3 +520,38 @@ Total number of changes (branches or tags) in a single push to determine whether individual push events or bulk push event will be created. More information can be found in the [Push event activities limit and bulk push events documentation](../user/admin_area/settings/push_event_activities_limit.md). + +## Package Registry Limits + +### File Size Limits + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218017) in GitLab 13.4. + +On GitLab.com, the maximum file size for a package that's uploaded to the [GitLab Package Registry](../user/packages/package_registry/index.md) +is 5 gigabytes. + +Limits are set per package type. + +To set this limit on a self-managed installation, run the following in the +[GitLab Rails console](troubleshooting/debug.md#starting-a-rails-console-session): + +```ruby +# File size limit is stored in bytes + +# For Conan Packages +Plan.default.actual_limits.update!(conan_max_file_size: 100.megabytes) + +# For NPM Packages +Plan.default.actual_limits.update!(npm_max_file_size: 100.megabytes) + +# For NuGet Packages +Plan.default.actual_limits.update!(nuget_max_file_size: 100.megabytes) + +# For Maven Packages +Plan.default.actual_limits.update!(maven_max_file_size: 100.megabytes) + +# For PyPI Packages +Plan.default.actual_limits.update!(pypi_max_file_size: 100.megabytes) +``` + +Set the limit to `0` to allow any file size. diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 66a7bcb90f6..2a79923b793 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -404,7 +404,7 @@ you can enable the `ci_disable_validates_dependencies` feature flag from a Rails ## Set the maximum file size of the artifacts If artifacts are enabled, you can change the maximum file size of the -artifacts through the [Admin Area settings](../user/admin_area/settings/continuous_integration.md#maximum-artifacts-size-core-only). +artifacts through the [Admin Area settings](../user/admin_area/settings/continuous_integration.md#maximum-artifacts-size). ## Storage statistics diff --git a/doc/administration/job_logs.md b/doc/administration/job_logs.md index 4dba33b796a..c34035e3c0c 100644 --- a/doc/administration/job_logs.md +++ b/doc/administration/job_logs.md @@ -9,7 +9,7 @@ type: reference > [Renamed from job traces to job logs](https://gitlab.com/gitlab-org/gitlab/-/issues/29121) in GitLab 12.5. -Job logs are sent by GitLab Runner while it's processing a job. You can see +Job logs are sent by a runner while it's processing a job. You can see logs in job pages, pipelines, email notifications, etc. ## Data flow @@ -19,8 +19,8 @@ In the following table you can see the phases a log goes through: | Phase | State | Condition | Data flow | Stored path | | -------------- | ------------ | ----------------------- | -----------------------------------------| ----------- | -| 1: patching | log | When a job is running | GitLab Runner => Puma => file storage | `#{ROOT_PATH}/gitlab-ci/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log` | -| 2: overwriting | log | When a job is finished | GitLab Runner => Puma => file storage | `#{ROOT_PATH}/gitlab-ci/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log` | +| 1: patching | log | When a job is running | Runner => Puma => file storage | `#{ROOT_PATH}/gitlab-ci/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log` | +| 2: overwriting | log | When a job is finished | Runner => Puma => file storage | `#{ROOT_PATH}/gitlab-ci/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log` | | 3: archiving | archived log | After a job is finished | Sidekiq moves log to artifacts folder | `#{ROOT_PATH}/gitlab-rails/shared/artifacts/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log` | | 4: uploading | archived log | After a log is archived | Sidekiq moves archived log to [object storage](#uploading-logs-to-object-storage) (if configured) | `#{bucket_name}/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log` | @@ -83,10 +83,9 @@ find /var/opt/gitlab/gitlab-rails/shared/artifacts -name "job.log" -mtime +60 -d ## New incremental logging architecture > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/18169) in GitLab 10.4. -> - [Announced as generally available](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/46097) in GitLab 11.0. NOTE: **Note:** -This feature is off by default. See below for how to [enable or disable](#enabling-incremental-logging) it. +This beta feature is off by default. See below for how to [enable or disable](#enabling-incremental-logging) it. By combining the process with object storage settings, we can completely bypass the local file storage. This is a useful option if GitLab is installed as @@ -103,8 +102,8 @@ The data are stored in the following Redis namespace: `Gitlab::Redis::SharedStat Here is the detailed data flow: -1. GitLab Runner picks a job from GitLab -1. GitLab Runner sends a piece of log to GitLab +1. The runner picks a job from GitLab +1. The runner sends a piece of log to GitLab 1. GitLab appends the data to Redis 1. Once the data in Redis reach 128KB, the data is flushed to a persistent store (object storage or the database). 1. The above steps are repeated until the job is finished. @@ -161,7 +160,7 @@ In some cases, having data stored on Redis could incur data loss: 1. **Case 1: When all data in Redis are accidentally flushed** - On going incremental logs could be recovered by re-sending logs (this is - supported by all versions of the GitLab Runner). + supported by all versions of GitLab Runner). - Finished jobs which have not archived incremental logs will lose the last part (~128KB) of log data. diff --git a/doc/administration/logs.md b/doc/administration/logs.md index 2e8d0bf7461..fcd6264dafd 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -87,7 +87,7 @@ In addition, the log contains the originating IP address, (`remote_ip`), the user's ID (`user_id`), and username (`username`). Some endpoints such as `/search` may make requests to Elasticsearch if using -[Advanced Global Search](../user/search/advanced_global_search.md). These will +[Advanced Search](../user/search/advanced_global_search.md). These will additionally log `elasticsearch_calls` and `elasticsearch_call_duration_s`, which correspond to: @@ -723,10 +723,15 @@ was initiated, such as `1509705644.log` ## `sidekiq_exporter.log` and `web_exporter.log` -If Prometheus metrics and the Sidekiq Exporter are both enabled, Sidekiq will -start a Web server and listen to the defined port (default: `8082`). Access logs -will be generated in `/var/log/gitlab/gitlab-rails/sidekiq_exporter.log` for -Omnibus GitLab packages or in `/home/git/gitlab/log/sidekiq_exporter.log` for +If Prometheus metrics and the Sidekiq Exporter are both enabled, Sidekiq +will start a Web server and listen to the defined port (default: +`8082`). By default, Sidekiq Exporter access logs are disabled but can +be enabled via the `sidekiq['exporter_log_enabled'] = true` option in `/etc/gitlab/gitlab.rb` +for Omnibus installations, or via the `sidekiq_exporter.log_enabled` option +in `gitlab.yml` for installations from source. When enabled, +access logs will be generated in +`/var/log/gitlab/gitlab-rails/sidekiq_exporter.log` for Omnibus GitLab +packages or in `/home/git/gitlab/log/sidekiq_exporter.log` for installations from source. If Prometheus metrics and the Web Exporter are both enabled, Puma/Unicorn will @@ -955,3 +960,38 @@ For Omnibus GitLab installations, GitLab Monitor logs reside in `/var/log/gitlab ## GitLab Exporter For Omnibus GitLab installations, GitLab Exporter logs reside in `/var/log/gitlab/gitlab-exporter/`. + +## Gathering logs + +When [troubleshooting](troubleshooting/index.md) issues that aren't localized to one of the +previously listed components, it's helpful to simultaneously gather multiple logs and statistics +from a GitLab instance. + +### GitLabSOS + +If performance degradations or cascading errors occur that can't readily be attributed to one +of the previously listed GitLab components, [GitLabSOS](https://gitlab.com/gitlab-com/support/toolbox/gitlabsos/) +can provide a perspective spanning all of Omnibus GitLab. For more details and instructions +to run it, see [the GitLabSOS documentation](https://gitlab.com/gitlab-com/support/toolbox/gitlabsos/#gitlabsos). + +NOTE: **Note:** +GitLab Support likes to use this custom-made tool. + +### Briefly tail the main logs + +If the bug or error is readily reproducible bug or error, save the main GitLab logs +[to a file](troubleshooting/linux_cheat_sheet.md#files--dirs) while reproducing the +problem once or more times: + +```shell +sudo gitlab-ctl tail | tee /tmp/<case-ID-and-keywords>.log +``` + +Conclude the log gathering with <kbd>Ctrl</kbd> + <kbd>C</kbd>. + +### Fast-stats + +[Fast-stats](https://gitlab.com/gitlab-com/support/toolbox/fast-stats) is a tool +for creating and comparing performance statistics from GitLab logs. +For more details and instructions to run it, see +[read the documentation for fast-stats](https://gitlab.com/gitlab-com/support/toolbox/fast-stats#usage). diff --git a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md index e272cccb7ce..efe31997b25 100644 --- a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md +++ b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md @@ -77,7 +77,7 @@ You can [add a webhook](../../../operations/metrics/alerts.md#external-prometheu to the Prometheus configuration in order for GitLab to receive notifications of any alerts. Once the webhook is setup, you can -[take action on incoming alerts](../../../operations/metrics/alerts.md#trigger-actions-from-alerts-ultimate). +[take action on incoming alerts](../../../operations/metrics/alerts.md#trigger-actions-from-alerts). ## Adding custom metrics to the self monitoring project @@ -93,7 +93,7 @@ You can add custom metrics in the self monitoring project by: There is [a bug](https://gitlab.com/gitlab-org/gitlab/-/issues/208676) which causes project creation to fail with the following error (which appears in the log file) when the first admin user is an -[external user](../../../user/permissions.md#external-users-core-only): +[external user](../../../user/permissions.md#external-users): ```plaintext Could not create instance administrators group. Errors: ["You don’t have permission to create groups."] @@ -108,6 +108,6 @@ User.admins.active.first.external? If this returns true, the first admin user is an external user. If you face this issue, you can temporarily -[make the admin user a non-external user](../../../user/permissions.md#external-users-core-only) +[make the admin user a non-external user](../../../user/permissions.md#external-users) and then try to create the project. Once the project is created, the admin user can be changed back to an external user. diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index bff689c0c0c..ae31a3db023 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -106,6 +106,13 @@ The following metrics are available: | `successful_login_captcha_total` | Gauge | 11.0 | Counter of successful CAPTCHA attempts during login | | | `auto_devops_pipelines_completed_total` | Counter | 12.7 | Counter of completed Auto DevOps pipelines, labeled by status | | | `gitlab_metrics_dashboard_processing_time_ms` | Summary | 12.10 | Metrics dashboard processing time in milliseconds | service, stages | +| `action_cable_active_connections` | Gauge | 13.4 | Number of ActionCable WS clients currently connected | `server_mode` | +| `action_cable_pool_min_size` | Gauge | 13.4 | Minimum number of worker threads in ActionCable thread pool | `server_mode` | +| `action_cable_pool_max_size` | Gauge | 13.4 | Maximum number of worker threads in ActionCable thread pool | `server_mode` | +| `action_cable_pool_current_size` | Gauge | 13.4 | Current number of worker threads in ActionCable thread pool | `server_mode` | +| `action_cable_pool_largest_size` | Gauge | 13.4 | Largest number of worker threads observed so far in ActionCable thread pool | `server_mode` | +| `action_cable_pool_pending_tasks` | Gauge | 13.4 | Number of tasks waiting to be executed in ActionCable thread pool | `server_mode` | +| `action_cable_pool_tasks_total` | Gauge | 13.4 | Total number of tasks executed in ActionCable thread pool | `server_mode` | ## Metrics controlled by a feature flag @@ -157,25 +164,46 @@ configuration option in `gitlab.yml`. These metrics are served from the | `geo_lfs_objects_synced_missing_on_primary` | Gauge | 10.7 | Number of LFS objects marked as synced due to the file missing on the primary | `url` | | `geo_job_artifacts_synced_missing_on_primary` | Gauge | 10.7 | Number of job artifacts marked as synced due to the file missing on the primary | `url` | | `geo_attachments_synced_missing_on_primary` | Gauge | 10.7 | Number of attachments marked as synced due to the file missing on the primary | `url` | -| `geo_repositories_checksummed_count` | Gauge | 10.7 | Number of repositories checksummed on primary | `url` | -| `geo_repositories_checksum_failed_count` | Gauge | 10.7 | Number of repositories failed to calculate the checksum on primary | `url` | -| `geo_wikis_checksummed_count` | Gauge | 10.7 | Number of wikis checksummed on primary | `url` | -| `geo_wikis_checksum_failed_count` | Gauge | 10.7 | Number of wikis failed to calculate the checksum on primary | `url` | -| `geo_repositories_verified_count` | Gauge | 10.7 | Number of repositories verified on secondary | `url` | -| `geo_repositories_verification_failed_count` | Gauge | 10.7 | Number of repositories failed to verify on secondary | `url` | -| `geo_repositories_checksum_mismatch_count` | Gauge | 10.7 | Number of repositories that checksum mismatch on secondary | `url` | -| `geo_wikis_verified_count` | Gauge | 10.7 | Number of wikis verified on secondary | `url` | -| `geo_wikis_verification_failed_count` | Gauge | 10.7 | Number of wikis failed to verify on secondary | `url` | -| `geo_wikis_checksum_mismatch_count` | Gauge | 10.7 | Number of wikis that checksum mismatch on secondary | `url` | -| `geo_repositories_checked_count` | Gauge | 11.1 | Number of repositories that have been checked via `git fsck` | `url` | -| `geo_repositories_checked_failed_count` | Gauge | 11.1 | Number of repositories that have a failure from `git fsck` | `url` | -| `geo_repositories_retrying_verification_count` | Gauge | 11.2 | Number of repositories verification failures that Geo is actively trying to correct on secondary | `url` | -| `geo_wikis_retrying_verification_count` | Gauge | 11.2 | Number of wikis verification failures that Geo is actively trying to correct on secondary | `url` | +| `geo_repositories_checksummed` | Gauge | 10.7 | Number of repositories checksummed on primary | `url` | +| `geo_repositories_checksum_failed` | Gauge | 10.7 | Number of repositories failed to calculate the checksum on primary | `url` | +| `geo_wikis_checksummed` | Gauge | 10.7 | Number of wikis checksummed on primary | `url` | +| `geo_wikis_checksum_failed` | Gauge | 10.7 | Number of wikis failed to calculate the checksum on primary | `url` | +| `geo_repositories_verified` | Gauge | 10.7 | Number of repositories verified on secondary | `url` | +| `geo_repositories_verification_failed` | Gauge | 10.7 | Number of repositories failed to verify on secondary | `url` | +| `geo_repositories_checksum_mismatch` | Gauge | 10.7 | Number of repositories that checksum mismatch on secondary | `url` | +| `geo_wikis_verified` | Gauge | 10.7 | Number of wikis verified on secondary | `url` | +| `geo_wikis_verification_failed` | Gauge | 10.7 | Number of wikis failed to verify on secondary | `url` | +| `geo_wikis_checksum_mismatch` | Gauge | 10.7 | Number of wikis that checksum mismatch on secondary | `url` | +| `geo_repositories_checked` | Gauge | 11.1 | Number of repositories that have been checked via `git fsck` | `url` | +| `geo_repositories_checked_failed` | Gauge | 11.1 | Number of repositories that have a failure from `git fsck` | `url` | +| `geo_repositories_retrying_verification` | Gauge | 11.2 | Number of repositories verification failures that Geo is actively trying to correct on secondary | `url` | +| `geo_wikis_retrying_verification` | Gauge | 11.2 | Number of wikis verification failures that Geo is actively trying to correct on secondary | `url` | +| `geo_package_files` | Gauge | 13.0 | Number of package files on primary | `url` | +| `geo_package_files_checksummed` | Gauge | 13.0 | Number of package files checksummed on primary | `url` | +| `geo_package_files_checksum_failed` | Gauge | 13.0 | Number of package files failed to calculate the checksum on primary | `url` | +| `geo_package_files_synced` | Gauge | 13.3 | Number of syncable package files synced on secondary | `url` | +| `geo_package_files_failed` | Gauge | 13.3 | Number of syncable package files failed to sync on secondary | `url` | +| `geo_package_files_registry` | Gauge | 13.3 | Number of package files in the registry | `url` | +| `geo_terraform_states` | Gauge | 13.3 | Number of terraform states on primary | `url` | +| `geo_terraform_states_checksummed` | Gauge | 13.3 | Number of terraform states checksummed on primary | `url` | +| `geo_terraform_states_checksum_failed` | Gauge | 13.3 | Number of terraform states failed to calculate the checksum on primary | `url` | +| `geo_terraform_states_synced` | Gauge | 13.3 | Number of syncable terraform states synced on secondary | `url` | +| `geo_terraform_states_failed` | Gauge | 13.3 | Number of syncable terraform states failed to sync on secondary | `url` | +| `geo_terraform_states_registry` | Gauge | 13.3 | Number of terraform states in the registry | `url` | | `global_search_bulk_cron_queue_size` | Gauge | 12.10 | Number of database records waiting to be synchronized to Elasticsearch | | | `global_search_awaiting_indexing_queue_size` | Gauge | 13.2 | Number of database updates waiting to be synchronized to Elasticsearch while indexing is paused | | -| `package_files_count` | Gauge | 13.0 | Number of package files on primary | `url` | -| `package_files_checksummed_count` | Gauge | 13.0 | Number of package files checksummed on primary | `url` | -| `package_files_checksum_failed_count` | Gauge | 13.0 | Number of package files failed to calculate the checksum on primary +| `geo_merge_request_diffs` | Gauge | 13.4 | Number of merge request diffs on primary | `url` | +| `geo_merge_request_diffs_checksummed` | Gauge | 13.4 | Number of merge request diffs checksummed on primary | `url` | +| `geo_merge_request_diffs_checksum_failed` | Gauge | 13.4 | Number of merge request diffs failed to calculate the checksum on primary | `url` | +| `geo_merge_request_diffs_synced` | Gauge | 13.4 | Number of syncable merge request diffs synced on secondary | `url` | +| `geo_merge_request_diffs_failed` | Gauge | 13.4 | Number of syncable merge request diffs failed to sync on secondary | `url` | +| `geo_merge_request_diffs_registry` | Gauge | 13.4 | Number of merge request diffs in the registry | `url` | +| `geo_snippet_repositories` | Gauge | 13.4 | Number of snippets on primary | `url` | +| `geo_snippet_repositories_checksummed` | Gauge | 13.4 | Number of snippets checksummed on primary | `url` | +| `geo_snippet_repositories_checksum_failed` | Gauge | 13.4 | Number of snippets failed to calculate the checksum on primary | `url` | +| `geo_snippet_repositories_synced` | Gauge | 13.4 | Number of syncable snippets synced on secondary | `url` | +| `geo_snippet_repositories_failed` | Gauge | 13.4 | Number of syncable snippets failed on secondary | `url` | +| `geo_snippet_repositories_registry` | Gauge | 13.4 | Number of syncable snippets in the registry | `url` | ## Database load balancing metrics **(PREMIUM ONLY)** @@ -185,6 +213,15 @@ The following metrics are available: |:--------------------------------- |:--------- |:------------------------------------------------------------- |:-------------------------------------- | | `db_load_balancing_hosts` | Gauge | [12.3](https://gitlab.com/gitlab-org/gitlab/-/issues/13630) | Current number of load balancing hosts | +## Database partitioning metrics **(PREMIUM ONLY)** + +The following metrics are available: + +| Metric | Type | Since | Description | +|:--------------------------------- |:--------- |:------------------------------------------------------------- |:----------------------------------------------------------------- | +| `db_partitions_present` | Gauge | [13.4](https://gitlab.com/gitlab-org/gitlab/-/issues/227353) | Number of database partitions present | +| `db_partitions_missing` | Gauge | [13.4](https://gitlab.com/gitlab-org/gitlab/-/issues/227353) | Number of database partitions currently expected, but not present | + ## Connection pool metrics These metrics record the status of the database diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md index bae6bd0dd6c..b54f05ad536 100644 --- a/doc/administration/nfs.md +++ b/doc/administration/nfs.md @@ -31,10 +31,10 @@ bug](https://bugzilla.redhat.com/show_bug.cgi?id=1783554) that causes following GitLab versions include a fix to work properly with that kernel version: -1. [12.10.12](https://about.gitlab.com/releases/2020/06/25/gitlab-12-10-12-released/) -1. [13.0.7](https://about.gitlab.com/releases/2020/06/25/gitlab-13-0-7-released/) -1. [13.1.1](https://about.gitlab.com/releases/2020/06/24/gitlab-13-1-1-released/) -1. 13.2 and up +- [12.10.12](https://about.gitlab.com/releases/2020/06/25/gitlab-12-10-12-released/) +- [13.0.7](https://about.gitlab.com/releases/2020/06/25/gitlab-13-0-7-released/) +- [13.1.1](https://about.gitlab.com/releases/2020/06/24/gitlab-13-1-1-released/) +- 13.2 and up If you are using that kernel version, be sure to upgrade GitLab to avoid errors. @@ -127,7 +127,9 @@ administrators to keep NFS server delegation disabled. ### Improving NFS performance with GitLab -#### Improving NFS performance with Unicorn +NFS performance with GitLab can in some cases be improved with +[direct Git access](gitaly/index.md#direct-access-to-git-in-gitlab) using +[Rugged](https://github.com/libgit2/rugged). NOTE: **Note:** From GitLab 12.1, it will automatically be detected if Rugged can and should be used per storage. @@ -138,18 +140,16 @@ If you previously enabled Rugged using the feature flag, you will need to unset sudo gitlab-rake gitlab:features:unset_rugged ``` -If the Rugged feature flag is explicitly set to either true or false, GitLab will use the value explicitly set. +If the Rugged feature flag is explicitly set to either `true` or `false`, GitLab will use the value explicitly set. #### Improving NFS performance with Puma NOTE: **Note:** -From GitLab 12.7, Rugged auto-detection is disabled if Puma thread count is greater than 1. +From GitLab 12.7, Rugged is not automatically enabled if Puma thread count is greater than `1`. -If you want to use Rugged with Puma, it is recommended to [set Puma thread count to 1](https://docs.gitlab.com/omnibus/settings/puma.html#puma-settings). +If you want to use Rugged with Puma, [set Puma thread count to `1`](https://docs.gitlab.com/omnibus/settings/puma.html#puma-settings). -If you want to use Rugged with Puma thread count more than 1, Rugged can be enabled using the [feature flag](../development/gitaly.md#legacy-rugged-code) - -If the Rugged feature flag is explicitly set to either true or false, GitLab will use the value explicitly set. +If you want to use Rugged with Puma thread count more than `1`, Rugged can be enabled using the [feature flag](../development/gitaly.md#legacy-rugged-code). ## NFS client @@ -173,6 +173,9 @@ Here is an example snippet to add to `/etc/fstab`: 10.1.0.1:/var/opt/gitlab/git-data /var/opt/gitlab/git-data nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 ``` +You can view information and options set for each of the mounted NFS file +systems by running `nfsstat -m` and `cat /etc/fstab`. + Note there are several options that you should consider using: | Setting | Description | @@ -271,9 +274,6 @@ Using bind mounts will require manually making sure the data directories are empty before attempting a restore. Read more about the [restore prerequisites](../raketasks/backup_restore.md). -You can view information and options set for each of the mounted NFS file -systems by running `nfsstat -m` and `cat /etc/fstab`. - ### Multiple NFS mounts When using default Omnibus configuration you will need to share 4 data locations diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index 49716883310..39365ffe404 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -18,6 +18,7 @@ GitLab has been tested on a number of object storage providers: - [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces/) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [Openstack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) +- [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) - On-premises hardware and appliances from various storage vendors. - MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. @@ -50,12 +51,17 @@ Using the consolidated object storage configuration has a number of advantages: - It enables the use of [encrypted S3 buckets](#encrypted-s3-buckets). - It [uploads files to S3 with proper `Content-MD5` headers](https://gitlab.com/gitlab-org/gitlab-workhorse/-/issues/222). -NOTE: **Note:** -Only AWS S3-compatible providers and Google are -supported at the moment since [direct upload -mode](../development/uploads.md#direct-upload) must be used. Background -upload is not supported in this mode. We recommend direct upload mode because -it does not require a shared folder, and [this setting may become the default](https://gitlab.com/gitlab-org/gitlab/-/issues/27331). +Because [direct upload mode](../development/uploads.md#direct-upload) +must be enabled, only the following providers can be used: + +- [Amazon S3-compatible providers](#s3-compatible-connection-settings) +- [Google Cloud Storage](#google-cloud-storage-gcs) +- [Azure Blob storage](#azure-blob-storage) + +Background upload is not supported with the consolidated object storage +configuration. We recommend enabling direct upload mode because it does +not require a shared folder, and [this setting may become the +default](https://gitlab.com/gitlab-org/gitlab/-/issues/27331). NOTE: **Note:** Consolidated object storage configuration cannot be used for @@ -112,7 +118,7 @@ See the section on [ETag mismatch errors](#etag-mismatch) for more details. AWS access key and secret access key/value pairs. For example: ```ruby - gitlab_rails['object_store_connection'] = { + gitlab_rails['object_store']['connection'] = { 'provider' => 'AWS', 'region' => '<eu-central-1>', 'use_iam_profile' => true @@ -158,7 +164,6 @@ See the section on [ETag mismatch errors](#etag-mismatch) for more details. ```toml [object_storage] - enabled = true provider = "AWS" [object_storage.s3] @@ -272,6 +277,61 @@ gitlab_rails['object_store']['connection'] = { } ``` +#### Azure Blob storage + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25877) in GitLab 13.4. + +Although Azure uses the word `container` to denote a collection of +blobs, GitLab standardizes on the term `bucket`. Be sure to configure +Azure container names in the `bucket` settings. + +The following are the valid connection parameters for Azure. Read the +[Azure Blob storage documentation](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) +to learn more. + +| Setting | Description | Example | +|---------|-------------|---------| +| `provider` | Provider name | `AzureRM` | +| `azure_storage_account_name` | Name of the Azure Blob Storage account used to access the storage | `azuretest` | +| `azure_storage_access_key` | Storage account access key used to access the container. This is typically a secret, 512-bit encryption key encoded in base64. | `czV2OHkvQj9FKEgrTWJRZVRoV21ZcTN0Nnc5eiRDJkYpSkBOY1JmVWpYbjJy\nNHU3eCFBJUQqRy1LYVBkU2dWaw==\n` | +| `azure_storage_domain` | Domain name used to contact the Azure Blob Storage API (optional). Defaults to `blob.core.windows.net`. Set this if you are using Azure China, Azure Germany, Azure US Government, or some other custom Azure domain. | `blob.core.windows.net` | + +##### Azure example (consolidated form) + +For Omnibus installations, this is an example of the `connection` setting: + +```ruby +gitlab_rails['object_store']['connection'] = { + 'provider' => 'AzureRM', + 'azure_storage_account_name' => '<AZURE STORAGE ACCOUNT NAME>', + 'azure_storage_access_key' => '<AZURE STORAGE ACCESS KEY>', + 'azure_storage_domain' => '<AZURE STORAGE DOMAIN>', +} +``` + +###### Azure Workhorse settings (source installs only) + +NOTE: **Note:** +For source installations, Workhorse needs to be configured with the +Azure credentials as well. This is not needed in Omnibus installs because +the Workhorse settings are populated from the settings above. + +1. Edit `/home/git/gitlab-workhorse/config.toml` and add or amend the following lines: + + ```toml + [object_storage] + provider = "AzureRM" + + [object_storage.azurerm] + azure_storage_account_name = "<AZURE STORAGE ACCOUNT NAME>" + azure_storage_access_key = "<AZURE STORAGE ACCESS KEY>" + ``` + +If you are using a custom Azure storage domain, note that +`azure_storage_domain` does **not** have to be set in the Workhorse +configuration. This information is exchanged in an API call between +GitLab Rails and Workhorse. + #### OpenStack-compatible connection settings NOTE: **Note:** @@ -279,7 +339,7 @@ This is not compatible with the consolidated object storage form. OpenStack Swift is only supported with the storage-specific form. See the [S3 settings](#s3-compatible-connection-settings) if you want to use the consolidated form. -While OpenStack Swift provides S3 compatibliity, some users may want to use the +While OpenStack Swift provides S3 compatibility, some users may want to use the [Swift API](https://docs.openstack.org/swift/latest/api/object_api_v1_overview.html). Here are the valid connection settings below for the Swift API, provided by [fog-openstack](https://github.com/fog/fog-openstack): @@ -445,15 +505,15 @@ supported by consolidated configuration form, refer to the following guides: | [Backups](../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage)|No| | [Job artifacts](job_artifacts.md#using-object-storage) and [incremental logging](job_logs.md#new-incremental-logging-architecture) | Yes | | [LFS objects](lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | -| [Uploads](uploads.md#using-object-storage-core-only) | Yes | +| [Uploads](uploads.md#using-object-storage) | Yes | | [Container Registry](packages/container_registry.md#use-object-storage) (optional feature) | No | | [Merge request diffs](merge_request_diffs.md#using-object-storage) | Yes | | [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | | [Packages](packages/index.md#using-object-storage) (optional feature) **(PREMIUM ONLY)** | Yes | | [Dependency Proxy](packages/dependency_proxy.md#using-object-storage) (optional feature) **(PREMIUM ONLY)** | Yes | | [Pseudonymizer](pseudonymizer.md#configuration) (optional feature) **(ULTIMATE ONLY)** | No | -| [Autoscale Runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | -| [Terraform state files](terraform_state.md#using-object-storage-core-only) | Yes | +| [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | +| [Terraform state files](terraform_state.md#using-object-storage) | Yes | ### Other alternatives to filesystem storage diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index b874a4257f0..6cd393be330 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -32,10 +32,10 @@ feature for CentOS 6, follow [the instructions on how to build and install a cus By default, GitLab manages an `authorized_keys` file, which contains all the public SSH keys for users allowed to access GitLab. However, to maintain a -single source of truth, [Geo](../geo/replication/index.md) needs to be configured to perform SSH fingerprint +single source of truth, [Geo](../geo/index.md) needs to be configured to perform SSH fingerprint lookups via database lookup. -As part of [setting up Geo](../geo/replication/index.md#setup-instructions), +As part of [setting up Geo](../geo/index.md#setup-instructions), you will be required to follow the steps outlined below for both the primary and secondary nodes, but note that the `Write to "authorized keys" file` checkbox only needs to be unchecked on the primary node since it will be reflected diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md index e7b4bb88faf..f5b09d7a978 100644 --- a/doc/administration/operations/puma.md +++ b/doc/administration/operations/puma.md @@ -1,6 +1,6 @@ # Switching to Puma -As of GitLab 12.9, [Puma](https://github.com/puma/puma) has replaced [Unicorn](https://yhbt.net/unicorn/). +As of GitLab 12.9, [Puma](https://github.com/puma/puma) has replaced [Unicorn](https://yhbt.net/unicorn/) as the default web server. From GitLab 13.0, the following run Puma instead of Unicorn unless explicitly configured not to: @@ -36,7 +36,8 @@ For deployments where NFS is used to store Git repository, we allow GitLab to us [Rugged](https://github.com/libgit2/rugged). Rugged usage is automatically enabled if direct Git access -[is available](../gitaly/index.md#how-it-works), unless it is disabled by +[is available](../gitaly/index.md#how-it-works) +and Puma is running single threaded, unless it is disabled by [feature flags](../../development/gitaly.md#legacy-rugged-code). MRI Ruby uses a GVL. This allows MRI Ruby to be multi-threaded, but running at @@ -49,7 +50,7 @@ We are actively working on removing Rugged usage. Even though performance withou is acceptable today, in some cases it might be still beneficial to run with it. Given the caveat of running Rugged with multi-threaded Puma, and acceptable -performance of Gitaly, we are disabling Rugged usage if Puma multi-threaded is +performance of Gitaly, we disable Rugged usage if Puma multi-threaded is used (when Puma is configured to run with more than one thread). This default behavior may not be the optimal configuration in some situations. If Rugged @@ -57,7 +58,6 @@ plays an important role in your deployment, we suggest you benchmark to find the optimal configuration: - The safest option is to start with single-threaded Puma. When working with -Rugged, single-threaded Puma does work the same as Unicorn. - -- To force Rugged auto detect with multi-threaded Puma, you can use [feature -flags](../../development/gitaly.md#legacy-rugged-code). + Rugged, single-threaded Puma works the same as Unicorn. +- To force Rugged to be used with multi-threaded Puma, you can use + [feature flags](../../development/gitaly.md#legacy-rugged-code). diff --git a/doc/administration/operations/sidekiq_memory_killer.md b/doc/administration/operations/sidekiq_memory_killer.md index e829d735c4f..d1ff98a0079 100644 --- a/doc/administration/operations/sidekiq_memory_killer.md +++ b/doc/administration/operations/sidekiq_memory_killer.md @@ -26,8 +26,8 @@ run as a process group leader (e.g., using `chpst -P`). If using Omnibus or the The MemoryKiller is controlled using environment variables. -- `SIDEKIQ_DAEMON_MEMORY_KILLER`: defaults to 0. When set to 1, the MemoryKiller - works in _daemon_ mode. Otherwise, the MemoryKiller works in _legacy_ mode. +- `SIDEKIQ_DAEMON_MEMORY_KILLER`: defaults to 1. When set to 0, the MemoryKiller + works in _legacy_ mode. Otherwise, the MemoryKiller works in _daemon_ mode. In _legacy_ mode, the MemoryKiller checks the Sidekiq process RSS after each job. diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index 1883f6659e6..74af5c8149b 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -750,11 +750,15 @@ U = <user_id> # Get required details / objects user = User.find_by_id(U) project = Project.find_by_id(P) -repo = ContainerRepository.find_by(project_id: P) policy = ContainerExpirationPolicy.find_by(project_id: P) -# Start the tag cleanup -Projects::ContainerRepository::CleanupTagsService.new(project, user, policy.attributes.except("created_at", "updated_at")).execute(repo) +# Loop through each container repository +project.container_repositories.find_each do |repo| + puts repo.attributes + + # Start the tag cleanup + puts Projects::ContainerRepository::CleanupTagsService.new(project, user, policy.attributes.except("created_at", "updated_at")).execute(repo) +end ``` NOTE: **Note:** @@ -988,8 +992,8 @@ thus the error above. While GitLab doesn't support using self-signed certificates with Container Registry out of the box, it is possible to make it work by [instructing the Docker daemon to trust the self-signed certificates](https://docs.docker.com/registry/insecure/#use-self-signed-certificates), -mounting the Docker daemon and setting `privileged = false` in the Runner's -`config.toml`. Setting `privileged = true` takes precedence over the Docker daemon: +mounting the Docker daemon and setting `privileged = false` in the GitLab Runner +`config.toml` file. Setting `privileged = true` takes precedence over the Docker daemon: ```toml [runners.docker] diff --git a/doc/administration/packages/index.md b/doc/administration/packages/index.md index 3af1f0c933b..1061f3c33db 100644 --- a/doc/administration/packages/index.md +++ b/doc/administration/packages/index.md @@ -109,7 +109,6 @@ We recommend using the [consolidated object storage settings](../object_storage. ```ruby gitlab_rails['packages_enabled'] = true - gitlab_rails['packages_storage_path'] = "/var/opt/gitlab/gitlab-rails/shared/packages" gitlab_rails['packages_object_store_enabled'] = true gitlab_rails['packages_object_store_remote_directory'] = "packages" # The bucket name. gitlab_rails['packages_object_store_direct_upload'] = false # Use Object Storage directly for uploads instead of background uploads if enabled (Default: false). @@ -123,6 +122,8 @@ We recommend using the [consolidated object storage settings](../object_storage. #'region' => 'eu-west-1', #'aws_access_key_id' => 'AWS_ACCESS_KEY_ID', #'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY', + ## If an IAM profile is being used with AWS, omit the aws_access_key_id and aws_secret_access_key and uncomment + #'use_iam_profile' => true, ## ## If the provider is other than AWS (an S3-compatible one), uncomment the following ## diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index 9e2aa602767..3c0030be629 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -235,6 +235,7 @@ control over how the Pages daemon runs and serves content in your environment. | `pages_path` | The directory on disk where pages are stored, defaults to `GITLAB-RAILS/shared/pages`. | `pages_nginx[]` | | | `enable` | Include a virtual host `server{}` block for Pages inside NGINX. Needed for NGINX to proxy traffic back to the Pages daemon. Set to `false` if the Pages daemon should directly receive all requests, for example, when using [custom domains](index.md#custom-domains). +| `FF_ENABLE_REDIRECTS` | Feature flag to enable redirects. See the [redirects documentation](../../user/project/pages/redirects.md#enable-or-disable-redirects) for more info. | --- @@ -424,6 +425,10 @@ Authority (CA) in the system certificate store. For Omnibus, this is fixed by [installing a custom CA in Omnibus GitLab](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). +## Enable redirects + +In GitLab Pages, you can [enable the redirects feature](../../user/project/pages/redirects.md#enable-or-disable-redirects) to configure rules to forward one URL to another using HTTP redirects. + ## Activate verbose logging for daemon Verbose logging was [introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/2533) in @@ -588,8 +593,9 @@ database encryption. Proceed with caution. 1. On the **GitLab server**, make the following changes to `/etc/gitlab/gitlab.rb`: ```ruby - gitlab_pages['enable'] = false pages_external_url "http://<pages_server_URL>" + gitlab_pages['enable'] = false + gitlab_rails['pages_enabled']=false gitlab_rails['pages_path'] = "/mnt/pages" ``` @@ -622,7 +628,7 @@ For more details see this [blog post](https://about.gitlab.com/blog/2020/08/03/h GitLab Pages can use an API-based configuration. This replaces disk source configuration, which was used prior to GitLab 13.0. Follow these steps to enable it: -1. Add the following to your `/etc/gitlab/gitlab.erb` file: +1. Add the following to your `/etc/gitlab/gitlab.rb` file: ```ruby gitlab_pages['domain_config_source'] = "gitlab" @@ -720,6 +726,24 @@ sudo cp /opt/gitlab/embedded/ssl/certs/cacert.pem /var/opt/gitlab/gitlab-rails/s sudo cp /opt/gitlab/embedded/ssl/certs/cacert.pem /var/opt/gitlab/gitlab-rails/shared/pages/etc/ssl/ca-bundle.pem ``` +### 502 error when connecting to GitLab Pages proxy when server does not listen over IPv6 + +In some cases, NGINX might default to using IPv6 to connect to the GitLab Pages +service even when the server does not listen over IPv6. You can identify when +this is happening if you see something similar to the log entry below in the +`gitlab_pages_error.log`: + +```plaintext +2020/02/24 16:32:05 [error] 112654#0: *4982804 connect() failed (111: Connection refused) while connecting to upstream, client: 123.123.123.123, server: ~^(?<group>.*)\.pages\.example\.com$, request: "GET /-/group/project/-/jobs/1234/artifacts/artifact.txt HTTP/1.1", upstream: "http://[::1]:8090//-/group/project/-/jobs/1234/artifacts/artifact.txt", host: "group.example.com" +``` + +To resolve this, set an explicit IP and port for the GitLab Pages `listen_proxy` setting +to define the explicit address that the GitLab Pages daemon should listen on: + +```ruby +gitlab_pages['listen_proxy'] = '127.0.0.1:8090' +``` + ### 404 error after transferring project to a different group or user If you encounter a `404 Not Found` error a Pages site after transferring a project to diff --git a/doc/administration/pages/source.md b/doc/administration/pages/source.md index 486bc7a8777..662817e7411 100644 --- a/doc/administration/pages/source.md +++ b/doc/administration/pages/source.md @@ -61,7 +61,7 @@ Before proceeding with the Pages configuration, make sure that: Pages artifacts. 1. (Optional) You have a **wildcard certificate** for the Pages domain if you decide to serve Pages (`*.example.io`) under HTTPS. -1. (Optional but recommended) You have configured and enabled the [Shared Runners](../../ci/runners/README.md) +1. (Optional but recommended) You have configured and enabled the [shared runners](../../ci/runners/README.md) so that your users don't have to bring their own. ### DNS configuration @@ -347,6 +347,10 @@ world. Custom domains and TLS are supported. 1. Restart NGINX 1. [Restart GitLab](../restart_gitlab.md#installations-from-source) +## Enable redirects + +In GitLab Pages, you can [enable the redirects feature](../../user/project/pages/redirects.md#enable-or-disable-redirects) to configure rules to forward one URL to another using HTTP redirects. + ## NGINX caveats NOTE: **Note:** @@ -421,7 +425,7 @@ Pages access control is disabled by default. To enable it: auth-server=<URL of the GitLab instance> ``` -1. Users can now configure it in their [projects' settings](../../user/project/pages/introduction.md#gitlab-pages-access-control-core). +1. Users can now configure it in their [projects' settings](../../user/project/pages/introduction.md#gitlab-pages-access-control). ## Change storage path diff --git a/doc/administration/postgresql/external.md b/doc/administration/postgresql/external.md index e2cfb95ec48..632b68fb014 100644 --- a/doc/administration/postgresql/external.md +++ b/doc/administration/postgresql/external.md @@ -11,8 +11,7 @@ If you use a cloud-managed service, or provide your own PostgreSQL instance: 1. Set up PostgreSQL according to the [database requirements document](../../install/requirements.md#database). -1. Set up a `gitlab` username with a password of your choice. The `gitlab` user - needs privileges to create the `gitlabhq_production` database. +1. Set up a `gitlab` user with a password of your choice, create the `gitlabhq_production` database, and make the user an owner of the database. You can see an example of this setup in the [installation from source documentation](../../install/installation.md#6-database). 1. If you are using a cloud-managed service, you may need to grant additional roles to your `gitlab` user: - Amazon RDS requires the [`rds_superuser`](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.html#Appendix.PostgreSQL.CommonDBATasks.Roles) role. diff --git a/doc/administration/raketasks/doctor.md b/doc/administration/raketasks/doctor.md index 2c1b6928663..62d0af70706 100644 --- a/doc/administration/raketasks/doctor.md +++ b/doc/administration/raketasks/doctor.md @@ -33,7 +33,6 @@ bundle exec rake gitlab:doctor:secrets RAILS_ENV=production **Example output** -<!-- vale gitlab.SentenceSpacing = NO --> ```plaintext I, [2020-06-11T17:17:54.951815 #27148] INFO -- : Checking encrypted values in the database I, [2020-06-11T17:18:12.677708 #27148] INFO -- : - ApplicationSetting failures: 0 @@ -45,7 +44,6 @@ I, [2020-06-11T17:18:15.575533 #27148] INFO -- : - ScimOauthAccessToken failure I, [2020-06-11T17:18:15.575678 #27148] INFO -- : Total: 1 row(s) affected I, [2020-06-11T17:18:15.575711 #27148] INFO -- : Done! ``` -<!-- vale gitlab.SentenceSpacing = YES --> ### Verbose mode diff --git a/doc/administration/raketasks/geo.md b/doc/administration/raketasks/geo.md index 71e4f922348..885d19903ed 100644 --- a/doc/administration/raketasks/geo.md +++ b/doc/administration/raketasks/geo.md @@ -1,6 +1,6 @@ # Geo Rake Tasks **(PREMIUM ONLY)** -The following Rake tasks are for [Geo installations](../geo/replication/index.md). +The following Rake tasks are for [Geo installations](../geo/index.md). ## Git housekeeping diff --git a/doc/administration/raketasks/ldap.md b/doc/administration/raketasks/ldap.md index 30bb9828aa0..6d04a786d3a 100644 --- a/doc/administration/raketasks/ldap.md +++ b/doc/administration/raketasks/ldap.md @@ -32,13 +32,13 @@ rake gitlab:ldap:check[50] > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/14735) in [GitLab Starter](https://about.gitlab.com/pricing/) 12.2. -The following task will run a [group sync](../auth/ldap/index.md#group-sync-starter-only) immediately. This is valuable +The following task will run a [group sync](../auth/ldap/index.md#group-sync) immediately. This is valuable when you'd like to update all configured group memberships against LDAP without waiting for the next scheduled group sync to be run. NOTE: **Note:** If you'd like to change the frequency at which a group sync is performed, -[adjust the cron schedule](../auth/ldap/index.md#adjusting-ldap-group-sync-schedule-starter-only) +[adjust the cron schedule](../auth/ldap/index.md#adjusting-ldap-group-sync-schedule) instead. **Omnibus Installation** diff --git a/doc/administration/raketasks/storage.md b/doc/administration/raketasks/storage.md index a97bff83290..896eafeb5f3 100644 --- a/doc/administration/raketasks/storage.md +++ b/doc/administration/raketasks/storage.md @@ -104,13 +104,13 @@ You can monitor the progress in the **Admin Area > Monitoring > Background Jobs* There is a specific queue you can watch to see how long it will take to finish: `hashed_storage:hashed_storage_project_migrate`. -After it reaches zero, you can confirm every project has been migrated by running the commands below. +After it reaches zero, you can confirm every project has been migrated by running the commands above. If you find it necessary, you can run this migration script again to schedule missing projects. Any error or warning will be logged in Sidekiq's log file. NOTE: **Note:** -If [Geo](../geo/replication/index.md) is enabled, each project that is successfully migrated +If [Geo](../geo/index.md) is enabled, each project that is successfully migrated generates an event to replicate the changes on any **secondary** nodes. You only need the `gitlab:storage:migrate_to_hashed` Rake task to migrate your repositories, but we have additional @@ -153,7 +153,7 @@ sudo gitlab-rake gitlab:storage:rollback_to_legacy ID_FROM=50 ID_TO=100 You can monitor the progress in the **Admin Area > Monitoring > Background Jobs** page. On the **Queues** tab, you can watch the `hashed_storage:hashed_storage_project_rollback` queue to see how long the process will take to finish. -After it reaches zero, you can confirm every project has been rolled back by running the commands bellow. +After it reaches zero, you can confirm every project has been rolled back by running the commands above. If some projects weren't rolled back, you can run this rollback script again to schedule further rollbacks. Any error or warning will be logged in Sidekiq's log file. diff --git a/doc/administration/raketasks/uploads/migrate.md b/doc/administration/raketasks/uploads/migrate.md index 8c020e91a15..8d61beea597 100644 --- a/doc/administration/raketasks/uploads/migrate.md +++ b/doc/administration/raketasks/uploads/migrate.md @@ -1,10 +1,13 @@ # Uploads migrate Rake tasks **(CORE ONLY)** -`gitlab:uploads:migrate` migrates uploads between different storage types. +There is a Rake task for migrating uploads between different storage types. + +- Migrate all uploads with [`gitlab:uploads:migrate:all`](#all-in-one-rake-task) or +- To only migrate specific upload types, use [`gitlab:uploads:migrate`](#individual-rake-tasks). ## Migrate to object storage -After [configuring the object storage](../../uploads.md#using-object-storage-core-only) for GitLab's +After [configuring the object storage](../../uploads.md#using-object-storage) for GitLab's uploads, use this task to migrate existing uploads from the local storage to the remote storage. Read more about using [object storage with GitLab](../../object_storage.md). @@ -165,4 +168,4 @@ To migrate uploads from object storage to local storage: ``` After running the Rake task, you can disable object storage by undoing the changes described -in the instructions to [configure object storage](../../uploads.md#using-object-storage-core-only). +in the instructions to [configure object storage](../../uploads.md#using-object-storage). diff --git a/doc/administration/redis/replication_and_failover_external.md b/doc/administration/redis/replication_and_failover_external.md index d530e6a8fd7..ce452d30fc2 100644 --- a/doc/administration/redis/replication_and_failover_external.md +++ b/doc/administration/redis/replication_and_failover_external.md @@ -17,9 +17,8 @@ Omnibus GitLab package. The following are the requirements for providing your own Redis instance: -- Redis version 5.0 or higher is recommended, as this is what ships with - Omnibus GitLab packages starting with GitLab 12.7. -- GitLab 13.0 and later requires Redis version 4.0 or higher. +- Find the minimum Redis version that is required in the + [requirements page](../../install/requirements.md). - Standalone Redis or Redis high availability with Sentinel are supported. Redis Cluster is not supported. - Managed Redis from cloud providers such as AWS ElastiCache will work. If these diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index fe2dad41066..5f8ab6683a9 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -167,6 +167,14 @@ added to GitLab to configure SSL certificates. See [NGINX HTTPS documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) for details on managing SSL certificates and configuring NGINX. +### Readiness checks + +Ensure the external load balancer only routes to working services with built +in monitoring endpoints. The [readiness checks](../../user/admin_area/monitoring/health_check.md) +all require [additional configuration](../monitoring/ip_whitelist.md) +on the nodes being checked, otherwise, the external load balancer will not be able to +connect. + ### Ports The basic ports to be used are shown in the table below. @@ -325,6 +333,9 @@ If you use a cloud-managed service, or provide your own PostgreSQL: 1. Configure the GitLab application servers with the appropriate details. This step is covered in [Configuring the GitLab Rails application](#configure-gitlab-rails). +See [Configure GitLab using an external PostgreSQL service](../postgresql/external.md) for +further configuration steps. + ### Standalone PostgreSQL using Omnibus GitLab The following IPs will be used as an example: @@ -1435,7 +1446,7 @@ On each node: gitlab_workhorse['enable'] = false grafana['enable'] = false - # If you run a seperate monitoring node you can disable these services + # If you run a separate monitoring node you can disable these services alertmanager['enable'] = false prometheus['enable'] = false @@ -1990,15 +2001,15 @@ based on what features you intend to use: 1. Configure [object storage for job artifacts](../job_artifacts.md#using-object-storage) including [incremental logging](../job_logs.md#new-incremental-logging-architecture). 1. Configure [object storage for LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage). -1. Configure [object storage for uploads](../uploads.md#using-object-storage-core-only). +1. Configure [object storage for uploads](../uploads.md#using-object-storage). 1. Configure [object storage for merge request diffs](../merge_request_diffs.md#using-object-storage). 1. Configure [object storage for Container Registry](../packages/container_registry.md#use-object-storage) (optional feature). 1. Configure [object storage for Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage) (optional feature). 1. Configure [object storage for packages](../packages/index.md#using-object-storage) (optional feature). **(PREMIUM ONLY)** 1. Configure [object storage for Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature). **(PREMIUM ONLY)** 1. Configure [object storage for Pseudonymizer](../pseudonymizer.md#configuration) (optional feature). **(ULTIMATE ONLY)** -1. Configure [object storage for autoscale Runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional - for improved performance). -1. Configure [object storage for Terraform state files](../terraform_state.md#using-object-storage-core-only). +1. Configure [object storage for autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional - for improved performance). +1. Configure [object storage for Terraform state files](../terraform_state.md#using-object-storage). Using separate buckets for each data type is the recommended approach for GitLab. diff --git a/doc/administration/reference_architectures/1k_users.md b/doc/administration/reference_architectures/1k_users.md index d3cf5f49413..d376c1b7575 100644 --- a/doc/administration/reference_architectures/1k_users.md +++ b/doc/administration/reference_architectures/1k_users.md @@ -12,7 +12,7 @@ full list of reference architectures, see If you need to serve up to 1,000 users and you don't have strict availability requirements, a single-node solution with -[frequent backups](index.md#automated-backups-core-only) is appropriate for +[frequent backups](index.md#automated-backups) is appropriate for many organizations . > - **Supported users (approximate):** 1,000 diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 1cfa2565893..2ef555bff29 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -167,6 +167,14 @@ added to GitLab to configure SSL certificates. See [NGINX HTTPS documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) for details on managing SSL certificates and configuring NGINX. +### Readiness checks + +Ensure the external load balancer only routes to working services with built +in monitoring endpoints. The [readiness checks](../../user/admin_area/monitoring/health_check.md) +all require [additional configuration](../monitoring/ip_whitelist.md) +on the nodes being checked, otherwise, the external load balancer will not be able to +connect. + ### Ports The basic ports to be used are shown in the table below. @@ -325,6 +333,9 @@ If you use a cloud-managed service, or provide your own PostgreSQL: 1. Configure the GitLab application servers with the appropriate details. This step is covered in [Configuring the GitLab Rails application](#configure-gitlab-rails). +See [Configure GitLab using an external PostgreSQL service](../postgresql/external.md) for +further configuration steps. + ### Standalone PostgreSQL using Omnibus GitLab The following IPs will be used as an example: @@ -1435,7 +1446,7 @@ On each node: gitlab_workhorse['enable'] = false grafana['enable'] = false - # If you run a seperate monitoring node you can disable these services + # If you run a separate monitoring node you can disable these services alertmanager['enable'] = false prometheus['enable'] = false @@ -1990,15 +2001,15 @@ based on what features you intend to use: 1. Configure [object storage for job artifacts](../job_artifacts.md#using-object-storage) including [incremental logging](../job_logs.md#new-incremental-logging-architecture). 1. Configure [object storage for LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage). -1. Configure [object storage for uploads](../uploads.md#using-object-storage-core-only). +1. Configure [object storage for uploads](../uploads.md#using-object-storage). 1. Configure [object storage for merge request diffs](../merge_request_diffs.md#using-object-storage). 1. Configure [object storage for Container Registry](../packages/container_registry.md#use-object-storage) (optional feature). 1. Configure [object storage for Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage) (optional feature). 1. Configure [object storage for packages](../packages/index.md#using-object-storage) (optional feature). **(PREMIUM ONLY)** 1. Configure [object storage for Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature). **(PREMIUM ONLY)** 1. Configure [object storage for Pseudonymizer](../pseudonymizer.md#configuration) (optional feature). **(ULTIMATE ONLY)** -1. Configure [object storage for autoscale Runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional - for improved performance). -1. Configure [object storage for Terraform state files](../terraform_state.md#using-object-storage-core-only). +1. Configure [object storage for autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional - for improved performance). +1. Configure [object storage for Terraform state files](../terraform_state.md#using-object-storage). Using separate buckets for each data type is the recommended approach for GitLab. diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index a7feb78a365..34b90964fbf 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -42,7 +42,7 @@ doesn't require you to provision and maintain a node. To set up GitLab and its components to accommodate up to 2,000 users: -1. [Configure the external load balancing node](#configure-the-load-balancer) +1. [Configure the external load balancing node](#configure-the-external-load-balancer) to handle the load balancing of the two GitLab application services nodes. 1. [Configure PostgreSQL](#configure-postgresql), the database for GitLab. 1. [Configure Redis](#configure-redis). @@ -60,7 +60,7 @@ To set up GitLab and its components to accommodate up to 2,000 users: storage. You can skip this step if you're not using GitLab Pages (which requires NFS). -## Configure the load balancer +## Configure the external load balancer NOTE: **Note:** This architecture has been tested and validated with [HAProxy](https://www.haproxy.org/). @@ -115,6 +115,14 @@ need to add a configuration to GitLab to configure SSL certificates. For details about managing SSL certificates and configuring NGINX, see the [NGINX HTTPS documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https). +### Readiness checks + +Ensure the external load balancer only routes to working services with built +in monitoring endpoints. The [readiness checks](../../user/admin_area/monitoring/health_check.md) +all require [additional configuration](../monitoring/ip_whitelist.md) +on the nodes being checked, otherwise, the external load balancer will not be able to +connect. + ### Ports The basic load balancer ports you should use are described in the following @@ -193,6 +201,9 @@ If you use a cloud-managed service, or provide your own PostgreSQL: 1. Configure the GitLab application servers with the appropriate details. This step is covered in [Configuring the GitLab Rails application](#configure-gitlab-rails). +See [Configure GitLab using an external PostgreSQL service](../postgresql/external.md) for +further configuration steps. + ### Standalone PostgreSQL using Omnibus GitLab 1. SSH into the PostgreSQL server. @@ -345,50 +356,51 @@ are supported and can be added if needed. ## Configure Gitaly -Deploying Gitaly in its own server can benefit GitLab installations that are -larger than a single machine. Gitaly node requirements are dependent on data, -specifically the number of projects and their sizes. It's recommended that each -Gitaly node store no more than 5TB of data. Your 2K setup may require one or more -nodes depending on your repository storage requirements. - -We strongly recommend that all Gitaly nodes should be set up with SSD disks with a throughput of at least -8,000 IOPS for read operations and 2,000 IOPS for write, as Gitaly has heavy I/O. -These IOPS values are recommended only as a starter as with time they may be -adjusted higher or lower depending on the scale of your environment's workload. -If you're running the environment on a Cloud provider -you may need to refer to their documentation on how configure IOPS correctly. - -Some things to note: - -- The GitLab Rails application shards repositories into [repository storages](../repository_storage_paths.md). -- A Gitaly server can host one or more storages. -- A GitLab server can use one or more Gitaly servers. -- Gitaly addresses must be specified in such a way that they resolve - correctly for ALL Gitaly clients. +[Gitaly](../gitaly/index.md) server node requirements are dependent on data, +specifically the number of projects and those projects' sizes. It's recommended +that a Gitaly server node stores no more than 5TB of data. Although this +reference architecture includes a single Gitaly server node, you may require +additional nodes depending on your repository storage requirements. + +Due to Gitaly having notable input and output requirements, we strongly +recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs +should have a throughput of at least 8,000 +input/output operations per second (IOPS) for read operations and 2,000 IOPS +for write operations. These IOPS values are initial recommendations, and may be +adjusted to greater or lesser values depending on the scale of your +environment's workload. If you're running the environment on a Cloud provider, +refer to their documentation about how to configure IOPS correctly. + +Be sure to note the following items: + +- The GitLab Rails application shards repositories into + [repository storage paths](../repository_storage_paths.md). +- A Gitaly server can host one or more storage paths. +- A GitLab server can use one or more Gitaly server nodes. +- Gitaly addresses must be specified to be correctly resolvable for *all* + Gitaly clients. - Gitaly servers must not be exposed to the public internet, as Gitaly's network traffic is unencrypted by default. The use of a firewall is highly recommended to restrict access to the Gitaly server. Another option is to [use TLS](#gitaly-tls-support). -TIP: **Tip:** -For more information about Gitaly's history and network architecture see the -[standalone Gitaly documentation](../gitaly/index.md). - -Note: **Note:** The token referred to throughout the Gitaly documentation is -just an arbitrary password selected by the administrator. It is unrelated to -tokens created for the GitLab API or other similar web API tokens. +NOTE: **Note:** +The token referred to throughout the Gitaly documentation is an arbitrary +password selected by the administrator. This token is unrelated to tokens +created for the GitLab API or other similar web API tokens. -Below we describe how to configure one Gitaly server `gitaly1.internal` with -secret token `gitalysecret`. We assume your GitLab installation has two -repository storages: `default` and `storage1`. +The following procedure describes how to configure a single Gitaly server named +`gitaly1.internal` with the secret token `gitalysecret`. We assume your GitLab +installation has two repository storages: `default` and `storage1`. To configure the Gitaly server: -1. [Download/Install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page but - **without** providing the `EXTERNAL_URL` value. -1. Edit `/etc/gitlab/gitlab.rb` to configure storage paths, enable - the network listener and configure the token: +1. On the server node you want to use for Gitaly, + [download and install](https://about.gitlab.com/install/) your selected + Omnibus GitLab package using *steps 1 and 2* from the GitLab downloads page, + but *without* providing the `EXTERNAL_URL` value. +1. Edit the Gitaly server node's `/etc/gitlab/gitlab.rb` file to configure + storage paths, enable the network listener, and to configure the token: <!-- updates to following example must also be made at @@ -402,7 +414,7 @@ To configure the Gitaly server: # to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API. # The following two values must be the same as their respective values # of the GitLab Rails application setup - gitaly['auth_token'] = 'gitlaysecret' + gitaly['auth_token'] = 'gitalysecret' gitlab_shell['secret_token'] = 'shellsecret' # Avoid running unnecessary services on the Gitaly server @@ -415,7 +427,7 @@ To configure the Gitaly server: gitlab_workhorse['enable'] = false grafana['enable'] = false - # If you run a seperate monitoring node you can disable these services + # If you run a separate monitoring node you can disable these services alertmanager['enable'] = false prometheus['enable'] = false @@ -437,11 +449,7 @@ To configure the Gitaly server: # Set the network addresses that the exporters used for monitoring will listen on node_exporter['listen_address'] = '0.0.0.0:9100' - ``` - -1. Append the following to `/etc/gitlab/gitlab.rb` on `gitaly1.internal`: - ```ruby git_data_dirs({ 'default' => { 'path' => '/var/opt/gitlab/git-data' @@ -452,12 +460,7 @@ To configure the Gitaly server: }) ``` - <!-- - updates to following example must also be made at - https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab - --> - -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). 1. Confirm that Gitaly can perform callbacks to the internal API: ```shell @@ -573,7 +576,7 @@ On each node perform the following: 1. Create/edit `/etc/gitlab/gitlab.rb` and use the following configuration. To maintain uniformity of links across nodes, the `external_url` on the application server should point to the external URL that users will use - to access GitLab. This would be the URL of the [load balancer](#configure-the-load-balancer) + to access GitLab. This would be the URL of the [load balancer](#configure-the-external-load-balancer) which will route traffic to the GitLab application server: ```ruby @@ -583,7 +586,7 @@ On each node perform the following: # to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API. # The following two values must be the same as their respective values # of the Gitaly setup - gitlab_rails['gitaly_token'] = 'gitalyecret' + gitlab_rails['gitaly_token'] = 'gitalysecret' gitlab_shell['secret_token'] = 'shellsecret' git_data_dirs({ @@ -818,15 +821,15 @@ on the features you intend to use: 1. [Object storage for job artifacts](../job_artifacts.md#using-object-storage) including [incremental logging](../job_logs.md#new-incremental-logging-architecture). 1. [Object storage for LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage). -1. [Object storage for uploads](../uploads.md#using-object-storage-core-only). +1. [Object storage for uploads](../uploads.md#using-object-storage). 1. [Object storage for merge request diffs](../merge_request_diffs.md#using-object-storage). 1. [Object storage for Container Registry](../packages/container_registry.md#use-object-storage) (optional feature). 1. [Object storage for Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage) (optional feature). 1. [Object storage for packages](../packages/index.md#using-object-storage) (optional feature). **(PREMIUM ONLY)** 1. [Object storage for Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature). **(PREMIUM ONLY)** 1. [Object storage for Pseudonymizer](../pseudonymizer.md#configuration) (optional feature). **(ULTIMATE ONLY)** -1. [Object storage for autoscale Runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional, for improved performance). -1. [Object storage for Terraform state files](../terraform_state.md#using-object-storage-core-only). +1. [Object storage for autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional, for improved performance). +1. [Object storage for Terraform state files](../terraform_state.md#using-object-storage). Using separate buckets for each data type is the recommended approach for GitLab. diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index 2f88413de6f..be944586e43 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -162,6 +162,14 @@ added to GitLab to configure SSL certificates. See [NGINX HTTPS documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) for details on managing SSL certificates and configuring NGINX. +### Readiness checks + +Ensure the external load balancer only routes to working services with built +in monitoring endpoints. The [readiness checks](../../user/admin_area/monitoring/health_check.md) +all require [additional configuration](../monitoring/ip_whitelist.md) +on the nodes being checked, otherwise, the external load balancer will not be able to +connect. + ### Ports The basic ports to be used are shown in the table below. @@ -600,6 +608,9 @@ If you use a cloud-managed service, or provide your own PostgreSQL: 1. Configure the GitLab application servers with the appropriate details. This step is covered in [Configuring the GitLab Rails application](#configure-gitlab-rails). +See [Configure GitLab using an external PostgreSQL service](../postgresql/external.md) for +further configuration steps. + ### Standalone PostgreSQL using Omnibus GitLab The following IPs will be used as an example: @@ -1128,7 +1139,7 @@ On each node: # to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API. # The following two values must be the same as their respective values # of the GitLab Rails application setup - gitaly['auth_token'] = 'gitlaysecret' + gitaly['auth_token'] = 'gitalysecret' gitlab_shell['secret_token'] = 'shellsecret' # Avoid running unnecessary services on the Gitaly server @@ -1142,7 +1153,7 @@ On each node: grafana['enable'] = false gitlab_exporter['enable'] = false - # If you run a seperate monitoring node you can disable these services + # If you run a separate monitoring node you can disable these services alertmanager['enable'] = false prometheus['enable'] = false @@ -1471,7 +1482,7 @@ On each node perform the following: # to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API. # The following two values must be the same as their respective values # of the Gitaly setup - gitlab_rails['gitaly_token'] = 'gitalyecret' + gitlab_rails['gitaly_token'] = 'gitalysecret' gitlab_shell['secret_token'] = 'shellsecret' git_data_dirs({ @@ -1716,15 +1727,15 @@ based on what features you intend to use: 1. Configure [object storage for job artifacts](../job_artifacts.md#using-object-storage) including [incremental logging](../job_logs.md#new-incremental-logging-architecture). 1. Configure [object storage for LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage). -1. Configure [object storage for uploads](../uploads.md#using-object-storage-core-only). +1. Configure [object storage for uploads](../uploads.md#using-object-storage). 1. Configure [object storage for merge request diffs](../merge_request_diffs.md#using-object-storage). 1. Configure [object storage for Container Registry](../packages/container_registry.md#use-object-storage) (optional feature). 1. Configure [object storage for Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage) (optional feature). 1. Configure [object storage for packages](../packages/index.md#using-object-storage) (optional feature). **(PREMIUM ONLY)** 1. Configure [object storage for Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature). **(PREMIUM ONLY)** 1. Configure [object storage for Pseudonymizer](../pseudonymizer.md#configuration) (optional feature). **(ULTIMATE ONLY)** -1. Configure [object storage for autoscale Runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional - for improved performance). -1. Configure [object storage for Terraform state files](../terraform_state.md#using-object-storage-core-only). +1. Configure [object storage for autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional - for improved performance). +1. Configure [object storage for Terraform state files](../terraform_state.md#using-object-storage). Using separate buckets for each data type is the recommended approach for GitLab. diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 565845b4bf5..e812eed0227 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -167,6 +167,14 @@ added to GitLab to configure SSL certificates. See [NGINX HTTPS documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) for details on managing SSL certificates and configuring NGINX. +### Readiness checks + +Ensure the external load balancer only routes to working services with built +in monitoring endpoints. The [readiness checks](../../user/admin_area/monitoring/health_check.md) +all require [additional configuration](../monitoring/ip_whitelist.md) +on the nodes being checked, otherwise, the external load balancer will not be able to +connect. + ### Ports The basic ports to be used are shown in the table below. @@ -325,6 +333,9 @@ If you use a cloud-managed service, or provide your own PostgreSQL: 1. Configure the GitLab application servers with the appropriate details. This step is covered in [Configuring the GitLab Rails application](#configure-gitlab-rails). +See [Configure GitLab using an external PostgreSQL service](../postgresql/external.md) for +further configuration steps. + ### Standalone PostgreSQL using Omnibus GitLab The following IPs will be used as an example: @@ -1435,7 +1446,7 @@ On each node: gitlab_workhorse['enable'] = false grafana['enable'] = false - # If you run a seperate monitoring node you can disable these services + # If you run a separate monitoring node you can disable these services alertmanager['enable'] = false prometheus['enable'] = false @@ -1990,15 +2001,15 @@ based on what features you intend to use: 1. Configure [object storage for job artifacts](../job_artifacts.md#using-object-storage) including [incremental logging](../job_logs.md#new-incremental-logging-architecture). 1. Configure [object storage for LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage). -1. Configure [object storage for uploads](../uploads.md#using-object-storage-core-only). +1. Configure [object storage for uploads](../uploads.md#using-object-storage). 1. Configure [object storage for merge request diffs](../merge_request_diffs.md#using-object-storage). 1. Configure [object storage for Container Registry](../packages/container_registry.md#use-object-storage) (optional feature). 1. Configure [object storage for Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage) (optional feature). 1. Configure [object storage for packages](../packages/index.md#using-object-storage) (optional feature). **(PREMIUM ONLY)** 1. Configure [object storage for Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature). **(PREMIUM ONLY)** 1. Configure [object storage for Pseudonymizer](../pseudonymizer.md#configuration) (optional feature). **(ULTIMATE ONLY)** -1. Configure [object storage for autoscale Runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional - for improved performance). -1. Configure [object storage for Terraform state files](../terraform_state.md#using-object-storage-core-only). +1. Configure [object storage for autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional - for improved performance). +1. Configure [object storage for Terraform state files](../terraform_state.md#using-object-storage). Using separate buckets for each data type is the recommended approach for GitLab. diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 14685ffa53d..6dfa588b092 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -162,6 +162,14 @@ added to GitLab to configure SSL certificates. See [NGINX HTTPS documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) for details on managing SSL certificates and configuring NGINX. +### Readiness checks + +Ensure the external load balancer only routes to working services with built +in monitoring endpoints. The [readiness checks](../../user/admin_area/monitoring/health_check.md) +all require [additional configuration](../monitoring/ip_whitelist.md) +on the nodes being checked, otherwise, the external load balancer will not be able to +connect. + ### Ports The basic ports to be used are shown in the table below. @@ -600,6 +608,9 @@ If you use a cloud-managed service, or provide your own PostgreSQL: 1. Configure the GitLab application servers with the appropriate details. This step is covered in [Configuring the GitLab Rails application](#configure-gitlab-rails). +See [Configure GitLab using an external PostgreSQL service](../postgresql/external.md) for +further configuration steps. + ### Standalone PostgreSQL using Omnibus GitLab The following IPs will be used as an example: @@ -1127,7 +1138,7 @@ On each node: # to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API. # The following two values must be the same as their respective values # of the GitLab Rails application setup - gitaly['auth_token'] = 'gitlaysecret' + gitaly['auth_token'] = 'gitalysecret' gitlab_shell['secret_token'] = 'shellsecret' # Avoid running unnecessary services on the Gitaly server @@ -1141,7 +1152,7 @@ On each node: grafana['enable'] = false gitlab_exporter['enable'] = false - # If you run a seperate monitoring node you can disable these services + # If you run a separate monitoring node you can disable these services alertmanager['enable'] = false prometheus['enable'] = false @@ -1470,7 +1481,7 @@ On each node perform the following: # to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API. # The following two values must be the same as their respective values # of the Gitaly setup - gitlab_rails['gitaly_token'] = 'gitalyecret' + gitlab_rails['gitaly_token'] = 'gitalysecret' gitlab_shell['secret_token'] = 'shellsecret' git_data_dirs({ @@ -1715,15 +1726,15 @@ based on what features you intend to use: 1. Configure [object storage for job artifacts](../job_artifacts.md#using-object-storage) including [incremental logging](../job_logs.md#new-incremental-logging-architecture). 1. Configure [object storage for LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage). -1. Configure [object storage for uploads](../uploads.md#using-object-storage-core-only). +1. Configure [object storage for uploads](../uploads.md#using-object-storage). 1. Configure [object storage for merge request diffs](../merge_request_diffs.md#using-object-storage). 1. Configure [object storage for Container Registry](../packages/container_registry.md#use-object-storage) (optional feature). 1. Configure [object storage for Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage) (optional feature). 1. Configure [object storage for packages](../packages/index.md#using-object-storage) (optional feature). **(PREMIUM ONLY)** 1. Configure [object storage for Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature). **(PREMIUM ONLY)** 1. Configure [object storage for Pseudonymizer](../pseudonymizer.md#configuration) (optional feature). **(ULTIMATE ONLY)** -1. Configure [object storage for autoscale Runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional - for improved performance). -1. Configure [object storage for Terraform state files](../terraform_state.md#using-object-storage-core-only). +1. Configure [object storage for autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional - for improved performance). +1. Configure [object storage for Terraform state files](../terraform_state.md#using-object-storage). Using separate buckets for each data type is the recommended approach for GitLab. diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index 4f7be2413dd..3964b8daeb7 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -32,7 +32,7 @@ per 1,000 users: - Git: 2 RPS For GitLab instances with less than 2,000 users, it's recommended that you use -the [default setup](#automated-backups-core-only) by +the [default setup](#automated-backups) by [installing GitLab](../../install/README.md) on a single machine to minimize maintenance and resource costs. @@ -73,11 +73,11 @@ The following reference architectures are available: GitLab comes with the following components for your use, listed from least to most complex: -- [Automated backups](#automated-backups-core-only) -- [Traffic load balancer](#traffic-load-balancer-starter-only) -- [Zero downtime updates](#zero-downtime-updates-starter-only) -- [Automated database failover](#automated-database-failover-premium-only) -- [Instance level replication with GitLab Geo](#instance-level-replication-with-gitlab-geo-premium-only) +- [Automated backups](#automated-backups) +- [Traffic load balancer](#traffic-load-balancer) +- [Zero downtime updates](#zero-downtime-updates) +- [Automated database failover](#automated-database-failover) +- [Instance level replication with GitLab Geo](#instance-level-replication-with-gitlab-geo) As you implement these components, begin with a single server and then do backups. Only after completing the first server should you proceed to the next. @@ -147,7 +147,7 @@ is recommended. > - Required domain knowledge: Storage replication > - Supported tiers: [GitLab Premium and Ultimate](https://about.gitlab.com/pricing/) -[GitLab Geo](../geo/replication/index.md) allows you to replicate your GitLab +[GitLab Geo](../geo/index.md) allows you to replicate your GitLab instance to other geographical locations as a read-only fully operational instance that can also be promoted in case of disaster. diff --git a/doc/administration/server_hooks.md b/doc/administration/server_hooks.md index ab808fc28d8..54b2cd43265 100644 --- a/doc/administration/server_hooks.md +++ b/doc/administration/server_hooks.md @@ -37,7 +37,7 @@ Note the following about server hooks: - [GitLab CI/CD](../ci/README.md). - [Push Rules](../push_rules/push_rules.md), for a user-configurable Git hook interface. **(STARTER)** -- Server hooks aren't replicated to [Geo](geo/replication/index.md) secondary nodes. +- Server hooks aren't replicated to [Geo](geo/index.md) secondary nodes. ## Create a server hook for a repository @@ -158,18 +158,18 @@ them as they can change. ## Transition to Go -> Introduced in GitLab 13.2 using feature flags. +> - Introduced in GitLab 13.2 using feature flags. +> - In GitLab 13.4, `update` Ruby [implementation removed](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/2501). +> - In GitLab 13.4, `post-receive` Go implementation [made default](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/2502). The following server hooks have been re-implemented in Go: - `pre-receive`, with the Go implementation used by default. To use the Ruby implementation instead, [disable](feature_flags.md#enable-or-disable-the-feature) the `:gitaly_go_preceive_hook` feature flag. -- `update`, with the Go implementation used by default. To use the Ruby implementation instead, - [disable](feature_flags.md#enable-or-disable-the-feature) the `:gitaly_go_update_hook` feature - flag. -- `post-receive`, however the Ruby implementation is used by default. To use the Go implementation - instead, [enable](feature_flags.md#enable-or-disable-the-feature) the +- `update`, with Go implementation always used. No Ruby implementation is available. +- `post-receive`, with the Go implementation used by default. To use the Ruby implementation + instead, [disable](feature_flags.md#enable-or-disable-the-feature) the `:gitaly_go_postreceive_hook` feature flag. ## Custom error messages diff --git a/doc/administration/troubleshooting/debug.md b/doc/administration/troubleshooting/debug.md index 5daf34c1011..a1b4df9b94e 100644 --- a/doc/administration/troubleshooting/debug.md +++ b/doc/administration/troubleshooting/debug.md @@ -64,10 +64,10 @@ easy to copy and save for future reference, you can run: puts Readline::HISTORY.to_a ``` -## Using the Rails Runner +## Using the Rails runner If you need to run some Ruby code in the context of your GitLab production -environment, you can do so using the [Rails Runner](https://guides.rubyonrails.org/command_line.html#rails-runner). When executing a script file, the script must be accessible by the `git` user. +environment, you can do so using the [Rails runner](https://guides.rubyonrails.org/command_line.html#rails-runner). When executing a script file, the script must be accessible by the `git` user. **For Omnibus installations** diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 22d699b424b..9a23a115765 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -43,10 +43,13 @@ instance_of_object.method(:foo).source_location project.method(:private?).source_location ``` -## Query an object +## Query the database using an ActiveRecord Model ```ruby -o = Object.where('attribute like ?', 'ex') +m = Model.where('attribute like ?', 'ex%') + +# for example to query the projects +projects = Project.where('path like ?', 'Oumua%') ``` ## View all keys in cache @@ -215,6 +218,17 @@ namespace = Namespace.find_by_full_path("") ::Projects::TransferService.new(p, current_user).execute(namespace) ``` +### For Removing webhooks that is getting timeout due to large webhook logs + +```ruby +# ID will be the webhook_id +WebHookLog.where(web_hook_id: ID).each_slice(ID) do |slice| + slice.each(&:destroy) +end + +WebHook.find(ID).destroy +``` + ### Bulk update service integration password for _all_ projects For example, change the Jira user's password for all projects that have the Jira @@ -242,6 +256,18 @@ p.each do |project| end ``` +### Incorrect repository statistics shown in the GUI + +After [reducing a repository size with third-party tools](../../user/project/repository/reducing_the_repo_size_using_git.md) +the displayed size may still show old sizes or commit numbers. To force an update, do: + +```ruby +p = Project.find_by_full_path('<namespace>/<project>') +pp p.statistics +p.statistics.refresh! +pp p.statistics # compare with earlier values +``` + ## Wikis ### Recreate diff --git a/doc/administration/troubleshooting/linux_cheat_sheet.md b/doc/administration/troubleshooting/linux_cheat_sheet.md index 06c49d67f40..f24234e1aff 100644 --- a/doc/administration/troubleshooting/linux_cheat_sheet.md +++ b/doc/administration/troubleshooting/linux_cheat_sheet.md @@ -179,11 +179,13 @@ strace -tt -T -f -y -yy -s 1024 -p <pid> ps auwx | grep unicorn | awk '{ print " -p " $2}' | xargs strace -tt -T -f -y -yy -s 1024 -o /tmp/unicorn.txt ``` -See the [strace zine](https://wizardzines.com/zines/strace/) for a quick walkthrough. +Be aware that strace can have major impacts to system performance when it is running. -Brendan Gregg has a more detailed explanation of [how to use strace](http://www.brendangregg.com/blog/2014-05-11/strace-wow-much-syscall.html). +#### Strace Resources -Be aware that strace can have major impacts to system performance when it is running. +- See the [strace zine](https://wizardzines.com/zines/strace/) for a quick walkthrough. +- Brendan Gregg has a more detailed explanation of [how to use strace](http://www.brendangregg.com/blog/2014-05-11/strace-wow-much-syscall.html). +- We have a [series of GitLab Unfiltered videos](https://www.youtube.com/playlist?list=PL05JrBw4t0KoC7cIkoAFcRhr4gsVesekg) on using strace to understand GitLab. ### The Strace Parser tool diff --git a/doc/administration/troubleshooting/postgresql.md b/doc/administration/troubleshooting/postgresql.md index b7e33e4501d..91ff6f6524a 100644 --- a/doc/administration/troubleshooting/postgresql.md +++ b/doc/administration/troubleshooting/postgresql.md @@ -34,7 +34,7 @@ This section is for links to information elsewhere in the GitLab documentation. - [More about external PostgreSQL](../postgresql/external.md) -- [Running Geo with external PostgreSQL](../geo/replication/external_database.md) +- [Running Geo with external PostgreSQL](../geo/setup/external_database.md) - [Upgrades when running PostgreSQL configured for HA.](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-gitlab-ha-cluster) diff --git a/doc/administration/troubleshooting/sidekiq.md b/doc/administration/troubleshooting/sidekiq.md index 9125ddf545f..404e806c5d9 100644 --- a/doc/administration/troubleshooting/sidekiq.md +++ b/doc/administration/troubleshooting/sidekiq.md @@ -212,12 +212,12 @@ the query details. ## Managing Sidekiq queues It is possible to use [Sidekiq API](https://github.com/mperham/sidekiq/wiki/API) -to perform a number of troubleshooting on Sidekiq. +to perform a number of troubleshooting steps on Sidekiq. These are the administrative commands and it should only be used if currently admin interface is not suitable due to scale of installation. -All this commands should be run using `gitlab-rails console`. +All these commands should be run using `gitlab-rails console`. ### View the queue size diff --git a/doc/administration/troubleshooting/tracing_correlation_id.md b/doc/administration/troubleshooting/tracing_correlation_id.md index 31f537beae5..03c342595a3 100644 --- a/doc/administration/troubleshooting/tracing_correlation_id.md +++ b/doc/administration/troubleshooting/tracing_correlation_id.md @@ -21,7 +21,7 @@ You can find your correlation ID by searching in either place. You can use your browser's developer tools to monitor and inspect network activity with the site that you're visiting. See the links below for network monitoring -documenation for some popular browsers. +documentation for some popular browsers. - [Network Monitor - Firefox Developer Tools](https://developer.mozilla.org/en-US/docs/Tools/Network_Monitor) - [Inspect Network Activity In Chrome DevTools](https://developers.google.com/web/tools/chrome-devtools/network/) @@ -38,7 +38,7 @@ value that was randomly generated by GitLab for the request. See the following example: - + ### Getting the correlation ID from your logs diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index d9902208e93..71a41719003 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -1,4 +1,4 @@ -# Uploads administration +# Uploads administration **(CORE ONLY)** Uploads represent all user data that may be sent to GitLab as a single file. As an example, avatars and notes' attachments are uploads. Uploads are integral to GitLab functionality, and therefore cannot be disabled. @@ -108,7 +108,7 @@ _The uploads are stored by default in ``` 1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate` Rake task](raketasks/uploads/migrate.md). +1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate:all` Rake task](raketasks/uploads/migrate.md). **In installations from source:** @@ -131,7 +131,7 @@ _The uploads are stored by default in ``` 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. -1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate` Rake task](raketasks/uploads/migrate.md). +1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate:all` Rake task](raketasks/uploads/migrate.md). ### OpenStack example @@ -157,7 +157,7 @@ _The uploads are stored by default in ``` 1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate` Rake task](raketasks/uploads/migrate.md). +1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate:all` Rake task](raketasks/uploads/migrate.md). --- @@ -188,4 +188,4 @@ _The uploads are stored by default in ``` 1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate` Rake task](raketasks/uploads/migrate.md). +1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate:all` Rake task](raketasks/uploads/migrate.md). |
