Welcome to mirror list, hosted at ThFree Co, Russian Federation.

gitlab.com/gitlab-org/gitlab-foss.git - Unnamed repository; edit this file 'description' to name the repository.
summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorGitLab Bot <gitlab-bot@gitlab.com>2022-12-20 17:22:11 +0300
committerGitLab Bot <gitlab-bot@gitlab.com>2022-12-20 17:22:11 +0300
commit0c872e02b2c822e3397515ec324051ff540f0cd5 (patch)
treece2fb6ce7030e4dad0f4118d21ab6453e5938cdd /doc/administration
parentf7e05a6853b12f02911494c4b3fe53d9540d74fc (diff)
Add latest changes from gitlab-org/gitlab@15-7-stable-eev15.7.0-rc42
Diffstat (limited to 'doc/administration')
-rw-r--r--doc/administration/application_settings_cache.md4
-rw-r--r--doc/administration/audit_event_streaming.md59
-rw-r--r--doc/administration/audit_events.md481
-rw-r--r--doc/administration/auth/ldap/index.md79
-rw-r--r--doc/administration/auth/ldap/ldap-troubleshooting.md60
-rw-r--r--doc/administration/auth/ldap/ldap_synchronization.md410
-rw-r--r--doc/administration/clusters/kas.md2
-rw-r--r--doc/administration/configure.md8
-rw-r--r--doc/administration/external_pipeline_validation.md3
-rw-r--r--doc/administration/feature_flags.md3
-rw-r--r--doc/administration/file_hooks.md4
-rw-r--r--doc/administration/geo/disaster_recovery/index.md6
-rw-r--r--doc/administration/geo/disaster_recovery/planned_failover.md2
-rw-r--r--doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md10
-rw-r--r--doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md2
-rw-r--r--doc/administration/geo/replication/configuration.md2
-rw-r--r--doc/administration/geo/replication/datatypes.md2
-rw-r--r--doc/administration/geo/replication/geo_validation_tests.md8
-rw-r--r--doc/administration/geo/replication/location_aware_git_url.md2
-rw-r--r--doc/administration/geo/replication/troubleshooting.md26
-rw-r--r--doc/administration/geo/replication/tuning.md2
-rw-r--r--doc/administration/geo/replication/version_specific_upgrades.md6
-rw-r--r--doc/administration/geo/secondary_proxy/index.md87
-rw-r--r--doc/administration/geo/setup/database.md152
-rw-r--r--doc/administration/gitaly/configure_gitaly.md62
-rw-r--r--doc/administration/gitaly/index.md56
-rw-r--r--doc/administration/gitaly/monitoring.md2
-rw-r--r--doc/administration/gitaly/praefect.md15
-rw-r--r--doc/administration/gitaly/reference.md6
-rw-r--r--doc/administration/gitaly/troubleshooting.md38
-rw-r--r--doc/administration/housekeeping.md27
-rw-r--r--doc/administration/img/audit_events_v14_5.pngbin33285 -> 0 bytes
-rw-r--r--doc/administration/img/impersonated_audit_events_v13_8.pngbin11908 -> 0 bytes
-rw-r--r--doc/administration/img/impersonated_audit_events_v15_7.pngbin0 -> 11909 bytes
-rw-r--r--doc/administration/inactive_project_deletion.md2
-rw-r--r--doc/administration/incoming_email.md2
-rw-r--r--doc/administration/index.md5
-rw-r--r--doc/administration/instance_limits.md10
-rw-r--r--doc/administration/integration/terminal.md6
-rw-r--r--doc/administration/issue_closing_pattern.md91
-rw-r--r--doc/administration/job_artifacts.md23
-rw-r--r--doc/administration/job_logs.md4
-rw-r--r--doc/administration/lfs/index.md86
-rw-r--r--doc/administration/logs/index.md67
-rw-r--r--doc/administration/maintenance_mode/index.md4
-rw-r--r--doc/administration/merge_request_diffs.md3
-rw-r--r--doc/administration/monitoring/gitlab_self_monitoring_project/index.md4
-rw-r--r--doc/administration/monitoring/performance/performance_bar.md4
-rw-r--r--doc/administration/monitoring/prometheus/gitlab_exporter.md4
-rw-r--r--doc/administration/monitoring/prometheus/gitlab_metrics.md13
-rw-r--r--doc/administration/nfs.md49
-rw-r--r--doc/administration/object_storage.md20
-rw-r--r--doc/administration/operations/extra_sidekiq_processes.md11
-rw-r--r--doc/administration/operations/extra_sidekiq_routing.md11
-rw-r--r--doc/administration/operations/filesystem_benchmarking.md6
-rw-r--r--doc/administration/operations/index.md5
-rw-r--r--doc/administration/operations/puma.md10
-rw-r--r--doc/administration/operations/sidekiq_memory_killer.md11
-rw-r--r--doc/administration/package_information/index.md6
-rw-r--r--doc/administration/package_information/licensing.md3
-rw-r--r--doc/administration/packages/container_registry.md42
-rw-r--r--doc/administration/packages/dependency_proxy.md9
-rw-r--r--doc/administration/packages/index.md253
-rw-r--r--doc/administration/pages/index.md50
-rw-r--r--doc/administration/postgresql/database_load_balancing.md20
-rw-r--r--doc/administration/postgresql/multiple_databases.md141
-rw-r--r--doc/administration/postgresql/pgbouncer.md4
-rw-r--r--doc/administration/postgresql/replication_and_failover.md186
-rw-r--r--doc/administration/raketasks/geo.md2
-rw-r--r--doc/administration/raketasks/github_import.md2
-rw-r--r--doc/administration/raketasks/maintenance.md2
-rw-r--r--doc/administration/reference_architectures/10k_users.md66
-rw-r--r--doc/administration/reference_architectures/25k_users.md68
-rw-r--r--doc/administration/reference_architectures/2k_users.md52
-rw-r--r--doc/administration/reference_architectures/3k_users.md67
-rw-r--r--doc/administration/reference_architectures/50k_users.md70
-rw-r--r--doc/administration/reference_architectures/5k_users.md73
-rw-r--r--doc/administration/reference_architectures/index.md9
-rw-r--r--doc/administration/restart_gitlab.md16
-rw-r--r--doc/administration/server_hooks.md19
-rw-r--r--doc/administration/sidekiq.md11
-rw-r--r--doc/administration/sidekiq/extra_sidekiq_processes.md286
-rw-r--r--doc/administration/sidekiq/extra_sidekiq_routing.md166
-rw-r--r--doc/administration/sidekiq/index.md6
-rw-r--r--doc/administration/sidekiq/processing_specific_job_classes.md337
-rw-r--r--doc/administration/sidekiq/sidekiq_job_migration.md19
-rw-r--r--doc/administration/sidekiq/sidekiq_troubleshooting.md171
-rw-r--r--doc/administration/sidekiq_health_check.md11
-rw-r--r--doc/administration/snippets/index.md4
-rw-r--r--doc/administration/terraform_state.md4
-rw-r--r--doc/administration/troubleshooting/elasticsearch.md11
-rw-r--r--doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md98
-rw-r--r--doc/administration/troubleshooting/linux_cheat_sheet.md4
-rw-r--r--doc/administration/troubleshooting/postgresql.md12
-rw-r--r--doc/administration/troubleshooting/sidekiq.md11
-rw-r--r--doc/administration/troubleshooting/tracing_correlation_id.md11
-rw-r--r--doc/administration/uploads.md22
-rw-r--r--doc/administration/user_settings.md2
98 files changed, 2684 insertions, 1739 deletions
diff --git a/doc/administration/application_settings_cache.md b/doc/administration/application_settings_cache.md
index d04056beb96..30019df44aa 100644
--- a/doc/administration/application_settings_cache.md
+++ b/doc/administration/application_settings_cache.md
@@ -20,7 +20,7 @@ To change the expiry value:
::Tabs
-:::TabTitle Omnibus package
+:::TabTitle Linux package (Omnibus)
1. Edit `/etc/gitlab/gitlab.rb`:
@@ -36,7 +36,7 @@ To change the expiry value:
gitlab-ctl restart
```
-:::TabTitle Source
+:::TabTitle Self-compiled (Source)
1. Edit `config/gitlab.yml`:
diff --git a/doc/administration/audit_event_streaming.md b/doc/administration/audit_event_streaming.md
index 0af1af12a60..4d0e6518ebb 100644
--- a/doc/administration/audit_event_streaming.md
+++ b/doc/administration/audit_event_streaming.md
@@ -18,6 +18,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w
> - Custom HTTP headers UI [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/365259) in GitLab 15.3. [Feature flag `custom_headers_streaming_audit_events_ui`](https://gitlab.com/gitlab-org/gitlab/-/issues/365259) removed.
> - [Improved user experience](https://gitlab.com/gitlab-org/gitlab/-/issues/367963) in GitLab 15.3.
> - User-specified verification token API support [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/360813) in GitLab 15.4.
+> - Event type filters API [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/344845) in GitLab 15.7.
Users can set a streaming destination for a top-level group to receive all audit events about the group, its subgroups, and
projects as structured JSON.
@@ -143,6 +144,7 @@ query {
id
}
}
+ eventTypeFilters
}
}
}
@@ -284,6 +286,63 @@ Users with the Owner role for a group can list streaming destinations and see th
1. On the main area, select the **Streams**.
1. View the verification token on the right side of each item.
+## Event type filters
+
+> Event type filters API [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/344845) in GitLab 15.7.
+
+When this feature is enabled for a group, you can use an API to permit users to filter streamed audit events per destination.
+If the feature is enabled with no filters, the destination receives all audit events.
+
+A streaming destination that has an event type filter set has a **filtered** (**{filter}**) label.
+
+### Use the API to add an event type filter
+
+Prerequisites:
+
+- You must have the Owner role for the group.
+
+You can add a list of event type filters using the `auditEventsStreamingDestinationEventsAdd` query type:
+
+```graphql
+mutation {
+ auditEventsStreamingDestinationEventsAdd(input: {
+ destinationId: "gid://gitlab/AuditEvents::ExternalAuditEventDestination/1",
+ eventTypeFilters: ["list of event type filters"]}){
+ errors
+ eventTypeFilters
+ }
+}
+```
+
+Event type filters are added if:
+
+- The returned `errors` object is empty.
+- The API responds with `200 OK`.
+
+### Use the API to remove an event type filter
+
+Prerequisites:
+
+- You must have the Owner role for the group.
+
+You can remove a list of event type filters using the `auditEventsStreamingDestinationEventsRemove` query type:
+
+```graphql
+mutation {
+ auditEventsStreamingDestinationEventsRemove(input: {
+ destinationId: "gid://gitlab/AuditEvents::ExternalAuditEventDestination/1",
+ eventTypeFilters: ["list of event type filters"]
+ }){
+ errors
+ }
+}
+```
+
+Event type filters are removed if:
+
+- The returned `errors` object is empty.
+- The API responds with `200 OK`.
+
## Payload schema
> Documentation for an audit event streaming schema was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/358149) in GitLab 15.3.
diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md
index 0aa0d163972..1951ab5e2c7 100644
--- a/doc/administration/audit_events.md
+++ b/doc/administration/audit_events.md
@@ -12,6 +12,8 @@ You can use audit events to track, for example:
- Who changed the permission level of a particular user for a GitLab project, and when.
- Who added a new user or removed a user, and when.
+Audit events are similar to the [log system](logs/index.md).
+
The GitLab API, database, and `audit_json.log` record many audit events. Some audit events are only available through
[streaming audit events](audit_event_streaming.md).
@@ -21,56 +23,148 @@ NOTE:
You can't configure a retention policy for audit events, but epic
[7917](https://gitlab.com/groups/gitlab-org/-/epics/7917) proposes to change this.
-## List of events
+## Time zones
-There are two kinds of events logged:
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/242014) in GitLab 15.7, GitLab UI shows dates and times in the user's local time zone instead of UTC.
-- Events scoped to the group or project, used by group and project managers
- to look up who made a change.
-- Instance events scoped to the whole GitLab instance, used by your Compliance team to
- perform formal audits.
+The time zone used for audit events depends on where you view them:
-NOTE:
-Some events are recorded and available only as [streaming audit events](audit_event_streaming.md).
+- In GitLab UI, your local time zone (GitLab 15.7 and later) or UTC (GitLab 15.6 and earlier) is used.
+- The [Audit Events API](../api/audit_events.md) returns dates and times in UTC by default, or the
+ [configured time zone](timezone.md) on a self-managed GitLab instance.
+- In `audit_json.log`, UTC is used.
+- In CSV exports, UTC is used.
-### Impersonation data
+## View audit events
-> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/536) in GitLab 13.0.
+Depending on the events you want to view, at a minimum you must have:
-When a user is being [impersonated](../user/admin_area/index.md#user-impersonation), their actions are logged as audit events as usual, with two additional details:
+- For group audit events of all users in the group, the Owner role for the group.
+- For project audit events of all users in the project, the Maintainer role for the project.
+- For group and project audit events based on your own actions, the Developer role for the group or project.
+- [Auditor users](auditor_users.md) can see group and project events for all users.
-1. Usual audit events include information about the impersonating administrator. These audit events are visible in their
- respective audit event pages depending on their type (group, project, or user).
-1. Extra audit events are recorded for the start and stop of the administrator's impersonation session. These audit events
- are visible in the:
- - Instance audit events.
- - Group audit events for all groups the user belongs to (GitLab 14.8 and later). For performance reasons, group audit
- events are limited to the oldest 20 groups to which you belong.
+You can view audit events scoped to a group or project.
-![audit events](img/impersonated_audit_events_v13_8.png)
+To view a group's audit events:
-### Group events
+1. Go to the group.
+1. On the left sidebar, select **Security & Compliance > Audit Events**.
-A user with:
+Group events do not include project audit events. Group events can also be accessed using the
+[Group Audit Events API](../api/audit_events.md#group-audit-events). Group event queries are limited to a maximum of 30
+days.
-- Owner role (or above) can retrieve group audit events of all users.
-- Developer or Maintainer role is limited to group audit events based on their individual actions.
+To view a project's audit events:
-Group events do not include project audit events.
+1. Go to the project.
+1. On the left sidebar, select **Security & Compliance > Audit Events**.
-To view a group's audit events:
+Project events can also be accessed using the [Project Audit Events API](../api/audit_events.md#project-audit-events).
+Project event queries are limited to a maximum of 30 days.
-1. Go to the group.
-1. On the left sidebar, select **Security & Compliance > Audit Events**.
+## View instance audit events **(PREMIUM SELF)**
-From there, you can see the following actions:
+You can view audit events from user actions across an entire GitLab instance.
+
+To view instance audit events:
+
+1. On the top bar, select **Main menu > Admin**.
+1. On the left sidebar, select **Monitoring > Audit Events**.
+
+### Export to CSV
+
+> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1449) in GitLab 13.4.
+> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/285441) in GitLab 13.7.
+
+You can export the current view (including filters) of your instance audit events as a CSV file. To export the instance
+audit events to CSV:
+
+1. On the top bar, select **Main menu > Admin**.
+1. On the left sidebar, select **Monitoring > Audit Events**.
+1. Select the available search [filters](#filter-audit-events).
+1. Select **Export as CSV**.
+
+The exported file:
+
+- Is sorted by `created_at` in ascending order.
+- Is limited to a maximum of 100 000 events. The remaining records are truncated when this limit is reached.
+
+Data is encoded with:
+
+- Comma as the column delimiter.
+- `"` to quote fields if necessary.
+- New lines separate rows.
+
+The first row contains the headers, which are listed in the following table along with a description of the values:
+
+| Column | Description |
+|:---------------------|:---------------------------------------------------|
+| **ID** | Audit event `id`. |
+| **Author ID** | ID of the author. |
+| **Author Name** | Full name of the author. |
+| **Entity ID** | ID of the scope. |
+| **Entity Type** | Type of the scope (`Project`, `Group`, or `User`). |
+| **Entity Path** | Path of the scope. |
+| **Target ID** | ID of the target. |
+| **Target Type** | Type of the target. |
+| **Target Details** | Details of the target. |
+| **Action** | Description of the action. |
+| **IP Address** | IP address of the author who performed the action. |
+| **Created At (UTC)** | Formatted as `YYYY-MM-DD HH:MM:SS`. |
+
+## View sign-in events **(FREE)**
+
+Successful sign-in events are the only audit events available at all tiers. To see successful sign-in events:
+
+1. Select your avatar.
+1. Select **Edit profile > Authentication log**.
+
+After upgrading to a paid tier, you can see also see successful sign-in events on audit event pages.
+
+## Filter audit events
+
+From audit events pages, different filters are available depending on the page you're on.
+
+| Audit event page | Available filter |
+|:-----------------|:-----------------------------------------------------------------------------------------------------------------------|
+| Project | User (member of the project) who performed the action. |
+| Group | User (member of the group) who performed the action. |
+| Instance | Group, project, or user. |
+| All | Date range buttons and pickers (maximum range of 31 days). Default is from the first day of the month to today's date. |
+
+## User impersonation
+
+> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/536) in GitLab 13.0.
+> - Impersonation session events included in group audit events in GitLab 14.8.
+
+When a user is [impersonated](../user/admin_area/index.md#user-impersonation), their actions are logged as audit events
+with additional details:
+
+- Audit events include information about the impersonating administrator. These audit events are visible in audit event
+ pages depending on the audit event type (group, project, or user).
+- Extra audit events are recorded for the start and end of the administrator's impersonation session. These audit events
+ are visible as:
+ - Instance audit events.
+ - Group audit events for all groups the user belongs to. For performance reasons, group audit events are limited to
+ the oldest 20 groups you belong to.
+
+![Audit event with impersonated user](img/impersonated_audit_events_v15_7.png)
+
+## Available audit events
+
+You can view different events depending on the version of GitLab you have.
+
+### Group events
+
+The following actions on groups generate group audit events:
- Group name or path changed.
- Group repository size limit changed.
- Group created or deleted.
- Group changed visibility.
- User was added to group and with which [permissions](../user/permissions.md).
-- User sign-in via [Group SAML](../user/group/saml_sso/index.md).
+- User sign-in using [Group SAML](../user/group/saml_sso/index.md).
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8071) in GitLab 14.5, changes to the following
[group SAML](../user/group/saml_sso/index.md) configuration:
- Enabled status.
@@ -85,8 +179,8 @@ From there, you can see the following actions:
- Permissions changes of a user assigned to a group.
- Removed user from group.
- Project repository imported into group.
-- [Project shared with group](../user/project/members/share_project_with_groups.md)
- and with which [permissions](../user/permissions.md).
+- [Project shared with group](../user/project/members/share_project_with_groups.md) and with which
+ [permissions](../user/permissions.md).
- Removal of a previously shared group with a project.
- LFS enabled or disabled.
- Shared runners minutes limit changed.
@@ -94,36 +188,36 @@ From there, you can see the following actions:
- Request access enabled or disabled.
- 2FA enforcement or grace period changed.
- Roles allowed to create project changed.
-- Group CI/CD variable added, removed, or protected status changed. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30857) in GitLab 13.3.
-- Compliance framework created, updated, or deleted. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340649) in GitLab 14.5.
-- Event streaming destination created, updated, or deleted. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/344664) in GitLab 14.6.
-- Instance administrator started or stopped impersonation of a group member. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/300961) in GitLab 14.8.
-- Group deploy token was successfully created, revoked, or deleted. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/353452) in GitLab 14.9.
-- Failed attempt to create a group deploy token. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/353452) in GitLab 14.9.
-- [IP restrictions](../user/group/access_and_permissions.md#restrict-access-to-groups-by-ip-address) changed. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/358986) in GitLab 15.0.
+- Group CI/CD variable added, removed, or protected status changed.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30857) in GitLab 13.3.
+- Compliance framework created, updated, or deleted.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340649) in GitLab 14.5.
+- Event streaming destination created, updated, or deleted.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/344664) in GitLab 14.6.
+- Instance administrator started or stopped impersonation of a group member.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/300961) in GitLab 14.8.
+- Group deploy token was successfully created, revoked, or deleted.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/353452) in GitLab 14.9.
+- Failed attempt to create a group deploy token. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/353452)
+ in GitLab 14.9.
+- [IP restrictions](../user/group/access_and_permissions.md#restrict-access-to-groups-by-ip-address) changed.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/358986) in GitLab 15.0.
- Changes to push rules. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/227629) in GitLab 15.0.
-- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/356152) in GitLab 15.1, changes to the following merge request approvals settings:
+- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/356152) in GitLab 15.1, changes to the following merge
+ request approvals settings:
- Prevent approval by author.
- Prevent approvals by users who add commits.
- Prevent editing approval rules in projects and merge requests.
- Require user password to approve.
- Remove all approvals when commits are added to the source branch.
-- Changes to streaming audit destination custom HTTP headers. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/366350) in GitLab 15.3.
-- Group had a security policy project linked, changed, or unlinked. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/377877) in GitLab 15.6)
-
-Group events can also be accessed via the [Group Audit Events API](../api/audit_events.md#group-audit-events)
+- Changes to streaming audit destination custom HTTP headers.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/366350) in GitLab 15.3.
+- Group had a security policy project linked, changed, or unlinked.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/377877) in GitLab 15.6.
### Project events
-A user with a Maintainer role (or above) can retrieve project audit events of all users.
-A user with a Developer role is limited to project audit events based on their individual actions.
-
-To view a project's audit events:
-
-1. Go to the project.
-1. On the left sidebar, select **Security & Compliance > Audit Events**.
-
-From there, you can see the following actions:
+The following actions on projects generate project audit events:
- Added or removed deploy keys
- Project created, deleted, renamed, moved (transferred), changed path
@@ -138,68 +232,87 @@ From there, you can see the following actions:
- Added, removed, or updated protected branches
- Release was added to a project
- Release was updated
-- Release was deleted ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/94793/) in GitLab 15.3)
+- Release was deleted. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/94793/) in GitLab 15.3.
- Release milestone associations changed
-- Permission to approve merge requests by committers was updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7531) in GitLab 12.9)
+- Permission to approve merge requests by committers was updated.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7531) in GitLab 12.9.
- Permission to approve merge requests by committers was updated.
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7531) in GitLab 12.9.
- Message for event [changed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/72623/diffs) in GitLab 14.6.
-
-- Permission to approve merge requests by authors was updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7531) in GitLab 12.9)
-- Number of required approvals was updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7531) in GitLab 12.9)
-- Added or removed users and groups from project approval groups ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213603) in GitLab 13.2)
-- Project CI/CD variable added, removed, or protected status changed ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30857) in GitLab 13.4)
-- Project access token was successfully created or revoked ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230007) in GitLab 13.9)
-- Failed attempt to create or revoke a project access token ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230007) in GitLab 13.9)
-- When default branch changes for a project ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/52339) in GitLab 13.9)
-- Created, updated, or deleted DAST profiles, DAST scanner profiles, and DAST site profiles
- ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217872) in GitLab 14.1)
-- Changed a project's compliance framework ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/329362) in GitLab 14.1)
-- User password required for approvals was updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336211) in GitLab 14.2)
-- Permission to modify merge requests approval rules in merge requests was updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336211) in GitLab 14.2)
-- New approvals requirement when new commits are added to an MR was updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336211) in GitLab 14.2)
-- When [strategies for feature flags](../operations/feature_flags.md#feature-flag-strategies) are changed ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68408) in GitLab 14.3)
-- Allowing force push to protected branch changed ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/338873) in GitLab 14.3)
-- Code owner approval requirement on merge requests targeting protected branch changed ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/338873) in GitLab 14.3)
-- Users and groups allowed to merge and push to protected branch added or removed ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/338873) in GitLab 14.3)
-- Project deploy token was successfully created, revoked or deleted ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/353451) in GitLab 14.9)
-- Failed attempt to create a project deploy token ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/353451) in GitLab 14.9)
-- When merge method is updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9)
-- Merged results pipelines enabled or disabled ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9)
-- Merge trains enabled or disabled ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9)
-- Automatically resolve merge request diff discussions enabled or disabled ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9)
-- Show link to create or view a merge request when pushing from the command line enabled or disabled ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9)
-- Delete source branch option by default enabled or disabled ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9)
-- Squash commits when merging is updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9)
-- Pipelines must succeed enabled or disabled ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9)
-- Skipped pipelines are considered successful enabled or disabled ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9)
-- All discussions must be resolved enabled or disabled ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9)
-- Commit message suggestion is updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9)
-- Status check is added, edited, or deleted ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355805) in GitLab 15.0)
-- Merge commit message template is updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355805) in GitLab 15.0)
-- Squash commit message template is updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355805) in GitLab 15.0)
-- Default description template for merge requests is updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355805) in GitLab 15.0)
-- Project was scheduled for deletion due to inactivity ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/85689) in GitLab 15.0)
-- Project had a security policy project linked, changed, or unlinked ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/377877) in GitLab 15.6)
-
-Project events can also be accessed via the [Project Audit Events API](../api/audit_events.md#project-audit-events).
-
-Project event queries are limited to a maximum of 30 days.
+- Permission to approve merge requests by authors was updated.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7531) in GitLab 12.9.
+- Number of required approvals was updated.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7531) in GitLab 12.9.
+- Added or removed users and groups from project approval groups.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213603) in GitLab 13.2.
+- Project CI/CD variable added, removed, or protected status changed.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30857) in GitLab 13.4.
+- Project access token was successfully created or revoked.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230007) in GitLab 13.9.
+- Failed attempt to create or revoke a project access token.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230007) in GitLab 13.9.
+- When default branch changes for a project.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/52339) in GitLab 13.9.
+- Created, updated, or deleted DAST profiles, DAST scanner profiles, and DAST site profiles.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217872) in GitLab 14.1.
+- Changed a project's compliance framework.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/329362) in GitLab 14.1.
+- User password required for approvals was updated.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336211) in GitLab 14.2.
+- Permission to modify merge requests approval rules in merge requests was updated.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336211) in GitLab 14.2.
+- New approvals requirement when new commits are added to an MR was updated.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336211) in GitLab 14.2.
+- When [strategies for feature flags](../operations/feature_flags.md#feature-flag-strategies) are changed.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68408) in GitLab 14.3.
+- Allowing force push to protected branch changed.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/338873) in GitLab 14.3.
+- Code owner approval requirement on merge requests targeting protected branch changed.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/338873) in GitLab 14.3.
+- Users and groups allowed to merge and push to protected branch added or removed.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/338873) in GitLab 14.3.
+- Project deploy token was successfully created, revoked or deleted.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/353451) in GitLab 14.9.
+- Failed attempt to create a project deploy token.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/353451) in GitLab 14.9.
+- When merge method is updated.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9.
+- Merged results pipelines enabled or disabled.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9.
+- Merge trains enabled or disabled.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9.
+- Automatically resolve merge request diff discussions enabled or disabled.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9.
+- Show link to create or view a merge request when pushing from the command line enabled or disabled.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9.
+- Delete source branch option by default enabled or disabled.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9.
+- Squash commits when merging is updated.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9.
+- Pipelines must succeed enabled or disabled.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9.
+- Skipped pipelines are considered successful enabled or disabled.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9.
+- All discussions must be resolved enabled or disabled.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9.
+- Commit message suggestion is updated.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9.
+- Status check is added, edited, or deleted.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355805) in GitLab 15.0.
+- Merge commit message template is updated.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355805) in GitLab 15.0.
+- Squash commit message template is updated.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355805) in GitLab 15.0.
+- Default description template for merge requests is updated.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355805) in GitLab 15.0.
+- Project was scheduled for deletion due to inactivity.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/85689) in GitLab 15.0.
+- Project had a security policy project linked, changed, or unlinked.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/377877) in GitLab 15.6.
### Instance events **(PREMIUM SELF)**
-Server-wide audit events introduce the ability to observe user actions across
-the entire instance of your GitLab server, making it easy to understand who
-changed what and when for audit purposes.
-
-Instance events do not include group or project audit events.
-
-To view the server-wide audit events:
-
-1. On the top bar, select **Main menu > Admin**.
-1. On the left sidebar, select **Monitoring > Audit Events**.
-
-The following user actions are recorded:
+The following user actions on a GitLab instance generate instance audit events:
- Sign-in events and the authentication type (such as standard, LDAP, or OmniAuth)
- Failed sign-ins
@@ -209,123 +322,47 @@ The following user actions are recorded:
- Ask for password reset
- Grant OAuth access
- Started or stopped user impersonation
-- Changed username ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7797) in GitLab 12.8)
-- User was deleted ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/251) in GitLab 12.8)
-- User was added ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/251) in GitLab 12.8)
-- User requests access to an instance ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/298783) in GitLab 13.9)
-- User was approved via Admin Area ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276250) in GitLab 13.6)
-- User was rejected via Admin Area ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/298783) in GitLab 13.9)
-- User was blocked via Admin Area ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/251) in GitLab 12.8)
-- User was blocked via API ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25872) in GitLab 12.9)
-- Failed second-factor authentication attempt ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16826) in GitLab 13.5)
-- A user's personal access token was successfully created or revoked ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276921) in GitLab 13.6)
-- A failed attempt to create or revoke a user's personal access token ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276921) in GitLab 13.6)
-- Administrator added or removed ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/323905) in GitLab 14.1)
-- Removed SSH key ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/220127) in GitLab 14.1)
-- Added or removed GPG key ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/220127) in GitLab 14.1)
-- A user's two-factor authentication was disabled ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/238177) in GitLab 15.1)
-
-Instance events can also be accessed via the [Instance Audit Events API](../api/audit_events.md#instance-audit-events).
-
-### Sign-in events **(FREE)**
-
-Successful sign-in events are the only Audit Events available at all tiers. To see
-successful sign-in events:
-
-1. Select your avatar.
-1. Select **Edit profile > Authentication log**.
-
-After upgrading from GitLab Free to a paid tier, successful sign-in events are the only Audit
-Events visible in Audit Events views until more events are logged.
-
-### "Deleted User" events
-
-Audit events can be created for a user after the user is deleted. The user name associated with the event is set to
-"Deleted User" because the actual user name is unknowable. For example, if a deleted user's access to a project is
-removed automatically due to expiration, the audit event is created for "Deleted User". We are [investigating](https://gitlab.com/gitlab-org/gitlab/-/issues/343933)
-whether this is avoidable.
-
-### Missing events
-
-Some events are not tracked in audit events. See the following
-epics for more detail on which events are not being tracked, and our progress
-on adding these events into GitLab:
-
-- [Project settings and activity](https://gitlab.com/groups/gitlab-org/-/epics/474)
-- [Group settings and activity](https://gitlab.com/groups/gitlab-org/-/epics/475)
-- [Instance-level settings and activity](https://gitlab.com/groups/gitlab-org/-/epics/476)
-
-Don't see the event you want in any of the epics linked above? You can either:
+- Changed username. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7797) in GitLab 12.8.
+- User was deleted. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/251) in GitLab 12.8.
+- User was added. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/251) in GitLab 12.8.
+- User requests access to an instance. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/298783) in GitLab 13.9.
+- User was approved using the Admin Area. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276250) in GitLab 13.6.
+- User was rejected using the Admin Area. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/298783) in GitLab 13.9.
+- User was blocked using the Admin Area. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/251) in GitLab 12.8.
+- User was blocked using the API. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25872) in GitLab 12.9.
+- Failed second-factor authentication attempt. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16826) in
+ GitLab 13.5.
+- A user's personal access token was successfully created or revoked.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276921) in GitLab 13.6.
+- A failed attempt to create or revoke a user's personal access token.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276921) in GitLab 13.6.
+- Administrator added or removed. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/323905) in GitLab 14.1.
+- Removed SSH key. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/220127) in GitLab 14.1.
+- Added or removed GPG key. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/220127) in GitLab 14.1.
+- A user's two-factor authentication was disabled. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/238177) in
+ GitLab 15.1.
+- Enabled Admin Mode. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/362101) in GitLab 15.7.
+
+Instance events can also be accessed using the [Instance Audit Events API](../api/audit_events.md#instance-audit-events).
+
+## "Deleted User" events
+
+Audit events created after users are deleted are created for "Deleted User". For example, if a deleted user's access to
+a project is removed automatically due to expiration.
+
+Issue [343933](https://gitlab.com/gitlab-org/gitlab/-/issues/343933) proposes to change this behavior.
+
+## Unsupported events
+
+Some events are not tracked in audit events. The following epics propose support for more events:
+
+- [Project settings and activity](https://gitlab.com/groups/gitlab-org/-/epics/474).
+- [Group settings and activity](https://gitlab.com/groups/gitlab-org/-/epics/475).
+- [Instance-level settings and activity](https://gitlab.com/groups/gitlab-org/-/epics/476).
+
+If you don't see the event you want in any of the epics, you can either:
- Use the **Audit Event Proposal** issue template to
[create an issue](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Audit%20Event%20Proposal) to
request it.
- [Add it yourself](../development/audit_event_guide/index.md).
-
-### Removed events
-
-> - Repositories push events was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/337993) in GitLab 14.3.
-> - Repositories push events was [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/337993) in GitLab 15.0.
-
-The repositories push events feature was:
-
-- [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/337993) in GitLab 14.3.
-- [Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/337993) in GitLab 15.0.
-
-## Search
-
-The search filters you can see depends on which audit level you are at.
-
-| Filter | Available options |
-| ------ | ----------------- |
-| Scope (Project level) | A specific user who performed the action. |
-| Scope (Group level) | A specific user (in a group) who performed the action. |
-| Scope (Instance level) | A specific group, project, or user that the action was scoped to. |
-| Date range | Either via the date range buttons or pickers (maximum range of 31 days). Default is from the first day of the month to today's date. |
-
-![audit events](img/audit_events_v14_5.png)
-
-## Export to CSV **(PREMIUM SELF)**
-
-> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1449) in GitLab 13.4.
-> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/285441) in GitLab 13.7.
-
-Export to CSV allows customers to export the current filter view of your audit events as a
-CSV file, which stores tabular data in plain text. The data provides a comprehensive view with respect to
-audit events.
-
-To export the audit events to CSV:
-
-1. On the top bar, select **Main menu > Admin**.
-1. On the left sidebar, select **Monitoring > Audit Events**.
-1. Select the available search [filters](#search).
-1. Select **Export as CSV**.
-
-### Sort
-
-Exported events are always sorted by `created_at` in ascending order.
-
-### Format
-
-Data is encoded with a comma as the column delimiter, with `"` used to quote fields if needed, and newlines to separate rows.
-The first row contains the headers, which are listed in the following table along with a description of the values:
-
-| Column | Description |
-|---------|-------------|
-| ID | Audit event `id` |
-| Author ID | ID of the author |
-| Author Name | Full name of the author |
-| Entity ID | ID of the scope |
-| Entity Type | Type of the scope (`Project`/`Group`/`User`) |
-| Entity Path | Path of the scope |
-| Target ID | ID of the target |
-| Target Type | Type of the target |
-| Target Details | Details of the target |
-| Action | Description of the action |
-| IP Address | IP address of the author who performed the action |
-| Created At (UTC) | Formatted as `YYYY-MM-DD HH:MM:SS` |
-
-### Limitation
-
-The audit events CSV file is limited to a maximum of `100,000` events.
-The remaining records are truncated when this limit is reached.
diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md
index 6243f3da2d2..2cb9bac7af9 100644
--- a/doc/administration/auth/ldap/index.md
+++ b/doc/administration/auth/ldap/index.md
@@ -87,9 +87,10 @@ with `start_tls` and `ssl` was replaced with `simple_tls`.
LDAP users must have a set email address, regardless of whether or not it's used
to sign in.
-### Example Omnibus GitLab configuration
+### Example Linux package (Omnibus) configuration
-This example shows configuration for Omnibus GitLab instances:
+This example shows a sample configuration for a GitLab instance that
+was installed by using the Linux package (Omnibus):
```ruby
gitlab_rails['ldap_enabled'] = true
@@ -135,9 +136,14 @@ gitlab_rails['ldap_servers'] = {
}
```
-### Example source install configuration
+### Example Helm chart (Kubernetes) configuration
-This example shows configuration for source install instances:
+View [how to configure LDAP for a GitLab instance that was installed by using the Helm chart](https://docs.gitlab.com/charts/charts/globals.html#ldap).
+
+### Example self-compiled (source) configuration
+
+This example shows a sample configuration for a GitLab instance that
+was installed by using the self-compiled source:
```yaml
production:
@@ -358,7 +364,9 @@ This can lead to several confusing issues such as creating links or namespaces w
GitLab can automatically lowercase usernames provided by the LDAP server by enabling
the configuration option `lowercase_usernames`. By default, this configuration option is `false`.
-**Omnibus configuration**
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
1. Edit `/etc/gitlab/gitlab.rb`:
@@ -373,7 +381,7 @@ the configuration option `lowercase_usernames`. By default, this configuration o
1. [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
-**Source configuration**
+:::TabTitle Self-compiled (source)
1. Edit `config/gitlab.yaml`:
@@ -388,6 +396,8 @@ the configuration option `lowercase_usernames`. By default, this configuration o
1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect.
+::EndTabs
+
### Disable LDAP web sign in
It can be useful to prevent using LDAP credentials through the web UI when
@@ -398,7 +408,9 @@ checks like custom 2FA.
When LDAP web sign in is disabled, users don't see an **LDAP** tab on the sign-in page.
This does not disable using LDAP credentials for Git access.
-**Omnibus configuration**
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
1. Edit `/etc/gitlab/gitlab.rb`:
@@ -408,7 +420,30 @@ This does not disable using LDAP credentials for Git access.
1. [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
-**Source configuration**
+:::TabTitle Helm chart (Kubernetes)
+
+1. Export the Helm values:
+
+ ```shell
+ helm get values gitlab > gitlab_values.yaml
+ ```
+
+1. Edit `gitlab_values.yaml`:
+
+ ```yaml
+ global:
+ appConfig:
+ ldap:
+ preventSignin: true
+ ```
+
+1. Save the file and apply the new values:
+
+ ```shell
+ helm upgrade -f gitlab_values.yaml gitlab gitlab/gitlab
+ ```
+
+:::TabTitle Self-compiled (source)
1. Edit `config/gitlab.yaml`:
@@ -420,6 +455,8 @@ This does not disable using LDAP credentials for Git access.
1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect.
+::EndTabs
+
### Use encrypted credentials
Instead of having the LDAP integration credentials stored in plaintext in the configuration files, you can optionally
@@ -439,7 +476,9 @@ The supported configuration items for the encrypted file are:
The encrypted contents can be configured with the [LDAP secret edit Rake command](../../raketasks/ldap.md#edit-secret).
-**Omnibus configuration**
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
If initially your LDAP configuration looked like:
@@ -473,7 +512,7 @@ If initially your LDAP configuration looked like:
1. [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
-**Source configuration**
+:::TabTitle Self-compiled (source)
If initially your LDAP configuration looked like:
@@ -507,6 +546,24 @@ If initially your LDAP configuration looked like:
1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect.
+::EndTabs
+
+## Updating LDAP DN and email
+
+When an LDAP server creates a user in GitLab, the user's LDAP distinguished name (DN) is linked to their GitLab account
+as an identifier.
+
+When a user tries to sign in with LDAP, GitLab tries to find the user using the DN saved on that user's account.
+
+- If GitLab finds the user by the DN and the user's email address:
+ - Matches the GitLab account's email address, GitLab does not take any further action.
+ - Has changed, GitLab updates its record of the user's email to match the one in LDAP.
+- If GitLab cannot find a user by their DN, it tries to find the user by their email. If GitLab:
+ - Finds the user by their email, GitLab updates the DN stored in the user's GitLab account. Both values now
+ match the information stored in LDAP.
+ - Cannot find the user by their email address (both the DN **and** the email address have changed), see
+ [User DN and email have changed](ldap-troubleshooting.md#user-dn-and-email-have-changed).
+
## Disable anonymous LDAP authentication
GitLab doesn't support TLS client authentication. Complete these steps on your LDAP server.
@@ -543,7 +600,7 @@ Updating user email addresses must be done on the LDAP server that manages the u
The updated user's previous email address becomes the secondary email address to preserve that user's commit history.
-You can find more details on the expected behavior of user updates in our [LDAP troubleshooting section](ldap-troubleshooting.md#user-dn-orand-email-have-changed).
+You can find more details on the expected behavior of user updates in our [LDAP troubleshooting section](ldap-troubleshooting.md#user-dn-and-email-have-changed).
## Google Secure LDAP
diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md
index 499c3c64af7..21ec4b293d4 100644
--- a/doc/administration/auth/ldap/ldap-troubleshooting.md
+++ b/doc/administration/auth/ldap/ldap-troubleshooting.md
@@ -541,7 +541,7 @@ Usually this is not a cause for concern.
If you think a particular user should already exist in GitLab, but you're seeing
this entry, it could be due to a mismatched DN stored in GitLab. See
-[User DN and/or email have changed](#user-dn-orand-email-have-changed) to update the user's LDAP identity.
+[User DN and email have changed](#user-dn-and-email-have-changed) to update the user's LDAP identity.
```shell
User with DN `uid=john0,ou=people,dc=example,dc=com` should have access
@@ -624,26 +624,16 @@ does not do this:
1. Wait until LDAP group synchronization has finished running.
1. Remove the user from the LDAP group.
-### User DN or/and email have changed
+### User DN and email have changed
-When an LDAP user is created in GitLab, their LDAP DN is stored for later reference.
+If both the primary email **and** the DN change in LDAP, GitLab cannot identify the correct LDAP record of a user. As a
+result, GitLab blocks that user. So that GitLab can find the LDAP record, update the user's existing GitLab profile with
+at least either:
-If GitLab cannot find a user by their DN, it falls back
-to finding the user by their email. If the lookup is successful, GitLab
-updates the stored DN to the new value so both values now match what's in
-LDAP.
+- The new primary email.
+- DN values.
-If the email has changed and the DN has not, GitLab finds the user with
-the DN and updates its own record of the user's email to match the one in LDAP.
-
-However, if the primary email _and_ the DN change in LDAP, then GitLab
-has no way of identifying the correct LDAP record of the user and, as a
-result, the user is blocked. To rectify this, the user's existing
-profile must be updated with at least one of the new values (primary
-email or DN) so the LDAP record can be found.
-
-The following script updates the emails for all provided users so they
-aren't blocked or unable to access their accounts.
+The following script updates the emails for all provided users so they aren't blocked or unable to access their accounts.
NOTE:
The following script requires that any new accounts with the new
@@ -669,7 +659,7 @@ end
You can then [run a UserSync](#sync-all-users) **(PREMIUM SELF)** to sync the latest DN
for each of these users.
-## Could not authenticate you from ldapmain because "Unknown provider"
+## `Could not authenticate you from Ldapmain because "Unknown provider"`
You can receive the following error when authenticating with an LDAP server:
@@ -831,6 +821,38 @@ ldapsearch -D "cn=admin,dc=ldap-testing,dc=example,dc=com" \
The `bind_dn`, `password`, `port`, `host`, and `base` are all
identical to what's configured in the `gitlab.rb`.
+#### Use ldapsearch with `start_tls` encryption
+
+The previous example performs an LDAP test in plaintext to port 389. If you are using [`start_tls` encryption](index.md#basic-configuration-settings), in
+the `ldapsearch` command include:
+
+- The `-Z` flag.
+- The FQDN of the LDAP server.
+
+You must include these because, during TLS negotiation, the FQDN of the LDAP server is evaluated against its certificate:
+
+```shell
+ldapsearch -D "cn=admin,dc=ldap-testing,dc=example,dc=com" \
+ -w Password1 \
+ -p 389 \
+ -h "testing.ldap.com" \
+ -b "dc=ldap-testing,dc=example,dc=com" -Z
+```
+
+#### Use ldapsearch with `simple_tls` encryption
+
+If you are using [`simple_tls` encryption](index.md#basic-configuration-settings) (usually on port 636), include the following in the `ldapsearch` command:
+
+- The LDAP server FQDN with the `-H` flag and the port.
+- The full constructed URI.
+
+```shell
+ldapsearch -D "cn=admin,dc=ldap-testing,dc=example,dc=com" \
+ -w Password1 \
+ -H "ldaps://testing.ldap.com:636" \
+ -b "dc=ldap-testing,dc=example,dc=com"
+```
+
For more information, see the [official `ldapsearch` documentation](https://linux.die.net/man/1/ldapsearch).
### Using **AdFind** (Windows)
diff --git a/doc/administration/auth/ldap/ldap_synchronization.md b/doc/administration/auth/ldap/ldap_synchronization.md
index 02b04861844..5c8c7f7baf1 100644
--- a/doc/administration/auth/ldap/ldap_synchronization.md
+++ b/doc/administration/auth/ldap/ldap_synchronization.md
@@ -55,7 +55,9 @@ use a [crontab generator](http://www.crontabgenerator.com).
The example below shows how to set LDAP user
sync to run once every 12 hours at the top of the hour.
-**Omnibus installations**
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
1. Edit `/etc/gitlab/gitlab.rb`:
@@ -63,19 +65,77 @@ sync to run once every 12 hours at the top of the hour.
gitlab_rails['ldap_sync_worker_cron'] = "0 */12 * * *"
```
-1. [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+1. Save the file and reconfigure GitLab:
+
+ ```shell
+ sudo gitlab-ctl reconfigure
+ ```
+
+:::TabTitle Helm chart (Kubernetes)
+
+1. Export the Helm values:
+
+ ```shell
+ helm get values gitlab > gitlab_values.yaml
+ ```
+
+1. Edit `gitlab_values.yaml`:
+
+ ```yaml
+ global:
+ appConfig:
+ cron_jobs:
+ ldap_sync_worker:
+ cron: "0 */12 * * *"
+ ```
+
+1. Save the file and apply the new values:
+
+ ```shell
+ helm upgrade -f gitlab_values.yaml gitlab gitlab/gitlab
+ ```
+
+:::TabTitle Docker
+
+1. Edit `docker-compose.yml`:
+
+ ```yaml
+ version: "3.6"
+ services:
+ gitlab:
+ environment:
+ GITLAB_OMNIBUS_CONFIG: |
+ gitlab_rails['ldap_sync_worker_cron'] = "0 */12 * * *"
+ ```
+
+1. Save the file and restart GitLab:
-**Source installations**
+ ```shell
+ docker compose up -d
+ ```
-1. Edit `config/gitlab.yaml`:
+:::TabTitle Self-compiled (source)
+
+1. Edit `/home/git/gitlab/config/gitlab.yml`:
```yaml
- cron_jobs:
- ldap_sync_worker_cron:
- "0 */12 * * *"
+ production: &base
+ ee_cron_jobs:
+ ldap_sync_worker:
+ cron: "0 */12 * * *"
```
-1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect.
+1. Save the file and restart GitLab:
+
+ ```shell
+ # For systems running systemd
+ sudo systemctl restart gitlab.target
+
+ # For systems running SysV init
+ sudo service gitlab restart
+ ```
+
+::EndTabs
## Group sync
@@ -91,41 +151,103 @@ GitLab group membership to be automatically updated based on LDAP group members.
The `group_base` configuration should be a base LDAP 'container', such as an
'organization' or 'organizational unit', that contains LDAP groups that should
be available to GitLab. For example, `group_base` could be
-`ou=groups,dc=example,dc=com`. In the configuration file it looks like the
+`ou=groups,dc=example,dc=com`. In the configuration file, it looks like the
following.
-**Omnibus configuration**
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
gitlab_rails['ldap_servers'] = {
- 'main' => {
- # snip...
- 'group_base' => 'ou=groups,dc=example,dc=com',
- }
+ 'main' => {
+ 'group_base' => 'ou=groups,dc=example,dc=com',
+ }
}
```
-1. [Apply your changes to GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure).
+1. Save the file and reconfigure GitLab:
+
+ ```shell
+ sudo gitlab-ctl reconfigure
+ ```
+
+:::TabTitle Helm chart (Kubernetes)
+
+1. Export the Helm values:
+
+ ```shell
+ helm get values gitlab > gitlab_values.yaml
+ ```
-**Source configuration**
+1. Edit `gitlab_values.yaml`:
+
+ ```yaml
+ global:
+ appConfig:
+ ldap:
+ servers:
+ main:
+ group_base: ou=groups,dc=example,dc=com
+ ```
+
+1. Save the file and apply the new values:
+
+ ```shell
+ helm upgrade -f gitlab_values.yaml gitlab gitlab/gitlab
+ ```
+
+:::TabTitle Docker
+
+1. Edit `docker-compose.yml`:
+
+ ```yaml
+ version: "3.6"
+ services:
+ gitlab:
+ environment:
+ GITLAB_OMNIBUS_CONFIG: |
+ gitlab_rails['ldap_servers'] = {
+ 'main' => {
+ 'group_base' => 'ou=groups,dc=example,dc=com',
+ }
+ }
+ ```
+
+1. Save the file and restart GitLab:
+
+ ```shell
+ docker compose up -d
+ ```
+
+:::TabTitle Self-compiled (source)
1. Edit `/home/git/gitlab/config/gitlab.yml`:
```yaml
- production:
+ production: &base
ldap:
servers:
main:
- # snip...
group_base: ou=groups,dc=example,dc=com
```
-1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect.
+1. Save the file and restart GitLab:
+
+ ```shell
+ # For systems running systemd
+ sudo systemctl restart gitlab.target
+
+ # For systems running SysV init
+ sudo service gitlab restart
+ ```
+
+::EndTabs
To take advantage of group sync, group Owners or users with the [Maintainer role](../../../user/permissions.md) must
-[create one or more LDAP group links](#add-group-links).
+[create one or more LDAP group links](../../../user/group/access_and_permissions.md#manage-group-memberships-via-ldap).
### Add group links
@@ -146,37 +268,101 @@ as opposed to the full DN.
Additionally, if an LDAP user has an `admin` role, but is not a member of the `admin_group`
group, GitLab revokes their `admin` role when syncing.
-**Omnibus configuration**
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
gitlab_rails['ldap_servers'] = {
- 'main' => {
- # snip...
- 'group_base' => 'ou=groups,dc=example,dc=com',
- 'admin_group' => 'my_admin_group',
- }
+ 'main' => {
+ 'group_base' => 'ou=groups,dc=example,dc=com',
+ 'admin_group' => 'my_admin_group',
+ }
}
```
-1. [Apply your changes to GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure).
+1. Save the file and reconfigure GitLab:
+
+ ```shell
+ sudo gitlab-ctl reconfigure
+ ```
+
+:::TabTitle Helm chart (Kubernetes)
+
+1. Export the Helm values:
+
+ ```shell
+ helm get values gitlab > gitlab_values.yaml
+ ```
+
+1. Edit `gitlab_values.yaml`:
+
+ ```yaml
+ global:
+ appConfig:
+ ldap:
+ servers:
+ main:
+ group_base: ou=groups,dc=example,dc=com
+ admin_group: my_admin_group
+ ```
+
+1. Save the file and apply the new values:
+
+ ```shell
+ helm upgrade -f gitlab_values.yaml gitlab gitlab/gitlab
+ ```
-**Source configuration**
+:::TabTitle Docker
+
+1. Edit `docker-compose.yml`:
+
+ ```yaml
+ version: "3.6"
+ services:
+ gitlab:
+ environment:
+ GITLAB_OMNIBUS_CONFIG: |
+ gitlab_rails['ldap_servers'] = {
+ 'main' => {
+ 'group_base' => 'ou=groups,dc=example,dc=com',
+ 'admin_group' => 'my_admin_group',
+ }
+ }
+ ```
+
+1. Save the file and restart GitLab:
+
+ ```shell
+ docker compose up -d
+ ```
+
+:::TabTitle Self-compiled (source)
1. Edit `/home/git/gitlab/config/gitlab.yml`:
```yaml
- production:
+ production: &base
ldap:
servers:
main:
- # snip...
group_base: ou=groups,dc=example,dc=com
admin_group: my_admin_group
```
-1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect.
+1. Save the file and restart GitLab:
+
+ ```shell
+ # For systems running systemd
+ sudo systemctl restart gitlab.target
+
+ # For systems running SysV init
+ sudo service gitlab restart
+ ```
+
+::EndTabs
### Global group memberships lock
@@ -218,7 +404,9 @@ You can manually configure LDAP group sync times by setting the
following configuration values. The example below shows how to set group
sync to run once every two hours at the top of the hour.
-**Omnibus installations**
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
1. Edit `/etc/gitlab/gitlab.rb`:
@@ -226,56 +414,176 @@ sync to run once every two hours at the top of the hour.
gitlab_rails['ldap_group_sync_worker_cron'] = "0 */2 * * * *"
```
-1. [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+1. Save the file and reconfigure GitLab:
+
+ ```shell
+ sudo gitlab-ctl reconfigure
+ ```
+
+:::TabTitle Helm chart (Kubernetes)
+
+1. Export the Helm values:
+
+ ```shell
+ helm get values gitlab > gitlab_values.yaml
+ ```
+
+1. Edit `gitlab_values.yaml`:
+
+ ```yaml
+ global:
+ appConfig:
+ cron_jobs:
+ ldap_group_sync_worker:
+ cron: "*/30 * * * *"
+ ```
+
+1. Save the file and apply the new values:
+
+ ```shell
+ helm upgrade -f gitlab_values.yaml gitlab gitlab/gitlab
+ ```
+
+:::TabTitle Docker
+
+1. Edit `docker-compose.yml`:
+
+ ```yaml
+ version: "3.6"
+ services:
+ gitlab:
+ environment:
+ GITLAB_OMNIBUS_CONFIG: |
+ gitlab_rails['ldap_group_sync_worker_cron'] = "0 */2 * * * *"
+ ```
+
+1. Save the file and restart GitLab:
+
+ ```shell
+ docker compose up -d
+ ```
-**Source installations**
+:::TabTitle Self-compiled (source)
-1. Edit `config/gitlab.yaml`:
+1. Edit `/home/git/gitlab/config/gitlab.yml`:
```yaml
- cron_jobs:
- ldap_group_sync_worker_cron:
- "*/30 * * * *"
+ production: &base
+ ee_cron_jobs:
+ ldap_group_sync_worker:
+ cron: "*/30 * * * *"
```
-1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect.
+1. Save the file and restart GitLab:
+
+ ```shell
+ # For systems running systemd
+ sudo systemctl restart gitlab.target
+
+ # For systems running SysV init
+ sudo service gitlab restart
+ ```
+
+::EndTabs
### External groups
Using the `external_groups` setting allows you to mark all users belonging
-to these groups as [external users](../../../user/permissions.md#external-users).
+to these groups as [external users](../../../user/admin_area/external_users.md).
Group membership is checked periodically through the `LdapGroupSync` background
task.
-**Omnibus configuration**
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
gitlab_rails['ldap_servers'] = {
- 'main' => {
- # snip...
- 'external_groups' => ['interns', 'contractors'],
- }
+ 'main' => {
+ 'external_groups' => ['interns', 'contractors'],
+ }
}
```
-1. [Apply your changes to GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure).
+1. Save the file and reconfigure GitLab:
+
+ ```shell
+ sudo gitlab-ctl reconfigure
+ ```
-**Source configuration**
+:::TabTitle Helm chart (Kubernetes)
-1. Edit `config/gitlab.yaml`:
+1. Export the Helm values:
+
+ ```shell
+ helm get values gitlab > gitlab_values.yaml
+ ```
+
+1. Edit `gitlab_values.yaml`:
```yaml
- production:
+ global:
+ appConfig:
+ ldap:
+ servers:
+ main:
+ external_groups: ['interns', 'contractors']
+ ```
+
+1. Save the file and apply the new values:
+
+ ```shell
+ helm upgrade -f gitlab_values.yaml gitlab gitlab/gitlab
+ ```
+
+:::TabTitle Docker
+
+1. Edit `docker-compose.yml`:
+
+ ```yaml
+ version: "3.6"
+ services:
+ gitlab:
+ environment:
+ GITLAB_OMNIBUS_CONFIG: |
+ gitlab_rails['ldap_servers'] = {
+ 'main' => {
+ 'external_groups' => ['interns', 'contractors'],
+ }
+ }
+ ```
+
+1. Save the file and restart GitLab:
+
+ ```shell
+ docker compose up -d
+ ```
+
+:::TabTitle Self-compiled (source)
+
+1. Edit `/home/git/gitlab/config/gitlab.yml`:
+
+ ```yaml
+ production: &base
ldap:
servers:
main:
- # snip...
external_groups: ['interns', 'contractors']
```
-1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect.
+1. Save the file and restart GitLab:
+
+ ```shell
+ # For systems running systemd
+ sudo systemctl restart gitlab.target
+
+ # For systems running SysV init
+ sudo service gitlab restart
+ ```
+
+::EndTabs
### Group sync technical details
diff --git a/doc/administration/clusters/kas.md b/doc/administration/clusters/kas.md
index d7e1c9af1de..79dd69183a6 100644
--- a/doc/administration/clusters/kas.md
+++ b/doc/administration/clusters/kas.md
@@ -69,7 +69,7 @@ To enable the agent server on multiple nodes:
- `gitlab_kas['api_secret_key']` is the shared secret used for authentication between KAS and GitLab. This value must be Base64-encoded and exactly 32 bytes long.
- `gitlab_kas['private_api_secret_key']` is the shared secret used for authentication between different KAS instances. This value must be Base64-encoded and exactly 32 bytes long.
-1. For each application node, follow the steps in: [Use an external installation](../clusters/kas.md#use-an-external-installation).
+1. For each application node, follow the steps in [Use an external installation](../clusters/kas.md#use-an-external-installation). If the agent server is enabled on the application node, do not include `gitlab_kas['enable'] = false` in the configuration for that node.
1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
### For GitLab Helm Chart
diff --git a/doc/administration/configure.md b/doc/administration/configure.md
index 91fec753f7e..bf618345a94 100644
--- a/doc/administration/configure.md
+++ b/doc/administration/configure.md
@@ -16,7 +16,7 @@ Customize and configure your self-managed GitLab installation. Here are some qui
- [Packages](packages/index.md)
The following tables are intended to guide you to choose the right combination of capabilities based on your requirements. It is common to want the most
-available, quickly recoverable, highly performant and fully data resilient solution. However, there are tradeoffs.
+available, quickly recoverable, highly performant, and fully data resilient solution. However, there are tradeoffs.
The tables lists features on the left and provides their capabilities to the right along with known trade-offs.
@@ -24,9 +24,9 @@ The tables lists features on the left and provides their capabilities to the rig
| | Availability | Recoverability | Data Resiliency | Performance | Risks/Trade-offs|
|-|--------------|----------------|-----------------|-------------|-----------------|
-|Gitaly Cluster | Very high - tolerant of node failures | RTO for a single node of 10s with no manual intervention | Data is stored on multiple nodes | Good - While writes may take slightly longer due to voting, read distribution improves read speeds | **Trade-off** - Slight decrease in write speed for redundant, strongly-consistent storage solution. **Risks** - [Does not currently support snapshot backups](gitaly/index.md#snapshot-backup-and-recovery-limitations), GitLab backup task can be slow for large data sets |
+|Gitaly Cluster | Very high - tolerant of node failures | RTO for a single node of 10 s with no manual intervention | Data is stored on multiple nodes | Good - While writes may take slightly longer due to voting, read distribution improves read speeds | **Trade-off** - Slight decrease in write speed for redundant, strongly-consistent storage solution. **Risks** - [Does not support snapshot backups](gitaly/index.md#snapshot-backup-and-recovery-limitations), GitLab backup task can be slow for large data sets |
|Gitaly Shards | Single storage location is a single point of failure | Would need to restore only shards which failed | Single point of failure | Good - can allocate repositories to shards to spread load | **Trade-off** - Need to manually configure repositories into different shards to balance loads / storage space **Risks** - Single point of failure relies on recovery process when single-node failure occurs |
-|Gitaly + NFS | Single storage location is a single point of failure | Single node failure requires restoration from backup | Single point of failure | Average - NFS is not ideally suited to large quantities of small reads / writes which can have a detrimental impact on performance | **Trade-off** - Easy and familiar administration though NFS is not ideally suited to Git demands **Risks** - Many instances of NFS compatibility issues which provide very poor customer experiences |
+|Gitaly + NFS | Single storage location is a single point of failure | Single node failure requires restoration from backup | Single point of failure | Average - NFS is not ideally suited to large quantities of small reads / writes which can have a detrimental impact on performance | **Trade-off** - Familiar administration though NFS is not ideally suited to Git demands **Risks** - Many instances of NFS compatibility issues which provide very poor customer experiences |
## Geo Capabilities
@@ -34,7 +34,7 @@ If your availability needs to span multiple zones or multiple locations, read ab
| | Availability | Recoverability | Data Resiliency | Performance | Risks/Trade-offs|
|-|--------------|----------------|-----------------|-------------|-----------------|
-|Geo| Depends on the architecture of the Geo site. It is possible to deploy secondaries in single and multiple node configurations. | Eventually consistent. Recovery point depends on replication lag, which depends on a number of factors such as network speeds. Geo supports failover from a primary to secondary site using manual commands that are scriptable. | Geo currently replicates 100% of planned data types and verifies 50%. See [limitations table](geo/replication/datatypes.md#limitations-on-replicationverification) for more detail. | Improves read/clone times for users of a secondary. | Geo is not intended to replace other backup/restore solutions. Because of replication lag and the possibility of replicating bad data from a primary, we recommend that customers also take regular backups of their primary site and test the restore process. |
+|Geo| Depends on the architecture of the Geo site. It is possible to deploy secondaries in single and multiple node configurations. | Eventually consistent. Recovery point depends on replication lag, which depends on a number of factors such as network speeds. Geo supports failover from a primary to secondary site using manual commands that are scriptable. | Geo replicates 100% of planned data types and verifies 50%. See [limitations table](geo/replication/datatypes.md#limitations-on-replicationverification) for more detail. | Improves read/clone times for users of a secondary. | Geo is not intended to replace other backup/restore solutions. Because of replication lag and the possibility of replicating bad data from a primary, customers should also take regular backups of their primary site and test the restore process. |
## Scenarios for failure modes and available mitigation paths
diff --git a/doc/administration/external_pipeline_validation.md b/doc/administration/external_pipeline_validation.md
index e6f825f7cb8..2dec3857f75 100644
--- a/doc/administration/external_pipeline_validation.md
+++ b/doc/administration/external_pipeline_validation.md
@@ -9,9 +9,6 @@ type: reference, howto
You can use an external service to validate a pipeline before it's created.
-WARNING:
-This is an experimental feature and subject to change without notice.
-
GitLab sends a POST request to the external service URL with the pipeline
data as payload. The response code from the external service determines if GitLab
should accept or reject the pipeline. If the response is:
diff --git a/doc/administration/feature_flags.md b/doc/administration/feature_flags.md
index f2a40b60536..f7237b167e5 100644
--- a/doc/administration/feature_flags.md
+++ b/doc/administration/feature_flags.md
@@ -45,8 +45,7 @@ Features that are disabled by default may change or be removed without notice in
Data corruption, stability degradation, performance degradation, or security issues might occur if
you enable a feature that's disabled by default. Problems caused by using a default
-disabled feature aren't covered by GitLab support, unless you were directed by GitLab
-to enable the feature.
+disabled feature aren't covered by GitLab Support.
Security issues found in features that are disabled by default are patched in regular releases
and do not follow our regular [maintenance policy](../policy/maintenance.md#security-releases)
diff --git a/doc/administration/file_hooks.md b/doc/administration/file_hooks.md
index f8b55a7c680..b74ee22d584 100644
--- a/doc/administration/file_hooks.md
+++ b/doc/administration/file_hooks.md
@@ -1,6 +1,6 @@
---
-stage: Create
-group: Source Code
+stage: Manage
+group: Integrations
info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments"
type: reference
---
diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md
index dfa8d09e6ef..a78350d9dba 100644
--- a/doc/administration/geo/disaster_recovery/index.md
+++ b/doc/administration/geo/disaster_recovery/index.md
@@ -174,7 +174,7 @@ Use `gitlab-ctl geo promote` instead.
#### Promoting a **secondary** site with multiple nodes running GitLab 14.5 and later
-1. SSH to every Sidekiq, PostgresSQL, and Gitaly node in the **secondary** site and run one of the following commands:
+1. SSH to every Sidekiq, PostgreSQL, and Gitaly node in the **secondary** site and run one of the following commands:
- To promote the node on the secondary site to primary:
@@ -252,7 +252,7 @@ do this manually.
#### Promoting a **secondary** site with a Patroni standby cluster running GitLab 14.5 and later
-1. SSH to every Sidekiq, PostgresSQL, and Gitaly node in the **secondary** site and run one of the following commands:
+1. SSH to every Sidekiq, PostgreSQL, and Gitaly node in the **secondary** site and run one of the following commands:
- To promote the secondary site to primary:
@@ -364,7 +364,7 @@ with the **secondary** site:
sudo -u $PG_SUPERUSER $PG_CTL_BINARY -D $PG_DATA_DIRECTORY promote
```
-1. SSH to every Sidekiq, PostgresSQL, and Gitaly node in the **secondary** site and run one of the following commands:
+1. SSH to every Sidekiq, PostgreSQL, and Gitaly node in the **secondary** site and run one of the following commands:
- To promote the secondary site to primary:
diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md
index 80707afacca..e9eb154d398 100644
--- a/doc/administration/geo/disaster_recovery/planned_failover.md
+++ b/doc/administration/geo/disaster_recovery/planned_failover.md
@@ -217,7 +217,7 @@ GitLab 13.9 through GitLab 14.3 are affected by a bug in which the Geo secondary
- All replication meters reach 100% replicated, 0% failures.
- All verification meters reach 100% verified, 0% failures.
- - Database replication lag is 0ms.
+ - Database replication lag is 0 ms.
- The Geo log cursor is up to date (0 events behind).
1. On the **secondary** site:
diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md
index 8c18aaa944d..cef0101dd40 100644
--- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md
+++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md
@@ -72,7 +72,7 @@ On the **secondary** site:
1. On the left sidebar, select **Geo > Sites** to see its status.
Replicated objects (shown in green) should be close to 100%,
and there should be no failures (shown in red). If a large proportion of
- objects aren't yet replicated (shown in gray), consider giving the site more
+ objects aren't replicated (shown in gray), consider giving the site more
time to complete.
![Replication status](../../replication/img/geo_dashboard_v14_0.png)
@@ -162,7 +162,7 @@ follow these steps to avoid unnecessary data loss:
- All replication meters reach 100% replicated, 0% failures.
- All verification meters reach 100% verified, 0% failures.
- - Database replication lag is 0ms.
+ - Database replication lag is 0 ms.
- The Geo log cursor is up to date (0 events behind).
1. On the **secondary** site:
@@ -215,7 +215,7 @@ follow these steps to avoid unnecessary data loss:
`initctl stop gitlab-runsvvdir && echo 'manual' > /etc/init/gitlab-runsvdir.override && initctl reload-configuration`.
- If you do not have SSH access to the **primary** site, take the machine offline and
- prevent it from rebooting. Since there are many ways you may prefer to accomplish
+ prevent it from rebooting. As there are many ways you may prefer to accomplish
this, we avoid a single recommendation. You may have to:
- Reconfigure the load balancers.
@@ -228,7 +228,7 @@ follow these steps to avoid unnecessary data loss:
### Promoting the **secondary** site running GitLab 14.5 and later
-1. SSH to every Sidekiq, PostgresSQL, and Gitaly node in the **secondary** site and run one of the following commands:
+1. SSH to every Sidekiq, PostgreSQL, and Gitaly node in the **secondary** site and run one of the following commands:
- To promote the secondary site to primary:
@@ -276,7 +276,7 @@ WARNING:
If you encounter an `ActiveRecord::RecordInvalid: Validation failed: Name has already been taken` error during this process, read
[the troubleshooting advice](../../replication/troubleshooting.md#fixing-errors-during-a-failover-or-when-promoting-a-secondary-to-a-primary-site).
-The `gitlab-ctl promote-to-primary-node` command cannot be used yet in
+The `gitlab-ctl promote-to-primary-node` command cannot be used in
conjunction with multiple servers, as it can only
perform changes on a **secondary** with only a single machine. Instead, you must
do this manually.
diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md
index f9d99095951..085765ec51e 100644
--- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md
+++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md
@@ -147,7 +147,7 @@ follow these steps to avoid unnecessary data loss:
- All replication meters reach 100% replicated, 0% failures.
- All verification meters reach 100% verified, 0% failures.
- - Database replication lag is 0ms.
+ - Database replication lag is 0 ms.
- The Geo log cursor is up to date (0 events behind).
1. On the **secondary** site:
diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md
index 55c5d3784c2..d625b2a0324 100644
--- a/doc/administration/geo/replication/configuration.md
+++ b/doc/administration/geo/replication/configuration.md
@@ -264,7 +264,7 @@ Install the correct certificate based on your certificate type:
- **Multi-domain certificate** that includes both primary and secondary site domains: Install the certificate at `/etc/gitlab/ssl` on all **Rails, Sidekiq, and Gitaly** nodes in the **secondary** site.
- **Single-domain certificate** where the certificates are specific to each Geo site domain: Generate a valid certificate for your **secondary** site's domain and install it at `/etc/gitlab/ssl` per [these instructions](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) on all **Rails, Sidekiq, and Gitaly** nodes in the **secondary** site.
-#### Connecting to external services that use customer certificates
+#### Connecting to external services that use custom certificates
A copy of the self-signed certificate for the external service needs to be added to the trust store on all the **primary** site's nodes that require access to the service.
diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md
index 0198d2a63e8..022fe114a33 100644
--- a/doc/administration/geo/replication/datatypes.md
+++ b/doc/administration/geo/replication/datatypes.md
@@ -201,7 +201,7 @@ successfully, you must replicate their data using some other means.
|[CI job artifacts](../../../ci/pipelines/job_artifacts.md) | **Yes** (10.4) | **Yes** (14.10) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Verification is behind the feature flag `geo_job_artifact_replication`, enabled by default in 14.10. |
|[CI Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/pipeline_artifact.rb) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Persists additional artifacts after a pipeline completes. |
|[CI Secure Files](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/secure_file.rb) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [No](object_storage.md#verification-of-files-in-object-storage) | Verification is behind the feature flag `geo_ci_secure_file_replication`, enabled by default in 15.3. |
-|[Container Registry](../../packages/container_registry.md) | **Yes** (12.3)* | No | No | No | Replication is behind feature flag `geo_container_repository_replication`, enabled by default. Requires additional configuration. See [instructions](container_registry.md) to set up the Container Registry replication. |
+|[Container Registry](../../packages/container_registry.md) | **Yes** (12.3)* | No | No | No | See [instructions](container_registry.md) to set up the Container Registry replication. |
|[Infrastructure Registry](../../../user/packages/infrastructure_registry/index.md) | **Yes** (14.0) | **Yes** (14.0) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Behind feature flag `geo_package_file_replication`, enabled by default. |
|[Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | N/A | N/A | Designs also require replication of LFS objects and Uploads. |
|[Package Registry](../../../user/packages/package_registry/index.md) | **Yes** (13.2) | **Yes** (13.10) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Behind feature flag `geo_package_file_replication`, enabled by default. |
diff --git a/doc/administration/geo/replication/geo_validation_tests.md b/doc/administration/geo/replication/geo_validation_tests.md
index f09422d1e26..a12dd8d9d68 100644
--- a/doc/administration/geo/replication/geo_validation_tests.md
+++ b/doc/administration/geo/replication/geo_validation_tests.md
@@ -198,7 +198,7 @@ The following are additional validation tests we performed.
[Validate Object storage replication using Azure based object storage](https://gitlab.com/gitlab-org/gitlab/-/issues/348804#note_821294631):
-- Description: Tested the average time it takes for a single image to replicate from the primary object storage location to the secondary when using Azure based object storage replication and [GitLab based object storage replication](object_storage.md#enabling-gitlab-managed-object-storage-replication). This was tested by uploading a 1mb image to a project on the primary site every second for 60 seconds. The time was then measured until a image was available on the secondary site. This was achieved using a [Ruby Script](https://gitlab.com/gitlab-org/quality/geo-replication-tester).
+- Description: Tested the average time it takes for a single image to replicate from the primary object storage location to the secondary when using Azure based object storage replication and [GitLab based object storage replication](object_storage.md#enabling-gitlab-managed-object-storage-replication). This was tested by uploading a 1 MB image to a project on the primary site every second for 60 seconds. The time was then measured until a image was available on the secondary site. This was achieved using a [Ruby Script](https://gitlab.com/gitlab-org/quality/geo-replication-tester).
- Outcome: When using Azure based replication the average time for an image to replicate from the primary object storage to the secondary was recorded as 40 seconds, the longest replication time was 70 seconds and the quickest was 11 seconds. When using GitLab based replication the average time for replication to complete was 5 seconds, the longest replication time was 10 seconds and the quickest was 3 seconds.
- Follow up issue:
- [Validate Cross Region Object storage replication using Azure based object storage](https://gitlab.com/gitlab-org/gitlab/-/issues/358154)
@@ -207,12 +207,12 @@ The following are additional validation tests we performed.
[Validate Object storage replication using AWS based object storage](https://gitlab.com/gitlab-org/gitlab/-/issues/351463):
-- Description: Tested the average time it takes for a single image to replicate from the primary object storage location to the secondary when using AWS based object storage replication and [GitLab based object storage replication](object_storage.md#enabling-gitlab-managed-object-storage-replication). This was tested by uploading a 1mb image to a project on the primary site every second for 60 seconds. The time was then measured until a image was available on the secondary site. This was achieved using a [Ruby Script](https://gitlab.com/gitlab-org/quality/geo-replication-tester).
-- Outcome: When using AWS managed replication the average time for an image to replicate between sites is about 49 seconds, this is true for when sites are located within the same region and when they are further apart (Europe to America). When using Geo managed replication within the same region the average time for replication took just 5 seconds, however when replicating cross region the average time rose to 33 seconds.
+- Description: Tested the average time it takes for a single image to replicate from the primary object storage location to the secondary when using AWS based object storage replication and [GitLab based object storage replication](object_storage.md#enabling-gitlab-managed-object-storage-replication). This was tested by uploading a 1 MB image to a project on the primary site every second for 60 seconds. The time was then measured until a image was available on the secondary site. This was achieved using a [Ruby Script](https://gitlab.com/gitlab-org/quality/geo-replication-tester).
+- Outcome: When using AWS managed replication the average time for an image to replicate between sites is about 49 seconds, this is true for when sites are located in the same region and when they are further apart (Europe to America). When using Geo managed replication in the same region the average time for replication took just 5 seconds, however when replicating cross region the average time rose to 33 seconds.
[Validate Object storage replication using GCP based object storage](https://gitlab.com/gitlab-org/gitlab/-/issues/351464):
-- Description: Tested the average time it takes for a single image to replicate from the primary object storage location to the secondary when using GCP based object storage replication and [GitLab based object storage replication](object_storage.md#enabling-gitlab-managed-object-storage-replication). This was tested by uploading a 1mb image to a project on the primary site every second for 60 seconds. The time was then measured until a image was available on the secondary site. This was achieved using a [Ruby Script](https://gitlab.com/gitlab-org/quality/geo-replication-tester).
+- Description: Tested the average time it takes for a single image to replicate from the primary object storage location to the secondary when using GCP based object storage replication and [GitLab based object storage replication](object_storage.md#enabling-gitlab-managed-object-storage-replication). This was tested by uploading a 1 MB image to a project on the primary site every second for 60 seconds. The time was then measured until a image was available on the secondary site. This was achieved using a [Ruby Script](https://gitlab.com/gitlab-org/quality/geo-replication-tester).
- Outcome: GCP handles replication differently than other Cloud Providers. In GCP, the process is to a create single bucket that is either multi, dual, or single region based. This means that the bucket automatically stores replicas in a region based on the option chosen. Even when using multi region, this only replicates in a single continent, the options being America, Europe, or Asia. At current there doesn't seem to be any way to replicate objects between continents using GCP based replication. For Geo managed replication the average time when replicating in the same region was 6 seconds, and when replicating cross region this rose to just 9 seconds.
## Other tests
diff --git a/doc/administration/geo/replication/location_aware_git_url.md b/doc/administration/geo/replication/location_aware_git_url.md
index dbe543f5a62..460de5f3232 100644
--- a/doc/administration/geo/replication/location_aware_git_url.md
+++ b/doc/administration/geo/replication/location_aware_git_url.md
@@ -104,7 +104,7 @@ on the external URL of the current host. For example:
You can customize the:
-- SSH remote URL to use the location-aware `git.example.com`. To do so, change the SSH remote URL's
+- SSH remote URL to use the location-aware `git.example.com`. To do so, change the SSH remote URL
host by setting `gitlab_rails['gitlab_ssh_host']` in `gitlab.rb` of web nodes.
- HTTP remote URL as shown in
[Custom Git clone URL for HTTP(S)](../../../user/admin_area/settings/visibility_and_access_controls.md#customize-git-clone-url-for-https).
diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md
index fa668091c90..1dcced781ce 100644
--- a/doc/administration/geo/replication/troubleshooting.md
+++ b/doc/administration/geo/replication/troubleshooting.md
@@ -49,6 +49,8 @@ health check manually to get this information and a few more details.
#### Health check Rake task
+> The use of a custom NTP server was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/105514) in GitLab 15.7.
+
This Rake task can be run on a **Rails** node in the **primary** or **secondary**
Geo sites:
@@ -80,6 +82,22 @@ All projects are in hashed storage? ... yes
Checking Geo ... Finished
```
+You can also specify a custom NTP server using environment variables. For example:
+
+```shell
+export NTP_HOST="ntp.ubuntu.com"
+export NTP_TIMEOUT="30"
+sudo gitlab-rake gitlab:geo:check
+```
+
+The following environment variables are supported.
+
+| Variable | Description | Default value |
+| ----------- | ----------- | ------------- |
+|`NTP_HOST` | The NTP host. | `pool.ntp.org` |
+|`NTP_PORT` | The NTP port the host listens on. |`ntp`|
+|`NTP_TIMEOUT`| The NTP timeout in seconds. | The value defined in the `net-ntp` Ruby library ([60 seconds](https://github.com/zencoder/net-ntp/blob/3d0990214f439a5127782e0f50faeaf2c8ca7023/lib/net/ntp/ntp.rb#L6)). |
+
#### Sync status Rake task
Current sync information can be found manually by running this Rake task on any
@@ -204,7 +222,7 @@ Commands that change data can cause damage if not run correctly or under the rig
```
1. This will cause the primary to start checksumming all Uploads.
-1. When a primary successfully checksums a record, then all secondaries rechecksum as well, and they compare the values.
+1. When a primary successfully checksums a record, then all secondaries recalculate the checksum as well, and they compare the values.
A similar thing can be done for all Models handled by the [Geo Self-Service Framework](../../../development/geo/framework.md) which have implemented verification:
@@ -1335,7 +1353,7 @@ status
```
1. This will cause the primary to start checksumming all Uploads.
-1. When a primary successfully checksums a record, then all secondaries rechecksum as well, and they compare the values.
+1. When a primary successfully checksums a record, then all secondaries recalculate the checksum as well, and they compare the values.
For other SSF data types replace `Upload` in the command above with the desired model class.
@@ -1476,7 +1494,7 @@ Failed to contact primary https://primary.domain.com/namespace/push_test.git\\nE
The partial failover to a secondary Geo *site* may be the result of a temporary/transient issue. Therefore, first attempt to run the promote command again.
-1. SSH into every Sidekiq, PostgresSQL, Gitaly, and Rails node in the **secondary** site and run one of the following commands:
+1. SSH into every Sidekiq, PostgreSQL, Gitaly, and Rails node in the **secondary** site and run one of the following commands:
- To promote the secondary site to primary:
@@ -1495,7 +1513,7 @@ The partial failover to a secondary Geo *site* may be the result of a temporary/
If the above steps are **not successful**, proceed through the next steps:
-1. SSH to every Sidekiq, PostgresSQL, Gitaly and Rails node in the **secondary** site and perform the following operations:
+1. SSH to every Sidekiq, PostgreSQL, Gitaly and Rails node in the **secondary** site and perform the following operations:
- Create a `/etc/gitlab/gitlab-cluster.json` file with the following content:
diff --git a/doc/administration/geo/replication/tuning.md b/doc/administration/geo/replication/tuning.md
index ab9263ad344..4dc3ba93d66 100644
--- a/doc/administration/geo/replication/tuning.md
+++ b/doc/administration/geo/replication/tuning.md
@@ -30,7 +30,7 @@ However, this may not lead to more downloads in parallel unless the number of
available Sidekiq threads is also increased. For example, if repository synchronization
concurrency is increased from 25 to 50, you may also want to increase the number
of Sidekiq threads from 25 to 50. See the
-[Sidekiq concurrency documentation](../../sidekiq/extra_sidekiq_processes.md#number-of-threads)
+[Sidekiq concurrency documentation](../../sidekiq/extra_sidekiq_processes.md#concurrency)
for more details.
## Repository re-verification
diff --git a/doc/administration/geo/replication/version_specific_upgrades.md b/doc/administration/geo/replication/version_specific_upgrades.md
index 9aad5cdeaa7..d981656f748 100644
--- a/doc/administration/geo/replication/version_specific_upgrades.md
+++ b/doc/administration/geo/replication/version_specific_upgrades.md
@@ -39,10 +39,6 @@ results in a loop that consistently fails for all objects stored in object stora
For information on how to fix this, see
[Troubleshooting - Failed syncs with GitLab-managed object storage replication](troubleshooting.md#failed-syncs-with-gitlab-managed-object-storage-replication).
-## Upgrading to 14.6
-
-[Geo proxying](../secondary_proxy/index.md) was [enabled by default for unified URLs](https://gitlab.com/gitlab-org/gitlab/-/issues/325732) in 14.6. This may be a breaking change. If needed, you may [disable Geo proxying](../secondary_proxy/index.md#disable-geo-proxying).
-
## Upgrading to 14.4
There is [an issue in GitLab 14.4.0 through 14.4.2](../../../update/index.md#1440) that can affect Geo and other features that rely on cronjobs. We recommend upgrading to GitLab 14.4.3 or later.
@@ -144,7 +140,7 @@ GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab
## Upgrading to GitLab 13.9
-### Error during zero-downtime upgrade: "cannot drop column asset_proxy_whitelist"
+### Error during zero-downtime upgrade: `cannot drop column asset_proxy_whitelist`
We've detected an issue [with a column rename](https://gitlab.com/gitlab-org/gitlab/-/issues/324160)
that prevents upgrades to GitLab 13.9.0, 13.9.1, 13.9.2 and 13.9.3 when following the zero-downtime steps. It is necessary
diff --git a/doc/administration/geo/secondary_proxy/index.md b/doc/administration/geo/secondary_proxy/index.md
index 2786982bb51..ac8b88a91d5 100644
--- a/doc/administration/geo/secondary_proxy/index.md
+++ b/doc/administration/geo/secondary_proxy/index.md
@@ -70,39 +70,6 @@ a single URL used by all Geo sites, including the primary.
In Kubernetes, you can use the same domain under `global.hosts.domain` as for the primary site.
-## Disable Geo proxying
-
-You can disable the secondary proxying on each Geo site, separately, by following these steps with Omnibus-based packages:
-
-1. SSH into each application node (serving user traffic directly) on your secondary Geo site
- and add the following environment variable:
-
- ```shell
- sudo editor /etc/gitlab/gitlab.rb
- ```
-
- ```ruby
- gitlab_workhorse['env'] = {
- "GEO_SECONDARY_PROXY" => "0"
- }
- ```
-
-1. Reconfigure the updated nodes for the change to take effect:
-
- ```shell
- gitlab-ctl reconfigure
- ```
-
-In Kubernetes, you can use `--set gitlab.webservice.extraEnv.GEO_SECONDARY_PROXY="0"`,
-or specify the following in your values file:
-
-```yaml
-gitlab:
- webservice:
- extraEnv:
- GEO_SECONDARY_PROXY: "0"
-```
-
## Geo proxying with Separate URLs
> Geo secondary proxying for separate URLs is [enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/346112) in GitLab 15.1.
@@ -116,11 +83,18 @@ You can also add feedback in the epic about any use-cases that
are not possible anymore with proxying enabled.
If you run into issues, to disable this feature, disable the `geo_secondary_proxy_separate_urls` feature flag.
-SSH into one node running Rails on your primary Geo site and run:
-```shell
-sudo gitlab-rails runner "Feature.disable(:geo_secondary_proxy_separate_urls)"
-```
+1. SSH into one node running Rails on your primary Geo site and run:
+
+ ```shell
+ sudo gitlab-rails runner "Feature.disable(:geo_secondary_proxy_separate_urls)"
+ ```
+
+1. Restart Puma on all of the nodes running Rails on your secondary Geo site:
+
+ ```shell
+ sudo gitlab-ctl restart puma
+ ```
In Kubernetes, you can run the same command in the toolbox pod. Refer to the
[Kubernetes cheat sheet](https://docs.gitlab.com/charts/troubleshooting/kubernetes_cheat_sheet.html#gitlab-specific-kubernetes-information)
@@ -192,3 +166,42 @@ It does not cover all data types, more will be added in the future as they are t
1. Git reads are served from the local secondary while pushes get proxied to the primary.
Selective sync or cases where repositories don't exist locally on the Geo secondary throw a "not found" error.
1. Pages can use the same URL (without access control), but must be configured separately and are not proxied.
+
+## Disable Geo proxying
+
+Secondary proxying is enabled by default on a secondary site when it uses a unified URL, meaning, the same `external_url` as the primary site. Disabling proxying in this case tends to not be helpful due to completely different behavior being served at the same URL, depending on routing.
+
+Secondary proxying is enabled by default in GitLab 15.1 on a secondary site even without a unified URL. If proxying needs to be disabled on a secondary site, it is much easier to disable the feature flag in [Geo proxying with Separate URLs](#geo-proxying-with-separate-urls). However, if there are multiple secondary sites, then the instructions in this section can be used to disable secondary proxying per site.
+
+Additionally, the `gitlab-workhorse` service polls `/api/v4/geo/proxy` every 10 seconds. In GitLab 15.2 and later, it is only polled once, if Geo is not enabled. Prior to GitLab 15.2, you can stop this polling by disabling secondary proxying.
+
+You can disable the secondary proxying on each Geo site, separately, by following these steps with Omnibus-based packages:
+
+1. SSH into each application node (serving user traffic directly) on your secondary Geo site
+ and add the following environment variable:
+
+ ```shell
+ sudo editor /etc/gitlab/gitlab.rb
+ ```
+
+ ```ruby
+ gitlab_workhorse['env'] = {
+ "GEO_SECONDARY_PROXY" => "0"
+ }
+ ```
+
+1. Reconfigure the updated nodes for the change to take effect:
+
+ ```shell
+ gitlab-ctl reconfigure
+ ```
+
+In Kubernetes, you can use `--set gitlab.webservice.extraEnv.GEO_SECONDARY_PROXY="0"`,
+or specify the following in your values file:
+
+```yaml
+gitlab:
+ webservice:
+ extraEnv:
+ GEO_SECONDARY_PROXY: "0"
+```
diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md
index 86caf5306b5..99f7b32be59 100644
--- a/doc/administration/geo/setup/database.md
+++ b/doc/administration/geo/setup/database.md
@@ -35,23 +35,23 @@ or trying to evaluate Geo for a future clusterized installation.
A single instance can be expanded to a clusterized version using Patroni, which is recommended for a
highly available architecture.
-Follow below the instructions on how to set up PostgreSQL replication as a single instance database.
+Follow the instructions below on how to set up PostgreSQL replication as a single instance database.
Alternatively, you can look at the [Multi-node database replication](#multi-node-database-replication)
instructions on setting up replication with a Patroni cluster.
### PostgreSQL replication
The GitLab **primary** site where the write operations happen connects to
-the **primary** database server, and **secondary** sites
+the **primary** database server. **Secondary** sites
connect to their own database servers (which are read-only).
-We recommend using [PostgreSQL replication slots](https://medium.com/@tk512/replication-slots-in-postgresql-b4b03d277c75)
+You should use [PostgreSQL's replication slots](https://medium.com/@tk512/replication-slots-in-postgresql-b4b03d277c75)
to ensure that the **primary** site retains all the data necessary for the **secondary** sites to
recover. See below for more details.
The following guide assumes that:
-- You are using Omnibus and therefore you are using PostgreSQL 12 or later
+- You are using Omnibus and therefore you are using PostgreSQL 12 or later,
which includes the [`pg_basebackup` tool](https://www.postgresql.org/docs/12/app-pgbasebackup.html).
- You have a **primary** site already set up (the GitLab server you are
replicating from), running Omnibus' PostgreSQL (or equivalent version), and
@@ -120,8 +120,8 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
1. Define a password for the database [replication user](https://wiki.postgresql.org/wiki/Streaming_Replication).
- We will use the username defined in `/etc/gitlab/gitlab.rb` under the `postgresql['sql_replication_user']`
- setting. The default value is `gitlab_replicator`, but if you changed it to something else, adapt
+ Use the username defined in `/etc/gitlab/gitlab.rb` under the `postgresql['sql_replication_user']`
+ setting. The default value is `gitlab_replicator`. If you changed the username to something else, adapt
the instructions below.
Generate a MD5 hash of the desired password:
@@ -141,7 +141,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
```
If you are using an external database not managed by Omnibus GitLab, you need
- to create the replicator user and define a password to it manually:
+ to create the `gitlab_replicator` user and define a password for that user manually:
```sql
--- Create a new user 'replicator'
@@ -155,16 +155,16 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
For security reasons, PostgreSQL does not listen on any network interfaces
by default. However, Geo requires the **secondary** site to be able to
- connect to the **primary** site's database. For this reason, we need the IP address of
+ connect to the **primary** site's database. For this reason, you need the IP address of
each site.
NOTE:
For external PostgreSQL instances, see [additional instructions](external_database.md).
- If you are using a cloud provider, you can lookup the addresses for each
+ If you are using a cloud provider, you can look up the addresses for each
Geo site through your cloud provider's management console.
- To lookup the address of a Geo site, SSH in to the Geo site and execute:
+ To look up the address of a Geo site, SSH into the Geo site and execute:
```shell
##
@@ -187,7 +187,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
| `postgresql['md5_auth_cidr_addresses']` | **Primary** and **Secondary** sites' public or VPC private addresses. |
If you are using Google Cloud Platform, SoftLayer, or any other vendor that
- provides a virtual private cloud (VPC) you can use the **primary** and **secondary** sites
+ provides a virtual private cloud (VPC), you can use the **primary** and **secondary** sites'
private addresses (corresponds to "internal address" for Google Cloud Platform) for
`postgresql['md5_auth_cidr_addresses']` and `postgresql['listen_address']`.
@@ -196,14 +196,14 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
for more details.
NOTE:
- If you need to use `0.0.0.0` or `*` as the listen_address, you also must add
+ If you need to use `0.0.0.0` or `*` as the `listen_address`, you also must add
`127.0.0.1/32` to the `postgresql['md5_auth_cidr_addresses']` setting, to allow Rails to connect through
`127.0.0.1`. For more information, see [omnibus-5258](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5258).
- Depending on your network configuration, the suggested addresses may not
- be correct. If your **primary** site and **secondary** sites connect over a local
+ Depending on your network configuration, the suggested addresses may
+ be incorrect. If your **primary** site and **secondary** sites connect over a local
area network, or a virtual network connecting availability zones like
- [Amazon's VPC](https://aws.amazon.com/vpc/) or [Google's VPC](https://cloud.google.com/vpc/)
+ [Amazon's VPC](https://aws.amazon.com/vpc/) or [Google's VPC](https://cloud.google.com/vpc/),
you should use the **secondary** site's private address for `postgresql['md5_auth_cidr_addresses']`.
Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP
@@ -286,12 +286,12 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
```
1. Now that the PostgreSQL server is set up to accept remote connections, run
- `netstat -plnt | grep 5432` to make sure that PostgreSQL is listening on port
+ `netstat -plnt | grep 5432` to ensure that PostgreSQL is listening on port
`5432` to the **primary** site's private address.
1. A certificate was automatically generated when GitLab was reconfigured. This
is used automatically to protect your PostgreSQL traffic from
- eavesdroppers, but to protect against active ("man-in-the-middle") attackers,
+ eavesdroppers. To protect against active ("man-in-the-middle") attackers,
the **secondary** site needs a copy of the certificate. Make a copy of the PostgreSQL
`server.crt` file on the **primary** site by running this command:
@@ -299,26 +299,26 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
cat ~gitlab-psql/data/server.crt
```
- Copy the output into a clipboard or into a local file. You
+ Copy the output to the clipboard or into a local file. You
need it when setting up the **secondary** site! The certificate is not sensitive
data.
However, this certificate is created with a generic `PostgreSQL` Common Name. For this,
you must use the `verify-ca` mode when replicating the database, otherwise,
- the hostname mismatch will cause errors.
+ the hostname mismatch causes errors.
1. Optional. Generate your own SSL certificate and manually
[configure SSL for PostgreSQL](https://docs.gitlab.com/omnibus/settings/database.html#configuring-ssl),
instead of using the generated certificate.
- You will need at least the SSL certificate and key, and set the `postgresql['ssl_cert_file']` and
+ You need at least the SSL certificate and key. Set the `postgresql['ssl_cert_file']` and
`postgresql['ssl_key_file']` values to their full paths, as per the Database SSL docs.
This allows you to use the `verify-full` SSL mode when replicating the database
and get the extra benefit of verifying the full hostname in the CN.
You can use this certificate (that you have also set in `postgresql['ssl_cert_file']`) instead
- of the certificate from the point above going forward. This will allow you to use `verify-full`
+ of the certificate from the point above going forward. This allows you to use `verify-full`
without replication errors if the CN matches.
#### Step 2. Configure the **secondary** server
@@ -337,7 +337,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
```
NOTE:
- This step is important so we don't try to execute anything before the site is fully configured.
+ This step is important so you don't try to execute anything before the site is fully configured.
1. [Check TCP connectivity](../../raketasks/maintenance.md) to the **primary** site's PostgreSQL server:
@@ -348,7 +348,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
NOTE:
If this step fails, you may be using the wrong IP address, or a firewall may
be preventing access to the site. Check the IP address, paying close
- attention to the difference between public and private addresses and ensure
+ attention to the difference between public and private addresses. Ensure
that, if a firewall is present, the **secondary** site is permitted to connect to the
**primary** site on port 5432.
@@ -389,14 +389,14 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
```
NOTE:
- If you are using manually generated certificates and plan on using
- `sslmode=verify-full` to benefit of the full hostname verification,
- make sure to replace `verify-ca` to `verify-full` when
+ If you are using manually generated certificates and want to use
+ `sslmode=verify-full` to benefit from the full hostname verification,
+ replace `verify-ca` with `verify-full` when
running the command.
- When prompted enter the _plaintext_ password you set in the first step for the
+ When prompted, enter the _plaintext_ password you set in the first step for the
`gitlab_replicator` user. If all worked correctly, you should see
- the list of **primary** site's databases.
+ the list of the **primary** site's databases.
A failure to connect here indicates that the TLS configuration is incorrect.
Ensure that the contents of `~gitlab-psql/data/server.crt` on the **primary** site
@@ -404,8 +404,8 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
1. Configure PostgreSQL:
- This step is similar to how we configured the **primary** instance.
- We must enable this, even if using a single node.
+ This step is similar to how you configured the **primary** instance.
+ You must enable this, even if using a single node.
Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP
addresses with addresses appropriate to your network configuration:
@@ -450,12 +450,12 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
#### Step 3. Initiate the replication process
-Below we provide a script that connects the database on the **secondary** site to
-the database on the **primary** site, replicates the database, and creates the
+Below is a script that connects the database on the **secondary** site to
+the database on the **primary** site. This script replicates the database and creates the
needed files for streaming replication.
The directories used are the defaults that are set up in Omnibus. If you have
-changed any defaults, configure it as you see fit replacing the directories and paths.
+changed any defaults, configure the script accordingly, replacing any directories and paths.
WARNING:
Make sure to run this on the **secondary** site as it removes all PostgreSQL's
@@ -469,7 +469,7 @@ data before running `pg_basebackup`.
1. Choose a database-friendly name to use for your **secondary** site to
use as the replication slot name. For example, if your domain is
- `secondary.geo.example.com`, you may use `secondary_example` as the slot
+ `secondary.geo.example.com`, use `secondary_example` as the slot
name as shown in the commands below.
1. Execute the command below to start a backup/restore and begin the replication
@@ -492,33 +492,36 @@ data before running `pg_basebackup`.
```
NOTE:
- If you have generated custom PostgreSQL certificates, you will want to use
+ If you have generated custom PostgreSQL certificates, you need to use
`--sslmode=verify-full` (or omit the `sslmode` line entirely), to benefit from the extra
validation of the full host name in the certificate CN / SAN for additional security.
- Otherwise, using the automatically created certificate with `verify-full` will fail,
- as it has a generic `PostgreSQL` CN which will not match the `--host` value in this command.
+ Otherwise, using the automatically created certificate with `verify-full` fails,
+ as it has a generic `PostgreSQL` CN which doesn't match the `--host` value in this command.
This command also takes a number of additional options. You can use `--help`
- to list them all, but here are a couple of tips:
+ to list them all, but here are some tips:
- - If PostgreSQL is listening on a non-standard port, add `--port=` as well.
+ - If PostgreSQL is listening on a non-standard port, add `--port=`.
- If your database is too large to be transferred in 30 minutes, you need
- to increase the timeout, for example, `--backup-timeout=3600` if you expect the
+ to increase the timeout. For example, use `--backup-timeout=3600` if you expect the
initial replication to take under an hour.
- Pass `--sslmode=disable` to skip PostgreSQL TLS authentication altogether
(for example, you know the network path is secure, or you are using a site-to-site
VPN). It is **not** safe over the public Internet!
- You can read more details about each `sslmode` in the
- [PostgreSQL documentation](https://www.postgresql.org/docs/12/libpq-ssl.html#LIBPQ-SSL-PROTECTION);
- the instructions above are carefully written to ensure protection against
+ [PostgreSQL documentation](https://www.postgresql.org/docs/12/libpq-ssl.html#LIBPQ-SSL-PROTECTION).
+ The instructions above are carefully written to ensure protection against
both passive eavesdroppers and active "man-in-the-middle" attackers.
- Change the `--slot-name` to the name of the replication slot
to be used on the **primary** database. The script attempts to create the
replication slot automatically if it does not exist.
- If you're repurposing an old site into a Geo **secondary** site, you must
add `--force` to the command line.
- - When not in a production machine you can disable backup step if you
- really sure this is what you want by adding `--skip-backup`
+ - When not in a production machine, you can disable the backup step (if you
+ are certain this is what you want) by adding `--skip-backup`.
+ - If you are using PgBouncer, you need to target the database host directly.
+ - If you are using Patroni on your primary site, you must target the current leader host.
+ - If you are using a load balancer proxy (for example HAProxy) and it is targeting the Patroni leader for the primary, you should target the load balancer proxy instead.
The replication process is now complete.
@@ -528,9 +531,9 @@ The replication process is now complete.
PostgreSQL connections, which can improve performance even when using in a
single instance installation.
-We recommend using PgBouncer if you use GitLab in a highly available
+You should use PgBouncer if you use GitLab in a highly available
configuration with a cluster of nodes supporting a Geo **primary** site and
-two other clusters of nodes supporting a Geo **secondary** site. One for the
+two other clusters of nodes supporting a Geo **secondary** site. You need two PgBouncer nodes: one for the
main database and the other for the tracking database. For more information,
see [High Availability with Omnibus GitLab](../../postgresql/replication_and_failover.md).
@@ -542,7 +545,7 @@ when using Omnibus-managed PostgreSQL instances:
On the GitLab Geo **primary** site:
1. The default value for the replication user is `gitlab_replicator`, but if you've set a custom replication
- user in your `/etc/gitlab/gitlab.rb` under the `postgresql['sql_replication_user']` setting, make sure to
+ user in your `/etc/gitlab/gitlab.rb` under the `postgresql['sql_replication_user']` setting, ensure you
adapt the following instructions for your own user.
Generate an MD5 hash of the desired password:
@@ -574,7 +577,7 @@ On the GitLab Geo **primary** site:
```
Until the password is updated on any **secondary** sites, the [PostgreSQL log](../../logs/index.md#postgresql-logs) on
-the secondaries will report the following error message:
+the secondaries report the following error message:
```console
FATAL: could not connect to the primary server: FATAL: password authentication failed for user "gitlab_replicator"
@@ -616,16 +619,16 @@ If you still haven't [migrated from repmgr to Patroni](#migrating-from-repmgr-to
### Migrating from repmgr to Patroni
-1. Before migrating, we recommend that there is no replication lag between the **primary** and **secondary** sites and that replication is paused. In GitLab 13.2 and later, you can pause and resume replication with `gitlab-ctl geo-replication-pause` and `gitlab-ctl geo-replication-resume` on a Geo secondary database node.
+1. Before migrating, you should ensure there is no replication lag between the **primary** and **secondary** sites and that replication is paused. In GitLab 13.2 and later, you can pause and resume replication with `gitlab-ctl geo-replication-pause` and `gitlab-ctl geo-replication-resume` on a Geo secondary database node.
1. Follow the [instructions to migrate repmgr to Patroni](../../postgresql/replication_and_failover.md#switching-from-repmgr-to-patroni). When configuring Patroni on each **primary** site database node, add `patroni['replication_slots'] = { '<slot_name>' => 'physical' }`
-to `gitlab.rb` where `<slot_name>` is the name of the replication slot for your **secondary** site. This ensures that Patroni recognizes the replication slot as permanent and not drop it upon restarting.
-1. If database replication to the **secondary** site was paused before migration, resume replication after Patroni is confirmed working on the **primary** site.
+to `gitlab.rb` where `<slot_name>` is the name of the replication slot for your **secondary** site. This ensures that Patroni recognizes the replication slot as permanent and doesn't drop it upon restarting.
+1. If database replication to the **secondary** site was paused before migration, resume replication after Patroni is confirmed as working on the **primary** site.
### Migrating a single PostgreSQL node to Patroni
Before the introduction of Patroni, Geo had no Omnibus support for HA setups on the **secondary** site.
-With Patroni it's now possible to support that. To migrate the existing PostgreSQL to Patroni:
+With Patroni, this support is now possible. To migrate the existing PostgreSQL to Patroni:
1. Make sure you have a Consul cluster setup on the secondary (similar to how you set it up on the **primary** site).
1. [Configure a permanent replication slot](#step-1-configure-patroni-permanent-replication-slot-on-the-primary-site).
@@ -634,23 +637,23 @@ With Patroni it's now possible to support that. To migrate the existing PostgreS
1. [Configure a Standby Cluster](#step-4-configure-a-standby-cluster-on-the-secondary-site)
on that single node machine.
-You end up with a “Standby Cluster” with a single node. That allows you to later on add additional Patroni nodes by following the same instructions above.
+You end up with a “Standby Cluster” with a single node. That allows you to add additional Patroni nodes by following the same instructions above.
### Patroni support
-Patroni is the official replication management solution for Geo. It
+Patroni is the official replication management solution for Geo. Patroni
can be used to build a highly available cluster on the **primary** and a **secondary** Geo site.
-Using Patroni on a **secondary** site is optional and you don't have to use the same amount of
+Using Patroni on a **secondary** site is optional and you don't have to use the same number of
nodes on each Geo site.
-For instructions about how to set up Patroni on the primary site, see the
+For instructions on how to set up Patroni on the primary site, see the
[PostgreSQL replication and failover with Omnibus GitLab](../../postgresql/replication_and_failover.md#patroni) page.
#### Configuring Patroni cluster for a Geo secondary site
In a Geo secondary site, the main PostgreSQL database is a read-only replica of the primary site's PostgreSQL database.
-If you are currently using `repmgr` on your Geo primary site, see [these instructions](#migrating-from-repmgr-to-patroni)
+If you are using `repmgr` on your Geo primary site, see [these instructions](#migrating-from-repmgr-to-patroni)
for migrating from `repmgr` to Patroni.
A production-ready and secure setup requires at least:
@@ -661,14 +664,14 @@ A production-ready and secure setup requires at least:
- 1 internal load-balancer _(primary site only)_
The internal load balancer provides a single endpoint for connecting to the Patroni cluster's leader whenever a new leader is
-elected, and it is required for enabling cascading replication from the secondary sites.
+elected. The load balancer is required for enabling cascading replication from the secondary sites.
Be sure to use [password credentials](../../postgresql/replication_and_failover.md#database-authorization-for-patroni)
and other database best practices.
##### Step 1. Configure Patroni permanent replication slot on the primary site
-To set up database replication with Patroni on a secondary site, we must
+To set up database replication with Patroni on a secondary site, you must
configure a _permanent replication slot_ on the primary site's Patroni cluster,
and ensure password authentication is used.
@@ -734,8 +737,8 @@ Leader instance**:
##### Step 2. Configure the internal load balancer on the primary site
To avoid reconfiguring the Standby Leader on the secondary site whenever a new
-Leader is elected on the primary site, we must set up a TCP internal load
-balancer which gives a single endpoint for connecting to the Patroni
+Leader is elected on the primary site, you should set up a TCP internal load
+balancer. This load balancer provides a single endpoint for connecting to the Patroni
cluster's Leader.
The Omnibus GitLab packages do not include a Load Balancer. Here's how you
@@ -773,14 +776,14 @@ backend postgresql
server patroni3.internal 10.6.0.23:5432 maxconn 100 check port 8008
```
-Refer to your preferred Load Balancer's documentation for further guidance.
+For further guidance, refer to the documentation for your preferred load balancer.
##### Step 3. Configure PgBouncer nodes on the secondary site
A production-ready and highly available configuration requires at least
-three Consul nodes, a minimum of one PgBouncer node, but it's recommended to have
-one per database node. An internal load balancer (TCP) is required when there is
-more than one PgBouncer service nodes. The internal load balancer provides a single
+three Consul nodes and a minimum of one PgBouncer node. However, it is recommended to have
+one PgBouncer node per database node. An internal load balancer (TCP) is required when there is
+more than one PgBouncer service node. The internal load balancer provides a single
endpoint for connecting to the PgBouncer cluster. For more information,
see [High Availability with Omnibus GitLab](../../postgresql/replication_and_failover.md).
@@ -841,7 +844,7 @@ On each node running a PgBouncer instance on the **secondary** site:
NOTE:
If you are converting a secondary site with a single PostgreSQL instance to a Patroni Cluster, you must start on the PostgreSQL instance. It becomes the Patroni Standby Leader instance,
-and then you can switch over to another replica if you need.
+and then you can switch over to another replica if you need to.
For each node running a Patroni instance on the secondary site:
@@ -895,7 +898,7 @@ For each node running a Patroni instance on the secondary site:
```
1. Reconfigure GitLab for the changes to take effect.
- This is required to bootstrap PostgreSQL users and settings.
+ This step is required to bootstrap PostgreSQL users and settings.
- If this is a fresh installation of Patroni:
@@ -915,13 +918,12 @@ For each node running a Patroni instance on the secondary site:
### Migrating a single tracking database node to Patroni
-Before the introduction of Patroni, Geo had no Omnibus support for HA setups on
+Before the introduction of Patroni, Geo provided no Omnibus support for HA setups on
the secondary site.
-With Patroni, it's now possible to support that. Due to some restrictions on the
-Patroni implementation on Omnibus that do not allow us to manage two different
-clusters on the same machine, we recommend setting up a new Patroni cluster for
-the tracking database by following the same instructions above.
+With Patroni, it's now possible to support HA setups. However, some restrictions in Patroni
+prevent the management of two different clusters on the same machine. You should set up a new
+Patroni cluster for the tracking database by following the same instructions above.
The secondary nodes backfill the new tracking database, and no data
synchronization is required.
@@ -935,8 +937,8 @@ Omnibus automatically configures a tracking database when `roles(['geo_secondary
If you want to run this database in a highly available configuration, don't use the `geo_secondary_role` above.
Instead, follow the instructions below.
-A production-ready and secure setup for the tracking PostgreSQL DB requires at least three Consul nodes, two
-Patroni nodes and one PgBouncer node on the secondary site.
+A production-ready and secure setup for the tracking PostgreSQL DB requires at least three Consul nodes: two
+Patroni nodes, and one PgBouncer node on the secondary site.
Because of [omnibus-6587](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6587), Consul can't track multiple
services, so these must be different than the nodes used for the Standby Cluster database.
@@ -1066,7 +1068,7 @@ On each node running a Patroni instance on the secondary site for the PostgreSQL
```
1. Reconfigure GitLab for the changes to take effect.
- This is required to bootstrap PostgreSQL users and settings:
+ This step is required to bootstrap PostgreSQL users and settings:
```shell
gitlab-ctl reconfigure
diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md
index e7a6fc35ced..d2f5282a69a 100644
--- a/doc/administration/gitaly/configure_gitaly.md
+++ b/doc/administration/gitaly/configure_gitaly.md
@@ -49,6 +49,9 @@ NOTE:
When configured to run on their own servers, Gitaly servers must be
[upgraded](../../update/package/index.md) before Gitaly clients in your cluster.
+NOTE:
+[Disk requirements](index.md#disk-requirements) apply to Gitaly nodes.
+
The process for setting up Gitaly on its own server is:
1. [Install Gitaly](#install-gitaly).
@@ -750,14 +753,25 @@ settings:
## Limit RPC concurrency
-Clone traffic can put a large strain on your Gitaly service. The bulk of the work gets done in the
-either of the following RPCs:
+WARNING:
+Enabling limits on your environment should be done with caution and only
+in select circumstances, such as to protect against unexpected traffic.
+When reached, limits _do_ result in disconnects that negatively impact users.
+For consistent and stable performance, you should first explore other options such as
+adjusting node specifications, and [reviewing large repositories](../../user/project/repository/managing_large_repositories.md) or workloads.
+
+When cloning or pulling repositories, various RPCs run in the background. In particular, the Git pack RPCs:
- `SSHUploadPackWithSidechannel` (for Git SSH).
- `PostUploadPackWithSidechannel` (for Git HTTP).
-To prevent such workloads from overwhelming your Gitaly server, you can set concurrency limits in
-Gitaly's configuration file. For example:
+These RPCs can consume a large amount of resources, which can have a significant impact in situations such as:
+
+- Unexpectedly high traffic.
+- Running against [large repositories](../../user/project/repository/managing_large_repositories.md) that don't follow best practices.
+
+You can limit these processes from overwhelming your Gitaly server in these scenarios using the concurrency limits in Gitaly's configuration file. For
+example:
```ruby
# in /etc/gitlab/gitlab.rb
@@ -795,30 +809,40 @@ repository. In the example above:
- If a request waits in the queue for more than 1 second, it is rejected with an error.
- If the queue grows beyond 10, subsequent requests are rejected with an error.
+NOTE:
+When these limits are reached, users are disconnected.
+
You can observe the behavior of this queue using the Gitaly logs and Prometheus. For more
information, see the [relevant documentation](monitoring.md#monitor-gitaly-concurrency-limiting).
## Control groups
+WARNING:
+Enabling limits on your environment should be done with caution and only
+in select circumstances, such as to protect against unexpected traffic.
+When reached, limits _do_ result in disconnects that negatively impact users.
+For consistent and stable performance, you should first explore other options such as
+adjusting node specifications, and [reviewing large repositories](../../user/project/repository/managing_large_repositories.md) or workloads.
+
FLAG:
On self-managed GitLab, by default repository cgroups are not available. To make it available, ask an administrator to
[enable the feature flag](../feature_flags.md) named `gitaly_run_cmds_in_cgroup`.
-Control groups (cgroups) in Linux allow limits to be imposed on how much memory and CPU can be consumed.
+When enabling cgroups for memory, you should ensure that no swap is configured on the Gitaly nodes as
+processes may switch to using that instead of being terminated. This situation could lead to notably compromised
+performance.
+
+You can use control groups (cgroups) in Linux to impose limits on how much memory and CPU can be consumed by Gitaly processes.
See the [`cgroups` Linux man page](https://man7.org/linux/man-pages/man7/cgroups.7.html) for more information.
-cgroups can be useful for protecting the system against resource exhaustion because of over consumption of memory and CPU.
+cgroups can be useful for protecting the system against unexpected resource exhaustion because of over consumption of memory and CPU.
-Some Git operations are expensive by nature. `git clone`, for instance,
-spawns a `git-upload-pack` process on the server that can consume a lot of memory
-for large repositories. For example, a client that keeps on cloning a
-large repository over and over again. This situation could potentially use up all of the
-memory on a server, causing other operations to fail for other users.
+Some Git operations can consume notable resources up to the point of exhaustion in situations such as:
-A repository can consume large amounts of memory for many reasons when cloned or downloaded.
-Using cgroups allows the kernel to kill these operations before they hog up all system resources.
+- Unexpectedly high traffic.
+- Operations running against large repositories that don't follow best practices.
-Gitaly shells out to Git for many of its operations. Git can consume a lot of resources for certain operations,
-especially for large repositories.
+As a hard protection, it's possible to use cgroups that configure the kernel to terminate these operations before they hog up all system resources
+and cause instability.
Gitaly has built-in cgroups control. When configured, Gitaly assigns Git processes to a cgroup based on the repository
the Git command is operating in. These cgroups are called repository cgroups. Each repository cgroup:
@@ -832,8 +856,8 @@ When a repository cgroup reaches its:
- Memory limit, the kernel looks through the processes for a candidate to kill.
- CPU limit, processes are not killed, but the processes are prevented from consuming more CPU than allowed.
-You configure repository cgroups for your GitLab installation to protect against system resource starvation from a few
-large repositories or bad actors.
+NOTE:
+When these limits are reached, performance may be reduced and users may be disconnected.
### Configure repository cgroups (new method)
@@ -913,8 +937,8 @@ gitaly['cgroups_cpu_enabled'] = true
In the previous example using the new configuration method:
-- The top level memory limit is capped at 60gb.
-- Each of the 1000 cgroups in the repositories pool is capped at 20gb.
+- The top level memory limit is capped at 60 GB.
+- Each of the 1000 cgroups in the repositories pool is capped at 20 GB.
This configuration leads to "oversubscription". Each cgroup in the pool has a much larger capacity than 1/1000th
of the top-level memory limit.
diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md
index 1b7e1f0f7c6..32b44552b1a 100644
--- a/doc/administration/gitaly/index.md
+++ b/doc/administration/gitaly/index.md
@@ -54,9 +54,8 @@ Before deploying Gitaly Cluster, review:
If you have:
-- Not yet migrated to Gitaly Cluster and want to continue using NFS, remain on the service you are using. NFS is
- supported in 14.x releases but is [deprecated](../../update/deprecations.md#nfs-for-git-repository-storage).
- Support for storing Git repository data on NFS is scheduled to end for all versions of GitLab on November 22, 2022.
+- Not yet migrated to Gitaly Cluster and want to continue using NFS, remain on the service you are using. However, NFS
+ is [no longer supported](../../update/removals.md#nfs-as-git-repository-storage-is-no-longer-supported).
- Not yet migrated to Gitaly Cluster but want to migrate away from NFS, you have two options:
- A sharded Gitaly instance.
- Gitaly Cluster.
@@ -90,6 +89,23 @@ If you are unable to use either method, contact customer support for restoration
Contact customer support for immediate help in restoration or recovery.
+## Disk requirements
+
+Gitaly and Gitaly Cluster require fast local storage to perform effectively because they are heavy I/O-based processes. Therefore,
+we strongly recommend that all Gitaly nodes use solid-state drives (SSDs).
+
+These SSDs should have a throughput of at least:
+
+- 8,000 input/output operations per second (IOPS) for read operations.
+- 2,000 IOPS for write operations.
+
+These IOPS values are initial recommendations, and may be adjusted to greater or lesser values
+depending on the scale of your environment's workload. If you’re running the environment on a
+cloud provider, refer to their documentation about how to configure IOPS correctly.
+
+For repository data, only local storage is supported for Gitaly and Gitaly Cluster for performance and consistency reasons.
+Alternatives such as [NFS](../nfs.md) or [cloud-based file systems](../nfs.md#avoid-using-cloud-based-file-systems) are not supported.
+
## Directly accessing repositories
GitLab doesn't advise directly accessing Gitaly repositories stored on disk with a Git client or any other tool,
@@ -405,33 +421,6 @@ The leftover state is eventually cleaned up.
Unlike Gitaly, Gitaly Cluster doesn't move the repositories in the storages but only virtually moves the repository by updating the
relative path of the repository in the metadata store.
-### Moving beyond NFS
-
-Engineering support for NFS for Git repositories is deprecated. Technical support is planned to be unavailable starting
-November 22, 2022. See our [statement of support](https://about.gitlab.com/support/statement-of-support/#gitaly-and-nfs)
-for more details.
-
-[Network File System (NFS)](https://en.wikipedia.org/wiki/Network_File_System)
-is not well suited to Git workloads which are CPU and IOPS sensitive.
-Specifically:
-
-- Git is sensitive to file system latency. Some operations require many
- read operations. Operations that are fast on block storage can become an order of
- magnitude slower. This significantly impacts GitLab application performance.
-- NFS performance optimizations that prevent the performance gap between
- block storage and NFS being even wider are vulnerable to race conditions. We have observed
- [data inconsistencies](https://gitlab.com/gitlab-org/gitaly/-/issues/2589)
- in production environments caused by simultaneous writes to different NFS
- clients. Data corruption is not an acceptable risk.
-
-Gitaly Cluster is purpose built to provide reliable, high performance, fault
-tolerant Git storage.
-
-Further reading:
-
-- Blog post: [The road to Gitaly v1.0 (aka, why GitLab doesn't require NFS for storing Git data anymore)](https://about.gitlab.com/blog/2018/09/12/the-road-to-gitaly-1-0/)
-- Blog post: [How we spent two weeks hunting an NFS bug in the Linux kernel](https://about.gitlab.com/blog/2018/11/14/how-we-spent-two-weeks-hunting-an-nfs-bug/)
-
### Components
Gitaly Cluster consists of multiple components:
@@ -672,7 +661,7 @@ To see if GitLab can access the repository file system directly, we use the foll
- Gitaly ensures that the file system has a metadata file in its root with a UUID in it.
- Gitaly reports this UUID to GitLab by using the `ServerInfo` RPC.
-- GitLab Rails tries to read the metadata file directly. If it exists, and if the UUID's match,
+- GitLab Rails tries to read the metadata file directly. If it exists, and if the UUIDs match,
assume we have direct access.
Direct Git access is:
@@ -696,8 +685,3 @@ There are two facets to our efforts to remove direct Git access in GitLab:
The second facet presents the only real solution. For this, we developed
[Gitaly Cluster](#gitaly-cluster).
-
-## NFS deprecation notice
-
-Engineering support for NFS for Git repositories is deprecated. Technical support is planned to be
-unavailable beginning November 22, 2022. For further information, see our [NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation) documentation.
diff --git a/doc/administration/gitaly/monitoring.md b/doc/administration/gitaly/monitoring.md
index 8d4f30c7c20..9024da269ca 100644
--- a/doc/administration/gitaly/monitoring.md
+++ b/doc/administration/gitaly/monitoring.md
@@ -49,7 +49,7 @@ the Gitaly logs and Prometheus:
You can observe the status of [control groups (cgroups)](configure_gitaly.md#control-groups) using Prometheus:
- `gitaly_cgroups_reclaim_attempts_total`, a gauge for the total number of times
- there has been a memory relcaim attempt. This number resets each time a server is
+ there has been a memory reclaim attempt. This number resets each time a server is
restarted.
- `gitaly_cgroups_cpu_usage`, a gauge that measures CPU usage per cgroup.
- `gitaly_cgroup_procs_total`, a gauge that measures the total number of
diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md
index 8cf32a6aaa3..9cc93b21ae9 100644
--- a/doc/administration/gitaly/praefect.md
+++ b/doc/administration/gitaly/praefect.md
@@ -28,6 +28,9 @@ The minimum recommended configuration for a Gitaly Cluster requires:
- 3 Praefect nodes
- 3 Gitaly nodes (1 primary, 2 secondary)
+NOTE:
+[Disk requirements](index.md#disk-requirements) apply to Gitaly nodes.
+
You should configure an odd number of Gitaly nodes so that transactions have a tie-breaker in case one of the
Gitaly nodes fails in a mutating RPC call.
@@ -43,15 +46,15 @@ default value. The default value depends on the GitLab version.
Network latency for Gitaly Cluster should ideally be measurable in single-digit milliseconds. Latency is particularly
important for:
-- Gitaly node health checks. Nodes must be able to respond within 1 second.
+- Gitaly node health checks. Nodes must be able to respond 1 second or faster.
- Reference transactions that enforce [strong consistency](index.md#strong-consistency). Lower latencies mean Gitaly
nodes can agree on changes faster.
Achieving acceptable latency between Gitaly nodes:
- On physical networks generally means high bandwidth, single location connections.
-- On the cloud generally means within the same region, including allowing cross availability zone replication. These links
- are designed for this type of synchronization. Latency of less than 2ms should be sufficient for Gitaly Cluster.
+- On the cloud generally means in the same region, including allowing cross availability zone replication. These links
+ are designed for this type of synchronization. Latency of less than 2 ms should be sufficient for Gitaly Cluster.
If you can't provide low network latencies for replication (for example, between distant locations), consider Geo. For
more information, see [Comparison to Geo](index.md#comparison-to-geo).
@@ -82,7 +85,7 @@ The requirements are relatively low because the database contains only metadata
- Where repositories are located.
- Some queued work.
-It depends on the number of repositories, but a useful minimum is 5-10 GB, similar to the main
+It depends on the number of repositories, but a good minimum is 5-10 GB, similar to the main
GitLab application database.
## Setup Instructions
@@ -103,7 +106,7 @@ If you [installed](https://about.gitlab.com/install/) GitLab using the Omnibus G
Before beginning, you should already have a working GitLab instance.
[Learn how to install GitLab](https://about.gitlab.com/install/).
-Provision a PostgreSQL server. We recommend using the PostgreSQL that is shipped
+Provision a PostgreSQL server. You should use the PostgreSQL that is shipped
with Omnibus GitLab and use it to configure the PostgreSQL database. You can use an
external PostgreSQL server (version 11 or newer) but you must set it up [manually](#manual-database-setup).
@@ -1453,7 +1456,7 @@ To migrate existing clusters:
1. On the Praefect nodes, configure the election strategy in `/etc/gitlab/gitlab.rb` with `praefect['failover_election_strategy'] = 'per_repository'`.
- 1. Run `gitlab-ctl reconfigure && gitlab-ctl start` to reconfigure and start the Praefects.
+ 1. Run `gitlab-ctl reconfigure && gitlab-ctl start` to reconfigure and start the Praefect nodes.
- If downtime is unacceptable:
diff --git a/doc/administration/gitaly/reference.md b/doc/administration/gitaly/reference.md
index 8f7dc688e56..cec18960dfd 100644
--- a/doc/administration/gitaly/reference.md
+++ b/doc/administration/gitaly/reference.md
@@ -57,7 +57,7 @@ an empty string.
It is possible to temporarily disable authentication with the `transitioning`
setting. This allows you to monitor if all clients are
authenticating correctly without causing a service outage for clients
-that are not configured correctly yet:
+that are still to be configured correctly:
```toml
[auth]
@@ -159,7 +159,7 @@ sum(rate(gitaly_catfile_cache_total{type="hit"}[5m])) / sum(rate(gitaly_catfile_
### `gitaly-ruby`
A Gitaly process uses one or more `gitaly-ruby` helper processes to
-execute RPC's implemented in Ruby instead of Go. The `[gitaly-ruby]`
+execute RPCs implemented in Ruby instead of Go. The `[gitaly-ruby]`
section of the configuration file contains settings for these helper processes.
These processes are known to occasionally suffer from memory leaks.
@@ -169,7 +169,7 @@ Gitaly restarts its `gitaly-ruby` helpers when their memory exceeds the
| Name | Type | Required | Description |
| ---- | ---- | -------- | ----------- |
| `dir` | string | yes | Path to where `gitaly-ruby` is installed (needed to boot the process).|
-| `max_rss` | integer | no | Resident set size limit that triggers a `gitaly-ruby` restart, in bytes. Default is `200000000` (200MB). |
+| `max_rss` | integer | no | Resident set size limit that triggers a `gitaly-ruby` restart, in bytes. Default is `200000000` (200 MB). |
| `graceful_restart_timeout` | string | no | Grace period before a `gitaly-ruby` process is forcibly terminated after exceeding `max_rss`. Default is `10m` (10 minutes).|
| `restart_delay` | string | no |Time that `gitaly-ruby` memory must remain high before a restart. Default is `5m` (5 minutes).|
| `num_workers` | integer | no |Number of `gitaly-ruby` worker processes. Try increasing this number in case of `ResourceExhausted` errors. Default is `2`, minimum is `2`.|
diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md
index 7edce840396..7f5d4b9e443 100644
--- a/doc/administration/gitaly/troubleshooting.md
+++ b/doc/administration/gitaly/troubleshooting.md
@@ -245,6 +245,28 @@ the application might be fetching this secret from a different file. Your Gitaly
If that setting is missing, GitLab defaults to using `.gitlab_shell_secret` under
`/opt/gitlab/embedded/service/gitlab-rails/.gitlab_shell_secret`.
+### Repository pushes fail
+
+When attempting `git push`, you can see:
+
+- `401 Unauthorized` errors.
+- The following in server logs:
+
+ ```json
+ {
+ ...
+ "exception.class":"JWT::VerificationError",
+ "exception.message":"Signature verification raised",
+ ...
+ }
+ ```
+
+This error occurs when the GitLab server has been upgraded to GitLab 15.5 or later but Gitaly has not yet been upgraded.
+
+From GitLab 15.5, GitLab [authenticates with GitLab Shell using a JWT token instead of a shared secret](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/86148).
+You should follow the [recommendations on upgrading external Gitaly](../../update/plan_your_upgrade.md#external-gitaly) and upgrade Gitaly before the GitLab
+server.
+
### Repository pushes fail with a `deny updating a hidden ref` error
Due to [a change](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3426)
@@ -456,6 +478,20 @@ in sync so the token check succeeds.
This check helps identify the root cause of `permission denied`
[errors being logged by Praefect](#permission-denied-errors-appearing-in-gitaly-or-praefect-logs-when-accessing-repositories).
+For offline environments where access to public [`pool.ntp.org`](https://pool.ntp.org) servers is not possible, the Praefect `check` sub-command fails this
+check with an error message similar to:
+
+```plaintext
+checking with NTP service at and allowed clock drift 60000ms [correlation_id: <XXX>]
+Failed (fatal) error: gitaly node at tcp://[gitlab.example-instance.com]:8075: rpc error: code = DeadlineExceeded desc = context deadline exceeded
+```
+
+To resolve this issue, set an environment variable on all Praefect servers to point to an accessible internal NTP server. For example:
+
+```shell
+export NTP_HOST=ntp.example.com
+```
+
### Praefect errors in logs
If you receive an error, check `/var/log/gitlab/gitlab-rails/production.log`.
@@ -608,7 +644,7 @@ Is [some cases](index.md#known-issues) the Praefect database can get out of sync
a given repository is fully synced on all nodes, run the [`gitlab:praefect:replicas` Rake task](../raketasks/praefect.md#replica-checksums)
that checksums the repository on all Gitaly nodes.
-The [Praefect dataloss](recovery.md#check-for-data-loss) command only checks the state of the repository in the Praefect database, and cannot
+The [Praefect `dataloss`](recovery.md#check-for-data-loss) command only checks the state of the repository in the Praefect database, and cannot
be relied to detect sync problems in this scenario.
### Relation does not exist errors
diff --git a/doc/administration/housekeeping.md b/doc/administration/housekeeping.md
index 2d3e937e047..584f06ef537 100644
--- a/doc/administration/housekeeping.md
+++ b/doc/administration/housekeeping.md
@@ -45,11 +45,12 @@ be slow.
### Heuristical housekeeping
-> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/2634) in GitLab 14.9 for the [manual trigger](#manual-trigger) and the [push-based trigger](#push-based-trigger) [with a flag](feature_flags.md) named `optimized_housekeeping`. Disabled by default.
+> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/2634) in GitLab 14.9 for the [manual trigger](#manual-trigger) and the [push-based trigger](#push-based-trigger) [with a flag](feature_flags.md) named `optimized_housekeeping`. Enabled by default.
> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/353607) in GitLab 14.10.
FLAG:
-On self-managed GitLab, by default this feature is not available for the [manual trigger](#manual-trigger) and the [push-based trigger](#push-based-trigger).
+On self-managed GitLab, by default this feature is available. To hide the feature, ask an administrator to [disable the feature flag](feature_flags.md) named `optimize_repository`.
+
To make it available, ask an administrator to [enable the feature flag](feature_flags.md) named `optimized_housekeeping`.
The heuristical (or "opportunistic") housekeeping strategy analyzes the
@@ -76,9 +77,22 @@ based on the size of the repository:
Gitaly does this to offset the fact that optimizing those data structures takes
more time the bigger they get. It is especially important in large
-monorepositories (which receive a lot of traffic) to avoid optimizing them too
+monorepos (which receive a lot of traffic) to avoid optimizing them too
frequently.
+You can change how often Gitaly is asked to optimize a repository.
+
+1. On the top bar, select **Main menu > Admin**.
+1. On the left sidebar, select **Settings > Repository**.
+1. Expand **Repository maintenance**.
+1. In the **Housekeeping** section, configure the housekeeping options.
+1. Select **Save changes**.
+
+- **Enable automatic repository housekeeping**: Regularly ask Gitaly to run repository optimization. If you
+ keep this setting disabled for a long time, Git repository access on your GitLab server becomes
+ slower and your repositories use more disk space.
+- **Optimize repository period**: Number of Git pushes after which Gitaly is asked to optimize a repository.
+
## Running housekeeping tasks
There are different ways in which GitLab runs housekeeping tasks:
@@ -108,8 +122,13 @@ To trigger housekeeping tasks manually:
This starts an asynchronous background worker for the project's repository. The
background worker executes `git gc`, which performs a number of optimizations.
+<!--- start_remove The following content will be removed on remove_date: '2023-04-22' -->
+
### Push-based trigger
+FLAG:
+On self-managed GitLab, by default this feature is not available and superseded by [heuristical housekeeping](#heuristical-housekeeping). It is planned to be removed in 15.8. To enable the feature, ask an administrator to [disable the feature flag](feature_flags.md) named `optimize_repository`.
+
GitLab automatically runs repository housekeeping tasks after a configured
number of pushes:
@@ -208,7 +227,7 @@ of a repository. When creating the first fork, we:
1. Create an object pool repository that contains all objects of the repository
that is about to be forked.
-1. Link the repository to this new object pool via Git's altenates mechanism.
+1. Link the repository to this new object pool via Git's alternates mechanism.
1. Repack the repository so that it uses objects from the object pool. It thus
can drop its own copy of the objects.
diff --git a/doc/administration/img/audit_events_v14_5.png b/doc/administration/img/audit_events_v14_5.png
deleted file mode 100644
index 57190463d05..00000000000
--- a/doc/administration/img/audit_events_v14_5.png
+++ /dev/null
Binary files differ
diff --git a/doc/administration/img/impersonated_audit_events_v13_8.png b/doc/administration/img/impersonated_audit_events_v13_8.png
deleted file mode 100644
index 0a8548d515d..00000000000
--- a/doc/administration/img/impersonated_audit_events_v13_8.png
+++ /dev/null
Binary files differ
diff --git a/doc/administration/img/impersonated_audit_events_v15_7.png b/doc/administration/img/impersonated_audit_events_v15_7.png
new file mode 100644
index 00000000000..ef7decd984d
--- /dev/null
+++ b/doc/administration/img/impersonated_audit_events_v15_7.png
Binary files differ
diff --git a/doc/administration/inactive_project_deletion.md b/doc/administration/inactive_project_deletion.md
index 88d8e3fc648..ea5658bef84 100644
--- a/doc/administration/inactive_project_deletion.md
+++ b/doc/administration/inactive_project_deletion.md
@@ -12,7 +12,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w
Administrators of large GitLab instances can find that over time, projects become inactive and are no longer used.
These projects take up unnecessary disk space. With inactive project deletion, you can identify these projects, warn
the maintainers ahead of time, and then delete the projects if they remain inactive. When an inactive project is
-deleted, the action generates an audit event that it was performed by the first active administrator.
+deleted, the action generates an audit event that it was performed by the @GitLab-Admin-Bot.
For the default setting on GitLab.com, see the [GitLab.com settings page](../user/gitlab_com/index.md#inactive-project-deletion).
diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md
index 433956bb066..826340ad967 100644
--- a/doc/administration/incoming_email.md
+++ b/doc/administration/incoming_email.md
@@ -10,7 +10,7 @@ GitLab has several features based on receiving incoming email messages:
- [Reply by Email](reply_by_email.md): allow GitLab users to comment on issues
and merge requests by replying to notification email.
-- [New issue by email](../user/project/issues/managing_issues.md#by-sending-an-email):
+- [New issue by email](../user/project/issues/create_issues.md#by-sending-an-email):
allow GitLab users to create a new issue by sending an email to a
user-specific email address.
- [New merge request by email](../user/project/merge_requests/creating_merge_requests.md#by-sending-an-email):
diff --git a/doc/administration/index.md b/doc/administration/index.md
index 95db2b7a2e0..1059424da27 100644
--- a/doc/administration/index.md
+++ b/doc/administration/index.md
@@ -121,6 +121,7 @@ Learn how to install, configure, update, and maintain your GitLab instance.
- [Creating users](../user/profile/account/create_accounts.md): Create users manually or through authentication integrations.
- [Libravatar](libravatar.md): Use Libravatar instead of Gravatar for user avatars.
- [Sign-up restrictions](../user/admin_area/settings/sign_up_restrictions.md): block email addresses of specific domains, or allow only specific domains.
+- [Admin mode](../user/admin_area/settings/sign_in_restrictions.md#admin-mode): require that administrators authenticate separately to use administrative access, like `sudo`.
- [Access restrictions](../user/admin_area/settings/visibility_and_access_controls.md#configure-enabled-git-access-protocols): Define which Git access protocols can be used to talk to GitLab (SSH, HTTP, HTTPS).
- [Authentication and Authorization](auth/index.md): Configure external authentication with LDAP, SAML, CAS, and additional providers.
- [Sync LDAP](auth/ldap/index.md)
@@ -133,7 +134,7 @@ Learn how to install, configure, update, and maintain your GitLab instance.
- Instances.
- [Auditor users](auditor_users.md): Users with read-only access to all projects, groups, and other resources on the GitLab instance.
- [Incoming email](incoming_email.md): Configure incoming emails to allow
- users to [reply by email](reply_by_email.md), create [issues by email](../user/project/issues/managing_issues.md#by-sending-an-email) and
+ users to [reply by email](reply_by_email.md), create [issues by email](../user/project/issues/create_issues.md#by-sending-an-email) and
[merge requests by email](../user/project/merge_requests/creating_merge_requests.md#by-sending-an-email), and to enable [Service Desk](../user/project/service_desk.md).
- [Postfix for incoming email](reply_by_email_postfix_setup.md): Set up a
basic Postfix mail server with IMAP authentication on Ubuntu for incoming
@@ -208,7 +209,7 @@ Learn how to install, configure, update, and maintain your GitLab instance.
## Troubleshooting
- [Log system](logs/index.md): Where to look for logs.
-- [Sidekiq Troubleshooting](troubleshooting/sidekiq.md): Debug when Sidekiq appears hung and is not processing jobs.
+- [Sidekiq Troubleshooting](sidekiq/sidekiq_troubleshooting.md): Debug when Sidekiq appears hung and is not processing jobs.
- [Navigating GitLab via Rails console](operations/rails_console.md)
- [GitLab application limits](instance_limits.md)
- [Responding to security incidents](../security/responding_to_security_incidents.md)
diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md
index 546c4667220..59746fc0f07 100644
--- a/doc/administration/instance_limits.md
+++ b/doc/administration/instance_limits.md
@@ -378,7 +378,7 @@ and to limit memory consumption.
When using offset-based pagination in the REST API, there is a limit to the maximum
requested offset into the set of results. This limit is only applied to endpoints that
-support keyset-based pagination. More information about pagination options can be
+also support keyset-based pagination. More information about pagination options can be
found in the [API documentation section on pagination](../api/index.md#pagination).
To set this limit for a self-managed installation, run the following in the
@@ -581,7 +581,8 @@ limit value. For example, for a maximum frequency of:
- Once per 10 minutes, the limit must be `144`.
- Once per 60 minutes, the limit must be `24`
-There is no limit for self-managed instances by default.
+The minimum value is `24`, or one pipeline per 60 minutes.
+There is no maximum value.
To set this limit to `1440` on a self-managed installation, run the following in the
[GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session):
@@ -672,6 +673,7 @@ setting is used:
| `ci_max_artifact_size_network_referee` | 0 |
| `ci_max_artifact_size_performance` | 0 |
| `ci_max_artifact_size_requirements` | 0 |
+| `ci_max_artifact_size_requirements_v2` | 0 |
| `ci_max_artifact_size_sast` | 0 |
| `ci_max_artifact_size_secret_detection` | 0 |
| `ci_max_artifact_size_terraform` | 5 MB ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37018) in GitLab 13.3) |
@@ -818,9 +820,9 @@ This limit is [enabled on GitLab.com](../user/gitlab_com/index.md#gitlab-cicd).
You can set a limit on the maximum size of a dotenv artifact. This limit is checked
every time a dotenv file is exported as an artifact.
-Set the limit to `0` to disable it. Defaults to 5KB.
+Set the limit to `0` to disable it. Defaults to 5 KB.
-To set this limit to 5KB on a self-managed installation, run the following in the
+To set this limit to 5 KB on a self-managed installation, run the following in the
[GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session):
```ruby
diff --git a/doc/administration/integration/terminal.md b/doc/administration/integration/terminal.md
index 14da70f6efb..a1522ccac31 100644
--- a/doc/administration/integration/terminal.md
+++ b/doc/administration/integration/terminal.md
@@ -41,7 +41,7 @@ In brief:
- The WebSocket is handled in [Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse),
rather than the Rails application server.
- Workhorse queries Rails for connection details and user permissions. Rails
- queries Kubernetes for them in the background using [Sidekiq](../troubleshooting/sidekiq.md).
+ queries Kubernetes for them in the background using [Sidekiq](../sidekiq/sidekiq_troubleshooting.md).
- Workhorse acts as a proxy server between the user's browser and the Kubernetes
API, passing WebSocket frames between the two.
- Workhorse regularly polls Rails, terminating the WebSocket connection if the
@@ -64,8 +64,8 @@ detail below.
## Enabling and disabling terminal support
NOTE:
-AWS Classic Load Balancers (CLBs) do not support web sockets.
-If you want web terminals to work, use AWS Network Load Balancers (NLBs).
+AWS Classic Load Balancers do not support web sockets.
+If you want web terminals to work, use AWS Network Load Balancers.
Read [AWS Elastic Load Balancing Product Comparison](https://aws.amazon.com/elasticloadbalancing/features/#compare)
for more information.
diff --git a/doc/administration/issue_closing_pattern.md b/doc/administration/issue_closing_pattern.md
index e9150ae0650..bdbcdd0093c 100644
--- a/doc/administration/issue_closing_pattern.md
+++ b/doc/administration/issue_closing_pattern.md
@@ -17,36 +17,93 @@ in the project's default branch.
## Change the issue closing pattern
-To change the pattern, you must have access to the server that GitLab
-is installed on.
-
-The default pattern can be located in [`gitlab.yml.example`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/gitlab.yml.example)
-under the "Automatic issue closing" section.
+The [default issue closing pattern](../user/project/issues/managing_issues.md#default-closing-pattern)
+covers a wide range of words. You can change the pattern to suit your needs.
NOTE:
You are advised to use <https://rubular.com> to test the issue closing pattern.
-Because Rubular doesn't understand `%{issue_ref}`, you can replace this by
+However, since Rubular doesn't understand `%{issue_ref}`, you can replace this by
`#\d+` when testing your patterns, which matches only local issue references like `#123`.
-**For Omnibus installations**
+To change the default issue closing pattern:
+
+::Tabs
-1. Open `/etc/gitlab/gitlab.rb` with your editor.
-1. Change the value of `gitlab_rails['gitlab_issue_closing_pattern']` to a regular
- expression of your liking:
+:::TabTitle Linux package (Omnibus)
+
+1. Edit `/etc/gitlab/gitlab.rb` and change the `gitlab_rails['gitlab_issue_closing_pattern']`
+ value:
```ruby
- gitlab_rails['gitlab_issue_closing_pattern'] = /\b((?:[Cc]los(?:e[sd]?|ing)|\b[Ff]ix(?:e[sd]|ing)?|\b[Rr]esolv(?:e[sd]?|ing)|\b[Ii]mplement(?:s|ed|ing)?)(:?) +(?:(?:issues? +)?%{issue_ref}(?:(?: *,? +and +| *,? *)?)|([A-Z][A-Z0-9_]+-\d+))+)/.source
+ gitlab_rails['gitlab_issue_closing_pattern'] = /<regular_expression>/.source
+ ```
+
+1. Save the file and reconfigure GitLab:
+
+ ```shell
+ sudo gitlab-ctl reconfigure
+ ```
+
+:::TabTitle Helm chart (Kubernetes)
+
+1. Export the Helm values:
+
+ ```shell
+ helm get values gitlab > gitlab_values.yaml
+ ```
+
+1. Edit `gitlab_values.yaml` and change the `issueClosingPattern` value:
+
+ ```yaml
+ global:
+ appConfig:
+ issueClosingPattern: "<regular_expression>"
```
-1. [Reconfigure](restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect.
+1. Save the file and apply the new values:
-**For installations from source**
+ ```shell
+ helm upgrade -f gitlab_values.yaml gitlab gitlab/gitlab
+ ```
+
+:::TabTitle Docker
+
+1. Edit `docker-compose.yml` and change the `gitlab_rails['gitlab_issue_closing_pattern']`
+ value:
+
+ ```yaml
+ version: "3.6"
+ services:
+ gitlab:
+ environment:
+ GITLAB_OMNIBUS_CONFIG: |
+ gitlab_rails['gitlab_issue_closing_pattern'] = /<regular_expression>/.source
+ ```
+
+1. Save the file and restart GitLab:
-1. Open `gitlab.yml` with your editor.
-1. Change the value of `issue_closing_pattern`:
+ ```shell
+ docker compose up -d
+ ```
+
+:::TabTitle Self-compiled (source)
+
+1. Edit `/home/git/gitlab/config/gitlab.yml` and change the `issue_closing_pattern` value:
```yaml
- issue_closing_pattern: "\b((?:[Cc]los(?:e[sd]?|ing)|\b[Ff]ix(?:e[sd]|ing)?|\b[Rr]esolv(?:e[sd]?|ing)|\b[Ii]mplement(?:s|ed|ing)?)(:?) +(?:(?:issues? +)?%{issue_ref}(?:(?: *,? +and +| *,? *)?)|([A-Z][A-Z0-9_]+-\d+))+)"
+ production: &base
+ gitlab:
+ issue_closing_pattern: "<regular_expression>"
+ ```
+
+1. Save the file and restart GitLab:
+
+ ```shell
+ # For systems running systemd
+ sudo systemctl restart gitlab.target
+
+ # For systems running SysV init
+ sudo service gitlab restart
```
-1. [Restart](restart_gitlab.md#installations-from-source) GitLab for the changes to take effect.
+::EndTabs
diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md
index d79f322c3f5..fb2f73876e8 100644
--- a/doc/administration/job_artifacts.md
+++ b/doc/administration/job_artifacts.md
@@ -100,8 +100,7 @@ In a multi-server setup you must use one of the options to
#### Object Storage Settings
-NOTE:
-In GitLab 13.2 and later, we recommend using the
+In GitLab 13.2 and later, you should use the
[consolidated object storage settings](object_storage.md#consolidated-object-storage-configuration).
This section describes the earlier configuration format.
@@ -235,7 +234,7 @@ by the `gitlab:artifacts:migrate` Rake task.
To migrate back to local storage:
1. Run `gitlab-rake gitlab:artifacts:migrate_to_local`.
-1. Disable object_storage for artifacts in `gitlab.rb`:
+1. Disable object storage for artifacts in `gitlab.rb`:
- Set `gitlab_rails['artifacts_object_store_enabled'] = false`.
- Comment out all other `artifacts_object_store` settings, including the entire
`artifacts_object_store_connection` section, including the closing `}`.
@@ -636,24 +635,6 @@ If you need to manually remove **all** job artifacts associated with multiple jo
- `3.months.ago`
- `1.year.ago`
-### Error `Downloading artifacts from coordinator... not found`
-
-When a job attempts to download artifacts from an earlier job, you might receive an error message similar to:
-
-```plaintext
-Downloading artifacts from coordinator... not found id=12345678 responseStatus=404 Not Found
-```
-
-This can be caused by a `gitlab.rb` file with the following configuration:
-
-```ruby
-gitlab_rails['artifacts_object_store_background_upload'] = false
-gitlab_rails['artifacts_object_store_direct_upload'] = true
-```
-
-To prevent this, comment out or remove those lines, or switch to their [default values](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template), and
-then run `sudo gitlab-ctl reconfigure`.
-
### Job artifact upload fails with error 500
If you are using object storage for artifacts and a job artifact fails to upload,
diff --git a/doc/administration/job_logs.md b/doc/administration/job_logs.md
index 5aa0e8f3948..9b25c70716b 100644
--- a/doc/administration/job_logs.md
+++ b/doc/administration/job_logs.md
@@ -158,7 +158,7 @@ To eliminate both file system requirements:
The data flow is the same as described in the [data flow section](#data-flow)
with one change: _the stored path of the first two phases is different_. This incremental
log architecture stores chunks of logs in Redis and a persistent store (object storage or database) instead of
-file storage. Redis is used as first-class storage, and it stores up-to 128KB
+file storage. Redis is used as first-class storage, and it stores up-to 128 KB
of data. After the full chunk is sent, it is flushed to a persistent store, either object storage (temporary directory) or database.
After a while, the data in Redis and a persistent store is archived to [object storage](#uploading-logs-to-object-storage).
@@ -169,7 +169,7 @@ Here is the detailed data flow:
1. The runner picks a job from GitLab
1. The runner sends a piece of log to GitLab
1. GitLab appends the data to Redis
-1. After the data in Redis reaches 128KB, the data is flushed to a persistent store (object storage or the database).
+1. After the data in Redis reaches 128 KB, the data is flushed to a persistent store (object storage or the database).
1. The above steps are repeated until the job is finished.
1. After the job is finished, GitLab schedules a Sidekiq worker to archive the log.
1. The Sidekiq worker archives the log to object storage and cleans up the log
diff --git a/doc/administration/lfs/index.md b/doc/administration/lfs/index.md
index 8fdc98bd12a..cf80b05a5e0 100644
--- a/doc/administration/lfs/index.md
+++ b/doc/administration/lfs/index.md
@@ -7,7 +7,8 @@ disqus_identifier: 'https://docs.gitlab.com/ee/workflow/lfs/lfs_administration.h
# GitLab Git Large File Storage (LFS) Administration **(FREE SELF)**
-Documentation about how to use Git LFS are under [Managing large binary files with Git LFS doc](../../topics/git/lfs/index.md).
+This page contains information about configuring Git LFS in self-managed GitLab instances.
+For user documentation about Git LFS, see [Git Large File Storage](../../topics/git/lfs/index.md).
LFS is enabled in GitLab self-managed instances by default.
@@ -62,27 +63,17 @@ to check which storage services can be integrated with GitLab.
You can also use external object storage in a private local network. For example,
[MinIO](https://min.io/) is a standalone object storage service that works with GitLab instances.
-GitLab provides two different options for the uploading mechanism: "Direct upload" and "Background upload".
-
[Read more about using object storage with GitLab](../object_storage.md).
NOTE:
-In GitLab 13.2 and later, we recommend using the
+In GitLab 13.2 and later, you should use the
[consolidated object storage settings](../object_storage.md#consolidated-object-storage-configuration).
-This section describes the earlier configuration format.
-
-**Option 1. Direct upload**
+This section describes the earlier configuration format. [Migration steps still apply](#migrating-to-object-storage).
1. User pushes an `lfs` file to the GitLab instance.
1. GitLab-workhorse uploads the file directly to the external object storage.
1. GitLab-workhorse notifies GitLab-rails that the upload process is complete.
-**Option 2. Background upload**
-
-1. User pushes an `lfs` file to the GitLab instance.
-1. GitLab-rails stores the file in the local file storage.
-1. GitLab-rails then uploads the file to the external object storage asynchronously.
-
The following general settings are supported.
| Setting | Description | Default |
@@ -119,9 +110,7 @@ On Omnibus GitLab installations, the settings are prefixed by `lfs_object_store_
1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
1. [Migrate any existing local LFS objects to the object storage](#migrating-to-object-storage).
- New LFS objects
- are forwarded to object storage unless
- `gitlab_rails['lfs_object_store_background_upload']` and `gitlab_rails['lfs_object_store_direct_upload']` is set to `false`.
+ New LFS objects are forwarded to object storage.
### S3 for installations from source
@@ -150,9 +139,7 @@ For source installations the settings are nested under `lfs:` and then
1. Save the file, and then [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect.
1. [Migrate any existing local LFS objects to the object storage](#migrating-to-object-storage).
- New LFS objects
- are forwarded to object storage unless
- `background_upload` and `direct_upload` is set to `false`.
+ New LFS objects are forwarded to object storage.
### Migrating to object storage
@@ -231,6 +218,12 @@ You can see the total storage used for LFS objects on groups and projects:
- In the administration area.
- In the [groups](../../api/groups.md) and [projects APIs](../../api/projects.md).
+## Related topics
+
+- Blog post: [Getting started with Git LFS](https://about.gitlab.com/blog/2017/01/30/getting-started-with-git-lfs-tutorial/)
+- User documentation: [Git Large File Storage (LFS)](../../topics/git/lfs/index.md)
+- [Git LFS developer information](../../development/lfs.md)
+
## Troubleshooting
### Missing LFS objects
@@ -276,39 +269,6 @@ To delete these references:
lfs_object.destroy
```
-### `Google::Apis::TransmissionError: execution expired`
-
-If LFS integration is configured with Google Cloud Storage and background uploads (`background_upload: true` and `direct_upload: false`),
-Sidekiq workers may encounter this error. This is because the uploading timed out with very large files.
-LFS files up to 6 GB can be uploaded without any extra steps, otherwise you need to use the following workaround.
-
-Sign in to Rails console:
-
-```shell
-sudo gitlab-rails console
-```
-
-Set up timeouts:
-
-- These settings are only in effect for the same session. For example, they are not effective for Sidekiq workers.
-- 20 minutes (1200 sec) is enough to upload 30GB LFS files:
-
-```ruby
-::Google::Apis::ClientOptions.default.open_timeout_sec = 1200
-::Google::Apis::ClientOptions.default.read_timeout_sec = 1200
-::Google::Apis::ClientOptions.default.send_timeout_sec = 1200
-```
-
-Upload LFS files manually (this process does not use Sidekiq at all):
-
-```ruby
-LfsObject.where(file_store: [nil, 1]).find_each do |lfs_object|
- lfs_object.file.migrate!(ObjectStorage::Store::REMOTE) if lfs_object.file.file.exists?
-end
-```
-
-See more information in [!19581](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/19581)
-
### LFS commands fail on TLS v1.3 server
If you configure GitLab to [disable TLS v1.2](https://docs.gitlab.com/omnibus/settings/nginx.html)
@@ -332,6 +292,28 @@ To check an installed Git LFS client's version, run this command:
git lfs version
```
+## Error viewing a PDF file
+
+When LFS has been configured with object storage and `proxy_download` set to
+`false`, [you may see an error when previewing a PDF file from the Web browser](https://gitlab.com/gitlab-org/gitlab/-/issues/248100):
+
+```plaintext
+An error occurred while loading the file. Please try again later.
+```
+
+This occurs due to Cross-Origin Resource Sharing (CORS) restrictions:
+the browser attempts to load the PDF from object storage, but the object
+storage provider rejects the request because the GitLab domain differs
+from the object storage domain.
+
+To fix this issue, configure your object storage provider's CORS
+settings to allow the GitLab domain. See the following documentation
+for more details:
+
+1. [AWS S3](https://aws.amazon.com/premiumsupport/knowledge-center/s3-configure-cors/)
+1. [Google Cloud Storage](https://cloud.google.com/storage/docs/configuring-cors)
+1. [Azure Storage](https://learn.microsoft.com/en-us/rest/api/storageservices/cross-origin-resource-sharing--cors--support-for-the-azure-storage-services).
+
## Known limitations
- Only compatible with the Git LFS client versions 1.1.0 and later, or 1.0.2.
diff --git a/doc/administration/logs/index.md b/doc/administration/logs/index.md
index b0631c52a47..9893c6935a3 100644
--- a/doc/administration/logs/index.md
+++ b/doc/administration/logs/index.md
@@ -6,9 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# Log system **(FREE SELF)**
-GitLab has an advanced log system where everything is logged, so you
-can analyze your instance using various system log files. In addition to
-system log files, GitLab Enterprise Edition provides [Audit Events](../audit_events.md).
+GitLab has an advanced log system where everything is logged, so you can analyze your instance using various system log
+files. The log system is similar to [audit events](../audit_events.md).
System log files are typically plain text in a standard log file format.
This guide talks about how to read and use these system log files.
@@ -86,24 +85,24 @@ except those captured by `runit`.
| Log type | Managed by logrotate | Managed by svlogd/runit |
|:------------------------------------------------|:------------------------|:------------------------|
-| [Alertmanager Logs](#alertmanager-logs) | **{dotted-circle}** No | **{check-circle}** Yes |
-| [Crond Logs](#crond-logs) | **{dotted-circle}** No | **{check-circle}** Yes |
+| [Alertmanager logs](#alertmanager-logs) | **{dotted-circle}** No | **{check-circle}** Yes |
+| [crond logs](#crond-logs) | **{dotted-circle}** No | **{check-circle}** Yes |
| [Gitaly](#gitaly-logs) | **{check-circle}** Yes | **{check-circle}** Yes |
| [GitLab Exporter for Omnibus](#gitlab-exporter) | **{dotted-circle}** No | **{check-circle}** Yes |
-| [GitLab Pages Logs](#pages-logs) | **{check-circle}** Yes | **{check-circle}** Yes |
+| [GitLab Pages logs](#pages-logs) | **{check-circle}** Yes | **{check-circle}** Yes |
| GitLab Rails | **{check-circle}** Yes | **{dotted-circle}** No |
-| [GitLab Shell Logs](#gitlab-shelllog) | **{check-circle}** Yes | **{dotted-circle}** No |
-| [Grafana Logs](#grafana-logs) | **{dotted-circle}** No | **{check-circle}** Yes |
-| [LogRotate Logs](#logrotate-logs) | **{dotted-circle}** No | **{check-circle}** Yes |
+| [GitLab Shell logs](#gitlab-shelllog) | **{check-circle}** Yes | **{dotted-circle}** No |
+| [Grafana logs](#grafana-logs) | **{dotted-circle}** No | **{check-circle}** Yes |
+| [LogRotate logs](#logrotate-logs) | **{dotted-circle}** No | **{check-circle}** Yes |
| [Mailroom](#mail_room_jsonlog-default) | **{check-circle}** Yes | **{check-circle}** Yes |
| [NGINX](#nginx-logs) | **{check-circle}** Yes | **{check-circle}** Yes |
-| [PostgreSQL Logs](#postgresql-logs) | **{dotted-circle}** No | **{check-circle}** Yes |
-| [Praefect Logs](#praefect-logs) | **{dotted-circle}** Yes | **{check-circle}** Yes |
-| [Prometheus Logs](#prometheus-logs) | **{dotted-circle}** No | **{check-circle}** Yes |
+| [PostgreSQL logs](#postgresql-logs) | **{dotted-circle}** No | **{check-circle}** Yes |
+| [Praefect logs](#praefect-logs) | **{dotted-circle}** Yes | **{check-circle}** Yes |
+| [Prometheus logs](#prometheus-logs) | **{dotted-circle}** No | **{check-circle}** Yes |
| [Puma](#puma-logs) | **{check-circle}** Yes | **{check-circle}** Yes |
-| [Redis Logs](#redis-logs) | **{dotted-circle}** No | **{check-circle}** Yes |
-| [Registry Logs](#registry-logs) | **{dotted-circle}** No | **{check-circle}** Yes |
-| [Workhorse Logs](#workhorse-logs) | **{check-circle}** Yes | **{check-circle}** Yes |
+| [Redis logs](#redis-logs) | **{dotted-circle}** No | **{check-circle}** Yes |
+| [Registry logs](#registry-logs) | **{dotted-circle}** No | **{check-circle}** Yes |
+| [Workhorse logs](#workhorse-logs) | **{check-circle}** Yes | **{check-circle}** Yes |
## `production_json.log`
@@ -161,10 +160,14 @@ seconds:
- `gitaly_duration_s`: Total time by Gitaly calls
- `gitaly_calls`: Total number of calls made to Gitaly
- `redis_calls`: Total number of calls made to Redis
+- `redis_cross_slot_calls`: Total number of cross-slot calls made to Redis
+- `redis_allowed_cross_slot_calls`: Total number of allowed cross-slot calls made to Redis
- `redis_duration_s`: Total time to retrieve data from Redis
- `redis_read_bytes`: Total bytes read from Redis
- `redis_write_bytes`: Total bytes written to Redis
- `redis_<instance>_calls`: Total number of calls made to a Redis instance
+- `redis_<instance>_cross_slot_calls`: Total number of cross-slot calls made to a Redis instance
+- `redis_<instance>_allowed_cross_slot_calls`: Total number of allowed cross-slot calls made to a Redis instance
- `redis_<instance>_duration_s`: Total time to retrieve data from a Redis instance
- `redis_<instance>_read_bytes`: Total bytes read from a Redis instance
- `redis_<instance>_write_bytes`: Total bytes written to a Redis instance
@@ -490,7 +493,7 @@ are logged to this file. For example:
}
```
-## Sidekiq Logs
+## Sidekiq logs
NOTE:
In Omnibus GitLab `12.10` or earlier, the Sidekiq log is at `/var/log/gitlab/gitlab-rails/sidekiq.log`.
@@ -670,7 +673,7 @@ I, [2015-02-13T06:17:00.679433 #9291] INFO -- : Moving existing hooks directory
User clone/fetch activity using SSH transport appears in this log as
`executing git command <gitaly-upload-pack...`.
-## Gitaly Logs
+## Gitaly logs
This file is in `/var/log/gitlab/gitaly/current` and is produced by [runit](http://smarden.org/runit/).
`runit` is packaged with Omnibus GitLab and a brief explanation of its purpose
@@ -697,7 +700,7 @@ This file is at `/var/log/gitlab/gitaly/gitaly_hooks.log` and is
produced by `gitaly-hooks` command. It also contains records about
failures received during processing of the responses from GitLab API.
-## Puma Logs
+## Puma logs
### `puma_stdout.log`
@@ -980,11 +983,11 @@ can be used.
}
```
-## Registry Logs
+## Registry logs
For Omnibus GitLab installations, Container Registry logs are in `/var/log/gitlab/registry/current`.
-## NGINX Logs
+## NGINX logs
For Omnibus GitLab installations, NGINX logs are in:
@@ -1003,7 +1006,7 @@ Below is the default GitLab NGINX access log format:
$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"
```
-## Pages Logs
+## Pages logs
For Omnibus GitLab installations, Pages logs are in `/var/log/gitlab/gitlab-pages/current`.
@@ -1032,50 +1035,50 @@ For example:
}
```
-## Mattermost Logs
+## Mattermost logs
For Omnibus GitLab installations, Mattermost logs are in these locations:
- `/var/log/gitlab/mattermost/mattermost.log`
- `/var/log/gitlab/mattermost/current`
-## Workhorse Logs
+## Workhorse logs
For Omnibus GitLab installations, Workhorse logs are in `/var/log/gitlab/gitlab-workhorse/current`.
-## PostgreSQL Logs
+## PostgreSQL logs
For Omnibus GitLab installations, PostgreSQL logs are in `/var/log/gitlab/postgresql/current`.
-## Prometheus Logs
+## Prometheus logs
For Omnibus GitLab installations, Prometheus logs are in `/var/log/gitlab/prometheus/current`.
-## Redis Logs
+## Redis logs
For Omnibus GitLab installations, Redis logs are in `/var/log/gitlab/redis/current`.
-## Alertmanager Logs
+## Alertmanager logs
For Omnibus GitLab installations, Alertmanager logs are in `/var/log/gitlab/alertmanager/current`.
<!-- vale gitlab.Spelling = NO -->
-## Crond Logs
+## crond logs
For Omnibus GitLab installations, crond logs are in `/var/log/gitlab/crond/`.
<!-- vale gitlab.Spelling = YES -->
-## Grafana Logs
+## Grafana logs
For Omnibus GitLab installations, Grafana logs are in `/var/log/gitlab/grafana/current`.
-## LogRotate Logs
+## LogRotate logs
For Omnibus GitLab installations, `logrotate` logs are in `/var/log/gitlab/logrotate/current`.
-## GitLab Monitor Logs
+## GitLab Monitor logs
For Omnibus GitLab installations, GitLab Monitor logs are in `/var/log/gitlab/gitlab-monitor/`.
@@ -1088,7 +1091,7 @@ For Omnibus GitLab installations, GitLab Exporter logs are in `/var/log/gitlab/g
For Omnibus GitLab installations, GitLab agent server logs are
in `/var/log/gitlab/gitlab-kas/current`.
-## Praefect Logs
+## Praefect logs
For Omnibus GitLab installations, Praefect logs are in `/var/log/gitlab/praefect/`.
diff --git a/doc/administration/maintenance_mode/index.md b/doc/administration/maintenance_mode/index.md
index 9adb8ce2cd9..c30465b4df9 100644
--- a/doc/administration/maintenance_mode/index.md
+++ b/doc/administration/maintenance_mode/index.md
@@ -84,7 +84,7 @@ them to disable Maintenance Mode after it's been enabled.
All users can sign in and out of the GitLab instance but no new users can be created.
-If there are [LDAP syncs](../auth/ldap/index.md) scheduled for that time, they fail since user creation is disabled. Similarly, [user creations based on SAML](../../integration/saml.md#general-setup) fail.
+If there are [LDAP syncs](../auth/ldap/index.md) scheduled for that time, they fail since user creation is disabled. Similarly, [user creations based on SAML](../../integration/saml.md#configure-saml-support-in-gitlab) fail.
### Git actions
@@ -178,7 +178,7 @@ To monitor queues and disable jobs:
### Incident management
-[Incident management](../../operations/incident_management/index.md) functions are limited. The creation of [alerts](../../operations/incident_management/alerts.md) and [incidents](../../operations/incident_management/incidents.md#incident-creation) are paused entirely. Notifications and paging on alerts and incidents are therefore disabled.
+[Incident management](../../operations/incident_management/index.md) functions are limited. The creation of [alerts](../../operations/incident_management/alerts.md) and [incidents](../../operations/incident_management/manage_incidents.md#create-an-incident) are paused entirely. Notifications and paging on alerts and incidents are therefore disabled.
### Feature flags
diff --git a/doc/administration/merge_request_diffs.md b/doc/administration/merge_request_diffs.md
index 4dec9e52c19..25dba00beb9 100644
--- a/doc/administration/merge_request_diffs.md
+++ b/doc/administration/merge_request_diffs.md
@@ -104,8 +104,7 @@ be configured already.
### Object Storage Settings
-NOTE:
-In GitLab 13.2 and later, we recommend using the
+In GitLab 13.2 and later, you should use the
[consolidated object storage settings](object_storage.md#consolidated-object-storage-configuration).
This section describes the earlier configuration format.
diff --git a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md
index 35dc64a0594..566bc070347 100644
--- a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md
+++ b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md
@@ -101,7 +101,7 @@ You can add custom metrics in the self-monitoring project by:
A [bug](https://gitlab.com/gitlab-org/gitlab/-/issues/208676) causes project creation to fail with
the following error in the log file when the first administrator user is an
-[external user](../../../user/permissions.md#external-users):
+[external user](../../../user/admin_area/external_users.md):
```plaintext
Could not create instance administrators group. Errors: ["You don't have permission to create groups."]
@@ -116,6 +116,6 @@ User.admins.active.first.external?
If this returns true, the first administrator user is an external user.
If you face this issue, you can temporarily
-[make the administrator user a non-external user](../../../user/permissions.md#external-users)
+[make the administrator user a non-external user](../../../user/admin_area/external_users.md)
and then try to create the project.
After the project is created, the administrator user can be changed back to an external user.
diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md
index de6178a3151..171a441eaec 100644
--- a/doc/administration/monitoring/performance/performance_bar.md
+++ b/doc/administration/monitoring/performance/performance_bar.md
@@ -1,6 +1,6 @@
---
-stage: Monitor
-group: Respond
+stage: Data Stores
+group: Application Performance
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
diff --git a/doc/administration/monitoring/prometheus/gitlab_exporter.md b/doc/administration/monitoring/prometheus/gitlab_exporter.md
index bce35306505..1f4156349c5 100644
--- a/doc/administration/monitoring/prometheus/gitlab_exporter.md
+++ b/doc/administration/monitoring/prometheus/gitlab_exporter.md
@@ -1,6 +1,6 @@
---
-stage: Monitor
-group: Respond
+stage: Data Stores
+group: Application Performance
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md
index d3b28fd15bc..53ee028bc32 100644
--- a/doc/administration/monitoring/prometheus/gitlab_metrics.md
+++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md
@@ -39,8 +39,9 @@ The following metrics are available:
| `gitlab_cache_misses_total` | Counter | 10.2 | Cache read miss | `controller`, `action` |
| `gitlab_cache_operation_duration_seconds` | Histogram | 10.2 | Cache access time | |
| `gitlab_cache_operations_total` | Counter | 12.2 | Cache operations by controller or action | `controller`, `action`, `operation` |
+| `gitlab_cache_read_multikey_count` | Histogram | 15.7 | Count of keys in multi-key cache read operations | `controller`, `action` |
| `gitlab_ci_pipeline_builder_scoped_variables_duration` | Histogram | 14.5 | Time in seconds it takes to create the scoped variables for a CI/CD job
-| `gitlab_ci_pipeline_creation_duration_seconds` | Histogram | 13.0 | Time in seconds it takes to create a CI/CD pipeline | |
+| `gitlab_ci_pipeline_creation_duration_seconds` | Histogram | 13.0 | Time in seconds it takes to create a CI/CD pipeline | `gitlab` |
| `gitlab_ci_pipeline_size_builds` | Histogram | 13.1 | Total number of builds within a pipeline grouped by a pipeline source | `source` |
| `gitlab_ci_runner_authentication_success_total` | Counter | 15.2 | Total number of times that runner authentication has succeeded | `type` |
| `gitlab_ci_runner_authentication_failure_total` | Counter | 15.2 | Total number of times that runner authentication has failed
@@ -331,6 +332,16 @@ configuration option in `gitlab.yml`. These metrics are served from the
| `geo_dependency_proxy_blob_verification_total` | Gauge | 15.6 | Number of dependency proxy blobs verifications tried on secondary | |
| `geo_dependency_proxy_blob_verified` | Gauge | 15.6 | Number of dependency proxy blobs verified on secondary | |
| `geo_dependency_proxy_blob_verification_failed` | Gauge | 15.6 | Number of dependency proxy blobs verifications failed on secondary | |
+| `geo_dependency_proxy_manifests` | Gauge | 15.6 | Number of dependency proxy manifests on primary | `url` |
+| `geo_dependency_proxy_manifests_checksum_total` | Gauge | 15.6 | Number of dependency proxy manifests tried to checksum on primary | `url` |
+| `geo_dependency_proxy_manifests_checksummed` | Gauge | 15.6 | Number of dependency proxy manifests successfully checksummed on primary | `url` |
+| `geo_dependency_proxy_manifests_checksum_failed` | Gauge | 15.6 | Number of dependency proxy manifests failed to calculate the checksum on primary | `url` |
+| `geo_dependency_proxy_manifests_synced` | Gauge | 15.6 | Number of syncable dependency proxy manifests synced on secondary | `url` |
+| `geo_dependency_proxy_manifests_failed` | Gauge | 15.6 | Number of syncable dependency proxy manifests failed to sync on secondary | `url` |
+| `geo_dependency_proxy_manifests_registry` | Gauge | 15.6 | Number of dependency proxy manifests in the registry | `url` |
+| `geo_dependency_proxy_manifests_verification_total` | Gauge | 15.6 | Number of dependency proxy manifests verifications tried on secondary | `url` |
+| `geo_dependency_proxy_manifests_verified` | Gauge | 15.6 | Number of dependency proxy manifests verified on secondary | `url` |
+| `geo_dependency_proxy_manifests_verification_failed` | Gauge | 15.6 | Number of dependency proxy manifests verifications failed on secondary | `url` |
## Database load balancing metrics **(PREMIUM SELF)**
diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md
index 85f35a1b188..972db315268 100644
--- a/doc/administration/nfs.md
+++ b/doc/administration/nfs.md
@@ -20,47 +20,14 @@ actions that read or write to Git repositories. For steps you can use to test
file system performance, see
[File System Performance Benchmarking](operations/filesystem_benchmarking.md).
-## Gitaly and NFS deprecation
+## Gitaly with NFS not supported
-Starting with GitLab version 14.0, support for NFS to store Git repository data is deprecated. Technical customer support and engineering support is available for the 14.x releases. Engineering is fixing bugs and security vulnerabilities consistent with our [release and maintenance policy](../policy/maintenance.md#security-releases).
+Technical and engineering support for using NFS to store Git repository data is officially at end-of-life. No product
+changes or troubleshooting is provided through engineering, security or paid support channels.
-Upon the release of GitLab 15.6 technical and engineering support for using NFS to store Git repository data is officially at end-of-life. There are no product changes or troubleshooting provided via Engineering, Security or Paid Support channels after the release date of 15.6, regardless of your GitLab version.
-
-Until the release of 15.6, for customers running 14.x releases, we continue to help with Git related tickets from customers running one or more Gitaly servers with its data stored on NFS. Examples may include:
-
-- Performance issues or timeouts accessing Git data
-- Commits or branches vanish
-- GitLab intermittently returns the wrong Git data (such as reporting that a repository has no branches)
-
-Assistance is limited to activities like:
-
-- Verifying developers' workflow uses features like protected branches
-- Reviewing GitLab event data from the database to advise if it looks like a force push over-wrote branches
-- Verifying that NFS client mount options match our [documented recommendations](#mount-options)
-- Analyzing the GitLab Workhorse and Rails logs, and determining that `500` errors being seen in the environment are caused by slow responses from Gitaly
-
-GitLab support is unable to continue with the investigation if both:
-
-- The date of the request is on or after the release of GitLab version 15.6.
-- Support Engineers and Management determine that all reasonable non-NFS root causes have been exhausted.
-
-If the issue is reproducible, or if it happens intermittently but regularly, GitLab Support can investigate providing the issue reproduces without the use of NFS. To reproduce without NFS, the affected repositories should be migrated to a different Gitaly shard, such as Gitaly cluster or a standalone Gitaly VM, backed with block storage.
-
-### Why remove NFS for Git repository data
-
-{:.no-toc}
-
-NFS is not well-suited to a workload consisting of many small files, like Git repositories. NFS does provide a number of configuration options designed to improve performance. However, over time, a number of these mount options have proven to result in inconsistencies across multiple nodes mounting the NFS volume, up to and including data loss. Addressing these inconsistencies consume extraordinary development and support engineer time that hamper our ability to develop [Gitaly Cluster](gitaly/praefect.md), our purpose-built solution to addressing the deficiencies of NFS in this environment.
-
-Gitaly Cluster provides highly-available Git repository storage. If this is not a requirement, single-node Gitaly backed by block storage is a suitable substitute.
-
-Engineering support for NFS for Git repositories is deprecated. Technical support is planned to be
-unavailable from GitLab 15.0. No further enhancements are planned for this feature.
-
-Read:
-
-- [Moving beyond NFS](gitaly/index.md#moving-beyond-nfs).
-- About the [correct mount options to use](#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss).
+If an issue is reproducible, or if it happens intermittently but regularly, GitLab Support can investigate providing the
+issue can be reproduced without NFS. To reproduce without NFS, migrate the affected repositories to a different Gitaly
+shard. For example, a Gitaly Cluster or a standalone Gitaly VM, backed with block storage.
## Known kernel version incompatibilities
@@ -397,8 +364,8 @@ sudo ufw allow from <client_ip_address> to any port nfs
### Upgrade to Gitaly Cluster or disable caching if experiencing data loss
WARNING:
-Engineering support for NFS for Git repositories is deprecated. Read about
-[moving beyond NFS](gitaly/index.md#moving-beyond-nfs).
+Engineering support for NFS for Git repositories
+[is unavailable](../update/removals.md#nfs-as-git-repository-storage-is-no-longer-supported).
Customers and users have reported data loss on high-traffic repositories when using NFS for Git repositories.
For example, we have seen:
diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md
index d2c9c35148c..0cae46faf28 100644
--- a/doc/administration/object_storage.md
+++ b/doc/administration/object_storage.md
@@ -70,7 +70,7 @@ must be enabled, only the following providers can be used:
- [Azure Blob storage](#azure-blob-storage)
When consolidated object storage is used, direct upload is enabled
-automatically. Background upload is not supported. For storage-specific
+automatically. For storage-specific
configuration, [direct upload may become the default](https://gitlab.com/gitlab-org/gitlab/-/issues/27331)
because it does not require a shared folder.
@@ -482,7 +482,7 @@ To migrate existing local data to object storage see the following guides:
- [LFS objects](lfs/index.md#migrating-to-object-storage)
- [Uploads](raketasks/uploads/migrate.md#migrate-to-object-storage)
- [Merge request diffs](merge_request_diffs.md#using-object-storage)
-- [Packages](packages/index.md#migrating-local-packages-to-object-storage) (optional feature)
+- [Packages](packages/index.md#migrate-local-packages-to-object-storage) (optional feature)
- [Dependency Proxy](packages/dependency_proxy.md#migrate-local-dependency-proxy-blobs-and-manifests-to-object-storage)
- [Terraform state files](terraform_state.md#migrate-to-object-storage)
- [Pages content](pages/index.md#migrate-pages-deployments-to-object-storage)
@@ -536,8 +536,8 @@ supported by consolidated configuration form, refer to the following guides:
| [Uploads](uploads.md#using-object-storage) | **{check-circle}** Yes |
| [Container Registry](packages/container_registry.md#use-object-storage) (optional feature) | **{dotted-circle}** No |
| [Merge request diffs](merge_request_diffs.md#using-object-storage) | **{check-circle}** Yes |
-| [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| **{dotted-circle}** No |
-| [Packages](packages/index.md#using-object-storage) (optional feature) | **{check-circle}** Yes |
+| [Mattermost](https://docs.mattermost.com/configure/file-storage-configuration-settings.html)| **{dotted-circle}** No |
+| [Packages](packages/index.md#use-object-storage) (optional feature) | **{check-circle}** Yes |
| [Dependency Proxy](packages/dependency_proxy.md#using-object-storage) (optional feature) | **{check-circle}** Yes |
| [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | **{dotted-circle}** No |
| [Terraform state files](terraform_state.md#using-object-storage) | **{check-circle}** Yes |
@@ -638,9 +638,15 @@ if this access is not in place include:
Received status code 403 from server: Forbidden
```
-Getting a `403 Forbidden` response is specifically called out on the
-[package repository documentation](packages/index.md#using-object-storage)
-as a side effect of how some build tools work.
+- Object storage buckets need to allow Cross-Origin Resource Sharing
+ (CORS) access from the URL of the GitLab instance. Attempting to load
+ a PDF in the repository page may show the following error:
+
+ ```plaintext
+ An error occurred while loading the file. Please try again later.
+ ```
+
+ See [the LFS documentation](lfs/index.md#error-viewing-a-pdf-file) for more details.
Additionally for a short time period users could share pre-signed, time-limited object storage URLs
with others without authentication. Also bandwidth charges may be incurred
diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md
deleted file mode 100644
index 58858c54843..00000000000
--- a/doc/administration/operations/extra_sidekiq_processes.md
+++ /dev/null
@@ -1,11 +0,0 @@
----
-redirect_to: '../sidekiq/extra_sidekiq_processes.md'
-remove_date: '2022-11-11'
----
-
-This document was moved to [another location](../sidekiq/extra_sidekiq_processes.md).
-
-<!-- This redirect file can be deleted after <2022-11-11>. -->
-<!-- Redirects that point to other docs in the same project expire in three months. -->
-<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. -->
-<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
diff --git a/doc/administration/operations/extra_sidekiq_routing.md b/doc/administration/operations/extra_sidekiq_routing.md
deleted file mode 100644
index 072b6f63537..00000000000
--- a/doc/administration/operations/extra_sidekiq_routing.md
+++ /dev/null
@@ -1,11 +0,0 @@
----
-redirect_to: '../sidekiq/extra_sidekiq_routing.md'
-remove_date: '2022-11-11'
----
-
-This document was moved to [another location](../sidekiq/extra_sidekiq_routing.md).
-
-<!-- This redirect file can be deleted after <2022-11-11>. -->
-<!-- Redirects that point to other docs in the same project expire in three months. -->
-<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. -->
-<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
diff --git a/doc/administration/operations/filesystem_benchmarking.md b/doc/administration/operations/filesystem_benchmarking.md
index ec2975baf52..cd4ab1a9cf8 100644
--- a/doc/administration/operations/filesystem_benchmarking.md
+++ b/doc/administration/operations/filesystem_benchmarking.md
@@ -19,7 +19,7 @@ I/O. The information on this page can be used for either scenario.
### Benchmarking with `fio`
-We recommend using
+You should use
[Fio](https://fio.readthedocs.io/en/latest/fio_doc.html) to test I/O
performance. This test should be run both on the NFS server and on the
application nodes that talk to the NFS server.
@@ -35,8 +35,8 @@ Then run the following:
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --bs=4k --iodepth=64 --readwrite=randrw --rwmixread=75 --size=4G --filename=/path/to/git-data/testfile
```
-This creates a 4GB file in `/path/to/git-data/testfile`. It performs
-4KB reads and writes using a 75%/25% split in the file, with 64
+This creates a 4 GB file in `/path/to/git-data/testfile`. It performs
+4 KB reads and writes using a 75%/25% split in the file, with 64
operations running at a time. Be sure to delete the file after the test
completes.
diff --git a/doc/administration/operations/index.md b/doc/administration/operations/index.md
index d18f41becd5..527a72c5933 100644
--- a/doc/administration/operations/index.md
+++ b/doc/administration/operations/index.md
@@ -13,10 +13,9 @@ Keep your GitLab instance up and running smoothly.
and more.
- [Moving repositories](moving_repositories.md): Moving all repositories managed
by GitLab to another file system or another server.
-- [Sidekiq MemoryKiller](sidekiq_memory_killer.md): Configure Sidekiq MemoryKiller
+- [Sidekiq MemoryKiller](../sidekiq/sidekiq_memory_killer.md): Configure Sidekiq MemoryKiller
to restart Sidekiq.
-- [Multiple Sidekiq processes](extra_sidekiq_processes.md): Configure multiple Sidekiq processes to ensure certain queues always have dedicated workers, no matter the number of jobs that must be processed. **(FREE SELF)**
-- [Sidekiq routing rules](extra_sidekiq_routing.md): Configure the routing rules to route a job from a worker to a desirable queue. **(FREE SELF)**
+- [Multiple Sidekiq processes](../sidekiq/extra_sidekiq_processes.md): Configure multiple Sidekiq processes to ensure certain queues always have dedicated workers, no matter the number of jobs that must be processed. **(FREE SELF)**
- [Puma](puma.md): Understand Puma and puma-worker-killer.
- Speed up SSH operations by
[Authorizing SSH users via a fast, indexed lookup to the GitLab database](fast_ssh_key_lookup.md), and/or
diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md
index eb326c06e6a..af595cdf297 100644
--- a/doc/administration/operations/puma.md
+++ b/doc/administration/operations/puma.md
@@ -66,9 +66,9 @@ From this output:
- The formula that calculates the maximum memory value results in workers
being killed before they reach the `per_worker_max_memory_mb` value.
-- In GitLab 13.4 and earlier, the default values for the formula were 550MB for the primary
- and 850MB for each worker.
-- In GitLab 13.5 and later, the values are primary: 800MB, worker: 1024MB.
+- In GitLab 13.4 and earlier, the default values for the formula were 550 MB for the primary
+ and 850 MB for each worker.
+- In GitLab 13.5 and later, the values are primary: 800 MB, worker: 1024 MB.
- The threshold for workers to be killed is set at 98% of the limit:
```plaintext
@@ -110,11 +110,11 @@ To change the worker timeout to 600 seconds:
WARNING:
This is an experimental [Alpha feature](../../policy/alpha-beta-support.md#alpha-features) and subject to change without notice. The feature
-is not ready for production use. If you want to use this feature, we recommend testing
+is not ready for production use. If you want to use this feature, you should test
outside of production first. See the [known issues](#puma-single-mode-known-issues)
for additional details.
-In a memory-constrained environment with less than 4GB of RAM available, consider disabling Puma
+In a memory-constrained environment with less than 4 GB of RAM available, consider disabling Puma
[clustered mode](https://github.com/puma/puma#clustered-mode).
Set the number of `workers` to `0` to reduce memory usage by hundreds of MB:
diff --git a/doc/administration/operations/sidekiq_memory_killer.md b/doc/administration/operations/sidekiq_memory_killer.md
deleted file mode 100644
index b1977589fae..00000000000
--- a/doc/administration/operations/sidekiq_memory_killer.md
+++ /dev/null
@@ -1,11 +0,0 @@
----
-redirect_to: '../sidekiq/sidekiq_memory_killer.md'
-remove_date: '2022-11-11'
----
-
-This document was moved to [another location](../sidekiq/sidekiq_memory_killer.md).
-
-<!-- This redirect file can be deleted after <2022-11-11>. -->
-<!-- Redirects that point to other docs in the same project expire in three months. -->
-<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. -->
-<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
diff --git a/doc/administration/package_information/index.md b/doc/administration/package_information/index.md
index 4758b453135..bfa751a051f 100644
--- a/doc/administration/package_information/index.md
+++ b/doc/administration/package_information/index.md
@@ -16,9 +16,9 @@ The released package versions are in the format `MAJOR.MINOR.PATCH-EDITION.OMNIB
| Component | Meaning | Example |
|---------------------|---------|---------|
-| `MAJOR.MINOR.PATCH` | The GitLab version this corresponds to. | 13.3.0 |
-| `EDITION` | The edition of GitLab this corresponds to. | ee |
-| `OMNIBUS_RELEASE` | The Omnibus GitLab release. Usually, this is 0. This is incremented if we need to build a new package without changing the GitLab version. | 0 |
+| `MAJOR.MINOR.PATCH` | The GitLab version this corresponds to. | `13.3.0` |
+| `EDITION` | The edition of GitLab this corresponds to. | `ee` |
+| `OMNIBUS_RELEASE` | The Omnibus GitLab release. Usually, this is 0. This is incremented if we need to build a new package without changing the GitLab version. | `0` |
## Licenses
diff --git a/doc/administration/package_information/licensing.md b/doc/administration/package_information/licensing.md
index 0e46289a308..ece1597c521 100644
--- a/doc/administration/package_information/licensing.md
+++ b/doc/administration/package_information/licensing.md
@@ -25,8 +25,7 @@ Starting with version 9.2, the Omnibus GitLab package ships a
`dependency_licenses.json` file containing version and license information of
all bundled software, including software libraries, Ruby gems that the rails
application uses, and JavaScript libraries that is required for the frontend
-components. This file, being in JSON format, is easily machine parseable and
-can be used for automated checks or validations. The file may be found at
+components. Because it's in JSON format, GitLab can parse this file easily and use it for automated checks or validations. The file may be found at
`/opt/gitlab/dependency_licenses.json`.
Starting with version 11.3, we have also made the license information available
diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md
index 2623f2afd8d..d76c728fb78 100644
--- a/doc/administration/packages/container_registry.md
+++ b/doc/administration/packages/container_registry.md
@@ -343,6 +343,9 @@ these deleted temporary upload artifacts are kept as non-current versions, there
storage bucket size. To ensure that non-current versions are deleted after a given amount of time,
you should configure an object lifecycle policy with your storage provider.
+WARNING:
+Do not directly modify the files or objects stored by the container registry. Anything other than the registry writing or deleting these entries can lead to instance-wide data consistency and instability issues from which recovery may not be possible.
+
You can configure the Container Registry to use various storage backends by
configuring a storage driver. By default the GitLab Container Registry
is configured to use the [file system driver](#use-file-system)
@@ -585,6 +588,38 @@ you can pull from the Container Registry, but you cannot push.
1. Configure your registry to [use the S3 bucket for storage](#use-object-storage).
1. For the changes to take effect, set the Registry back to `read-write` mode and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+#### Moving to Azure Object Storage
+
+When moving from an existing file system or another object storage provider to Azure Object Storage, you must configure the registry to use the standard root directory.
+This configuration is done by setting [`trimlegacyrootprefix: true]`](https://gitlab.com/gitlab-org/container-registry/-/blob/a3f64464c3ec1c5a599c0a2daa99ebcbc0100b9a/docs-gitlab/README.md#azure-storage-driver) in the Azure storage driver section of the registry configuration.
+Without this configuration, the Azure storage driver uses `//` instead of `/` as the first section of the root path, rendering the migrated images inaccessible.
+
+**Omnibus GitLab installations**
+
+```ruby
+registry['storage'] = {
+ 'azure' => {
+ 'accountname' => 'accountname',
+ 'accesskey' => 'base64encodedaccountkey',
+ 'container' => 'containername',
+ 'rootdirectory' => '/azure/virtual/container',
+ 'trimlegacyrootprefix' => 'true'
+ }
+}
+```
+
+**Installations from source**
+
+```yaml
+storage:
+ azure:
+ accountname: accountname
+ accountkey: base64encodedaccountkey
+ container: containername
+ rootdirectory: /azure/virtual/container
+ trimlegacyrootprefix: true
+```
+
### Disable redirect for storage driver
By default, users accessing a registry configured with a remote backend are redirected to the default backend for the storage driver. For example, registries can be configured using the `s3` storage driver, which redirects requests to a remote S3 bucket to alleviate load on the GitLab server.
@@ -882,7 +917,7 @@ WARNING:
If you're using a distributed architecture and Sidekiq is running on a different node, the cleanup
policies don't work. To fix this, you must configure the `gitlab.rb` file on the Sidekiq nodes to
point to the correct registry URL and copy the `registry.key` file to each Sidekiq node. For more
-information, see the [Sidekiq configuration](../sidekiq.md)
+information, see the [Sidekiq configuration](../sidekiq/index.md)
page.
To reduce the amount of [Container Registry disk space used by a given project](#registry-disk-space-usage-by-project),
@@ -1091,6 +1126,9 @@ To enable the read-only mode:
1. Next, trigger one of the garbage collect commands:
+ WARNING:
+ You must use `/opt/gitlab/embedded/bin/registry` to recycle unused tags. If you use `gitlab-ctl registry-garbage-collect`, you **will bring the container registry down**.
+
```shell
# Recycling unused tags
sudo /opt/gitlab/embedded/bin/registry garbage-collect /var/opt/gitlab/registry/config.yml
@@ -1348,7 +1386,7 @@ level=error msg="response completed with error" err.code=unknown err.detail="une
```
To resolve the error specify a `chunksize` value in the Registry configuration.
-Start with a value between `25000000` (25MB) and `50000000` (50MB).
+Start with a value between `25000000` (25 MB) and `50000000` (50 MB).
**For Omnibus installations**
diff --git a/doc/administration/packages/dependency_proxy.md b/doc/administration/packages/dependency_proxy.md
index f6eb6f85274..78efd75bbe3 100644
--- a/doc/administration/packages/dependency_proxy.md
+++ b/doc/administration/packages/dependency_proxy.md
@@ -121,15 +121,12 @@ To change the local storage path:
### Using object storage
Instead of relying on the local storage, you can use an object storage to
-store the blobs of the Dependency Proxy.
+store the blobs of the Dependency Proxy. In GitLab 13.2 and later, you should use the
+[consolidated object storage settings](../object_storage.md#consolidated-object-storage-configuration).
+This section describes the earlier configuration format. [Migration steps still apply](#migrate-local-dependency-proxy-blobs-and-manifests-to-object-storage).
[Read more about using object storage with GitLab](../object_storage.md).
-NOTE:
-In GitLab 13.2 and later, we recommend using the
-[consolidated object storage settings](../object_storage.md#consolidated-object-storage-configuration).
-This section describes the earlier configuration format.
-
**Omnibus GitLab installations**
1. Edit `/etc/gitlab/gitlab.rb` and add the following lines (uncomment where
diff --git a/doc/administration/packages/index.md b/doc/administration/packages/index.md
index 74d835eb744..6e53d77109f 100644
--- a/doc/administration/packages/index.md
+++ b/doc/administration/packages/index.md
@@ -6,11 +6,13 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# GitLab Package Registry administration **(FREE SELF)**
-GitLab Packages allows organizations to use GitLab as a private repository
-for a variety of common package managers. Users are able to build and publish
-packages, which can be easily consumed as a dependency in downstream projects.
+To use GitLab as a private repository for a variety of common package managers, use the Package Registry.
+You can build and publish
+packages, which can be consumed as dependencies in downstream projects.
-The Packages feature allows GitLab to act as a repository and supports the following formats:
+## Supported formats
+
+The Package Registry supports the following formats:
| Package type | GitLab version |
|-------------------------------------------------------------------|----------------|
@@ -49,204 +51,155 @@ guides you through the process.
<!-- vale gitlab.Spelling = YES -->
-## Enabling the Packages feature
+## Rate limits
-NOTE:
-After the Packages feature is enabled, the repositories are available
-for all new projects by default. To enable it for existing projects, users
-explicitly do so in the project's settings.
+When downloading packages as dependencies in downstream projects, many requests are made through the
+Packages API. You may therefore reach enforced user and IP rate limits. To address this issue, you
+can define specific rate limits for the Packages API. For more details, see [Package Registry Rate Limits](../../user/admin_area/settings/package_registry_rate_limits.md).
-To enable the Packages feature:
+## Change the storage path
-**Omnibus GitLab installations**
+By default, the packages are stored locally, but you can change the default
+local location or even use object storage.
-1. Edit `/etc/gitlab/gitlab.rb` and add the following line:
+### Change the local storage path
- ```ruby
- gitlab_rails['packages_enabled'] = true
- ```
+By default, the packages are stored in a local path, relative to the GitLab
+installation:
-1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+- Linux package (Omnibus): `/var/opt/gitlab/gitlab-rails/shared/packages/`
+- Self-compiled (source): `/home/git/gitlab/shared/packages/`
-**Installations from source**
+To change the local storage path:
-1. After the installation is complete, you configure the `packages`
- section in `config/gitlab.yml`. Set to `true` to enable it:
+::Tabs
- ```yaml
- packages:
- enabled: true
+:::TabTitle Linux package (Omnibus)
+
+1. Edit `/etc/gitlab/gitlab.rb` and add the following line:
+
+ ```ruby
+ gitlab_rails['packages_storage_path'] = "/mnt/packages"
```
-1. [Restart GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+1. Save the file and reconfigure GitLab:
+
+ ```shell
+ sudo gitlab-ctl reconfigure
+ ```
-**Helm Chart installations**
+:::TabTitle Self-compiled (source)
-1. After the installation is complete, you configure the `packages`
- section in `global.appConfig.packages`. Set to `true` to enable it:
+1. Edit `/home/git/gitlab/config/gitlab.yml`:
```yaml
- packages:
- enabled: true
+ production: &base
+ packages:
+ enabled: true
+ storage_path: /mnt/packages
```
-1. [Restart GitLab](../restart_gitlab.md#helm-chart-installations) for the changes to take effect.
+1. Save the file and restart GitLab:
-## Rate limits
+ ```shell
+ # For systems running systemd
+ sudo systemctl restart gitlab.target
-When downloading packages as dependencies in downstream projects, many requests are made through the
-Packages API. You may therefore reach enforced user and IP rate limits. To address this issue, you
-can define specific rate limits for the Packages API. For more details, see [Package Registry Rate Limits](../../user/admin_area/settings/package_registry_rate_limits.md).
+ # For systems running SysV init
+ sudo service gitlab restart
+ ```
-## Changing the storage path
+::EndTabs
-By default, the packages are stored locally, but you can change the default
-local location or even use object storage.
+Docker and Kubernetes do not use local storage.
-### Changing the local storage path
+- For the Helm chart (Kubernetes): Use object storage instead.
+- For Docker: The `/var/opt/gitlab/` directory is already
+ mounted in a directory on the host. There's no need to change the local
+ storage path inside the container.
-The packages for Omnibus GitLab installations are stored under
-`/var/opt/gitlab/gitlab-rails/shared/packages/` and for source
-installations under `shared/packages/` (relative to the Git home directory).
-To change the local storage path:
+### Use object storage
-**Omnibus GitLab installations**
+Instead of relying on the local storage, you can use an object storage to store
+packages.
-1. Edit `/etc/gitlab/gitlab.rb` and add the following line:
+For more information, see how to use the
+[consolidated object storage settings](../object_storage.md#consolidated-object-storage-configuration).
- ```ruby
- gitlab_rails['packages_storage_path'] = "/mnt/packages"
- ```
+### Migrate local packages to object storage
+
+After [configuring the object storage](#use-object-storage), use the following task to
+migrate existing packages from the local storage to the remote storage.
+The processing is done in a background worker and requires **no downtime**.
-1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure)
- for the changes to take effect.
+1. Migrate the packages.
-**Installations from source**
+ ::Tabs
-1. Edit the `packages` section in `config/gitlab.yml`:
+ :::TabTitle Linux package (Omnibus)
- ```yaml
- packages:
- enabled: true
- storage_path: shared/packages
+ ```shell
+ sudo gitlab-rake "gitlab:packages:migrate"
```
-1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect.
-
-### Using object storage
+ :::TabTitle Self-compiled (source)
-Instead of relying on the local storage, you can use an object storage to
-store packages.
+ ```shell
+ RAILS_ENV=production sudo -u git -H bundle exec rake gitlab:packages:migrate
+ ```
-[Read more about using object storage with GitLab](../object_storage.md).
+ ::EndTabs
-NOTE:
-We recommend using the [consolidated object storage settings](../object_storage.md#consolidated-object-storage-configuration). The following instructions apply to the original configuration format.
+1. Track the progress and verify that all packages migrated successfully using
+ the PostgreSQL console.
-**Omnibus GitLab installations**
+ ::Tabs
-1. Edit `/etc/gitlab/gitlab.rb` and add the following lines (uncomment where
- necessary):
+ :::TabTitle Linux package (Omnibus) 14.1 and earlier
- ```ruby
- gitlab_rails['packages_enabled'] = true
- gitlab_rails['packages_object_store_enabled'] = true
- gitlab_rails['packages_object_store_remote_directory'] = "packages" # The bucket name.
- gitlab_rails['packages_object_store_proxy_download'] = false # Passthrough all downloads via GitLab instead of using Redirects to Object Storage.
- gitlab_rails['packages_object_store_connection'] = {
- ##
- ## If the provider is AWS S3, uncomment the following
- ##
- #'provider' => 'AWS',
- #'region' => 'eu-west-1',
- #'aws_access_key_id' => 'AWS_ACCESS_KEY_ID',
- #'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY',
- ## If an IAM profile is being used with AWS, omit the aws_access_key_id and aws_secret_access_key and uncomment
- #'use_iam_profile' => true,
- ##
- ## If the provider is other than AWS (an S3-compatible one), uncomment the following
- ##
- #'host' => 's3.amazonaws.com',
- #'aws_signature_version' => 4 # For creation of signed URLs. Set to 2 if provider does not support v4.
- #'endpoint' => 'https://s3.amazonaws.com' # Useful for S3-compliant services such as DigitalOcean Spaces.
- #'path_style' => false # If true, use 'host/bucket_name/object' instead of 'bucket_name.host/object'.
- }
+ ```shell
+ sudo gitlab-rails dbconsole
```
-1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure)
- for the changes to take effect.
-
-**Installations from source**
+ :::TabTitle Linux package (Omnibus) 14.2 and later
-1. Edit the `packages` section in `config/gitlab.yml` (uncomment where necessary):
-
- ```yaml
- packages:
- enabled: true
- ##
- ## The location where build packages are stored (default: shared/packages).
- ##
- # storage_path: shared/packages
- object_store:
- enabled: false
- remote_directory: packages # The bucket name.
- # proxy_download: false # Passthrough all downloads via GitLab instead of using Redirects to Object Storage.
- connection:
- ##
- ## If the provider is AWS S3, use the following:
- ##
- provider: AWS
- region: us-east-1
- aws_access_key_id: AWS_ACCESS_KEY_ID
- aws_secret_access_key: AWS_SECRET_ACCESS_KEY
- ##
- ## If the provider is other than AWS (an S3-compatible one), comment out the previous 4 lines and use the following instead:
- ##
- # host: 's3.amazonaws.com' # default: s3.amazonaws.com.
- # aws_signature_version: 4 # For creation of signed URLs. Set to 2 if provider does not support v4.
- # endpoint: 'https://s3.amazonaws.com' # Useful for S3-compliant services such as DigitalOcean Spaces.
- # path_style: false # If true, use 'host/bucket_name/object' instead of 'bucket_name.host/object'.
+ ```shell
+ sudo gitlab-rails dbconsole --database main
```
-1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect.
-
-### Migrating local packages to object storage
+ :::TabTitle Self-compiled (source)
-After [configuring the object storage](#using-object-storage), use the following task to
-migrate existing packages from the local storage to the remote storage.
-The processing is done in a background worker and requires **no downtime**.
+ ```shell
+ RAILS_ENV=production sudo -u git -H psql -d gitlabhq_production
+ ```
-For Omnibus GitLab:
+ ::EndTabs
-```shell
-sudo gitlab-rake "gitlab:packages:migrate"
-```
+1. Verify that all packages migrated to object storage with the following SQL
+ query. The number of `objectstg` should be the same as `total`:
-For installations from source:
+ ```shell
+ gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM packages_package_files;
-```shell
-RAILS_ENV=production sudo -u git -H bundle exec rake gitlab:packages:migrate
-```
+ total | filesystem | objectstg
+ ------+------------+-----------
+ 34 | 0 | 34
+ ```
-You can optionally track progress and verify that all packages migrated successfully using the
-[PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database):
+1. Finally, verify that there are no files on disk in the `packages` directory:
-- `sudo gitlab-rails dbconsole` for Omnibus GitLab 14.1 and earlier.
-- `sudo gitlab-rails dbconsole --database main` for Omnibus GitLab 14.2 and later.
-- `sudo -u git -H psql -d gitlabhq_production` for source-installed instances.
+ ::Tabs
-Verify `objectstg` below (where `file_store = '2'`) has count of all packages:
+ :::TabTitle Linux package (Omnibus)
-```shell
-gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM packages_package_files;
+ ```shell
+ sudo find /var/opt/gitlab/gitlab-rails/shared/packages -type f | grep -v tmp | wc -l
+ ```
-total | filesystem | objectstg
-------+------------+-----------
- 34 | 0 | 34
-```
+ :::TabTitle Self-compiled (source)
-Verify that there are no files on disk in the `packages` folder:
+ ```shell
+ sudo -u git find /home/git/gitlab/shared/packages -type f | grep -v tmp | wc -l
+ ```
-```shell
-sudo find /var/opt/gitlab/gitlab-rails/shared/packages -type f | grep -v tmp | wc -l
-```
+ ::EndTabs
diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md
index 3d31491a9d2..2a28df96ef4 100644
--- a/doc/administration/pages/index.md
+++ b/doc/administration/pages/index.md
@@ -247,20 +247,20 @@ control over how the Pages daemon runs and serves content in your environment.
| `enable` | Enable or disable GitLab Pages on the current system. |
| `external_http` | Configure Pages to bind to one or more secondary IP addresses, serving HTTP requests. Multiple addresses can be given as an array, along with exact ports, for example `['1.2.3.4', '1.2.3.5:8063']`. Sets value for `listen_http`. |
| `external_https` | Configure Pages to bind to one or more secondary IP addresses, serving HTTPS requests. Multiple addresses can be given as an array, along with exact ports, for example `['1.2.3.4', '1.2.3.5:8063']`. Sets value for `listen_https`. |
-| `server_shutdown_timeout` | GitLab Pages server shutdown timeout in seconds (default: 30s). |
-| `gitlab_client_http_timeout` | GitLab API HTTP client connection timeout in seconds (default: 10s). |
-| `gitlab_client_jwt_expiry` | JWT Token expiry time in seconds (default: 30s). |
-| `gitlab_cache_expiry` | The maximum time a domain's configuration is stored in the cache (default: 600s). |
-| `gitlab_cache_refresh` | The interval at which a domain's configuration is set to be due to refresh (default: 60s). |
-| `gitlab_cache_cleanup` | The interval at which expired items are removed from the cache (default: 60s). |
-| `gitlab_retrieval_timeout` | The maximum time to wait for a response from the GitLab API per request (default: 30s). |
-| `gitlab_retrieval_interval` | The interval to wait before retrying to resolve a domain's configuration via the GitLab API (default: 1s). |
+| `server_shutdown_timeout` | GitLab Pages server shutdown timeout in seconds (default: 30 s). |
+| `gitlab_client_http_timeout` | GitLab API HTTP client connection timeout in seconds (default: 10 s). |
+| `gitlab_client_jwt_expiry` | JWT Token expiry time in seconds (default: 30 s). |
+| `gitlab_cache_expiry` | The maximum time a domain's configuration is stored in the cache (default: 600 s). |
+| `gitlab_cache_refresh` | The interval at which a domain's configuration is set to be due to refresh (default: 60 s). |
+| `gitlab_cache_cleanup` | The interval at which expired items are removed from the cache (default: 60 s). |
+| `gitlab_retrieval_timeout` | The maximum time to wait for a response from the GitLab API per request (default: 30 s). |
+| `gitlab_retrieval_interval` | The interval to wait before retrying to resolve a domain's configuration via the GitLab API (default: 1 s). |
| `gitlab_retrieval_retries` | The maximum number of times to retry to resolve a domain's configuration via the API (default: 3). |
| `domain_config_source` | This parameter was removed in 14.0, on earlier versions it can be used to enable and test API domain configuration source |
| `gitlab_id` | The OAuth application public ID. Leave blank to automatically fill when Pages authenticates with GitLab. |
| `gitlab_secret` | The OAuth application secret. Leave blank to automatically fill when Pages authenticates with GitLab. |
| `auth_scope` | The OAuth application scope to use for authentication. Must match GitLab Pages OAuth application settings. Leave blank to use `api` scope by default. |
-| `auth_cookie_session_timeout` | Authentication cookie session timeout in seconds (default: 600s). A value of `0` means the cookie is deleted after the browser session ends. |
+| `auth_cookie_session_timeout` | Authentication cookie session timeout in seconds (default: 600 s). A value of `0` means the cookie is deleted after the browser session ends. |
| `gitlab_server` | Server to use for authentication when access control is enabled; defaults to GitLab `external_url`. |
| `headers` | Specify any additional http headers that should be sent to the client with each response. Multiple headers can be given as an array, header and value as one string, for example `['my-header: myvalue', 'my-other-header: my-other-value']` |
| `enable_disk` | Allows the GitLab Pages daemon to serve content from disk. Shall be disabled if shared disk storage isn't available. |
@@ -275,9 +275,9 @@ control over how the Pages daemon runs and serves content in your environment.
| `max_uri_length` | The maximum length of URIs accepted by GitLab Pages. Set to 0 for unlimited length. [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/659) in GitLab 14.5.
| `metrics_address` | The address to listen on for metrics requests. |
| `redirect_http` | Redirect pages from HTTP to HTTPS, true/false. |
-| `redirects_max_config_size` | The maximum size of the _redirects file, in bytes (default: 65536). |
-| `redirects_max_path_segments` | The maximum number of path segments allowed in _redirects rules URLs (default: 25). |
-| `redirects_max_rule_count` | The maximum number of rules allowed in _redirects (default: 1000). |
+| `redirects_max_config_size` | The maximum size of the `_redirects` file, in bytes (default: 65536). |
+| `redirects_max_path_segments` | The maximum number of path segments allowed in `_redirects` rules URLs (default: 25). |
+| `redirects_max_rule_count` | The maximum number of rules allowed in `_redirects` (default: 1000). |
| `sentry_dsn` | The address for sending Sentry crash reporting to. |
| `sentry_enabled` | Enable reporting and logging with Sentry, true/false. |
| `sentry_environment` | The environment for Sentry crash reporting. |
@@ -294,7 +294,7 @@ control over how the Pages daemon runs and serves content in your environment.
| `pages_path` | The directory on disk where pages are stored, defaults to `GITLAB-RAILS/shared/pages`. |
| **`pages_nginx[]`** | |
| `enable` | Include a virtual host `server{}` block for Pages inside NGINX. Needed for NGINX to proxy traffic back to the Pages daemon. Set to `false` if the Pages daemon should directly receive all requests, for example, when using [custom domains](index.md#custom-domains). |
-| `FF_ENABLE_PLACEHOLDERS` | Feature flag to enable/disable rewrites (disabled by default). Read the [redirects documentation](../../user/project/pages/redirects.md#feature-flag-for-rewrites) for more information. |
+| `FF_ENABLE_PLACEHOLDERS` | Feature flag for rewrites (enabled by default). See [Rewrites](../../user/project/pages/redirects.md#rewrites) for more information. |
| `use_legacy_storage` | Temporarily-introduced parameter allowing to use legacy domain configuration source and storage. [Removed in 14.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6166). |
| `rate_limit_source_ip` | Rate limit per source IP in number of requests per second. Set to `0` to disable this feature. |
| `rate_limit_source_ip_burst` | Rate limit per source IP maximum burst allowed per second. |
@@ -540,22 +540,22 @@ archive. You can modify the cache behavior by changing the following configurati
| Setting | Description |
| ------- | ----------- |
-| `zip_cache_expiration` | The cache expiration interval of ZIP archives. Must be greater than zero to avoid serving stale content. Default is 60s. |
-| `zip_cache_cleanup` | The interval at which archives are cleaned from memory if they have already expired. Default is 30s. |
-| `zip_cache_refresh` | The time interval in which an archive is extended in memory if accessed before `zip_cache_expiration`. This works together with `zip_cache_expiration` to determine if an archive is extended in memory. See the [example below](#zip-cache-refresh-example) for important details. Default is 30s. |
-| `zip_open_timeout` | The maximum time allowed to open a ZIP archive. Increase this time for big archives or slow network connections, as doing so may affect the latency of serving Pages. Default is 30s. |
-| `zip_http_client_timeout` | The maximum time for the ZIP HTTP client. Default is 30m. |
+| `zip_cache_expiration` | The cache expiration interval of ZIP archives. Must be greater than zero to avoid serving stale content. Default is 60 s. |
+| `zip_cache_cleanup` | The interval at which archives are cleaned from memory if they have already expired. Default is 30 s. |
+| `zip_cache_refresh` | The time interval in which an archive is extended in memory if accessed before `zip_cache_expiration`. This works together with `zip_cache_expiration` to determine if an archive is extended in memory. See the [example below](#zip-cache-refresh-example) for important details. Default is 30 s. |
+| `zip_open_timeout` | The maximum time allowed to open a ZIP archive. Increase this time for big archives or slow network connections, as doing so may affect the latency of serving Pages. Default is 30 s. |
+| `zip_http_client_timeout` | The maximum time for the ZIP HTTP client. Default is 30 m. |
#### ZIP cache refresh example
Archives are refreshed in the cache (extending the time they are held in memory) if they're accessed
before `zip_cache_expiration`, and the time left before expiring is less than or equal to
-`zip_cache_refresh`. For example, if `archive.zip` is accessed at time 0s, it expires in 60s (the
-default for `zip_cache_expiration`). In the example below, if the archive is opened again after 15s
-it is **not** refreshed because the time left for expiry (45s) is greater than `zip_cache_refresh`
-(default 30s). However, if the archive is accessed again after 45s (from the first time it was
+`zip_cache_refresh`. For example, if `archive.zip` is accessed at time 0 s, it expires in 60 s (the
+default for `zip_cache_expiration`). In the example below, if the archive is opened again after 15 s
+it is **not** refreshed because the time left for expiry (45 s) is greater than `zip_cache_refresh`
+(default 30 s). However, if the archive is accessed again after 45 s (from the first time it was
opened) it's refreshed. This extends the time the archive remains in memory from
-`45s + zip_cache_expiration (60s)`, for a total of 105s.
+`45s + zip_cache_expiration (60s)`, for a total of 105 s.
After an archive reaches `zip_cache_expiration`, it's marked as expired and removed on the next
`zip_cache_cleanup` interval.
@@ -941,6 +941,10 @@ If you want to stop using and disconnect the NFS server, you need to
#### S3-compatible connection settings
+In GitLab 13.2 and later, you should use the
+[consolidated object storage settings](../object_storage.md#consolidated-object-storage-configuration).
+This section describes the earlier configuration format.
+
See [the available connection settings for different providers](../object_storage.md#connection-settings).
In Omnibus installations:
diff --git a/doc/administration/postgresql/database_load_balancing.md b/doc/administration/postgresql/database_load_balancing.md
index 805c4b2023f..15129770888 100644
--- a/doc/administration/postgresql/database_load_balancing.md
+++ b/doc/administration/postgresql/database_load_balancing.md
@@ -76,18 +76,20 @@ Database Load Balancing can be configured in one of two ways:
### Hosts
-To configure a list of hosts, add the `gitlab_rails['db_load_balancing']` setting into the
-`gitlab.rb` file in the GitLab Rails / Sidekiq nodes for each environment you want to balance.
+To configure a list of hosts, perform these steps on all GitLab Rails (Sidekiq)
+nodes for each environment you want to balance:
-For example, on an environment that has PostgreSQL running on the hosts `host1.example.com`,
-`host2.example.com` and `host3.example.com` and reachable on the same port configured with
-`gitlab_rails['db_port']`:
+1. Edit the `/etc/gitlab/gitlab.rb` file.
+1. In `gitlab_rails['db_load_balancing']`, create an array of the read-only
+ replicas you want to balance. Do not add the primary host. For example, on
+ an environment with PostgreSQL running on the hosts `primary.example.com`,
+ `host1.example.com`, `host2.example.com`, and `host3.example.com`:
-1. On each GitLab Rails / Sidekiq node, edit `/etc/gitlab/gitlab.rb` and add the following line:
+ ```ruby
+ gitlab_rails['db_load_balancing'] = { 'hosts' => ['host1.example.com', 'host2.example.com', `host3.example.com`] }
+ ```
- ```ruby
- gitlab_rails['db_load_balancing'] = { 'hosts' => ['host1.example.com', 'host2.example.com', `host3.example.com`] }
- ```
+ These replicas must be reachable on the same port configured with `gitlab_rails['db_port']`.
1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
diff --git a/doc/administration/postgresql/multiple_databases.md b/doc/administration/postgresql/multiple_databases.md
new file mode 100644
index 00000000000..ffccbf861e7
--- /dev/null
+++ b/doc/administration/postgresql/multiple_databases.md
@@ -0,0 +1,141 @@
+---
+stage: Data Stores
+group: Pods
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Multiple Databases **(FREE SELF)**
+
+> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/6168) in GitLab 15.7.
+
+WARNING:
+This feature is not ready for production use
+
+By default, GitLab uses a single application database, referred to as the `main` database.
+
+To scale GitLab, you can configure GitLab to use multiple application databases.
+
+Due to [known issues](#known-issues), configuring GitLab with multiple databases is in [**Alpha**](../../policy/alpha-beta-support.md#alpha-features).
+
+## Known issues
+
+- Migrating data from the `main` database to the `ci` database is not supported or documented yet.
+- Once data is migrated to the `ci` database, you cannot migrate it back.
+
+## Set up multiple databases
+
+Use the following content to set up multiple databases with a new GitLab installation.
+
+There is no documentation for existing GitLab installations yet.
+
+After you have set up multiple databases, GitLab uses a second application database for
+[CI/CD features](../../ci/index.md), referred to as the `ci` database. For
+example, GitLab reads and writes to the `ci_pipelines` table in the `ci`
+database.
+
+WARNING:
+You must stop GitLab before setting up multiple databases. This prevents
+split-brain situations, where `main` data is written to the `ci` database, and
+the other way around.
+
+### Installations from source
+
+1. [Back up GitLab](../../raketasks/backup_restore.md)
+ in case of unforeseen issues.
+
+1. Stop GitLab:
+
+ ```shell
+ sudo service gitlab stop
+ ```
+
+1. Open `config/database.yml`, and add a `ci:` section under
+ `production:`. See `config/database.yml.decomposed-postgresql` for possible
+ values for this new `ci:` section. Once modified, the `config/database.yml` should
+ look like:
+
+ ```yaml
+ production:
+ main:
+ # ...
+ ci:
+ adapter: postgresql
+ encoding: unicode
+ database: gitlabhq_production_ci
+ # ...
+ ```
+
+1. Save the `config/database.yml` file.
+
+1. Create the `gitlabhq_production_ci` database:
+
+ ```shell
+ sudo -u postgres psql -d template1 -c "CREATE DATABASE gitlabhq_production OWNER git;"
+ sudo -u git -H bundle exec rake db:schema:load:ci
+ ```
+
+1. Lock writes for `ci` tables in `main` database, and the other way around:
+
+ ```shell
+ sudo -u git -H bundle exec rake gitlab:db:lock_writes
+ ```
+
+1. Restart GitLab:
+
+ ```shell
+ sudo service gitlab restart
+ ```
+
+### Omnibus GitLab installations
+
+1. [Back up GitLab](../../raketasks/backup_restore.md)
+ in case of unforeseen issues.
+
+1. Stop GitLab:
+
+ ```shell
+ sudo gitlab-ctl stop
+ ```
+
+1. Edit `/etc/gitlab/gitlab.rb` and add the following lines:
+
+ ```ruby
+ gitlab_rails['databases']['ci']['enable'] = true
+ gitlab_rails['databases']['ci']['db_database'] = 'gitlabhq_production_ci'
+ ```
+
+1. Save the `/etc/gitlab/gitlab.rb` file.
+
+1. Reconfigure GitLab:
+
+ ```shell
+ sudo gitlab-ctl reconfigure
+ ```
+
+1. Optional. Reconfiguring GitLab should create the `gitlabhq_production_ci`. If it did not, manually create the `gitlabhq_production_ci`:
+
+ ```shell
+ sudo gitlab-ctl start postgresql
+ sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql -d template1 -c "CREATE DATABASE gitlabhq_production_ci OWNER gitlab;"
+ sudo gitlab-rake db:schema:load:ci
+
+1. Lock writes for `ci` tables in `main` database, and the other way around:
+
+ ```shell
+ sudo gitlab-ctl start postgresql
+ sudo gitlab-rake gitlab:db:lock_writes
+ ```
+
+1. Restart GitLab:
+
+ ```shell
+ sudo gitlab-ctl restart
+ ```
+
+## Further information
+
+For more information on multiple databases, see [issue 6168](https://gitlab.com/groups/gitlab-org/-/epics/6168).
+
+For more information on how multiple databases work in GitLab, see the [development guide for multiple databases](../../development/database/multiple_databases.md).
+
+Since 2022-07-02, GitLab.com has been running with two separate databases. For more information, see this [blog post](https://about.gitlab.com/blog/2022/06/02/splitting-database-into-main-and-ci/).
diff --git a/doc/administration/postgresql/pgbouncer.md b/doc/administration/postgresql/pgbouncer.md
index 91c689fadea..25c4c940b97 100644
--- a/doc/administration/postgresql/pgbouncer.md
+++ b/doc/administration/postgresql/pgbouncer.md
@@ -220,8 +220,8 @@ the database. Each of the listed services below use the following formula to def
- `puma` : `max_threads + headroom` (default `14`)
- `max_threads` is configured via: `gitlab['puma']['max_threads']` (default: `4`)
- `headroom` can be configured via `DB_POOL_HEADROOM` environment variable (default to `10`)
-- `sidekiq` : `max_concurrency + 1 + headroom` (default: `61`)
- - `max_concurrency` is configured via: `sidekiq['max_concurrency']` (default: `50`)
+- `sidekiq` : `max_concurrency + 1 + headroom` (default: `31`)
+ - `max_concurrency` is configured via: `sidekiq['max_concurrency']` (default: `20`)
- `headroom` can be configured via `DB_POOL_HEADROOM` environment variable (default to `10`)
- `geo-logcursor`: `1+headroom` (default: `11`)
- `headroom` can be configured via `DB_POOL_HEADROOM` environment variable (default to `10`)
diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md
index ee90b120d05..ececa12f3ad 100644
--- a/doc/administration/postgresql/replication_and_failover.md
+++ b/doc/administration/postgresql/replication_and_failover.md
@@ -608,7 +608,7 @@ Here is a list and description of each machine and the assigned IP:
All passwords are set to `toomanysecrets`. Do not use this password or derived hashes and the `external_url` for GitLab is `http://gitlab.example.com`.
-After the initial configuration, if a failover occurs, the PostgresSQL leader node changes to one of the available secondaries until it is failed back.
+After the initial configuration, if a failover occurs, the PostgreSQL leader node changes to one of the available secondaries until it is failed back.
#### Example recommended setup for Consul servers
@@ -1081,6 +1081,190 @@ Reverting the PostgreSQL upgrade with `gitlab-ctl revert-pg-upgrade` has the sam
`gitlab-ctl pg-upgrade`. You should follow the same procedure by first stopping the replicas,
then reverting the leader, and finally reverting the replicas.
+### Near zero downtime upgrade of PostgreSQL in a Patroni cluster (Experimental)
+
+Patroni enables you to run a major PostgreSQL upgrade without shutting down the cluster. However, this
+requires additional resources to host the new Patroni nodes with the upgraded PostgreSQL. In practice, with this
+procedure, you are:
+
+- Creating a new Patroni cluster with a new version of PostgreSQL.
+- Migrating the data from the existing cluster.
+
+This procedure is non-invasive, and does not impact your existing cluster before switching it off.
+However, it can be both time- and resource-consuming. Consider their trade-offs with availability.
+
+The steps, in order:
+
+1. [Provision resources for the new cluster](#provision-resources-for-the-new-cluster).
+1. [Preflight check](#preflight-check).
+1. [Configure the leader of the new cluster](#configure-the-leader-of-the-new-cluster).
+1. [Start publisher on the existing leader](#start-publisher-on-the-existing-leader).
+1. [Copy the data from the existing cluster](#copy-the-data-from-the-existing-cluster).
+1. [Replicate data from the existing cluster](#replicate-data-from-the-existing-cluster).
+1. [Grow the new cluster](#grow-the-new-cluster).
+1. [Switch the application to use the new cluster](#switch-the-application-to-use-the-new-cluster).
+1. [Clean up](#clean-up).
+
+#### Provision resources for the new cluster
+
+You need a new set of resources for Patroni nodes. The new Patroni cluster does not require exactly the same number
+of nodes as the existing cluster. You may choose a different number of nodes based on your requirements. The new
+cluster uses the existing Consul cluster (with a different `patroni['scope']`) and PgBouncer nodes.
+
+Make sure that at least the leader node of the existing cluster is accessible from the nodes of the new
+cluster.
+
+#### Preflight check
+
+We rely on PostgreSQL [logical replication](https://www.postgresql.org/docs/current/logical-replication.html)
+to support near-zero downtime upgrades of Patroni clusters. The of
+[logical replication requirements](https://www.postgresql.org/docs/current/logical-replication-restrictions.html)
+must be met. In particular, `wal_level` must be `logical`. To check the `wal_level`,
+run the following command with `gitlab-psql` on any node of the existing cluster:
+
+```sql
+SHOW wal_level;
+```
+
+By default, Patroni sets `wal_level` to `replica`. You must increase it to `logical`.
+Changing `wal_level` requires restarting PostgreSQL, so this step leads to a short
+downtime (hence near-zero downtime). To do this on the Patroni **leader** node:
+
+1. Edit `gitlab.rb` by setting:
+
+ ```ruby
+ patroni['postgresql']['wal_level'] = 'logical'
+ ```
+
+1. Run `gitlab-ctl reconfigure`. This writes the configuration but does not restart PostgreSQL service.
+1. Run `gitlab-ctl patroni restart` to restart PostgreSQL and apply the new `wal_level` without triggering
+ failover. For the duration of restart cycle, the cluster leader is unavailable.
+1. Verify the change by running `SHOW wal_level` with `gitlab-psql`.
+
+#### Configure the leader of the new cluster
+
+Configure the first node of the new cluster. It becomes the leader of the new cluster.
+You can use the configuration of the existing cluster, if it is compatible with the new
+PostgreSQL version. Refer to the documentation on [configuring Patroni clusters](#configuring-patroni-cluster).
+
+In addition to the common configuration, you must apply the following in `gitlab.rb` to:
+
+1. Make sure that the new Patroni cluster uses a different scope. The scope is used to namespace the Patroni settings
+ in Consul, making it possible to use the same Consul cluster for the existing and the new clusters.
+
+ ```ruby
+ patroni['scope'] = 'postgresql_new-ha'
+ ```
+
+1. Make sure that Consul agents don't mix PostgreSQL services offered by the existing and the new Patroni
+ clusters. For this purpose, you must use an internal attribute that is currently undocumented:
+
+ ```ruby
+ consul['internal']['postgresql_service_name'] = 'postgresql_new'
+ ```
+
+#### Start publisher on the existing leader
+
+On the existing leader, run this SQL statement with `gitlab-psql` to start a logical replication publisher:
+
+```sql
+CREATE PUBLICATION patroni_upgrade FOR ALL TABLES;
+```
+
+#### Copy the data from the existing cluster
+
+To dump the current database from the existing cluster, run these commands on the
+**leader** of the new cluster:
+
+1. Optional. Copy global database objects:
+
+ ```shell
+ pg_dumpall -h ${EXISTING_CLUSTER_LEADER} -U gitlab-psql -g | gitlab-psql
+ ```
+
+ You can ignore the errors about existing database objects, such as roles. They are
+ created when the node is configured for the first time.
+
+1. Copy the current database:
+
+ ```shell
+ pg_dump -h ${EXISTING_CLUSTER_LEADER} -U gitlab-psql -d gitlabhq_production -s | gitlab-psql
+ ```
+
+ Depending on the size of your database, this command may take a while to complete.
+
+The `pg_dump` and `pg_dumpall` commands are in `/opt/gitlab/embedded/bin`. In these commands,
+`EXISTING_CLUSTER_LEADER` is the host address of the leader node of the existing cluster.
+
+NOTE:
+The `gitlab-psql` user must be able to authenticate the existing leader from the new leader node.
+
+#### Replicate data from the existing cluster
+
+After taking the initial data dump, you must keep the new leader in sync with the
+latest changes of your existing cluster. On the new leader, run this SQL statement
+with `gitlab-psql` to subscribe to publication of the existing leader:
+
+```sql
+CREATE SUBSCRIPTION patroni_upgrade
+ CONNECTION 'host=EXISTING_CLUSTER_LEADER dbname=gitlabhq_production user=gitlab-psql'
+ PUBLICATION patroni_upgrade;
+```
+
+In this statement, `EXISTING_CLUSTER_LEADER` is the host address of the leader node
+of the existing cluster. You can also use
+[other parameters](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS)
+to change the connection string. For example, you can pass the authentication password.
+
+To check the status of replication, run these queries:
+
+- `SELECT * FROM pg_replication_slots WHERE slot_name = 'patroni_upgrade'` on the existing leader (the publisher).
+- `SELECT * FROM pg_stat_subscription` on the new leader (the subscriber).
+
+#### Grow the new cluster
+
+Configure other nodes of the new cluster in the way you
+[configured the leader](#configure-the-leader-of-the-new-cluster).
+Make sure that you use the same `patroni['scope']` and
+`consul['internal']['postgresql_service_name']`.
+
+What happens here:
+
+- The application still uses the existing leader as its database backend.
+- The logical replication ensures that the new leader keeps in sync.
+- When other nodes are added to the new cluster, Patroni handles
+ the replication to the these nodes.
+
+It is a good idea to wait until the replica nodes of the new cluster are initialized and caught up on the replication
+lag.
+
+#### Switch the application to use the new cluster
+
+Up to this point, you can stop the upgrade procedure without losing data on the
+existing cluster. When you switch the database backend of the application and point
+it to the new cluster, the old cluster does not receive new updates. It falls behind
+the new cluster. After this point, any recovery must be done from the nodes of the new cluster.
+
+To do the switch on **all** PgBouncer nodes:
+
+1. Edit `gitlab.rb` by setting:
+
+ ```ruby
+ consul['watchers'] = %w(postgresql_new)
+ consul['internal']['postgresql_service_name'] = 'postgresql_new'
+ ```
+
+1. Run `gitlab-ctl reconfigure`.
+1. You must also run `rm /var/opt/gitlab/consul/config.d/watcher_postgresql.json`.
+ This is a [known issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7293).
+
+#### Clean up
+
+After completing these steps, then you can clean up the resources of the old Patroni cluster.
+They are no longer needed. However, before removing the resources, remove the
+logical replication subscription on the new leader by running `DROP SUBSCRIPTION patroni_upgrade`
+with `gitlab-psql`.
+
## Troubleshooting
### Consul and PostgreSQL changes not taking effect
diff --git a/doc/administration/raketasks/geo.md b/doc/administration/raketasks/geo.md
index a4d027101dd..03b09e00f1c 100644
--- a/doc/administration/raketasks/geo.md
+++ b/doc/administration/raketasks/geo.md
@@ -4,7 +4,7 @@ group: Geo
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
-# Geo Rake Tasks **(PREMIUM SELF)**
+# Geo Rake tasks **(PREMIUM SELF)**
The following Rake tasks are for [Geo installations](../geo/index.md).
See also [troubleshooting Geo](../geo/replication/troubleshooting.md) for additional Geo Rake tasks.
diff --git a/doc/administration/raketasks/github_import.md b/doc/administration/raketasks/github_import.md
index 224ed63d3e6..61f6137e1ed 100644
--- a/doc/administration/raketasks/github_import.md
+++ b/doc/administration/raketasks/github_import.md
@@ -4,7 +4,7 @@ group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
-# GitHub import **(FREE SELF)**
+# GitHub import Rake task **(FREE SELF)**
To retrieve and import GitHub repositories, you need a [GitHub personal access token](https://github.com/settings/tokens).
A username should be passed as the second argument to the Rake task,
diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md
index 293efb1b7ae..ba095b33bf5 100644
--- a/doc/administration/raketasks/maintenance.md
+++ b/doc/administration/raketasks/maintenance.md
@@ -301,7 +301,7 @@ sudo gitlab-rake gitlab:exclusive_lease:clear[project_housekeeping:4]
## Display status of database migrations
-See the [upgrade documentation](../../update/index.md#checking-for-background-migrations-before-upgrading)
+See the [background migrations documentation](../../update/background_migrations.md)
for how to check that migrations are complete when upgrading GitLab.
To check the status of specific migrations, you can use the following Rake task:
diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md
index a2463c6ff88..88913eb1f7f 100644
--- a/doc/administration/reference_architectures/10k_users.md
+++ b/doc/administration/reference_architectures/10k_users.md
@@ -35,13 +35,13 @@ full list of reference architectures, see
| GitLab Rails | 3 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` |
| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` |
| Object storage<sup>4</sup> | - | - | - | - |
-| NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` |
<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix -->
<!-- markdownlint-disable MD029 -->
1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
- [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work.
- - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB clusters](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo.
+ - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo.
+ - Note that [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported.
- [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440).
- Consul is primarily used for Omnibus PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery.
2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
@@ -189,7 +189,7 @@ CI pipelines alike.
As such, large repositories come with notable cost and typically will require more resources to handle,
significantly so in some cases. It's therefore **strongly** recommended then to review large repositories
-to ensure they maintain good repo health and reduce their size wherever possible.
+to ensure they maintain good health and reduce their size wherever possible.
NOTE:
If best practices aren't followed and large repositories are present on the environment,
@@ -227,9 +227,6 @@ To set up GitLab and its components to accommodate up to 10,000 users:
environment.
1. [Configure the object storage](#configure-the-object-storage)
used for shared data objects.
-1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended)
- to have shared disk storage service as an alternative to Gitaly or object
- storage.
1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster,
more advanced code search across your entire GitLab instance.
@@ -1275,7 +1272,7 @@ in the second step, do not supply the `EXTERNAL_URL` value.
# PostgreSQL configuration
postgresql['listen_address'] = '0.0.0.0'
- postgresql['max_connections'] = 200
+ postgresql['max_connections'] = 500
# Prevent database migrations from running on upgrade automatically
gitlab_rails['auto_migrate'] = false
@@ -1759,9 +1756,8 @@ To configure Praefect with TLS:
Sidekiq requires connection to the [Redis](#configure-redis),
[PostgreSQL](#configure-postgresql) and [Gitaly](#configure-gitaly) instances.
-Since it's recommended to use [Object storage](#configure-the-object-storage)
-over [NFS](#configure-nfs-optional) for data objects, the following examples
-include the Object storage configuration.
+Because you must use [Object storage](#configure-the-object-storage) instead of NFS for data objects, the following
+examples include the Object storage configuration.
- `10.6.0.101`: Sidekiq 1
- `10.6.0.102`: Sidekiq 2
@@ -1859,8 +1855,8 @@ Updates to example must be made at:
# Set number of Sidekiq queue processes to the same number as available CPUs
sidekiq['queue_groups'] = ['*'] * 4
- # Set number of Sidekiq threads per queue process to the recommend number of 10
- sidekiq['max_concurrency'] = 10
+ # Set number of Sidekiq threads per queue process to the recommend number of 20
+ sidekiq['max_concurrency'] = 20
# Monitoring
consul['enable'] = true
@@ -1918,7 +1914,7 @@ Updates to example must be made at:
NOTE:
If you find that the environment's Sidekiq job processing is slow with long queues,
more nodes can be added as required. You can also tune your Sidekiq nodes to
-run [multiple Sidekiq processes](../operations/extra_sidekiq_processes.md).
+run [multiple Sidekiq processes](../sidekiq/extra_sidekiq_processes.md).
<div align="right">
<a type="button" class="btn btn-default" href="#setup-components">
@@ -1929,9 +1925,8 @@ run [multiple Sidekiq processes](../operations/extra_sidekiq_processes.md).
## Configure GitLab Rails
This section describes how to configure the GitLab application (Rails) component.
-Since it's recommended to use [Object storage](#configure-the-object-storage)
-over [NFS](#configure-nfs-optional) for data objects, the following examples
-include the Object storage configuration.
+Because you must use [Object storage](#configure-the-object-storage) instead of NFS for data objects, the following
+examples include the Object storage configuration.
The following IPs will be used as an example:
@@ -2070,7 +2065,6 @@ On each node perform the following:
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
-
1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run:
```shell
@@ -2081,9 +2075,7 @@ On each node perform the following:
[GitLab Rails post-configuration](#gitlab-rails-post-configuration) section.
1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
-
-1. [Enable incremental logging](#enable-incremental-logging), unless you are using [NFS](#configure-nfs-optional).
-
+1. [Enable incremental logging](#enable-incremental-logging).
1. Confirm the node can connect to Gitaly:
```shell
@@ -2209,9 +2201,6 @@ To configure the Monitoring node:
## Configure the object storage
GitLab supports using an object storage service for holding numerous types of data.
-It's recommended over [NFS](#configure-nfs-optional) and in general it's better
-in larger setups as object storage is typically much more performant, reliable,
-and scalable.
GitLab has been tested on a number of object storage providers:
@@ -2235,7 +2224,7 @@ NOTE:
When using the [storage-specific form](../object_storage.md#storage-specific-configuration)
in GitLab 14.x and earlier, you should enable [direct upload mode](../../development/uploads/index.md#direct-upload).
The previous [background upload](../../development/uploads/index.md#direct-upload) mode,
-which was deprecated in 14.9, requires shared storage such as [NFS](#configure-nfs-optional).
+which was deprecated in 14.9, requires shared storage such as NFS.
Using separate buckets for each data type is the recommended approach for GitLab.
This ensures there are no collisions across the various types of data GitLab stores.
@@ -2254,22 +2243,6 @@ GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily
While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs.
-## Configure NFS (optional)
-
-[Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly)
-are recommended over NFS wherever possible for improved performance.
-
-See how to [configure NFS](../nfs.md).
-
-WARNING:
-Engineering support for NFS for Git repositories is deprecated, and [technical support is scheduled to be unavailable](../nfs.md#gitaly-and-nfs-deprecation)
-after the release of GitLab 15.6. No further enhancements are planned for this feature.
-
-Read:
-
-- [Gitaly and NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation).
-- About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss).
-
## Configure Advanced Search
You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md)
@@ -2319,12 +2292,10 @@ Refer to [epic 6127](https://gitlab.com/groups/gitlab-org/-/epics/6127) for more
The following tables and diagram detail the hybrid environment using the same formats
as the normal environment above.
-First are the components that run in Kubernetes. The recommendation at this time is to
-use Google Cloud's Kubernetes Engine (GKE) or AWS Elastic Kubernetes Service (EKS) and associated machine types, but the memory
-and CPU requirements should translate to most other providers. We hope to update this in the
-future with further specific cloud provider details.
+First are the components that run in Kubernetes. These run across several node groups, although you can change
+the overall makeup as desired as long as the minimum CPU and Memory requirements are observed.
-| Service | Nodes | Configuration | GCP | AWS | Min Allocatable CPUs and Memory |
+| Service Node Group | Nodes | Configuration | GCP | AWS | Min Allocatable CPUs and Memory |
|---------------------|-------|-------------------------|-----------------|--------------|---------------------------------|
| Webservice | 4 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | 127.5 vCPU, 118 GB memory |
| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | 15.5 vCPU, 50 GB memory |
@@ -2333,7 +2304,7 @@ future with further specific cloud provider details.
- For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results)
[Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary.
- Nodes configuration is shown as it is forced to ensure pod vCPU / memory ratios and avoid scaling during **performance testing**.
- - In production deployments, there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices.
+ - In production deployments, there is no need to assign pods to specific nodes. A minimum of three nodes per node group in three different availability zones is strongly recommended to align with resilient cloud architecture practices.
Next are the backend components that run on static compute VMs via Omnibus (or External PaaS
services where applicable):
@@ -2355,7 +2326,8 @@ services where applicable):
<!-- markdownlint-disable MD029 -->
1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
- [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work.
- - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB clusters](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo.
+ - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo.
+ - Note that [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported.
- [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440).
- Consul is primarily used for Omnibus PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery.
2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md
index 84eba01fe11..02739904f5e 100644
--- a/doc/administration/reference_architectures/25k_users.md
+++ b/doc/administration/reference_architectures/25k_users.md
@@ -35,13 +35,13 @@ full list of reference architectures, see
| GitLab Rails | 5 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` |
| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` |
| Object storage<sup>4</sup> | - | - | - | - |
-| NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` |
<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix -->
<!-- markdownlint-disable MD029 -->
1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
- [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work.
- - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB clusters](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo.
+ - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo.
+ - Note that [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported.
- [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440).
- Consul is primarily used for Omnibus PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery.
2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
@@ -189,7 +189,7 @@ CI pipelines alike.
As such, large repositories come with notable cost and typically will require more resources to handle,
significantly so in some cases. It's therefore **strongly** recommended then to review large repositories
-to ensure they maintain good repo health and reduce their size wherever possible.
+to ensure they maintain good health and reduce their size wherever possible.
NOTE:
If best practices aren't followed and large repositories are present on the environment,
@@ -227,9 +227,6 @@ To set up GitLab and its components to accommodate up to 25,000 users:
environment.
1. [Configure the object storage](#configure-the-object-storage)
used for shared data objects.
-1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended)
- to have shared disk storage service as an alternative to Gitaly or object
- storage.
1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster,
more advanced code search across your entire GitLab instance.
@@ -1295,7 +1292,7 @@ in the second step, do not supply the `EXTERNAL_URL` value.
# PostgreSQL configuration
postgresql['listen_address'] = '0.0.0.0'
- postgresql['max_connections'] = 200
+ postgresql['max_connections'] = 500
# Prevent database migrations from running on upgrade automatically
gitlab_rails['auto_migrate'] = false
@@ -1777,9 +1774,8 @@ To configure Praefect with TLS:
Sidekiq requires connection to the [Redis](#configure-redis),
[PostgreSQL](#configure-postgresql) and [Gitaly](#configure-gitaly) instances.
-Since it's recommended to use [Object storage](#configure-the-object-storage)
-over [NFS](#configure-nfs-optional) for data objects, the following examples
-include the Object storage configuration.
+Because you must use [Object storage](#configure-the-object-storage) instead of NFS for data objects, the following
+examples include the Object storage configuration.
- `10.6.0.101`: Sidekiq 1
- `10.6.0.102`: Sidekiq 2
@@ -1877,8 +1873,8 @@ Updates to example must be made at:
# Set number of Sidekiq queue processes to the same number as available CPUs
sidekiq['queue_groups'] = ['*'] * 4
- # Set number of Sidekiq threads per queue process to the recommend number of 10
- sidekiq['max_concurrency'] = 10
+ # Set number of Sidekiq threads per queue process to the recommend number of 20
+ sidekiq['max_concurrency'] = 20
# Monitoring
consul['enable'] = true
@@ -1936,7 +1932,7 @@ Updates to example must be made at:
NOTE:
If you find that the environment's Sidekiq job processing is slow with long queues,
more nodes can be added as required. You can also tune your Sidekiq nodes to
-run [multiple Sidekiq processes](../operations/extra_sidekiq_processes.md).
+run [multiple Sidekiq processes](../sidekiq/extra_sidekiq_processes.md).
<div align="right">
<a type="button" class="btn btn-default" href="#setup-components">
@@ -1947,9 +1943,8 @@ run [multiple Sidekiq processes](../operations/extra_sidekiq_processes.md).
## Configure GitLab Rails
This section describes how to configure the GitLab application (Rails) component.
-Since it's recommended to use [Object storage](#configure-the-object-storage)
-over [NFS](#configure-nfs-optional) for data objects, the following examples
-include the Object storage configuration.
+Because you must use [Object storage](#configure-the-object-storage) instead of NFS for data objects, the following
+examples include the Object storage configuration.
The following IPs will be used as an example:
@@ -2090,7 +2085,6 @@ On each node perform the following:
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
-
1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run:
```shell
@@ -2101,9 +2095,7 @@ On each node perform the following:
[GitLab Rails post-configuration](#gitlab-rails-post-configuration) section.
1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
-
-1. [Enable incremental logging](#enable-incremental-logging), unless you are using [NFS](#configure-nfs-optional).
-
+1. [Enable incremental logging](#enable-incremental-logging).
1. Confirm the node can connect to Gitaly:
```shell
@@ -2228,9 +2220,6 @@ To configure the Monitoring node:
## Configure the object storage
GitLab supports using an object storage service for holding numerous types of data.
-It's recommended over [NFS](#configure-nfs-optional) and in general it's better
-in larger setups as object storage is typically much more performant, reliable,
-and scalable.
GitLab has been tested on a number of object storage providers:
@@ -2254,7 +2243,7 @@ NOTE:
When using the [storage-specific form](../object_storage.md#storage-specific-configuration)
in GitLab 14.x and earlier, you should enable [direct upload mode](../../development/uploads/index.md#direct-upload).
The previous [background upload](../../development/uploads/index.md#direct-upload) mode,
-which was deprecated in 14.9, requires shared storage such as [NFS](#configure-nfs-optional).
+which was deprecated in 14.9, requires shared storage such as NFS.
Using separate buckets for each data type is the recommended approach for GitLab.
This ensures there are no collisions across the various types of data GitLab stores.
@@ -2273,22 +2262,6 @@ GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily
While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs.
-## Configure NFS (optional)
-
-[Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly)
-are recommended over NFS wherever possible for improved performance.
-
-See how to [configure NFS](../nfs.md).
-
-WARNING:
-Engineering support for NFS for Git repositories is deprecated, and [technical support is scheduled to be unavailable](../nfs.md#gitaly-and-nfs-deprecation)
-after the release of GitLab 15.6. No further enhancements are planned for this feature.
-
-Read:
-
-- [Gitaly and NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation).
-- About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss).
-
## Configure Advanced Search
You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md)
@@ -2338,12 +2311,10 @@ Refer to [epic 6127](https://gitlab.com/groups/gitlab-org/-/epics/6127) for more
The following tables and diagram detail the hybrid environment using the same formats
as the normal environment above.
-First are the components that run in Kubernetes. The recommendation at this time is to
-use Google Cloud's Kubernetes Engine (GKE) or AWS Elastic Kubernetes Service (EKS) and associated machine types, but the memory
-and CPU requirements should translate to most other providers. We hope to update this in the
-future with further specific cloud provider details.
+First are the components that run in Kubernetes. These run across several node groups, although you can change
+the overall makeup as desired as long as the minimum CPU and Memory requirements are observed.
-| Service | Nodes | Configuration | GCP | AWS | Min Allocatable CPUs and Memory |
+| Service Node Group | Nodes | Configuration | GCP | AWS | Min Allocatable CPUs and Memory |
|---------------------|-------|-------------------------|-----------------|--------------|---------------------------------|
| Webservice | 7 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | 223 vCPU, 206.5 GB memory |
| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | 15.5 vCPU, 50 GB memory |
@@ -2352,7 +2323,7 @@ future with further specific cloud provider details.
- For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results)
[Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary.
- Nodes configuration is shown as it is forced to ensure pod vCPU / memory ratios and avoid scaling during **performance testing**.
- - In production deployments, there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices.
+ - In production deployments, there is no need to assign pods to specific nodes. A minimum of three nodes per node group in three different availability zones is strongly recommended to align with resilient cloud architecture practices.
Next are the backend components that run on static compute VMs via Omnibus (or External PaaS
services where applicable):
@@ -2362,7 +2333,7 @@ services where applicable):
| Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` |
| PostgreSQL<sup>1</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` |
| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` |
-| Internal load balancing node<sup>3</sup> | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | `c5.xlarge` |
+| Internal load balancing node<sup>3</sup> | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` |
| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` |
| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` |
| Gitaly<sup>5 6</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` |
@@ -2374,7 +2345,8 @@ services where applicable):
<!-- markdownlint-disable MD029 -->
1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
- [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work.
- - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB clusters](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo.
+ - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo.
+ - Note that [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported.
- [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440).
- Consul is primarily used for Omnibus PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery.
2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md
index 1acae93f764..f41c8e9cb24 100644
--- a/doc/administration/reference_architectures/2k_users.md
+++ b/doc/administration/reference_architectures/2k_users.md
@@ -29,12 +29,12 @@ For a full list of reference architectures, see
| GitLab Rails | 2 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` |
| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` |
| Object storage<sup>4</sup> | - | - | - | - | - |
-| NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` |
<!-- markdownlint-disable MD029 -->
1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
- [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work.
- - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB clusters](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo.
+ - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo.
+ - Note that [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported.
- [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and [Azure Database for PostgreSQL](https://azure.microsoft.com/en-gb/products/postgresql/#overview) is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61).
- Consul is primarily used for Omnibus PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery.
2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
@@ -125,7 +125,7 @@ CI pipelines alike.
As such, large repositories come with notable cost and typically will require more resources to handle,
significantly so in some cases. It's therefore **strongly** recommended then to review large repositories
-to ensure they maintain good repo health and reduce their size wherever possible.
+to ensure they maintain good health and reduce their size wherever possible.
NOTE:
If best practices aren't followed and large repositories are present on the environment,
@@ -151,9 +151,6 @@ To set up GitLab and its components to accommodate up to 2,000 users:
environment.
1. [Configure the object storage](#configure-the-object-storage) used for
shared data objects.
-1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended)
- to have shared disk storage service as an alternative to Gitaly or object
- storage.
1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster,
more advanced code search across your entire GitLab instance.
@@ -700,8 +697,8 @@ On each node perform the following:
puma['listen'] = '0.0.0.0'
sidekiq['listen_address'] = "0.0.0.0"
- # Configure Sidekiq with 2 workers and 10 max concurrency
- sidekiq['max_concurrency'] = 10
+ # Configure Sidekiq with 2 workers and 20 max concurrency
+ sidekiq['max_concurrency'] = 20
sidekiq['queue_groups'] = ['*'] * 2
# Add the monitoring node's IP address to the monitoring whitelist and allow it to
@@ -780,9 +777,7 @@ On each node perform the following:
[GitLab Rails post-configuration](#gitlab-rails-post-configuration) section.
1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
-
-1. [Enable incremental logging](#enable-incremental-logging), unless you are using [NFS](#configure-nfs-optional).
-
+1. [Enable incremental logging](#enable-incremental-logging).
1. Run `sudo gitlab-rake gitlab:gitaly:check` to confirm the node can connect to Gitaly.
1. Tail the logs to see the requests:
@@ -930,9 +925,6 @@ running [Prometheus](../monitoring/prometheus/index.md) and
## Configure the object storage
GitLab supports using an object storage service for holding numerous types of data.
-It's recommended over [NFS](#configure-nfs-optional) and in general it's better
-in larger setups as object storage is typically much more performant, reliable,
-and scalable.
GitLab has been tested on a number of object storage providers:
@@ -957,7 +949,7 @@ NOTE:
When using the [storage-specific form](../object_storage.md#storage-specific-configuration)
in GitLab 14.x and earlier, you should enable [direct upload mode](../../development/uploads/index.md#direct-upload).
The previous [background upload](../../development/uploads/index.md#direct-upload) mode,
-which was deprecated in 14.9, requires shared storage such as [NFS](#configure-nfs-optional).
+which was deprecated in 14.9, requires shared storage such as NFS.
Using separate buckets for each data type is the recommended approach for GitLab.
This ensures there are no collisions across the various types of data GitLab stores.
@@ -976,23 +968,6 @@ GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily
While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs.
-## Configure NFS (optional)
-
-For improved performance, [object storage](#configure-the-object-storage),
-along with [Gitaly](#configure-gitaly), are recommended over using NFS whenever
-possible.
-
-See how to [configure NFS](../nfs.md).
-
-WARNING:
-Engineering support for NFS for Git repositories is deprecated, and [technical support is scheduled to be unavailable](../nfs.md#gitaly-and-nfs-deprecation)
-after the release of GitLab 15.6. No further enhancements are planned for this feature.
-
-Read:
-
-- [Gitaly and NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation).
-- About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss).
-
## Configure Advanced Search **(PREMIUM SELF)**
You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md)
@@ -1046,12 +1021,10 @@ Refer to [epic 6127](https://gitlab.com/groups/gitlab-org/-/epics/6127) for more
The following tables and diagram detail the hybrid environment using the same formats
as the normal environment above.
-First are the components that run in Kubernetes. The recommendation at this time is to
-use Google Cloud's Kubernetes Engine (GKE) or AWS Elastic Kubernetes Service (EKS) and associated machine types, but the memory
-and CPU requirements should translate to most other providers. We hope to update this in the
-future with further specific cloud provider details.
+First are the components that run in Kubernetes. These run across several node groups, although you can change
+the overall makeup as desired as long as the minimum CPU and Memory requirements are observed.
-| Service | Nodes | Configuration | GCP | AWS | Min Allocatable CPUs and Memory |
+| Service Node Group | Nodes | Configuration | GCP | AWS | Min Allocatable CPUs and Memory |
|---------------------|-------|------------------------|-----------------|--------------|---------------------------------|
| Webservice | 3 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | 23.7 vCPU, 16.9 GB memory |
| Sidekiq | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | 7.8 vCPU, 25.9 GB memory |
@@ -1060,7 +1033,7 @@ future with further specific cloud provider details.
- For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results)
[Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary.
- Nodes configuration is shown as it is forced to ensure pod vCPU / memory ratios and avoid scaling during **performance testing**.
- - In production deployments, there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices.
+ - In production deployments, there is no need to assign pods to specific nodes. A minimum of three nodes per node group in three different availability zones is strongly recommended to align with resilient cloud architecture practices.
Next are the backend components that run on static compute VMs via Omnibus (or External PaaS
services where applicable):
@@ -1076,7 +1049,8 @@ services where applicable):
<!-- markdownlint-disable MD029 -->
1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
- [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work.
- - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB clusters](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo.
+ - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo.
+ - Note that [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported.
- [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and [Azure Database for PostgreSQL](https://azure.microsoft.com/en-gb/products/postgresql/#overview) is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61).
- Consul is primarily used for Omnibus PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery.
2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md
index 4fc6af3f72e..008b5ffcc0e 100644
--- a/doc/administration/reference_architectures/3k_users.md
+++ b/doc/administration/reference_architectures/3k_users.md
@@ -44,13 +44,13 @@ For a full list of reference architectures, see
| GitLab Rails | 3 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` |
| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` |
| Object storage<sup>4</sup> | - | - | - | - |
-| NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` |
<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix -->
<!-- markdownlint-disable MD029 -->
1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
- [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work.
- - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB clusters](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo.
+ - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo.
+ - Note that [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported.
- [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440).
- Consul is primarily used for Omnibus PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery.
2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
@@ -195,7 +195,7 @@ CI pipelines alike.
As such, large repositories come with notable cost and typically will require more resources to handle,
significantly so in some cases. It's therefore **strongly** recommended then to review large repositories
-to ensure they maintain good repo health and reduce their size wherever possible.
+to ensure they maintain good health and reduce their size wherever possible.
NOTE:
If best practices aren't followed and large repositories are present on the environment,
@@ -233,9 +233,6 @@ To set up GitLab and its components to accommodate up to 3,000 users:
environment.
1. [Configure the object storage](#configure-the-object-storage)
used for shared data objects.
-1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended)
- to have shared disk storage service as an alternative to Gitaly or object
- storage.
1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster,
more advanced code search across your entire GitLab instance.
@@ -1230,7 +1227,7 @@ in the second step, do not supply the `EXTERNAL_URL` value.
# PostgreSQL configuration
postgresql['listen_address'] = '0.0.0.0'
- postgresql['max_connections'] = 200
+ postgresql['max_connections'] = 500
# Prevent database migrations from running on upgrade automatically
gitlab_rails['auto_migrate'] = false
@@ -1711,9 +1708,8 @@ To configure Praefect with TLS:
Sidekiq requires connection to the [Redis](#configure-redis),
[PostgreSQL](#configure-postgresql) and [Gitaly](#configure-gitaly) instances.
-Since it's recommended to use [Object storage](#configure-the-object-storage)
-over [NFS](#configure-nfs-optional) for data objects, the following examples
-include the Object storage configuration.
+Because you must use [Object storage](#configure-the-object-storage) instead of NFS for data objects, the following
+examples include the Object storage configuration.
The following IPs will be used as an example:
@@ -1794,8 +1790,8 @@ Updates to example must be made at:
## Set number of Sidekiq queue processes to the same number as available CPUs
sidekiq['queue_groups'] = ['*'] * 2
- ## Set number of Sidekiq threads per queue process to the recommend number of 10
- sidekiq['max_concurrency'] = 10
+ ## Set number of Sidekiq threads per queue process to the recommend number of 20
+ sidekiq['max_concurrency'] = 20
# Monitoring
consul['enable'] = true
@@ -1869,7 +1865,7 @@ Updates to example must be made at:
NOTE:
If you find that the environment's Sidekiq job processing is slow with long queues,
more nodes can be added as required. You can also tune your Sidekiq nodes to
-run [multiple Sidekiq processes](../operations/extra_sidekiq_processes.md).
+run [multiple Sidekiq processes](../sidekiq/extra_sidekiq_processes.md).
<div align="right">
<a type="button" class="btn btn-default" href="#setup-components">
@@ -1880,9 +1876,8 @@ run [multiple Sidekiq processes](../operations/extra_sidekiq_processes.md).
## Configure GitLab Rails
This section describes how to configure the GitLab application (Rails) component.
-Since it's recommended to use [Object storage](#configure-the-object-storage)
-over [NFS](#configure-nfs-optional) for data objects, the following examples
-include the Object storage configuration.
+Because you must use [Object storage](#configure-the-object-storage) instead of NFS for data objects, the following
+examples include the Object storage configuration.
On each node perform the following:
@@ -2021,7 +2016,6 @@ On each node perform the following:
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
-
1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run:
```shell
@@ -2032,11 +2026,8 @@ On each node perform the following:
[GitLab Rails post-configuration](#gitlab-rails-post-configuration) section.
1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
-
-1. [Enable incremental logging](#enable-incremental-logging), unless you are using [NFS](#configure-nfs-optional).
-
+1. [Enable incremental logging](#enable-incremental-logging).
1. Run `sudo gitlab-rake gitlab:gitaly:check` to confirm the node can connect to Gitaly.
-
1. Tail the logs to see the requests:
```shell
@@ -2175,9 +2166,6 @@ running [Prometheus](../monitoring/prometheus/index.md) and
## Configure the object storage
GitLab supports using an object storage service for holding numerous types of data.
-It's recommended over [NFS](#configure-nfs-optional) and in general it's better
-in larger setups as object storage is typically much more performant, reliable,
-and scalable.
GitLab has been tested on a number of object storage providers:
@@ -2201,7 +2189,7 @@ NOTE:
When using the [storage-specific form](../object_storage.md#storage-specific-configuration)
in GitLab 14.x and earlier, you should enable [direct upload mode](../../development/uploads/index.md#direct-upload).
The previous [background upload](../../development/uploads/index.md#direct-upload) mode,
-which was deprecated in 14.9, requires shared storage such as [NFS](#configure-nfs-optional).
+which was deprecated in 14.9, requires shared storage such as NFS.
Using separate buckets for each data type is the recommended approach for GitLab.
This ensures there are no collisions across the various types of data GitLab stores.
@@ -2220,22 +2208,6 @@ GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily
While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs.
-## Configure NFS (optional)
-
-[Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly)
-are recommended over NFS wherever possible for improved performance.
-
-See how to [configure NFS](../nfs.md).
-
-WARNING:
-Engineering support for NFS for Git repositories is deprecated, and [technical support is scheduled to be unavailable](../nfs.md#gitaly-and-nfs-deprecation)
-after the release of GitLab 15.6. No further enhancements are planned for this feature.
-
-Read:
-
-- [Gitaly and NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation).
-- About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss).
-
## Configure Advanced Search
You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md)
@@ -2309,12 +2281,10 @@ Refer to [epic 6127](https://gitlab.com/groups/gitlab-org/-/epics/6127) for more
The following tables and diagram detail the hybrid environment using the same formats
as the normal environment above.
-First are the components that run in Kubernetes. The recommendation at this time is to
-use Google Cloud's Kubernetes Engine (GKE) or AWS Elastic Kubernetes Service (EKS) and associated machine types, but the memory
-and CPU requirements should translate to most other providers. We hope to update this in the
-future with further specific cloud provider details.
+First are the components that run in Kubernetes. These run across several node groups, although you can change
+the overall makeup as desired as long as the minimum CPU and Memory requirements are observed.
-| Service | Nodes | Configuration | GCP | AWS | Min Allocatable CPUs and Memory |
+| Service Node Group | Nodes | Configuration | GCP | AWS | Min Allocatable CPUs and Memory |
|---------------------|-------|-------------------------|-----------------|--------------|---------------------------------|
| Webservice | 2 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | `c5.4xlarge` | 31.8 vCPU, 24.8 GB memory |
| Sidekiq | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | 11.8 vCPU, 38.9 GB memory |
@@ -2323,7 +2293,7 @@ future with further specific cloud provider details.
- For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results)
[Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary.
- Nodes configuration is shown as it is forced to ensure pod vCPU / memory ratios and avoid scaling during **performance testing**.
- - In production deployments, there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices.
+ - In production deployments, there is no need to assign pods to specific nodes. A minimum of three nodes per node group in three different availability zones is strongly recommended to align with resilient cloud architecture practices.
Next are the backend components that run on static compute VMs via Omnibus (or External PaaS
services where applicable):
@@ -2344,7 +2314,8 @@ services where applicable):
<!-- markdownlint-disable MD029 -->
1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
- [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work.
- - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB clusters](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo.
+ - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo.
+ - Note that [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported.
- [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440).
- Consul is primarily used for Omnibus PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery.
2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md
index ca159d62f1f..87d1408b568 100644
--- a/doc/administration/reference_architectures/50k_users.md
+++ b/doc/administration/reference_architectures/50k_users.md
@@ -35,13 +35,13 @@ full list of reference architectures, see
| GitLab Rails | 12 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` |
| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` |
| Object storage<sup>4</sup> | - | - | - | - |
-| NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` |
<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix -->
<!-- markdownlint-disable MD029 -->
1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
- [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work.
- - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB clusters](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo.
+ - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo.
+ - Note that [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported.
- [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440).
- Consul is primarily used for Omnibus PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery.
2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
@@ -189,7 +189,7 @@ CI pipelines alike.
As such, large repositories come with notable cost and typically will require more resources to handle,
significantly so in some cases. It's therefore **strongly** recommended then to review large repositories
-to ensure they maintain good repo health and reduce their size wherever possible.
+to ensure they maintain good health and reduce their size wherever possible.
NOTE:
If best practices aren't followed and large repositories are present on the environment,
@@ -227,9 +227,6 @@ To set up GitLab and its components to accommodate up to 50,000 users:
environment.
1. [Configure the object storage](#configure-the-object-storage)
used for shared data objects.
-1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended)
- to have shared disk storage service as an alternative to Gitaly or object
- storage.
1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster,
more advanced code search across your entire GitLab instance.
@@ -296,8 +293,8 @@ could also be used, those load balancers have not been validated.
### Balancing algorithm
-We recommend that a least-connection load balancing algorithm or equivalent
-is used wherever possible to ensure equal spread of calls to the nodes and good performance.
+You should use a least-connection load balancing algorithm or equivalent
+wherever possible to ensure equal spread of calls to the nodes and good performance.
We don't recommend the use of round-robin algorithms as they are known to not
spread connections equally in practice.
@@ -1288,7 +1285,7 @@ in the second step, do not supply the `EXTERNAL_URL` value.
# PostgreSQL configuration
postgresql['listen_address'] = '0.0.0.0'
- postgresql['max_connections'] = 200
+ postgresql['max_connections'] = 500
# Prevent database migrations from running on upgrade automatically
gitlab_rails['auto_migrate'] = false
@@ -1772,9 +1769,8 @@ To configure Praefect with TLS:
Sidekiq requires connection to the [Redis](#configure-redis),
[PostgreSQL](#configure-postgresql) and [Gitaly](#configure-gitaly) instances.
-Since it's recommended to use [Object storage](#configure-the-object-storage)
-over [NFS](#configure-nfs-optional) for data objects, the following examples
-include the Object storage configuration.
+Because you must use [Object storage](#configure-the-object-storage) instead of NFS for data objects, the following
+examples include the Object storage configuration.
- `10.6.0.101`: Sidekiq 1
- `10.6.0.102`: Sidekiq 2
@@ -1872,8 +1868,8 @@ Updates to example must be made at:
## Set number of Sidekiq queue processes to the same number as available CPUs
sidekiq['queue_groups'] = ['*'] * 4
- ## Set number of Sidekiq threads per queue process to the recommend number of 10
- sidekiq['max_concurrency'] = 10
+ ## Set number of Sidekiq threads per queue process to the recommend number of 20
+ sidekiq['max_concurrency'] = 20
# Monitoring
consul['enable'] = true
@@ -1931,7 +1927,7 @@ Updates to example must be made at:
NOTE:
If you find that the environment's Sidekiq job processing is slow with long queues,
more nodes can be added as required. You can also tune your Sidekiq nodes to
-run [multiple Sidekiq processes](../operations/extra_sidekiq_processes.md).
+run [multiple Sidekiq processes](../sidekiq/extra_sidekiq_processes.md).
<div align="right">
<a type="button" class="btn btn-default" href="#setup-components">
@@ -1942,9 +1938,8 @@ run [multiple Sidekiq processes](../operations/extra_sidekiq_processes.md).
## Configure GitLab Rails
This section describes how to configure the GitLab application (Rails) component.
-Since it's recommended to use [Object storage](#configure-the-object-storage)
-over [NFS](#configure-nfs-optional) for data objects, the following examples
-include the Object storage configuration.
+Because you must use [Object storage](#configure-the-object-storage) instead of NFS for data objects, the following
+examples include the Object storage configuration.
The following IPs will be used as an example:
@@ -2092,7 +2087,6 @@ On each node perform the following:
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
-
1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run:
```shell
@@ -2103,9 +2097,7 @@ On each node perform the following:
[GitLab Rails post-configuration](#gitlab-rails-post-configuration) section.
1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
-
-1. [Enable incremental logging](#enable-incremental-logging), unless you are using [NFS](#configure-nfs-optional).
-
+1. [Enable incremental logging](#enable-incremental-logging).
1. Confirm the node can connect to Gitaly:
```shell
@@ -2230,9 +2222,6 @@ To configure the Monitoring node:
## Configure the object storage
GitLab supports using an object storage service for holding numerous types of data.
-It's recommended over [NFS](#configure-nfs-optional) and in general it's better
-in larger setups as object storage is typically much more performant, reliable,
-and scalable.
GitLab has been tested on a number of object storage providers:
@@ -2256,7 +2245,7 @@ NOTE:
When using the [storage-specific form](../object_storage.md#storage-specific-configuration)
in GitLab 14.x and earlier, you should enable [direct upload mode](../../development/uploads/index.md#direct-upload).
The previous [background upload](../../development/uploads/index.md#direct-upload) mode,
-which was deprecated in 14.9, requires shared storage such as [NFS](#configure-nfs-optional).
+which was deprecated in 14.9, requires shared storage such as NFS.
Using separate buckets for each data type is the recommended approach for GitLab.
This ensures there are no collisions across the various types of data GitLab stores.
@@ -2275,22 +2264,6 @@ GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily
While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs.
-## Configure NFS (optional)
-
-[Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly)
-are recommended over NFS wherever possible for improved performance.
-
-See how to [configure NFS](../nfs.md).
-
-WARNING:
-Engineering support for NFS for Git repositories is deprecated, and [technical support is scheduled to be unavailable](../nfs.md#gitaly-and-nfs-deprecation)
-after the release of GitLab 15.6. No further enhancements are planned for this feature.
-
-Read:
-
-- [Gitaly and NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation).
-- About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss).
-
## Configure Advanced Search
You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md)
@@ -2340,12 +2313,10 @@ Refer to [epic 6127](https://gitlab.com/groups/gitlab-org/-/epics/6127) for more
The following tables and diagram detail the hybrid environment using the same formats
as the normal environment above.
-First are the components that run in Kubernetes. The recommendation at this time is to
-use Google Cloud's Kubernetes Engine (GKE) or AWS Elastic Kubernetes Service (EKS) and associated machine types, but the memory
-and CPU requirements should translate to most other providers. We hope to update this in the
-future with further specific cloud provider details.
+First are the components that run in Kubernetes. These run across several node groups, although you can change
+the overall makeup as desired as long as the minimum CPU and Memory requirements are observed.
-| Service | Nodes | Configuration | GCP | AWS | Min Allocatable CPUs and Memory |
+| Service Node Group | Nodes | Configuration | GCP | AWS | Min Allocatable CPUs and Memory |
|---------------------|-------|-------------------------|-----------------|--------------|---------------------------------|
| Webservice | 16 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `m5.8xlarge` | 510 vCPU, 472 GB memory |
| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | 15.5 vCPU, 50 GB memory |
@@ -2354,7 +2325,7 @@ future with further specific cloud provider details.
- For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results)
[Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary.
- Nodes configuration is shown as it is forced to ensure pod vCPU / memory ratios and avoid scaling during **performance testing**.
- - In production deployments, there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices.
+ - In production deployments, there is no need to assign pods to specific nodes. A minimum of three nodes per node group in three different availability zones is strongly recommended to align with resilient cloud architecture practices.
Next are the backend components that run on static compute VMs via Omnibus (or External PaaS
services where applicable):
@@ -2376,7 +2347,8 @@ services where applicable):
<!-- markdownlint-disable MD029 -->
1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
- [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work.
- - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB clusters](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo.
+ - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo.
+ - Note that [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported.
- [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440).
- Consul is primarily used for Omnibus PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery.
2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md
index a2b92f9c300..182edb82b5f 100644
--- a/doc/administration/reference_architectures/5k_users.md
+++ b/doc/administration/reference_architectures/5k_users.md
@@ -41,13 +41,13 @@ costly-to-operate environment by using the
| GitLab Rails | 3 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | `c5.4xlarge` |
| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` |
| Object storage<sup>4</sup> | - | - | - | - |
-| NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` |
<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix -->
<!-- markdownlint-disable MD029 -->
1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
- [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work.
- - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB clusters](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo.
+ - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo.
+ - Note that [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported.
- [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440).
- Consul is primarily used for Omnibus PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery.
2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
@@ -192,7 +192,7 @@ CI pipelines alike.
As such, large repositories come with notable cost and typically will require more resources to handle,
significantly so in some cases. It's therefore **strongly** recommended then to review large repositories
-to ensure they maintain good repo health and reduce their size wherever possible.
+to ensure they maintain good health and reduce their size wherever possible.
NOTE:
If best practices aren't followed and large repositories are present on the environment,
@@ -230,9 +230,6 @@ To set up GitLab and its components to accommodate up to 5,000 users:
environment.
1. [Configure the object storage](#configure-the-object-storage)
used for shared data objects.
-1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended)
- to have shared disk storage service as an alternative to Gitaly or object
- storage.
1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster,
more advanced code search across your entire GitLab instance.
@@ -1226,7 +1223,7 @@ in the second step, do not supply the `EXTERNAL_URL` value.
# PostgreSQL configuration
postgresql['listen_address'] = '0.0.0.0'
- postgresql['max_connections'] = 200
+ postgresql['max_connections'] = 500
# Prevent database migrations from running on upgrade automatically
gitlab_rails['auto_migrate'] = false
@@ -1708,9 +1705,8 @@ To configure Praefect with TLS:
Sidekiq requires connection to the [Redis](#configure-redis),
[PostgreSQL](#configure-postgresql) and [Gitaly](#configure-gitaly) instances.
-Since it's recommended to use [Object storage](#configure-the-object-storage)
-over [NFS](#configure-nfs-optional) for data objects, the following examples
-include the Object storage configuration.
+Because you must use [Object storage](#configure-the-object-storage) instead of NFS for data objects, the following
+examples include the Object storage configuration.
- `10.6.0.71`: Sidekiq 1
- `10.6.0.72`: Sidekiq 2
@@ -1790,8 +1786,8 @@ Updates to example must be made at:
## Set number of Sidekiq queue processes to the same number as available CPUs
sidekiq['queue_groups'] = ['*'] * 4
- ## Set number of Sidekiq threads per queue process to the recommend number of 10
- sidekiq['max_concurrency'] = 10
+ ## Set number of Sidekiq threads per queue process to the recommend number of 20
+ sidekiq['max_concurrency'] = 20
# Monitoring
consul['enable'] = true
@@ -1865,7 +1861,7 @@ Updates to example must be made at:
NOTE:
If you find that the environment's Sidekiq job processing is slow with long queues,
more nodes can be added as required. You can also tune your Sidekiq nodes to
-run [multiple Sidekiq processes](../operations/extra_sidekiq_processes.md).
+run [multiple Sidekiq processes](../sidekiq/extra_sidekiq_processes.md).
<div align="right">
<a type="button" class="btn btn-default" href="#setup-components">
@@ -1876,9 +1872,8 @@ run [multiple Sidekiq processes](../operations/extra_sidekiq_processes.md).
## Configure GitLab Rails
This section describes how to configure the GitLab application (Rails) component.
-Since it's recommended to use [Object storage](#configure-the-object-storage)
-over [NFS](#configure-nfs-optional) for data objects, the following examples
-include the Object storage configuration.
+Because you must use [Object storage](#configure-the-object-storage) instead of NFS for data objects, the following
+examples include the Object storage configuration.
On each node perform the following:
@@ -2020,7 +2015,6 @@ On each node perform the following:
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
-
1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run:
```shell
@@ -2031,11 +2025,8 @@ On each node perform the following:
[GitLab Rails post-configuration](#gitlab-rails-post-configuration) section.
1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
-
-1. [Enable incremental logging](#enable-incremental-logging), unless you are using [NFS](#configure-nfs-optional).
-
+1. [Enable incremental logging](#enable-incremental-logging).
1. Run `sudo gitlab-rake gitlab:gitaly:check` to confirm the node can connect to Gitaly.
-
1. Tail the logs to see the requests:
```shell
@@ -2174,9 +2165,6 @@ running [Prometheus](../monitoring/prometheus/index.md) and
## Configure the object storage
GitLab supports using an object storage service for holding numerous types of data.
-It's recommended over [NFS](#configure-nfs-optional) and in general it's better
-in larger setups as object storage is typically much more performant, reliable,
-and scalable.
GitLab has been tested on a number of object storage providers:
@@ -2200,7 +2188,7 @@ NOTE:
When using the [storage-specific form](../object_storage.md#storage-specific-configuration)
in GitLab 14.x and earlier, you should enable [direct upload mode](../../development/uploads/index.md#direct-upload).
The previous [background upload](../../development/uploads/index.md#direct-upload) mode,
-which was deprecated in 14.9, requires shared storage such as [NFS](#configure-nfs-optional).
+which was deprecated in 14.9, requires shared storage such as NFS.
Using separate buckets for each data type is the recommended approach for GitLab.
This ensures there are no collisions across the various types of data GitLab stores.
@@ -2219,22 +2207,6 @@ GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily
While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs.
-## Configure NFS (optional)
-
-[Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly)
-are recommended over NFS wherever possible for improved performance.
-
-See how to [configure NFS](../nfs.md).
-
-WARNING:
-Engineering support for NFS for Git repositories is deprecated, and [technical support is scheduled to be unavailable](../nfs.md#gitaly-and-nfs-deprecation)
-after the release of GitLab 15.6. No further enhancements are planned for this feature.
-
-Read:
-
-- [Gitaly and NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation).
-- About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss).
-
## Configure Advanced Search
You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md)
@@ -2284,16 +2256,14 @@ Refer to [epic 6127](https://gitlab.com/groups/gitlab-org/-/epics/6127) for more
The following tables and diagram detail the hybrid environment using the same formats
as the normal environment above.
-First are the components that run in Kubernetes. The recommendation at this time is to
-use Google Cloud's Kubernetes Engine (GKE) or AWS Elastic Kubernetes Service (EKS) and associated machine types, but the memory
-and CPU requirements should translate to most other providers. We hope to update this in the
-future with further specific cloud provider details.
+First are the components that run in Kubernetes. These run across several node groups, although you can change
+the overall makeup as desired as long as the minimum CPU and Memory requirements are observed.
-| Service | Nodes | Configuration | GCP | AWS | Min Allocatable CPUs and Memory |
-|-----------------------------------------------|-------|-------------------------|-----------------|--------------|---------------------------------|
-| Webservice | 5 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | `c5.4xlarge` | 79.5 vCPU, 62 GB memory |
-| Sidekiq | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | 11.8 vCPU, 38.9 GB memory |
-| Supporting services such as NGINX, Prometheus | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | 3.9 vCPU, 11.8 GB memory |
+| Service Node Group | Nodes | Configuration | GCP | AWS | Min Allocatable CPUs and Memory |
+|-------------------- |-------|-------------------------|-----------------|--------------|---------------------------------|
+| Webservice | 5 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | `c5.4xlarge` | 79.5 vCPU, 62 GB memory |
+| Sidekiq | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | 11.8 vCPU, 38.9 GB memory |
+| Supporting services | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | 3.9 vCPU, 11.8 GB memory |
- For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results)
[Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary.
@@ -2319,7 +2289,8 @@ services where applicable):
<!-- markdownlint-disable MD029 -->
1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
- [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work.
- - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB clusters](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo.
+ - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo.
+ - Note that [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported.
- [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440).
- Consul is primarily used for Omnibus PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery.
2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md
index 467cc332e25..60258fb5a09 100644
--- a/doc/administration/reference_architectures/index.md
+++ b/doc/administration/reference_architectures/index.md
@@ -207,7 +207,8 @@ Several cloud provider services are known not to support the above or have been
- [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is incompatible and not supported. See [14.4.0](../../update/index.md#1440) for more details.
- [Azure Database for PostgreSQL Single Server](https://azure.microsoft.com/en-gb/products/postgresql/#overview) (Single / Flexible) is **strongly not recommended** for use due to notable performance / stability issues or missing functionality. See [Recommendation Notes for Azure](#recommendation-notes-for-azure) for more details.
-- [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB clusters](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo.
+- [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo.
+ - Note that [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported.
### Recommendation notes for Azure
@@ -216,7 +217,7 @@ Due to performance issues that we found with several key Azure services, we only
In addition to the above, you should be aware of the additional specific guidance for Azure:
- **We outright strongly do not recommend [Azure Database for PostgreSQL Single Server](https://learn.microsoft.com/en-us/azure/postgresql/single-server/overview-single-server)** specifically due to significant performance and stability issues found. **For GitLab 14.0 and higher the service is not supported** due to it only supporting up to PostgreSQL 11.
- - A new service, [Azure Database for Postgres Flexible Server](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/) has been released but due to it missing some functionality we don't recommend it at this time.
+ - A new service, [Azure Database for PostgreSQL Flexible Server](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/) has been released but due to it missing some functionality we don't recommend it at this time.
- [Azure Blob Storage](https://azure.microsoft.com/en-gb/products/storage/blobs/) has been found to have performance limits that can impact production use at certain times. However, this has only been seen in larger architectures.
## Validation and test results
@@ -240,11 +241,11 @@ Testing occurs against all reference architectures and cloud providers in an aut
- The [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/gitlab-environment-toolkit) for building the environments.
- The [GitLab Performance Tool](https://gitlab.com/gitlab-org/quality/performance) for performance testing.
-Network latency on the test environments between components on all Cloud Providers were measured at <5ms. Note that this is shared as an observation and not as an implicit recommendation.
+Network latency on the test environments between components on all Cloud Providers were measured at <5 ms. Note that this is shared as an observation and not as an implicit recommendation.
We aim to have a "test smart" approach where architectures tested have a good range that can also apply to others. Testing focuses on 10k Omnibus on GCP as the testing has shown this is a good bellwether for the other architectures and cloud providers as well as Cloud Native Hybrids.
-The Standard Reference Architectures are designed to be platform-agnostic, with everything being run on VMs via [Omnibus GitLab](https://docs.gitlab.com/omnibus/). While testing occurs primarily on GCP, ad-hoc testing has shown that they perform similarly on equivalently specced hardware on other Cloud Providers or if run on premises (bare-metal).
+The Standard Reference Architectures are designed to be platform-agnostic, with everything being run on VMs via [Omnibus GitLab](https://docs.gitlab.com/omnibus/). While testing occurs primarily on GCP, ad-hoc testing has shown that they perform similarly on hardware with equivalent specs on other Cloud Providers or if run on premises (bare-metal).
Testing on these reference architectures is performed with the
[GitLab Performance Tool](https://gitlab.com/gitlab-org/quality/performance)
diff --git a/doc/administration/restart_gitlab.md b/doc/administration/restart_gitlab.md
index 1a1194e16a9..7996db3d1e1 100644
--- a/doc/administration/restart_gitlab.md
+++ b/doc/administration/restart_gitlab.md
@@ -14,7 +14,7 @@ A short downtime is expected for all methods.
## Omnibus installations
-If you have used the [Omnibus packages](https://about.gitlab.com/install/) to install GitLab, then
+If you have used the [Omnibus packages](https://about.gitlab.com/install/) to install GitLab,
you should already have `gitlab-ctl` in your `PATH`.
`gitlab-ctl` interacts with the Omnibus packages and can be used to restart the
@@ -88,16 +88,14 @@ sudo gitlab-ctl reconfigure
Reconfiguring GitLab should occur in the event that something in its
configuration (`/etc/gitlab/gitlab.rb`) has changed.
-When you run this command, [Chef](https://www.chef.io/products/chef-infra), the underlying configuration management
-application that powers Omnibus GitLab, makes sure that all things like directories,
-permissions, and services are in place and in the same shape that they were
-initially shipped.
+When you run `gitlab-ctl reconfigure`, [Chef](https://www.chef.io/products/chef-infra),
+the underlying configuration management application that powers Omnibus GitLab, runs some checks.
+Chef ensures directories, permissions, and services are in place and working.
-It also [restarts GitLab components](#how-to-restart-gitlab)
-where needed, if any of their configuration files have changed.
+Chef also [restarts GitLab components](#how-to-restart-gitlab) if any of their configuration files have changed.
If you manually edit any files in `/var/opt/gitlab` that are managed by Chef,
-running reconfigure reverts the changes and restarts the services that
+running `reconfigure` reverts the changes and restarts the services that
depend on those files.
## Installations from source
@@ -118,7 +116,7 @@ This should restart Puma, Sidekiq, GitLab Workhorse, and [Mailroom](reply_by_ema
## Helm chart installations
-There is no single command to restart the entire GitLab application installed via
+There is no single command to restart the entire GitLab application installed through
the [cloud-native Helm chart](https://docs.gitlab.com/charts/). Usually, it should be
enough to restart a specific component separately (for example, `gitaly`, `puma`,
`workhorse`, or `gitlab-shell`) by deleting all the pods related to it:
diff --git a/doc/administration/server_hooks.md b/doc/administration/server_hooks.md
index 448becb32dc..3d4f39b5ff0 100644
--- a/doc/administration/server_hooks.md
+++ b/doc/administration/server_hooks.md
@@ -47,15 +47,30 @@ To create server hooks for a repository:
`pre-receive` server hook, the filename should be `pre-receive` with no extension.
- To create many server hooks, create a directory for the hooks that matches the hook type. For example, for a
`pre-receive` server hook, the directory name should be `pre-receive.d`. Put the files for the hook in that directory.
-1. Make the server hook files executable and ensure that they are owned by the Git user.
+1. **Make the server hook files executable** and ensure that they are owned by the Git user.
1. Write the code to make the server hook function as expected. Git server hooks can be in any programming language. Ensure
the [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) at the top reflects the language type. For
example, if the script is in Ruby the shebang is probably `#!/usr/bin/env ruby`.
-1. Make the hook file executable, ensure that it's owned by the Git user, and ensure it does not match the backup file
+1. Ensure the hook file does not match the backup file
pattern (`*~`).
If the server hook code is properly implemented, it should execute when the Git hook is next triggered.
+### Gitaly Cluster
+
+If you use [Gitaly Cluster](gitaly/index.md), the scripts must be copied to every Gitaly node that has a replica of the repository. Every Gitaly node
+needs a copy because any node can be made a primary at any time. Server hooks only run on primary nodes.
+
+The location to copy the scripts to depends on where repositories are stored:
+
+- In GitLab 15.2 and earlier, Gitaly Cluster uses the [hashed storage path](repository_storage_types.md#hashed-storage)
+ reported by the GitLab application.
+- In GitLab 15.3 and later, new repositories are created using
+ [Praefect-generated replica paths](gitaly/index.md#praefect-generated-replica-paths-gitlab-150-and-later),
+ which are not the hashed storage path. The replica path can be identified by
+ [querying the Praefect repository metadata](../administration/gitaly/troubleshooting.md#view-repository-metadata)
+ using `-relative-path` to specify the expected GitLab hashed storage path.
+
## Create global server hooks for all repositories
To create a Git hook that applies to all repositories, set a global server hook. Global server hooks also apply to:
diff --git a/doc/administration/sidekiq.md b/doc/administration/sidekiq.md
deleted file mode 100644
index 01f83f98607..00000000000
--- a/doc/administration/sidekiq.md
+++ /dev/null
@@ -1,11 +0,0 @@
----
-redirect_to: 'sidekiq/index.md'
-remove_date: '2022-11-11'
----
-
-This document was moved to [another location](sidekiq/index.md).
-
-<!-- This redirect file can be deleted after <2022-11-11>. -->
-<!-- Redirects that point to other docs in the same project expire in three months. -->
-<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. -->
-<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
diff --git a/doc/administration/sidekiq/extra_sidekiq_processes.md b/doc/administration/sidekiq/extra_sidekiq_processes.md
index feaaa55aa59..d5007e9a3e9 100644
--- a/doc/administration/sidekiq/extra_sidekiq_processes.md
+++ b/doc/administration/sidekiq/extra_sidekiq_processes.md
@@ -6,91 +6,41 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# Run multiple Sidekiq processes **(FREE SELF)**
-GitLab allows you to start multiple Sidekiq processes.
-These processes can be used to consume a dedicated set
-of queues. This can be used to ensure certain queues always have dedicated
-workers, no matter the number of jobs to be processed.
+GitLab allows you to start multiple Sidekiq processes to process background jobs
+at a higher rate on a single instance. By default, Sidekiq starts one worker
+process and only uses a single core.
NOTE:
The information in this page applies only to Omnibus GitLab.
-## Available Sidekiq queues
-
-For a list of the existing Sidekiq queues, check the following files:
-
-- [Queues for both GitLab Community and Enterprise Editions](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/all_queues.yml)
-- [Queues for GitLab Enterprise Editions only](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/all_queues.yml)
-
-Each entry in the above files represents a queue on which Sidekiq processes
-can be started.
-
## Start multiple processes
> - [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4006) in GitLab 12.10, starting multiple processes with Sidekiq cluster.
> - [Sidekiq cluster moved](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181) to GitLab Free in 12.10.
> - [Sidekiq cluster became default](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4140) in GitLab 13.0.
-When starting multiple processes, the number of processes should
-equal (and **not** exceed) the number of CPU cores you want to
-dedicate to Sidekiq. Each Sidekiq process can use only 1 CPU
-core, subject to the available workload and concurrency settings.
+When starting multiple processes, the number of processes should at most equal
+(and **not** exceed) the number of CPU cores you want to dedicate to Sidekiq.
+The Sidekiq worker process uses no more than one CPU core.
-To start multiple processes:
+To start multiple processes, use the `sidekiq['queue_groups']` array setting to
+specify how many processes to create using `sidekiq-cluster` and which queues
+they should handle. Each item in the array equates to one additional Sidekiq
+process, and values in each item determine the queues it works on. In the vast
+majority of cases, all processes should listen to all queues (see
+[processing specific job classes](processing_specific_job_classes.md) for more
+details).
-1. Using the `sidekiq['queue_groups']` array setting, specify how many processes to
- create using `sidekiq-cluster` and which queue they should handle.
- Each item in the array equates to one additional Sidekiq
- process, and values in each item determine the queues it works on.
+For example, to create four Sidekiq processes, each listening
+to all available queues:
- For example, the following setting creates three Sidekiq processes, one to run on
- `elastic_commit_indexer`, one to run on `mailers`, and one process running on all queues:
+1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
- sidekiq['queue_groups'] = [
- "elastic_commit_indexer",
- "mailers",
- "*"
- ]
+ sidekiq['queue_groups'] = ['*'] * 4
```
- To have an additional Sidekiq process handle multiple queues, add multiple
- queue names to its item delimited by commas. For example:
-
- ```ruby
- sidekiq['queue_groups'] = [
- "elastic_commit_indexer, elastic_association_indexer",
- "mailers",
- "*"
- ]
- ```
-
- [In GitLab 12.9](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26594) and
- later, the special queue name `*` means all queues. This starts two
- processes, each handling all queues:
-
- ```ruby
- sidekiq['queue_groups'] = [
- "*",
- "*"
- ]
- ```
-
- `*` which matches all workers.
- As a result, the wildcard query must stay at the end of the list or the rules after it are ignored.
-
- `*` cannot be combined with concrete queue names - `*, mailers`
- just handles the `mailers` queue.
-
- When `sidekiq-cluster` is only running on a single node, make sure that at least
- one process is running on all queues using `*`. This ensures a process
- automatically picks up jobs in queues created in the future,
- including queues that have dedicated processes.
-
- If `sidekiq-cluster` is running on more than one node, you can also use
- [`--negate`](#negate-settings) and list all the queues that are already being
- processed.
-
-1. Save the file and reconfigure GitLab for the changes to take effect:
+1. Save the file and reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
@@ -101,125 +51,38 @@ To view the Sidekiq processes in GitLab:
1. On the top bar, select **Main menu > Admin**.
1. On the left sidebar, select **Monitoring > Background Jobs**.
-## Negate settings
+## Concurrency
-To have the Sidekiq process work on every queue **except** the ones
-you list. In this example, we exclude all import-related jobs from a Sidekiq node:
+By default each process defined under `sidekiq` starts with a number of threads
+that equals the number of queues, plus one spare thread, up to a maximum of 50.
+For example, a process that handles all queues will use 50 threads by default.
-1. Edit `/etc/gitlab/gitlab.rb` and add:
-
- ```ruby
- sidekiq['negate'] = true
- sidekiq['queue_selector'] = true
- sidekiq['queue_groups'] = [
- "feature_category=importers"
- ]
- ```
-
-1. Save the file and reconfigure GitLab for the changes to take effect:
-
- ```shell
- sudo gitlab-ctl reconfigure
- ```
-
-## Queue selector
-
-> - [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/45) in GitLab 12.8.
-> - [Sidekiq cluster, including queue selector, moved](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181) to GitLab Free in 12.10.
-> - [Renamed from `experimental_queue_selector` to `queue_selector`](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/147) in GitLab 13.6.
-
-In addition to selecting queues by name, as above, the `queue_selector` option
-allows queue groups to be selected in a more general way using a
-[worker matching query](extra_sidekiq_routing.md#worker-matching-query). After `queue_selector`
-is set, all `queue_groups` must follow the aforementioned syntax.
-
-In `/etc/gitlab/gitlab.rb`:
-
-```ruby
-sidekiq['enable'] = true
-sidekiq['queue_selector'] = true
-sidekiq['queue_groups'] = [
- # Run all non-CPU-bound queues that are high urgency
- 'resource_boundary!=cpu&urgency=high',
- # Run all continuous integration and pages queues that are not high urgency
- 'feature_category=continuous_integration,pages&urgency!=high',
- # Run all queues
- '*'
-]
-```
-
-## Ignore all import queues
-
-When [importing from GitHub](../../user/project/import/github.md) or
-other sources, Sidekiq might use all of its resources to perform those
-operations. To set up two separate `sidekiq-cluster` processes, where
-one only processes imports and the other processes all other queues:
-
-1. Edit `/etc/gitlab/gitlab.rb` and add:
-
- ```ruby
- sidekiq['enable'] = true
- sidekiq['queue_selector'] = true
- sidekiq['queue_groups'] = [
- "feature_category=importers",
- "feature_category!=importers"
- ]
- ```
-
-1. Save the file and reconfigure GitLab for the changes to take effect:
-
- ```shell
- sudo gitlab-ctl reconfigure
- ```
-
-## Number of threads
-
-By default each process defined under `sidekiq` starts with a
-number of threads that equals the number of queues, plus one spare thread.
-For example, a process that handles the `process_commit` and `post_receive`
-queues uses three threads in total.
-
-These thread run inside a single Ruby process, and each process
-can only use a single CPU core. The usefulness of threading depends
-on the work having some external dependencies to wait on, like database queries or
-HTTP requests. Most Sidekiq deployments benefit from this threading, and when
-running fewer queues in a process, increasing the thread count might be
-even more desirable to make the most effective use of CPU resources.
+These threads run inside a single Ruby process, and each process can only use a
+single CPU core. The usefulness of threading depends on the work having some
+external dependencies to wait on, like database queries or HTTP requests. Most
+Sidekiq deployments benefit from this threading.
### Manage thread counts explicitly
-The correct maximum thread count (also called concurrency) depends on the workload.
-Typical values range from `1` for highly CPU-bound tasks to `15` or higher for mixed
-low-priority work. A reasonable starting range is `15` to `25` for a non-specialized
-deployment.
+The correct maximum thread count (also called concurrency) depends on the
+workload. Typical values range from `5` for highly CPU-bound tasks to `15` or
+higher for mixed low-priority work. A reasonable starting range is `15` to `25`
+for a non-specialized deployment.
-You can find example values used by GitLab.com by searching for `concurrency:` in
-[the Helm charts](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/blob/master/releases/gitlab/values/gprd.yaml.gotmpl).
-The values vary according to the work each specific deployment of Sidekiq does.
-Any other specialized deployments with processes dedicated to specific queues should
-have the concurrency tuned according to:
-have the concurrency tuned according to:
+We only recommend setting explicit concurrency by setting `min_concurrency` and
+`max_concurrency` to the same value. The two values are kept for backwards
+compatibility reasons, but for more predictable results, use the same value.
-- The CPU usage of each type of process.
-- The throughput achieved.
-
-Each thread requires a Redis connection, so adding threads may increase Redis
-latency and potentially cause client timeouts. See the
-[Sidekiq documentation about Redis](https://github.com/mperham/sidekiq/wiki/Using-Redis) for more
-details.
-
-#### When running Sidekiq cluster (default)
+For example, to set the concurrency to `20`:
-Running Sidekiq cluster is the default in GitLab 13.0 and later.
-
-1. Edit `/etc/gitlab/gitlab.rb` and add:
+1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
- sidekiq['min_concurrency'] = 15
- sidekiq['max_concurrency'] = 25
+ sidekiq['min_concurrency'] = 20
+ sidekiq['max_concurrency'] = 20
```
-1. Save the file and reconfigure GitLab for the changes to take effect:
+1. Save the file and reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
@@ -231,50 +94,45 @@ the other. Setting `min_concurrency` to `0` disables the limit.
For each queue group, let `N` be one more than the number of queues. The
concurrency is set to:
+1. `min_concurrency`, if it's equal to `max_concurrency`.
1. `N`, if it's between `min_concurrency` and `max_concurrency`.
1. `max_concurrency`, if `N` exceeds this value.
1. `min_concurrency`, if `N` is less than this value.
-If `min_concurrency` is equal to `max_concurrency`, then this value is used
-regardless of the number of queues.
-
When `min_concurrency` is greater than `max_concurrency`, it is treated as
being equal to `max_concurrency`.
-#### When running a single Sidekiq process
-
-Running a single Sidekiq process is the default in GitLab 12.10 and earlier.
-
-WARNING:
-Running Sidekiq directly was removed in GitLab
-[14.0](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/240).
-
-1. Edit `/etc/gitlab/gitlab.rb` and add:
-
- ```ruby
- sidekiq['cluster'] = false
- sidekiq['concurrency'] = 25
- ```
-
-1. Save the file and reconfigure GitLab for the changes to take effect:
+You can find example values used by GitLab.com by searching for `concurrency:`
+in [the Helm charts](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/blob/master/releases/gitlab/values/gprd.yaml.gotmpl).
+The values vary according to the work each specific deployment of Sidekiq does.
+Any other specialized deployments with processes dedicated to specific queues
+should have the concurrency tuned according to:
- ```shell
- sudo gitlab-ctl reconfigure
- ```
+- The CPU usage of each type of process.
+- The throughput achieved.
-This sets the concurrency (number of threads) for the Sidekiq process.
+Each thread requires a Redis connection, so adding threads may increase Redis
+latency and potentially cause client timeouts. See the [Sidekiq documentation about Redis](https://github.com/mperham/sidekiq/wiki/Using-Redis)
+for more details.
## Modify the check interval
-To modify `sidekiq-cluster`'s health check interval for the additional Sidekiq processes:
+To modify Sidekiq's health check interval for the additional Sidekiq
+processes:
-1. Edit `/etc/gitlab/gitlab.rb` and add (the value can be any integer number of seconds):
+1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
sidekiq['interval'] = 5
```
-1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+ The value can be any integer number of seconds.
+
+1. Save the file and reconfigure GitLab:
+
+ ```shell
+ sudo gitlab-ctl reconfigure
+ ```
## Troubleshoot using the CLI
@@ -291,6 +149,9 @@ takes arguments using the following syntax:
/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster [QUEUE,QUEUE,...] [QUEUE, ...]
```
+The `--dryrun` argument allows viewing the command to be executed without
+actually starting it.
+
Each separate argument denotes a group of queues that have to be processed by a
Sidekiq process. Multiple queues can be processed by the same process by
separating them with a comma instead of a space.
@@ -301,29 +162,6 @@ explicitly list all the queue names. For more information about queue namespaces
see the relevant section in the
[Sidekiq development documentation](../../development/sidekiq/index.md#queue-namespaces).
-For example, say you want to start 2 extra processes: one to process the
-`process_commit` queue, and one to process the `post_receive` queue. This can be
-done as follows:
-
-```shell
-/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster process_commit post_receive
-```
-
-If you instead want to start one process processing both queues, you'd use the
-following syntax:
-
-```shell
-/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster process_commit,post_receive
-```
-
-If you want to have one Sidekiq process dealing with the `process_commit` and
-`post_receive` queues, and one process to process the `gitlab_shell` queue,
-you'd use the following:
-
-```shell
-/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster process_commit,post_receive gitlab_shell
-```
-
### Monitor the `sidekiq-cluster` command
The `sidekiq-cluster` command does not terminate once it has started the desired
diff --git a/doc/administration/sidekiq/extra_sidekiq_routing.md b/doc/administration/sidekiq/extra_sidekiq_routing.md
index 56c51beb758..d1d65498fcc 100644
--- a/doc/administration/sidekiq/extra_sidekiq_routing.md
+++ b/doc/administration/sidekiq/extra_sidekiq_routing.md
@@ -1,163 +1,11 @@
---
-stage: Systems
-group: Distribution
-info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+redirect_to: 'processing_specific_job_classes.md#routing-rules'
+remove_date: '2023-02-01'
---
-# Queue routing rules **(FREE SELF)**
+This document was moved to [another location](processing_specific_job_classes.md#routing-rules).
-When the number of Sidekiq jobs increases to a certain scale, the system faces
-some scalability issues. One of them is that the length of the queue tends to get
-longer. High-urgency jobs have to wait longer until other less urgent jobs
-finish. This head-of-line blocking situation may eventually affect the
-responsiveness of the system, especially critical actions. In another scenario,
-the performance of some jobs is degraded due to other long running or CPU-intensive jobs
-(computing or rendering ones) in the same machine.
-
-To counter the aforementioned issues, one effective solution is to split
-Sidekiq jobs into different queues and assign machines handling each queue
-exclusively. For example, all CPU-intensive jobs could be routed to the
-`cpu-bound` queue and handled by a fleet of CPU optimized instances. The queue
-topology differs between companies depending on the workloads and usage
-patterns. Therefore, GitLab supports a flexible mechanism for the
-administrator to route the jobs based on their characteristics.
-
-As an alternative to [Queue selector](extra_sidekiq_processes.md#queue-selector), which
-configures Sidekiq cluster to listen to a specific set of workers or queues,
-GitLab also supports routing a job from a worker to the desired queue when it
-is scheduled. Sidekiq clients try to match a job against a configured list of
-routing rules. Rules are evaluated from first to last, and as soon as we find a
-match for a given worker we stop processing for that worker (first match wins).
-If the worker doesn't match any rule, it falls back to the queue name generated
-from the worker name.
-
-By default, if the routing rules are not configured (or denoted with an empty
-array), all the jobs are routed to the queue generated from the worker name.
-
-## Example configuration
-
-In `/etc/gitlab/gitlab.rb`:
-
-```ruby
-sidekiq['routing_rules'] = [
- # Do not re-route workers that require their own queue
- ['tags=needs_own_queue', nil],
- # Route all non-CPU-bound workers that are high urgency to `high-urgency` queue
- ['resource_boundary!=cpu&urgency=high', 'high-urgency'],
- # Route all database, gitaly and global search workers that are throttled to `throttled` queue
- ['feature_category=database,gitaly,global_search&urgency=throttled', 'throttled'],
- # Route all workers having contact with outside work to a `network-intenstive` queue
- ['has_external_dependencies=true|feature_category=hooks|tags=network', 'network-intensive'],
- # Route all import workers to the queues generated by the worker name, for
- # example, JiraImportWorker to `jira_import`, SVNWorker to `svn_worker`
- ['feature_category=import', nil],
- # Wildcard matching, route the rest to `default` queue
- ['*', 'default']
-]
-```
-
-The routing rules list is an order-matter array of tuples of query and
-corresponding queue:
-
-- The query is following a [worker matching query](#worker-matching-query) syntax.
-- The `<queue_name>` must be a valid Sidekiq queue name. If the queue name
- is `nil`, or an empty string, the worker is routed to the queue generated
- by the name of the worker instead.
-
-The query supports wildcard matching `*`, which matches all workers. As a
-result, the wildcard query must stay at the end of the list or the rules after it
-are ignored.
-
-NOTE:
-Mixing queue routing rules and queue selectors requires care to
-ensure all jobs that are scheduled and picked up by appropriate Sidekiq
-workers.
-
-## Worker matching query
-
-GitLab provides a query syntax to match a worker based on its
-attributes. This query syntax is employed by both
-[Queue routing rules](#queue-routing-rules) and
-[Queue selector](extra_sidekiq_processes.md#queue-selector). A query includes two
-components:
-
-- Attributes that can be selected.
-- Operators used to construct a query.
-
-### Available attributes
-
-> [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/261) in GitLab 13.1 (`tags`).
-
-Queue matching query works upon the worker attributes, described in
-[Sidekiq style guide](../../development/sidekiq/index.md). We support querying
-based on a subset of worker attributes:
-
-- `feature_category` - the
- [GitLab feature category](https://about.gitlab.com/direction/maturity/#category-maturity) the
- queue belongs to. For example, the `merge` queue belongs to the
- `source_code_management` category.
-- `has_external_dependencies` - whether or not the queue connects to external
- services. For example, all importers have this set to `true`.
-- `urgency` - how important it is that this queue's jobs run
- quickly. Can be `high`, `low`, or `throttled`. For example, the
- `authorized_projects` queue is used to refresh user permissions, and
- is `high` urgency.
-- `worker_name` - the worker name. Use this attribute to select a specific worker.
-- `name` - the queue name generated from the worker name. Use this attribute to select a specific queue. Because this is generated from
- the worker name, it does not change based on the result of other routing
- rules.
-- `resource_boundary` - if the queue is bound by `cpu`, `memory`, or
- `unknown`. For example, the `ProjectExportWorker` is memory bound as it has
- to load data in memory before saving it for export.
-- `tags` - short-lived annotations for queues. These are expected to frequently
- change from release to release, and may be removed entirely.
-
-`has_external_dependencies` is a boolean attribute: only the exact
-string `true` is considered true, and everything else is considered
-false.
-
-`tags` is a set, which means that `=` checks for intersecting sets, and
-`!=` checks for disjoint sets. For example, `tags=a,b` selects queues
-that have tags `a`, `b`, or both. `tags!=a,b` selects queues that have
-neither of those tags.
-
-The attributes of each worker are hard-coded in the source code. For
-convenience, we generate a
-[list of all available attributes in GitLab Community Edition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/all_queues.yml)
-and a
-[list of all available attributes in GitLab Enterprise Edition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/all_queues.yml).
-
-### Available operators
-
-`queue_selector` supports the following operators, listed from highest
-to lowest precedence:
-
-- `|` - the logical `OR` operator. For example, `query_a|query_b` (where `query_a`
- and `query_b` are queries made up of the other operators here) includes
- queues that match either query.
-- `&` - the logical `AND` operator. For example, `query_a&query_b` (where
- `query_a` and `query_b` are queries made up of the other operators here) will
- only include queues that match both queries.
-- `!=` - the `NOT IN` operator. For example, `feature_category!=issue_tracking`
- excludes all queues from the `issue_tracking` feature category.
-- `=` - the `IN` operator. For example, `resource_boundary=cpu` includes all
- queues that are CPU bound.
-- `,` - the concatenate set operator. For example,
- `feature_category=continuous_integration,pages` includes all queues from
- either the `continuous_integration` category or the `pages` category. This
- example is also possible using the OR operator, but allows greater brevity, as
- well as being lower precedence.
-
-The operator precedence for this syntax is fixed: it's not possible to make `AND`
-have higher precedence than `OR`.
-
-[In GitLab 12.9](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26594) and
-later, as with the standard queue group syntax above, a single `*` as the
-entire queue group selects all queues.
-
-### Migration
-
-After the Sidekiq routing rules are changed, administrators must take care
-with the migration to avoid losing jobs entirely, especially in a system with
-long queues of jobs. The migration can be done by following the migration steps
-mentioned in [Sidekiq job migration](sidekiq_job_migration.md)
+<!-- This redirect file can be deleted after <2023-02-01>. -->
+<!-- Redirects that point to other docs in the same project expire in three months. -->
+<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. -->
+<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
diff --git a/doc/administration/sidekiq/index.md b/doc/administration/sidekiq/index.md
index f17c248e60e..7b3ecdd0890 100644
--- a/doc/administration/sidekiq/index.md
+++ b/doc/administration/sidekiq/index.md
@@ -379,11 +379,11 @@ To enable LDAP with the synchronization worker for Sidekiq:
## Configure SAML Groups for SAML Group Sync
-If you use [SAML Group Sync](../../user/group/saml_sso/group_sync.md), you must configure [SAML Groups](../../integration/saml.md#saml-groups) on all your Sidekiq nodes.
+If you use [SAML Group Sync](../../user/group/saml_sso/group_sync.md), you must configure [SAML Groups](../../integration/saml.md#configure-users-based-on-saml-group-membership) on all your Sidekiq nodes.
## Disable Rugged
-Calls into Rugged, Ruby bindings for `libgit2`, [lock the Sidekiq processes's GVL](https://silverhammermba.github.io/emberb/c/#c-in-ruby-threads),
+Calls into Rugged, Ruby bindings for `libgit2`, [lock the Sidekiq processes (GVL)](https://silverhammermba.github.io/emberb/c/#c-in-ruby-threads),
blocking all jobs on that worker from proceeding. If Rugged calls performed by Sidekiq are slow, this can cause significant delays in
background task processing.
@@ -398,7 +398,7 @@ sudo gitlab-rake gitlab:features:disable_rugged
## Related topics
- [Extra Sidekiq processes](extra_sidekiq_processes.md)
-- [Extra Sidekiq routing](extra_sidekiq_routing.md)
+- [Processing specific job classes](processing_specific_job_classes.md)
- [Sidekiq health checks](sidekiq_health_check.md)
- [Using the GitLab-Sidekiq chart](https://docs.gitlab.com/charts/charts/gitlab/sidekiq/)
diff --git a/doc/administration/sidekiq/processing_specific_job_classes.md b/doc/administration/sidekiq/processing_specific_job_classes.md
new file mode 100644
index 00000000000..080ad7d4eae
--- /dev/null
+++ b/doc/administration/sidekiq/processing_specific_job_classes.md
@@ -0,0 +1,337 @@
+---
+stage: Systems
+group: Distribution
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Processing specific job classes
+
+WARNING:
+These are advanced settings. While they are used on GitLab.com, most GitLab
+instances should add more processes that all listen to all queues. This is the
+same approach we take in our [Reference Architectures](../reference_architectures/index.md).
+
+GitLab has two options for creating Sidekiq processes that only handle specific
+job classes:
+
+1. [Routing rules](#routing-rules) are used on GitLab.com. They direct jobs
+ inside the application to queue names configured by administrators. This
+ lowers the load on Redis, which is important on very large-scale deployments.
+1. [Queue selectors](#queue-selectors) perform the job selection outside the
+ application, when starting the Sidekiq process. This was used on GitLab.com
+ until September 2021, and is retained for compatibility reasons.
+
+Both of these use the same [worker matching query](#worker-matching-query)
+syntax. While they can technically be used together, most deployments should
+choose one or the other; there is no particular benefit in combining them.
+
+Routing rules must be the same across all GitLab nodes as they are part of the
+application configuration. Queue selectors can be different across GitLab nodes
+because they only change the arguments to the launched Sidekiq process.
+
+## Routing rules
+
+> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/59604) in GitLab 13.12.
+> - [Default routing rule value](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/97908) added in GitLab 15.4.
+
+NOTE:
+Mailer jobs cannot be routed by routing rules, and always go to the
+`mailers` queue. When using routing rules, ensure that at least one process is
+listening to the `mailers` queue. Typically this can be placed alongside the
+`default` queue.
+
+We recommend most GitLab instances using routing rules to manage their Sidekiq
+queues. This allows administrators to choose single queue names for groups of
+job classes based on their attributes. The syntax is an ordered array of pairs of `[query, queue]`:
+
+1. The query is a [worker matching query](#worker-matching-query).
+1. The queue name must be a valid Sidekiq queue name. If the queue name
+ is `nil`, or an empty string, the worker is routed to the queue generated
+ by the name of the worker instead. (See [list of available job classes](#list-of-available-job-classes)
+ for more information).
+ The queue name does not have to match any existing queue name in the
+ list of available job classes.
+1. The first query matching a worker is chosen for that worker; later rules are
+ ignored.
+
+### Routing rules migration
+
+After the Sidekiq routing rules are changed, administrators must take care with
+the migration to avoid losing jobs entirely, especially in a system with long
+queues of jobs. The migration can be done by following the migration steps
+mentioned in [Sidekiq job migration](sidekiq_job_migration.md).
+
+### Detailed example
+
+This is a comprehensive example intended to show different possibilities. It is
+not a recommendation.
+
+1. Edit `/etc/gitlab/gitlab.rb`:
+
+ ```ruby
+ sidekiq['routing_rules'] = [
+ # Route all non-CPU-bound workers that are high urgency to `high-urgency` queue
+ ['resource_boundary!=cpu&urgency=high', 'high-urgency'],
+ # Route all database, gitaly and global search workers that are throttled to `throttled` queue
+ ['feature_category=database,gitaly,global_search&urgency=throttled', 'throttled'],
+ # Route all workers having contact with outside world to a `network-intenstive` queue
+ ['has_external_dependencies=true|feature_category=hooks|tags=network', 'network-intensive'],
+ # Route all import workers to the queues generated by the worker name, for
+ # example, JiraImportWorker to `jira_import`, SVNWorker to `svn_worker`
+ ['feature_category=import', 'import'],
+ # Wildcard matching, route the rest to `default` queue
+ ['*', 'default']
+ ]
+ ```
+
+ The `queue_groups` can then be set to match these generated queue names. For
+ instance:
+
+ ```ruby
+ sidekiq['queue_selector'] = false
+ sidekiq['queue_groups'] = [
+ # Run two high-urgency processes
+ 'high-urgency',
+ 'high-urgency',
+ # Run one process for throttled, network-intensive, import
+ 'throttled,network-intensive,import',
+ # Run one 'catchall' process on the default and mailers queues
+ 'default,mailers'
+ ]
+ ```
+
+1. Save the file and reconfigure GitLab:
+
+ ```shell
+ sudo gitlab-ctl reconfigure
+ ```
+
+## Queue selectors
+
+> - [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/45) in GitLab 12.8.
+> - [Sidekiq cluster, including queue selector, moved](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181) to GitLab Free in 12.10.
+> - [Renamed from `experimental_queue_selector` to `queue_selector`](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/147) in GitLab 13.6.
+
+The `queue_selector` option allows queue groups to be selected in a more general
+way using a [worker matching query](#worker-matching-query). After
+`queue_selector` is set, all `queue_groups` must follow the aforementioned
+syntax.
+
+### Using queue selectors
+
+1. Edit `/etc/gitlab/gitlab.rb`:
+
+ ```ruby
+ sidekiq['enable'] = true
+ sidekiq['routing_rules'] = [['*', nil]]
+ sidekiq['queue_selector'] = true
+ sidekiq['queue_groups'] = [
+ # Run all non-CPU-bound queues that are high urgency
+ 'resource_boundary!=cpu&urgency=high',
+ # Run all continuous integration and pages queues that are not high urgency
+ 'feature_category=continuous_integration,pages&urgency!=high',
+ # Run all queues
+ '*'
+ ]
+ ```
+
+1. Save the file and reconfigure GitLab:
+
+ ```shell
+ sudo gitlab-ctl reconfigure
+ ```
+
+### Negate settings
+
+This allows you to have the Sidekiq process work on every queue **except** the
+ones you list. This is generally only used when there are multiple Sidekiq
+nodes. In this example, we exclude all import-related jobs from a Sidekiq node.
+
+1. Edit `/etc/gitlab/gitlab.rb`:
+
+ ```ruby
+ sidekiq['routing_rules'] = [['*', nil]]
+ sidekiq['negate'] = true
+ sidekiq['queue_selector'] = true
+ sidekiq['queue_groups'] = [
+ "feature_category=importers"
+ ]
+ ```
+
+1. Save the file and reconfigure GitLab:
+
+ ```shell
+ sudo gitlab-ctl reconfigure
+ ```
+
+### Migrating from queue selectors to routing rules
+
+We recommend GitLab deployments add more Sidekiq processes listening to all queues, as in the
+[Reference Architectures](../reference_architectures/index.md). For very large-scale deployments, we recommend
+[routing rules](#routing-rules) instead of [queue selectors](#queue-selectors). We use routing rules on GitLab.com as
+it helps to lower the load on Redis.
+
+To migrate from queue selectors to routing rules:
+
+1. Open `/etc/gitlab/gitlab.rb`.
+1. Set `sidekiq['queue_selector']` to `false`.
+1. Take all queue `selector`s in the `sidekiq['queue_groups']`.
+1. Give each `selector` a `queue_name` and put them in `[selector, queue_name]` format.
+1. Replace `sidekiq['routing_rules']` with an array of `[selector, queue_name]` entries.
+1. Add a wildcard match of `['*', 'default']` as the last entry in `sidekiq['routing_rules']`. This "catchall" queue has
+ to be named as `default`.
+1. Replace `sidekiq['queue_groups']` with `queue_name`s.
+1. Add at least one `default` queue and at least one `mailers` queue to the `sidekiq['queue_groups']`.
+1. Save the file and reconfigure GitLab:
+
+ ```shell
+ sudo gitlab-ctl reconfigure
+ ```
+
+1. Run the Rake task to [migrate existing jobs](sidekiq_job_migration.md):
+
+ ```shell
+ sudo gitlab-rake gitlab:sidekiq:migrate_jobs:retry gitlab:sidekiq:migrate_jobs:schedule gitlab:sidekiq:migrate_jobs:queued
+ ```
+
+NOTE:
+It is important to run the Rake task immediately after reconfiguring GitLab.
+After reconfiguring GitLab, existing jobs are not processed until the Rake task starts to migrate the jobs.
+
+The following example better illustrates the migration process above:
+
+1. Check the following content of `/etc/gitlab/gitlab.rb`:
+
+ ```ruby
+ sidekiq['routing_rules'] = []
+ sidekiq['queue_selector'] = true
+ sidekiq['queue_groups'] = [
+ 'urgency=high',
+ 'urgency=low',
+ 'urgency=throttled',
+ '*'
+ ]
+ ```
+
+1. Update `/etc/gitlab/gitlab.rb` to use routing rules:
+
+ ```ruby
+ sidekiq['min_concurrency'] = 20
+ sidekiq['max_concurrency'] = 20
+
+ sidekiq['routing_rules'] = [
+ ['urgency=high', 'high_urgency'],
+ ['urgency=low', 'low_urgency'],
+ ['urgency=throttled', 'throttled_urgency'],
+ # Wildcard matching, route the rest to `default` queue
+ ['*', 'default']
+ ]
+
+ sidekiq['queue_selector'] = false
+ sidekiq['queue_groups'] = [
+ 'high_urgency',
+ 'low_urgency',
+ 'throttled_urgency',
+ 'default,mailers'
+ ]
+ ```
+
+1. Save the file and reconfigure GitLab:
+
+ ```shell
+ sudo gitlab-ctl reconfigure
+ ```
+
+1. Run the Rake task to [migrate existing jobs](sidekiq_job_migration.md):
+
+ ```shell
+ sudo gitlab-rake gitlab:sidekiq:migrate_jobs:retry gitlab:sidekiq:migrate_jobs:schedule gitlab:sidekiq:migrate_jobs:queued
+ ```
+
+WARNING:
+As described in [the concurrency section](extra_sidekiq_processes.md#manage-thread-counts-explicitly), we
+recommend setting `min_concurrency` and `max_concurrency` to the same value. For example, if the number of queues
+in a queue group entry is 1, while `min_concurrency` is set to `0`, and `max_concurrency` is set to `20`, the resulting
+concurrency will be set to `2` instead. A concurrency of `2` might be too low in most cases, except for very highly-CPU
+bound tasks.
+
+## Worker matching query
+
+GitLab provides a query syntax to match a worker based on its attributes. This
+query syntax is employed by both [routing rules](#routing-rules) and
+[queue selectors](#queue-selectors). A query includes two components:
+
+- Attributes that can be selected.
+- Operators used to construct a query.
+
+### Available attributes
+
+> [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/261) in GitLab 13.1 (`tags`).
+
+Queue matching query works upon the worker attributes, described in
+[Sidekiq style guide](../../development/sidekiq/index.md). We support querying
+based on a subset of worker attributes:
+
+- `feature_category` - the
+ [GitLab feature category](https://about.gitlab.com/direction/maturity/#category-maturity) the
+ queue belongs to. For example, the `merge` queue belongs to the
+ `source_code_management` category.
+- `has_external_dependencies` - whether or not the queue connects to external
+ services. For example, all importers have this set to `true`.
+- `urgency` - how important it is that this queue's jobs run
+ quickly. Can be `high`, `low`, or `throttled`. For example, the
+ `authorized_projects` queue is used to refresh user permissions, and
+ is `high` urgency.
+- `worker_name` - the worker name. Use this attribute to select a specific worker. Find all available names in [the job classes lists](#list-of-available-job-classes) below.
+- `name` - the queue name generated from the worker name. Use this attribute to select a specific queue. Because this is generated from
+ the worker name, it does not change based on the result of other routing
+ rules.
+- `resource_boundary` - if the queue is bound by `cpu`, `memory`, or
+ `unknown`. For example, the `ProjectExportWorker` is memory bound as it has
+ to load data in memory before saving it for export.
+- `tags` - short-lived annotations for queues. These are expected to frequently
+ change from release to release, and may be removed entirely.
+
+`has_external_dependencies` is a boolean attribute: only the exact
+string `true` is considered true, and everything else is considered
+false.
+
+`tags` is a set, which means that `=` checks for intersecting sets, and
+`!=` checks for disjoint sets. For example, `tags=a,b` selects queues
+that have tags `a`, `b`, or both. `tags!=a,b` selects queues that have
+neither of those tags.
+
+### Available operators
+
+Routing rules and queue selectors support the following operators, listed from
+highest to lowest precedence:
+
+- `|` - the logical `OR` operator. For example, `query_a|query_b` (where `query_a`
+ and `query_b` are queries made up of the other operators here) includes
+ queues that match either query.
+- `&` - the logical `AND` operator. For example, `query_a&query_b` (where
+ `query_a` and `query_b` are queries made up of the other operators here) will
+ only include queues that match both queries.
+- `!=` - the `NOT IN` operator. For example, `feature_category!=issue_tracking`
+ excludes all queues from the `issue_tracking` feature category.
+- `=` - the `IN` operator. For example, `resource_boundary=cpu` includes all
+ queues that are CPU bound.
+- `,` - the concatenate set operator. For example,
+ `feature_category=continuous_integration,pages` includes all queues from
+ either the `continuous_integration` category or the `pages` category. This
+ example is also possible using the OR operator, but allows greater brevity, as
+ well as being lower precedence.
+
+The operator precedence for this syntax is fixed: it's not possible to make `AND`
+have higher precedence than `OR`.
+
+As with the standard queue group syntax above, a single `*` as the
+entire queue group selects all queues.
+
+### List of available job classes
+
+For a list of the existing Sidekiq job classes and queues, check the following
+files:
+
+- [Queues for all GitLab editions](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/all_queues.yml)
+- [Queues for GitLab Enterprise Editions only](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/all_queues.yml)
diff --git a/doc/administration/sidekiq/sidekiq_job_migration.md b/doc/administration/sidekiq/sidekiq_job_migration.md
index f61021ad4e7..b93d86d4c86 100644
--- a/doc/administration/sidekiq/sidekiq_job_migration.md
+++ b/doc/administration/sidekiq/sidekiq_job_migration.md
@@ -4,7 +4,7 @@ group: unassigned
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
-# Sidekiq job migration **(FREE SELF)**
+# Sidekiq job migration Rake tasks **(FREE SELF)**
WARNING:
This operation should be very uncommon. We do not recommend it for the vast majority of GitLab instances.
@@ -17,24 +17,27 @@ If the Sidekiq routing rules are changed, administrators need to take care with
1. Listen to both the old and new queues.
1. Update the routing rules.
-1. Wait until there are no publishers dispatching jobs to the old queues.
-1. Run the [Rake tasks for future jobs](#future-jobs).
-1. Wait for the old queues to be empty.
+1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+1. Run the [Rake tasks for migrating queued and future jobs](#migrate-queued-and-future-jobs).
1. Stop listening to the old queues.
-## Future jobs
+## Migrate queued and future jobs
Step 4 involves rewriting some Sidekiq job data for jobs that are already stored in Redis, but due to run in future. There are two sets of jobs to run in future: scheduled jobs and jobs to be retried. We provide a separate Rake task to migrate each set:
- `gitlab:sidekiq:migrate_jobs:retry` for jobs to be retried.
- `gitlab:sidekiq:migrate_jobs:scheduled` for scheduled jobs.
-Most of the time, running both at the same time is the correct choice. There are two separate tasks to allow for more fine-grained control where needed. To run both at once:
+Queued jobs that are yet to be run can also be migrated with a Rake task:
+
+- `gitlab:sidekiq:migrate_jobs:queued` for queued jobs to be performed asynchronously.
+
+Most of the time, running all three at the same time is the correct choice. There are three separate tasks to allow for more fine-grained control where needed. To run all three at once:
```shell
# omnibus-gitlab
-sudo gitlab-rake gitlab:sidekiq:migrate_jobs:retry gitlab:sidekiq:migrate_jobs:schedule
+sudo gitlab-rake gitlab:sidekiq:migrate_jobs:retry gitlab:sidekiq:migrate_jobs:schedule gitlab:sidekiq:migrate_jobs:queued
# source installations
-bundle exec rake gitlab:sidekiq:migrate_jobs:retry gitlab:sidekiq:migrate_jobs:schedule RAILS_ENV=production
+bundle exec rake gitlab:sidekiq:migrate_jobs:retry gitlab:sidekiq:migrate_jobs:schedule gitlab:sidekiq:migrate_jobs:queued RAILS_ENV=production
```
diff --git a/doc/administration/sidekiq/sidekiq_troubleshooting.md b/doc/administration/sidekiq/sidekiq_troubleshooting.md
index d2afe171e9c..b261e385949 100644
--- a/doc/administration/sidekiq/sidekiq_troubleshooting.md
+++ b/doc/administration/sidekiq/sidekiq_troubleshooting.md
@@ -56,6 +56,120 @@ gitlab_rails['env'] = {"SIDEKIQ_LOG_ARGUMENTS" => "0"}
In GitLab 13.5 and earlier, set `SIDEKIQ_LOG_ARGUMENTS` to `1` to start logging arguments passed to Sidekiq.
+## Investigating Sidekiq queue backlogs or slow performance
+
+Symptoms of slow Sidekiq performance include problems with merge request status updates,
+and delays before CI pipelines start running.
+
+Potential causes include:
+
+- The GitLab instance may need more Sidekiq workers. By default, a single-node Omnibus GitLab
+ runs one worker, restricting the execution of Sidekiq jobs to a maximum of one CPU core.
+ [Read more about running multiple Sidekiq workers](extra_sidekiq_processes.md).
+
+- The instance is configured with more Sidekiq workers, but most of the extra workers are
+ not configured to run any job that is queued. This can result in a backlog of jobs
+ when the instance is busy, if the workload has changed in the months or years since
+ the workers were configured, or as a result of GitLab product changes.
+
+Gather data on the state of the Sidekiq workers with the following Ruby script.
+
+1. Create the script:
+
+ ```ruby
+ cat > /var/opt/gitlab/sidekiqcheck.rb <<EOF
+ require 'sidekiq/monitor'
+ Sidekiq::Monitor::Status.new.display('overview')
+ Sidekiq::Monitor::Status.new.display('processes'); nil
+ Sidekiq::Monitor::Status.new.display('queues'); nil
+ puts "----------- workers ----------- "
+ workers = Sidekiq::Workers.new
+ workers.each do |_process_id, _thread_id, work|
+ pp work
+ end
+ puts "----------- Queued Jobs ----------- "
+ Sidekiq::Queue.all.each do |queue|
+ queue.each do |job|
+ pp job
+ end
+ end ;nil
+ puts "----------- done! ----------- "
+ EOF
+ ```
+
+1. Execute and capture the output:
+
+ ```shell
+ sudo gitlab-rails runner /var/opt/gitlab/sidekiqcheck.rb > /tmp/sidekiqcheck_$(date '+%Y%m%d-%H:%M').out
+ ```
+
+ If the performance issue is intermittent:
+
+ - Run this in a cron job every five minutes. Write the files to a location with enough space: allow for 500KB per file.
+ - Refer back to the data to see what went wrong.
+
+1. Analyze the output. The following commands assume that you have a directory of output files.
+
+ 1. `grep 'Busy: ' *` shows how many jobs were being run. `grep 'Enqueued: ' *`
+ shows the backlog of work at that time.
+
+ 1. Look at the number of busy threads across the workers in samples where Sidekiq is under load:
+
+ ```shell
+ ls | while read f ; do if grep -q 'Enqueued: 0' $f; then :
+ else echo $f; egrep 'Busy:|Enqueued:|---- Processes' $f
+ grep 'Threads:' $f ; fi
+ done | more
+ ```
+
+ Example output:
+
+ ```plaintext
+ sidekiqcheck_20221024-14:00.out
+ Busy: 47
+ Enqueued: 363
+ ---- Processes (13) ----
+ Threads: 30 (0 busy)
+ Threads: 30 (0 busy)
+ Threads: 30 (0 busy)
+ Threads: 30 (0 busy)
+ Threads: 23 (0 busy)
+ Threads: 30 (0 busy)
+ Threads: 30 (0 busy)
+ Threads: 30 (0 busy)
+ Threads: 30 (0 busy)
+ Threads: 30 (0 busy)
+ Threads: 30 (0 busy)
+ Threads: 30 (24 busy)
+ Threads: 30 (23 busy)
+ ```
+
+ - In this output file, 47 threads were busy, and there was a backlog of 363 jobs.
+ - Of the 13 worker processes, only two were busy.
+ - This indicates that the other workers are configured too specifically.
+ - Look at the full output to work out which workers were busy.
+ Correlate with your `sidekiq_queues` configuration in `gitlab.rb`.
+ - An overloaded single-worker environment might look like this:
+
+ ```plaintext
+ sidekiqcheck_20221024-14:00.out
+ Busy: 25
+ Enqueued: 363
+ ---- Processes (1) ----
+ Threads: 25 (25 busy)
+ ```
+
+ 1. Look at the `---- Queues (xxx) ----` section of the output file to
+ determine what jobs were queued up at the time.
+
+ 1. The files also include low level details about the state of Sidekiq at the time.
+ This could be useful for identifying where spikes in workload are coming from.
+
+ - The `----------- workers -----------` section details the jobs that make up the
+ `Busy` count in the summary.
+ - The `----------- Queued Jobs -----------` section provides details on
+ jobs that are `Enqueued`.
+
## Thread dump
Send the Sidekiq process ID the `TTIN` signal to output thread
@@ -379,3 +493,60 @@ has number of drawbacks, as mentioned in [Why Ruby's Timeout is dangerous (and T
> - in any of your code, regardless of whether it could have possibly raised an exception before
>
> Nobody writes code to defend against an exception being raised on literally any line. That's not even possible. So Thread.raise is basically like a sneak attack on your code that could result in almost anything. It would probably be okay if it were pure-functional code that did not modify any state. But this is Ruby, so that's unlikely :)
+
+## Omnibus GitLab 14.0 and later: remove the `sidekiq-cluster` service
+
+Omnibus GitLab instances that were configured to run `sidekiq-cluster` prior to GitLab 14.0
+might still have this service running along side `sidekiq` in later releases.
+
+The code to manage `sidekiq-cluster` was removed in GitLab 14.0.
+The configuration files remain on disk so the `sidekiq-cluster` process continues
+to be started by the GitLab systemd service .
+
+The extra service can be identified as running by:
+
+- `gitlab-ctl status` showing both services:
+
+ ```plaintext
+ run: sidekiq: (pid 1386) 445s; run: log: (pid 1385) 445s
+ run: sidekiq-cluster: (pid 1388) 445s; run: log: (pid 1381) 445s
+ ```
+
+- `ps -ef | grep 'runsv sidekiq'` showing two processes:
+
+ ```plaintext
+ root 31047 31045 0 13:54 ? 00:00:00 runsv sidekiq-cluster
+ root 31054 31045 0 13:54 ? 00:00:00 runsv sidekiq
+ ```
+
+To remove the `sidekiq-cluster` service from servers running GitLab 14.0 and later:
+
+1. Stop GitLab and the systemd service:
+
+ ```shell
+ sudo gitlab-ctl stop
+ sudo systemctl stop gitlab-runsvdir.service
+ ```
+
+1. Remove the `runsv` service definition:
+
+ ```shell
+ sudo rm -rf /opt/gitlab/sv/sidekiq-cluster
+ ```
+
+1. Restart GitLab:
+
+ ```shell
+ sudo systemctl start gitlab-runsvdir.service
+ ```
+
+1. Check that all services are up, and the `sidekiq-cluster` service is not listed:
+
+ ```shell
+ sudo gitlab-ctl status
+ ```
+
+This change might reduce the amount of work Sidekiq can do. Symptoms like delays creating pipelines
+indicate that additional Sidekiq processes would be beneficial.
+Consider [adding additional Sidekiq processes](extra_sidekiq_processes.md)
+to compensate for removing the `sidekiq-cluster` service.
diff --git a/doc/administration/sidekiq_health_check.md b/doc/administration/sidekiq_health_check.md
deleted file mode 100644
index 3294eb663f2..00000000000
--- a/doc/administration/sidekiq_health_check.md
+++ /dev/null
@@ -1,11 +0,0 @@
----
-redirect_to: 'sidekiq/sidekiq_health_check.md'
-remove_date: '2022-11-11'
----
-
-This document was moved to [another location](sidekiq/sidekiq_health_check.md).
-
-<!-- This redirect file can be deleted after <2022-11-11>. -->
-<!-- Redirects that point to other docs in the same project expire in three months. -->
-<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. -->
-<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
diff --git a/doc/administration/snippets/index.md b/doc/administration/snippets/index.md
index 7bf828afedd..4bd03aeb8c8 100644
--- a/doc/administration/snippets/index.md
+++ b/doc/administration/snippets/index.md
@@ -72,3 +72,7 @@ You can also use the API to [retrieve the current value](../../api/settings.md#g
```shell
curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/application/settings"
```
+
+## Related topics
+
+- [User snippets](../../user/snippets.md)
diff --git a/doc/administration/terraform_state.md b/doc/administration/terraform_state.md
index 02a95e28747..d3b941bd129 100644
--- a/doc/administration/terraform_state.md
+++ b/doc/administration/terraform_state.md
@@ -160,6 +160,10 @@ sudo find /var/opt/gitlab/gitlab-rails/shared/terraform_state -type f | grep -v
### S3-compatible connection settings
+In GitLab 13.2 and later, you should use the
+[consolidated object storage settings](object_storage.md#consolidated-object-storage-configuration).
+This section describes the earlier configuration format.
+
See [the available connection settings for different providers](object_storage.md#connection-settings).
**In Omnibus installations:**
diff --git a/doc/administration/troubleshooting/elasticsearch.md b/doc/administration/troubleshooting/elasticsearch.md
deleted file mode 100644
index 7390f4bc816..00000000000
--- a/doc/administration/troubleshooting/elasticsearch.md
+++ /dev/null
@@ -1,11 +0,0 @@
----
-redirect_to: '../../integration/advanced_search/elasticsearch_troubleshooting.md'
-remove_date: '2022-11-02'
----
-
-This document was moved to [another location](../../integration/advanced_search/elasticsearch_troubleshooting.md).
-
-<!-- This redirect file can be deleted after <2022-11-02>. -->
-<!-- Redirects that point to other docs in the same project expire in three months. -->
-<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. -->
-<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md
index 6993d48b450..7c5feb24e15 100644
--- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md
+++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md
@@ -1,11 +1,95 @@
---
-redirect_to: 'index.md'
-remove_date: '2023-02-01'
+stage: Systems
+group: Distribution
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
-This document was moved to [another location](index.md).
+# GitLab Rails Console Cheat Sheet **(FREE SELF)**
-<!-- This redirect file can be deleted after 2023-02-01. -->
-<!-- Redirects that point to other docs in the same project expire in three months. -->
-<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. -->
-<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
+This was the GitLab Support Team's collection of information regarding the GitLab Rails
+console, for use while troubleshooting. It is listed here for posterity,
+as most content has been moved to feature-specific troubleshooting pages and sections,
+see epic [&8147](https://gitlab.com/groups/gitlab-org/-/epics/8147#tree).
+Please update your bookmarks accordingly.
+
+If you are currently having an issue with GitLab,
+it is highly recommended that you first check
+our guide on [the Rails console](../operations/rails_console.md),
+and your [support options](https://about.gitlab.com/support/),
+before attempting the information pointed to from here.
+
+WARNING:
+Some of these scripts could be damaging if not run correctly,
+or under the right conditions. We highly recommend running them under the
+guidance of a Support Engineer, or running them in a test environment with a
+backup of the instance ready to be restored, just in case.
+
+WARNING:
+As GitLab changes, changes to the code are inevitable,
+and so some scripts may not work as they once used to. These are not kept
+up-to-date as these scripts/commands were added as they were found/needed. As
+mentioned above, we recommend running these scripts under the supervision of a
+Support Engineer, who can also verify that they continue to work as they
+should and, if needed, update the script for the latest version of GitLab.
+
+## Mirrors
+
+### Find mirrors with "bad decrypt" errors
+
+This content has been converted to a Rake task, see [verify database values can be decrypted using the current secrets](../raketasks/check.md#verify-database-values-can-be-decrypted-using-the-current-secrets).
+
+### Transfer mirror users and tokens to a single service account
+
+This content has been moved to [Troubleshooting Repository mirroring](../../user/project/repository/mirror/index.md#transfer-mirror-users-and-tokens-to-a-single-service-account-in-rails-console).
+
+## Merge requests
+
+## CI
+
+This content has been moved to [Troubleshooting CI/CD](../../ci/troubleshooting.md).
+
+## License
+
+This content has been moved to [Activate GitLab EE with a license file or key](../../user/admin_area/license_file.md).
+
+## Registry
+
+### Registry Disk Space Usage by Project
+
+Find this content in the [Container Registry troubleshooting documentation](../packages/container_registry.md#registry-disk-space-usage-by-project).
+
+### Run the Cleanup policy now
+
+Find this content in the [Container Registry troubleshooting documentation](../packages/container_registry.md#run-the-cleanup-policy-now).
+
+## Sidekiq
+
+This content has been moved to [Troubleshooting Sidekiq](../sidekiq/sidekiq_troubleshooting.md).
+
+## Geo
+
+### Reverify all uploads (or any SSF data type which is verified)
+
+Moved to [Geo replication troubleshooting](../geo/replication/troubleshooting.md#reverify-all-uploads-or-any-ssf-data-type-which-is-verified).
+
+### Artifacts
+
+Moved to [Geo replication troubleshooting](../geo/replication/troubleshooting.md#find-failed-artifacts).
+
+### Repository verification failures
+
+Moved to [Geo replication troubleshooting](../geo/replication/troubleshooting.md#find-repository-verification-failures).
+
+### Resync repositories
+
+Moved to [Geo replication troubleshooting - Resync repository types except for project or project wiki repositories](../geo/replication/troubleshooting.md#repository-types-except-for-project-or-project-wiki-repositories).
+
+Moved to [Geo replication troubleshooting - Resync project and project wiki repositories](../geo/replication/troubleshooting.md#resync-project-and-project-wiki-repositories).
+
+### Blob types
+
+Moved to [Geo replication troubleshooting](../geo/replication/troubleshooting.md#blob-types).
+
+## Generate Service Ping
+
+This content has been moved to [Service Ping Troubleshooting](../../development/service_ping/troubleshooting.md).
diff --git a/doc/administration/troubleshooting/linux_cheat_sheet.md b/doc/administration/troubleshooting/linux_cheat_sheet.md
index 90cd1e24c79..ae0ef44f0b1 100644
--- a/doc/administration/troubleshooting/linux_cheat_sheet.md
+++ b/doc/administration/troubleshooting/linux_cheat_sheet.md
@@ -294,8 +294,8 @@ small differences should not be considered significant.
|Setup | access times |
|:--------------|:--------------|
-| EFS | 10 - 30ms |
-| Local Storage | 0.01 - 1ms |
+| EFS | 10 - 30 ms |
+| Local Storage | 0.01 - 1 ms |
## Networking
diff --git a/doc/administration/troubleshooting/postgresql.md b/doc/administration/troubleshooting/postgresql.md
index 829fed38060..d5288bfead8 100644
--- a/doc/administration/troubleshooting/postgresql.md
+++ b/doc/administration/troubleshooting/postgresql.md
@@ -44,10 +44,6 @@ This section is for links to information elsewhere in the GitLab documentation.
- Consuming PostgreSQL from [within CI runners](../../ci/services/postgres.md).
-- [Using Slony to update PostgreSQL](../../update/upgrading_postgresql_using_slony.md).
- - Uses replication to handle PostgreSQL upgrades if the schemas are the same.
- - Reduces downtime to a short window for switching to the newer version.
-
- Managing Omnibus PostgreSQL versions [from the development docs](https://docs.gitlab.com/omnibus/development/managing-postgresql-versions.html).
- [PostgreSQL scaling](../postgresql/replication_and_failover.md)
@@ -104,14 +100,14 @@ or `statement_timeout`, but to leave the third setting at 60 seconds. Setting
`idle_in_transaction` protects the database from sessions potentially hanging for
days. There's more discussion in [the issue relating to introducing this timeout on GitLab.com](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1053).
-PostgresSQL defaults:
+PostgreSQL defaults:
- `statement_timeout = 0` (never)
- `idle_in_transaction_session_timeout = 0` (never)
Comments in issue [#30528](https://gitlab.com/gitlab-org/gitlab/-/issues/30528)
indicate that these should both be set to at least a number of minutes for all
-Omnibus GitLab installations (so they don't hang indefinitely). However, 15s
+Omnibus GitLab installations (so they don't hang indefinitely). However, 15 s
for `statement_timeout` is very short, and is only effective if the
underlying infrastructure is very performant.
@@ -188,7 +184,7 @@ To temporarily change the statement timeout:
### Database is not accepting commands to avoid wraparound data loss
-This error likely means that AUTOVACUUM is failing to complete its run:
+This error likely means that `autovacuum` is failing to complete its run:
```plaintext
ERROR: database is not accepting commands to avoid wraparound data loss in database "gitlabhq_production"
@@ -211,7 +207,7 @@ To resolve the error, run `VACUUM` manually:
The [database requirements](../../install/requirements.md#database) for GitLab include:
-- Support for MySQL was removed in GitLab 12.1; [migrate to PostgreSQL](../../update/mysql_to_postgresql.md).
+- Support for MySQL was removed in [GitLab 12.1](../../update/index.md#1210).
- Review and install the [required extension list](../../install/postgresql_extensions.md).
### Serialization errors in the `production/sidekiq` log
diff --git a/doc/administration/troubleshooting/sidekiq.md b/doc/administration/troubleshooting/sidekiq.md
deleted file mode 100644
index e49e0ed4f1c..00000000000
--- a/doc/administration/troubleshooting/sidekiq.md
+++ /dev/null
@@ -1,11 +0,0 @@
----
-redirect_to: '../sidekiq/sidekiq_troubleshooting.md'
-remove_date: '2022-11-11'
----
-
-This document was moved to [another location](../sidekiq/sidekiq_troubleshooting.md).
-
-<!-- This redirect file can be deleted after <2022-11-11>. -->
-<!-- Redirects that point to other docs in the same project expire in three months. -->
-<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. -->
-<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
diff --git a/doc/administration/troubleshooting/tracing_correlation_id.md b/doc/administration/troubleshooting/tracing_correlation_id.md
deleted file mode 100644
index 917e27bab70..00000000000
--- a/doc/administration/troubleshooting/tracing_correlation_id.md
+++ /dev/null
@@ -1,11 +0,0 @@
----
-redirect_to: '../logs/tracing_correlation_id.md'
-remove_date: '2022-11-12'
----
-
-This document was moved to [another location](../logs/tracing_correlation_id.md).
-
-<!-- This redirect file can be deleted after 2022-11-12. -->
-<!-- Redirects that point to other docs in the same project expire in three months. -->
-<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. -->
-<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> \ No newline at end of file
diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md
index f0def7320cc..ff0b8ecf178 100644
--- a/doc/administration/uploads.md
+++ b/doc/administration/uploads.md
@@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# Uploads administration **(FREE SELF)**
-Uploads represent all user data that may be sent to GitLab as a single file. As an example, avatars and notes' attachments are uploads. Uploads are integral to GitLab functionality, and therefore cannot be disabled.
+Uploads represent all user data that may be sent to GitLab as a single file. For example, avatars and note attachments are uploads. Uploads are integral to GitLab functionality and therefore cannot be disabled.
## Using local storage
@@ -14,15 +14,15 @@ This is the default configuration. To change the location where the uploads are
stored locally, use the steps in this section based on your installation method:
NOTE:
-For historical reasons, instance level uploads (for example the [favicon](../user/admin_area/appearance.md#favicon)) are stored into a base directory,
-which by default is `uploads/-/system`. It is strongly discouraged to change the base
-directory on an existing GitLab installation.
+For historical reasons, uploads for the whole instance (for example the [favicon](../user/admin_area/appearance.md#favicon)) are stored in a base directory,
+which by default is `uploads/-/system`. Changing the base
+directory on an existing GitLab installation is strongly discouraged.
**In Omnibus GitLab installations:**
_The uploads are stored by default in `/var/opt/gitlab/gitlab-rails/uploads`._
-1. To change the storage path for example to `/mnt/storage/uploads`, edit
+1. To change the storage path, for example to `/mnt/storage/uploads`, edit
`/etc/gitlab/gitlab.rb` and add the following line:
```ruby
@@ -38,7 +38,7 @@ _The uploads are stored by default in `/var/opt/gitlab/gitlab-rails/uploads`._
_The uploads are stored by default in
`/home/git/gitlab/public/uploads`._
-1. To change the storage path for example to `/mnt/storage/uploads`, edit
+1. To change the storage path, for example to `/mnt/storage/uploads`, edit
`/home/git/gitlab/config/gitlab.yml` and add or amend the following lines:
```yaml
@@ -57,10 +57,12 @@ This configuration relies on valid AWS credentials to be configured already.
[Read more about using object storage with GitLab](object_storage.md).
-We recommend using the [consolidated object storage settings](object_storage.md#consolidated-object-storage-configuration). The following instructions apply to the original configuration format.
-
### Object Storage Settings
+In GitLab 13.2 and later, you should use the
+[consolidated object storage settings](object_storage.md#consolidated-object-storage-configuration).
+This section describes the earlier configuration format.
+
For source installations the following settings are nested under `uploads:` and then `object_store:`. On Omnibus GitLab installs they are prefixed by `uploads_object_store_`.
| Setting | Description | Default |
@@ -104,7 +106,7 @@ _The uploads are stored by default in
```
1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
-1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate:all` Rake task](raketasks/uploads/migrate.md).
+1. Migrate any existing local uploads to the object storage with the [`gitlab:uploads:migrate:all` Rake task](raketasks/uploads/migrate.md).
**In installations from source:**
@@ -127,4 +129,4 @@ _The uploads are stored by default in
```
1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect.
-1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate:all` Rake task](raketasks/uploads/migrate.md).
+1. Migrate any existing local uploads to the object storage with the [`gitlab:uploads:migrate:all` Rake task](raketasks/uploads/migrate.md).
diff --git a/doc/administration/user_settings.md b/doc/administration/user_settings.md
index c96a6311208..b02e1c7244b 100644
--- a/doc/administration/user_settings.md
+++ b/doc/administration/user_settings.md
@@ -18,7 +18,7 @@ ability to create top-level groups (does not affect existing users' setting), Gi
- The [application setting API](../api/settings.md#change-application-settings).
- In GitLab 15.4 and earlier, in a configuration file by following the steps in this section.
-To disable new users' ability to create top-level groups using the configuation file:
+To disable new users' ability to create top-level groups using the configuration file:
**Omnibus GitLab installations**