Welcome to mirror list, hosted at ThFree Co, Russian Federation.

gitlab.com/gitlab-org/gitlab-foss.git - Unnamed repository; edit this file 'description' to name the repository.
summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorGitLab Bot <gitlab-bot@gitlab.com>2023-05-17 19:05:49 +0300
committerGitLab Bot <gitlab-bot@gitlab.com>2023-05-17 19:05:49 +0300
commit43a25d93ebdabea52f99b05e15b06250cd8f07d7 (patch)
treedceebdc68925362117480a5d672bcff122fb625b /doc/administration
parent20c84b99005abd1c82101dfeff264ac50d2df211 (diff)
Add latest changes from gitlab-org/gitlab@16-0-stable-eev16.0.0-rc42
Diffstat (limited to 'doc/administration')
-rw-r--r--doc/administration/audit_event_streaming.md64
-rw-r--r--doc/administration/audit_events.md311
-rw-r--r--doc/administration/auth/atlassian.md2
-rw-r--r--doc/administration/auth/authentiq.md12
-rw-r--r--doc/administration/auth/cognito.md2
-rw-r--r--doc/administration/auth/crowd.md4
-rw-r--r--doc/administration/auth/index.md5
-rw-r--r--doc/administration/auth/jwt.md2
-rw-r--r--doc/administration/auth/ldap/google_secure_ldap.md6
-rw-r--r--doc/administration/auth/ldap/index.md21
-rw-r--r--doc/administration/auth/ldap/ldap-troubleshooting.md6
-rw-r--r--doc/administration/auth/ldap/ldap_synchronization.md150
-rw-r--r--doc/administration/auth/oidc.md469
-rw-r--r--doc/administration/auth/test_oidc_oauth.md57
-rw-r--r--doc/administration/clusters/kas.md75
-rw-r--r--doc/administration/compliance.md100
-rw-r--r--doc/administration/configure.md71
-rw-r--r--doc/administration/dedicated/index.md127
-rw-r--r--doc/administration/docs_self_host.md43
-rw-r--r--doc/administration/environment_variables.md2
-rw-r--r--doc/administration/external_pipeline_validation.md3
-rw-r--r--doc/administration/feature_flags.md10
-rw-r--r--doc/administration/file_hooks.md2
-rw-r--r--doc/administration/geo/disaster_recovery/background_verification.md24
-rw-r--r--doc/administration/geo/disaster_recovery/planned_failover.md4
-rw-r--r--doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md2
-rw-r--r--doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md2
-rw-r--r--doc/administration/geo/index.md5
-rw-r--r--doc/administration/geo/replication/configuration.md27
-rw-r--r--doc/administration/geo/replication/container_registry.md16
-rw-r--r--doc/administration/geo/replication/datatypes.md22
-rw-r--r--doc/administration/geo/replication/geo_validation_tests.md3
-rw-r--r--doc/administration/geo/replication/img/adding_a_secondary_v13_3.pngbin20195 -> 0 bytes
-rw-r--r--doc/administration/geo/replication/img/adding_a_secondary_v15_8.pngbin0 -> 14698 bytes
-rw-r--r--doc/administration/geo/replication/single_sign_on.md9
-rw-r--r--doc/administration/geo/replication/troubleshooting.md29
-rw-r--r--doc/administration/geo/replication/upgrading_the_geo_sites.md1
-rw-r--r--doc/administration/geo/secondary_proxy/index.md1
-rw-r--r--doc/administration/geo/setup/database.md57
-rw-r--r--doc/administration/geo/setup/external_database.md18
-rw-r--r--doc/administration/geo/setup/index.md24
-rw-r--r--doc/administration/get_started.md8
-rw-r--r--doc/administration/git_protocol.md4
-rw-r--r--doc/administration/gitaly/configure_gitaly.md560
-rw-r--r--doc/administration/gitaly/gitaly_geo_capabilities.md41
-rw-r--r--doc/administration/gitaly/index.md28
-rw-r--r--doc/administration/gitaly/monitoring.md15
-rw-r--r--doc/administration/gitaly/praefect.md558
-rw-r--r--doc/administration/gitaly/recovery.md72
-rw-r--r--doc/administration/gitaly/reference.md42
-rw-r--r--doc/administration/gitaly/troubleshooting.md58
-rw-r--r--doc/administration/housekeeping.md31
-rw-r--r--doc/administration/incoming_email.md3
-rw-r--r--doc/administration/index.md239
-rw-r--r--doc/administration/instance_limits.md120
-rw-r--r--doc/administration/integration/kroki.md15
-rw-r--r--doc/administration/integration/plantuml.md15
-rw-r--r--doc/administration/integration/terminal.md6
-rw-r--r--doc/administration/job_artifacts.md33
-rw-r--r--doc/administration/lfs/index.md19
-rw-r--r--doc/administration/load_balancer.md10
-rw-r--r--doc/administration/logs/index.md78
-rw-r--r--doc/administration/logs/log_parsing.md20
-rw-r--r--doc/administration/maintenance_mode/index.md28
-rw-r--r--doc/administration/merge_request_diffs.md4
-rw-r--r--doc/administration/monitoring/github_imports.md4
-rw-r--r--doc/administration/monitoring/gitlab_self_monitoring_project/img/self_monitoring_overview_dashboard.pngbin51508 -> 0 bytes
-rw-r--r--doc/administration/monitoring/gitlab_self_monitoring_project/index.md119
-rw-r--r--doc/administration/monitoring/index.md3
-rw-r--r--doc/administration/monitoring/performance/grafana_configuration.md31
-rw-r--r--doc/administration/monitoring/performance/index.md4
-rw-r--r--doc/administration/monitoring/prometheus/gitlab_metrics.md141
-rw-r--r--doc/administration/monitoring/prometheus/index.md7
-rw-r--r--doc/administration/monitoring/prometheus/node_exporter.md4
-rw-r--r--doc/administration/monitoring/prometheus/pgbouncer_exporter.md4
-rw-r--r--doc/administration/monitoring/prometheus/postgres_exporter.md4
-rw-r--r--doc/administration/monitoring/prometheus/redis_exporter.md4
-rw-r--r--doc/administration/monitoring/prometheus/registry_exporter.md4
-rw-r--r--doc/administration/nfs.md2
-rw-r--r--doc/administration/object_storage.md1037
-rw-r--r--doc/administration/operations/gitlab_sshd.md19
-rw-r--r--doc/administration/operations/index.md40
-rw-r--r--doc/administration/operations/moving_repositories.md16
-rw-r--r--doc/administration/operations/puma.md4
-rw-r--r--doc/administration/operations/rails_console.md2
-rw-r--r--doc/administration/package_information/defaults.md8
-rw-r--r--doc/administration/package_information/supported_os.md18
-rw-r--r--doc/administration/packages/container_registry.md172
-rw-r--r--doc/administration/packages/dependency_proxy.md2
-rw-r--r--doc/administration/packages/index.md2
-rw-r--r--doc/administration/pages/index.md456
-rw-r--r--doc/administration/pages/source.md46
-rw-r--r--doc/administration/pages/troubleshooting.md304
-rw-r--r--doc/administration/postgresql/database_load_balancing.md5
-rw-r--r--doc/administration/postgresql/external.md22
-rw-r--r--doc/administration/postgresql/multiple_databases.md91
-rw-r--r--doc/administration/postgresql/pgbouncer.md21
-rw-r--r--doc/administration/postgresql/replication_and_failover.md2
-rw-r--r--doc/administration/raketasks/github_import.md8
-rw-r--r--doc/administration/raketasks/maintenance.md6
-rw-r--r--doc/administration/raketasks/project_import_export.md144
-rw-r--r--doc/administration/raketasks/service_desk_email.md2
-rw-r--r--doc/administration/raketasks/uploads/sanitize.md2
-rw-r--r--doc/administration/read_only_gitlab.md44
-rw-r--r--doc/administration/redis/replication_and_failover.md43
-rw-r--r--doc/administration/redis/replication_and_failover_external.md22
-rw-r--r--doc/administration/redis/troubleshooting.md8
-rw-r--r--doc/administration/reference_architectures/10k_users.md288
-rw-r--r--doc/administration/reference_architectures/1k_users.md4
-rw-r--r--doc/administration/reference_architectures/25k_users.md290
-rw-r--r--doc/administration/reference_architectures/2k_users.md117
-rw-r--r--doc/administration/reference_architectures/3k_users.md252
-rw-r--r--doc/administration/reference_architectures/50k_users.md290
-rw-r--r--doc/administration/reference_architectures/5k_users.md248
-rw-r--r--doc/administration/reference_architectures/index.md242
-rw-r--r--doc/administration/reply_by_email.md6
-rw-r--r--doc/administration/repository_checks.md35
-rw-r--r--doc/administration/restart_gitlab.md14
-rw-r--r--doc/administration/server_hooks.md72
-rw-r--r--doc/administration/sidekiq/extra_sidekiq_routing.md11
-rw-r--r--doc/administration/sidekiq/index.md43
-rw-r--r--doc/administration/sidekiq/processing_specific_job_classes.md27
-rw-r--r--doc/administration/sidekiq/sidekiq_memory_killer.md4
-rw-r--r--doc/administration/silent_mode/index.md64
-rw-r--r--doc/administration/static_objects_external_storage.md2
-rw-r--r--doc/administration/system_hooks.md2
-rw-r--r--doc/administration/terraform_state.md30
-rw-r--r--doc/administration/troubleshooting/index.md25
-rw-r--r--doc/administration/uploads.md11
-rw-r--r--doc/administration/wikis/index.md4
130 files changed, 5266 insertions, 3487 deletions
diff --git a/doc/administration/audit_event_streaming.md b/doc/administration/audit_event_streaming.md
index 92ccbe8b1a6..afc4926fec0 100644
--- a/doc/administration/audit_event_streaming.md
+++ b/doc/administration/audit_event_streaming.md
@@ -41,13 +41,13 @@ Streaming destinations receive **all** audit event data, which could include sen
Users with the Owner role for a group can add streaming destinations for it:
1. On the top bar, select **Main menu > Groups** and find your group.
-1. On the left sidebar, select **Security & Compliance > Audit events**.
+1. On the left sidebar, select **Security and Compliance > Audit events**.
1. On the main area, select **Streams** tab.
1. Select **Add streaming destination** to show the section for adding destinations.
1. Enter the destination URL to add.
1. Optional. Locate the **Custom HTTP headers** table.
-1. Ignore the **Active** checkbox because it isn't functional. To track progress on adding functionality to the **Active** checkbox, see the
- [relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/361925).
+1. Ignore the **Active** checkbox because it isn't functional. To track progress on adding functionality to the
+ **Active** checkbox, see [issue 367509](https://gitlab.com/gitlab-org/gitlab/-/issues/367509).
1. Select **Add header** to create a new name and value pair. Enter as many name and value pairs as required. You can add up to
20 headers per streaming destination.
1. After all headers have been filled out, select **Add** to add the new streaming destination.
@@ -119,7 +119,7 @@ Users with the Owner role for a group can list streaming destinations.
To list the streaming destinations:
1. On the top bar, select **Main menu > Groups** and find your group.
-1. On the left sidebar, select **Security & Compliance > Audit events**.
+1. On the left sidebar, select **Security and Compliance > Audit events**.
1. On the main area, select **Streams** tab.
1. To the right of the item, select **Edit** (**{pencil}**) to see all the custom HTTP headers.
@@ -164,13 +164,13 @@ Users with the Owner role for a group can update streaming destinations.
To update a streaming destinations custom HTTP headers:
1. On the top bar, select **Main menu > Groups** and find your group.
-1. On the left sidebar, select **Security & Compliance > Audit events**.
+1. On the left sidebar, select **Security and Compliance > Audit events**.
1. On the main area, select **Streams** tab.
1. To the right of the item, select **Edit** (**{pencil}**).
1. Locate the **Custom HTTP headers** table.
1. Locate the header that you wish to update.
-1. Ignore the **Active** checkbox because it isn't functional. To track progress on adding functionality to the **Active** checkbox, see the
- [relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/361925).
+1. Ignore the **Active** checkbox because it isn't functional. To track progress on adding functionality to the
+ **Active** checkbox, see [issue 367509](https://gitlab.com/gitlab-org/gitlab/-/issues/367509).
1. Select **Add header** to create a new name and value pair. Enter as many name and value pairs as required. You can add up to
20 headers per streaming destination.
1. Select **Save** to update the streaming destination.
@@ -218,14 +218,14 @@ When the last destination is successfully deleted, streaming is disabled for the
To delete a streaming destination:
1. On the top bar, select **Main menu > Groups** and find your group.
-1. On the left sidebar, select **Security & Compliance > Audit events**.
+1. On the left sidebar, select **Security and Compliance > Audit events**.
1. On the main area, select the **Streams** tab.
1. To the right of the item, select **Delete** (**{remove}**).
To delete only the custom HTTP headers for a streaming destination:
1. On the top bar, select **Main menu > Groups** and find your group.
-1. On the left sidebar, select **Security & Compliance > Audit events**.
+1. On the left sidebar, select **Security and Compliance > Audit events**.
1. On the main area, select the **Streams** tab.
1. To the right of the item, **Edit** (**{pencil}**).
1. Locate the **Custom HTTP headers** table.
@@ -282,7 +282,7 @@ the destination's value when [listing streaming destinations](#list-streaming-de
Users with the Owner role for a group can list streaming destinations and see the verification tokens:
1. On the top bar, select **Main menu > Groups** and find your group.
-1. On the left sidebar, select **Security & Compliance > Audit events**.
+1. On the left sidebar, select **Security and Compliance > Audit events**.
1. On the main area, select the **Streams**.
1. View the verification token on the right side of each item.
@@ -365,6 +365,50 @@ Streamed audit events have a predictable schema in the body of the response.
| `target_id` | ID of the audit event's target | |
| `target_type` | String representation of the target's type | |
+### JSON payload schema
+
+```json
+{
+ "properties": {
+ "id": {
+ "type": "string"
+ },
+ "author_id": {
+ "type": "integer"
+ },
+ "author_name": {
+ "type": "string"
+ },
+ "details": {},
+ "ip_address": {
+ "type": "string"
+ },
+ "entity_id": {
+ "type": "integer"
+ },
+ "entity_path": {
+ "type": "string"
+ },
+ "entity_type": {
+ "type": "string"
+ },
+ "event_type": {
+ "type": "string"
+ },
+ "target_id": {
+ "type": "integer"
+ },
+ "target_type": {
+ "type": "string"
+ },
+ "target_details": {
+ "type": "string"
+ },
+ },
+ "type": "object"
+}
+```
+
## Audit event streaming on Git operations
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332747) in GitLab 14.9 [with a flag](../administration/feature_flags.md) named `audit_event_streaming_git_operations`. Disabled by default.
diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md
index c51b5fbb431..50bd943b8e4 100644
--- a/doc/administration/audit_events.md
+++ b/doc/administration/audit_events.md
@@ -49,7 +49,7 @@ You can view audit events scoped to a group or project.
To view a group's audit events:
1. Go to the group.
-1. On the left sidebar, select **Security & Compliance > Audit Events**.
+1. On the left sidebar, select **Security and Compliance > Audit Events**.
Group events do not include project audit events. Group events can also be accessed using the
[Group Audit Events API](../api/audit_events.md#group-audit-events). Group event queries are limited to a maximum of 30
@@ -120,7 +120,7 @@ Successful sign-in events are the only audit events available at all tiers. To s
1. Select your avatar.
1. Select **Edit profile > Authentication log**.
-After upgrading to a paid tier, you can see also see successful sign-in events on audit event pages.
+After upgrading to a paid tier, you can also see successful sign-in events on audit event pages.
## Filter audit events
@@ -159,11 +159,35 @@ You can view different events depending on the version of GitLab you have.
The following actions on groups generate group audit events:
+#### Group management
+
- Group name or path changed.
- Group repository size limit changed.
- Group created or deleted.
- Group changed visibility.
- User was added to group and with which [permissions](../user/permissions.md).
+- Removed user from group.
+- Project repository imported into group.
+- [Project shared with group](../user/project/members/share_project_with_groups.md) and with which
+ [permissions](../user/permissions.md).
+- Removal of a previously shared group with a project.
+- LFS enabled or disabled.
+- Membership lock enabled or disabled.
+- Request access enabled or disabled.
+- Roles allowed to create project changed.
+- Group deploy token was successfully created, revoked, or deleted.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/353452) in GitLab 14.9.
+- Failed attempt to create a group deploy token. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/353452)
+ in GitLab 14.9.
+- [IP restrictions](../user/group/access_and_permissions.md#restrict-group-access-by-ip-address) changed.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/358986) in GitLab 15.0.
+- Group had a security policy project linked, changed, or unlinked.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/377877) in GitLab 15.6.
+- An environment is protected or unprotected.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216164) in GitLab 15.8.
+
+#### Authentication and authorization
+
- User sign-in using [Group SAML](../user/group/saml_sso/index.md).
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8071) in GitLab 14.5, changes to the following
[group SAML](../user/group/saml_sso/index.md) configuration:
@@ -177,32 +201,28 @@ The following actions on groups generate group audit events:
- Default membership role.
- SSO-SAML group sync configuration.
- Permissions changes of a user assigned to a group.
-- Removed user from group.
-- Project repository imported into group.
-- [Project shared with group](../user/project/members/share_project_with_groups.md) and with which
- [permissions](../user/permissions.md).
-- Removal of a previously shared group with a project.
-- LFS enabled or disabled.
-- Shared runners minutes limit changed.
-- Membership lock enabled or disabled.
-- Request access enabled or disabled.
- 2FA enforcement or grace period changed.
-- Roles allowed to create project changed.
-- Group CI/CD variable added, removed, or protected status changed.
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30857) in GitLab 13.3.
+
+#### Compliance and security
+
- Compliance framework created, updated, or deleted.
[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340649) in GitLab 14.5.
- Event streaming destination created, updated, or deleted.
[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/344664) in GitLab 14.6.
+- Changes to push rules. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/227629) in GitLab 15.0.
+- Changes to streaming audit destination custom HTTP headers.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/366350) in GitLab 15.3.
- Instance administrator started or stopped impersonation of a group member.
[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/300961) in GitLab 14.8.
-- Group deploy token was successfully created, revoked, or deleted.
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/353452) in GitLab 14.9.
-- Failed attempt to create a group deploy token. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/353452)
- in GitLab 14.9.
-- [IP restrictions](../user/group/access_and_permissions.md#restrict-group-access-by-ip-address) changed.
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/358986) in GitLab 15.0.
-- Changes to push rules. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/227629) in GitLab 15.0.
+
+#### CI/CD
+
+- Shared runners minutes limit changed.
+- Group CI/CD variable added, removed, or protected status changed.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30857) in GitLab 13.3.
+
+#### Code collaboration
+
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/356152) in GitLab 15.1, changes to the following merge
request approvals settings:
- Prevent approval by author.
@@ -210,145 +230,187 @@ The following actions on groups generate group audit events:
- Prevent editing approval rules in projects and merge requests.
- Require user password to approve.
- Remove all approvals when commits are added to the source branch.
-- Changes to streaming audit destination custom HTTP headers.
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/366350) in GitLab 15.3.
-- Group had a security policy project linked, changed, or unlinked.
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/377877) in GitLab 15.6.
-- An environment is protected or unprotected.
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216164) in GitLab 15.8.
+- Changes to Code Suggestions.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/405295) in GitLab 15.11.
### Project events
The following actions on projects generate project audit events:
-- Added or removed deploy keys
-- Project created, deleted, renamed, moved (transferred), changed path
-- Project changed visibility level
-- User was added to project and with which [permissions](../user/permissions.md)
-- Permission changes of a user assigned to a project
-- User was removed from project
-- Project export was downloaded
-- Project repository was downloaded
-- Project was archived
-- Project was unarchived
-- Added, removed, or updated protected branches
-- Release was added to a project
-- Release was updated
-- Release was deleted. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/94793/) in GitLab 15.3.
-- Release milestone associations changed
-- Permission to approve merge requests by committers was updated.
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7531) in GitLab 12.9.
-- Permission to approve merge requests by committers was updated.
- - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7531) in GitLab 12.9.
- - Message for event [changed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/72623/diffs) in GitLab 14.6.
-- Permission to approve merge requests by authors was updated.
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7531) in GitLab 12.9.
-- Number of required approvals was updated.
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7531) in GitLab 12.9.
-- Added or removed users and groups from project approval groups.
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213603) in GitLab 13.2.
-- Project CI/CD variable added, removed, or protected status changed.
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30857) in GitLab 13.4.
-- Project access token was successfully created or revoked.
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230007) in GitLab 13.9.
-- Failed attempt to create or revoke a project access token.
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230007) in GitLab 13.9.
-- When default branch changes for a project.
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/52339) in GitLab 13.9.
-- Created, updated, or deleted DAST profiles, DAST scanner profiles, and DAST site profiles.
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217872) in GitLab 14.1.
-- Changed a project's compliance framework.
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/329362) in GitLab 14.1.
-- User password required for approvals was updated.
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336211) in GitLab 14.2.
-- Permission to modify merge requests approval rules in merge requests was updated.
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336211) in GitLab 14.2.
-- New approvals requirement when new commits are added to an MR was updated.
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336211) in GitLab 14.2.
-- When [strategies for feature flags](../operations/feature_flags.md#feature-flag-strategies) are changed.
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68408) in GitLab 14.3.
-- Allowing force push to protected branch changed.
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/338873) in GitLab 14.3.
-- Code owner approval requirement on merge requests targeting protected branch changed.
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/338873) in GitLab 14.3.
-- Users and groups allowed to merge and push to protected branch added or removed.
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/338873) in GitLab 14.3.
+#### Project management
+
+- Added or removed deploy keys.
+- Project created, deleted, renamed, moved (transferred), changed path.
+- Project changed visibility level.
+- Project export was downloaded.
+- Project repository was downloaded.
+- Project was archived.
+- Project was unarchived.
+- Project had a security policy project linked, changed, or unlinked.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/377877) in GitLab 15.6.
+- Project was scheduled for deletion due to inactivity.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/85689) in GitLab 15.0.
- Project deploy token was successfully created, revoked or deleted.
[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/353451) in GitLab 14.9.
- Failed attempt to create a project deploy token.
[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/353451) in GitLab 14.9.
-- When merge method is updated.
+- When [strategies for feature flags](../operations/feature_flags.md#feature-flag-strategies) are changed.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68408) in GitLab 14.3.
+
+#### User management
+
+- User was added to project and with which [permissions](../user/permissions.md).
+- Permission changes of a user assigned to a project.
+- User was removed from project.
+- Users and groups allowed to merge and push to protected branch added or removed.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/338873) in GitLab 14.3.
+
+#### Access control
+
+- Branch protection was added, removed, or updated.
+- Failed attempt to create or revoke a project access token.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230007) in GitLab 13.9.
+- Allowing force push to protected branch changed.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/338873) in GitLab 14.3.
+- An environment is protected or unprotected.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216164) in GitLab 15.8.
+- User password required for approvals was updated.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336211) in GitLab 14.2.
+- Project access token was successfully created or revoked.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230007) in GitLab 13.9.
+
+#### Code collaboration
+
+- Default description template for merge requests is updated.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355805) in GitLab 15.0.
+- Merge commit message template is updated.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355805) in GitLab 15.0.
+- Squash commit message template is updated.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355805) in GitLab 15.0.
+- Delete source branch option by default enabled or disabled.
[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9.
-- Merged results pipelines enabled or disabled.
+- Squash commits when merging is updated.
[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9.
-- Merge trains enabled or disabled.
+- All discussions must be resolved enabled or disabled.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9.
+- Commit message suggestion is updated.
[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9.
- Automatically resolve merge request diff discussions enabled or disabled.
[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9.
- Show link to create or view a merge request when pushing from the command line enabled or disabled.
[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9.
-- Delete source branch option by default enabled or disabled.
+- When merge method is updated.
[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9.
-- Squash commits when merging is updated.
+- Merge trains enabled or disabled.
[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9.
+- Code owner approval requirement on merge requests targeting protected branch changed.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/338873) in GitLab 14.3.
+- Permission to modify merge requests approval rules in merge requests was updated.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336211) in GitLab 14.2.
+- New approvals requirement when new commits are added to an MR was updated.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336211) in GitLab 14.2.
+- Added or removed users and groups from project approval groups.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213603) in GitLab 13.2.
+- Permission to approve merge requests by committers was updated.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7531) in GitLab 12.9.
+ - Message for event [changed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/72623/diffs) in GitLab 14.6.
+- Permission to approve merge requests by authors was updated.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7531) in GitLab 12.9.
+- Number of required approvals was updated.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7531) in GitLab 12.9.
+
+#### Release management
+
+- Release was added to a project.
+- Release was updated.
+- Release was deleted. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/94793/) in GitLab 15.3.
+- Release milestone associations changed.
+
+#### CI/CD
+
+- Project CI/CD variable added, removed, or protected status changed.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30857) in GitLab 13.4.
+- When default branch changes for a project.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/52339) in GitLab 13.9.
- Pipelines must succeed enabled or disabled.
[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9.
- Skipped pipelines are considered successful enabled or disabled.
[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9.
-- All discussions must be resolved enabled or disabled.
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9.
-- Commit message suggestion is updated.
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9.
- Status check is added, edited, or deleted.
[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355805) in GitLab 15.0.
-- Merge commit message template is updated.
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355805) in GitLab 15.0.
-- Squash commit message template is updated.
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355805) in GitLab 15.0.
-- Default description template for merge requests is updated.
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355805) in GitLab 15.0.
-- Project was scheduled for deletion due to inactivity.
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/85689) in GitLab 15.0.
-- Project had a security policy project linked, changed, or unlinked.
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/377877) in GitLab 15.6.
-- An environment is protected or unprotected.
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216164) in GitLab 15.8.
+- Merged results pipelines enabled or disabled.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9.
+
+#### Compliance and security
+
+- Created, updated, or deleted DAST profiles, DAST scanner profiles, and DAST site profiles.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217872) in GitLab 14.1.
+- Changed a project's compliance framework.
+ [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/329362) in GitLab 14.1.
+
+### GitLab agent for Kubernetes events
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/382133) in GitLab 15.10.
+
+GitLab generates audit events when a cluster agent token is created or revoked.
### Instance events **(PREMIUM SELF)**
+> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16826) in GitLab 13.5, audit events for failed second-factor authentication attempt.
+> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276250) in GitLab 13.6, audit events for when a user is approved using the Admin Area.
+> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276921) in GitLab 13.6, audit events for when a user's personal access token is successfully or unsuccessfully created or revoked.
+> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/298783) in GitLab 13.9, audit events for when a user requests access to an instance or is rejected using the Admin Area.
+
The following user actions on a GitLab instance generate instance audit events:
-- Sign-in events and the authentication type (such as standard, LDAP, or OmniAuth)
-- Failed sign-ins
-- Added SSH key
-- Added or removed email
-- Changed password
-- Ask for password reset
-- Grant OAuth access
-- Started or stopped user impersonation
-- Changed username. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7797) in GitLab 12.8.
-- User was deleted. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/251) in GitLab 12.8.
-- User was added. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/251) in GitLab 12.8.
-- User requests access to an instance. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/298783) in GitLab 13.9.
-- User was approved using the Admin Area. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276250) in GitLab 13.6.
-- User was rejected using the Admin Area. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/298783) in GitLab 13.9.
-- User was blocked using the Admin Area. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/251) in GitLab 12.8.
-- User was blocked using the API. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25872) in GitLab 12.9.
-- Failed second-factor authentication attempt. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16826) in
- GitLab 13.5.
-- A user's personal access token was successfully created or revoked.
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276921) in GitLab 13.6.
-- A failed attempt to create or revoke a user's personal access token.
- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276921) in GitLab 13.6.
+#### Authentication
+
+- Sign-in events and the authentication type such as standard, LDAP, or OmniAuth.
+- Failed sign-ins.
+- Ask for password reset.
+- Grant OAuth access.
+- Failed second-factor authentication attempt.
+- A user's personal access token was successfully or unsuccessfully created or revoked.
+- A user's two-factor authentication was disabled. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/238177) in
+ GitLab 15.1.
+
+#### User management
+
+- Added SSH key.
+- Added or removed email.
+- Changed password.
+- Started or stopped user impersonation.
+- Changed username.
+- User was added or deleted.
+- User requests access to an instance.
+- User was approved, rejected, or blocked using the Admin Area.
+- User was blocked using the API.
- Administrator added or removed. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/323905) in GitLab 14.1.
- Removed SSH key. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/220127) in GitLab 14.1.
- Added or removed GPG key. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/220127) in GitLab 14.1.
-- A user's two-factor authentication was disabled. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/238177) in
- GitLab 15.1.
- Enabled Admin Mode. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/362101) in GitLab 15.7.
+- All [group events](#group-events) and [project events](#project-events).
+- User was unblocked using the Admin Area or API. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/115727) in GitLab 15.11.
+- User was banned using the Admin Area or API. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/116103) in GitLab 15.11.
+- User was unbanned using the Admin Area or API. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/116221) in GitLab 15.11.
Instance events can also be accessed using the [Instance Audit Events API](../api/audit_events.md#instance-audit-events).
+### GitLab Runner events
+
+> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/335509) in GitLab 14.8, audit events for when a runner is registered.
+> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/349540) in GitLab 14.9, audit events for when a runner is unregistered.
+> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/349542) in GitLab 14.9, audit events for when a runner is assigned to or unassigned from a project.
+> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355637) in GitLab 15.0, audit events for when a runner registration token is reset.
+
+GitLab generates audit events for the following GitLab Runner actions:
+
+- Instance, group, or project runner is registered.
+- Instance, group, or project runner is unregistered.
+- Runner is assigned to or unassigned from a project.
+- Instance, group, or project runner registration token is reset.
+ [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/102579) in GitLab 15.6.
+
## "Deleted User" events
Audit events created after users are deleted are created for "Deleted User". For example, if a deleted user's access to
@@ -364,6 +426,7 @@ Some events are not tracked in audit events. The following epics and issues prop
- [Group settings and activity](https://gitlab.com/groups/gitlab-org/-/epics/475).
- [Instance-level settings and activity](https://gitlab.com/groups/gitlab-org/-/epics/476).
- [Deployment Approval activity](https://gitlab.com/gitlab-org/gitlab/-/issues/354782).
+- [Approval rules processing by a non GitLab user](https://gitlab.com/gitlab-org/gitlab/-/issues/407384).
If you don't see the event you want in any of the epics, you can either:
diff --git a/doc/administration/auth/atlassian.md b/doc/administration/auth/atlassian.md
index 58412078802..45617b9965c 100644
--- a/doc/administration/auth/atlassian.md
+++ b/doc/administration/auth/atlassian.md
@@ -41,7 +41,7 @@ To enable the Atlassian OmniAuth provider for passwordless authentication you mu
sudo -u git -H editor /home/git/gitlab/config/gitlab.yml
```
-1. Edit the [common configuration file settings](../../integration/omniauth.md#configure-common-settings)
+1. Configure the [common settings](../../integration/omniauth.md#configure-common-settings)
to add `atlassian_oauth2` as a single sign-on provider. This enables
Just-In-Time account provisioning for users who do not have an existing
GitLab account.
diff --git a/doc/administration/auth/authentiq.md b/doc/administration/auth/authentiq.md
deleted file mode 100644
index a32d2a2cf94..00000000000
--- a/doc/administration/auth/authentiq.md
+++ /dev/null
@@ -1,12 +0,0 @@
----
-stage: Manage
-group: Authentication and Authorization
-info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
-remove_date: '2023-02-22'
-redirect_to: '../../integration/omniauth.md'
----
-
-# Authentiq OmniAuth Provider (removed) **(FREE SELF)**
-
-This feature was [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/389452) in 15.9.
-Use another [OmniAuth provider](../../integration/omniauth.md) instead.
diff --git a/doc/administration/auth/cognito.md b/doc/administration/auth/cognito.md
index a73595c731b..cfac958e297 100644
--- a/doc/administration/auth/cognito.md
+++ b/doc/administration/auth/cognito.md
@@ -41,7 +41,7 @@ To enable AWS Cognito as an authentication provider, complete the following step
## Configure GitLab
-1. Edit the [common configuration file settings](../../integration/omniauth.md#configure-common-settings)
+1. Configure the [common settings](../../integration/omniauth.md#configure-common-settings)
to add `cognito` as a single sign-on provider. This enables Just-In-Time
account provisioning for users who do not have an existing GitLab account.
1. On your GitLab server, open the configuration file.
diff --git a/doc/administration/auth/crowd.md b/doc/administration/auth/crowd.md
index 277e59b1a8a..f89e1a00928 100644
--- a/doc/administration/auth/crowd.md
+++ b/doc/administration/auth/crowd.md
@@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w
WARNING:
This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/369117) in GitLab 15.3 and is planned for
-removal in 16.0.
+removal in 17.0.
Authenticate to GitLab using the Atlassian Crowd OmniAuth provider. Enabling
this provider also allows Crowd authentication for Git-over-https requests.
@@ -40,7 +40,7 @@ this provider also allows Crowd authentication for Git-over-https requests.
sudo -u git -H editor config/gitlab.yml
```
-1. Edit the [common configuration file settings](../../integration/omniauth.md#configure-common-settings)
+1. Configure the [common settings](../../integration/omniauth.md#configure-common-settings)
to add `crowd` as a single sign-on provider. This enables Just-In-Time
account provisioning for users who do not have an existing GitLab account.
diff --git a/doc/administration/auth/index.md b/doc/administration/auth/index.md
index 2b978dd1f2c..4a8e230a944 100644
--- a/doc/administration/auth/index.md
+++ b/doc/administration/auth/index.md
@@ -1,5 +1,4 @@
---
-comments: false
type: index
stage: Manage
group: Authentication and Authorization
@@ -34,3 +33,7 @@ For more information, see the links shown on this page for each external provide
| **User Removal** | SCIM (remove user from top-level group) | LDAP (remove user from groups and block from the instance)<br>SCIM |
1. Using Just-In-Time (JIT) provisioning, user accounts are created when the user first signs in.
+
+## Test OIDC/OAuth in GitLab
+
+See [Test OIDC/OAuth in GitLab](test_oidc_oauth.md) to learn how to test OIDC/OAuth authentication in your GitLab instance using your client application.
diff --git a/doc/administration/auth/jwt.md b/doc/administration/auth/jwt.md
index 8c9800aa792..9994b374038 100644
--- a/doc/administration/auth/jwt.md
+++ b/doc/administration/auth/jwt.md
@@ -25,7 +25,7 @@ JWT provides you with a secret key for you to use.
sudo -u git -H editor config/gitlab.yml
```
-1. Edit the [common configuration file settings](../../integration/omniauth.md#configure-common-settings)
+1. Configure the [common settings](../../integration/omniauth.md#configure-common-settings)
to add `jwt` as a single sign-on provider. This enables Just-In-Time
account provisioning for users who do not have an existing GitLab account.
1. Add the provider configuration.
diff --git a/doc/administration/auth/ldap/google_secure_ldap.md b/doc/administration/auth/ldap/google_secure_ldap.md
index e0612099221..042a65be500 100644
--- a/doc/administration/auth/ldap/google_secure_ldap.md
+++ b/doc/administration/auth/ldap/google_secure_ldap.md
@@ -18,6 +18,8 @@ The steps below cover:
- Configuring the Secure LDAP Client in the Google administrator console.
- Required GitLab configuration.
+Secure LDAP is only available on specific Google Workspace editions. For more information, see the [Google Secure LDAP service documentation](https://support.google.com/a/answer/9048516).
+
## Configuring Google LDAP client
1. Go to <https://admin.google.com/Dashboard> and sign in as a Google Workspace domain administrator.
@@ -88,7 +90,7 @@ values obtained during the LDAP client configuration earlier:
encryption: 'simple_tls'
verify_certificates: true
retry_empty_result_with_codes: [80]
-
+ base: "DC=example,DC=com"
tls_options:
cert: |
-----BEGIN CERTIFICATE-----
@@ -152,7 +154,7 @@ values obtained during the LDAP client configuration earlier:
servers:
main: # 'main' is the GitLab 'provider ID' of this LDAP server
label: 'Google Secure LDAP'
-
+ base: "DC=example,DC=com"
host: 'ldap.google.com'
port: 636
uid: 'uid'
diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md
index 0c7bd33c2c1..7687f7c9340 100644
--- a/doc/administration/auth/ldap/index.md
+++ b/doc/administration/auth/ldap/index.md
@@ -270,7 +270,7 @@ These LDAP sync configuration settings are available:
| Setting | Description | Required | Examples |
|-------------------|-------------|----------|----------|
-| `group_base` | Base used to search for groups. | **{dotted-circle}** No | `'ou=groups,dc=gitlab,dc=example'` |
+| `group_base` | Base used to search for groups. | **{dotted-circle}** No (required when `external_groups` is configured) | `'ou=groups,dc=gitlab,dc=example'` |
| `admin_group` | The CN of a group containing GitLab administrators. Not `cn=administrators` or the full DN. | **{dotted-circle}** No | `'administrators'` |
| `external_groups` | An array of CNs of groups containing users that should be considered external. Not `cn=interns` or the full DN. | **{dotted-circle}** No | `['interns', 'contractors']` |
| `sync_ssh_keys` | The LDAP attribute containing a user's public SSH key. | **{dotted-circle}** No | `'sshPublicKey'` or false if not set |
@@ -1031,6 +1031,25 @@ See [Google Secure LDAP](google_secure_ldap.md) for detailed configuration instr
For more information on synchronizing users and groups between LDAP and GitLab, see
[LDAP synchronization](ldap_synchronization.md).
+## Move from LDAP to SAML
+
+1. [Configure SAML](../../../integration/saml.md). Add `auto_link_ldap_user` to:
+ - [`gitlab.rb` for Omnibus](../../../integration/saml.html?tab=Linux+package+%28Omnibus%29).
+ - [`values.yml` for Kubernetes](../../../integration/saml.html?tab=Helm+chart+%28Kubernetes%29).
+ For more information, see the [initial settings for all providers](../../../integration/omniauth.md#configure-initial-settings).
+
+1. Optional. [Disable the LDAP auth from the sign-in page](#disable-ldap-web-sign-in).
+
+1. Optional. To fix issues with linking users, you can first [remove those users' LDAP identities](ldap-troubleshooting.md#remove-the-identity-records-that-relate-to-the-removed-ldap-server).
+
+1. Confirm that users are able to sign in to their accounts. If a user cannot sign in, check if that user's LDAP is still there and remove it if necessary. If this issue persists, check the logs to identify the problem.
+
+1. In the configuration file, change:
+ - `omniauth_auto_link_user` to `saml` only.
+ - `omniauth_auto_link_ldap_user` to false.
+ - `ldap_enabled` to `false`.
+ You can also comment out the LDAP provider settings.
+
## Troubleshooting
See our [administrator guide to troubleshooting LDAP](ldap-troubleshooting.md).
diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md
index 7e21d3c6655..909ab5245ca 100644
--- a/doc/administration/auth/ldap/ldap-troubleshooting.md
+++ b/doc/administration/auth/ldap/ldap-troubleshooting.md
@@ -170,10 +170,10 @@ We have a workaround, based on toggling the access level of affected users:
1. As an administrator, on the top bar, select **Main menu > Admin**.
1. On the left sidebar, select **Overview > Users**.
1. Select the name of the affected user.
-1. In the upper right, select **Edit**.
+1. In the upper-right corner, select **Edit**.
1. Change the user's access level from `Regular` to `Administrator` (or vice versa).
1. At the bottom of the page, select **Save changes**.
-1. In the upper right, select **Edit** again.
+1. In the upper-right corner, select **Edit** again.
1. Restore the user's original access level (`Regular` or `Administrator`)
and select **Save changes** again.
@@ -761,7 +761,7 @@ users, [see what to do when no users are found](#no-users-are-found).
### GitLab logs
If a user account is blocked or unblocked due to the LDAP configuration, a
-message is [logged to `application.log`](../../logs/index.md#applicationlog).
+message is [logged to `application_json.log`](../../logs/index.md#application_jsonlog).
If there is an unexpected error during an LDAP lookup (configuration error,
timeout), the sign-in is rejected and a message is [logged to `production.log`](../../logs/index.md#productionlog).
diff --git a/doc/administration/auth/ldap/ldap_synchronization.md b/doc/administration/auth/ldap/ldap_synchronization.md
index cc300941470..f32a4af9e27 100644
--- a/doc/administration/auth/ldap/ldap_synchronization.md
+++ b/doc/administration/auth/ldap/ldap_synchronization.md
@@ -7,12 +7,16 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# LDAP synchronization **(PREMIUM SELF)**
If you have [configured LDAP to work with GitLab](index.md), GitLab can automatically synchronize
-users and groups. This process updates user and group information.
+users and groups.
+
+LDAP synchronization updates user and group information for existing GitLab users that have an LDAP identity assigned. It does not create new GitLab users through LDAP.
You can change when synchronization occurs.
## User sync
+> Preventing LDAP username synchronization [introduced](<https://gitlab.com/gitlab-org/gitlab/-/issues/11336>) in GitLab 15.11.
+
Once per day, GitLab runs a worker to check and update GitLab
users against LDAP.
@@ -33,17 +37,132 @@ For more information, see [Bitmask Searches in LDAP](https://ctovswild.com/2009/
<!-- vale gitlab.Spelling = YES -->
-The user is set to an `ldap_blocked` state in GitLab if the previous conditions
-fail. This means the user cannot sign in or push or pull code.
-
The process also updates the following user information:
- Name. Because of a [sync issue](https://gitlab.com/gitlab-org/gitlab/-/issues/342598), `name` is not synchronized if
- [**Prevent users from changing their profile name**](../../../user/admin_area/settings/account_and_limit_settings.md#disable-user-profile-name-changes) is enabled.
+ [**Prevent users from changing their profile name**](../../../user/admin_area/settings/account_and_limit_settings.md#disable-user-profile-name-changes) is enabled or `sync_name` is set to `false`.
- Email address.
- SSH public keys if `sync_ssh_keys` is set.
- Kerberos identity if Kerberos is enabled.
+### Synchronize LDAP username
+
+By default, GitLab synchronizes the LDAP username field.
+
+To prevent this synchronization, you can set `sync_name` to `false`.
+
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
+
+1. Edit `/etc/gitlab/gitlab.rb`:
+
+ ```ruby
+ gitlab_rails['ldap_servers'] = {
+ 'main' => {
+ 'sync_name' => false,
+ }
+ }
+ ```
+
+1. Save the file and reconfigure GitLab:
+
+ ```shell
+ sudo gitlab-ctl reconfigure
+ ```
+
+:::TabTitle Helm chart (Kubernetes)
+
+1. Export the Helm values:
+
+ ```shell
+ helm get values gitlab > gitlab_values.yaml
+ ```
+
+1. Edit `gitlab_values.yaml`:
+
+ ```yaml
+ global:
+ appConfig:
+ ldap:
+ servers:
+ main:
+ sync_name: false
+ ```
+
+1. Save the file and apply the new values:
+
+ ```shell
+ helm upgrade -f gitlab_values.yaml gitlab gitlab/gitlab
+ ```
+
+:::TabTitle Docker
+
+1. Edit `docker-compose.yml`:
+
+ ```yaml
+ version: "3.6"
+ services:
+ gitlab:
+ environment:
+ GITLAB_OMNIBUS_CONFIG: |
+ gitlab_rails['ldap_servers'] = {
+ 'main' => {
+ 'sync_name' => false,
+ }
+ }
+ ```
+
+1. Save the file and restart GitLab:
+
+ ```shell
+ docker compose up -d
+ ```
+
+:::TabTitle Self-compiled (source)
+
+1. Edit `/home/git/gitlab/config/gitlab.yml`:
+
+ ```yaml
+ production: &base
+ ldap:
+ servers:
+ main:
+ sync_name: false
+ ```
+
+1. Save the file and restart GitLab:
+
+ ```shell
+ # For systems running systemd
+ sudo systemctl restart gitlab.target
+
+ # For systems running SysV init
+ sudo service gitlab restart
+ ```
+
+::EndTabs
+
+### Blocked users
+
+A user is blocked if either the:
+
+- [Access check fails](#user-sync) and that user is set to an `ldap_blocked` state in GitLab.
+- LDAP server is not available when that user signs in.
+
+If a user is blocked, that user cannot sign in or push or pull code.
+
+A blocked user is unblocked when they sign in with LDAP if all of the following are true:
+
+- All the access check conditions are true.
+- The LDAP server is available when the user signs in.
+
+**All users** are blocked if the LDAP server is unavailable when an LDAP user synchronization is run.
+
+NOTE:
+If all users are blocked due to the LDAP server not being available when an LDAP user synchronization is run,
+a subsequent LDAP user synchronization does not automatically unblock those users.
+
### Adjust LDAP user sync schedule
By default, GitLab runs a worker once per day at 01:30 a.m. server time to
@@ -387,6 +506,23 @@ To enable global group memberships lock:
1. Expand the **Visibility and access controls** section.
1. Ensure the **Lock memberships to LDAP synchronization** checkbox is selected.
+### Change LDAP group synchronization settings management
+
+By default, group members with the Owner role can manage [LDAP group synchronization settings](../../../user/group/access_and_permissions.md#manage-group-memberships-via-ldap).
+
+GitLab administrators can remove this permission from group Owners:
+
+1. [Configure LDAP](index.md#configure-ldap).
+1. On the top bar, select **Main menu > Admin**.
+1. On the left sidebar, select **Settings > General**.
+1. Expand **Visibility and access controls**.
+1. Ensure the **Allow group owners to manage LDAP-related settings** checkbox is not checked.
+
+When **Allow group owners to manage LDAP-related settings** is disabled:
+
+- Group Owners cannot change LDAP synchronization settings for either top-level groups and subgroups.
+- Instance administrators can manage LDAP group synchronization settings on all groups on an instance.
+
### Adjust LDAP group sync schedule
By default, GitLab runs a group sync process every hour, on the hour.
@@ -411,7 +547,7 @@ sync to run once every two hours at the top of the hour.
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
- gitlab_rails['ldap_group_sync_worker_cron'] = "0 */2 * * * *"
+ gitlab_rails['ldap_group_sync_worker_cron'] = "0 */2 * * *"
```
1. Save the file and reconfigure GitLab:
@@ -454,7 +590,7 @@ sync to run once every two hours at the top of the hour.
gitlab:
environment:
GITLAB_OMNIBUS_CONFIG: |
- gitlab_rails['ldap_group_sync_worker_cron'] = "0 */2 * * * *"
+ gitlab_rails['ldap_group_sync_worker_cron'] = "0 */2 * * *"
```
1. Save the file and restart GitLab:
diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md
index ea6922629d8..106cc6c23eb 100644
--- a/doc/administration/auth/oidc.md
+++ b/doc/administration/auth/oidc.md
@@ -29,7 +29,7 @@ The OpenID Connect provides you with a client's details and secret for you to us
sudo -u git -H editor config/gitlab.yml
```
-1. Edit the [common configuration file settings](../../integration/omniauth.md#configure-common-settings)
+1. Configure the [common settings](../../integration/omniauth.md#configure-common-settings)
to add `openid_connect` as a single sign-on provider. This enables Just-In-Time
account provisioning for users who do not have an existing GitLab account.
@@ -40,7 +40,7 @@ The OpenID Connect provides you with a client's details and secret for you to us
```ruby
gitlab_rails['omniauth_providers'] = [
{
- name: "openid_connect",
+ name: "openid_connect", # do not change this parameter
label: "Provider name", # optional label for login button, defaults to "Openid Connect"
icon: "<custom_provider_icon>",
args: {
@@ -63,10 +63,55 @@ The OpenID Connect provides you with a client's details and secret for you to us
]
```
+ For Omnibus GitLab with multiple identity providers:
+
+ ```ruby
+ { 'name' => 'openid_connect',
+ 'label' => '...',
+ 'icon' => '...',
+ 'args' => {
+ 'name' => 'openid_connect',
+ 'strategy_class': 'OmniAuth::Strategies::OpenIDConnect',
+ 'scope' => ['openid', 'profile', 'email'],
+ 'discovery' => true,
+ 'response_type' => 'code',
+ 'issuer' => 'https://...',
+ 'client_auth_method' => 'query',
+ 'uid_field' => '...',
+ 'client_options' => {
+ `identifier`: "<your_oidc_client_id>",
+ `secret`: "<your_oidc_client_secret>",
+ 'redirect_uri' => 'https://.../users/auth/openid_connect/callback'
+ }
+ }
+ },
+ { 'name' => 'openid_connect_2fa',
+ 'label' => '...',
+ 'icon' => '...',
+ 'args' => {
+ 'name' => 'openid_connect_2fa',
+ 'strategy_class': 'OmniAuth::Strategies::OpenIDConnect',
+ 'scope' => ['openid', 'profile', 'email'],
+ 'discovery' => true,
+ 'response_type' => 'code',
+ 'issuer' => 'https://...',
+ 'client_auth_method' => 'query',
+ 'uid_field' => '...',
+ 'client_options' => {
+ ...
+ 'redirect_uri' => 'https://.../users/auth/openid_connect_2fa/callback'
+ }
+ }
+ }
+ ```
+
+ NOTE:
+ For more information on using multiple identity providers with OIDC, see [issue 5992](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5992).
+
For installation from source:
```yaml
- - { name: 'openid_connect',
+ - { name: 'openid_connect', # do not change this parameter
label: 'Provider name', # optional label for login button, defaults to "Openid Connect"
icon: '<custom_provider_icon>',
args: {
@@ -89,10 +134,7 @@ The OpenID Connect provides you with a client's details and secret for you to us
```
NOTE:
- For more information on each configuration option, refer to the:
-
- - [OmniAuth OpenID Connect usage documentation](https://github.com/m0n9oose/omniauth_openid_connect#usage).
- - [OpenID Connect Core 1.0 specification](https://openid.net/specs/openid-connect-core-1_0.html).
+ For more information on each configuration option, refer to the [OmniAuth OpenID Connect usage documentation](https://github.com/omniauth/omniauth_openid_connect#usage) and [OpenID Connect Core 1.0 specification](https://openid.net/specs/openid-connect-core-1_0.html).
1. For the provider configuration, change the values for the provider to match your
OpenID Connect client setup. Use the following as a guide:
@@ -129,7 +171,7 @@ The OpenID Connect provides you with a client's details and secret for you to us
- `client_options` are the OpenID Connect client-specific options. Specifically:
- `identifier` is the client identifier as configured in the OpenID Connect service provider.
- `secret` is the client secret as configured in the OpenID Connect service provider. For example,
- [OmniAuth OpenIDConnect](https://github.com/omniauth/omniauth_openid_connect)) requires this. If the service provider doesn't require a secret,
+ [OmniAuth OpenID Connect](https://github.com/omniauth/omniauth_openid_connect) requires this. If the service provider doesn't require a secret,
provide any value and it is ignored.
- `redirect_uri` is the GitLab URL to redirect the user to after successful login
(for example, `http://example.com/users/auth/openid_connect/callback`).
@@ -165,7 +207,7 @@ for more details:
```ruby
gitlab_rails['omniauth_providers'] = [
{
- name: "openid_connect",
+ name: "openid_connect", # do not change this parameter
label: "Google OpenID", # optional label for login button, defaults to "Openid Connect"
args: {
name: "openid_connect",
@@ -203,7 +245,7 @@ Example Omnibus configuration block:
```ruby
gitlab_rails['omniauth_providers'] = [
{
- name: "openid_connect",
+ name: "openid_connect", # do not change this parameter
label: "Azure OIDC", # optional label for login button, defaults to "Openid Connect"
args: {
name: "openid_connect",
@@ -335,7 +377,7 @@ but `LocalAccounts` authenticates against local Active Directory accounts. Befor
```ruby
gitlab_rails['omniauth_providers'] = [
{
- name: "openid_connect",
+ name: "openid_connect", # do not change this parameter
label: "Azure B2C OIDC", # optional label for login button, defaults to "Openid Connect"
args: {
name: "openid_connect",
@@ -395,7 +437,7 @@ Example Omnibus configuration block:
```ruby
gitlab_rails['omniauth_providers'] = [
{
- name: "openid_connect",
+ name: "openid_connect", # do not change this parameter
label: "Keycloak", # optional label for login button, defaults to "Openid Connect"
args: {
name: "openid_connect",
@@ -475,7 +517,7 @@ To use symmetric key encryption:
```ruby
gitlab_rails['omniauth_providers'] = [
{
- name: "openid_connect",
+ name: "openid_connect", # do not change this parameter
label: "Keycloak", # optional label for login button, defaults to "Openid Connect"
args: {
name: "openid_connect",
@@ -519,7 +561,7 @@ Example Omnibus GitLab configuration (file path: `/etc/gitlab/gitlab.rb`):
```ruby
gitlab_rails['omniauth_providers'] = [
{
- name: "openid_connect",
+ name: "openid_connect", # do not change this parameter
label: "Casdoor", # optional label for login button, defaults to "Openid Connect"
args: {
name: "openid_connect",
@@ -542,7 +584,7 @@ gitlab_rails['omniauth_providers'] = [
Example installations from source configuration (file path: `config/gitlab.yml`):
```yaml
- - { name: 'openid_connect',
+ - { name: 'openid_connect', # do not change this parameter
label: 'Casdoor', # optional label for login button, defaults to "Openid Connect"
args: {
name: 'openid_connect',
@@ -561,6 +603,401 @@ Example installations from source configuration (file path: `config/gitlab.yml`)
}
```
+## Configure multiple OpenID Connect providers
+
+You can configure your application to use multiple OpenID Connect (OIDC) providers. You do this by explicitly setting the `strategy_class` in your configuration file.
+
+You should do this in either of the following scenarios:
+
+- [Migrating to the OpenID Connect protocol](../../integration/azure.md#migrate-to-the-openid-connect-protocol).
+- Offering different levels of authentication.
+
+NOTE:
+This is not compatible with [configuring users based on OIDC group membership](#configure-users-based-on-oidc-group-membership). For more information, see [issue 408248](https://gitlab.com/gitlab-org/gitlab/-/issues/408248).
+
+The following example configurations show how to offer different levels of authentication, one option with 2FA and one without 2FA.
+
+For Omnibus GitLab:
+
+```ruby
+gitlab_rails['omniauth_providers'] = [
+ {
+ name: "openid_connect",
+ label: "Provider name", # optional label for login button, defaults to "Openid Connect"
+ icon: "<custom_provider_icon>",
+ args: {
+ name: "openid_connect",
+ strategy_class: "OmniAuth::Strategies::OpenIDConnect",
+ scope: ["openid","profile","email"],
+ response_type: "code",
+ issuer: "<your_oidc_url>",
+ discovery: true,
+ client_auth_method: "query",
+ uid_field: "<uid_field>",
+ send_scope_to_token_endpoint: "false",
+ pkce: true,
+ client_options: {
+ identifier: "<your_oidc_client_id>",
+ secret: "<your_oidc_client_secret>",
+ redirect_uri: "<your_gitlab_url>/users/auth/openid_connect/callback"
+ }
+ }
+ },
+ {
+ name: "openid_connect_2fa",
+ label: "Provider name 2FA", # optional label for login button, defaults to "Openid Connect"
+ icon: "<custom_provider_icon>",
+ args: {
+ name: "openid_connect_2fa",
+ strategy_class: "OmniAuth::Strategies::OpenIDConnect",
+ scope: ["openid","profile","email"],
+ response_type: "code",
+ issuer: "<your_oidc_url>",
+ discovery: true,
+ client_auth_method: "query",
+ uid_field: "<uid_field>",
+ send_scope_to_token_endpoint: "false",
+ pkce: true,
+ client_options: {
+ identifier: "<your_oidc_client_id>",
+ secret: "<your_oidc_client_secret>",
+ redirect_uri: "<your_gitlab_url>/users/auth/openid_connect_2fa/callback"
+ }
+ }
+ }
+]
+```
+
+For installation from source:
+
+```yaml
+ - { name: 'openid_connect',
+ label: 'Provider name', # optional label for login button, defaults to "Openid Connect"
+ icon: '<custom_provider_icon>',
+ args: {
+ name: 'openid_connect',
+ strategy_class: "OmniAuth::Strategies::OpenIDConnect",
+ scope: ['openid','profile','email'],
+ response_type: 'code',
+ issuer: '<your_oidc_url>',
+ discovery: true,
+ client_auth_method: 'query',
+ uid_field: '<uid_field>',
+ send_scope_to_token_endpoint: false,
+ pkce: true,
+ client_options: {
+ identifier: '<your_oidc_client_id>',
+ secret: '<your_oidc_client_secret>',
+ redirect_uri: '<your_gitlab_url>/users/auth/openid_connect/callback'
+ }
+ }
+ }
+ - { name: 'openid_connect_2fa',
+ label: 'Provider name 2FA', # optional label for login button, defaults to "Openid Connect"
+ icon: '<custom_provider_icon>',
+ args: {
+ name: 'openid_connect_2fa',
+ strategy_class: "OmniAuth::Strategies::OpenIDConnect",
+ scope: ['openid','profile','email'],
+ response_type: 'code',
+ issuer: '<your_oidc_url>',
+ discovery: true,
+ client_auth_method: 'query',
+ uid_field: '<uid_field>',
+ send_scope_to_token_endpoint: false,
+ pkce: true,
+ client_options: {
+ identifier: '<your_oidc_client_id>',
+ secret: '<your_oidc_client_secret>',
+ redirect_uri: '<your_gitlab_url>/users/auth/openid_connect_2fa/callback'
+ }
+ }
+ }
+```
+
+In this use case, you might want to synchronize the `extern_uid` across the
+different providers based on an existing known identifier in your
+corporate directory.
+
+To do this, you set the `uid_field`. The following example code shows how to
+do this:
+
+```python
+def sync_missing_provider(self, user: User, extern_uid: str)
+ existing_identities = []
+ for identity in user.identities:
+ existing_identities.append(identity.get("provider"))
+
+ local_extern_uid = extern_uid.lower()
+ for provider in ("openid_connect_2fa", "openid_connect"):
+ identity = [
+ identity
+ for identity in user.identities
+ if identity.get("provider") == provider
+ and identity.get("extern_uid").lower() != local_extern_uid
+ ]
+ if provider not in existing_identities or identity:
+ if identity and identity[0].get("extern_uid") != "":
+ logger.error(f"Found different identity for provider {provider} for user {user.id}")
+ continue
+ else:
+ logger.info(f"Add identity 'provider': {provider}, 'extern_uid': {extern_uid} for user {user.id}")
+ user.provider = provider
+ user.extern_uid = extern_uid
+ user = self.save_user(user)
+ return user
+```
+
+For more information, see the [GitLab API user method documentation](https://python-gitlab.readthedocs.io/en/stable/gl_objects/users.html#examples).
+
+## Configure users based on OIDC group membership **(PREMIUM)**
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/209898) in GitLab 15.10.
+
+You can configure OIDC group membership to:
+
+- Require users to be members of a certain group.
+- Assign users [external roles](../../user/admin_area/external_users.md), or as
+ administrators based on group membership.
+
+GitLab checks these groups on each sign in and updates user attributes as necessary.
+This feature **does not** allow you to automatically add users to GitLab
+[groups](../../user/group/index.md).
+
+### Required groups
+
+Your identity provider (IdP) must pass group information to GitLab in the OIDC response. To use this
+response to require users to be members of a certain group, configure GitLab to identify:
+
+- Where to look for the groups in the OIDC response, using the `groups_attribute` setting.
+- Which group membership is required to sign in, using the `required_groups` setting.
+
+If you do not set `required_groups` or leave the setting empty, any user authenticated by the IdP through OIDC can use GitLab.
+
+For Omnibus GitLab:
+
+1. Edit `/etc/gitlab/gitlab.rb`:
+
+ ```ruby
+ gitlab_rails['omniauth_providers'] = [
+ {
+ name: "openid_connect",
+ label: "Provider name",
+ args: {
+ name: "openid_connect",
+ scope: ["openid","profile","email"],
+ response_type: "code",
+ issuer: "<your_oidc_url>",
+ discovery: true,
+ client_auth_method: "query",
+ uid_field: "<uid_field>",
+ client_options: {
+ identifier: "<your_oidc_client_id>",
+ secret: "<your_oidc_client_secret>",
+ redirect_uri: "<your_gitlab_url>/users/auth/openid_connect/callback",
+ gitlab: {
+ groups_attribute: "groups",
+ required_groups: ["Developer"]
+ }
+ }
+ }
+ }
+ ]
+ ```
+
+1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure)
+ for the changes to take effect.
+
+For installation from source:
+
+1. Edit `/home/git/gitlab/config/gitlab.yml`:
+
+ ```yaml
+ production: &base
+ omniauth:
+ providers:
+ - { name: 'openid_connect',
+ label: 'Provider name',
+ args: {
+ name: 'openid_connect',
+ scope: ['openid','profile','email'],
+ response_type: 'code',
+ issuer: '<your_oidc_url>',
+ discovery: true,
+ client_auth_method: 'query',
+ uid_field: '<uid_field>',
+ client_options: {
+ identifier: '<your_oidc_client_id>',
+ secret: '<your_oidc_client_secret>',
+ redirect_uri: '<your_gitlab_url>/users/auth/openid_connect/callback',
+ gitlab: {
+ groups_attribute: "groups",
+ required_groups: ["Developer"]
+ }
+ }
+ }
+ }
+ ```
+
+1. Save the file and [reconfigure GitLab](../restart_gitlab.md#installations-from-source)
+ for the changes to take effect.
+
+### External groups
+
+Your IdP must pass group information to GitLab in the OIDC response. To use this
+response to identify users as [external users](../../user/admin_area/external_users.md)
+based on group membership, configure GitLab to identify:
+
+- Where to look for the groups in the OIDC response, using the `groups_attribute` setting.
+- Which group memberships should identify a user as an
+ [external user](../../user/admin_area/external_users.md), using the
+ `external_groups` setting.
+
+For Omnibus GitLab:
+
+1. Edit `/etc/gitlab/gitlab.rb`:
+
+ ```ruby
+ gitlab_rails['omniauth_providers'] = [
+ {
+ name: "openid_connect",
+ label: "Provider name",
+ args: {
+ name: "openid_connect",
+ scope: ["openid","profile","email"],
+ response_type: "code",
+ issuer: "<your_oidc_url>",
+ discovery: true,
+ client_auth_method: "query",
+ uid_field: "<uid_field>",
+ client_options: {
+ identifier: "<your_oidc_client_id>",
+ secret: "<your_oidc_client_secret>",
+ redirect_uri: "<your_gitlab_url>/users/auth/openid_connect/callback",
+ gitlab: {
+ groups_attribute: "groups",
+ external_groups: ["Freelancer"]
+ }
+ }
+ }
+ }
+ ]
+ ```
+
+1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure)
+ for the changes to take effect.
+
+For installation from source:
+
+1. Edit `/home/git/gitlab/config/gitlab.yml`:
+
+ ```yaml
+ production: &base
+ omniauth:
+ providers:
+ - { name: 'openid_connect',
+ label: 'Provider name',
+ args: {
+ name: 'openid_connect',
+ scope: ['openid','profile','email'],
+ response_type: 'code',
+ issuer: '<your_oidc_url>',
+ discovery: true,
+ client_auth_method: 'query',
+ uid_field: '<uid_field>',
+ client_options: {
+ identifier: '<your_oidc_client_id>',
+ secret: '<your_oidc_client_secret>',
+ redirect_uri: '<your_gitlab_url>/users/auth/openid_connect/callback',
+ gitlab: {
+ groups_attribute: "groups",
+ external_groups: ["Freelancer"]
+ }
+ }
+ }
+ }
+ ```
+
+1. Save the file and [reconfigure GitLab](../restart_gitlab.md#installations-from-source)
+ for the changes to take effect.
+
+### Administrator groups
+
+Your IdP must pass group information to GitLab in the OIDC response. To use this
+response to assign users as administrator based on group membership, configure GitLab to identify:
+
+- Where to look for the groups in the OIDC response, using the `groups_attribute` setting.
+- Which group memberships grant the user administrator access, using the
+ `admin_groups` setting.
+
+For Omnibus GitLab:
+
+1. Edit `/etc/gitlab/gitlab.rb`:
+
+ ```ruby
+ gitlab_rails['omniauth_providers'] = [
+ {
+ name: "openid_connect",
+ label: "Provider name",
+ args: {
+ name: "openid_connect",
+ scope: ["openid","profile","email"],
+ response_type: "code",
+ issuer: "<your_oidc_url>",
+ discovery: true,
+ client_auth_method: "query",
+ uid_field: "<uid_field>",
+ client_options: {
+ identifier: "<your_oidc_client_id>",
+ secret: "<your_oidc_client_secret>",
+ redirect_uri: "<your_gitlab_url>/users/auth/openid_connect/callback",
+ gitlab: {
+ groups_attribute: "groups",
+ admin_groups: ["Admin"]
+ }
+ }
+ }
+ }
+ ]
+ ```
+
+1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure)
+ for the changes to take effect.
+
+For installation from source:
+
+1. Edit `/home/git/gitlab/config/gitlab.yml`:
+
+ ```yaml
+ production: &base
+ omniauth:
+ providers:
+ - { name: 'openid_connect',
+ label: 'Provider name',
+ args: {
+ name: 'openid_connect',
+ scope: ['openid','profile','email'],
+ response_type: 'code',
+ issuer: '<your_oidc_url>',
+ discovery: true,
+ client_auth_method: 'query',
+ uid_field: '<uid_field>',
+ client_options: {
+ identifier: '<your_oidc_client_id>',
+ secret: '<your_oidc_client_secret>',
+ redirect_uri: '<your_gitlab_url>/users/auth/openid_connect/callback',
+ gitlab: {
+ groups_attribute: "groups",
+ admin_groups: ["Admin"]
+ }
+ }
+ }
+ }
+ ```
+
+1. Save the file and [reconfigure GitLab](../restart_gitlab.md#installations-from-source)
+ for the changes to take effect.
+
## Troubleshooting
1. Ensure `discovery` is set to `true`. If you set it to `false`, you must
@@ -568,7 +1005,7 @@ Example installations from source configuration (file path: `config/gitlab.yml`)
1. Check your system clock to ensure the time is synchronized properly.
-1. As mentioned in [the OmniAuth OpenID Connect documentation](https://github.com/m0n9oose/omniauth_openid_connect),
+1. As mentioned in [the OmniAuth OpenID Connect documentation](https://github.com/omniauth/omniauth_openid_connect),
make sure `issuer` corresponds to the base URL of the Discovery URL. For
example, `https://accounts.google.com` is used for the URL
`https://accounts.google.com/.well-known/openid-configuration`.
diff --git a/doc/administration/auth/test_oidc_oauth.md b/doc/administration/auth/test_oidc_oauth.md
new file mode 100644
index 00000000000..95cca1ced86
--- /dev/null
+++ b/doc/administration/auth/test_oidc_oauth.md
@@ -0,0 +1,57 @@
+---
+stage: Manage
+group: Authentication and Authorization
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Test OIDC/OAuth in GitLab **(FREE)**
+
+To test OIDC/OAuth in GitLab, you must:
+
+1. [Enable OIDC/OAuth](#enable-oidcoauth-in-gitlab)
+1. [Test OIDC/OAuth with your client application](#test-oidcoauth-with-your-client-application)
+1. [Verify OIDC/OAuth authentication](#verify-oidcoauth-authentication)
+
+## Prerequisites
+
+Before you can test OIDC/OAuth on GitLab, you'll need the following:
+
+- Publicly accessible GitLab instance
+- A client application that you want to use to test OIDC/OAuth
+- A user account on the GitLab instance that you can use to log in and test OIDC/OAuth
+
+## Enable OIDC/OAuth in GitLab
+
+First, you must create OIDC/OAuth application on your GitLab instance. To do this:
+
+1. Sign in to GitLab as an administrator.
+1. Select **Profile > Preferences > Applications**.
+1. Fill in the details for your client application, including the name, redirect URI, and allowed scopes.
+1. Make sure the `openid` scope is enabled.
+1. Select **Save application** to create the new OAuth application.
+
+## Test OIDC/OAuth with your client application
+
+After you've created your OAuth application in GitLab, you can use it to test OIDC/OAuth:
+
+1. You can use <https://openidconnect.net> as the OIDC/OAuth playground.
+1. Sign out of GitLab.
+1. Visit your client application and initiate the OIDC/OAuth flow, using the GitLab OAuth application you created in the previous step.
+1. Follow the prompts to sign in to GitLab and authorize the client application to access your GitLab account.
+1. After you've completed the OIDC/OAuth flow, your client application should have received an access token that it can use to authenticate with GitLab.
+
+## Verify OIDC/OAuth authentication
+
+To verify that OIDC/OAuth authentication is working correctly on GitLab, you can perform the following checks:
+
+1. Check that the access token you received in the previous step is valid and can be used to authenticate with GitLab. You can do this by making a test API request to GitLab, using the access token to authenticate. For example:
+
+ ```shell
+ curl --header "Authorization: Bearer <access_token>" https://mygitlabinstance.com/api/v4/user
+ ```
+
+ Replace `<access_token>` with the actual access token you received in the previous step. If the API request succeeds and returns information about the authenticated user, then OIDC/OAuth authentication is working correctly.
+
+1. Check that the scopes you specified in your OAuth application are being enforced correctly. You can do this by making API requests that require the specific scopes and checking that they succeed or fail as expected.
+
+That's it! With these steps, you should be able to test OIDC/OAuth authentication on your GitLab instance using your client application.
diff --git a/doc/administration/clusters/kas.md b/doc/administration/clusters/kas.md
index a7f8f8e712b..6d6e8e5513c 100644
--- a/doc/administration/clusters/kas.md
+++ b/doc/administration/clusters/kas.md
@@ -1,6 +1,6 @@
---
-stage: Configure
-group: Configure
+stage: Deploy
+group: Environments
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
@@ -43,6 +43,33 @@ To enable the agent server on a single node:
For additional configuration options, see the **Enable GitLab KAS** section of the
[`gitlab.rb.template`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/files/gitlab-config-template/gitlab.rb.template).
+##### Configure KAS to listen on a UNIX socket
+
+If you use GitLab behind a proxy, KAS might not work correctly. You can resolve this issue on a single-node installation, you can configure KAS to listen on a UNIX socket.
+
+To configure KAS to listen on a UNIX socket:
+
+1. Create a directory for the KAS sockets:
+
+ ```shell
+ sudo mkdir -p /var/opt/gitlab/gitlab-kas/sockets/
+ ```
+
+1. Edit `/etc/gitlab/gitlab.rb`:
+
+ ```ruby
+ gitlab_kas['internal_api_listen_network'] = 'unix'
+ gitlab_kas['internal_api_listen_address'] = '/var/opt/gitlab/gitlab-kas/sockets/internal-api.socket'
+ gitlab_kas['private_api_listen_network'] = 'unix'
+ gitlab_kas['private_api_listen_address'] = '/var/opt/gitlab/gitlab-kas/sockets/private-api.socket'
+ gitlab_kas['env'] = {
+ 'SSL_CERT_DIR' => "/opt/gitlab/embedded/ssl/certs/",
+ 'OWN_PRIVATE_API_URL' => 'unix:///var/opt/gitlab/gitlab-kas/sockets/private-api.socket'
+ }
+ ```
+
+1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+
#### Enable on multiple nodes
To enable the agent server on multiple nodes:
@@ -50,6 +77,8 @@ To enable the agent server on multiple nodes:
1. For each agent server node, edit `/etc/gitlab/gitlab.rb`:
```ruby
+ gitlab_kas_external_url 'wss://kas.gitlab.example.com/'
+
gitlab_kas['enable'] = true
gitlab_kas['api_secret_key'] = '<32_bytes_long_base64_encoded_value>'
gitlab_kas['private_api_secret_key'] = '<32_bytes_long_base64_encoded_value>'
@@ -65,17 +94,23 @@ To enable the agent server on multiple nodes:
gitlab_rails['gitlab_kas_external_k8s_proxy_url'] = 'https://gitlab.example.com/-/kubernetes-agent/k8s-proxy/'
```
- In this configuration:
+1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
- - `gitlab_kas['private_api_listen_address']` is the address the agent server listens on. You can set it to `0.0.0.0` or an IP address reachable by other nodes in the cluster.
- - `OWN_PRIVATE_API_URL` is the environment variable used by the KAS process for service discovery. You can set it to a hostname or IP address of the node you're configuring. The node must be reachable by other nodes in the cluster.
- - `gitlab_kas['api_secret_key']` is the shared secret used for authentication between KAS and GitLab. This value must be Base64-encoded and exactly 32 bytes long.
- - `gitlab_kas['private_api_secret_key']` is the shared secret used for authentication between different KAS instances. This value must be Base64-encoded and exactly 32 bytes long.
- - `gitlab_rails['gitlab_kas_external_url']` is the user-facing URL for the in-cluster `agentk`.
- - `gitlab_rails['gitlab_kas_internal_url']` is the internal URL the GitLab backend uses to communicate with KAS.
- - `gitlab_rails['gitlab_kas_external_k8s_proxy_url']` is the user-facing URL for Kubernetes API proxying.
+##### Agent server node settings
-1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+| Setting | Description |
+|---------|-------------|
+| `gitlab_kas['private_api_listen_address']` | The address the agent server listens on. Set to `0.0.0.0` or to an IP address reachable by other nodes in the cluster. |
+| `gitlab_kas['api_secret_key']` | The shared secret used for authentication between KAS and GitLab. The value must be Base64-encoded and exactly 32 bytes long. |
+| `gitlab_kas['private_api_secret_key']` | The shared secret used for authentication between different KAS instances. The value must be Base64-encoded and exactly 32 bytes long. |
+| `OWN_PRIVATE_API_URL` | The environment variable used by KAS for service discovery. Set to the hostname or IP address of the node you're configuring. The node must be reachable by other nodes in the cluster. |
+| `gitlab_kas_external_url` | The user-facing URL for the in-cluster `agentk`. Can be a fully qualified domain or subdomain, <sup>**1**</sup> or a GitLab external URL. <sup>**2**</sup> If blank, defaults to a GitLab external URL. |
+| `gitlab_rails['gitlab_kas_external_url']` | The user-facing URL for the in-cluster `agentk`. If blank, defaults to the `gitlab_kas_external_url`. |
+| `gitlab_rails['gitlab_kas_external_k8s_proxy_url']` | The user-facing URL for Kubernetes API proxying. If blank, defaults to a URL based on `gitlab_kas_external_url`. |
+| `gitlab_rails['gitlab_kas_internal_url']` | The internal URL the GitLab backend uses to communicate with KAS. |
+
+1. For example, `wss://kas.gitlab.example.com/`.
+1. For example, `wss://gitlab.example.com/-/kubernetes-agent/`.
### For GitLab Helm Chart
@@ -105,6 +140,24 @@ For GitLab [Helm Chart](https://docs.gitlab.com/charts/) installations:
For details, see [how to use the GitLab-KAS chart](https://docs.gitlab.com/charts/charts/gitlab/kas/).
+## Kubernetes API proxy cookie
+
+> Introduced in GitLab 15.10 [with feature flags](../feature_flags.md) named `kas_user_access` and `kas_user_access_project`. Disabled by default.
+
+FLAG:
+On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flags](../feature_flags.md) named `kas_user_access` and `kas_user_access_project`.
+
+KAS proxies Kubernetes API requests to the GitLab agent with either:
+
+- A [CI/CD job](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/kubernetes_ci_access.md).
+- [GitLab user credentials](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/kubernetes_user_access.md).
+
+To authenticate with user credentials, Rails sets a cookie for the GitLab frontend.
+This cookie is called `_gitlab_kas` and it contains an encrypted
+session ID, like the [`_gitlab_session` cookie](../../user/profile/index.md#cookies-used-for-sign-in).
+The `_gitlab_kas` cookie must be sent to the KAS proxy endpoint with every request
+to authenticate and authorize the user.
+
## Troubleshooting
If you have issues while using the agent server for Kubernetes, view the
diff --git a/doc/administration/compliance.md b/doc/administration/compliance.md
index 592c110b446..9ae0e9d7790 100644
--- a/doc/administration/compliance.md
+++ b/doc/administration/compliance.md
@@ -18,25 +18,13 @@ standards or mandates from regulatory bodies. The following features help you
define rules and policies to adhere to workflow requirements, separation of duties,
and secure supply chain best practices:
-- [**Credentials inventory**](../user/admin_area/credentials_inventory.md) (for
- instances): With a credentials inventory, GitLab administrators can keep track
- of the credentials used by all of the users in their GitLab instance.
-- [**Granular user roles and flexible permissions**](../user/permissions.md)
- (for instances, groups, and projects): Manage access and permissions with five
- different user roles and settings for external users. Set permissions according
- to people's role, rather than either read or write access to a repository. Don't
- share the source code with people that only need access to the issue tracker.
-- [**Merge request approvals**](../user/project/merge_requests/approvals/index.md)
- (for instances, groups, and projects): Configure approvals required for
- merge requests.
-- [**Push rules**](../user/project/repository/push_rules.md) (for instances, groups, and
- projects): Control pushes to your repositories.
-- Separation of duties using [**protected branches**](../user/project/protected_branches.md#require-code-owner-approval-on-a-protected-branch)
- and [**custom CI/CD configuration paths**](../ci/pipelines/settings.md#specify-a-custom-cicd-configuration-file) (for projects): Users can leverage the GitLab cross-project YAML configurations
- to define deployers of code and developers of code. See how to use this setup
- to define these roles in:
- - The [Separation of Duties deploy project](https://gitlab.com/guided-explorations/separation-of-duties-deploy/blob/master/README.md).
- - The [Separation of Duties project](https://gitlab.com/guided-explorations/separation-of-duties/blob/master/README.md).
+| Feature | Instances | Groups | Projects | Description |
+|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------|:-----------------------|:-----------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| [Credentials inventory](../user/admin_area/credentials_inventory.md) | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Keep track of the credentials used by all of the users in a GitLab instance. |
+| [Granular user roles<br/>and flexible permissions](../user/permissions.md) | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Manage access and permissions with five different user roles and settings for external users. Set permissions according to people's role, rather than either read or write access to a repository. Don't share the source code with people that only need access to the issue tracker. |
+| [Merge request approvals](../user/project/merge_requests/approvals/index.md) | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Configure approvals required for merge requests. |
+| [Push rules](../user/project/repository/push_rules.md) | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Control pushes to your repositories. |
+| Separation of duties using<br/>[protected branches](../user/project/protected_branches.md#require-code-owner-approval-on-a-protected-branch) and<br/>[custom CI/CD configuration paths](../ci/pipelines/settings.md#specify-a-custom-cicd-configuration-file) | **{dotted-circle}** No | **{dotted-circle}** No | **{check-circle}** Yes | Leverage the GitLab cross-project YAML configurations to define deployers of code and developers of code. See how to use this setup to define these roles in the [Separation of Duties deploy project](https://gitlab.com/guided-explorations/separation-of-duties-deploy/blob/master/README.md) and the [Separation of Duties project](https://gitlab.com/guided-explorations/separation-of-duties/blob/master/README.md). |
## Compliant workflow automation
@@ -48,66 +36,42 @@ settings and automation to ensure that whatever a compliance team has configured
stays configured and working correctly. These features can help you automate
compliance:
-- [**Compliance frameworks**](../user/group/compliance_frameworks.md) (for groups): Create a custom
- compliance framework at the group level to describe the type of compliance requirements any child project needs to follow.
-- [**Compliance pipelines**](../user/group/compliance_frameworks.md#compliance-pipelines) (for groups): Define a
- pipeline configuration to run for any projects with a given compliance framework.
+| Feature | Instances | Groups | Projects | Description |
+|:------------------------------------------------------------------------------------|:-----------------------|:-----------------------|:-----------------------|:-------------------------------------------------------------------------------------------|
+| [Compliance frameworks](../user/group/compliance_frameworks.md) | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Describe the type of compliance requirements projects must follow. |
+| [Compliance pipelines](../user/group/compliance_frameworks.md#compliance-pipelines) | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Define a pipeline configuration to run for any projects with a given compliance framework. |
## Audit management
An important part of any compliance program is being able to go back and understand
-what happened, when it happened, and who was responsible. This is useful in audit
+what happened, when it happened, and who was responsible. You can use this in audit
situations as well as for understanding the root cause of issues when they occur.
-It is useful to have both low-level, raw lists of audit data as well as high-level,
+
+It is helpful to have both low-level, raw lists of audit data as well as high-level,
summary lists of audit data. Between these two, compliance teams can quickly
identify if problems exist and then drill down into the specifics of those issues.
These features can help provide visibility into GitLab and audit what is happening:
-- [**Audit events**](audit_events.md) (for instances, groups, and projects): To
- maintain the integrity of your code, audit events give administrators the
- ability to view any modifications made within the GitLab server in an advanced
- audit events system, so you can control, analyze, and track every change.
-- [**Audit reports**](audit_reports.md) (for instances, groups, and projects):
- Create and access reports based on the audit events that have occurred. Use
- pre-built GitLab reports or the API to build your own.
-- [**Auditor users**](auditor_users.md) (for instances): Auditor users are users
- who are given read-only access to all projects, groups, and other resources on
- the GitLab instance.
-- [**Compliance report**](../user/compliance/compliance_report/index.md) (for
- groups): Quickly get visibility into the compliance posture of your organization.
+| Feature | Instances | Groups | Projects | Description |
+|:-------------------------------------------------------------------|:-----------------------|:-----------------------|:-----------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| [Audit events](audit_events.md) | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | To maintain the integrity of your code, audit events give administrators the ability to view any modifications made in the GitLab server in an advanced audit events system, so you can control, analyze, and track every change. |
+| [Audit reports](audit_reports.md) | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Create and access reports based on the audit events that have occurred. Use pre-built GitLab reports or the API to build your own. |
+| [Auditor users](auditor_users.md) | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Auditor users are users who are given read-only access to all projects, groups, and other resources on the GitLab instance. |
+| [Compliance report](../user/compliance/compliance_report/index.md) | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Quickly get visibility into the compliance posture of your organization. |
## Other compliance features
These features can also help with compliance requirements:
-- [**Email all users of a project, group, or entire server**](../user/admin_area/email_from_gitlab.md)
- (for instances): An administrator can email groups of users based on project
- or group membership, or email everyone using the GitLab instance. These emails
- are great for scheduled maintenance or upgrades.
-- [**Enforce ToS acceptance**](../user/admin_area/settings/terms.md) (for
- instances): Enforce your users accepting new terms of service by blocking GitLab
- traffic.
-- [**External Status Checks**](../user/project/merge_requests/status_checks.md)
- (for projects): Interface with third-party systems you already use during
- development to ensure you remain compliant.
-- [**Generate reports on permission levels of users**](../user/admin_area/index.md#user-permission-export)
- (for instances): Administrators can generate a report listing all users' access
- permissions for groups and projects in the instance.
-- [**License compliance**](../user/compliance/license_compliance/index.md) (for
- projects): Search dependencies for their licenses. This lets you determine if
- the licenses of your project's dependencies are compatible with your project's
- license.
-- [**Lock project membership to group**](../user/group/access_and_permissions.md#prevent-members-from-being-added-to-projects-in-a-group)
- (for groups): Group owners can prevent new members from being added to projects
- within a group.
-- [**LDAP group sync**](auth/ldap/ldap_synchronization.md#group-sync) (for
- instances): Gives administrators the ability to automatically sync groups and
- manage SSH keys, permissions, and authentication, so you can focus on building
- your product, not configuring your tools.
-- [**LDAP group sync filters**](auth/ldap/ldap_synchronization.md#group-sync)
- (for instances): Gives more flexibility to synchronize with LDAP based on
- filters, meaning you can leverage LDAP attributes to map GitLab permissions.
-- [**Omnibus GitLab package supports log forwarding**](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-forwarding)
- (for instances): Forward your logs to a central system.
-- [**Restrict SSH Keys**](../security/ssh_keys_restrictions.md) (for instances):
- Control the technology and key length of SSH keys used to access GitLab.
+| Feature | Instances | Groups | Projects | Description |
+|:------------------------------------------------------------------------------------------------------------------------------------|:-----------------------|:-----------------------|:-----------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| [Email all users of a project,<br/>group, or entire server](../user/admin_area/email_from_gitlab.md) | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Email groups of users based on project or group membership, or email everyone using the GitLab instance. These emails are great for scheduled maintenance or upgrades. |
+| [Enforce ToS acceptance](../user/admin_area/settings/terms.md) | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Enforce your users accepting new terms of service by blocking GitLab traffic. |
+| [External Status Checks](../user/project/merge_requests/status_checks.md) | **{dotted-circle}** No | **{dotted-circle}** No | **{check-circle}** Yes | Interface with third-party systems you already use during development to ensure you remain compliant. |
+| [Generate reports on permission<br/>levels of users](../user/admin_area/index.md#user-permission-export) | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Generate a report listing all users' access permissions for groups and projects in the instance. |
+| [License compliance](../user/compliance/license_compliance/index.md) | **{dotted-circle}** No | **{dotted-circle}** No | **{check-circle}** Yes | Search dependencies for their licenses. This lets you determine if the licenses of your project's dependencies are compatible with your project's license. |
+| [Lock project membership to group](../user/group/access_and_permissions.md#prevent-members-from-being-added-to-projects-in-a-group) | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Group owners can prevent new members from being added to projects in a group. |
+| [LDAP group sync](auth/ldap/ldap_synchronization.md#group-sync) | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Automatically synchronize groups and manage SSH keys, permissions, and authentication, so you can focus on building your product, not configuring your tools. |
+| [LDAP group sync filters](auth/ldap/ldap_synchronization.md#group-sync) | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Gives more flexibility to synchronize with LDAP based on filters, meaning you can leverage LDAP attributes to map GitLab permissions. |
+| [Omnibus GitLab package supports<br/>log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-forwarding) | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Forward your logs to a central system. |
+| [Restrict SSH Keys](../security/ssh_keys_restrictions.md) | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Control the technology and key length of SSH keys used to access GitLab. |
diff --git a/doc/administration/configure.md b/doc/administration/configure.md
index bf618345a94..d68e81ebf69 100644
--- a/doc/administration/configure.md
+++ b/doc/administration/configure.md
@@ -7,44 +7,39 @@ type: reference
# Configure your GitLab installation **(FREE SELF)**
-Customize and configure your self-managed GitLab installation. Here are some quick links to get you started:
+Customize and configure your self-managed GitLab installation.
- [Authentication](auth/index.md)
+- [CI/CD](../administration/cicd.md)
- [Configuration](../user/admin_area/index.md)
-- [Repository storage](repository_storage_paths.md)
-- [Geo](geo/index.md)
-- [Packages](packages/index.md)
-
-The following tables are intended to guide you to choose the right combination of capabilities based on your requirements. It is common to want the most
-available, quickly recoverable, highly performant, and fully data resilient solution. However, there are tradeoffs.
-
-The tables lists features on the left and provides their capabilities to the right along with known trade-offs.
-
-## Gitaly Capabilities
-
-| | Availability | Recoverability | Data Resiliency | Performance | Risks/Trade-offs|
-|-|--------------|----------------|-----------------|-------------|-----------------|
-|Gitaly Cluster | Very high - tolerant of node failures | RTO for a single node of 10 s with no manual intervention | Data is stored on multiple nodes | Good - While writes may take slightly longer due to voting, read distribution improves read speeds | **Trade-off** - Slight decrease in write speed for redundant, strongly-consistent storage solution. **Risks** - [Does not support snapshot backups](gitaly/index.md#snapshot-backup-and-recovery-limitations), GitLab backup task can be slow for large data sets |
-|Gitaly Shards | Single storage location is a single point of failure | Would need to restore only shards which failed | Single point of failure | Good - can allocate repositories to shards to spread load | **Trade-off** - Need to manually configure repositories into different shards to balance loads / storage space **Risks** - Single point of failure relies on recovery process when single-node failure occurs |
-|Gitaly + NFS | Single storage location is a single point of failure | Single node failure requires restoration from backup | Single point of failure | Average - NFS is not ideally suited to large quantities of small reads / writes which can have a detrimental impact on performance | **Trade-off** - Familiar administration though NFS is not ideally suited to Git demands **Risks** - Many instances of NFS compatibility issues which provide very poor customer experiences |
-
-## Geo Capabilities
-
-If your availability needs to span multiple zones or multiple locations, read about [Geo](geo/index.md).
-
-| | Availability | Recoverability | Data Resiliency | Performance | Risks/Trade-offs|
-|-|--------------|----------------|-----------------|-------------|-----------------|
-|Geo| Depends on the architecture of the Geo site. It is possible to deploy secondaries in single and multiple node configurations. | Eventually consistent. Recovery point depends on replication lag, which depends on a number of factors such as network speeds. Geo supports failover from a primary to secondary site using manual commands that are scriptable. | Geo replicates 100% of planned data types and verifies 50%. See [limitations table](geo/replication/datatypes.md#limitations-on-replicationverification) for more detail. | Improves read/clone times for users of a secondary. | Geo is not intended to replace other backup/restore solutions. Because of replication lag and the possibility of replicating bad data from a primary, customers should also take regular backups of their primary site and test the restore process. |
-
-## Scenarios for failure modes and available mitigation paths
-
-The following table outlines failure modes and mitigation paths for the product offerings detailed in the tables above. Note - Gitaly Cluster install assumes an odd number replication factor of 3 or greater
-
-| Gitaly Mode | Loss of Single Gitaly Node | Application / Data Corruption | Regional Outage (Loss of Instance) | Notes |
-| ----------- | -------------------------- | ----------------------------- | ---------------------------------- | ----- |
-| Single Gitaly Node | Downtime - Must restore from backup | Downtime - Must restore from Backup | Downtime - Must wait for outage to end | |
-| Single Gitaly Node + Geo Secondary | Downtime - Must restore from backup, can perform a manual failover to secondary | Downtime - Must restore from Backup, errors could have propagated to secondary | Manual intervention - failover to Geo secondary | |
-| Sharded Gitaly Install | Partial Downtime - Only repositories on impacted node affected, must restore from backup | Partial Downtime - Only repositories on impacted node affected, must restore from backup | Downtime - Must wait for outage to end | |
-| Sharded Gitaly Install + Geo Secondary | Partial Downtime - Only repositories on impacted node affected, must restore from backup, could perform manual failover to secondary for impacted repositories | Partial Downtime - Only repositories on impacted node affected, must restore from backup, errors could have propagated to secondary | Manual intervention - failover to Geo secondary | |
-| Gitaly Cluster Install* | No Downtime - swaps repository primary to another node after 10 seconds | Not applicable; All writes are voted on by multiple Gitaly Cluster nodes | Downtime - Must wait for outage to end | Snapshot backups for Gitaly Cluster nodes not supported at this time |
-| Gitaly Cluster Install* + Geo Secondary | No Downtime - swaps repository primary to another node after 10 seconds | Not applicable; All writes are voted on by multiple Gitaly Cluster nodes | Manual intervention - failover to Geo secondary | Snapshot backups for Gitaly Cluster nodes not supported at this time |
+- [Consul](../administration/consul.md)
+- [Environment variables](../administration/environment_variables.md)
+- [File hooks](../administration/file_hooks.md)
+- [Git protocol v2](../administration/git_protocol.md)
+- [Incoming email](../administration/incoming_email.md)
+- [Instance limits](../administration/instance_limits.md)
+- [Instance Review](../administration/instance_review.md)
+- [PostgreSQL](../administration/postgresql/index.md)
+- [Load balancer](../administration/load_balancer.md)
+- [NFS](../administration/nfs.md)
+- [Postfix](../administration/reply_by_email_postfix_setup.md)
+- [Redis](../administration/redis/index.md)
+- [Sidekiq](../administration/sidekiq/index.md)
+- [S/MIME signing](../administration/smime_signing_email.md)
+- [Repository storage](../administration/repository_storage_paths.md)
+- [Object storage](../administration/object_storage.md)
+- [Merge request diffs storage](../administration/merge_request_diffs.md)
+- [Static objects external storage](../administration/static_objects_external_storage.md)
+- [Geo](../administration/geo/index.md)
+- [Disaster recovery (Geo)](../administration/geo/disaster_recovery/index.md)
+- [Agent server for Kubernetes](../administration/clusters/kas.md)
+- [Server hooks](../administration/server_hooks.md)
+- [Terraform state](../administration/terraform_state.md)
+- [Terraform limits](../user/admin_area/settings/terraform_limits.md)
+- [Packages](../administration/packages/index.md)
+- [Web terminals](../administration/integration/terminal.md)
+- [Wikis](../administration/wikis/index.md)
+- [Invalidate Markdown cache](../administration/invalidate_markdown_cache.md)
+- [Issue closing pattern](../administration/issue_closing_pattern.md)
+- [Snippets](../administration/snippets/index.md)
+- [Host the product documentation](../administration/docs_self_host.md)
diff --git a/doc/administration/dedicated/index.md b/doc/administration/dedicated/index.md
index 22b8cc13637..f2caa5fa368 100644
--- a/doc/administration/dedicated/index.md
+++ b/doc/administration/dedicated/index.md
@@ -26,11 +26,12 @@ To request the creation of a new GitLab Dedicated environment for your organizat
- Desired backup region: An AWS region where the primary backups of your data are replicated. This can be the same as the primary or secondary region or different.
- Desired instance subdomain: The main domain for GitLab Dedicated instances is `gitlab-dedicated.com`. You get to choose the subdomain name where your instance is accessible from (for example, `customer_name.gitlab-dedicated.com`).
- Initial storage: Initial storage size for your repositories in GB.
-- Availability Zone IDs for PrivateLink: If you plan to later add a PrivateLink connection (either [inbound](#inbound-private-link) or [outbound](#outbound-private-link)) to your environment, and you require the connections to be available in specific Availability Zones, you must provide up to two [Availability Zone IDs](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#az-ids) during onboarding. If not specified, GitLab will select two random Availability Zone IDs in which the connections will be available.
+- Availability Zone IDs for PrivateLink: If you plan to later add a PrivateLink connection (either [inbound](#inbound-private-link) or [outbound](#outbound-private-link)) to your environment, and you require the connections to be available in specific Availability Zones, you must provide up to two [Availability Zone IDs](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#az-ids) during onboarding. If not specified, GitLab selects two random Availability Zone IDs where the connections are available.
+- [KMS keys](https://docs.aws.amazon.com/kms/latest/developerguide/overview.html) for encrypted AWS services (if you are using that functionality).
### Maintenance window
-When onboarding, you must also specify your preference for the weekly four-hour timeslot that GitLab uses to perform maintenance and upgrade operations on the tenant instance.
+When onboarding, you must also specify your preference for the weekly four-hour time slot that GitLab uses to perform maintenance and upgrade operations on the tenant instance.
- APAC (outside working hours): Wednesday 1pm-5pm UTC
- EU (outside working hours): Tuesday 1am-5am UTC
@@ -45,13 +46,118 @@ To change or update the configuration for your GitLab Dedicated instance, open a
The turnaround time for processing configuration change requests is [documented in the GitLab handbook](https://about.gitlab.com/handbook/engineering/infrastructure/team/gitlab-dedicated/#handling-configuration-changes-for-tenant-environments).
+### Encrypted Data At Rest (BYOK)
+
+If you want your GitLab data to be encrypted at rest, the KMS keys used must be accessible by GitLab services. KMS keys can be used in two modes for this purpose:
+
+1. Per-service KMS keys (Backup, EBS, RDS, S3), or
+1. One KMS key for all services.
+
+If you use a key per service, all services must be encrypted at rest. Selective enablement of this feature is not supported.
+
+The keys provided have to reside in the same primary and secondary region specified during [onboarding](#onboarding).
+
+For instructions on how to create and manage KMS keys, visit [Managing keys](https://docs.aws.amazon.com/kms/latest/developerguide/getting-started.html) in the AWS KMS documentation.
+
+To create a KMS key using the AWS Console:
+
+1. In `Configure key`, select:
+ 1. Key type: **Symmetrical**
+ 1. Key usage: **Encrypt and decrypt**
+ 1. `Advanced options`:
+ 1. Key material origin: **KMS**
+ 1. Regionality: **Multi-Region key**
+1. Enter your values for key alias, description, and tags.
+1. Select Key administrators (optionally allow or deny key administrators to delete the key).
+1. For Key usage permissions, add the GitLab AWS account using the **Other AWS accounts** dialog.
+
+The last page asks you to confirm the KMS key policy. It should look similar to the following example, populated with your account IDs and usernames:
+
+```json
+{
+ "Version": "2012-10-17",
+ "Id": "byok-key-policy",
+ "Statement": [
+ {
+ "Sid": "Enable IAM User Permissions",
+ "Effect": "Allow",
+ "Principal": {
+ "AWS": "arn:aws:iam::<CUSTOMER-ACCOUNT-ID>:root"
+ },
+ "Action": "kms:*",
+ "Resource": "*"
+ },
+ {
+ "Sid": "Allow access for Key Administrators",
+ "Effect": "Allow",
+ "Principal": {
+ "AWS": [
+ "arn:aws:iam::<CUSTOMER-ACCOUNT-ID>:user/<CUSTOMER-USER>"
+ ]
+ },
+ "Action": [
+ "kms:Create*",
+ "kms:Describe*",
+ "kms:Enable*",
+ "kms:List*",
+ "kms:Put*",
+ "kms:Update*",
+ "kms:Revoke*",
+ "kms:Disable*",
+ "kms:Get*",
+ "kms:Delete*",
+ "kms:TagResource",
+ "kms:UntagResource",
+ "kms:ScheduleKeyDeletion",
+ "kms:CancelKeyDeletion",
+ "kms:ReplicateKey",
+ "kms:UpdatePrimaryRegion"
+ ],
+ "Resource": "*"
+ },
+ {
+ "Sid": "Allow use of the key",
+ "Effect": "Allow",
+ "Principal": {
+ "AWS": [
+ "arn:aws:iam::<GITLAB-ACCOUNT-ID>:root"
+ ]
+ },
+ "Action": [
+ "kms:Encrypt",
+ "kms:Decrypt",
+ "kms:ReEncrypt*",
+ "kms:GenerateDataKey*",
+ "kms:DescribeKey"
+ ],
+ "Resource": "*"
+ },
+ {
+ "Sid": "Allow attachment of persistent resources",
+ "Effect": "Allow",
+ "Principal": {
+ "AWS": [
+ "arn:aws:iam::<GITLAB-ACCOUNT-ID>:root"
+ ]
+ },
+ "Action": [
+ "kms:CreateGrant",
+ "kms:ListGrants",
+ "kms:RevokeGrant"
+ ],
+ "Resource": "*"
+ }
+ ]
+}
+```
+
### Inbound Private Link
[AWS Private Link](https://docs.aws.amazon.com/vpc/latest/privatelink/what-is-privatelink.html) allows users and applications in your VPC on AWS to securely connect to the GitLab Dedicated endpoint without network traffic going over the public internet.
To enable the Inbound Private Link:
-1. In the body of your [support ticket](#configuration-changes), include the IAM Principal for the AWS user or role in your own AWS Organization that will be establishing the VPC endpoint within your AWS account. GitLab Dedicated uses this IAM Principal for access-control. This IAM principal will be the only one able to set up an endpoint to the service.
+1. In the body of your [support ticket](#configuration-changes), include the IAM Principal for the AWS user or role in your own AWS Organization that's establishing the VPC endpoint in your AWS account. GitLab Dedicated uses this IAM Principal for access-control. This IAM principal is the only one able to set up an endpoint to the service.
1. After your IAM Principal has been allowlisted, GitLab [creates the Endpoint Service](https://docs.aws.amazon.com/vpc/latest/privatelink/create-endpoint-service.html) and communicates the `Service Endpoint Name` on the support ticket. The service name is generated by AWS upon creation of the service endpoint.
- GitLab handles the domain verification for the Private DNS name, so that DNS resolution of the tenant instance domain name in your VPC resolves to the PrivateLink endpoint.
- GitLab makes the Endpoint Service available in the Availability Zones you specified during the initial onboarding. If you did not specify any Availability Zones, GitLab randomly selects the Availability Zones IDs.
@@ -69,8 +175,8 @@ Outbound Private Links allow the GitLab Dedicated instance to securely communica
To enable an Outbound Private Link:
1. In your [support ticket](#configuration-changes), GitLab provides you with an IAM role ARN that connects to your endpoint service. You can then add this ARN to the allowlist on your side to restrict connections to your endpoint service.
-1. [Create the Endpoint service](https://docs.aws.amazon.com/vpc/latest/privatelink/create-endpoint-service.html) through which your internal service will be made available to GitLab Dedicated. Provide the associated `Service Endpoint Name` on the [support ticket](#configuration-changes).
-1. When creating the Endpoint service, you must provide GitLab with a [verified Private DNS Name](https://docs.aws.amazon.com/vpc/latest/privatelink/manage-dns-names.html#verify-domain-ownership) for your service. Optionally, if you would like GitLab Dedicated to reach your service via other aliases, you have the ability to specify a list of Private Hosted Zone (PHZ) entries. With this option, GitLab sets up a Private Hosted Zone with DNS names aliased to the verified Private DNS name. To enable this functionality, you must provide GitLab a list of PHZ entries on your support ticket. After the PHZ has been created in the tenant environment, DNS resolution of any of the provided records will correctly resolve to the PrivateLink endpoint.
+1. [Create the Endpoint service](https://docs.aws.amazon.com/vpc/latest/privatelink/create-endpoint-service.html) through which your internal service is available to GitLab Dedicated. Provide the associated `Service Endpoint Name` on the [support ticket](#configuration-changes).
+1. When creating the Endpoint service, you must provide GitLab with a [verified Private DNS Name](https://docs.aws.amazon.com/vpc/latest/privatelink/manage-dns-names.html#verify-domain-ownership) for your service. Optionally, if you would like GitLab Dedicated to reach your service via other aliases, you have the ability to specify a list of Private Hosted Zone (PHZ) entries. With this option, GitLab sets up a Private Hosted Zone with DNS names aliased to the verified Private DNS name. To enable this functionality, you must provide GitLab a list of PHZ entries on your support ticket. After the PHZ is created in the tenant environment, DNS resolution of any of the provided records correctly resolves to the PrivateLink endpoint.
1. GitLab then configures the tenant instance to create the necessary Endpoint Interfaces based on the service names you provided. Any outbound calls made from the tenant GitLab instance are directed through the PrivateLink into your VPC.
#### Custom certificates
@@ -97,14 +203,13 @@ Prerequisites:
To activate SAML for your GitLab Dedicated instance:
-1. Read the [GitLab documentation about SAML](../../integration/saml.md#https://docs.gitlab.com/ee/integration/saml.html#configure-saml-on-your-idp) to gather all data your identity provider requires for configuration. You can also find some providers and their requirements in the [group SAML documentation](../../user/group/saml_sso/index.md#providers).
-1. To make the necessary changes, include in your [support ticket](#configuration-changes) the desired [SAML configuration block](../../integration/saml.md#configure-saml-support-in-gitlab) that will be set on the GitLab application. At a minimum, GitLab needs the following information to enable SAML for your instance:
+1. To make the necessary changes, include the desired [SAML configuration block](../../integration/saml.md#configure-saml-support-in-gitlab) for your GitLab application in your [support ticket](#configuration-changes). At a minimum, GitLab needs the following information to enable SAML for your instance:
- Assertion consumer service URL
- Certificate fingerprint or certificate
- NameID format
- SSO login button description
-1. After GitLab deploys the SAML configuration to your instance, you will be notified on your support ticket.
+1. After GitLab deploys the SAML configuration to your instance, you are notified on your support ticket.
1. To verify the SAML configuration is successful:
- Check that the SSO login button description is displayed on your instance's login page.
- Go to the metadata URL of your instance (`https://INSTANCE-URL/users/auth/saml/metadata`). This page can be used to simplify much of the configuration of the identity provider, as well as manually validate the settings.
@@ -113,7 +218,7 @@ To activate SAML for your GitLab Dedicated instance:
If [SAML request signing](../../integration/saml.md#sign-saml-authentication-requests-optional) is desired, a certificate must be obtained. This certificate can be self-signed which has the advantage of not having to prove ownership of an arbitrary Common Name (CN) to a public Certificate Authority (CA)).
-To enable SAML request signing, indicate on your SAML [support ticket](#configuration-changes) that you would like request signing enabled. GitLab will then work with you on sending the Certificate Signing Request (CSR) for you to sign. Alternatively, the CSR can be signed with a public CA. After the certificate is signed, GitLab will add the certificate and its associated private key to the `security` section of the SAML configuration. Authentication requests from GitLab to your identity provider will then be signed.
+To enable SAML request signing, indicate on your SAML [support ticket](#configuration-changes) that you want request signing enabled. GitLab works with you on sending the Certificate Signing Request (CSR) for you to sign. Alternatively, the CSR can be signed with a public CA. After the certificate is signed, GitLab adds the certificate and its associated private key to the `security` section of the SAML configuration. Authentication requests from GitLab to your identity provider can then be signed.
#### SAML groups
@@ -136,5 +241,5 @@ GitLab [application logs](../../administration/logs/index.md) are delivered to a
To gain read only access to this bucket:
-1. Open a [support ticket](#configuration-changes) with the title "Customer Log Access". In the body of the ticket, include a list of IAM Principal ARNs (users or roles) that will be fetching the logs from S3.
-1. GitLab will then inform you of the name of the S3 bucket. Your nominated users/roles will then be able to list and get all objects in the S3 bucket.
+1. Open a [support ticket](#configuration-changes) with the title "Customer Log Access". In the body of the ticket, include a list of IAM Principal ARNs (users or roles) that are fetching the logs from S3.
+1. GitLab then informs you of the name of the S3 bucket. Your nominated users/roles can then able to list and get all objects in the S3 bucket.
diff --git a/doc/administration/docs_self_host.md b/doc/administration/docs_self_host.md
index 97eff35da91..d1ad36880dd 100644
--- a/doc/administration/docs_self_host.md
+++ b/doc/administration/docs_self_host.md
@@ -22,7 +22,7 @@ To host the GitLab product documentation, you can use:
- GitLab Pages
- Your own web server
-After you create a website by using one of these methods, you redirect the UI links
+After you create a website by using one of these methods, redirect the UI links
in the product to point to your website.
NOTE:
@@ -41,7 +41,7 @@ In the following example, we expose this on the host under the same port.
Make sure you either:
- Allow port `4000` in your firewall.
-- Use a different port. In following examples, replace the leftmost `4000` with the port different port.
+- Use a different port. In following examples, replace the leftmost `4000` with a different port number.
To run the GitLab product documentation website in a Docker container:
@@ -74,9 +74,9 @@ To run the GitLab product documentation website in a Docker container:
docker-compose up -d
```
-1. Visit `http://0.0.0.0:4000` to view the documentation website and verify
+1. Visit `http://0.0.0.0:4000` to view the documentation website and verify that
it works.
-1. [Redirect the help links to the new Docs site](#redirect-the-help-links-to-the-new-docs-site).
+1. [Redirect the help links to the new documentation site](#redirect-the-help-links-to-the-new-docs-site).
### Self-host the product documentation with GitLab Pages
@@ -84,7 +84,7 @@ You can use GitLab Pages to host the GitLab product documentation.
Prerequisite:
-- Ensure the Pages site URL does not use a subfolder. Because of how the docs
+- Ensure the Pages site URL does not use a subfolder. Because of the way the
site is pre-compiled, the CSS and JavaScript files are relative to the
main domain or subdomain. For example, URLs like `https://example.com/docs/`
are not supported.
@@ -114,13 +114,13 @@ To host the product documentation site with GitLab Pages:
| [Project website](../user/project/pages/getting_started_part_one.md#project-website-examples) | Not supported | Supported |
| [User or group website](../user/project/pages/getting_started_part_one.md#user-and-group-website-examples) | Supported | Supported |
-1. [Redirect the help links to the new Docs site](#redirect-the-help-links-to-the-new-docs-site).
+1. [Redirect the help links to the new documentation site](#redirect-the-help-links-to-the-new-docs-site).
### Self-host the product documentation on your own web server
Because the product documentation site is static, you can take the contents of
`/usr/share/nginx/html` from inside the container, and use your own web server to host
-the docs wherever you want.
+the documentation wherever you want.
The `html` directory should be served as is and it has the following structure:
@@ -135,7 +135,7 @@ In this example:
- `index.html` is a simple HTML file that redirects to the directory containing the documentation. In this
case, `14.5/`.
-To extract the HTML files of the Docs site:
+To extract the HTML files of the documentation site:
1. Create the container that holds the HTML files of the documentation website:
@@ -158,36 +158,40 @@ To extract the HTML files of the Docs site:
```
1. Point your web server to serve the contents of `/srv/gitlab/html/`.
-1. [Redirect the help links to the new Docs site](#redirect-the-help-links-to-the-new-docs-site).
+1. [Redirect the help links to the new documentation site](#redirect-the-help-links-to-the-new-docs-site).
## Redirect the `/help` links to the new Docs site
After your local product documentation site is running,
[redirect the help links](../user/admin_area/settings/help_page.md#redirect-help-pages)
in the GitLab application to your local site, by using the fully qualified domain
-name as the docs URL. For example, if you used the
+name as the documentation URL. For example, if you used the
[Docker method](#self-host-the-product-documentation-with-docker), enter `http://0.0.0.0:4000`.
You don't need to append the version. GitLab detects it and appends it to
documentation URL requests as needed. For example, if your GitLab version is
14.5:
-- The GitLab Docs URL becomes `http://0.0.0.0:4000/14.5/`.
+- The GitLab documentation URL becomes `http://0.0.0.0:4000/14.5/`.
- The link in GitLab displays as `<instance_url>/help/user/admin_area/settings/help_page#destination-requirements`.
- When you select the link, you are redirected to
`http://0.0.0.0:4000/14.5/ee/user/admin_area/settings/help_page/#destination-requirements`.
-To test the setting, select a **Learn more** link in the GitLab application.
+To test the setting, in GitLab, select a **Learn more** link. For example:
+
+1. On the top bar, in the upper-right corner, select your avatar.
+1. Select **Preferences**.
+1. In the **Syntax highlighting theme** section, select **Learn more**.
## Upgrade the product documentation to a later version
-Upgrading the Docs site to a later version requires downloading the newer Docker image tag.
+Upgrading the documentation site to a later version requires downloading the newer Docker image tag.
### Upgrade using Docker
To upgrade to a later version [using Docker](#self-host-the-product-documentation-with-docker):
-- If you use plain Docker:
+- If you use Docker:
1. Stop the running container:
@@ -207,7 +211,7 @@ To upgrade to a later version [using Docker](#self-host-the-product-documentatio
docker run --detach --name gitlab_docs -it --rm -p 4000:4000 registry.gitlab.com/gitlab-org/gitlab-docs:14.6
```
-- If you use Docker compose:
+- If you use Docker Compose:
1. Change the version in `docker-compose.yaml`, for example 14.6:
@@ -231,19 +235,19 @@ To upgrade to a later version [using Docker](#self-host-the-product-documentatio
To upgrade to a later version [using GitLab Pages](#self-host-the-product-documentation-with-gitlab-pages):
-1. Edit your existing `.gitlab-ci.yml` file, and replace the `image`'s version number:
+1. Edit your existing `.gitlab-ci.yml` file, and replace the `image` version number:
```yaml
image: registry.gitlab.com/gitlab-org/gitlab-docs:14.5
```
-1. Commit the changes, push, and GitLab Pages pulls the new Docs site version.
+1. Commit the changes, push, and GitLab Pages pulls the new documentation site version.
### Upgrade using your own web-server
To upgrade to a later version [using your own web-server](#self-host-the-product-documentation-on-your-own-web-server):
-1. Copy the HTML files of the Docs site:
+1. Copy the HTML files of the documentation site:
```shell
docker create -it --name gitlab_docs registry.gitlab.com/gitlab-org/gitlab-docs:14.6
@@ -261,8 +265,5 @@ To upgrade to a later version [using your own web-server](#self-host-the-product
If you self-host the product documentation:
-- The version dropdown list displays additional versions that don't exist. Selecting
- these versions displays a `404 Not Found` page.
-- The search displays results from `docs.gitlab.com` and not the local site.
- By default, the landing page redirects to the
respective version (for example, `/14.5/`). This causes the landing page <https://docs.gitlab.com> to not be displayed.
diff --git a/doc/administration/environment_variables.md b/doc/administration/environment_variables.md
index d2edc3085f1..a453d703a18 100644
--- a/doc/administration/environment_variables.md
+++ b/doc/administration/environment_variables.md
@@ -35,8 +35,8 @@ You can use the following environment variables to override certain values:
| `GITLAB_ROOT_PASSWORD` | string | Sets the password for the `root` user on installation. |
| `GITLAB_SHARED_RUNNERS_REGISTRATION_TOKEN` | string | Sets the initial registration token used for runners. |
| `RAILS_ENV` | string | The Rails environment; can be one of `production`, `development`, `staging`, or `test`. |
-| `UNSTRUCTURED_RAILS_LOG` | string | Enables the unstructured log in addition to JSON logs (defaults to `true`). |
| `GITLAB_RAILS_CACHE_DEFAULT_TTL_SECONDS` | integer | The default TTL used for entries stored in the Rails-cache. Default is `28800`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/95042) in 15.3. |
+| `GITLAB_CI_CONFIG_FETCH_TIMEOUT_SECONDS` | integer | Timeout for resolving remote includes in CI config in seconds. Must be between `0` and `60`. Default is `30`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/116383) in 15.11. |
## Adding more variables
diff --git a/doc/administration/external_pipeline_validation.md b/doc/administration/external_pipeline_validation.md
index 2dec3857f75..9d182a95fc9 100644
--- a/doc/administration/external_pipeline_validation.md
+++ b/doc/administration/external_pipeline_validation.md
@@ -160,5 +160,4 @@ required number of seconds.
}
```
-The `namespace` field is only available in [GitLab Premium](https://about.gitlab.com/pricing/)
-and higher.
+The `namespace` field is only available in [GitLab Premium and Ultimate](https://about.gitlab.com/pricing/).
diff --git a/doc/administration/feature_flags.md b/doc/administration/feature_flags.md
index f7237b167e5..26deff0ca84 100644
--- a/doc/administration/feature_flags.md
+++ b/doc/administration/feature_flags.md
@@ -41,11 +41,15 @@ GitLab, the feature flag status may change.
## Risks when enabling features still in development
+Before enabling a disabled feature flag in a production GitLab environment, it is crucial to understand the potential risks involved.
+
+WARNING:
+Data corruption, stability degradation, performance degradation, and security issues may occur if you enable a feature that's disabled by default.
+
Features that are disabled by default may change or be removed without notice in a future version of GitLab.
-Data corruption, stability degradation, performance degradation, or security issues might occur if
-you enable a feature that's disabled by default. Problems caused by using a default
-disabled feature aren't covered by GitLab Support.
+Features behind default-disabled feature flags are not recommended for use in a production environment
+and problems caused by using a default disabled features aren't covered by GitLab Support.
Security issues found in features that are disabled by default are patched in regular releases
and do not follow our regular [maintenance policy](../policy/maintenance.md#security-releases)
diff --git a/doc/administration/file_hooks.md b/doc/administration/file_hooks.md
index b74ee22d584..dfddf38475c 100644
--- a/doc/administration/file_hooks.md
+++ b/doc/administration/file_hooks.md
@@ -1,6 +1,6 @@
---
stage: Manage
-group: Integrations
+group: Import and Integrate
info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments"
type: reference
---
diff --git a/doc/administration/geo/disaster_recovery/background_verification.md b/doc/administration/geo/disaster_recovery/background_verification.md
index 99ac95cd536..ea4523c66e6 100644
--- a/doc/administration/geo/disaster_recovery/background_verification.md
+++ b/doc/administration/geo/disaster_recovery/background_verification.md
@@ -6,12 +6,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# Automatic background verification **(PREMIUM SELF)**
-NOTE:
-Automatic background verification of repositories and wikis was added in
-GitLab EE 10.6 but is enabled by default only on GitLab EE 11.1. You can
-disable or enable this feature manually by following
-[these instructions](#disabling-or-enabling-the-automatic-background-verification).
-
Automatic background verification ensures that the transferred data matches a
calculated checksum. If the checksum of the data on the **primary** site matches checksum of the
data on the **secondary** site, the data transferred successfully. Following a planned failover,
@@ -179,19 +173,5 @@ If the **primary** and **secondary** sites have a checksum verification mismatch
## Current limitations
-Automatic background verification doesn't cover attachments, LFS objects,
-job artifacts, and user uploads in file storage. You can keep track of the
-progress to include them in [Geo: Verify all replicated data](https://gitlab.com/groups/gitlab-org/-/epics/1430).
-
-For now, you can verify their integrity
-manually by following [these instructions](../../raketasks/check.md) on both
-sites, and comparing the output between them.
-
-In GitLab EE 12.1, Geo calculates checksums for attachments, LFS objects, and
-archived traces on secondary sites after the transfer, compares it with the
-stored checksums, and rejects transfers if mismatched. Geo
-currently does not support an automatic way to verify these data if they have
-been synced before GitLab EE 12.1.
-
-Data in object storage is **not verified**, as the object store is responsible
-for ensuring the integrity of the data.
+For more information on what replication and verification methods are supported,
+see the [supported Geo data types](../replication/datatypes.md).
diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md
index e9eb154d398..a52bdc759a2 100644
--- a/doc/administration/geo/disaster_recovery/planned_failover.md
+++ b/doc/administration/geo/disaster_recovery/planned_failover.md
@@ -38,9 +38,9 @@ performed ahead of the maintenance window; subsequent `rsync`s (including a
final transfer inside the maintenance window) then transfers only the
*changes* between the **primary** site and the **secondary** sites.
-Repository-centric strategies for using `rsync` effectively can be found in the
+Git repository-centric strategies for using `rsync` effectively can be found in the
[moving repositories](../../operations/moving_repositories.md) documentation; these strategies can
-be adapted for use with any other file-based data, such as [GitLab Pages](../../pages/index.md#change-storage-path).
+be adapted for use with any other file-based data.
### Container registry
diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md
index cef0101dd40..feb5e030b80 100644
--- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md
+++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md
@@ -6,7 +6,7 @@ type: howto
---
WARNING:
-This runbook is in [**Alpha**](../../../../policy/alpha-beta-support.md#alpha-features). For complete, production-ready documentation, see the
+This runbook is an [Experiment](../../../../policy/alpha-beta-support.md#experiment). For complete, production-ready documentation, see the
[disaster recovery documentation](../index.md).
# Disaster Recovery (Geo) promotion runbooks **(PREMIUM SELF)**
diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md
index 085765ec51e..19150027cca 100644
--- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md
+++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md
@@ -6,7 +6,7 @@ type: howto
---
WARNING:
-This runbook is in [**Alpha**](../../../../policy/alpha-beta-support.md#alpha-features). For complete, production-ready documentation, see the
+This runbook is an [Experiment](../../../../policy/alpha-beta-support.md#experiment). For complete, production-ready documentation, see the
[disaster recovery documentation](../index.md).
# Disaster Recovery (Geo) promotion runbooks **(PREMIUM SELF)**
diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md
index 3f98f1e12fe..31de7f5c62f 100644
--- a/doc/administration/geo/index.md
+++ b/doc/administration/geo/index.md
@@ -9,8 +9,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w
Geo is the solution for widely distributed development teams and for providing
a warm-standby as part of a disaster recovery strategy.
-## Overview
-
WARNING:
Geo undergoes significant changes from release to release. Upgrades are
supported and [documented](#upgrading-geo), but you should ensure that you're
@@ -123,7 +121,8 @@ The following are required to run Geo:
- Note,[PostgreSQL 12 is deprecated](../../update/deprecations.md#postgresql-12-deprecated) and is removed in GitLab 16.0.
- Git 2.9 or later
- Git-lfs 2.4.2 or later on the user side when using LFS
-- All sites must run [the same GitLab and PostgreSQL versions](setup/database.md#postgresql-replication).
+- All sites must run the same GitLab version.
+- All sites must run [the same PostgreSQL versions](setup/database.md#postgresql-replication).
- If using different operating system versions between Geo sites,
[check OS locale data compatibility](replication/troubleshooting.md#check-os-locale-data-compatibility)
across Geo sites to avoid silent corruption of database indexes.
diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md
index 912de360e43..5fa6df393b9 100644
--- a/doc/administration/geo/replication/configuration.md
+++ b/doc/administration/geo/replication/configuration.md
@@ -205,15 +205,16 @@ keys must be manually replicated to the **secondary** site.
1. On the top bar, select **Main menu > Admin**.
1. On the left sidebar, select **Geo > Sites**.
1. Select **Add site**.
- ![Add secondary site](img/adding_a_secondary_v13_3.png)
- 1. Fill in **Name** with the `gitlab_rails['geo_node_name']` in
- `/etc/gitlab/gitlab.rb`. These values must always match *exactly*, character
+ ![Add secondary site](img/adding_a_secondary_v15_8.png)
+ 1. In **Name**, enter the value for `gitlab_rails['geo_node_name']` in
+ `/etc/gitlab/gitlab.rb`. These values must always match **exactly**, character
for character.
- 1. Fill in **URL** with the `external_url` in `/etc/gitlab/gitlab.rb`. These
+ 1. In **External URL**, enter the value for `external_url` in `/etc/gitlab/gitlab.rb`. These
values must always match, but it doesn't matter if one ends with a `/` and
the other doesn't.
- 1. (Optional) Choose which groups or storage shards should be replicated by the
- **secondary** site. Leave blank to replicate all. Read more in
+ 1. Optional. In **Internal URL (optional)**, enter an internal URL for the primary site.
+ 1. Optional. Select which groups or storage shards should be replicated by the
+ **secondary** site. Leave blank to replicate all. For more information, see
[selective synchronization](#selective-synchronization).
1. Select **Save changes** to add the **secondary** site.
1. SSH into **each Rails, and Sidekiq node on your secondary** site and restart the services:
@@ -257,12 +258,12 @@ You can safely skip this step if:
#### Custom or self-signed certificate for inbound connections
-If your GitLab Geo **primary** site uses a custom or [self-signed certificate to secure inbound HTTPS connections](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates), this certificate can either be single-domain certificate or multi-domain.
+If your GitLab Geo **primary** site uses a custom or [self-signed certificate to secure inbound HTTPS connections](https://docs.gitlab.com/omnibus/settings/ssl/index.html#install-custom-public-certificates), this can be either a single-domain or multi-domain certificate.
Install the correct certificate based on your certificate type:
- **Multi-domain certificate** that includes both primary and secondary site domains: Install the certificate at `/etc/gitlab/ssl` on all **Rails, Sidekiq, and Gitaly** nodes in the **secondary** site.
-- **Single-domain certificate** where the certificates are specific to each Geo site domain: Generate a valid certificate for your **secondary** site's domain and install it at `/etc/gitlab/ssl` per [these instructions](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) on all **Rails, Sidekiq, and Gitaly** nodes in the **secondary** site.
+- **Single-domain certificate** where the certificates are specific to each Geo site domain: Generate a valid certificate for your **secondary** site's domain and install it at `/etc/gitlab/ssl` following [these instructions](https://docs.gitlab.com/omnibus/settings/ssl/index.html#install-custom-public-certificates) on all **Rails, Sidekiq, and Gitaly** nodes in the **secondary** site.
#### Connecting to external services that use custom certificates
@@ -303,7 +304,7 @@ If your **primary** site is using a [custom or self-signed certificate for inbou
sudo gitlab-ctl reconfigure
```
-### Step 5. Enable Git access over HTTP/HTTPS
+### Step 5. Enable Git access over HTTP/HTTPS and SSH
Geo synchronizes repositories over HTTP/HTTPS, and therefore requires this clone
method to be enabled. This is enabled by default, but if converting an existing site to Geo it should be checked:
@@ -313,7 +314,10 @@ On the **primary** site:
1. On the top bar, select **Main menu > Admin**.
1. On the left sidebar, select **Settings > General**.
1. Expand **Visibility and access controls**.
-1. Ensure "Enabled Git access protocols" is set to either "Both SSH and HTTP(S)" or "Only HTTP(S)".
+1. If using Git over SSH, then:
+ 1. Ensure "Enabled Git access protocols" is set to "Both SSH and HTTP(S)".
+ 1. Follow [Fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md) on both primary and secondary sites.
+1. If not using Git over SSH, then set "Enabled Git access protocols" to "Only HTTP(S)".
### Step 6. Verify proper functioning of the **secondary** site
@@ -366,6 +370,9 @@ former is ideal for replicating data belonging to a subset of users, while the
latter is more suited to progressively rolling out Geo to a large GitLab
instance.
+NOTE:
+Geo's synchronization logic is outlined in the [documentation](../index.md). Both the solution and the documentation is subject to change from time to time. You must independently determine your legal obligations in regard to privacy and cybersecurity laws, and applicable trade control law on an ongoing basis.
+
Selective synchronization:
1. Does not restrict permissions from **secondary** sites.
diff --git a/doc/administration/geo/replication/container_registry.md b/doc/administration/geo/replication/container_registry.md
index 88ca8781dc3..66c67f29c1c 100644
--- a/doc/administration/geo/replication/container_registry.md
+++ b/doc/administration/geo/replication/container_registry.md
@@ -7,7 +7,7 @@ type: howto
# Container Registry for a secondary site **(PREMIUM SELF)**
-You can set up a Container Registry on your **secondary** Geo site that mirrors the one on the **primary** Geo site.
+You can set up a Container Registry on your **secondary** Geo site that mirrors the one on the **primary** Geo site.
NOTE:
The Container Registry replication is used only for disaster recovery purposes. We do not recommend
@@ -76,7 +76,9 @@ the **primary** site before following the next steps.
We need to make Container Registry send notification events to the
**primary** site.
-1. SSH into your GitLab **primary** server and login as root:
+For each application and Sidekiq node on the **primary** site:
+
+1. SSH into the node and login as the `root` user:
```shell
sudo -i
@@ -105,17 +107,15 @@ We need to make Container Registry send notification events to the
that starts with a letter. You can generate one with `< /dev/urandom tr -dc _A-Z-a-z-0-9 | head -c 32 | sed "s/^[0-9]*//"; echo`
NOTE:
- If you use an external Registry (not the one integrated with GitLab), you must add
- these settings to its configuration yourself. In this case, you also have to specify
- notification secret in `registry.notification_secret` section of
+ If you use an external Registry (not the one integrated with GitLab), you also have to specify
+ the notification secret (`registry['notification_secret']`) in the
`/etc/gitlab/gitlab.rb` file.
NOTE:
- If you use GitLab HA, you also have to specify
- the notification secret in `registry.notification_secret` section of
+ If you use GitLab HA, you also have to specify the notification secret (`registry['notification_secret']`) in
`/etc/gitlab/gitlab.rb` file for every web node.
-1. Reconfigure the **primary** node for the change to take effect:
+1. Reconfigure each node:
```shell
gitlab-ctl reconfigure
diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md
index 27e1b990b2b..d6ce5ef5067 100644
--- a/doc/administration/geo/replication/datatypes.md
+++ b/doc/administration/geo/replication/datatypes.md
@@ -43,12 +43,12 @@ verification methods:
| Blobs | CI job artifacts _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ |
| Blobs | Archived CI build traces _(file system)_ | Geo with API | _Not implemented_ |
| Blobs | Archived CI build traces _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ |
-| Blobs | Container registry _(file system)_ | Geo with API/Docker API | _Not implemented_ |
-| Blobs | Container registry _(object storage)_ | Geo with API/Managed/Docker API (*2*) | _Not implemented_ |
+| Blobs | Container registry _(file system)_ | Geo with API/Docker API | _SHA256 checksum_ |
+| Blobs | Container registry _(object storage)_ | Geo with API/Managed/Docker API (*2*) | _SHA256 checksum_ |
| Blobs | Package registry _(file system)_ | Geo with API | SHA256 checksum |
| Blobs | Package registry _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ |
-| Blobs | Infrastructure registry _(file system)_ | Geo with API | SHA256 checksum |
-| Blobs | Infrastructure registry _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ |
+| Blobs | Terraform Module Registry _(file system)_ | Geo with API | SHA256 checksum |
+| Blobs | Terraform Module Registry _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ |
| Blobs | Versioned Terraform State _(file system)_ | Geo with API | SHA256 checksum |
| Blobs | Versioned Terraform State _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ |
| Blobs | External Merge Request Diffs _(file system)_ | Geo with API | SHA256 checksum |
@@ -139,7 +139,7 @@ We use PostgreSQL's own replication functionality to replicate data from the **p
We use Redis both as a cache store and to hold persistent data for our background jobs system. Because both
use-cases have data that are exclusive to the same Geo site, we don't replicate it between sites.
-Elasticsearch is an optional database that for advanced searching capabilities. It can improve search
+Elasticsearch is an optional database for advanced search. It can improve search
in both source-code level, and user generated content in issues, merge requests, and discussions.
Elasticsearch is not supported in Geo.
@@ -192,17 +192,17 @@ successfully, you must replicate their data using some other means.
|:--------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------|:---------------------------------------------------------------------------|:--------------------------------------------------------------------|:----------------------------------------------------------------|:------|
|[Application data in PostgreSQL](../../postgresql/index.md) | **Yes** (10.2) | **Yes** (10.2) | N/A | N/A | |
|[Project repository](../../../user/project/repository/index.md) | **Yes** (10.2) | **Yes** (10.7) | N/A | N/A | |
-|[Project wiki repository](../../../user/project/wiki/index.md) | **Yes** (10.2) | **Yes** (10.7) | N/A | N/A | |
+|[Project wiki repository](../../../user/project/wiki/index.md) | **Yes** (10.2)<sup>2</sup> | **Yes** (10.7)<sup>2</sup> | N/A | N/A | Migrated to [self-service framework](../../../development/geo/framework.md) in 15.11. See GitLab issue [#367925](https://gitlab.com/gitlab-org/gitlab/-/issues/367925) for more details.<br /><br />Behind feature flag geo_project_wiki_repository_replication, enabled by default in (15.11). |
|[Group wiki repository](../../../user/project/wiki/group.md) | [**Yes** (13.10)](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | No | N/A | N/A | Behind feature flag `geo_group_wiki_repository_replication`, enabled by default. |
|[Uploads](../../uploads.md) | **Yes** (10.2) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Replication is behind the feature flag `geo_upload_replication`, enabled by default. Verification was behind the feature flag `geo_upload_verification`, removed in 14.8. |
|[LFS objects](../../lfs/index.md) | **Yes** (10.2) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696).<br /><br />Replication is behind the feature flag `geo_lfs_object_replication`, enabled by default. Verification was behind the feature flag `geo_lfs_object_verification`, removed in 14.7. |
|[Personal snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | N/A | N/A | |
|[Project snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | N/A | N/A | |
-|[CI job artifacts](../../../ci/pipelines/job_artifacts.md) | **Yes** (10.4) | **Yes** (14.10) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Verification is behind the feature flag `geo_job_artifact_replication`, enabled by default in 14.10. |
+|[CI job artifacts](../../../ci/jobs/job_artifacts.md) | **Yes** (10.4) | **Yes** (14.10) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Verification is behind the feature flag `geo_job_artifact_replication`, enabled by default in 14.10. |
|[CI Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/pipeline_artifact.rb) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Persists additional artifacts after a pipeline completes. |
|[CI Secure Files](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/secure_file.rb) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [No](object_storage.md#verification-of-files-in-object-storage) | Verification is behind the feature flag `geo_ci_secure_file_replication`, enabled by default in 15.3. |
-|[Container Registry](../../packages/container_registry.md) | **Yes** (12.3)* | No | No | No | See [instructions](container_registry.md) to set up the Container Registry replication. |
-|[Infrastructure Registry](../../../user/packages/infrastructure_registry/index.md) | **Yes** (14.0) | **Yes** (14.0) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Behind feature flag `geo_package_file_replication`, enabled by default. |
+|[Container Registry](../../packages/container_registry.md) | **Yes** (12.3)<sup>1</sup> | **Yes** (15.10) | **Yes** (12.3)<sup>1</sup> | **Yes** (15.10) | See [instructions](container_registry.md) to set up the Container Registry replication. |
+|[Terraform Module Registry](../../../user/packages/terraform_module_registry/index.md) | **Yes** (14.0) | **Yes** (14.0) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Behind feature flag `geo_package_file_replication`, enabled by default. |
|[Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | N/A | N/A | Designs also require replication of LFS objects and Uploads. |
|[Package Registry](../../../user/packages/package_registry/index.md) | **Yes** (13.2) | **Yes** (13.10) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Behind feature flag `geo_package_file_replication`, enabled by default. |
|[Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | **Yes** (13.12) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Replication is behind the feature flag `geo_terraform_state_version_replication`, enabled by default. Verification was behind the feature flag `geo_terraform_state_version_verification`, which was removed in 14.0. |
@@ -217,4 +217,6 @@ successfully, you must replicate their data using some other means.
|[Dependency Proxy Images](../../../user/packages/dependency_proxy/index.md) | [**Yes** (15.7)](https://gitlab.com/groups/gitlab-org/-/epics/8833) | [**Yes** (15.7)](https://gitlab.com/groups/gitlab-org/-/epics/8833) | [**Yes** (15.7)](https://gitlab.com/groups/gitlab-org/-/epics/8833) | [No](object_storage.md#verification-of-files-in-object-storage) | |
|[Vulnerability Export](../../../user/application_security/vulnerability_report/index.md#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | No | No | Not planned because they are ephemeral and sensitive information. They can be regenerated on demand. |
-\* Migrated to [self-service framework](../../../development/geo/framework.md) in 15.5. See GitLab issue [#337436](https://gitlab.com/gitlab-org/gitlab/-/issues/337436) for more details.
+<sup>1</sup> Migrated to [self-service framework](../../../development/geo/framework.md) in 15.5. See GitLab issue [#337436](https://gitlab.com/gitlab-org/gitlab/-/issues/337436) for more details.
+
+<sup>2</sup> Migrated to [self-service framework](../../../development/geo/framework.md) in 15.11. Behind feature flag `geo_project_wiki_repository_replication`, enabled by default. See GitLab issue [#367925](https://gitlab.com/gitlab-org/gitlab/-/issues/367925) for more details.
diff --git a/doc/administration/geo/replication/geo_validation_tests.md b/doc/administration/geo/replication/geo_validation_tests.md
index a12dd8d9d68..cad3a396bfc 100644
--- a/doc/administration/geo/replication/geo_validation_tests.md
+++ b/doc/administration/geo/replication/geo_validation_tests.md
@@ -127,8 +127,7 @@ The following are PostgreSQL upgrade validation tests we performed.
- Description: With PostgreSQL 12 available as an opt-in version in GitLab 13.3, we tested upgrading
existing Geo installations from PostgreSQL 11 to 12. We also re-tested fresh installations of GitLab
with Geo after fixes were made to support PostgreSQL 12. These tests were done using a
- [nightly build](https://packages.gitlab.com/gitlab/nightly-builds/packages/ubuntu/bionic/gitlab-ee_13.3.6+rnightly.169516.d5209202-0_amd64.deb)
- of GitLab 13.4.
+ nightly build of GitLab 13.4.
- Outcome: Tests were successful for Geo deployments with a single database node on the primary and secondary.
We encountered known issues with repmgr and Patroni managed PostgreSQL clusters on the Geo primary. Using
PostgreSQL 12 with a database cluster on the primary is not recommended until the issues are resolved.
diff --git a/doc/administration/geo/replication/img/adding_a_secondary_v13_3.png b/doc/administration/geo/replication/img/adding_a_secondary_v13_3.png
deleted file mode 100644
index e43ace99666..00000000000
--- a/doc/administration/geo/replication/img/adding_a_secondary_v13_3.png
+++ /dev/null
Binary files differ
diff --git a/doc/administration/geo/replication/img/adding_a_secondary_v15_8.png b/doc/administration/geo/replication/img/adding_a_secondary_v15_8.png
new file mode 100644
index 00000000000..d7c99e6551e
--- /dev/null
+++ b/doc/administration/geo/replication/img/adding_a_secondary_v15_8.png
Binary files differ
diff --git a/doc/administration/geo/replication/single_sign_on.md b/doc/administration/geo/replication/single_sign_on.md
index fc2f23552db..55e77d5657c 100644
--- a/doc/administration/geo/replication/single_sign_on.md
+++ b/doc/administration/geo/replication/single_sign_on.md
@@ -112,6 +112,15 @@ After configuring your SAML IdP to allow the secondary site's SAML callback URL,
If you have configured SAML on the primary site correctly, then it should work on the secondary site without additional configuration.
+## OpenID Connect
+
+If you use an [OpenID Connect (OIDC)](../../auth/oidc.md) OmniAuth provider,
+in most cases, it should work without an issue:
+
+- **OIDC with Unified URL**: If you have configured OIDC on the primary site correctly, then it should work on the secondary site without additional configuration.
+- **OIDC with separate URL with proxying disabled**: If you have configured OIDC on the primary site correctly, then it should work on the secondary site without additional configuration.
+- **OIDC with separate URL with proxying enabled**: Geo with separate URL with proxying enabled does not support [OpenID Connect](../../auth/oidc.md). For more information, see [issue 396745](https://gitlab.com/gitlab-org/gitlab/-/issues/396745).
+
## LDAP
If you use LDAP on your **primary** site, you should also set up secondary LDAP servers on each **secondary** site. Otherwise, users cannot perform Git operations over HTTP(s) on the **secondary** site using HTTP basic authentication. However, users can still use Git with SSH and personal access tokens.
diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md
index 59a67fecfcd..a8736b3ed1d 100644
--- a/doc/administration/geo/replication/troubleshooting.md
+++ b/doc/administration/geo/replication/troubleshooting.md
@@ -42,7 +42,7 @@ to help identify if something is wrong:
![Geo health check](img/geo_site_health_v14_0.png)
-A site shows as "Unhealthy" if the site's status is more than 10 minutes old. In that case, try running the following in the [Rails console](../../operations/rails_console.md) on the affected site:
+A site shows as "Unhealthy" if the site's status is more than 10 minutes old. In that case, try running the following in the [Rails console](../../operations/rails_console.md) on the affected secondary site:
```ruby
Geo::MetricsUpdateWorker.new.perform
@@ -52,7 +52,7 @@ If it raises an error, then the error is probably also preventing the jobs from
If it successfully updates the status, then something may be wrong with Sidekiq. Is it running? Do the logs show errors? This job is supposed to be enqueued every minute. It takes an exclusive lease in Redis to ensure that only one of these jobs can run at a time. The primary site updates its status directly in the PostgreSQL database. Secondary sites send an HTTP Post request to the primary site with their status data.
-A site also shows as "Unhealthy" if certain health checks fail. You can reveal the failure by running the following in the [Rails console](../../operations/rails_console.md) on the affected site:
+A site also shows as "Unhealthy" if certain health checks fail. You can reveal the failure by running the following in the [Rails console](../../operations/rails_console.md) on the affected secondary site:
```ruby
Gitlab::Geo::HealthCheck.new.perform_checks
@@ -718,6 +718,8 @@ To solve this:
### Very large repositories never successfully synchronize on the **secondary** site
+#### GitLab 10.1 and earlier
+
GitLab places a timeout on all repository clones, including project imports
and Geo synchronization operations. If a fresh `git clone` of a repository
on the **primary** takes more than the default three hours, you may be affected by this.
@@ -740,6 +742,10 @@ add the following line to `/etc/gitlab/gitlab.rb`:
This increases the timeout to four hours (14400 seconds). Choose a time
long enough to accommodate a full clone of your largest repositories.
+#### GitLab 10.2 and later
+
+Geo [replicates Git repositories over HTTPS](../index.md#how-it-works). GitLab does not place a timeout on these requests. If a Git repository is failing to replicate, with a consistent job execution time, then you should check for timeouts applied by external components such as load balancers.
+
### New LFS objects are never replicated
If new LFS objects are never replicated to secondary Geo sites, check the version of
@@ -959,7 +965,7 @@ This behavior affects only the following data types through GitLab 14.6:
| Package Registry | 13.10 |
| CI Pipeline Artifacts | 13.11 |
| Terraform State Versions | 13.12 |
-| Infrastructure Registry | 14.0 |
+| Infrastructure Registry (renamed to Terraform Module Registry in GitLab 15.11) | 14.0 |
| External MR diffs | 14.6 |
| LFS Objects | 14.6 |
| Pages Deployments | 14.6 |
@@ -1373,6 +1379,23 @@ The bug causes all wildcard domains (`.example.com`) to be ignored except for th
You can have only one wildcard domain in the `no_proxy` list.
+### Secondary site shows "Unhealthy" in UI after changing the value of `external_url` for the primary site
+
+If you have updated the value of `external_url` in `/etc/gitlab/gitlab.rb` for the primary site or changed the protocol from `http` to `https`, you may see that secondary sites are shown as `Unhealthy`. You may also find the following error in `geo.log`:
+
+```plaintext
+"class": "Geo::NodeStatusRequestService",
+...
+"message": "Failed to Net::HTTP::Post to primary url: http://primary-site.gitlab.tld/api/v4/geo/status",
+ "error": "Failed to open TCP connection to <PRIMARY_IP_ADDRESS>:80 (Connection refused - connect(2) for \"<PRIMARY_ID_ADDRESS>\" port 80)"
+```
+
+In this case, make sure to update the changed URL on all your sites:
+
+1. On the top bar, select **Main menu > Admin**.
+1. On the left sidebar, select **Admin > Geo > Sites**.
+1. Change the URL and save the change.
+
## Fixing non-PostgreSQL replication failures
If you notice replication failures in `Admin > Geo > Sites` or the [Sync status Rake task](#sync-status-rake-task), you can try to resolve the failures with the following general steps:
diff --git a/doc/administration/geo/replication/upgrading_the_geo_sites.md b/doc/administration/geo/replication/upgrading_the_geo_sites.md
index 55395a55857..644232a2c9e 100644
--- a/doc/administration/geo/replication/upgrading_the_geo_sites.md
+++ b/doc/administration/geo/replication/upgrading_the_geo_sites.md
@@ -33,6 +33,7 @@ and all **secondary** sites:
1. SSH into each node of the **primary** site.
1. [Upgrade GitLab on the **primary** site](../../../update/package/index.md#upgrade-using-the-official-repositories).
1. Perform testing on the **primary** site, particularly if you paused replication in step 1 to protect DR. [There are some suggestions for post-upgrade testing](../../../update/plan_your_upgrade.md#pre-upgrade-and-post-upgrade-checks) in the upgrade documentation.
+1. Ensure that the secrets in the `/etc/gitlab/gitlab-secrets.json` file of both the primary site and the secondary site are the same. The file must be the same on all of a site’s nodes.
1. SSH into each node of **secondary** sites.
1. [Upgrade GitLab on each **secondary** site](../../../update/package/index.md#upgrade-using-the-official-repositories).
1. If you paused replication in step 1, [resume replication on each **secondary**](../index.md#pausing-and-resuming-replication).
diff --git a/doc/administration/geo/secondary_proxy/index.md b/doc/administration/geo/secondary_proxy/index.md
index addf894f0a5..35ab1d8252c 100644
--- a/doc/administration/geo/secondary_proxy/index.md
+++ b/doc/administration/geo/secondary_proxy/index.md
@@ -165,6 +165,7 @@ It does not cover all data types.
| LFS objects (using Git) | **{check-circle}** Yes |
| Pages | **{dotted-circle}** No <sup>2</sup> |
| Advanced search (using the web UI) | **{dotted-circle}** No |
+| Container registry | **{dotted-circle}** No |
1. Git reads are served from the local secondary while pushes get proxied to the primary.
Selective sync or cases where repositories don't exist locally on the Geo secondary throw a "not found" error.
diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md
index 99f7b32be59..cf0c40dbbc5 100644
--- a/doc/administration/geo/setup/database.md
+++ b/doc/administration/geo/setup/database.md
@@ -21,7 +21,7 @@ NOTE:
The stages of the setup process must be completed in the documented order.
If not, [complete all prior stages](../setup/index.md#using-omnibus-gitlab) before proceeding.
-Ensure the **secondary** site is running the same version of GitLab Enterprise Edition as the **primary** site. Confirm you have added the [premium or higher licenses](https://about.gitlab.com/pricing/) to your **primary** site.
+Ensure the **secondary** site is running the same version of GitLab Enterprise Edition as the **primary** site. Confirm you have added a license for a [Premium or Ultimate subscription](https://about.gitlab.com/pricing/) to your **primary** site.
Be sure to read and review all of these steps before you execute them in your
testing or production environments.
@@ -29,10 +29,10 @@ testing or production environments.
## Single instance database replication
A single instance database replication is easier to set up and still provides the same Geo capabilities
-as a clusterized alternative. It's useful for setups running on a single machine
-or trying to evaluate Geo for a future clusterized installation.
+as a clustered alternative. It's useful for setups running on a single machine
+or trying to evaluate Geo for a future clustered installation.
-A single instance can be expanded to a clusterized version using Patroni, which is recommended for a
+A single instance can be expanded to a clustered version using Patroni, which is recommended for a
highly available architecture.
Follow the instructions below on how to set up PostgreSQL replication as a single instance database.
@@ -151,6 +151,13 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
ALTER USER gitlab_replicator WITH REPLICATION ENCRYPTED PASSWORD '<replication_password>';
```
+1. Edit `/etc/gitlab/gitlab.rb` and set the role to `geo_primary_role` (for more information, see [Geo roles](https://docs.gitlab.com/omnibus/roles/#gitlab-geo-roles)):
+
+ ```ruby
+ ## Geo Primary role
+ roles(['geo_primary_role'])
+ ```
+
1. Configure PostgreSQL to listen on network interfaces:
For security reasons, PostgreSQL does not listen on any network interfaces
@@ -211,17 +218,6 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
```ruby
##
- ## Geo Primary role
- ## - Configures Postgres settings for replication
- ## - Prevents automatic upgrade of Postgres since it requires downtime of
- ## streaming replication to Geo secondary sites
- ## - Enables standard single-node GitLab services like NGINX, Puma, Redis,
- ## or Sidekiq. If you are segregating services, then you will need to
- ## explicitly disable unwanted services.
- ##
- roles(['geo_primary_role'])
-
- ##
## Primary address
## - replace '<primary_node_ip>' with the public or VPC address of your Geo primary node
##
@@ -239,11 +235,13 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
# postgresql['max_replication_slots'] = 1 # Set this to be the number of Geo secondary nodes if you have more than one
# postgresql['max_wal_senders'] = 10
# postgresql['wal_keep_segments'] = 10
+ ```
- ##
- ## Disable automatic database migrations temporarily
- ## (until PostgreSQL is restarted and listening on the private address).
- ##
+1. Disable automatic database migrations temporarily until PostgreSQL is restarted and listening on the private address.
+ Edit `/etc/gitlab/gitlab.rb` and change the configuration to false:
+
+ ```ruby
+ ## Disable automatic database migrations
gitlab_rails['auto_migrate'] = false
```
@@ -402,6 +400,16 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
Ensure that the contents of `~gitlab-psql/data/server.crt` on the **primary** site
match the contents of `~gitlab-psql/.postgresql/root.crt` on the **secondary** site.
+1. Edit `/etc/gitlab/gitlab.rb` and set the role to `geo_secondary_role` (for more information, see [Geo roles](https://docs.gitlab.com/omnibus/roles/#gitlab-geo-roles)):
+
+ ```ruby
+ ##
+ ## Geo Secondary role
+ ## - configure dependent flags automatically to enable Geo
+ ##
+ roles(['geo_secondary_role'])
+ ```
+
1. Configure PostgreSQL:
This step is similar to how you configured the **primary** instance.
@@ -412,12 +420,6 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
```ruby
##
- ## Geo Secondary role
- ## - configure dependent flags automatically to enable Geo
- ##
- roles(['geo_secondary_role'])
-
- ##
## Secondary address
## - replace '<secondary_site_ip>' with the public or VPC address of your Geo secondary site
##
@@ -863,6 +865,7 @@ For each node running a Patroni instance on the secondary site:
consul['configuration'] = {
retry_join: %w[CONSUL_SECONDARY1_IP CONSUL_SECONDARY2_IP CONSUL_SECONDARY3_IP]
}
+ consul['services'] = %w(postgresql)
postgresql['md5_auth_cidr_addresses'] = [
'PATRONI_SECONDARY1_IP/32', 'PATRONI_SECONDARY2_IP/32', 'PATRONI_SECONDARY3_IP/32', 'PATRONI_SECONDARY_PGBOUNCER/32',
@@ -892,7 +895,7 @@ For each node running a Patroni instance on the secondary site:
postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH'
postgresql['listen_address'] = '0.0.0.0' # You can use a public or VPC address here instead
- gitlab_rails['dbpassword'] = 'POSTGRESQL_PASSWORD'
+ gitlab_rails['db_password'] = 'POSTGRESQL_PASSWORD'
gitlab_rails['enable'] = true
gitlab_rails['auto_migrate'] = false
```
@@ -937,6 +940,8 @@ Omnibus automatically configures a tracking database when `roles(['geo_secondary
If you want to run this database in a highly available configuration, don't use the `geo_secondary_role` above.
Instead, follow the instructions below.
+If you want to run the Geo tracking database on a single node, see [Configure the Geo tracking database on the Geo secondary site](../replication/multiple_servers.md#step-3-configure-the-geo-tracking-database-on-the-geo-secondary-site).
+
A production-ready and secure setup for the tracking PostgreSQL DB requires at least three Consul nodes: two
Patroni nodes, and one PgBouncer node on the secondary site.
diff --git a/doc/administration/geo/setup/external_database.md b/doc/administration/geo/setup/external_database.md
index 0fefc11f078..65ea2e6e5af 100644
--- a/doc/administration/geo/setup/external_database.md
+++ b/doc/administration/geo/setup/external_database.md
@@ -211,20 +211,20 @@ the tracking database on port 5432.
1. Set up PostgreSQL according to the
[database requirements document](../../../install/requirements.md#database).
-1. Set up a `gitlab_geo` user with a password of your choice, create the `gitlabhq_geo_production` database, and make the user an owner of the database. You can see an example of this setup in the [installation from source documentation](../../../install/installation.md#6-database).
+1. Set up a `gitlab_geo` user with a password of your choice, create the `gitlabhq_geo_production` database, and make the user an owner of the database. You can see an example of this setup in the [installation from source documentation](../../../install/installation.md#7-database).
1. If you are **not** using a cloud-managed PostgreSQL database, ensure that your secondary
site can communicate with your tracking database by manually changing the
`pg_hba.conf` that is associated with your tracking database.
Remember to restart PostgreSQL afterwards for the changes to take effect:
- ```plaintext
- ##
- ## Geo Tracking Database Role
- ## - pg_hba.conf
- ##
- host all all <trusted tracking IP>/32 md5
- host all all <trusted secondary IP>/32 md5
- ```
+ ```plaintext
+ ##
+ ## Geo Tracking Database Role
+ ## - pg_hba.conf
+ ##
+ host all all <trusted tracking IP>/32 md5
+ host all all <trusted secondary IP>/32 md5
+ ```
1. SSH into a GitLab **secondary** server and login as root:
diff --git a/doc/administration/geo/setup/index.md b/doc/administration/geo/setup/index.md
index c794b8ef219..da7e55509e7 100644
--- a/doc/administration/geo/setup/index.md
+++ b/doc/administration/geo/setup/index.md
@@ -7,33 +7,33 @@ type: howto
# Setting up Geo **(PREMIUM SELF)**
-These instructions assume you have a working instance of GitLab. They guide you through:
+## Prerequisites
-1. Making your existing instance the **primary** site.
-1. Adding **secondary** sites.
+- Two (or more) independently working GitLab sites:
+ - One GitLab site serves as the Geo **primary** site. Use the [GitLab reference architectures documentation](../../reference_architectures/index.md) to set this up. You can use different reference architecture sizes for each Geo site. If you already have a working GitLab instance that is in-use, it can be used as a **primary** site.
+ - The second GitLab site serves as the Geo **secondary** site. Use the [GitLab reference architectures documentation](../../reference_architectures/index.md) to set this up. It's a good idea to sign in and test it. However, be aware that **all of the data on the secondary are lost** as part of the process of replicating from the **primary** site.
-You must use a [GitLab Premium](https://about.gitlab.com/pricing/) license or higher,
-but you only need one license for all the sites.
+ NOTE:
+ Geo supports multiple secondaries. You can follow the same steps and make any changes accordingly.
-WARNING:
-The steps below should be followed in the order they appear. **Make sure the GitLab version is the same on all sites. Do not create an account or sign in to the new secondary.**
+- Ensure the **primary** site has a [GitLab Premium or Ultimate](https://about.gitlab.com/pricing/) subscription to unlock Geo. You only need one license for all the sites.
+- Confirm the [requirements for running Geo](../index.md#requirements-for-running-geo) are met by all sites. For example, sites must use the same GitLab version, and sites must be able to communicate with each other over certain ports.
## Using Omnibus GitLab
If you installed GitLab using the Omnibus packages (highly recommended):
-1. Confirm the [requirements for running Geo](../index.md#requirements-for-running-geo) are met.
-1. [Install GitLab Enterprise Edition](https://about.gitlab.com/install/) on the nodes that serve as the **secondary** site. **Do not create an account or sign in** to the new **secondary** site. The **GitLab version must match** across primary and secondary sites.
-1. [Add the GitLab License](../../../user/admin_area/license.md) on the **primary** site to unlock Geo. The license must be for [GitLab Premium](https://about.gitlab.com/pricing/) or higher.
-1. [Confirm network connectivity](../index.md#firewall-rules) between the **primary** and **secondary** site.
1. [Set up the database replication](database.md) (`primary (read-write) <-> secondary (read-only)` topology).
-1. [Configure fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md). This step is required and needs to be done on **both** the **primary** and **secondary** sites.
1. [Configure GitLab](../replication/configuration.md) to set the **primary** and **secondary** sites.
1. Optional: [Configure Object storage](../../object_storage.md)
1. Optional: [Configure a secondary LDAP server](../../auth/ldap/index.md) for the **secondary** sites. See [notes on LDAP](../index.md#ldap).
1. Optional: [Configure Geo secondary proxying](../secondary_proxy/index.md) to use a single, unified URL for all Geo sites. This step is recommended to accelerate most read requests while transparently proxying writes to the primary Geo site.
1. Follow the [Using a Geo Site](../replication/usage.md) guide.
+## Using GitLab Charts
+
+[Configure the GitLab chart with GitLab Geo](https://docs.gitlab.com/charts/advanced/geo/).
+
## Post-installation documentation
After installing GitLab on the **secondary** sites and performing the initial configuration, see the [following documentation for post-installation information](../index.md#post-installation-documentation).
diff --git a/doc/administration/get_started.md b/doc/administration/get_started.md
index d9191440a24..0b5de38a78b 100644
--- a/doc/administration/get_started.md
+++ b/doc/administration/get_started.md
@@ -1,7 +1,7 @@
---
-info: For assistance with this CSM Onboarding page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-other-projects-and-subjects.
+info: For assistance with this tutorial, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-other-projects-and-subjects.
stage: none
-group: unassigned
+group: Tutorials
---
# Get started administering GitLab **(FREE)**
@@ -100,7 +100,6 @@ Unlike other monitoring solutions (for example, Zabbix or New Relic), Prometheus
- Prometheus and its exporters are on by default. However, you need to [configure the service](../administration/monitoring/prometheus/index.md#configuring-prometheus).
- Learn more about [GitLab architecture](../development/architecture.md).
- Find out why [application performance metrics](https://about.gitlab.com/blog/2020/05/07/working-with-performance-metrics/) matter.
-- Create a [self-monitoring project](../administration/monitoring/gitlab_self_monitoring_project/index.md) to track the health of your instance.
- Integrate Grafana to [build visual dashboards](https://youtu.be/f4R7s0An1qE) based on performance metrics.
### Components of monitoring
@@ -151,7 +150,8 @@ Backups of GitLab databases and file systems are taken every 24 hours, and are k
- You can use the project export option in:
- [The UI](../user/project/settings/import_export.md#export-a-project-and-its-data).
- [The API](../api/project_import_export.md#schedule-an-export).
-- [Group export](../user/group/settings/import_export.md) does *not* export the projects in it, but does export:
+- [Group export by uploading a file export](../user/group/import/index.md#migrate-groups-by-uploading-an-export-file-deprecated)
+ does **not** export the projects in it, but does export:
- Epics
- Milestones
- Boards
diff --git a/doc/administration/git_protocol.md b/doc/administration/git_protocol.md
index 349a92de51e..b996b9efae9 100644
--- a/doc/administration/git_protocol.md
+++ b/doc/administration/git_protocol.md
@@ -17,7 +17,7 @@ More details about the new features and improvements are available in
the [Google Open Source Blog](https://opensource.googleblog.com/2018/05/introducing-git-protocol-version-2.html)
and the [protocol documentation](https://github.com/git/git/blob/master/Documentation/gitprotocol-v2.txt).
-## Requirements
+## Prerequisites
From the client side, `git` `v2.18.0` or newer must be installed.
@@ -111,4 +111,4 @@ URL to use SSH.
### Observe Git protocol version of connections
For information on observing the Git protocol versions are being used in a production environment,
-see the [relevant documentation](gitaly/monitoring.md#useful-queries).
+see the [relevant documentation](gitaly/monitoring.md#queries).
diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md
index 143f7dca7d3..f3eb2de3f1d 100644
--- a/doc/administration/gitaly/configure_gitaly.md
+++ b/doc/administration/gitaly/configure_gitaly.md
@@ -6,25 +6,26 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# Configure Gitaly **(FREE SELF)**
-The Gitaly service itself is configured by using a [TOML configuration file](reference.md).
+Configure Gitaly in one of two ways:
-To change Gitaly settings:
+::Tabs
-**For Omnibus GitLab**
+:::TabTitle Linux package (Omnibus)
1. Edit `/etc/gitlab/gitlab.rb` and add or change the
[Gitaly settings](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/1dd07197c7e5ae23626aad5a4a070a800b670380/files/gitlab-config-template/gitlab.rb.template#L1622-1676).
1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
-**For installations from source**
+:::TabTitle Self-compiled (source)
1. Edit `/home/git/gitaly/config.toml` and add or change the [Gitaly settings](https://gitlab.com/gitlab-org/gitaly/blob/master/config.toml.example).
1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).
+::EndTabs
+
The following configuration options are also available:
- Enabling [TLS support](#enable-tls-support).
-- Configuring the [number of `gitaly-ruby` workers](#configure-number-of-gitaly-ruby-workers).
- Limiting [RPC concurrency](#limit-rpc-concurrency).
## About the Gitaly token
@@ -138,15 +139,25 @@ Gitaly and GitLab use two shared secrets for authentication:
- _Gitaly token_: used to authenticate gRPC requests to Gitaly
- _GitLab Shell token_: used for authentication callbacks from GitLab Shell to the GitLab internal API
-**For Omnibus GitLab**
+Configure authentication in one of two ways:
+
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
To configure the _Gitaly token_, edit `/etc/gitlab/gitlab.rb`:
```ruby
- gitaly['auth_token'] = 'abc123secret'
+ gitaly['configuration'] = {
+ # ...
+ auth: {
+ # ...
+ token: 'abc123secret',
+ },
+ }
```
-There are two ways to configure the _GitLab Shell token_.
+Configure the _GitLab Shell token_ in one of two ways.
Method 1 (recommended):
@@ -161,7 +172,7 @@ Edit `/etc/gitlab/gitlab.rb`:
gitlab_shell['secret_token'] = 'shellsecret'
```
-**For installations from source**
+:::TabTitle Self-compiled (source)
1. Copy `/home/git/gitlab/.gitlab_shell_secret` from the Gitaly client to the same path on the
Gitaly servers (and any other Gitaly clients).
@@ -183,19 +194,26 @@ Edit `/etc/gitlab/gitlab.rb`:
1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).
-#### Configure Gitaly server
-
-**For Omnibus GitLab**
+::EndTabs
-1. Edit `/etc/gitlab/gitlab.rb`:
+#### Configure Gitaly server
<!--
Updates to example must be made at:
+
- https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab
- https://gitlab.com/gitlab-org/gitlab/blob/master/doc/administration/gitaly/index.md#gitaly-server-configuration
-- all reference architecture pages
+- All reference architecture pages
-->
+Configure Gitaly server in one of two ways:
+
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
+
+1. Edit `/etc/gitlab/gitlab.rb`:
+
```ruby
# Avoid running unnecessary services on the Gitaly server
postgresql['enable'] = false
@@ -230,14 +248,21 @@ Updates to example must be made at:
# Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from Gitaly client to Gitaly server.
gitlab_rails['internal_api_url'] = 'https://gitlab.example.com'
- # Make Gitaly accept connections on all network interfaces. You must use
- # firewalls to restrict access to this address/port.
- # Comment out following line if you only want to support TLS connections
- gitaly['listen_addr'] = "0.0.0.0:8075"
-
- # Authentication token to ensure only authorized servers can communicate with
- # Gitaly server
- gitaly['auth_token'] = 'AUTH_TOKEN'
+ gitaly['configuration'] = {
+ # ...
+ #
+ # Make Gitaly accept connections on all network interfaces. You must use
+ # firewalls to restrict access to this address/port.
+ # Comment out following line if you only want to support TLS connections
+ listen_addr: '0.0.0.0:8075',
+ auth: {
+ # ...
+ #
+ # Authentication token to ensure only authorized servers can communicate with
+ # Gitaly server
+ token: 'AUTH_TOKEN',
+ },
+ }
```
1. Append the following to `/etc/gitlab/gitlab.rb` for each respective Gitaly server:
@@ -247,24 +272,33 @@ Updates to example must be made at:
On `gitaly1.internal`:
```ruby
- git_data_dirs({
- 'default' => {
- 'path' => '/var/opt/gitlab/git-data'
- },
- 'storage1' => {
- 'path' => '/mnt/gitlab/git-data'
- },
- })
+ gitaly['configuration'] = {
+ # ...
+ storage: [
+ {
+ name: 'default',
+ path: '/var/opt/gitlab/git-data',
+ },
+ {
+ name: 'storage1',
+ path: '/mnt/gitlab/git-data',
+ },
+ ],
+ }
```
On `gitaly2.internal`:
```ruby
- git_data_dirs({
- 'storage2' => {
- 'path' => '/srv/gitlab/git-data'
- },
- })
+ gitaly['configuration'] = {
+ # ...
+ storage: [
+ {
+ name: 'storage2',
+ path: '/srv/gitlab/git-data',
+ },
+ ],
+ }
```
1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
@@ -272,7 +306,7 @@ Updates to example must be made at:
- For GitLab 15.3 and later, run `sudo /opt/gitlab/embedded/bin/gitaly check /var/opt/gitlab/gitaly/config.toml`.
- For GitLab 15.2 and earlier, run `sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml`.
-**For installations from source**
+:::TabTitle Self-compiled (source)
1. Edit `/home/git/gitaly/config.toml`:
@@ -323,6 +357,8 @@ Updates to example must be made at:
- For GitLab 15.3 and later, run `sudo /opt/gitlab/embedded/bin/gitaly check /var/opt/gitlab/gitaly/config.toml`.
- For GitLab 15.2 and earlier, run `sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml`.
+::EndTabs
+
WARNING:
If directly copying repository data from a GitLab server to Gitaly, ensure that the metadata file,
default path `/var/opt/gitlab/git-data/repositories/.gitaly-metadata`, is not included in the transfer.
@@ -359,7 +395,11 @@ You can't define Gitaly servers with some as a local Gitaly server
server (with `gitaly_address`) unless you use
[mixed configuration](#mixed-configuration).
-**For Omnibus GitLab**
+Configure Gitaly clients in one of two ways:
+
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
1. Edit `/etc/gitlab/gitlab.rb`:
@@ -393,7 +433,7 @@ server (with `gitaly_address`) unless you use
sudo gitlab-ctl tail gitaly
```
-**For installations from source**
+:::TabTitle Self-compiled (source)
1. Edit `/home/git/gitlab/config/gitlab.yml`:
@@ -429,6 +469,8 @@ server (with `gitaly_address`) unless you use
tail -f /home/git/gitlab/log/gitaly.log
```
+::EndTabs
+
When you tail the Gitaly logs on your Gitaly server, you should see requests coming in. One sure way
to trigger a Gitaly request is to clone a repository from GitLab over HTTP or HTTPS.
@@ -461,17 +503,28 @@ example:
git_data_dirs({
'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' },
# Address of the GitLab server that also has Gitaly running on it
- 'storage1' => { 'gitaly_address' => 'tcp://gitlab.internal:8075', 'path' => '/mnt/gitlab/git-data' },
+ 'storage1' => { 'gitaly_address' => 'tcp://gitlab.internal:8075' },
'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' },
})
-# Make Gitaly accept connections on all network interfaces
-gitaly['listen_addr'] = "0.0.0.0:8075"
-
-# Or for TLS
-gitaly['tls_listen_addr'] = "0.0.0.0:9999"
-gitaly['certificate_path'] = "/etc/gitlab/ssl/cert.pem"
-gitaly['key_path'] = "/etc/gitlab/ssl/key.pem"
+gitaly['configuration'] = {
+ # ...
+ #
+ # Make Gitaly accept connections on all network interfaces
+ listen_addr: '0.0.0.0:8075',
+ # Or for TLS
+ tls_listen_addr: '0.0.0.0:9999',
+ tls: {
+ certificate_path: '/etc/gitlab/ssl/cert.pem',
+ key_path: '/etc/gitlab/ssl/key.pem',
+ },
+ storage: [
+ {
+ name: 'storage1',
+ path: '/mnt/gitlab/git-data',
+ },
+ ],
+}
```
`path` can be included only for storage shards on the local Gitaly server.
@@ -499,9 +552,11 @@ Disabling Gitaly on the GitLab instance makes sense only when you run GitLab in
Gitaly runs on a separate machine from the GitLab instance. Disabling Gitaly on all machines in the cluster is not
a valid configuration (some machines much act as Gitaly servers).
-To disable Gitaly on a GitLab server:
+Disable Gitaly on a GitLab server in one of two ways:
+
+::Tabs
-**For Omnibus GitLab**
+:::TabTitle Linux package (Omnibus)
1. Edit `/etc/gitlab/gitlab.rb`:
@@ -511,7 +566,7 @@ To disable Gitaly on a GitLab server:
1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
-**For installations from source**
+:::TabTitle Self-compiled (source)
1. Edit `/etc/default/gitlab`:
@@ -521,10 +576,9 @@ To disable Gitaly on a GitLab server:
1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).
-## Enable TLS support
+::EndTabs
-> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/22602) in GitLab 11.8.
-> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/3160) in GitLab 13.6, outgoing TLS connections to GitLab provide client certificates if configured.
+## Enable TLS support
Gitaly supports TLS encryption. To communicate with a Gitaly instance that listens for secure
connections, use the `tls://` URL scheme in the `gitaly_address` of the corresponding
@@ -543,6 +597,8 @@ Additionally, the certificate (or its certificate authority) must be installed o
- Gitaly servers.
- Gitaly clients that communicate with it.
+If you use a load balancer, it must be able to negotiate HTTP/2 using the ALPN TLS extension.
+
### Certificate requirements
- The certificate must specify the address you use to access the Gitaly server. You must add the hostname or IP address as a Subject Alternative Name to the certificate.
@@ -553,9 +609,11 @@ Additionally, the certificate (or its certificate authority) must be installed o
### Configure Gitaly with TLS
-To configure Gitaly with TLS:
+Configure Gitaly with TLS in one of two ways:
+
+::Tabs
-**For Omnibus GitLab**
+:::TabTitle Linux package (Omnibus)
1. Create certificates for Gitaly servers.
1. On the Gitaly clients, copy the certificates (or their certificate authority) into
@@ -600,21 +658,26 @@ To configure Gitaly with TLS:
<!-- Updates to following example must also be made at https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab -->
```ruby
- gitaly['tls_listen_addr'] = "0.0.0.0:9999"
- gitaly['certificate_path'] = "/etc/gitlab/ssl/cert.pem"
- gitaly['key_path'] = "/etc/gitlab/ssl/key.pem"
+ gitaly['configuration'] = {
+ # ...
+ tls_listen_addr: '0.0.0.0:9999',
+ tls: {
+ certificate_path: '/etc/gitlab/ssl/cert.pem',
+ key_path: '/etc/gitlab/ssl/key.pem',
+ },
+ }
```
1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
1. Verify Gitaly traffic is being served over TLS by
[observing the types of Gitaly connections](#observe-type-of-gitaly-connections).
1. Optional. Improve security by:
- 1. Disabling non-TLS connections by commenting out or deleting `gitaly['listen_addr']` in
+ 1. Disabling non-TLS connections by commenting out or deleting `gitaly['configuration'][:listen_addr]` in
`/etc/gitlab/gitlab.rb`.
1. Saving the file.
1. [Reconfiguring GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
-**For installations from source**
+:::TabTitle Self-compiled (source)
1. Create certificates for Gitaly servers.
1. On the Gitaly clients, copy the certificates into the system trusted certificates:
@@ -690,66 +753,12 @@ To configure Gitaly with TLS:
1. Saving the file.
1. [Restarting GitLab](../restart_gitlab.md#installations-from-source).
+::EndTabs
+
### Observe type of Gitaly connections
For information on observing the type of Gitaly connections being served, see the
-[relevant documentation](monitoring.md#useful-queries).
-
-## `gitaly-ruby`
-
-Gitaly was developed to replace the Ruby application code in GitLab.
-
-To save time and avoid the risk of rewriting existing application logic, we chose to copy some
-application code from GitLab into Gitaly.
-
-To be able to run that code, `gitaly-ruby` was created, which is a "sidecar" process for the main
-Gitaly Go process. Some examples of things that are implemented in `gitaly-ruby` are:
-
-- RPCs that deal with wikis.
-- RPCs that create commits on behalf of a user, such as merge commits.
-
-We recommend:
-
-- At least 300 MB memory per worker.
-- No more than one worker per core.
-
-NOTE:
-`gitaly-ruby` is planned to be eventually removed. To track progress, see the
-[Remove the Gitaly-Ruby sidecar](https://gitlab.com/groups/gitlab-org/-/epics/2862) epic.
-
-### Configure number of `gitaly-ruby` workers
-
-`gitaly-ruby` has much less capacity than Gitaly implemented in Go. If your Gitaly server has to handle lots of
-requests, the default setting of having just one active `gitaly-ruby` sidecar might not be enough.
-
-If you see `ResourceExhausted` errors from Gitaly, it's very likely that you have not enough
-`gitaly-ruby` capacity.
-
-You can increase the number of `gitaly-ruby` processes on your Gitaly server with the following
-settings:
-
-**For Omnibus GitLab**
-
-1. Edit `/etc/gitlab/gitlab.rb`:
-
- ```ruby
- # Default is 2 workers. The minimum is 2; 1 worker is always reserved as
- # a passive stand-by.
- gitaly['ruby_num_workers'] = 4
- ```
-
-1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
-
-**For installations from source**
-
-1. Edit `/home/git/gitaly/config.toml`:
-
- ```toml
- [gitaly-ruby]
- num_workers = 4
- ```
-
-1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).
+[relevant documentation](monitoring.md#queries).
## Limit RPC concurrency
@@ -775,21 +784,23 @@ example:
```ruby
# in /etc/gitlab/gitlab.rb
-
-gitaly['concurrency'] = [
- {
- 'rpc' => "/gitaly.SmartHTTPService/PostUploadPackWithSidechannel",
- 'max_per_repo' => 20,
- 'max_queue_time' => "1s",
- 'max_queue_size' => 10
- },
- {
- 'rpc' => "/gitaly.SSHService/SSHUploadPackWithSidechannel",
- 'max_per_repo' => 20
- 'max_queue_time' => "1s",
- 'max_queue_size' => 10
- }
-]
+gitaly['configuration'] = {
+ # ...
+ concurrency: [
+ {
+ rpc: '/gitaly.SmartHTTPService/PostUploadPackWithSidechannel',
+ max_per_repo: 20,
+ max_queue_time: '1s',
+ max_queue_size: 10,
+ },
+ {
+ rpc: '/gitaly.SSHService/SSHUploadPackWithSidechannel',
+ max_per_repo: 20,
+ max_queue_time: '1s',
+ max_queue_size: 10,
+ },
+ ],
+}
```
- `rpc` is the name of the RPC to set a concurrency limit for per repository.
@@ -834,7 +845,7 @@ performance.
You can use control groups (cgroups) in Linux to impose limits on how much memory and CPU can be consumed by Gitaly processes.
See the [`cgroups` Linux man page](https://man7.org/linux/man-pages/man7/cgroups.7.html) for more information.
-cgroups can be useful for protecting the system against unexpected resource exhaustion because of over consumption of memory and CPU.
+cgroups can help protect the system against unexpected resource exhaustion because of over consumption of memory and CPU.
Some Git operations can consume notable resources up to the point of exhaustion in situations such as:
@@ -861,43 +872,60 @@ When these limits are reached, performance may be reduced and users may be disco
### Configure repository cgroups (new method)
-> This method of configuring repository cgroups was introduced in GitLab 15.1.
+> - This method of configuring repository cgroups was introduced in GitLab 15.1.
+> - `cpu_quota_us`[introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5422) in GitLab 15.10.
To configure repository cgroups in Gitaly using the new method, use the following settings for the new configuration method
-to `gitaly['cgroups']` in `/etc/gitlab/gitlab.rb`:
+to `gitaly['configuration'][:cgroups]` in `/etc/gitlab/gitlab.rb`:
-- `cgroups_mountpoint` is where the parent cgroup directory is mounted. Defaults to `/sys/fs/cgroup`.
-- `cgroups_hierarchy_root` is the parent cgroup under which Gitaly creates groups, and
+- `mountpoint` is where the parent cgroup directory is mounted. Defaults to `/sys/fs/cgroup`.
+- `hierarchy_root` is the parent cgroup under which Gitaly creates groups, and
is expected to be owned by the user and group Gitaly runs as. Omnibus GitLab
creates the set of directories `mountpoint/<cpu|memory>/hierarchy_root`
when Gitaly starts.
-- `cgroups_memory_bytes` is the total memory limit that is imposed collectively on all
+- `memory_bytes` is the total memory limit that is imposed collectively on all
Git processes that Gitaly spawns. 0 implies no limit.
-- `cgroups_cpu_shares` is the CPU limit that is imposed collectively on all Git
+- `cpu_shares` is the CPU limit that is imposed collectively on all Git
processes that Gitaly spawns. 0 implies no limit. The maximum is 1024 shares,
which represents 100% of CPU.
-- `cgroups_repositories_count` is the number of cgroups in the cgroups pool. Each time a new Git
+- `cpu_quota_us` is the [`cfs_quota_us`](https://docs.kernel.org/scheduler/sched-bwc.html#management)
+ to throttle the cgroups' processes if they exceed this quota value. We set
+ `cfs_period_us` to `100ms` so 1 core is `100000`. 0 implies no limit.
+- `repositories.count` is the number of cgroups in the cgroups pool. Each time a new Git
command is spawned, Gitaly assigns it to one of these cgroups based
on the repository the command is for. A circular hashing algorithm assigns
Git commands to these cgroups, so a Git command for a repository is
always assigned to the same cgroup.
-- `cgroups_repositories_memory_bytes` is the total memory limit imposed on all Git processes contained in a repository cgroup.
- 0 implies no limit. This value cannot exceed that of the top level `cgroups_memory_bytes`.
-- `cgroups_repositories_cpu_shares` is the CPU limit that is imposed on all Git processes contained in a repository cgroup.
+- `repositories.memory_bytes` is the total memory limit imposed on all Git processes contained in a repository cgroup.
+ 0 implies no limit. This value cannot exceed that of the top level `memory_bytes`.
+- `repositories.cpu_shares` is the CPU limit that is imposed on all Git processes contained in a repository cgroup.
0 implies no limit. The maximum is 1024 shares, which represents 100% of CPU.
- This value cannot exceed that of the top level`cgroups_cpu_shares`.
+ This value cannot exceed that of the top level`cpu_shares`.
+- `repositories.cpu_quota_us` is the [`cfs_quota_us`](https://docs.kernel.org/scheduler/sched-bwc.html#management)
+ that is imposed on all Git processes contained in a repository cgroup. A Git
+ process can't use more then the given quota. We set
+ `cfs_period_us` to `100ms` so 1 core is `100000`. 0 implies no limit.
For example:
```ruby
# in /etc/gitlab/gitlab.rb
-gitaly['cgroups_mountpoint'] = "/sys/fs/cgroup"
-gitaly['cgroups_hierarchy_root'] => "gitaly"
-gitaly['cgroups_memory_bytes'] = 64424509440, # 60gb
-gitaly['cgroups_cpu_shares'] = 1024
-gitaly['cgroups_repositories_count'] => 1000,
-gitaly['cgroups_repositories_memory_bytes'] => 32212254720 # 20gb
-gitaly['cgroups_repositories_cpu_shares'] => 512
+gitaly['configuration'] = {
+ # ...
+ cgroups: {
+ mountpoint: '/sys/fs/cgroup',
+ hierarchy_root: 'gitaly',
+ memory_bytes: 64424509440, # 60gb
+ cpu_shares: 1024,
+ cpu_quota_us: 400000 # 4 cores
+ repositories: {
+ count: 1000,
+ memory_bytes: 32212254720, # 20gb
+ cpu_shares: 512,
+ cpu_quota_us: 200000 # 2 cores
+ },
+ },
+}
```
### Configure repository cgroups (legacy method)
@@ -953,28 +981,38 @@ This strategy has two main benefits:
to 3 child cgroups can concurrently burst up to their max. In general, all
1000 cgroups would use much less than the 20 GB.
-## Background Repository Optimization
+## Background repository optimization
Empty directories and unneeded configuration settings may accumulate in a repository and
slow down Git operations. Gitaly can schedule a daily background task with a maximum duration
to clean up these items and improve performance.
WARNING:
-This is an experimental feature and may place significant load on the host while running.
+Background repository optimization is an experimental feature and may place significant load on the host while running.
Make sure to schedule this during off-peak hours and keep the duration short (for example, 30-60 minutes).
-**For Omnibus GitLab**
+Configure background repository optimization in one of two ways:
+
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
Edit `/etc/gitlab/gitlab.rb` and add:
```ruby
-gitaly['daily_maintenance_start_hour'] = 4
-gitaly['daily_maintenance_start_minute'] = 30
-gitaly['daily_maintenance_duration'] = '30m'
-gitaly['daily_maintenance_storages'] = ["default"]
+gitaly['configuration'] = {
+ # ...
+ daily_maintenance: {
+ # ...
+ start_hour: 4,
+ start_minute: 30,
+ duration: '30m',
+ storages: ['default'],
+ },
+}
```
-**For installations from source**
+:::TabTitle Self-compiled (source)
Edit `/home/git/gitaly/config.toml` and add:
@@ -986,6 +1024,8 @@ duration = '30m'
storages = ["default"]
```
+::EndTabs
+
## Rotate Gitaly authentication token
Rotating credentials in a production environment often requires downtime, causes outages, or both.
@@ -1006,7 +1046,7 @@ server" and "Gitaly client" refers to the same machine.
### Verify authentication monitoring
Before rotating a Gitaly authentication token, verify that you can
-[monitor the authentication behavior](monitoring.md#useful-queries) of your GitLab installation using
+[monitor the authentication behavior](monitoring.md#queries) of your GitLab installation using
Prometheus.
You can then continue the rest of the procedure.
@@ -1018,7 +1058,13 @@ transitioning" mode as follows:
```ruby
# in /etc/gitlab/gitlab.rb
-gitaly['auth_transitioning'] = true
+gitaly['configuration'] = {
+ # ...
+ auth: {
+ # ...
+ transitioning: true,
+ },
+}
```
After you have made this change, your [Prometheus query](#verify-authentication-monitoring)
@@ -1038,8 +1084,13 @@ To update to a new Gitaly authentication token, on each Gitaly client **and** Gi
```ruby
# in /etc/gitlab/gitlab.rb
-
- gitaly['auth_token'] = '<new secret token>'
+ gitaly['configuration'] = {
+ # ...
+ auth: {
+ # ...
+ token: '<new secret token>',
+ },
+ }
```
1. Restart Gitaly:
@@ -1069,7 +1120,13 @@ your Gitaly servers as follows:
```ruby
# in /etc/gitlab/gitlab.rb
-gitaly['auth_transitioning'] = false
+gitaly['configuration'] = {
+ # ...
+ auth: {
+ # ...
+ transitioning: false,
+ },
+}
```
WARNING:
@@ -1088,17 +1145,11 @@ result as you did at the start. For example:
## Pack-objects cache **(FREE SELF)**
-> - [Introduced](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/372) in GitLab 13.11.
-> - It's enabled on GitLab.com.
-> - It's recommended for production use.
-
[Gitaly](index.md), the service that provides storage for Git
repositories, can be configured to cache a short rolling window of Git
fetch responses. This can reduce server load when your server receives
lots of CI fetch traffic.
-### Overview
-
The pack-objects cache wraps `git pack-objects`, an internal part of
Git that gets invoked indirectly via the PostUploadPack and
SSHUploadPack Gitaly RPCs. Gitaly runs PostUploadPack when a
@@ -1131,31 +1182,40 @@ automatically keep parts of the pack-objects cache files in RAM,
making it faster.
Because the pack-objects cache can lead to a significant increase in
-disk write IO, it is off by default.
+disk write IO, it is off by default. In GitLab 15.11 and later,
+the write workload is approximately 50% lower, but the cache is still disabled by default.
### Configure the cache
-These are the configuration settings for the pack-objects cache. Each
-setting is discussed in greater detail below.
+These configuration settings are available for the pack-objects cache. Each setting is discussed in greater detail
+below.
-|Setting|Default|Description|
-|:---|:---|:---|
-|`enabled`|`false`|Turns on the cache. When off, Gitaly runs a dedicated `git pack-objects` process for each request. |
-|`dir`|`<PATH TO FIRST STORAGE>/+gitaly/PackObjectsCache`|Local directory where cache files get stored.|
-|`max_age`|`5m` (5 minutes)|Cache entries older than this get evicted and removed from disk.|
+| Setting | Default | Description |
+|:----------|:---------------------------------------------------|:---------------------------------------------------------------------------------------------------|
+| `enabled` | `false` | Turns on the cache. When off, Gitaly runs a dedicated `git pack-objects` process for each request. |
+| `dir` | `<PATH TO FIRST STORAGE>/+gitaly/PackObjectsCache` | Local directory where cache files get stored. |
+| `max_age` | `5m` (5 minutes) | Cache entries older than this get evicted and removed from disk. |
+| `min_occurrences` | 1 | Minimum times a key must occur before a cache entry is created. |
In `/etc/gitlab/gitlab.rb`, set:
```ruby
-gitaly['pack_objects_cache_enabled'] = true
-## gitaly['pack_objects_cache_dir'] = '/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache'
-## gitaly['pack_objects_cache_max_age'] = '5m'
+gitaly['configuration'] = {
+ # ...
+ pack_objects_cache: {
+ # ...
+ enabled: true,
+ # dir: '/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache',
+ # max_age: '5m',
+ # min_occurrences: 1,
+ },
+}
```
#### `enabled` defaults to `false`
-The cache is disabled by default. This is because in some cases, it
-can create an [extreme increase](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4010#note_534564684)
+The cache is disabled by default because in some cases, it can create an
+[extreme increase](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4010#note_534564684)
in the number of bytes written to disk. On GitLab.com, we have verified
that our repository storage disks can handle this extra workload, but
we felt we cannot assume this is true everywhere.
@@ -1198,21 +1258,37 @@ The amount of space required depends on:
- The size of the `max_age` cache eviction window.
If your users pull 100 MB/s and you use a 5 minute window, then on average you have
-`5*60*100MB = 30GB` of data in your cache directory. This is an expected average, not
+`5*60*100MB = 30GB` of data in your cache directory. This average is an expected average, not
a guarantee. Peak size may exceed this average.
#### Cache eviction window `max_age`
The `max_age` configuration setting lets you control the chance of a
cache hit and the average amount of storage used by cache files.
-Entries older than `max_age` get evicted from the in-memory metadata
-store, and deleted from disk.
+Entries older than `max_age` get deleted from the disk.
+
+Eviction does not interfere with ongoing requests. It is OK for `max_age` to be less than the time it takes to do a
+fetch over a slow connection because Unix filesystems do not truly delete a file until all processes that are reading
+the deleted file have closed it.
+
+#### Minimum key occurrences `min_occurrences`
+
+> [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/2222) in GitLab 15.11.
-Eviction does not interfere with ongoing requests, so it is OK
-for `max_age` to be less than the time it takes to do a fetch over a
-slow connection. This is because Unix filesystems do not truly delete
-a file until all processes that are reading the deleted file have
-closed it.
+The `min_occurrences` setting controls how often an identical request
+must occur before we create a new cache entry. The default value is `1`,
+meaning that unique requests do not get written into the cache.
+
+If you:
+
+- Increase this number, your cache hit rate goes down and the
+cache uses less disk space.
+- Decrease this number, your cache hit
+rate goes up and the cache uses more disk space.
+
+You should set `min_occurrences` to `1`. On GitLab.com,
+going from 0 to 1 saved us 50% cache disk space while barely affecting
+the cache hit rate.
### Observe the cache
@@ -1315,7 +1391,32 @@ process repositories that do not pass consistency checks.
For Omnibus GitLab installations, edit `/etc/gitlab/gitlab.rb` and set the
following keys (in this example, to disable the `hasDotgit` consistency check):
-- In [GitLab 15.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6800) and later:
+- In [GitLab 15.10](https://gitlab.com/gitlab-org/gitaly/-/issues/4754) and later:
+
+ ```ruby
+ ignored_blobs = "/etc/gitlab/instance_wide_ignored_git_blobs.txt"
+
+ gitaly['configuration'] = {
+ # ...
+ git: {
+ # ...
+ config: [
+ # Populate a file with one unabbreviated SHA-1 per line.
+ # See https://git-scm.com/docs/git-config#Documentation/git-config.txt-fsckskipList
+ { key: "fsck.skipList", value: ignored_blobs },
+ { key: "fetch.fsck.skipList", value: ignored_blobs },
+ { key: "receive.fsck.skipList", value: ignored_blobs },
+
+ { key: "fsck.hasDotgit", value: "ignore" },
+ { key: "fetch.fsck.hasDotgit", value: "ignore" },
+ { key: "receive.fsck.hasDotgit", value: "ignore" },
+ { key: "fsck.missingSpaceBeforeEmail", value: "ignore" },
+ ],
+ },
+ }
+ ```
+
+- In [GitLab 15.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6800) to GitLab 15.9:
```ruby
ignored_blobs = "/etc/gitlab/instance_wide_ignored_git_blobs.txt"
@@ -1407,10 +1508,14 @@ By default, Gitaly doesn't sign commits made using GitLab UI. For example, commi
- Web IDE.
- Merge requests.
-You can configure Gitaly to sign commits made using GitLab UI. The commits show as unverified and signed by an unknown user. Support for improvements is
-proposed in issue [19185](https://gitlab.com/gitlab-org/gitlab/-/issues/19185).
+You can configure Gitaly to sign commits made with the GitLab UI. The commits show as unverified and signed by an unknown
+user. Support for improvements is proposed in [issue 19185](https://gitlab.com/gitlab-org/gitlab/-/issues/19185).
+
+Configure Gitaly to sign commits made with the GitLab UI in one of two ways:
+
+::Tabs
-**For Omnibus GitLab**
+:::TabTitle Linux package (Omnibus)
1. [Create a GPG key](../../user/project/repository/gpg_signed_commits/index.md#create-a-gpg-key)
and export it. For optimal performance, consider using an EdDSA key.
@@ -1423,12 +1528,18 @@ proposed in issue [19185](https://gitlab.com/gitlab-org/gitlab/-/issues/19185).
1. Edit `/etc/gitlab/gitlab.rb` and configure `gitaly['gpg_signing_key_path']`:
```ruby
- gitaly['gpg_signing_key_path'] = "/etc/gitlab/gitaly/signing_key.gpg"
+ gitaly['configuration'] = {
+ # ...
+ git: {
+ # ...
+ signing_key: '/etc/gitlab/gitaly/signing_key.gpg',
+ },
+ }
```
1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
-**For installations from source**
+:::TabTitle Self-compiled (source)
1. [Create a GPG key](../../user/project/repository/gpg_signed_commits/index.md#create-a-gpg-key)
and export it. For optimal performance, consider using an EdDSA key.
@@ -1446,3 +1557,60 @@ proposed in issue [19185](https://gitlab.com/gitlab-org/gitlab/-/issues/19185).
```
1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).
+
+::EndTabs
+
+## Generate configuration using an external command
+
+> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4828) in GitLab 15.11.
+
+You can generate parts of the Gitaly configuration using an external command. You might do this:
+
+- To configure nodes without having to distribute the full configuration to each of them.
+- To configure using auto-discovery of the node's settings. For example, using DNS entries.
+- To configure secrets at startup of the node, so that don't need to be visible in plain text.
+
+To generate configuration using an external command, you must provide a script that dumps the
+desired configuration of the Gitaly node in JSON format to its standard output.
+
+For example, the following command configures the HTTP password used to connect to the
+GitLab internal API using an AWS secret:
+
+```ruby
+#!/usr/bin/env ruby
+require 'json'
+JSON.generate({"gitlab": {"http_settings": {"password": `aws get-secret-value --secret-id ...`}}})
+```
+
+You must then make the script path known to Gitaly in one of two ways:
+
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
+
+Edit `/etc/gitlab/gitlab.rb` and configure the `config_command`:
+
+```ruby
+gitaly['configuration'] = {
+ config_command: '/path/to/config_command',
+}
+```
+
+:::TabTitle Self-compiled (source)
+
+Edit `/home/git/gitaly/config.toml` and configure `config_command`:
+
+```toml
+config_command = "/path/to/config_command"
+```
+
+::EndTabs
+
+After configuration, Gitaly executes the command on startup and parses its
+standard output as JSON. The resulting configuration is then merged back into
+the other Gitaly configuration.
+
+Gitaly fails to start up if either:
+
+- The configuration command fails.
+- The output produced by the command cannot be parsed as valid JSON.
diff --git a/doc/administration/gitaly/gitaly_geo_capabilities.md b/doc/administration/gitaly/gitaly_geo_capabilities.md
new file mode 100644
index 00000000000..e4147eec162
--- /dev/null
+++ b/doc/administration/gitaly/gitaly_geo_capabilities.md
@@ -0,0 +1,41 @@
+---
+stage: Systems
+group: Gitaly
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Gitaly and Geo capabilities
+
+It is common to want the most available, quickly recoverable, highly performant,
+and fully resilient solution for your data. However, there are tradeoffs.
+
+The following tables are intended to guide you to choose the right combination of capabilities based on your requirements.
+
+## Gitaly capabilities
+
+| Capability | Availability | Recoverability | Data Resiliency | Performance | Risks/Trade-offs|
+|------------|--------------|----------------|-----------------|-------------|-----------------|
+|Gitaly Cluster | Very high - tolerant of node failures | RTO for a single node of 10 s with no manual intervention | Data is stored on multiple nodes | Good - While writes may take slightly longer due to voting, read distribution improves read speeds | **Trade-off** - Slight decrease in write speed for redundant, strongly-consistent storage solution. **Risks** - [Does not support snapshot backups](../gitaly/index.md#snapshot-backup-and-recovery-limitations), GitLab backup task can be slow for large data sets |
+|Gitaly Shards | Single storage location is a single point of failure | Would need to restore only shards which failed | Single point of failure | Good - can allocate repositories to shards to spread load | **Trade-off** - Need to manually configure repositories into different shards to balance loads / storage space **Risks** - Single point of failure relies on recovery process when single-node failure occurs |
+|Gitaly + NFS | Single storage location is a single point of failure | Single node failure requires restoration from backup | Single point of failure | Average - NFS is not ideally suited to large quantities of small reads / writes which can have a detrimental impact on performance | **Trade-off** - Familiar administration though NFS is not ideally suited to Git demands **Risks** - Many instances of NFS compatibility issues which provide very poor customer experiences |
+
+## Geo capabilities
+
+If your availability needs to span multiple zones or multiple locations, read about [Geo](../geo/index.md).
+
+| Capability | Availability | Recoverability | Data Resiliency | Performance | Risks/Trade-offs|
+|------------|--------------|----------------|-----------------|-------------|-----------------|
+|Geo| Depends on the architecture of the Geo site. It is possible to deploy secondaries in single and multiple node configurations. | Eventually consistent. Recovery point depends on replication lag, which depends on a number of factors such as network speeds. Geo supports failover from a primary to secondary site using manual commands that are scriptable. | Geo replicates 100% of planned data types and verifies 50%. See [limitations table](../geo/replication/datatypes.md#limitations-on-replicationverification) for more detail. | Improves read/clone times for users of a secondary. | Geo is not intended to replace other backup/restore solutions. Because of replication lag and the possibility of replicating bad data from a primary, customers should also take regular backups of their primary site and test the restore process. |
+
+## Scenarios for failure modes and available mitigation paths
+
+The following table outlines failure modes and mitigation paths for the product offerings detailed in the tables above. Note - Gitaly Cluster install assumes an odd number replication factor of 3 or greater
+
+| Gitaly Mode | Loss of Single Gitaly Node | Application / Data Corruption | Regional Outage (Loss of Instance) | Notes |
+| ----------- | -------------------------- | ----------------------------- | ---------------------------------- | ----- |
+| Single Gitaly Node | Downtime - Must restore from backup | Downtime - Must restore from Backup | Downtime - Must wait for outage to end | |
+| Single Gitaly Node + Geo Secondary | Downtime - Must restore from backup, can perform a manual failover to secondary | Downtime - Must restore from Backup, errors could have propagated to secondary | Manual intervention - failover to Geo secondary | |
+| Sharded Gitaly Install | Partial Downtime - Only repositories on impacted node affected, must restore from backup | Partial Downtime - Only repositories on impacted node affected, must restore from backup | Downtime - Must wait for outage to end | |
+| Sharded Gitaly Install + Geo Secondary | Partial Downtime - Only repositories on impacted node affected, must restore from backup, could perform manual failover to secondary for impacted repositories | Partial Downtime - Only repositories on impacted node affected, must restore from backup, errors could have propagated to secondary | Manual intervention - failover to Geo secondary | |
+| Gitaly Cluster Install* | No Downtime - swaps repository primary to another node after 10 seconds | Not applicable; All writes are voted on by multiple Gitaly Cluster nodes | Downtime - Must wait for outage to end | Snapshot backups for Gitaly Cluster nodes not supported at this time |
+| Gitaly Cluster Install* + Geo Secondary | No Downtime - swaps repository primary to another node after 10 seconds | Not applicable; All writes are voted on by multiple Gitaly Cluster nodes | Manual intervention - failover to Geo secondary | Snapshot backups for Gitaly Cluster nodes not supported at this time |
diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md
index 405c56284cf..359d4ef90dc 100644
--- a/doc/administration/gitaly/index.md
+++ b/doc/administration/gitaly/index.md
@@ -69,7 +69,7 @@ the current status of these issues, refer to the referenced issues and epics.
| Issue | Summary | How to avoid |
|:--------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------|
-| Gitaly Cluster + Geo - Issues retrying failed syncs | If Gitaly Cluster is used on a Geo secondary site, repositories that have failed to sync could continue to fail when Geo tries to resync them. Recovering from this state requires assistance from support to run manual steps. | No known solution prior to GitLab 15.0. In GitLab 15.0 to 15.2, enable the [`gitaly_praefect_generated_replica_paths` feature flag](#praefect-generated-replica-paths-gitlab-150-and-later). In GitLab 15.3, the feature flag is enabled by default. |
+| Gitaly Cluster + Geo - Issues retrying failed syncs | If Gitaly Cluster is used on a Geo secondary site, repositories that have failed to sync could continue to fail when Geo tries to resync them. Recovering from this state requires assistance from support to run manual steps. | No known solution prior to GitLab 15.0. In GitLab 15.0 to 15.2, enable the [`gitaly_praefect_generated_replica_paths` feature flag](#praefect-generated-replica-paths-gitlab-150-and-later) on your Geo primary site. In GitLab 15.3, the feature flag is enabled by default. |
| Praefect unable to insert data into the database due to migrations not being applied after an upgrade | If the database is not kept up to date with completed migrations, then the Praefect node is unable to perform standard operation. | Make sure the Praefect database is up and running with all migrations completed (For example: `/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migrate-status` should show a list of all applied migrations). Consider [requesting upgrade assistance](https://about.gitlab.com/support/scheduling-upgrade-assistance/) so your upgrade plan can be reviewed by support. |
| Restoring a Gitaly Cluster node from a snapshot in a running cluster | Because the Gitaly Cluster runs with consistent state, introducing a single node that is behind results in the cluster not being able to reconcile the nodes data and other nodes data | Don't restore a single Gitaly Cluster node from a backup snapshot. If you must restore from backup:<br/><br/>1. [Shut down GitLab](../read_only_gitlab.md#shut-down-the-gitlab-ui).<br/>2. Snapshot all Gitaly Cluster nodes at the same time.<br/>3. Take a database dump of the Praefect database. |
| Rebuilding or replacing an existing Gitaly Cluster node | There is no way to replace existing nodes in place because the Praefect database is relied on to determine the current state of each Gitaly node. Changing how Gitaly Cluster stores repositories is proposed in issue [4218](https://gitlab.com/gitlab-org/gitaly/-/issues/4218). Turning on [repository verification](praefect.md#repository-verification) is proposed in issue [4429](https://gitlab.com/gitlab-org/gitaly/-/issues/4429) so Praefect can detect that data is missing and needs to replicated to a new Gitaly node. | No known solution at this time. |
@@ -78,7 +78,7 @@ the current status of these issues, refer to the referenced issues and epics.
Gitaly Cluster does not support snapshot backups. Snapshot backups can cause issues where the Praefect database becomes
out of sync with the disk storage. Because of how Praefect rebuilds the replication metadata of Gitaly disk information
-during a restore, we recommend using the [official backup and restore Rake tasks](../../raketasks/backup_restore.md).
+during a restore, you should use the [official backup and restore Rake tasks](../../raketasks/backup_restore.md).
The [incremental backup method](../../raketasks/backup_gitlab.md#incremental-repository-backups)
can be used to speed up Gitaly Cluster backups.
@@ -100,7 +100,7 @@ These SSDs should have a throughput of at least:
- 2,000 IOPS for write operations.
These IOPS values are initial recommendations, and may be adjusted to greater or lesser values
-depending on the scale of your environment's workload. If you’re running the environment on a
+depending on the scale of your environment's workload. If you're running the environment on a
cloud provider, refer to their documentation about how to configure IOPS correctly.
For repository data, only local storage is supported for Gitaly and Gitaly Cluster for performance and consistency reasons.
@@ -363,10 +363,10 @@ For example, `@cluster/repositories/6f/96/54771`.
The last component of the replica path, `54771`, is the repository ID. This can be used to identify the repository on the disk.
-`<xx>/<xx>` are the first four hex digits of the SHA256 hash of the string representation of the repository ID. This is used to balance
-the repositories evenly into subdirectories to avoid overly large directories that might cause problems on some file
-systems. In this case, `54771` hashes to `6f960ab01689464e768366d3315b3d3b2c28f38761a58a70110554eb04d582f7` so the
-first four digits are `6f` and `96`.
+`<xx>/<xx>` are the first four hex digits of the SHA256 hash of the string representation of the repository ID. These
+digits are used to balance the repositories evenly into subdirectories to avoid overly large directories that might
+cause problems on some file systems. In this case, `54771` hashes to
+`6f960ab01689464e768366d3315b3d3b2c28f38761a58a70110554eb04d582f7` so the first four digits are `6f` and `96`.
#### Identify repositories on disk
@@ -387,8 +387,9 @@ Follow the [instructions in hashed storage's documentation](../repository_storag
Gitaly Cluster uses the PostgreSQL metadata store with the storage layout to ensure atomicity of repository creation,
deletion, and move operations. The disk operations can't be atomically applied across multiple storages. However, PostgreSQL guarantees
the atomicity of the metadata operations. Gitaly Cluster models the operations in a manner that the failing operations always leave
-the metadata consistent. The disks may contain stale state even after successful operations. This is expected and the leftover state
-does not interfere with future operations but may use up disk space unnecessarily until a clean up is performed.
+the metadata consistent. The disks may contain stale state even after successful operations. This situation is expected and
+the leftover state does not interfere with future operations but may use up disk space unnecessarily until a clean up is
+performed.
There is on-going work on a [background crawler](https://gitlab.com/gitlab-org/gitaly/-/issues/3719) that cleans up the leftover
repositories from the storages.
@@ -397,7 +398,7 @@ repositories from the storages.
When creating repositories, Praefect:
-1. Reserves a repository ID from PostgreSQL. This is atomic and no two creations receive the same ID.
+1. Reserves a repository ID from PostgreSQL, which is atomic and no two creations receive the same ID.
1. Creates replicas on the Gitaly storages in the replica path derived from the repository ID.
1. Creates metadata records after the repository is successfully created on disk.
@@ -470,9 +471,12 @@ _Up to date_ in this context means that:
The primary node is chosen to serve the request if:
-- There are no up to date nodes.
+- No up-to-date nodes exist.
- Any other error occurs during node selection.
+If you have a large, heavily modified repository (like a multi-gigabyte monorepo), the primary node can service most or all requests if changes come in faster than Praefect
+can replicate to the secondaries. When this occurs, CI/CD jobs and other repository traffic are bottlenecked by the capacity of the primary node.
+
You can [monitor distribution of reads](monitoring.md#monitor-gitaly-cluster) using Prometheus.
#### Strong consistency
@@ -677,7 +681,7 @@ Direct Git access is:
For the sake of removing complexity, we must remove direct Git access in GitLab. However, we can't
remove it as long some GitLab installations require Git repositories on NFS.
-There are two facets to our efforts to remove direct Git access in GitLab:
+Two facets of our efforts to remove direct Git access in GitLab are:
- Reduce the number of inefficient Gitaly queries made by GitLab.
- Persuade administrators of fault-tolerant or horizontally-scaled GitLab instances to migrate off
diff --git a/doc/administration/gitaly/monitoring.md b/doc/administration/gitaly/monitoring.md
index 0fd34d5c89f..00d0499faa2 100644
--- a/doc/administration/gitaly/monitoring.md
+++ b/doc/administration/gitaly/monitoring.md
@@ -54,6 +54,9 @@ You can observe the status of [control groups (cgroups)](configure_gitaly.md#con
- `gitaly_cgroups_cpu_usage`, a gauge that measures CPU usage per cgroup.
- `gitaly_cgroup_procs_total`, a gauge that measures the total number of
processes Gitaly has spawned under the control of cgroups.
+- `gitaly_cgroup_cpu_cfs_periods_total`, a counter that for the value of [`nr_periods`](https://docs.kernel.org/scheduler/sched-bwc.html#statistics).
+- `gitaly_cgroup_cpu_cfs_throttled_periods_total`, a counter for the value of [`nr_throttled`](https://docs.kernel.org/scheduler/sched-bwc.html#statistics).
+- `gitaly_cgroup_cpu_cfs_throttled_seconds_total`, a counter for the value of [`throttled_time`](https://docs.kernel.org/scheduler/sched-bwc.html#statistics) in seconds.
## `pack-objects` cache
@@ -86,9 +89,9 @@ gitaly_streamcache_filestore_removed_total{dir="/var/opt/gitlab/git-data/reposit
gitaly_streamcache_index_entries{dir="/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache"} 1
```
-## Useful queries
+## Queries
-The following are useful queries for monitoring Gitaly:
+The following are some queries for monitoring Gitaly:
- Use the following Prometheus query to observe the
[type of connections](configure_gitaly.md#enable-tls-support) Gitaly is serving a production
@@ -130,8 +133,8 @@ The following are useful queries for monitoring Gitaly:
## Monitor Gitaly Cluster
-To monitor Gitaly Cluster (Praefect), you can use these Prometheus metrics. There are two separate metrics
-endpoints from which metrics can be scraped:
+To monitor Gitaly Cluster (Praefect), you can use these Prometheus metrics. Two separate metrics endpoints are
+available from which metrics can be scraped:
- The default `/metrics` endpoint.
- `/db_metrics`, which contains metrics that require database queries.
@@ -153,6 +156,7 @@ The following metrics are available from the `/metrics` endpoint:
- `gitaly_praefect_replication_delay_bucket`, a histogram measuring how much time passes between
when the replication job is created and when it starts. Available in GitLab 12.10 and later.
- `gitaly_praefect_connections_total`, the total number of connections to Praefect. [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4220) in GitLab 14.7.
+- `gitaly_praefect_method_types`, a count of accessor and mutator RPCs per node.
To monitor [strong consistency](index.md#strong-consistency), you can use the following Prometheus metrics:
@@ -179,9 +183,6 @@ To monitor [repository verification](praefect.md#repository-verification), use t
- `gitaly_praefect_stale_verification_leases_released_total`, the number of stale verification leases
released.
-The `/metrics` endpoint also provides all the metrics available under the `/db_metrics` endpoint. Using `/metrics` for `/db_metrics` metrics
-is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/390266) in GitLab 15.9 and will be removed in GitLab 16.0.
-
You can also monitor the [Praefect logs](../logs/index.md#praefect-logs).
### Database metrics `/db_metrics` endpoint
diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md
index 440bd7427ae..51201ec442f 100644
--- a/doc/administration/gitaly/praefect.md
+++ b/doc/administration/gitaly/praefect.md
@@ -142,7 +142,7 @@ with secure tokens as you complete the setup process.
1. `PRAEFECT_EXTERNAL_TOKEN`: repositories hosted on your Praefect cluster can
only be accessed by Gitaly clients that carry this token.
1. `PRAEFECT_INTERNAL_TOKEN`: this token is used for replication traffic inside
- your Praefect cluster. This is distinct from `PRAEFECT_EXTERNAL_TOKEN`
+ your Praefect cluster. This token is distinct from `PRAEFECT_EXTERNAL_TOKEN`
because Gitaly clients must not be able to access internal nodes of the
Praefect cluster directly; that could lead to data loss.
1. `PRAEFECT_SQL_PASSWORD`: this password is used by Praefect to connect to
@@ -269,11 +269,16 @@ The database used by Praefect is now configured.
You can now configure Praefect to use the database:
```ruby
-praefect['database_host'] = POSTGRESQL_HOST
-praefect['database_port'] = 5432
-praefect['database_user'] = 'praefect'
-praefect['database_password'] = PRAEFECT_SQL_PASSWORD
-praefect['database_dbname'] = 'praefect_production'
+praefect['configuration'] = {
+ # ...
+ database: {
+ # ...
+ host: POSTGRESQL_HOST,
+ port: 5432,
+ password: PRAEFECT_SQL_PASSWORD,
+ dbname: 'praefect_production',
+ }
+}
```
If you see Praefect database errors after configuring PostgreSQL, see
@@ -285,19 +290,27 @@ Praefect performance can be improved by additionally configuring the `database_d
settings:
```ruby
-praefect['database_direct_host'] = POSTGRESQL_HOST
-praefect['database_direct_port'] = 5432
-
-# Use the following to override parameters of direct database connection.
-# Comment out where the parameters are the same for both connections.
-
-praefect['database_direct_user'] = 'praefect'
-praefect['database_direct_password'] = PRAEFECT_SQL_PASSWORD
-praefect['database_direct_dbname'] = 'praefect_production'
-#praefect['database_direct_sslmode'] = '...'
-#praefect['database_direct_sslcert'] = '...'
-#praefect['database_direct_sslkey'] = '...'
-#praefect['database_direct_sslrootcert'] = '...'
+praefect['configuration'] = {
+ # ...
+ database: {
+ # ...
+ session_pooled: {
+ # ...
+ host: POSTGRESQL_HOST,
+ port: 5432
+
+ # Use the following to override parameters of direct database connection.
+ # Comment out where the parameters are the same for both connections.
+ user: 'praefect',
+ password: PRAEFECT_SQL_PASSWORD,
+ dbname: 'praefect_production',
+ # sslmode: '...',
+ # sslcert: '...',
+ # sslkey: '...',
+ # sslrootcert: '...',
+ }
+ }
+}
```
When configured, this connection is automatically used for the
@@ -313,8 +326,8 @@ reads distribution caching is enabled by configuration
#### Use PgBouncer
-To reduce PostgreSQL resource consumption, we recommend setting up and configuring
-[PgBouncer](https://www.pgbouncer.org/) in front of the PostgreSQL instance. However, PgBouncer isn't required because
+To reduce PostgreSQL resource consumption, you should set up and configure [PgBouncer](https://www.pgbouncer.org/) in
+front of the PostgreSQL instance. However, PgBouncer isn't required because
Praefect makes a low number of connections. If you choose to use PgBouncer, you can use the same PgBouncer instance for
both the GitLab application database and the Praefect database.
@@ -322,15 +335,21 @@ To configure PgBouncer in front of the PostgreSQL instance, you must point Praef
parameters on Praefect configuration:
```ruby
-praefect['database_host'] = PGBOUNCER_HOST
-praefect['database_port'] = 6432
-praefect['database_user'] = 'praefect'
-praefect['database_password'] = PRAEFECT_SQL_PASSWORD
-praefect['database_dbname'] = 'praefect_production'
-#praefect['database_sslmode'] = '...'
-#praefect['database_sslcert'] = '...'
-#praefect['database_sslkey'] = '...'
-#praefect['database_sslrootcert'] = '...'
+praefect['configuration'] = {
+ # ...
+ database: {
+ # ...
+ host: PGBOUNCER_HOST,
+ port: 6432,
+ user: 'praefect',
+ password: PRAEFECT_SQL_PASSWORD,
+ dbname: 'praefect_production',
+ # sslmode: '...',
+ # sslcert: '...',
+ # sslkey: '...',
+ # sslrootcert: '...',
+ }
+}
```
Praefect requires an additional connection to the PostgreSQL that supports the
@@ -341,12 +360,12 @@ It is not supported in `transaction` pool mode (`pool_mode = transaction`).
To configure the additional connection, you must either:
- Configure a new PgBouncer database that uses to the same PostgreSQL database endpoint,
- but with different pool mode. That is, `pool_mode = session`.
+ but with different pool mode (`pool_mode = session`).
- Connect Praefect directly to PostgreSQL and bypass PgBouncer.
#### Configure a new PgBouncer database with `pool_mode = session`
-We recommend using PgBouncer with `session` pool mode. You can use the
+You should use PgBouncer with `session` pool mode. You can use the
[bundled PgBouncer](../postgresql/pgbouncer.md) or use an external PgBouncer and
[configure it manually](https://www.pgbouncer.org/config.html).
@@ -399,23 +418,30 @@ praefect_production_direct = host=POSTGRESQL_HOST auth_user=pgbouncer dbname=pra
Now you can configure Praefect to use PgBouncer for both connections:
```ruby
-praefect['database_host'] = PGBOUNCER_HOST
-praefect['database_port'] = 6432
-praefect['database_user'] = 'praefect'
-# `PRAEFECT_SQL_PASSWORD` is the plain-text password of
-# Praefect user. Not to be confused with `PRAEFECT_SQL_PASSWORD_HASH`.
-praefect['database_password'] = PRAEFECT_SQL_PASSWORD
-
-praefect['database_dbname'] = 'praefect_production'
-praefect['database_direct_dbname'] = 'praefect_production_direct'
-
-# There is no need to repeat the following. Parameters of direct
-# database connection will fall back to the values above.
-
-#praefect['database_direct_host'] = PGBOUNCER_HOST
-#praefect['database_direct_port'] = 6432
-#praefect['database_direct_user'] = 'praefect'
-#praefect['database_direct_password'] = PRAEFECT_SQL_PASSWORD
+praefect['configuration'] = {
+ # ...
+ database: {
+ # ...
+ host: PGBOUNCER_HOST,
+ port: 6432,
+ user: 'praefect',
+ # `PRAEFECT_SQL_PASSWORD` is the plain-text password of
+ # Praefect user. Not to be confused with `PRAEFECT_SQL_PASSWORD_HASH`.
+ password: PRAEFECT_SQL_PASSWORD,
+ dbname: 'praefect_production',
+ session_pooled: {
+ # ...
+ dbname: 'praefect_production_direct',
+ # There is no need to repeat the following. Parameters of direct
+ # database connection will fall back to the values above.
+ #
+ # host: PGBOUNCER_HOST,
+ # port: 6432,
+ # user: 'praefect',
+ # password: PRAEFECT_SQL_PASSWORD,
+ },
+ },
+}
```
With this configuration, Praefect uses PgBouncer for both connection types.
@@ -428,25 +454,34 @@ configuration option is set. For more details, consult the PgBouncer documentati
#### Configure Praefect to connect directly to PostgreSQL
-As an alternative to configuring PgBouncer with `session` pool mode, Praefect can be configured to use different connection parameters for direct access
-to PostgreSQL. This is the connection that supports the `LISTEN` feature.
+As an alternative to configuring PgBouncer with `session` pool mode, Praefect can be configured to use different
+connection parameters for direct access to PostgreSQL. This connection supports the `LISTEN` feature.
An example of Praefect configuration that bypasses PgBouncer and directly connects to PostgreSQL:
```ruby
-praefect['database_direct_host'] = POSTGRESQL_HOST
-praefect['database_direct_port'] = 5432
-
-# Use the following to override parameters of direct database connection.
-# Comment out where the parameters are the same for both connections.
-
-praefect['database_direct_user'] = 'praefect'
-praefect['database_direct_password'] = PRAEFECT_SQL_PASSWORD
-praefect['database_direct_dbname'] = 'praefect_production'
-#praefect['database_direct_sslmode'] = '...'
-#praefect['database_direct_sslcert'] = '...'
-#praefect['database_direct_sslkey'] = '...'
-#praefect['database_direct_sslrootcert'] = '...'
+praefect['configuration'] = {
+ # ...
+ database: {
+ # ...
+ session_pooled: {
+ # ...
+ host: POSTGRESQL_HOST,
+ port: 5432,
+
+ # Use the following to override parameters of direct database connection.
+ # Comment out where the parameters are the same for both connections.
+ #
+ user: 'praefect',
+ password: PRAEFECT_SQL_PASSWORD,
+ dbname: 'praefect_production',
+ # sslmode: '...',
+ # sslcert: '...',
+ # sslkey: '...',
+ # sslrootcert: '...',
+ },
+ },
+}
```
### Praefect
@@ -501,30 +536,42 @@ Updates to example must be made at:
`/etc/gitlab/gitlab.rb`:
```ruby
- praefect['listen_addr'] = '0.0.0.0:2305'
+ praefect['configuration'] = {
+ # ...
+ listen_addr: '0.0.0.0:2305',
+ }
```
1. Configure Prometheus metrics by editing
`/etc/gitlab/gitlab.rb`:
```ruby
- # Enable Prometheus metrics access to Praefect. You must use firewalls
- # to restrict access to this address/port.
- # The default metrics endpoint is /metrics
- praefect['prometheus_listen_addr'] = '0.0.0.0:9652'
-
- # Some metrics run queries against the database. Enabling separate database metrics allows
- # these metrics to be collected when the metrics are
- # scraped on a separate /db_metrics endpoint.
- praefect['separate_database_metrics'] = true
+ praefect['configuration'] = {
+ # ...
+ #
+ # Enable Prometheus metrics access to Praefect. You must use firewalls
+ # to restrict access to this address/port.
+ # The default metrics endpoint is /metrics
+ prometheus_listen_addr: '0.0.0.0:9652',
+ # Some metrics run queries against the database. Enabling separate database metrics allows
+ # these metrics to be collected when the metrics are
+ # scraped on a separate /db_metrics endpoint.
+ prometheus_exclude_database_from_default_metrics: true,
+ }
```
-1. Configure a strong `auth_token` for **Praefect** by editing
- `/etc/gitlab/gitlab.rb`. This is needed by clients outside the cluster
+1. Configure a strong authentication token for **Praefect** by editing
+ `/etc/gitlab/gitlab.rb`, which is needed by clients outside the cluster
(like GitLab Shell) to communicate with the Praefect cluster:
```ruby
- praefect['auth_token'] = 'PRAEFECT_EXTERNAL_TOKEN'
+ praefect['configuration'] = {
+ # ...
+ auth: {
+ # ...
+ token: 'PRAEFECT_EXTERNAL_TOKEN',
+ },
+ }
```
1. Configure **Praefect** to [connect to the PostgreSQL database](#postgresql). We
@@ -533,19 +580,32 @@ Updates to example must be made at:
If you want to use a TLS client certificate, the options below can be used:
```ruby
- # Connect to PostgreSQL using a TLS client certificate
- # praefect['database_sslcert'] = '/path/to/client-cert'
- # praefect['database_sslkey'] = '/path/to/client-key'
-
- # Trust a custom certificate authority
- # praefect['database_sslrootcert'] = '/path/to/rootcert'
+ praefect['configuration'] = {
+ # ...
+ database: {
+ # ...
+ #
+ # Connect to PostgreSQL using a TLS client certificate
+ # sslcert: '/path/to/client-cert',
+ # sslkey: '/path/to/client-key',
+ #
+ # Trust a custom certificate authority
+ # sslrootcert: '/path/to/rootcert',
+ },
+ }
```
By default, Praefect refuses to make an unencrypted connection to
PostgreSQL. You can override this by uncommenting the following line:
```ruby
- # praefect['database_sslmode'] = 'disable'
+ praefect['configuration'] = {
+ # ...
+ database: {
+ # ...
+ # sslmode: 'disable',
+ },
+ }
```
1. Configure the **Praefect** cluster to connect to each Gitaly node in the
@@ -573,29 +633,37 @@ Updates to example must be made at:
NOTE:
When adding additional Gitaly nodes to a virtual storage, all storage names
- within that virtual storage must be unique. Additionally, all Gitaly node
+ in that virtual storage must be unique. Additionally, all Gitaly node
addresses referenced in the Praefect configuration must be unique.
```ruby
# Name of storage hash must match storage name in git_data_dirs on GitLab
- # server ('default') and in git_data_dirs on Gitaly nodes ('gitaly-1')
- praefect['virtual_storages'] = {
- 'default' => {
- 'nodes' => {
- 'gitaly-1' => {
- 'address' => 'tcp://GITALY_HOST_1:8075',
- 'token' => 'PRAEFECT_INTERNAL_TOKEN',
- },
- 'gitaly-2' => {
- 'address' => 'tcp://GITALY_HOST_2:8075',
- 'token' => 'PRAEFECT_INTERNAL_TOKEN'
+ # server ('default') and in gitaly['configuration'][:storage][INDEX][:name] on Gitaly nodes ('gitaly-1')
+ praefect['configuration'] = {
+ # ...
+ virtual_storage: [
+ {
+ # ...
+ name: 'default',
+ node: [
+ {
+ storage: 'gitaly-1',
+ address: 'tcp://GITALY_HOST_1:8075',
+ token: 'PRAEFECT_INTERNAL_TOKEN'
+ },
+ {
+ storage: 'gitaly-2',
+ address: 'tcp://GITALY_HOST_2:8075',
+ token: 'PRAEFECT_INTERNAL_TOKEN'
+ },
+ {
+ storage: 'gitaly-3',
+ address: 'tcp://GITALY_HOST_3:8075',
+ token: 'PRAEFECT_INTERNAL_TOKEN'
+ },
+ ],
},
- 'gitaly-3' => {
- 'address' => 'tcp://GITALY_HOST_3:8075',
- 'token' => 'PRAEFECT_INTERNAL_TOKEN'
- }
- }
- }
+ ],
}
```
@@ -662,7 +730,7 @@ for secure connections, you must:
Additionally the certificate, or its certificate authority, must be installed on all Gitaly servers
and on all Praefect clients that communicate with it following the procedure described in
-[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) (and repeated below).
+[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl/index.html#install-custom-public-certificates) (and repeated below).
Note the following:
@@ -681,7 +749,14 @@ Note the following:
This allows you to do a gradual transition from unencrypted to encrypted traffic, if
necessary.
- To disable the unencrypted listener, set `praefect['listen_addr'] = nil`.
+ To disable the unencrypted listener, set:
+
+ ```ruby
+ praefect['configuration'] = {
+ # ...
+ listen_addr: nil,
+ }
+ ```
To configure Praefect with TLS:
@@ -702,9 +777,15 @@ To configure Praefect with TLS:
1. Edit `/etc/gitlab/gitlab.rb` and add:
```ruby
- praefect['tls_listen_addr'] = "0.0.0.0:3305"
- praefect['certificate_path'] = "/etc/gitlab/ssl/cert.pem"
- praefect['key_path'] = "/etc/gitlab/ssl/key.pem"
+ praefect['configuration'] = {
+ # ...
+ tls_listen_addr: '0.0.0.0:3305',
+ tls: {
+ # ...
+ certificate_path: '/etc/gitlab/ssl/cert.pem',
+ key_path: '/etc/gitlab/ssl/key.pem',
+ },
+ }
```
1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure).
@@ -790,6 +871,125 @@ To configure Praefect with TLS:
1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).
+#### Service discovery
+
+> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/8971) in GitLab 15.10.
+
+Prerequisites:
+
+- A DNS server.
+
+GitLab uses service discovery to retrieve a list of Praefect hosts. Service
+discovery involves periodic checks of a DNS A or AAAA record, with the IPs
+retrieved from the record serving as the addresses of the target nodes.
+Praefect does not support service discovery by SRV record.
+
+By default, the minimum time between checks is 5 minutes, regardless of the
+records' TTLs. Praefect does not support customizing this interval. When clients
+receive an update, they:
+
+- Establish new connections to the new IP addresses.
+- Keep existing connections to intact IP addresses.
+- Drop connections to removed IP addresses.
+
+In-flight requests on to-be-removed connections are still handled until they
+finish. Workhorse has a 10-minute timeout, while other clients do not specify a
+graceful timeout.
+
+The DNS server should return all IP addresses instead of load-balancing itself.
+Clients can distribute requests to IP addresses in a round-robin fashion.
+
+Before updating client configuration, ensure that DNS service discovery works
+correctly. It should return the list of IP addresses correctly. `dig` is a good
+tool to use to verify.
+
+```console
+❯ dig A praefect.service.consul @127.0.0.1
+
+; <<>> DiG 9.10.6 <<>> A praefect.service.consul @127.0.0.1
+;; global options: +cmd
+;; Got answer:
+;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29210
+;; flags: qr aa rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
+
+;; OPT PSEUDOSECTION:
+; EDNS: version: 0, flags:; udp: 4096
+;; QUESTION SECTION:
+;praefect.service.consul. IN A
+
+;; ANSWER SECTION:
+praefect.service.consul. 0 IN A 10.0.0.3
+praefect.service.consul. 0 IN A 10.0.0.2
+praefect.service.consul. 0 IN A 10.0.0.1
+
+;; Query time: 0 msec
+;; SERVER: ::1#53(::1)
+;; WHEN: Wed Dec 14 12:53:58 +07 2022
+;; MSG SIZE rcvd: 86
+```
+
+##### Configure service discovery
+
+By default, Praefect delegates DNS resolution to the operating system. In such
+cases, the Gitaly address can be set in either of these formats:
+
+- `dns:[host]:[port]`
+- `dns:///[host]:[port]` (note the three slashes)
+
+You can also appoint an authoritative name server by setting it in this format:
+
+- `dns://[authority_host]:[authority_port]/[host]:[port]`
+
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
+
+1. Edit `/etc/gitlab/gitlab.rb` and add:
+
+ ```ruby
+ praefect['consul_service_name'] = 'praefect'
+ ```
+
+1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+1. On the Praefect clients (except Gitaly servers), edit `git_data_dirs` in
+`/etc/gitlab/gitlab.rb` as follows. Replace `PRAEFECT_SERVICE_DISCOVERY_ADDRESS`
+with Praefect service discovery address, such as `praefect.service.consul`.
+
+ ```ruby
+ git_data_dirs({
+ "default" => {
+ "gitaly_address" => 'dns:PRAEFECT_SERVICE_DISCOVERY_ADDRESS:2305',
+ "gitaly_token" => 'PRAEFECT_EXTERNAL_TOKEN'
+ }
+ })
+ ```
+
+1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+
+:::TabTitle Self-compiled (source)
+
+1. Install a DNS service discovery service. Register all Praefect nodes with the service.
+1. On the Praefect clients (except Gitaly servers), edit `storages` in
+ `/home/git/gitlab/config/gitlab.yml` as follows:
+
+ ```yaml
+ gitlab:
+ repositories:
+ storages:
+ default:
+ gitaly_address: dns:PRAEFECT_SERVICE_DISCOVERY_ADDRESS:2305
+ path: /some/local/path
+ ```
+
+ NOTE:
+ `/some/local/path` should be set to a local folder that exists, however no
+ data is stored in this folder. [Issue 375254](https://gitlab.com/gitlab-org/gitlab/-/issues/375254)
+ proposes to remove this requirement.
+
+1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).
+
+::EndTabs
+
### Gitaly
NOTE:
@@ -813,12 +1013,11 @@ because we rely on Praefect to route operations correctly.
Particular attention should be shown to:
-- The `gitaly['auth_token']` configured in this section must match the `token`
- value under `praefect['virtual_storages']['nodes']` on the Praefect node. This was set
- in the [previous section](#praefect). This document uses the placeholder
- `PRAEFECT_INTERNAL_TOKEN` throughout.
-- The storage names in `git_data_dirs` configured in this section must match the
- storage names under `praefect['virtual_storages']` on the Praefect node. This
+- The `gitaly['configuration'][:auth][:token]` configured in this section must match the `token`
+ value under `praefect['configuration'][:virtual_storage][<index>][:node][<index>][:token]` on the Praefect node. This value was
+ set in the [previous section](#praefect). This document uses the placeholder `PRAEFECT_INTERNAL_TOKEN` throughout.
+- The storage names in `gitaly['configuration'][:storage]` configured in this section must match the
+ storage names under `praefect['configuration'][:virtual_storage]` on the Praefect node. This
was set in the [previous section](#praefect). This document uses `gitaly-1`,
`gitaly-2`, and `gitaly-3` as Gitaly storage names.
@@ -834,7 +1033,7 @@ For more information on Gitaly server configuration, see our
1. Disable all other services by editing `/etc/gitlab/gitlab.rb`:
```ruby
- # Disable all other services on the Praefect node
+ # Disable all other services on the Gitaly node
postgresql['enable'] = false
redis['enable'] = false
nginx['enable'] = false
@@ -859,22 +1058,31 @@ For more information on Gitaly server configuration, see our
`/etc/gitlab/gitlab.rb`:
```ruby
- # Make Gitaly accept connections on all network interfaces.
- # Use firewalls to restrict access to this address/port.
- gitaly['listen_addr'] = '0.0.0.0:8075'
-
- # Enable Prometheus metrics access to Gitaly. You must use firewalls
- # to restrict access to this address/port.
- gitaly['prometheus_listen_addr'] = '0.0.0.0:9236'
+ gitaly['configuration'] = {
+ # ...
+ #
+ # Make Gitaly accept connections on all network interfaces.
+ # Use firewalls to restrict access to this address/port.
+ listen_addr: '0.0.0.0:8075',
+ # Enable Prometheus metrics access to Gitaly. You must use firewalls
+ # to restrict access to this address/port.
+ prometheus_listen_addr: '0.0.0.0:9236',
+ }
```
1. Configure a strong `auth_token` for **Gitaly** by editing
- `/etc/gitlab/gitlab.rb`. This is needed by clients to communicate with
+ `/etc/gitlab/gitlab.rb`, which is needed by clients to communicate with
this Gitaly nodes. Typically, this token is the same for all Gitaly
nodes.
```ruby
- gitaly['auth_token'] = 'PRAEFECT_INTERNAL_TOKEN'
+ gitaly['configuration'] = {
+ # ...
+ auth: {
+ # ...
+ token: 'PRAEFECT_INTERNAL_TOKEN',
+ },
+ }
```
1. Configure the GitLab Shell secret token, which is needed for `git push` operations. Either:
@@ -903,13 +1111,13 @@ For more information on Gitaly server configuration, see our
gitlab_rails['internal_api_url'] = 'http://GITLAB_HOST'
```
-1. Configure the storage location for Git data by setting `git_data_dirs` in
+1. Configure the storage location for Git data by setting `gitaly['configuration'][:storage]` in
`/etc/gitlab/gitlab.rb`. Each Gitaly node should have a unique storage name
(such as `gitaly-1`).
- Instead of configuring `git_data_dirs` uniquely for each Gitaly node, it is
+ Instead of configuring `gitaly['configuration'][:storage]` uniquely for each Gitaly node, it is
often easier to have include the configuration for all Gitaly nodes on every
- Gitaly node. This is supported because the Praefect `virtual_storages`
+ Gitaly node. You can do this because the Praefect `virtual_storage`
configuration maps each storage name (such as `gitaly-1`) to a specific node, and
requests are routed accordingly. This means every Gitaly node in your fleet
can share the same configuration.
@@ -918,17 +1126,23 @@ For more information on Gitaly server configuration, see our
# You can include the data dirs for all nodes in the same config, because
# Praefect will only route requests according to the addresses provided in the
# prior step.
- git_data_dirs({
- "gitaly-1" => {
- "path" => "/var/opt/gitlab/git-data"
- },
- "gitaly-2" => {
- "path" => "/var/opt/gitlab/git-data"
- },
- "gitaly-3" => {
- "path" => "/var/opt/gitlab/git-data"
- }
- })
+ gitaly['configuration'] = {
+ # ...
+ storage: [
+ {
+ name: 'gitaly-1',
+ path: '/var/opt/gitlab/git-data',
+ },
+ {
+ name: 'gitaly-2',
+ path: '/var/opt/gitlab/git-data',
+ },
+ {
+ name: 'gitaly-3',
+ path: '/var/opt/gitlab/git-data',
+ },
+ ],
+ }
```
1. Save the changes to `/etc/gitlab/gitlab.rb` and
@@ -967,10 +1181,7 @@ scope of the GitLab documentation.
NOTE:
The load balancer must be configured to accept traffic from the Gitaly nodes in
-addition to the GitLab nodes. Some requests handled by
-[`gitaly-ruby`](configure_gitaly.md#gitaly-ruby) sidecar processes call into the main Gitaly
-process. `gitaly-ruby` uses the Gitaly address set in the GitLab server's
-`git_data_dirs` setting to make this connection.
+addition to the GitLab nodes.
We hope that if you're managing fault-tolerant systems like GitLab, you have a load balancer
of choice already. Some examples include [HAProxy](https://www.haproxy.org/)
@@ -980,8 +1191,8 @@ Big-IP LTM, and Citrix Net Scaler. This documentation outlines what ports
and protocols you need configure.
NOTE:
-We recommend the equivalent of HAProxy `leastconn` load-balancing strategy because long-running operations (for example,
-clones) keep some connections open for extended periods.
+You should use the equivalent of HAProxy `leastconn` load-balancing strategy because long-running operations (for
+example, clones) keep some connections open for extended periods.
| LB Port | Backend Port | Protocol |
|:--------|:-------------|:---------|
@@ -995,12 +1206,12 @@ To complete this section you need:
- [Configured Gitaly nodes](#gitaly)
The Praefect cluster needs to be exposed as a storage location to the GitLab
-application. This is done by updating the `git_data_dirs`.
+application, which is done by updating the `git_data_dirs`.
Particular attention should be shown to:
- the storage name added to `git_data_dirs` in this section must match the
- storage name under `praefect['virtual_storages']` on the Praefect nodes. This
+ storage name under `praefect['configuration'][:virtual_storage]` on the Praefect nodes. This
was set in the [Praefect](#praefect) section of this guide. This document uses
`default` as the Praefect storage name.
@@ -1219,12 +1430,16 @@ You can configure:
The configuration is added to the `/etc/gitlab/gitlab.rb` file:
```ruby
- praefect['virtual_storages'] = {
- 'default' => {
- 'default_replication_factor' => 1,
+ praefect['configuration'] = {
# ...
- }
- }
+ virtual_storage: [
+ {
+ # ...
+ name: 'default',
+ default_replication_factor: 1,
+ },
+ ],
+ }
```
- A replication factor for an existing repository using the `set-replication-factor` sub-command.
@@ -1313,29 +1528,50 @@ interval is configurable with any valid [Go duration string](https://pkg.go.dev/
To verify the metadata every three days:
```ruby
-praefect['background_verification_verification_interval'] = '72h'
+praefect['configuration'] = {
+ # ...
+ background_verification: {
+ # ...
+ verification_interval: '72h',
+ },
+}
```
Values of 0 and below disable the background verifier.
```ruby
-praefect['background_verification_verification_interval'] = '0'
+praefect['configuration'] = {
+ # ...
+ background_verification: {
+ # ...
+ verification_interval: '0',
+ },
+}
```
#### Enable deletions
+> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4080) and disabled by default in GitLab 15.0
+> - [Default enabled](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5321) in GitLab 15.9.
+
WARNING:
-Deletions are disabled by default due to a race condition with repository renames that can cause incorrect
-deletions. This is especially prominent in Geo instances as Geo performs more renames than instances without Geo.
-You should enable deletions only if the [`gitaly_praefect_generated_replica_paths` feature flag](index.md#praefect-generated-replica-paths-gitlab-150-and-later) is enabled.
+Deletions were disabled by default prior to GitLab 15.9 due to a race condition with repository renames
+that can cause incorrect deletions, which is especially prominent in Geo instances as Geo performs more renames
+than instances without Geo. In GitLab 15.0 to 15.5, you should enable deletions only if the [`gitaly_praefect_generated_replica_paths` feature flag](index.md#praefect-generated-replica-paths-gitlab-150-and-later) is enabled. The feature flag was removed in GitLab 15.6 making deletions always safe to enable.
-By default, the worker does not delete invalid metadata records but logs them and outputs Prometheus
-metrics for them.
+By default, the worker deletes invalid metadata records. It also logs the deleted records and outputs Prometheus
+metrics.
-You can enable deleting invalid metadata records with:
+You can disable deleting invalid metadata records with:
```ruby
-praefect['background_verification_delete_invalid_records'] = true
+praefect['configuration'] = {
+ # ...
+ background_verification: {
+ # ...
+ delete_invalid_records: false,
+ },
+}
```
### Prioritize verification manually
@@ -1370,10 +1606,10 @@ The output includes the number of replicas that were marked unverified.
## Automatic failover and primary election strategies
-Praefect regularly checks the health of each Gitaly node. This is used to automatically fail over
+Praefect regularly checks the health of each Gitaly node, which is used to automatically fail over
to a newly-elected primary Gitaly node if the current primary node is found to be unhealthy.
-We recommend using [repository-specific primary nodes](#repository-specific-primary-nodes). This is
+You should use [repository-specific primary nodes](#repository-specific-primary-nodes). This is
[the only available election strategy](https://gitlab.com/gitlab-org/gitaly/-/issues/3574) from GitLab 14.0.
### Repository-specific primary nodes
diff --git a/doc/administration/gitaly/recovery.md b/doc/administration/gitaly/recovery.md
index 1207d7af3e7..7d27a633512 100644
--- a/doc/administration/gitaly/recovery.md
+++ b/doc/administration/gitaly/recovery.md
@@ -74,7 +74,7 @@ Cluster:
### Unavailable repositories
> - From GitLab 13.0 through 14.0, repositories became read-only if they were outdated on the primary but fully up to date on a healthy secondary. `dataloss` sub-command displays read-only repositories by default through these versions.
-> - Since GitLab 14.1, Praefect contains more responsive failover logic which immediately fails over to one of the fully up to date secondaries rather than placing the repository in read-only mode. Since GitLab 14.1, the `dataloss` sub-command displays repositories which are unavailable due to having no fully up to date copies on healthy Gitaly nodes.
+> - From GitLab 14.1, Praefect contains more responsive failover logic which immediately fails over to one of the fully up to date secondaries rather than placing the repository in read-only mode. From GitLab 14.1, the `dataloss` sub-command displays repositories which are unavailable due to having no fully up to date copies on healthy Gitaly nodes.
A repository is unavailable if all of its up to date replicas are unavailable. Unavailable repositories are
not accessible through Praefect to prevent serving stale data that may break automated tooling.
@@ -100,7 +100,7 @@ The following parameters are available:
some assigned copies that are not available.
NOTE:
-`dataloss` is still in [Beta](../../policy/alpha-beta-support.md#beta-features) and the output format is subject to change.
+`dataloss` is still in [Beta](../../policy/alpha-beta-support.md#beta) and the output format is subject to change.
To check for repositories with outdated primaries or for unavailable repositories, run:
@@ -277,15 +277,33 @@ The reconciliation frequency can be changed via the configuration. The value can
Examples:
```ruby
-praefect['reconciliation_scheduling_interval'] = '5m' # the default value
+praefect['configuration'] = {
+ # ...
+ reconciliation: {
+ # ...
+ scheduling_interval: '5m', # the default value
+ },
+}
```
```ruby
-praefect['reconciliation_scheduling_interval'] = '30s' # reconcile every 30 seconds
+praefect['configuration'] = {
+ # ...
+ reconciliation: {
+ # ...
+ scheduling_interval: '30s', # reconcile every 30 seconds
+ },
+}
```
```ruby
-praefect['reconciliation_scheduling_interval'] = '0' # disable the feature
+praefect['configuration'] = {
+ # ...
+ reconciliation: {
+ # ...
+ scheduling_interval: '0', # disable the feature
+ },
+}
```
### Manual reconciliation
@@ -334,16 +352,21 @@ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.t
sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml remove-repository -virtual-storage <virtual-storage> -repository <repository> -apply
```
-- `-virtual-storage` is the virtual storage the repository is located in. Virtual storages are configured in `/etc/gitlab/gitlab.rb` under `praefect['virtual_storages]` and looks like the following:
+- `-virtual-storage` is the virtual storage the repository is located in. Virtual storages are configured in `/etc/gitlab/gitlab.rb` under `praefect['configuration']['virtual_storage]` and looks like the following:
```ruby
- praefect['virtual_storages'] = {
- 'default' => {
- ...
- },
- 'storage-1' => {
- ...
- }
+ praefect['configuration'] = {
+ # ...
+ virtual_storage: [
+ {
+ # ...
+ name: 'default',
+ },
+ {
+ # ...
+ name: 'storage-1',
+ },
+ ],
}
```
@@ -415,16 +438,21 @@ The `track-repository` Praefect sub-command adds repositories on disk to the Pra
sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml track-repository -virtual-storage <virtual-storage> -authoritative-storage <storage-name> -repository <repository> -replicate-immediately
```
-- `-virtual-storage` is the virtual storage the repository is located in. Virtual storages are configured in `/etc/gitlab/gitlab.rb` under `praefect['virtual_storages]` and looks like the following:
+- `-virtual-storage` is the virtual storage the repository is located in. Virtual storages are configured in `/etc/gitlab/gitlab.rb` under `praefect['configuration'][:virtual_storage]` and looks like the following:
```ruby
- praefect['virtual_storages'] = {
- 'default' => {
- ...
- },
- 'storage-1' => {
- ...
- }
+ praefect['configuration'] = {
+ # ...
+ virtual_storage: [
+ {
+ # ...
+ name: 'default',
+ },
+ {
+ # ...
+ name: 'storage-1',
+ },
+ ],
}
```
@@ -473,7 +501,7 @@ The command validates that all entries:
- Are formatted correctly and contain required fields.
- Correspond to a valid Git repository on disk.
-- Are not currently tracked in the Praefect tracking database.
+- Are not tracked in the Praefect tracking database.
If any entry fails these checks, the command aborts prior to attempting to track a repository.
diff --git a/doc/administration/gitaly/reference.md b/doc/administration/gitaly/reference.md
index 1516b82a906..81b3faf859e 100644
--- a/doc/administration/gitaly/reference.md
+++ b/doc/administration/gitaly/reference.md
@@ -8,11 +8,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w
Gitaly is configured via a [TOML](https://github.com/toml-lang/toml)
configuration file. Unlike installations from source, in Omnibus GitLab, you
-would not edit this file directly.
+would not edit this file directly. For Omnibus GitLab installations, the default file location is `/var/opt/gitlab/gitaly/config.toml`.
-The configuration file is passed as an argument to the `gitaly`
-executable. This is usually done by either Omnibus GitLab or your
-[init](https://en.wikipedia.org/wiki/Init) script.
+The configuration file is passed as an argument to the `gitaly` executable, which is usually done by either Omnibus
+GitLab or your [init](https://en.wikipedia.org/wiki/Init) script.
An [example configuration file](https://gitlab.com/gitlab-org/gitaly/blob/master/config.toml.example)
can be found in the Gitaly project.
@@ -42,7 +41,7 @@ prometheus_listen_addr = "localhost:9236"
### Authentication
Gitaly can be configured to reject requests that do not contain a
-specific bearer token in their headers. This is a security measure to
+specific bearer token in their headers, which is a security measure to
be used when serving requests over TCP:
```toml
@@ -70,7 +69,7 @@ Remember to disable `transitioning` when you are done
changing your token settings.
All authentication attempts are counted in Prometheus under
-the [`gitaly_authentications_total` metric](monitoring.md#useful-queries).
+the [`gitaly_authentications_total` metric](monitoring.md#queries).
### TLS
@@ -156,36 +155,6 @@ Prometheus query to see the hit rate:
sum(rate(gitaly_catfile_cache_total{type="hit"}[5m])) / sum(rate(gitaly_catfile_cache_total{type=~"(hit)|(miss)"}[5m]))
```
-### `gitaly-ruby`
-
-A Gitaly process uses one or more `gitaly-ruby` helper processes to
-execute RPCs implemented in Ruby instead of Go. The `[gitaly-ruby]`
-section of the configuration file contains settings for these helper processes.
-
-These processes are known to occasionally suffer from memory leaks.
-Gitaly restarts its `gitaly-ruby` helpers when their memory exceeds the
-`max_rss` limit.
-
-| Name | Type | Required | Description |
-| ---- | ---- | -------- | ----------- |
-| `dir` | string | yes | Path to where `gitaly-ruby` is installed (needed to boot the process).|
-| `max_rss` | integer | no | Resident set size limit that triggers a `gitaly-ruby` restart, in bytes. Default is `200000000` (200 MB). |
-| `graceful_restart_timeout` | string | no | Grace period before a `gitaly-ruby` process is forcibly terminated after exceeding `max_rss`. Default is `10m` (10 minutes).|
-| `restart_delay` | string | no |Time that `gitaly-ruby` memory must remain high before a restart. Default is `5m` (5 minutes).|
-| `num_workers` | integer | no |Number of `gitaly-ruby` worker processes. Try increasing this number in case of `ResourceExhausted` errors. Default is `2`, minimum is `2`.|
-| `linguist_languages_path` | string | no | Override for dynamic `languages.json` discovery. Defaults to an empty string (use of dynamic discovery).|
-
-Example:
-
-```toml
-[gitaly-ruby]
-dir = "/home/git/gitaly/ruby"
-max_rss = 200000000
-graceful_restart_timeout = "10m"
-restart_delay = "5m"
-num_workers = 2
-```
-
### GitLab Shell
For historical reasons
@@ -233,7 +202,6 @@ The following values configure logging in Gitaly under the `[logging]` section.
| `level` | string | no | Log level: `debug`, `info`, `warn`, `error`, `fatal`, or `panic`. Default: `info`. |
| `sentry_dsn` | string | no | Sentry DSN (Data Source Name) for exception monitoring. |
| `sentry_environment` | string | no | [Sentry Environment](https://docs.sentry.io/product/sentry-basics/environments/) for exception monitoring. |
-| `ruby_sentry_dsn` | string | no | Sentry DSN (Data Source Name) for `gitaly-ruby` exception monitoring. |
While the main Gitaly application logs go to `stdout`, there are some extra log
files that go to a configured directory, like the GitLab Shell logs.
diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md
index 5f05b6322b0..dbbd348556c 100644
--- a/doc/administration/gitaly/troubleshooting.md
+++ b/doc/administration/gitaly/troubleshooting.md
@@ -66,7 +66,7 @@ for details.
### Client side gRPC logs
Gitaly uses the [gRPC](https://grpc.io/) RPC framework. The Ruby gRPC
-client has its own log file which may contain useful information when
+client has its own log file which may contain helpful information when
you are seeing Gitaly errors. You can control the log level of the
gRPC client with the `GRPC_LOG_LEVEL` environment variable. The
default level is `WARN`.
@@ -85,7 +85,7 @@ check for an SSL or TLS problem:
```
Check whether `Verify return code` field indicates a
-[known Omnibus GitLab configuration problem](https://docs.gitlab.com/omnibus/settings/ssl.html).
+[known Omnibus GitLab configuration problem](https://docs.gitlab.com/omnibus/settings/ssl/index.html).
If `openssl` succeeds but `gitlab-rake gitlab:gitaly:check` fails,
check [certificate requirements](configure_gitaly.md#certificate-requirements) for Gitaly.
@@ -123,27 +123,6 @@ sudo cat /proc/$PID/environ | tr '\0' '\n' | grep ^CORRELATION_ID=
This method isn't reliable for `git cat-file` processes, because Gitaly
internally pools and re-uses those across RPCs.
-### Observing `gitaly-ruby` traffic
-
-[`gitaly-ruby`](configure_gitaly.md#gitaly-ruby) is an internal implementation detail of Gitaly,
-so, there's not that much visibility into what goes on inside
-`gitaly-ruby` processes.
-
-If you have Prometheus set up to scrape your Gitaly process, you can see
-request rates and error codes for individual RPCs in `gitaly-ruby` by
-querying `grpc_client_handled_total`.
-
-All gRPC calls made by `gitaly-ruby` itself are internal calls from the main Gitaly process to one of its `gitaly-ruby`
-sidecars.
-
-Assuming your `grpc_client_handled_total` counter only observes Gitaly,
-the following query shows you RPCs are (most likely) internally
-implemented as calls to `gitaly-ruby`:
-
-```prometheus
-sum(rate(grpc_client_handled_total[5m])) by (grpc_method) > 0
-```
-
### Repository changes fail with a `401 Unauthorized` error
If you run Gitaly on its own server and notice these conditions:
@@ -345,7 +324,7 @@ You might see the following in Gitaly and Praefect logs:
}
```
-This is a gRPC call
+This information in the logs is a gRPC call
[error response code](https://grpc.github.io/grpc/core/md_doc_statuscodes.html).
If this error occurs, even though
@@ -358,7 +337,7 @@ server to keep them synchronized.
### Gitaly not listening on new address after reconfiguring
-When updating the `gitaly['listen_addr']` or `gitaly['prometheus_listen_addr']` values, Gitaly may
+When updating the `gitaly['configuration'][:listen_addr]` or `gitaly['configuration'][:prometheus_listen_addr]` values, Gitaly may
continue to listen on the old address after a `sudo gitlab-ctl reconfigure`.
When this occurs, run `sudo gitlab-ctl restart` to resolve the issue. This should no longer be
@@ -401,6 +380,12 @@ push, which causes a significant delay.
If Git pushes are too slow when Dynatrace is enabled, disable Dynatrace.
+### `gitaly check` fails with `401` status code
+
+`gitaly check` can fail with `401` status code if Gitaly can't access the internal GitLab API.
+
+One way to resolve this is to make sure the entry is correct for the GitLab internal API URL configured in `gitlab.rb` with `gitlab_rails['internal_api_url']`.
+
## Gitaly fails to fork processes stored on `noexec` file systems
Because of changes [introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5999) in GitLab 14.10, applying the `noexec` option to a mount
@@ -492,7 +477,7 @@ in sync so the token check succeeds.
This check helps identify the root cause of `permission denied`
[errors being logged by Praefect](#permission-denied-errors-appearing-in-gitaly-or-praefect-logs-when-accessing-repositories).
-For offline environments where access to public [`pool.ntp.org`](https://pool.ntp.org) servers is not possible, the Praefect `check` sub-command fails this
+For offline environments where access to public `pool.ntp.org` servers is not possible, the Praefect `check` sub-command fails this
check with an error message similar to:
```plaintext
@@ -514,9 +499,9 @@ Here are common errors and potential causes:
- 500 response code
- `ActionView::Template::Error (7:permission denied)`
- - `praefect['auth_token']` and `gitlab_rails['gitaly_token']` do not match on the GitLab server.
+ - `praefect['configuration'][:auth][:token]` and `gitlab_rails['gitaly_token']` do not match on the GitLab server.
- `Unable to save project. Error: 7:permission denied`
- - Secret token in `praefect['storage_nodes']` on GitLab server does not match the
+ - Secret token in `praefect['configuration'][:virtual_storage]` on GitLab server does not match the
value in `gitaly['auth_token']` on one or more Gitaly servers.
- 503 response code
- `GRPC::Unavailable (14:failed to connect to all addresses)`
@@ -530,7 +515,7 @@ Here are common errors and potential causes:
Some common reasons for the Praefect database to experience elevated CPU usage include:
- Prometheus metrics scrapes [running an expensive query](https://gitlab.com/gitlab-org/gitaly/-/issues/3796). If you have GitLab 14.2
- or above, set `praefect['separate_database_metrics'] = true` in `gitlab.rb`.
+ or above, set `praefect['configuration'][:prometheus_exclude_database_from_default_metrics] = true` in `gitlab.rb`.
- [Read distribution caching](praefect.md#reads-distribution-caching) is disabled, increasing the number of queries made to the
database when user traffic is high. Ensure read distribution caching is enabled.
@@ -544,9 +529,8 @@ To determine the primary node of a repository:
- With legacy election strategies in GitLab 13.12 and earlier, the primary was the same for all repositories in a virtual storage.
To determine the current primary Gitaly node for a specific virtual storage:
- - Use the `Shard Primary Election` [Grafana chart](praefect.md#grafana) on the
+ - (Recommended) Use the `Shard Primary Election` [Grafana chart](praefect.md#grafana) on the
[`Gitlab Omnibus - Praefect` dashboard](https://gitlab.com/gitlab-org/grafana-dashboards/-/blob/master/omnibus/praefect.json).
- This is recommended.
- If you do not have Grafana set up, use the following command on each host of each
Praefect node:
@@ -650,7 +634,7 @@ If the supplied value for `-virtual-storage` is incorrect, the command returns t
get metadata: rpc error: code = NotFound desc = repository not found
```
-The documented examples specify `-virtual-storage default`. Check the Praefect server setting `praefect['virtual_storages']` in `/etc/gitlab/gitlab.rb`.
+The documented examples specify `-virtual-storage default`. Check the Praefect server setting `praefect['configuration'][:virtual_storage]` in `/etc/gitlab/gitlab.rb`.
### Check that repositories are in sync
@@ -669,7 +653,7 @@ However, the Praefect database tables are not created on initial reconfigure and
errors that relations do not exist if either:
- The `gitlab-ctl reconfigure` command isn't executed.
-- There are errors during the execution.
+- Errors occur during the execution.
For example:
@@ -693,7 +677,7 @@ praefect sql-migrate: OK (applied 21 migrations)
This indicates that the virtual storage name used in the
[Praefect configuration](praefect.md#praefect) does not match the storage name used in
-[`git_data_dirs` setting](praefect.md#gitaly) for GitLab.
+[`gitaly['configuration'][:storage][<index>][:name]` setting](praefect.md#gitaly) for GitLab.
Resolve this by matching the virtual storage names used in Praefect and GitLab configuration.
@@ -715,9 +699,13 @@ Possible solutions:
- Provision larger VMs to gain access to larger network traffic allowances.
- Use your cloud service's monitoring and logging to check that the Praefect nodes are not exhausting their traffic allowances.
+### `gitlab-ctl reconfigure` fails with error: `STDOUT: praefect: configuration error: error reading config file: toml: cannot store TOML string into a Go int`
+
+This error occurs when `praefect['database_port']` or `praefect['database_direct_port']` are configured as a string instead of an integer.
+
## Profiling Gitaly
-Gitaly exposes several of the Golang built-in performance profiling tools on the Prometheus listen port. For example, if Prometheus is listening
+Gitaly exposes several of the Go built-in performance profiling tools on the Prometheus listen port. For example, if Prometheus is listening
on port `9236` of the GitLab server:
- Get a list of running `goroutines` and their backtraces:
diff --git a/doc/administration/housekeeping.md b/doc/administration/housekeeping.md
index d58989db70c..3a2b3657145 100644
--- a/doc/administration/housekeeping.md
+++ b/doc/administration/housekeeping.md
@@ -166,6 +166,10 @@ You can change this default in Gitaly configuration. The following snippet
enables daily background repository maintenance starting at 23:00 for 1 hour
for the `default` storage:
+::Tabs
+
+:::TabTitle Self-compiled (source)
+
```toml
[daily_maintenance]
start_hour = 23
@@ -182,6 +186,33 @@ maintenance:
disabled = true
```
+:::TabTitle Linux package (Omnibus)
+
+```ruby
+gitaly['configuration'] = {
+ daily_maintenance: {
+ disabled: false,
+ start_hour: 23,
+ start_minute: 00,
+ duration: '1h',
+ storages: ['default'],
+ },
+}
+```
+
+Use the following snippet to completely disable background repository
+maintenance:
+
+```ruby
+gitaly['configuration'] = {
+ daily_maintenance: {
+ disabled: true,
+ },
+}
+```
+
+::EndTabs
+
## Object pool repositories
Object pool repositories are used by GitLab to deduplicate objects across forks
diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md
index ea051e2067d..3b4eaeb161c 100644
--- a/doc/administration/incoming_email.md
+++ b/doc/administration/incoming_email.md
@@ -68,8 +68,7 @@ this method only supports replies, and not the other features of [incoming email
## Accepted headers
-> - Accepting `Received` headers [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/81489) in GitLab 14.9 [with a flag](feature_flags.md) named `use_received_header_for_incoming_emails`. Enabled by default.
-> - Accepting `Received` headers: [feature flag](feature_flags.md) named `use_received_header_for_incoming_emails` [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/362596) in GitLab 14.1.
+> Accepting `Received` headers [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/81489) in GitLab 14.9.
Email is processed correctly when a configured email address is present in one of the following headers
(sorted in the order they are checked):
diff --git a/doc/administration/index.md b/doc/administration/index.md
index 9f4477bcbf7..ded6eabdba7 100644
--- a/doc/administration/index.md
+++ b/doc/administration/index.md
@@ -7,235 +7,14 @@ description: 'Learn how to install, configure, update, and maintain your GitLab
# Administer GitLab **(FREE SELF)**
-If you use GitLab.com, only GitLab team members have access to administration tools and settings.
-If you use a self-managed GitLab instance, learn how to administer it.
+Learn how to administer a self-managed GitLab instance.
-Only administrator users can access GitLab administration tools and settings.
+- [Get started](../administration/get_started.md)
+- [All feature flags](../user/feature_flags.md)
+- [Configure your installation](../administration/configure.md)
+- [Configure GitLab Dedicated](../administration/dedicated/index.md)
+- [Maintain your installation](../administration/operations/index.md)
+- [Secure your installation](../security/index.md)
+- [Administer users](../user/profile/account/create_accounts.md)
-## Available distributions
-
-GitLab has two product distributions available through [different subscriptions](https://about.gitlab.com/pricing/):
-
-- The open source [GitLab Community Edition (CE)](https://gitlab.com/gitlab-org/gitlab-foss).
-- The open core [GitLab Enterprise Edition (EE)](https://gitlab.com/gitlab-org/gitlab).
-
-You can [install either GitLab CE or GitLab EE](https://about.gitlab.com/install/ce-or-ee/).
-However, the features you have access to depend on your chosen subscription.
-
-GitLab Community Edition installations have access only to Free features.
-
-## Installing and maintaining GitLab
-
-Learn how to install, configure, update, and maintain your GitLab instance.
-
-### Installing GitLab
-
-- [Install](../install/index.md): Requirements, directory structures, and installation methods.
-- [Reference architectures](reference_architectures/index.md): Add additional resources to support more users.
-- [Installing GitLab on Amazon Web Services (AWS)](../install/aws/index.md): Set up GitLab on Amazon AWS.
-- [Geo](geo/index.md): Replicate your GitLab instance to other geographic locations as a read-only fully operational version.
-- [Disaster Recovery](geo/disaster_recovery/index.md): Quickly fail-over to a different site with minimal effort in a disaster situation.
-- [Add License](../user/admin_area/license.md): Add a license at install time to unlock features that are in paid tiers of GitLab.
-
-### Configuring GitLab
-
-- [Adjust your instance's time zone](timezone.md): Customize the default time zone of GitLab.
-- [System hooks](system_hooks.md): Notifications when users, projects and keys are changed.
-- [Security](../security/index.md): Learn what you can do to further secure your GitLab instance.
-- [Usage statistics, version check, and Service Ping](../user/admin_area/settings/usage_statistics.md): Enable or disable information about your instance to be sent to GitLab, Inc.
-- [Global user settings](user_settings.md): Configure instance-wide user permissions.
-- [Polling](polling.md): Configure how often the GitLab UI polls for updates.
-- [GitLab Pages configuration](pages/index.md): Enable and configure GitLab Pages.
-- [GitLab Pages configuration for GitLab source installations](pages/source.md):
- Enable and configure GitLab Pages on [source installations](../install/installation.md#installation-from-source).
-- [Uploads administration](uploads.md): Configure GitLab uploads storage.
-- [Environment variables](environment_variables.md): Supported environment
- variables that can be used to override their default values to configure
- GitLab.
-- [File hooks](file_hooks.md): With custom file hooks, GitLab administrators can
- introduce custom integrations without modifying GitLab source code.
-- [Enforcing Terms of Service](../user/admin_area/settings/terms.md)
-- [Customer experience improvement and third-party offers](../user/admin_area/settings/third_party_offers.md)
-- [Compliance](compliance.md): A collection of features from across the
- application that you may configure to help ensure that your GitLab instance
- and DevOps workflow meet compliance standards.
-- [Diff limits](../user/admin_area/diff_limits.md): Configure the diff rendering
- size limits of branch comparison pages.
-- [Merge request diffs storage](merge_request_diffs.md): Configure merge
- requests diffs external storage.
-- [Broadcast Messages](../user/admin_area/broadcast_messages.md): Send messages
- to GitLab users through the UI.
-- [Elasticsearch](../integration/advanced_search/elasticsearch.md): Enable Elasticsearch to
- empower Advanced Search. Use when you deal with a huge amount of data.
-- [External Classification Policy Authorization](../user/admin_area/settings/external_authorization.md)
-- [Add a license](../user/admin_area/license.md): Add a license to unlock
- features that are in paid tiers of GitLab.
-- [Admin Area](../user/admin_area/index.md): for self-managed instance-wide
- configuration and maintenance.
-- [S/MIME Signing](smime_signing_email.md): how to sign all outgoing notification
- emails with S/MIME.
-- [Enabling and disabling features flags](feature_flags.md): how to enable and
- disable GitLab features deployed behind feature flags.
-- [Application settings cache expiry interval](application_settings_cache.md)
-- [Database Load Balancing](postgresql/database_load_balancing.md): Distribute database queries among multiple database servers.
-- [Omnibus support for log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-shipping-gitlab-enterprise-edition-only).
-
-#### Customizing GitLab appearance
-
-- [Header logo](../user/admin_area/appearance.md#top-bar): Change the logo on all pages and email headers.
-- [Favicon](../user/admin_area/appearance.md#favicon): Change the default favicon to your own logo.
-- [Branded login page](../user/admin_area/appearance.md#sign-in--sign-up-pages): Customize the login page with your own logo, title, and description.
-- ["New Project" page](../user/admin_area/appearance.md#new-project-pages): Customize the text to be displayed on the page that opens whenever your users create a new project.
-- [Additional custom email text](../user/admin_area/settings/email.md#custom-additional-text): Add additional custom text to emails sent from GitLab.
-
-### Maintaining GitLab
-
-- [Rake tasks](../raketasks/index.md): Perform various tasks for maintenance, backups, automatic webhooks setup, and more.
- - [Backup and restore](../raketasks/backup_restore.md): Backup and restore your GitLab instance.
-- [Operations](operations/index.md): Keeping GitLab up and running (clean up Redis sessions, moving repositories, Sidekiq MemoryKiller, Puma).
-- [Restart GitLab](restart_gitlab.md): Learn how to restart GitLab and its components.
-- [Invalidate Markdown cache](invalidate_markdown_cache.md): Invalidate any cached Markdown.
-- [Instance review](instance_review.md): Request a free review of your GitLab instance.
-
-#### Updating GitLab
-
-- [GitLab versions and maintenance policy](../policy/maintenance.md): Understand GitLab versions and releases (Major, Minor, Patch, Security), as well as update recommendations.
-- [GitLab in maintenance mode](maintenance_mode/index.md): Put GitLab in maintenance mode.
-- [Update GitLab](../update/index.md): Update guides to upgrade your installation to a new version.
-- [Upgrading without downtime](../update/index.md#upgrading-without-downtime): Upgrade to a newer major, minor, or patch version of GitLab without taking your GitLab instance offline.
-
-### Upgrading or downgrading GitLab
-
-- [Upgrade from GitLab CE to GitLab EE](../update/index.md#upgrading-between-editions): Learn how to upgrade GitLab Community Edition to GitLab Enterprise Editions.
-- [Downgrade from GitLab EE to GitLab CE](../downgrade_ee_to_ce/index.md): Learn how to downgrade GitLab Enterprise Editions to Community Edition.
-
-### GitLab platform integrations
-
-- [Mattermost](../integration/mattermost/index.md): Integrate with [Mattermost](https://mattermost.com), an open source, private cloud workplace for web messaging.
-- [PlantUML](integration/plantuml.md): Create diagrams in AsciiDoc and Markdown documents
- created in snippets, wikis, and repositories.
-- [Web terminals](integration/terminal.md): Provide terminal access to your applications deployed to Kubernetes from GitLab CI/CD [environments](../ci/environments/index.md#web-terminals-deprecated).
-
-## User settings and permissions
-
-- [Creating users](../user/profile/account/create_accounts.md): Create users manually or through authentication integrations.
-- [Libravatar](libravatar.md): Use Libravatar instead of Gravatar for user avatars.
-- [Sign-up restrictions](../user/admin_area/settings/sign_up_restrictions.md): block email addresses of specific domains, or allow only specific domains.
-- [Admin mode](../user/admin_area/settings/sign_in_restrictions.md#admin-mode): require that administrators authenticate separately to use administrative access, like `sudo`.
-- [Access restrictions](../user/admin_area/settings/visibility_and_access_controls.md#configure-enabled-git-access-protocols): Define which Git access protocols can be used to talk to GitLab (SSH, HTTP, HTTPS).
-- [Authentication and Authorization](auth/index.md): Configure external authentication with LDAP, SAML, CAS, and additional providers.
- - [Sync LDAP](auth/ldap/index.md)
- - [Kerberos authentication](../integration/kerberos.md)
- - See also other [authentication](../topics/authentication/index.md#gitlab-administrators) topics (for example, enforcing 2FA).
-- [Email users](../user/admin_area/email_from_gitlab.md): Email GitLab users from GitLab.
-- [User Cohorts](../user/admin_area/user_cohorts.md): Display the monthly cohorts of new users and their activities over time.
-- [Audit events](audit_events.md): View the changes made on the GitLab server for:
- - Groups and projects.
- - Instances.
-- [Auditor users](auditor_users.md): Users with read-only access to all projects, groups, and other resources on the GitLab instance.
-- [Incoming email](incoming_email.md): Configure incoming emails to allow
- users to [reply by email](reply_by_email.md), create [issues by email](../user/project/issues/create_issues.md#by-sending-an-email) and
- [merge requests by email](../user/project/merge_requests/creating_merge_requests.md#by-sending-an-email), and to enable [Service Desk](../user/project/service_desk.md).
- - [Postfix for incoming email](reply_by_email_postfix_setup.md): Set up a
- basic Postfix mail server with IMAP authentication on Ubuntu for incoming
- emails.
-- [Abuse reports](../user/admin_area/review_abuse_reports.md): View and resolve abuse reports from your users.
-- [Credentials Inventory](../user/admin_area/credentials_inventory.md): With Credentials inventory, GitLab administrators can keep track of the credentials used by their users in their GitLab self-managed instance.
-
-## Project settings
-
-- [Issue closing pattern](issue_closing_pattern.md): Customize how to close an issue from commit messages.
-- [Gitaly](gitaly/index.md): Configuring Gitaly, the Git repository storage service for GitLab.
-- [Default labels](../user/admin_area/labels.md): Create labels that are automatically added to every new project.
-- [Restrict the use of public or internal projects](../user/public_access.md#restrict-use-of-public-or-internal-projects): Restrict the use of visibility levels for users when they create a project or a snippet.
-- [Custom project templates](../user/admin_area/custom_project_templates.md): Configure a set of projects to be used as custom templates when creating a new project.
-
-## Package Registry administration
-
-- [Container Registry](packages/container_registry.md): Configure GitLab to act as a registry for containers.
-- [Package Registry](packages/index.md): Enable GitLab to act as a registry for packages.
-- [Dependency Proxy](packages/dependency_proxy.md): Configure the Dependency Proxy, a local proxy for frequently used upstream images/packages.
-
-### Repository settings
-
-- [Repository checks](repository_checks.md): Check your repository for data corruption.
-- [Repository storage paths](repository_storage_paths.md): Manage the paths used to store repositories.
-- [Repository storage types](repository_storage_types.md): Information about the different repository storage types.
-- [Repository storage Rake tasks](raketasks/storage.md): A collection of Rake tasks to list and migrate existing projects and attachments associated with it from Legacy storage to Hashed storage.
-- [Limit repository size](../user/admin_area/settings/account_and_limit_settings.md): Set a hard limit for your repositories' size.
-- [Static objects external storage](static_objects_external_storage.md): Set external storage for static objects in a repository.
-
-## CI/CD settings
-
-- [Enable or disable GitLab CI/CD](cicd.md#disable-gitlab-cicd-in-new-projects): Set new projects in your instance to have GitLab CI/CD enabled or disabled by default.
-- [GitLab CI/CD administration settings](../user/admin_area/settings/continuous_integration.md): Enable or disable Auto DevOps site-wide and define the artifacts' max size and expiration time.
-- [External Pipeline Validation](external_pipeline_validation.md): Enable, disable, and configure external pipeline validation.
-- [Job artifacts](job_artifacts.md): Enable, disable, and configure job artifacts (a set of files and directories which are outputted by a job when it completes successfully).
-- [Job logs](job_logs.md): Information about the job logs.
-- [Register runners](../ci/runners/runners_scope.md): Learn how to register and configure runners.
-- [Shared runners quota of CI/CD minutes](../ci/pipelines/cicd_minutes.md): Limit the usage of CI/CD minutes for shared runners.
-- [Enable or disable Auto DevOps](../topics/autodevops/index.md#enable-or-disable-auto-devops): Enable or disable Auto DevOps for your instance.
-
-## Snippet settings
-
-- [Setting snippet content size limit](snippets/index.md): Set a maximum content size limit for snippets.
-
-## Wiki settings
-
-- [Setting wiki page content size limit](wikis/index.md): Set a maximum content size limit for wiki pages.
-
-## Git configuration options
-
-- [Git server hooks](server_hooks.md): Git server hooks (on the file system) for when webhooks aren't enough. Previously called server hooks.
-- [Git LFS configuration](lfs/index.md): Learn how to configure LFS for GitLab.
-- [Housekeeping](housekeeping.md): Keep your Git repositories tidy and fast.
-- [Configuring Git Protocol v2](git_protocol.md): Git protocol version 2 support.
-
-## Monitoring GitLab
-
-- [Monitoring GitLab](monitoring/index.md):
- - [Monitoring uptime](../user/admin_area/monitoring/health_check.md): Check the server status using the health check endpoint.
- - [IP allowlist](monitoring/ip_allowlist.md): Monitor endpoints that provide health check information when probed.
- - [Monitoring GitHub imports](monitoring/github_imports.md): The GitLab GitHub Importer displays Prometheus metrics to monitor the health and progress of the importer.
-
-### Performance Monitoring
-
-- [GitLab Performance Monitoring](monitoring/performance/index.md):
- - [Enable Performance Monitoring](monitoring/performance/gitlab_configuration.md): Enable GitLab Performance Monitoring.
- - [GitLab performance monitoring with Prometheus](monitoring/prometheus/index.md): Configure GitLab and Prometheus for measuring performance metrics.
- - [GitLab performance monitoring with Grafana](monitoring/performance/grafana_configuration.md): Configure GitLab to visualize time series metrics through graphs and dashboards.
- - [Performance Bar](monitoring/performance/performance_bar.md): Get performance information for the current page.
-
-## Troubleshooting
-
-- [Log system](logs/index.md): Where to look for logs.
-- [Sidekiq Troubleshooting](sidekiq/sidekiq_troubleshooting.md): Debug when Sidekiq appears hung and is not processing jobs.
-- [Navigating GitLab via Rails console](operations/rails_console.md)
-- [GitLab application limits](instance_limits.md)
-- [Responding to security incidents](../security/responding_to_security_incidents.md)
-
-### Support Team Documentation
-
-The GitLab Support Team has collected a lot of information about troubleshooting GitLab.
-The following documents are used by the Support Team or by customers
-with direct guidance from a Support Team member. GitLab administrators may find the
-information useful for troubleshooting. However, if you are experiencing trouble with your
-GitLab instance, you should check your [support options](https://about.gitlab.com/support/)
-before referring to these documents.
-
-WARNING:
-The commands in the following documentation might result in data loss or
-other damage to a GitLab instance. They should be used only by experienced administrators
-who are aware of the risks.
-
-- [Diagnostics tools](troubleshooting/diagnostics_tools.md)
-- [Linux commands](troubleshooting/linux_cheat_sheet.md)
-- [Troubleshooting Kubernetes](https://docs.gitlab.com/charts/troubleshooting/kubernetes_cheat_sheet.html)
-- [Troubleshooting PostgreSQL](troubleshooting/postgresql.md)
-- [Guide to test environments](troubleshooting/test_environments.md) (for Support Engineers)
-- [Troubleshooting SSL](troubleshooting/ssl.md)
-- Related links:
- - [GitLab Developer Documentation](../development/index.md)
- - [Repairing and recovering broken Git repositories](https://git.seveas.net/repairing-and-recovering-broken-git-repositories.html)
- - [Testing with OpenSSL](https://www.feistyduck.com/library/openssl-cookbook/online/testing-with-openssl/index.html)
- - [`strace` zine](https://wizardzines.com/zines/strace/)
+Only GitLab team members have access to administration tools and settings on GitLab.com.
diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md
index b1d97fd16a9..03c7c51251b 100644
--- a/doc/administration/instance_limits.md
+++ b/doc/administration/instance_limits.md
@@ -126,7 +126,7 @@ Read more about [import/export rate limits](../user/admin_area/settings/import_e
Limit the maximum daily member invitations allowed per group hierarchy.
-- GitLab.com: Free members may invite 20 members per day.
+- GitLab.com: Free members may invite 20 members per day, Premium trial and Ultimate trial members may invite 50 members per day.
- Self-managed: Invites are not limited.
### Webhook rate limit
@@ -157,16 +157,17 @@ Set the limit to `0` to disable it.
### Search rate limit
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/80631) in GitLab 14.9.
-> - [Changed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/104208) to include issue, merge request, and epic searches to the rate limit in GitLab 15.9.
+> - [Changed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/104208) in GitLab 15.9 to include issue, merge request, and epic searches in the rate limit.
+> - [Changed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/118525) in GitLab 16.0 to apply rate limits to [search scopes](../user/search/index.md#global-search-scopes) for authenticated requests.
This setting limits search requests as follows:
| Limit | Default (requests per minute) |
|-------------------------|-------------------------------|
-| Authenticated user | 30 |
-| Unauthenticated user | 10 |
+| Authenticated user | 300 |
+| Unauthenticated user | 100 |
-Depending on the number of enabled [scopes](../user/search/index.md#global-search-scopes), a global search request can consume two to seven requests per minute. You may want to disable one or more scopes to use fewer requests. Search requests that exceed the search rate limit per minute return the following error:
+Search requests that exceed the search rate limit per minute return the following error:
```plaintext
This endpoint has been requested too many times. Try again later.
@@ -290,7 +291,7 @@ Plan.default.actual_limits.update!(group_hooks: 100)
Set the limit to `0` to disable it.
-The default maximum number of webhooks is `100` per project, `50` per group.
+The default maximum number of webhooks is `100` per project and `50` per group. Webhooks in a child group do not count towards the webhook limit of their parent group.
For GitLab.com, see the [webhook limits for GitLab.com](../user/gitlab_com/index.md#webhooks).
@@ -430,38 +431,6 @@ Plan.default.actual_limits.update!(ci_active_jobs: 500)
Set the limit to `0` to disable it.
-### Number of pipelines running concurrently
-
-> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32823) in GitLab 12.5.
-
-The total number of pipelines running concurrently can be limited per project.
-When enabled, the limit is checked each time a new pipeline is created.
-Without a concurrent pipelines limit, a sudden flood of triggered pipelines could
-overwhelm the instance resources.
-
-If a new pipeline would cause the total number of pipelines to exceed the limit,
-the pipeline fails with a `The pipeline activity limit was exceeded.` error.
-
-On [GitLab Premium](https://about.gitlab.com/pricing/) self-managed or
-higher installations, this limit is defined under a `default` plan that affects all
-projects. This limit is disabled (`0`) by default. GitLab SaaS subscribers have different
-limits [defined per plan](../user/gitlab_com/index.md#gitlab-cicd), and they affect
-all projects under that plan.
-
-To set this limit for a self-managed installation, enable the **Maximum number of active pipelines per project**
-[setting in the Admin Area](../user/admin_area/settings/continuous_integration.md#set-cicd-limits).
-
-Alternatively, you can run the following in the [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session):
-
-```ruby
-# If limits don't exist for the default plan, you can create one with:
-# Plan.default.create_limits!
-
-Plan.default.actual_limits.update!(ci_active_pipelines: 100)
-```
-
-Set the limit to `0` to disable it.
-
### Maximum time jobs can run
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/16777) in GitLab 12.3.
@@ -509,10 +478,10 @@ checked each time a new subscription is created.
If a new subscription would cause the total number of subscription to exceed the
limit, the subscription is considered invalid.
-- GitLab SaaS subscribers have different limits [defined per plan](../user/gitlab_com/index.md#gitlab-cicd),
+- On GitLab SaaS: Limits are [defined per plan](../user/gitlab_com/index.md#gitlab-cicd),
and they affect all projects under that plan.
-- On [GitLab Premium](https://about.gitlab.com/pricing/) self-managed
- or higher installations, this limit is defined under a `default` plan that
+- On self-managed: On [GitLab Premium or Ultimate](https://about.gitlab.com/pricing/),
+ this limit is defined under a `default` plan that
affects all projects. By default, there is a limit of `2` subscriptions.
To set this limit for a self-managed installation, run the following in the
@@ -632,6 +601,42 @@ To update this limit to a new value on a self-managed installation, run the foll
Plan.default.actual_limits.update!(ci_instance_level_variables: 30)
```
+### Number of group level variables
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/362227) in GitLab 15.7.
+
+The total number of group level CI/CD variables is limited at the instance level.
+This limit is checked each time a new group level variable is created. If a new variable
+would cause the total number of variables to exceed the limit, the new variable is not created.
+
+On self-managed instances this limit is defined for the `default` plan. By default,
+this limit is set to `30000`.
+
+To update this limit to a new value on a self-managed installation, run the following in the
+[GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session):
+
+```ruby
+Plan.default.actual_limits.update!(group_ci_variables: 40000)
+```
+
+### Number of project level variables
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/362227) in GitLab 15.7.
+
+The total number of project level CI/CD variables is limited at the instance level.
+This limit is checked each time a new project level variable is created. If a new variable
+would cause the total number of variables to exceed the limit, the new variable is not created.
+
+On self-managed instances this limit is defined for the `default` plan. By default,
+this limit is set to `8000`.
+
+To update this limit to a new value on a self-managed installation, run the following in the
+[GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session):
+
+```ruby
+Plan.default.actual_limits.update!(project_ci_variables: 10000)
+```
+
### Maximum file size per type of artifact
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37226) in GitLab 13.3.
@@ -723,22 +728,23 @@ GitLab checks these limits against runners that have been active in the last 3 m
A runner's registration fails if it exceeds the limit for the scope determined by the runner registration token.
If the limit value is set to zero, the limit is disabled.
-- GitLab SaaS subscribers have different limits defined per plan, affecting all projects using that plan.
-- Self-managed GitLab Premium and Ultimate limits are defined by a default plan that affects all projects:
+GitLab SaaS subscribers have different limits defined per plan, affecting all projects using that plan.
- | Runner scope | Default value |
- |---------------------------------------------|---------------|
- | `ci_registered_group_runners` | 1000 |
- | `ci_registered_project_runners` | 1000 |
+Self-managed GitLab Premium and Ultimate limits are defined by a default plan that affects all projects:
- To update these limits, run the following in the
- [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session):
+| Runner scope | Default value |
+|---------------------------------------------|---------------|
+| `ci_registered_group_runners` | 1000 |
+| `ci_registered_project_runners` | 1000 |
+
+To update these limits, run the following in the
+[GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session):
- ```ruby
- # Use ci_registered_group_runners or ci_registered_project_runners
- # depending on desired scope
- Plan.default.actual_limits.update!(ci_registered_project_runners: 100)
- ```
+```ruby
+# Use ci_registered_group_runners or ci_registered_project_runners
+# depending on desired scope
+Plan.default.actual_limits.update!(ci_registered_project_runners: 100)
+```
### Maximum file size for job logs
@@ -789,7 +795,7 @@ You can change these limits in the [GitLab Rails console](operations/rails_conso
The `max_yaml_size_bytes` value is not directly tied to the size of the YAML file,
but rather the memory allocated for the relevant objects.
-- To update the maximum YAML depth, update `max_yaml_depth` with the new value in megabytes:
+- To update the maximum YAML depth, update `max_yaml_depth` with the new value in number of lines:
```ruby
ApplicationSetting.update(max_yaml_depth: 125)
@@ -922,7 +928,7 @@ Reports that go over the 20 MB limit aren't loaded. Affected reports:
- [CI/CD parameter `artifacts:expose_as`](../ci/yaml/index.md#artifactsexpose_as)
- [Unit test reports](../ci/testing/unit_test_reports.md)
-## Advanced Search limits
+## Advanced search limits
### Maximum file size indexed
@@ -945,7 +951,7 @@ is pre-allocated during indexing.
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/201826) in GitLab 12.8.
-You can set a limit on the content of text fields indexed for Advanced Search.
+You can set a limit on the content of text fields indexed for advanced search.
Setting a maximum helps to reduce the load of the indexing processes. If any
text field exceeds this limit, then the text is truncated to this number of
characters. The rest of the text is not indexed, and not searchable.
diff --git a/doc/administration/integration/kroki.md b/doc/administration/integration/kroki.md
index 081c4d8a08c..f90458200b3 100644
--- a/doc/administration/integration/kroki.md
+++ b/doc/administration/integration/kroki.md
@@ -9,12 +9,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/241744) in GitLab 13.7.
> - Support for reStructuredText and Textile documents [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/324766) in GitLab 13.12.
-When [Kroki](https://kroki.io) integration is enabled and configured in
-GitLab, you can use it to create diagrams in AsciiDoc, Markdown, reStructuredText, and Textile documents.
+With the [Kroki](https://kroki.io) integration,
+you can create diagrams in AsciiDoc, Markdown, reStructuredText, and Textile.
-## Kroki Server
+## Kroki server
-When Kroki is enabled, GitLab sends diagrams to an instance of Kroki to display them as images.
+When you enable Kroki, GitLab sends diagrams to an instance of Kroki to display them as images.
You can use the free public cloud instance `https://kroki.io` or you can [install Kroki](https://docs.kroki.io/kroki/setup/install/)
on your own infrastructure.
After you've installed Kroki, make sure to update the server URL to point to your instance.
@@ -29,11 +29,16 @@ docker run -d --name kroki -p 8080:8000 yuzutech/kroki
The **Kroki URL** is the hostname of the server running the container.
-The [`yuzutech/kroki`](https://hub.docker.com/r/yuzutech/kroki) image contains the following diagrams libraries out-of-the-box:
+The [`yuzutech/kroki`](https://hub.docker.com/r/yuzutech/kroki) Docker image contains several diagram
+libraries out of the box. For a complete list, see the
+[`asciidoctor-kroki` README](https://github.com/ggrossetie/asciidoctor-kroki/blob/master/README.md#supported-diagram-types).
+Supported libraries include:
<!-- vale gitlab.Spelling = NO -->
- [Bytefield](https://bytefield-svg.deepsymmetry.org/)
+- [D2](https://d2lang.com/tour/intro/)
+- [DBML](https://www.dbml.org/home/)
- [Ditaa](https://ditaa.sourceforge.net)
- [Erd](https://github.com/BurntSushi/erd)
- [GraphViz](https://www.graphviz.org/)
diff --git a/doc/administration/integration/plantuml.md b/doc/administration/integration/plantuml.md
index 33434e15c4e..fcfae6cbe70 100644
--- a/doc/administration/integration/plantuml.md
+++ b/doc/administration/integration/plantuml.md
@@ -7,9 +7,8 @@ type: reference, howto
# PlantUML **(FREE)**
-When the [PlantUML](https://plantuml.com) integration is enabled and configured in
-GitLab, you can create diagrams in snippets, wikis, and repositories. This integration
-is enabled on GitLab.com for all SaaS users and does not require any additional configuration.
+With the [PlantUML](https://plantuml.com) integration, you can create diagrams in snippets, wikis, and repositories.
+This integration is enabled on GitLab.com for all SaaS users and does not require any additional configuration.
To set up the integration on a self-managed instance, you must:
@@ -117,7 +116,7 @@ services:
image: 'gitlab/gitlab-ee:12.2.5-ee.0'
environment:
GITLAB_OMNIBUS_CONFIG: |
- nginx['custom_gitlab_server_config'] = "location /-/plantuml/ { \n proxy_cache off; \n proxy_pass http://plantuml:8080/; \n}\n"
+ nginx['custom_gitlab_server_config'] = "location /-/plantuml/ { \n rewrite ^/-/plantuml/(.*) /$1 break;\n proxy_cache off; \n proxy_pass http://plantuml:8080/; \n}\n"
plantuml:
image: 'plantuml/plantuml-server:tomcat'
@@ -148,7 +147,7 @@ using Tomcat:
```
The Tomcat service should restart. After the restart is complete, the
-PlantUML service is ready and listening for requests on port 8080:
+PlantUML integration is ready and listening for requests on port 8080:
`http://localhost:8080/plantuml`
To change these defaults, edit the `/etc/tomcat8/server.xml` file.
@@ -169,7 +168,7 @@ following:
- `http://plantuml:8080/`
- `http://localhost:8080/plantuml/`
-If you're running [GitLab with TLS](https://docs.gitlab.com/omnibus/settings/ssl.html)
+If you're running [GitLab with TLS](https://docs.gitlab.com/omnibus/settings/ssl/index.html)
you must configure this redirection, because PlantUML uses the insecure HTTP protocol.
Newer browsers such as [Google Chrome 86+](https://www.chromestatus.com/feature/4926989725073408)
don't load insecure HTTP resources on pages served over HTTPS.
@@ -180,10 +179,10 @@ To enable this redirection:
```ruby
# Docker deployment
- nginx['custom_gitlab_server_config'] = "location /-/plantuml/ { \n proxy_cache off; \n proxy_pass http://plantuml:8080/; \n}\n"
+ nginx['custom_gitlab_server_config'] = "location /-/plantuml/ { \n rewrite ^/-/plantuml/(.*) /$1 break;\n proxy_cache off; \n proxy_pass http://plantuml:8080/; \n}\n"
# Built from source
- nginx['custom_gitlab_server_config'] = "location /-/plantuml { \n rewrite ^/-/(plantuml.*) /$1 break;\n proxy_cache off; \n proxy_pass http://localhost:8080/plantuml; \n}\n"
+ nginx['custom_gitlab_server_config'] = "location /-/plantuml { \n rewrite ^/-/plantuml/(.*) /$1 break;\n proxy_cache off; \n proxy_pass http://localhost:8080/plantuml; \n}\n"
```
1. To activate the changes, run the following command:
diff --git a/doc/administration/integration/terminal.md b/doc/administration/integration/terminal.md
index a1522ccac31..df9bb6b6e17 100644
--- a/doc/administration/integration/terminal.md
+++ b/doc/administration/integration/terminal.md
@@ -1,10 +1,10 @@
---
-stage: Configure
-group: Configure
+stage: Deploy
+group: Environments
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
-# Web terminals (DEPRECATED) **(FREE)**
+# Web terminals (deprecated) **(FREE)**
> - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5.
> - [Disabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/353410) in GitLab 15.0.
diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md
index 4f2eb20877e..099e7ec9e6f 100644
--- a/doc/administration/job_artifacts.md
+++ b/doc/administration/job_artifacts.md
@@ -1,13 +1,13 @@
---
stage: Verify
-group: Pipeline Insights
+group: Pipeline Security
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Jobs artifacts administration **(FREE SELF)**
This is the administration documentation. To learn how to use job artifacts in your GitLab CI/CD pipeline,
-see the [job artifacts configuration documentation](../ci/pipelines/job_artifacts.md).
+see the [job artifacts configuration documentation](../ci/jobs/job_artifacts.md).
An artifact is a list of files and directories attached to a job after it
finishes. This feature is enabled by default in all GitLab installations.
@@ -174,7 +174,7 @@ In a multi-server setup you must use one of the options to
[eliminate local disk usage for job logs](job_logs.md#prevent-local-disk-usage), or job logs could be lost.
In GitLab 13.2 and later, you should use the
-[consolidated object storage settings](object_storage.md#consolidated-object-storage-configuration).
+[consolidated object storage settings](object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form).
### Migrating to object storage
@@ -233,7 +233,7 @@ processing is done in a background worker and requires **no downtime**.
::EndTabs
- 1. Verify that all packages migrated to object storage with the following
+ 1. Verify that all artifacts migrated to object storage with the following
SQL query. The number of `objectstg` should be the same as `total`:
```shell
@@ -276,7 +276,7 @@ to clean up orphaned artifacts.
### Migrating from object storage to local storage
To migrate back to local storage, you must
-[selectively disable the artifacts storage](object_storage.md#selectively-disabling-object-storage).
+[selectively disable the artifacts storage](object_storage.md#disable-object-storage-for-specific-features).
## Expiring artifacts
@@ -419,6 +419,8 @@ reasons are:
to remove these. This script should always find work to do, as it also removes empty directories (see above).
- [Artifact housekeeping was changed significantly](#artifacts-housekeeping-disabled-in-gitlab-146-to-152),
and you might need to enable a feature flag to used the updated system.
+- The [keep latest artifacts from most recent success jobs](../ci/jobs/job_artifacts.md#keep-artifacts-from-most-recent-successful-jobs)
+ feature is enabled.
In these and other cases, identify the projects most responsible
for disk space usage, figure out what types of artifacts are using the most
@@ -461,7 +463,7 @@ To check if the feature flags are enabled:
```
These changes include switching artifacts from `unlocked` to `locked` if
-they [should be retained](../ci/pipelines/job_artifacts.md#keep-artifacts-from-most-recent-successful-jobs).
+they [should be retained](../ci/jobs/job_artifacts.md#keep-artifacts-from-most-recent-successful-jobs).
Artifacts created before this feature was introduced have a status of `unknown`. After they expire,
these artifacts are not processed by the new housekeeping jobs.
@@ -673,7 +675,7 @@ If you need to manually remove job artifacts associated with multiple jobs while
NOTE:
This step also erases artifacts that users have chosen to
- ["keep"](../ci/pipelines/job_artifacts.md#download-job-artifacts).
+ ["keep"](../ci/jobs/job_artifacts.md#download-job-artifacts).
```ruby
builds_to_clear = builds_with_artifacts.where("finished_at < ?", 1.week.ago)
@@ -779,7 +781,7 @@ In both cases, you might need to add `region` to the job artifact [object storag
### Job artifact upload fails with `500 Internal Server Error (Missing file)`
-Bucket names that include folder paths are not supported with [consolidated object storage](object_storage.md#consolidated-object-storage-configuration).
+Bucket names that include folder paths are not supported with [consolidated object storage](object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form).
For example, `bucket/path`. If a bucket name has a path in it, you might receive an error similar to:
```plaintext
@@ -807,3 +809,18 @@ To work around this issue, you can try:
- For older kernels, using the `nolease` mount option to disable file leasing.
For more information, [see the investigation details](https://gitlab.com/gitlab-org/gitlab/-/issues/389995).
+
+### Usage quota shows incorrect artifact storage usage
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/238536) in GitLab 14.10.
+
+Sometimes the [artifacts storage usage](../user/usage_quotas.md) displays an incorrect
+value for the total storage space used by artifacts. To recalculate the artifact
+usage statistics for all projects in the instance, you can run this background script:
+
+```shell
+bin/rake 'gitlab:refresh_project_statistics_build_artifacts_size[file.csv]'
+```
+
+The artifact usage value can fluctuate to `0` while the script is running. After
+recalculation, usage should display as expected again.
diff --git a/doc/administration/lfs/index.md b/doc/administration/lfs/index.md
index 3638729a6b6..ee5cd04f3d6 100644
--- a/doc/administration/lfs/index.md
+++ b/doc/administration/lfs/index.md
@@ -2,7 +2,6 @@
stage: Create
group: Source Code
info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments"
-disqus_identifier: 'https://docs.gitlab.com/ee/workflow/lfs/lfs_administration.html'
---
# GitLab Git Large File Storage (LFS) Administration **(FREE SELF)**
@@ -157,14 +156,14 @@ You can store LFS objects in remote object storage. This allows you
to reduce reads and writes to the local disk, and free up disk space significantly.
In GitLab 13.2 and later, you should use the
-[consolidated object storage settings](../object_storage.md#consolidated-object-storage-configuration).
+[consolidated object storage settings](../object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form).
### Migrating to object storage
You can migrate the LFS objects from local storage to object storage. The
processing is done in the background and requires **no downtime**.
-1. [Configure the object storage](../object_storage.md#consolidated-object-storage-configuration).
+1. [Configure the object storage](../object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form).
1. Migrate the LFS objects:
::Tabs
@@ -272,7 +271,7 @@ To migrate back to local storage:
```
1. Edit `/etc/gitlab/gitlab.rb` and
- [disable object storage](../object_storage.md#selectively-disabling-object-storage)
+ [disable object storage](../object_storage.md#disable-object-storage-for-specific-features)
for LFS objects:
```ruby
@@ -421,6 +420,18 @@ To check an installed Git LFS client's version, run this command:
git lfs version
```
+### `Connection refused` errors
+
+If you push or mirror LFS objects and receive errors like the following:
+
+- `dial tcp <IP>:443: connect: connection refused`
+- `Connection refused - connect(2) for \"<target-or-proxy-IP>\" port 443`
+
+a firewall or proxy rule may be terminating the connection.
+
+If connection checks with standard Unix tools or manual Git pushes are successful,
+the rule may be related to the size of the request.
+
## Error viewing a PDF file
When LFS has been configured with object storage and `proxy_download` set to
diff --git a/doc/administration/load_balancer.md b/doc/administration/load_balancer.md
index 298d22f1da5..e43fe851aa2 100644
--- a/doc/administration/load_balancer.md
+++ b/doc/administration/load_balancer.md
@@ -31,18 +31,18 @@ Configure your load balancers to pass connections on port 443 as 'TCP' rather
than 'HTTP(S)' protocol. This passes the connection to the application nodes
NGINX service untouched. NGINX has the SSL certificate and listen on port 443.
-See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html)
+See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html)
for details on managing SSL certificates and configuring NGINX.
### Load Balancers terminate SSL without backend SSL
Configure your load balancers to use the `HTTP(S)` protocol rather than `TCP`.
-The load balancers is be responsible for managing SSL certificates and
+The load balancers are responsible for managing SSL certificates and
terminating SSL.
Because communication between the load balancers and GitLab isn't secure,
there is some additional configuration needed. See the
-[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination)
+[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination)
for details.
### Load Balancers terminate SSL with backend SSL
@@ -55,7 +55,7 @@ Traffic is secure between the load balancers and NGINX in this
scenario. There is no need to add configuration for proxied SSL because the
connection is secure all the way. However, configuration must be
added to GitLab to configure SSL certificates. See
-the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html)
+the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html)
for details on managing SSL certificates and configuring NGINX.
## Ports
@@ -131,5 +131,5 @@ The default ciphers for a GitLab version can be
viewed in the [`files/gitlab-cookbooks/gitlab/attributes/default.rb`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/files/gitlab-cookbooks/gitlab/attributes/default.rb)
file and selecting the Git tag that correlates with your target GitLab version
(for example `15.0.5+ee.0`). If required by your load balancer, you can then define
-[custom SSL ciphers](https://docs.gitlab.com/omnibus/settings/ssl.html#use-custom-ssl-ciphers)
+[custom SSL ciphers](https://docs.gitlab.com/omnibus/settings/ssl/index.html#use-custom-ssl-ciphers)
for NGINX.
diff --git a/doc/administration/logs/index.md b/doc/administration/logs/index.md
index eab4c9b7d83..7fab97f76da 100644
--- a/doc/administration/logs/index.md
+++ b/doc/administration/logs/index.md
@@ -96,6 +96,7 @@ except those captured by `runit`.
| [LogRotate logs](#logrotate-logs) | **{dotted-circle}** No | **{check-circle}** Yes |
| [Mailroom](#mail_room_jsonlog-default) | **{check-circle}** Yes | **{check-circle}** Yes |
| [NGINX](#nginx-logs) | **{check-circle}** Yes | **{check-circle}** Yes |
+| [PgBouncer logs](#pgbouncer-logs) | **{dotted-circle}** No | **{check-circle}** Yes |
| [PostgreSQL logs](#postgresql-logs) | **{dotted-circle}** No | **{check-circle}** Yes |
| [Praefect logs](#praefect-logs) | **{dotted-circle}** Yes | **{check-circle}** Yes |
| [Prometheus logs](#prometheus-logs) | **{dotted-circle}** No | **{check-circle}** Yes |
@@ -180,7 +181,7 @@ In addition, the log contains the originating IP address,
(`remote_ip`), the user's ID (`user_id`), and username (`username`).
Some endpoints (such as `/search`) may make requests to Elasticsearch if using
-[Advanced Search](../../user/search/advanced_search.md). These
+[advanced search](../../user/search/advanced_search.md). These
additionally log `elasticsearch_calls` and `elasticsearch_call_duration_s`,
which correspond to:
@@ -349,15 +350,17 @@ the `view_duration_s` is calculated by [`duration_s - db_duration_s`](https://gi
Therefore, `view_duration_s` can be affected by multiple different factors, like read-write
process on Redis or external HTTP, not only the serialization process.
-## `application.log`
+## `application.log` (deprecated)
+
+> [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/111046) in GitLab 15.10.
Depending on your installation method, this file is located at:
- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/application.log`
- Installations from source: `/home/git/gitlab/log/application.log`
-It helps you discover events happening in your instance such as user creation
-and project deletion. For example:
+It contains a less structured version of the logs in
+[`application_json.log`](#application_jsonlog), like this example:
```plaintext
October 06, 2014 11:56: User "Administrator" (admin@example.com) was created
@@ -376,7 +379,8 @@ Depending on your installation method, this file is located at:
- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/application_json.log`
- Installations from source: `/home/git/gitlab/log/application_json.log`
-It contains the JSON version of the logs in `application.log`, like this example:
+It helps you discover events happening in your instance such as user creation
+and project deletion. For example:
```json
{
@@ -426,7 +430,7 @@ like this example:
}
```
-## `kubernetes.log` (DEPRECATED)
+## `kubernetes.log` (deprecated)
> [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5.
@@ -686,14 +690,6 @@ Unix timestamp format, and `gzip`-compressed (like `@1584057562.s`).
This file is at `/var/log/gitlab/gitlab-rails/grpc.log` for Omnibus GitLab
packages. Native [gRPC](https://grpc.io/) logging used by Gitaly.
-### `gitaly_ruby_json.log`
-
-> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/2678) in GitLab 13.6.
-
-This file is at `/var/log/gitlab/gitaly/gitaly_ruby_json.log` and is
-produced by [`gitaly-ruby`](../gitaly/reference.md#gitaly-ruby). It contains an
-access log of gRPC calls made by Gitaly to `gitaly-ruby`.
-
### `gitaly_hooks.log`
This file is at `/var/log/gitlab/gitaly/gitaly_hooks.log` and is
@@ -733,7 +729,7 @@ Depending on your installation method, this file is located at:
- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/importer.log`
- Installations from source: `/home/git/gitlab/log/importer.log`
-It logs the progress of the import process.
+This file logs the progress of [project imports and migrations](../../user/project/import/index.md).
## `exporter.log`
@@ -771,6 +767,24 @@ are recorded in this file. For example:
{"severity":"INFO","time":"2020-11-24T02:31:29.329Z","correlation_id":null,"key":"cd_auto_rollback","action":"remove"}
```
+## `ci_resource_groups_json.log`
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/384180) in GitLab 15.9.
+
+Depending on your installation method, this file is located at:
+
+- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/ci_resource_group_json.log`
+- Installations from source: `/home/git/gitlab/log/ci_resource_group_json.log`
+
+It contains information about [resource group](../../ci/resource_groups/index.md) acquisition. For example:
+
+```json
+{"severity":"INFO","time":"2023-02-10T23:02:06.095Z","correlation_id":"01GRYS10C2DZQ9J1G12ZVAD4YD","resource_group_id":1,"processable_id":288,"message":"attempted to assign resource to processable","success":true}
+{"severity":"INFO","time":"2023-02-10T23:02:08.945Z","correlation_id":"01GRYS138MYEG32C0QEWMC4BDM","resource_group_id":1,"processable_id":288,"message":"attempted to release resource from processable","success":true}
+```
+
+The examples show the `resource_group_id`, `processable_id`, `message`, and `success` fields for each entry.
+
## `auth.log`
> Introduced in GitLab 12.0.
@@ -787,6 +801,28 @@ This log records:
- In GitLab versions [12.3](https://gitlab.com/gitlab-org/gitlab/-/issues/29239) and later,
user ID and username, if available.
+## `auth_json.log`
+
+Depending on your installation method, this file is located at:
+
+- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/auth_json.log`
+- Installations from source: `/home/git/gitlab/log/auth_json.log`
+
+This file contains the JSON version of the logs in `auth.log`, for example:
+
+```json
+{
+ "severity":"ERROR",
+ "time":"2023-04-19T22:14:25.893Z",
+ "correlation_id":"01GYDSAKAN2SPZPAMJNRWW5H8S",
+ "message":"Rack_Attack",
+ "env":"blocklist",
+ "remote_ip":"x.x.x.x",
+ "request_method":"GET",
+ "path":"/group/project.git/info/refs?service=git-upload-pack"
+}
+```
+
## `graphql_json.log`
> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/59587) in GitLab 12.0.
@@ -811,6 +847,8 @@ Depending on your installation method, this file is located at:
- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/migrations.log`
- Installations from source: `/home/git/gitlab/log/migrations.log`
+This file logs the progress of [database migrations](../raketasks/maintenance.md#display-status-of-database-migrations).
+
## `mail_room_json.log` (default)
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/19186) in GitLab 12.6.
@@ -1016,9 +1054,13 @@ For Omnibus GitLab installations, NGINX logs are in:
Below is the default GitLab NGINX access log format:
```plaintext
-$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"
+'$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"'
```
+The `$request` and `$http_referer` are
+[filtered](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/support/nginx/gitlab)
+for sensitive query string parameters such as secret tokens.
+
## Pages logs
For Omnibus GitLab installations, Pages logs are in `/var/log/gitlab/gitlab-pages/current`.
@@ -1059,6 +1101,10 @@ For Omnibus GitLab installations, Mattermost logs are in these locations:
For Omnibus GitLab installations, Workhorse logs are in `/var/log/gitlab/gitlab-workhorse/current`.
+## PgBouncer logs
+
+For Omnibus GitLab installations, PgBouncer logs are in `/var/log/gitlab/pgbouncer/current`.
+
## PostgreSQL logs
For Omnibus GitLab installations, PostgreSQL logs are in `/var/log/gitlab/postgresql/current`.
diff --git a/doc/administration/logs/log_parsing.md b/doc/administration/logs/log_parsing.md
index 84c517e6120..d9520ea9bc0 100644
--- a/doc/administration/logs/log_parsing.md
+++ b/doc/administration/logs/log_parsing.md
@@ -158,7 +158,7 @@ CT: 297 ROUTE: /api/:version/projects/:id/repository/tags DURS: 731.39,
CT: 190 ROUTE: /api/:version/projects/:id/repository/commits DURS: 1079.02, 979.68, 958.21
```
-### Print top API user agents
+#### Print top API user agents
```shell
jq --raw-output '[.route, .ua] | @tsv' api_json.log | sort | uniq -c | sort -n
@@ -180,9 +180,20 @@ if the output contains many:
You can also [use `fast-stats top`](#parsing-gitlab-logs-with-jq) to extract performance statistics.
+### Parsing `gitlab-rails/importer.log`
+
+To troubleshoot [project imports](../../administration/raketasks/project_import_export.md) or
+[migrations](../../user/project/import/index.md), run this command:
+
+```shell
+jq 'select(.project_path == "<namespace>/<project>").error_messages' importer.log
+```
+
+For common issues, see [troubleshooting](../../administration/raketasks/project_import_export.md#troubleshooting).
+
### Parsing `gitlab-workhorse/current`
-### Print top Workhorse user agents
+#### Print top Workhorse user agents
```shell
jq --raw-output '[.uri, .user_agent] | @tsv' current | sort | uniq -c | sort -n
@@ -212,9 +223,12 @@ repeatedly reports that some items never reach 100%,
the following command helps to focus on the most common errors.
```shell
-jq --raw-output 'select(.severity == "ERROR") | [.project_path, .message] | @tsv' geo.log | sort | uniq -c | sort | tail
+jq --raw-output 'select(.severity == "ERROR") | [.project_path, .class, .message, .error] | @tsv' geo.log | sort | uniq -c | sort | tail
```
+Refer to our [Geo troubleshooting page](../geo/replication/troubleshooting.md)
+for advice about specific error messages.
+
### Parsing `gitaly/current`
Use the following examples to [troubleshoot Gitaly](../gitaly/troubleshooting.md).
diff --git a/doc/administration/maintenance_mode/index.md b/doc/administration/maintenance_mode/index.md
index c30465b4df9..8347015a8a3 100644
--- a/doc/administration/maintenance_mode/index.md
+++ b/doc/administration/maintenance_mode/index.md
@@ -8,17 +8,17 @@ info: To determine the technical writer assigned to the Stage/Group associated w
> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2149) in GitLab 13.9.
-Maintenance Mode allows administrators to reduce write operations to a minimum while maintenance tasks are performed. The main goal is to block all external actions that change the internal state, including the PostgreSQL database, but especially files, Git repositories, and Container repositories.
+Maintenance Mode allows administrators to reduce write operations to a minimum while maintenance tasks are performed. The main goal is to block all external actions that change the internal state. The internal state includes the PostgreSQL database, but especially files, Git repositories, and Container repositories.
-When Maintenance Mode is enabled, in-progress actions finish relatively quickly since no new actions are coming in, and internal state changes are minimal.
-In that state, various maintenance tasks are easier, and services can be stopped completely or be
-further degraded for a much shorter period of time than might otherwise be needed. For example, stopping cron jobs and draining queues should be fairly quick.
+When Maintenance Mode is enabled, in-progress actions finish relatively quickly because no new actions are coming in, and internal state changes are minimal.
+In that state, various maintenance tasks are easier. Services can be stopped completely or
+further degraded for a shorter period of time than might otherwise be needed. For example, stopping cron jobs and draining queues should be fairly quick.
Maintenance Mode allows most external actions that do not change internal state. On a high-level, HTTP `POST`, `PUT`, `PATCH`, and `DELETE` requests are blocked and a detailed overview of [how special cases are handled](#rest-api) is available.
## Enable Maintenance Mode
-There are three ways to enable Maintenance Mode as an administrator:
+Enable Maintenance Mode as an administrator in one of these ways:
- **Web UI**:
1. On the top bar, select **Main menu > Admin**.
@@ -42,7 +42,7 @@ There are three ways to enable Maintenance Mode as an administrator:
## Disable Maintenance Mode
-There are three ways to disable Maintenance Mode:
+Disable Maintenance Mode in one of three ways:
- **Web UI**:
1. On the top bar, select **Main menu > Admin**.
@@ -73,7 +73,7 @@ An error is displayed when a user tries to perform a write operation that isn't
![Maintenance Mode banner and error message](img/maintenance_mode_error_message.png)
NOTE:
-In some cases, the visual feedback from an action could be misleading, for example when starring a project, the **Star** button changes to show the **Unstar** action, however, this is only the frontend update, and it doesn't take into account the failed status of the POST request. These visual bugs are to be fixed [in follow-up iterations](https://gitlab.com/gitlab-org/gitlab/-/issues/295197).
+In some cases, the visual feedback from an action could be misleading. For example, when starring a project, the **Star** button changes to show the **Unstar** action. However, this is only the frontend update, and it doesn't take into account the failed status of the POST request. These visual bugs are to be fixed [in follow-up iterations](https://gitlab.com/gitlab-org/gitlab/-/issues/295197).
### Administrator functions
@@ -84,7 +84,7 @@ them to disable Maintenance Mode after it's been enabled.
All users can sign in and out of the GitLab instance but no new users can be created.
-If there are [LDAP syncs](../auth/ldap/index.md) scheduled for that time, they fail since user creation is disabled. Similarly, [user creations based on SAML](../../integration/saml.md#configure-saml-support-in-gitlab) fail.
+If there are [LDAP syncs](../auth/ldap/index.md) scheduled for that time, they fail because user creation is disabled. Similarly, [user creations based on SAML](../../integration/saml.md#configure-saml-support-in-gitlab) fail.
### Git actions
@@ -135,22 +135,22 @@ For most JSON requests, `POST`, `PUT`, `PATCH`, and `DELETE` are blocked, and th
even if they finish running on the GitLab Runner.
- Jobs in the `running` state for longer than the project's time limit do not time out.
- Pipelines cannot be started, retried or canceled. No new jobs can be created either.
-- The status of the runners in `/admin/runners` won't be updated.
-- `gitlab-runner verify` will return the error `ERROR: Verifying runner... is removed`.
+- The status of the runners in `/admin/runners` isn't updated.
+- `gitlab-runner verify` returns the error `ERROR: Verifying runner... is removed`.
After Maintenance Mode is disabled, new jobs are picked up again. Jobs that were
in the `running` state before enabling Maintenance Mode resume and their logs start
updating again.
NOTE:
-It is recommended that you restart previously `running` pipelines after Maintenance Mode
+You should restart previously `running` pipelines after Maintenance Mode
is turned off.
### Deployments
Deployments don't go through because pipelines are unfinished.
-It is recommended to disable auto deploys during Maintenance Mode, and enable them when it is disabled.
+You should disable auto deploys during Maintenance Mode, and enable them when it is disabled.
#### Terraform integration
@@ -169,7 +169,7 @@ Package Registry allows you to install but not publish packages.
Background jobs (cron jobs, Sidekiq) continue running as is, because background jobs are not automatically disabled.
[During a planned Geo failover](../geo/disaster_recovery/planned_failover.md#prevent-updates-to-the-primary-site),
-it is recommended that you disable all cron jobs except for those related to Geo.
+you should disable all cron jobs except for those related to Geo.
To monitor queues and disable jobs:
@@ -206,7 +206,7 @@ SAST and Secret Detection cannot be initiated because they depend on passing CI
## An example use case: a planned failover
-In the use case of [a planned failover](../geo/disaster_recovery/planned_failover.md), a few writes in the primary database are acceptable, since they are replicated quickly and are not significant in number.
+In the use case of [a planned failover](../geo/disaster_recovery/planned_failover.md), a few writes in the primary database are acceptable, because they are replicated quickly and are not significant in number.
For the same reason we don't automatically block background jobs when Maintenance Mode is enabled.
diff --git a/doc/administration/merge_request_diffs.md b/doc/administration/merge_request_diffs.md
index 85677512860..3250a2339e2 100644
--- a/doc/administration/merge_request_diffs.md
+++ b/doc/administration/merge_request_diffs.md
@@ -105,7 +105,7 @@ be configured already.
### Object Storage Settings
In GitLab 13.2 and later, you should use the
-[consolidated object storage settings](object_storage.md#consolidated-object-storage-configuration).
+[consolidated object storage settings](object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form).
This section describes the earlier configuration format.
For source installations, these settings are nested under `external_diffs:` and
@@ -121,7 +121,7 @@ then `object_store:`. On Omnibus installations, they are prefixed by
#### S3 compatible connection settings
-See [the available connection settings for different providers](object_storage.md#connection-settings).
+See [the available connection settings for different providers](object_storage.md#configure-the-connection-settings).
**In Omnibus installations:**
diff --git a/doc/administration/monitoring/github_imports.md b/doc/administration/monitoring/github_imports.md
index 097109585af..74e584db3a0 100644
--- a/doc/administration/monitoring/github_imports.md
+++ b/doc/administration/monitoring/github_imports.md
@@ -1,6 +1,6 @@
---
-stage: Monitor
-group: Respond
+stage: Manage
+group: Import and Integrate
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
diff --git a/doc/administration/monitoring/gitlab_self_monitoring_project/img/self_monitoring_overview_dashboard.png b/doc/administration/monitoring/gitlab_self_monitoring_project/img/self_monitoring_overview_dashboard.png
deleted file mode 100644
index 1d61823ce41..00000000000
--- a/doc/administration/monitoring/gitlab_self_monitoring_project/img/self_monitoring_overview_dashboard.png
+++ /dev/null
Binary files differ
diff --git a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md
index 566bc070347..dbdcdf22007 100644
--- a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md
+++ b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md
@@ -2,120 +2,11 @@
stage: Monitor
group: Respond
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+remove_date: '2023-08-22'
+redirect_to: '../index.md'
---
-# Self-monitoring project (DEPRECATED) **(FREE SELF)**
+# Self-monitoring project (removed) **(FREE SELF)**
-> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32351) in GitLab 12.7 [with a flag](../../feature_flags.md) named `self_monitoring_project`. Disabled by default.
-> - Generally available in GitLab 12.8. [Feature flag `self_monitoring_project`](https://gitlab.com/gitlab-org/gitlab/-/issues/198511) removed.
-> - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/348909) in GitLab 14.9. Planned for removal in GitLab 16.0.
-
-WARNING:
-This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/348909)
-in GitLab 14.9, and is planned for removal in GitLab 16.0.
-
-GitLab provides administrators insights into the health of their GitLab instance.
-
-To provide a native experience (similar interacting with an application deployed using GitLab), a
-project called **Monitoring** is created:
-
-- With [internal visibility](../../../user/public_access.md#internal-projects-and-groups).
-- Under a group called **GitLab Instance**.
-
-The project is created specifically for visualizing and configuring the monitoring of your GitLab
-instance.
-
-When the project and group are created, all administrators are given the [Maintainer role](../../../user/permissions.md).
-As an administrator, you can add new members to the group to give them the Maintainer role for the project.
-
-This project can be used to:
-
-- Self-monitor your GitLab instance. The metrics dashboard of the project shows some basic resource
- usage charts, such as CPU and memory usage of each server in
- [Omnibus GitLab](https://docs.gitlab.com/omnibus/) installations.
-- Also configure your own [custom metrics](../../../operations/metrics/index.md#adding-custom-metrics)
- using metrics exposed by the [GitLab exporter](../prometheus/gitlab_metrics.md#metrics-available).
-
-## Create the self-monitoring project
-
-1. On the top bar, select **Main menu > Admin**.
-1. On the left sidebar, select **Settings > Metrics and profiling** and expand **Self-monitoring**.
-1. Toggle **Self-monitoring** on.
-1. After your GitLab instance creates the project, GitLab displays a link to the
- project in the text above the **Self-monitoring** toggle. You can also find it
- from the top bar by selecting **Main menu > Projects**.
-
-## Delete the self-monitoring project
-
-WARNING:
-Deleting the self-monitoring project removes any changes made to the project. If
-you create the project again, it's created in its default state.
-
-1. On the top bar, select **Main menu > Admin**.
-1. On the left sidebar, go to **Settings > Metrics and profiling** and expand **Self-monitoring**.
-1. Toggle **Self-monitoring** off.
-1. In the confirmation dialog that opens, select **Delete self-monitoring project**.
- It can take a few seconds for it to be deleted.
-1. After the project is deleted, GitLab displays a message confirming your action.
-
-## Dashboards available in Omnibus GitLab
-
-Omnibus GitLab provides a dashboard that displays CPU and memory usage
-of each GitLab server. To select the servers to be displayed in the
-panels, provide a regular expression in the **Instance label regex** field.
-The dashboard uses metrics available in
-[Omnibus GitLab](https://docs.gitlab.com/omnibus/) installations.
-
-![GitLab self-monitoring overview dashboard](img/self_monitoring_overview_dashboard.png)
-
-You can also
-[create your own dashboards](../../../operations/metrics/dashboards/index.md).
-
-## Connect to Prometheus
-
-The project is automatically configured to connect to the
-[internal Prometheus](../prometheus/index.md) instance if the Prometheus instance is present.
-This should be the case if GitLab was installed using Omnibus GitLab and you haven't disabled it.
-
-If that's not the case, or if you have an external Prometheus instance or a customized setup,
-you [configure it manually](../../../user/project/integrations/prometheus.md#manual-configuration-of-prometheus).
-
-## Take action on Prometheus alerts **(ULTIMATE)**
-
-You can [add a Prometheus integration](../../../operations/incident_management/integrations.md)
-to GitLab to receive notifications of any alerts.
-
-When the integration is set up, you can
-[take action on incoming alerts](../../../operations/metrics/alerts.md#trigger-actions-from-alerts).
-
-## Add custom metrics to the self-monitoring project
-
-You can add custom metrics in the self-monitoring project by:
-
-1. [Duplicating](../../../operations/metrics/dashboards/index.md#duplicate-a-gitlab-defined-dashboard) the overview dashboard.
-1. [Editing](../../../operations/metrics/index.md) the newly created dashboard file and configuring it with [dashboard YAML properties](../../../operations/metrics/dashboards/yaml.md).
-
-## Troubleshooting
-
-### Error message in logs: `Could not create instance administrators group. Errors: ["You don't have permission to create groups."]`
-
-A [bug](https://gitlab.com/gitlab-org/gitlab/-/issues/208676) causes project creation to fail with
-the following error in the log file when the first administrator user is an
-[external user](../../../user/admin_area/external_users.md):
-
-```plaintext
-Could not create instance administrators group. Errors: ["You don't have permission to create groups."]
-```
-
-Run the following in a Rails console to check if the first administrator user is an external user:
-
-```ruby
-User.admins.active.first.external?
-```
-
-If this returns true, the first administrator user is an external user.
-
-If you face this issue, you can temporarily
-[make the administrator user a non-external user](../../../user/admin_area/external_users.md)
-and then try to create the project.
-After the project is created, the administrator user can be changed back to an external user.
+This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/348909)
+in GitLab 14.9 and [removed](https://gitlab.com/groups/gitlab-org/-/epics/10030) in 16.0.
diff --git a/doc/administration/monitoring/index.md b/doc/administration/monitoring/index.md
index e57156e6513..1b23d6b7f49 100644
--- a/doc/administration/monitoring/index.md
+++ b/doc/administration/monitoring/index.md
@@ -8,9 +8,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w
Explore our features to monitor your GitLab instance:
-- [GitLab self-monitoring](gitlab_self_monitoring_project/index.md): The
- GitLab instance administration project helps to monitor the GitLab instance and
- take action on alerts.
- [Performance monitoring](performance/index.md): GitLab Performance Monitoring
makes it possible to measure a wide variety of statistics of your instance.
- [Prometheus](prometheus/index.md): Prometheus is a powerful time-series monitoring
diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md
index 3dec34ebace..1113dcfef32 100644
--- a/doc/administration/monitoring/performance/grafana_configuration.md
+++ b/doc/administration/monitoring/performance/grafana_configuration.md
@@ -6,10 +6,41 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# Grafana Configuration **(FREE SELF)**
+> [Deprecated](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7772) in GitLab 16.0.
+
+WARNING:
+Bundled Grafana was deprecated GitLab 16.0 and is no longer supported. It will be removed in GitLab 16.3.
+For more information, see [deprecation notes](#deprecation-of-bundled-grafana).
+
[Grafana](https://grafana.com/) is a tool that enables you to visualize time
series metrics through graphs and dashboards. GitLab writes performance data to Prometheus,
and Grafana allows you to query the data to display useful graphs.
+## Deprecation of bundled Grafana
+
+Bundled Grafana was an optional Omnibus GitLab service that provided a user interface to GitLab metrics.
+
+The version of Grafana that is bundled with Omnibus GitLab is no longer supported. If you're using the bundled Grafana, you
+should switch to a newer version from [Grafana Labs](https://grafana.com/grafana/).
+
+### Switch to new Grafana instance
+
+To switch away from bundled Grafana to a newer version of Grafana from Grafana Labs:
+
+1. Set up a version of Grafana from Grafana Labs.
+1. [Export the existing dashboards](https://grafana.com/docs/grafana/latest/dashboards/manage-dashboards/#export-a-dashboard) from bundled Grafana.
+1. [Import the existing dashboards](https://grafana.com/docs/grafana/latest/dashboards/manage-dashboards/#import-a-dashboard) in the new Grafana instance.
+1. [Configure GitLab](#integration-with-gitlab-ui) to use the new Grafana instance.
+
+### Temporary workaround
+
+In GitLab versions 16.0 to 16.2, you can still force Omnibus GitLab to enable and configure Grafana by setting the following:
+
+- `grafana['enable'] = true`.
+- `grafana['enable_deprecated_service'] = true`.
+
+You see a deprecation message when reconfiguring GitLab.
+
## Installation
Omnibus GitLab can [help you install Grafana (recommended)](https://docs.gitlab.com/omnibus/settings/grafana.html)
diff --git a/doc/administration/monitoring/performance/index.md b/doc/administration/monitoring/performance/index.md
index 89f1270532a..cbdd68804e8 100644
--- a/doc/administration/monitoring/performance/index.md
+++ b/doc/administration/monitoring/performance/index.md
@@ -1,6 +1,6 @@
---
-stage: Monitor
-group: Respond
+stage: Data Stores
+group: Application Performance
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md
index ed55e7d7ff3..c0a175c76cd 100644
--- a/doc/administration/monitoring/prometheus/gitlab_metrics.md
+++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md
@@ -1,6 +1,6 @@
---
-stage: Monitor
-group: Respond
+stage: Shared responsibility based on functional area
+group: Shared responsibility based on functional area
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
@@ -30,16 +30,16 @@ For enabling and viewing metrics from Sidekiq nodes, see [Sidekiq metrics](#side
## Metrics available
+> `caller_id` [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/392622) from `redis_hit_miss_operations_total` and `redis_cache_generation_duration_seconds` in GitLab 15.11.
+
The following metrics are available:
| Metric | Type | Since | Description | Labels |
| :--------------------------------------------------------------- | :---------- | ------: | :-------------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------- |
-| `gitlab_banzai_cached_render_real_duration_seconds` | Histogram | 9.4 | Duration of rendering Markdown into HTML when cached output exists | `controller`, `action` |
-| `gitlab_banzai_cacheless_render_real_duration_seconds` | Histogram | 9.4 | Duration of rendering Markdown into HTML when cached output does not exist | `controller`, `action` |
| `gitlab_cache_misses_total` | Counter | 10.2 | Cache read miss | `controller`, `action` |
-| `gitlab_cache_operation_duration_seconds` | Histogram | 10.2 | Cache access time | |
-| `gitlab_cache_operations_total` | Counter | 12.2 | Cache operations by controller or action | `controller`, `action`, `operation` |
-| `gitlab_cache_read_multikey_count` | Histogram | 15.7 | Count of keys in multi-key cache read operations | `controller`, `action` |
+| `gitlab_cache_operation_duration_seconds` | Histogram | 10.2 | Cache access time | `operation`, `store` |
+| `gitlab_cache_operations_total` | Counter | 12.2 | Cache operations by controller or action | `controller`, `action`, `operation`, `store` |
+| `gitlab_cache_read_multikey_count` | Histogram | 15.7 | Count of keys in multi-key cache read operations | `controller`, `action`, `store` |
| `gitlab_ci_pipeline_builder_scoped_variables_duration` | Histogram | 14.5 | Time in seconds it takes to create the scoped variables for a CI/CD job
| `gitlab_ci_pipeline_creation_duration_seconds` | Histogram | 13.0 | Time in seconds it takes to create a CI/CD pipeline | `gitlab` |
| `gitlab_ci_pipeline_size_builds` | Histogram | 13.1 | Total number of builds within a pipeline grouped by a pipeline source | `source` |
@@ -129,10 +129,12 @@ The following metrics are available:
| `action_cable_single_client_transmissions_total` | Counter | 13.10 | The number of ActionCable messages transmitted to any client in any channel | `server_mode` |
| `action_cable_subscription_confirmations_total` | Counter | 13.10 | The number of ActionCable subscriptions from clients confirmed | `server_mode` |
| `action_cable_subscription_rejections_total` | Counter | 13.10 | The number of ActionCable subscriptions from clients rejected | `server_mode` |
-| `action_cable_transmitted_bytes` | Histogram | 14.1 | Message size, in bytes, transmitted over action cable | `operation`, `channel` |
+| `action_cable_transmitted_bytes_total` | Counter | 16.0 | Total number of bytes transmitted over ActionCable | `operation`, `channel` |
| `gitlab_issuable_fast_count_by_state_total` | Counter | 13.5 | Total number of row count operations on issue/merge request list pages | |
| `gitlab_issuable_fast_count_by_state_failures_total` | Counter | 13.5 | Number of soft-failed row count operations on issue/merge request list pages | |
| `gitlab_ci_trace_finalize_duration_seconds` | Histogram | 13.6 | Duration of build trace chunks migration to object storage | |
+| `gitlab_vulnerability_report_branch_comparison_real_duration_seconds` | Histogram | 15.11 | Execution duration of vulnerability report present on default branch sql query | |
+| `gitlab_vulnerability_report_branch_comparison_cpu_duration_seconds` | Histogram | 15.11 | Execution duration of vulnerability report present on default branch sql query | |
| `gitlab_external_http_total` | Counter | 13.8 | Total number of HTTP calls to external systems | `controller`, `action` |
| `gitlab_external_http_duration_seconds` | Counter | 13.8 | Duration in seconds spent on each HTTP call to external systems | |
| `gitlab_external_http_exception_total` | Counter | 13.8 | Total number of exceptions raised when making external HTTP calls | |
@@ -151,8 +153,8 @@ The following metrics are available:
| `gitlab_ci_build_trace_errors_total` | Counter | 14.4 | Total amount of different error types on a build trace | `error_reason` |
| `gitlab_presentable_object_cacheless_render_real_duration_seconds` | Histogram | 15.3 | Duration of real time spent caching and representing specific web request objects | `controller`, `action` |
| `cached_object_operations_total` | Counter | 15.3 | Total number of objects cached for specific web requests | `controller`, `action` |
-| `redis_hit_miss_operations_total` | Counter | 15.6 | Total number of Redis cache hits and misses | `cache_hit`, `caller_id`, `cache_identifier`, `feature_category`, `backing_resource` |
-| `redis_cache_generation_duration_seconds` | Histogram | 15.6 | Time to generate Redis cache | `cache_hit`, `caller_id`, `cache_identifier`, `feature_category`, `backing_resource` |
+| `redis_hit_miss_operations_total` | Counter | 15.6 | Total number of Redis cache hits and misses | `cache_hit`, `cache_identifier`, `feature_category`, `backing_resource` |
+| `redis_cache_generation_duration_seconds` | Histogram | 15.6 | Time to generate Redis cache | `cache_hit`, `cache_identifier`, `feature_category`, `backing_resource` |
| `gitlab_diffs_reorder_real_duration_seconds` | Histogram | 15.8 | Duration in seconds spend on reordering of diff files on diffs batch request | `controller`, `action` |
| `gitlab_diffs_collection_real_duration_seconds` | Histogram | 15.8 | Duration in seconds spent on querying merge request diff files on diffs batch request | `controller`, `action` |
| `gitlab_diffs_comparison_real_duration_seconds` | Histogram | 15.8 | Duration in seconds spent on getting comparison data on diffs batch request | `controller`, `action` |
@@ -163,12 +165,15 @@ The following metrics are available:
| `gitlab_diffs_render_real_duration_seconds` | Histogram | 15.8 | Duration in seconds spent on serializing and rendering diffs on diffs batch request | `controller`, `action` |
| `gitlab_memwd_violations_total` | Counter | 15.9 | Total number of times a Ruby process violated a memory threshold | |
| `gitlab_memwd_violations_handled_total` | Counter | 15.9 | Total number of times Ruby process memory violations were handled | |
+| `gitlab_sli_rails_request_apdex_total` | Counter | 14.4 | Total number of request Apdex measurements. For more information, see [Rails request SLIs](../../../development/application_slis/rails_request.md) | `endpoint_id`, `feature_category`, `request_urgency` |
+| `gitlab_sli_rails_request_apdex_success_total` | Counter | 14.4 | Total number of successful requests that met the target duration for their urgency. Divide by `gitlab_sli_rails_requests_apdex_total` to get a success ratio | `endpoint_id`, `feature_category`, `request_urgency` |
+| `gitlab_sli_rails_request_error_total` | Counter | 15.7 | Total number of request error measurements. For more information, see [Rails request SLIs](../../../development/application_slis/rails_request.md) | `endpoint_id`, `feature_category`, `request_urgency`, `error` |
## Metrics controlled by a feature flag
The following metrics can be controlled by feature flags:
-| Metric | Feature Flag |
+| Metric | Feature flag |
|:---------------------------------------------------------------|:-------------------------------------------------------------------|
| `gitlab_view_rendering_duration_seconds` | `prometheus_metrics_view_instrumentation` |
@@ -206,16 +211,16 @@ configuration option in `gitlab.yml`. These metrics are served from the
| `geo_repositories` | Gauge | 10.2 | Total number of repositories available on primary | `url` |
| `geo_repositories_synced` | Gauge | 10.2 | Number of repositories synced on secondary | `url` |
| `geo_repositories_failed` | Gauge | 10.2 | Number of repositories failed to sync on secondary | `url` |
-| `geo_lfs_objects` | Gauge | 10.2 | Number of LFS objects on primary | `url` |
+| `geo_lfs_objects` | Gauge | 10.2 | Number of LFS objects on primary | `url` |
| `geo_lfs_objects_checksummed` | Gauge | 14.6 | Number of LFS objects checksummed successfully on primary | `url` |
| `geo_lfs_objects_checksum_failed` | Gauge | 14.6 | Number of LFS objects failed to calculate the checksum on primary | `url` |
-| `geo_lfs_objects_checksum_total` | Gauge | 14.6 | Number of LFS objects tried to checksum on primary | `url` |
+| `geo_lfs_objects_checksum_total` | Gauge | 14.6 | Number of LFS objects that need to be checksummed on primary | `url` |
| `geo_lfs_objects_synced` | Gauge | 10.2 | Number of syncable LFS objects synced on secondary | `url` |
| `geo_lfs_objects_failed` | Gauge | 10.2 | Number of syncable LFS objects failed to sync on secondary | `url` |
| `geo_lfs_objects_registry` | Gauge | 14.6 | Number of LFS objects in the registry | `url` |
-| `geo_lfs_objects_verified` | Gauge | 14.6 | Number of LFS objects verified on secondary | `url` |
-| `geo_lfs_objects_verification_failed` | Gauge | 14.6 | Number of LFS objects' verifications failed on secondary | `url` |
-| `geo_lfs_objects_verification_total` | Gauge | 14.6 | Number of LFS objects' verifications tried on secondary | `url` |LFS objects failed to sync on secondary | `url` |
+| `geo_lfs_objects_verified` | Gauge | 14.6 | Number of LFS objects successfully verified on secondary | `url` |
+| `geo_lfs_objects_verification_failed` | Gauge | 14.6 | Number of LFS objects that failed verifications on secondary | `url` |
+| `geo_lfs_objects_verification_total` | Gauge | 14.6 | Number of LFS objects to attempt to verify on secondary | `url` |
| `geo_attachments` | Gauge | 10.2 | Total number of file attachments available on primary | `url` |
| `geo_attachments_synced` | Gauge | 10.2 | Number of attachments synced on secondary | `url` |
| `geo_attachments_failed` | Gauge | 10.2 | Number of attachments failed to sync on secondary | `url` |
@@ -230,11 +235,11 @@ configuration option in `gitlab.yml`. These metrics are served from the
| `geo_repositories_checksum_failed` | Gauge | 10.7 | Number of repositories failed to calculate the checksum on primary | `url` |
| `geo_wikis_checksummed` | Gauge | 10.7 | Number of wikis checksummed on primary | `url` |
| `geo_wikis_checksum_failed` | Gauge | 10.7 | Number of wikis failed to calculate the checksum on primary | `url` |
-| `geo_repositories_verified` | Gauge | 10.7 | Number of repositories verified on secondary | `url` |
-| `geo_repositories_verification_failed` | Gauge | 10.7 | Number of repositories failed to verify on secondary | `url` |
+| `geo_repositories_verified` | Gauge | 10.7 | Number of repositories successfully verified on secondary | `url` |
+| `geo_repositories_verification_failed` | Gauge | 10.7 | Number of repositories that failed verification on secondary | `url` |
| `geo_repositories_checksum_mismatch` | Gauge | 10.7 | Number of repositories that checksum mismatch on secondary | `url` |
-| `geo_wikis_verified` | Gauge | 10.7 | Number of wikis verified on secondary | `url` |
-| `geo_wikis_verification_failed` | Gauge | 10.7 | Number of wikis failed to verify on secondary | `url` |
+| `geo_wikis_verified` | Gauge | 10.7 | Number of wikis successfully verified on secondary | `url` |
+| `geo_wikis_verification_failed` | Gauge | 10.7 | Number of wikis that failed verification on secondary | `url` |
| `geo_wikis_checksum_mismatch` | Gauge | 10.7 | Number of wikis that checksum mismatch on secondary | `url` |
| `geo_repositories_checked` | Gauge | 11.1 | Number of repositories that have been checked via `git fsck` | `url` |
| `geo_repositories_checked_failed` | Gauge | 11.1 | Number of repositories that have a failure from `git fsck` | `url` |
@@ -249,25 +254,25 @@ configuration option in `gitlab.yml`. These metrics are served from the
| `geo_terraform_state_versions` | Gauge | 13.5 | Number of terraform state versions on primary | `url` |
| `geo_terraform_state_versions_checksummed` | Gauge | 13.5 | Number of terraform state versions checksummed successfully on primary | `url` |
| `geo_terraform_state_versions_checksum_failed` | Gauge | 13.5 | Number of terraform state versions failed to calculate the checksum on primary | `url` |
-| `geo_terraform_state_versions_checksum_total` | Gauge | 13.12 | Number of terraform state versions tried to checksum on primary | `url` |
+| `geo_terraform_state_versions_checksum_total` | Gauge | 13.12 | Number of terraform state versions that need to be checksummed on primary | `url` |
| `geo_terraform_state_versions_synced` | Gauge | 13.5 | Number of syncable terraform state versions synced on secondary | `url` |
| `geo_terraform_state_versions_failed` | Gauge | 13.5 | Number of syncable terraform state versions failed to sync on secondary | `url` |
| `geo_terraform_state_versions_registry` | Gauge | 13.5 | Number of terraform state versions in the registry | `url` |
-| `geo_terraform_state_versions_verified` | Gauge | 13.12 | Number of terraform state versions verified on secondary | `url` |
-| `geo_terraform_state_versions_verification_failed` | Gauge | 13.12 | Number of terraform state versions verifications failed on secondary | `url` |
-| `geo_terraform_state_versions_verification_total` | Gauge | 13.12 | Number of terraform state versions verifications tried on secondary | `url` |
+| `geo_terraform_state_versions_verified` | Gauge | 13.12 | Number of terraform state versions successfully verified on secondary | `url` |
+| `geo_terraform_state_versions_verification_failed` | Gauge | 13.12 | Number of terraform state versions that failed verification on secondary | `url` |
+| `geo_terraform_state_versions_verification_total` | Gauge | 13.12 | Number of terraform state versions to attempt to verify on secondary | `url` |
| `global_search_bulk_cron_queue_size` | Gauge | 12.10 | Number of database records waiting to be synchronized to Elasticsearch | |
| `global_search_awaiting_indexing_queue_size` | Gauge | 13.2 | Number of database updates waiting to be synchronized to Elasticsearch while indexing is paused | |
| `geo_merge_request_diffs` | Gauge | 13.4 | Number of merge request diffs on primary | `url` |
-| `geo_merge_request_diffs_checksum_total` | Gauge | 13.12 | Number of merge request diffs tried to checksum on primary | `url` |
-| `geo_merge_request_diffs_checksummed` | Gauge | 13.4 | Number of merge request diffs successfully checksummed on primary | `url` |
+| `geo_merge_request_diffs_checksum_total` | Gauge | 13.12 | Number of merge request diffs to checksum on primary | `url` |
+| `geo_merge_request_diffs_checksummed` | Gauge | 13.4 | Number of merge request diffs that successfully calculated the checksum on primary | `url` |
| `geo_merge_request_diffs_checksum_failed` | Gauge | 13.4 | Number of merge request diffs failed to calculate the checksum on primary | `url` |
| `geo_merge_request_diffs_synced` | Gauge | 13.4 | Number of syncable merge request diffs synced on secondary | `url` |
| `geo_merge_request_diffs_failed` | Gauge | 13.4 | Number of syncable merge request diffs failed to sync on secondary | `url` |
| `geo_merge_request_diffs_registry` | Gauge | 13.4 | Number of merge request diffs in the registry | `url` |
-| `geo_merge_request_diffs_verification_total` | Gauge | 13.12 | Number of merge request diffs verifications tried on secondary | `url` |
-| `geo_merge_request_diffs_verified` | Gauge | 13.12 | Number of merge request diffs verified on secondary | `url` |
-| `geo_merge_request_diffs_verification_failed` | Gauge | 13.12 | Number of merge request diffs verifications failed on secondary | `url` |
+| `geo_merge_request_diffs_verification_total` | Gauge | 13.12 | Number of merge request diffs to attempt to verify on secondary | `url` |
+| `geo_merge_request_diffs_verified` | Gauge | 13.12 | Number of merge request diffs successfully verified on secondary | `url` |
+| `geo_merge_request_diffs_verification_failed` | Gauge | 13.12 | Number of merge request diffs that failed verification on secondary | `url` |
| `geo_snippet_repositories` | Gauge | 13.4 | Number of snippets on primary | `url` |
| `geo_snippet_repositories_checksummed` | Gauge | 13.4 | Number of snippets checksummed on primary | `url` |
| `geo_snippet_repositories_checksum_failed` | Gauge | 13.4 | Number of snippets failed to calculate the checksum on primary | `url` |
@@ -281,25 +286,25 @@ configuration option in `gitlab.yml`. These metrics are served from the
| `geo_group_wiki_repositories_failed` | Gauge | 13.10 | Number of syncable group wikis failed on secondary | `url` |
| `geo_group_wiki_repositories_registry` | Gauge | 13.10 | Number of syncable group wikis in the registry | `url` |
| `geo_pages_deployments` | Gauge | 14.3 | Number of pages deployments on primary | `url` |
-| `geo_pages_deployments_checksum_total` | Gauge | 14.6 | Number of pages deployments tried to checksum on primary | `url` |
-| `geo_pages_deployments_checksummed` | Gauge | 14.6 | Number of pages deployments successfully checksummed on primary | `url` |
+| `geo_pages_deployments_checksum_total` | Gauge | 14.6 | Number of pages deployments to checksum on primary | `url` |
+| `geo_pages_deployments_checksummed` | Gauge | 14.6 | Number of pages deployments that successfully calculated the checksum on primary | `url` |
| `geo_pages_deployments_checksum_failed` | Gauge | 14.6 | Number of pages deployments failed to calculate the checksum on primary | `url` |
| `geo_pages_deployments_synced` | Gauge | 14.3 | Number of syncable pages deployments synced on secondary | `url` |
| `geo_pages_deployments_failed` | Gauge | 14.3 | Number of syncable pages deployments failed to sync on secondary | `url` |
| `geo_pages_deployments_registry` | Gauge | 14.3 | Number of pages deployments in the registry | `url` |
-| `geo_pages_deployments_verification_total` | Gauge | 14.6 | Number of pages deployments verifications tried on secondary | `url` |
-| `geo_pages_deployments_verified` | Gauge | 14.6 | Number of pages deployments verified on secondary | `url` |
+| `geo_pages_deployments_verification_total` | Gauge | 14.6 | Number of pages deployments to attempt to verify on secondary | `url` |
+| `geo_pages_deployments_verified` | Gauge | 14.6 | Number of pages deployments successfully verified on secondary | `url` |
| `geo_pages_deployments_verification_failed` | Gauge | 14.6 | Number of pages deployments verifications failed on secondary | `url` |
| `geo_job_artifacts` | Gauge | 14.8 | Number of job artifacts on primary | `url` |
-| `geo_job_artifacts_checksum_total` | Gauge | 14.8 | Number of job artifacts tried to checksum on primary | `url` |
-| `geo_job_artifacts_checksummed` | Gauge | 14.8 | Number of job artifacts successfully checksummed on primary | `url` |
+| `geo_job_artifacts_checksum_total` | Gauge | 14.8 | Number of job artifacts to checksum on primary | `url` |
+| `geo_job_artifacts_checksummed` | Gauge | 14.8 | Number of job artifacts that successfully calculated the checksum on primary | `url` |
| `geo_job_artifacts_checksum_failed` | Gauge | 14.8 | Number of job artifacts failed to calculate the checksum on primary | `url` |
| `geo_job_artifacts_synced` | Gauge | 14.8 | Number of syncable job artifacts synced on secondary | `url` |
| `geo_job_artifacts_failed` | Gauge | 14.8 | Number of syncable job artifacts failed to sync on secondary | `url` |
| `geo_job_artifacts_registry` | Gauge | 14.8 | Number of job artifacts in the registry | `url` |
-| `geo_job_artifacts_verification_total` | Gauge | 14.8 | Number of job artifacts verifications tried on secondary | `url` |
-| `geo_job_artifacts_verified` | Gauge | 14.8 | Number of job artifacts verified on secondary | `url` |
-| `geo_job_artifacts_verification_failed` | Gauge | 14.8 | Number of job artifacts verifications failed on secondary | `url` |
+| `geo_job_artifacts_verification_total` | Gauge | 14.8 | Number of job artifacts to attempt to verify on secondary | `url` |
+| `geo_job_artifacts_verified` | Gauge | 14.8 | Number of job artifacts successfully verified on secondary | `url` |
+| `geo_job_artifacts_verification_failed` | Gauge | 14.8 | Number of job artifacts that failed verification on secondary | `url` |
| `limited_capacity_worker_running_jobs` | Gauge | 13.5 | Number of running jobs | `worker` |
| `limited_capacity_worker_max_running_jobs` | Gauge | 13.5 | Maximum number of running jobs | `worker` |
| `limited_capacity_worker_remaining_work_count` | Gauge | 13.5 | Number of jobs waiting to be enqueued | `worker` |
@@ -310,51 +315,66 @@ configuration option in `gitlab.yml`. These metrics are served from the
| `geo_uploads_synced` | Gauge | 14.1 | Number of uploads synced on secondary | `url` |
| `geo_uploads_failed` | Gauge | 14.1 | Number of syncable uploads failed to sync on secondary | `url` |
| `geo_uploads_registry` | Gauge | 14.1 | Number of uploads in the registry | `url` |
-| `geo_uploads_checksum_total` | Gauge | 14.6 | Number of uploads tried to checksum on primary | `url` |
-| `geo_uploads_checksummed` | Gauge | 14.6 | Number of uploads successfully checksummed on primary | `url` |
+| `geo_uploads_checksum_total` | Gauge | 14.6 | Number of uploads to checksum on primary | `url` |
+| `geo_uploads_checksummed` | Gauge | 14.6 | Number of uploads that successfully calculated the checksum on primary | `url` |
| `geo_uploads_checksum_failed` | Gauge | 14.6 | Number of uploads failed to calculate the checksum on primary | `url` |
-| `geo_uploads_verification_total` | Gauge | 14.6 | Number of uploads verifications tried on secondary | `url` |
-| `geo_uploads_verified` | Gauge | 14.6 | Number of uploads verified on secondary | `url` |
-| `geo_uploads_verification_failed` | Gauge | 14.6 | Number of uploads verifications failed on secondary | `url` |
+| `geo_uploads_verification_total` | Gauge | 14.6 | Number of uploads to attempt to verify on secondary | `url` |
+| `geo_uploads_verified` | Gauge | 14.6 | Number of uploads successfully verified on secondary | `url` |
+| `geo_uploads_verification_failed` | Gauge | 14.6 | Number of uploads that failed verification on secondary | `url` |
| `geo_container_repositories` | Gauge | 15.4 | Number of container repositories on primary | `url` |
| `geo_container_repositories_synced` | Gauge | 15.4 | Number of container repositories synced on secondary | `url` |
| `geo_container_repositories_failed` | Gauge | 15.4 | Number of syncable container repositories failed to sync on secondary | `url` |
| `geo_container_repositories_registry` | Gauge | 15.4 | Number of container repositories in the registry | `url` |
-| `gitlab_sli:rails_request_apdex:total` | Counter | 14.4 | The number of request-apdex measurements, [more information the development documentation](../../../development/application_slis/rails_request_apdex.md) | `endpoint_id`, `feature_category`, `request_urgency` |
-| `gitlab_sli:rails_request_apdex:success_total` | Counter | 14.4 | The number of successful requests that met the target duration for their urgency. Divide by `gitlab_sli:rails_requests_apdex:total` to get a success ratio | `endpoint_id`, `feature_category`, `request_urgency` |
+| `geo_container_repositories_checksum_total` | Gauge | 15.10 | Number of container repositories checksummed successfully on primary | `url` |
+| `geo_container_repositories_checksummed` | Gauge | 15.10 | Number of container repositories tried to checksum on primary | `url` |
+| `geo_container_repositories_checksum_failed` | Gauge | 15.10 | Number of container repositories failed to calculate the checksum on primary | `url` |
+| `geo_container_repositories_verification_total` | Gauge | 15.10 | Number of container repositories' verifications tried on secondary | `url` |
+| `geo_container_repositories_verified` | Gauge | 15.10 | Number of container repositories verified on secondary | `url` |
+| `geo_container_repositories_verification_failed` | Gauge | 15.10 | Number of container repositories' failed verifications on secondary | `url` |
| `geo_ci_secure_files` | Gauge | 15.3 | Number of secure files on primary | `url` |
-| `geo_ci_secure_files_checksum_total` | Gauge | 15.3 | Number of secure files tried to checksum on primary | `url` |
-| `geo_ci_secure_files_checksummed` | Gauge | 15.3 | Number of secure files successfully checksummed on primary | `url` |
+| `geo_ci_secure_files_checksum_total` | Gauge | 15.3 | Number of secure files to checksum on primary | `url` |
+| `geo_ci_secure_files_checksummed` | Gauge | 15.3 | Number of secure files that successfully calculated the checksum on primary | `url` |
| `geo_ci_secure_files_checksum_failed` | Gauge | 15.3 | Number of secure files failed to calculate the checksum on primary | `url` |
| `geo_ci_secure_files_synced` | Gauge | 15.3 | Number of syncable secure files synced on secondary | `url` |
| `geo_ci_secure_files_failed` | Gauge | 15.3 | Number of syncable secure files failed to sync on secondary | `url` |
| `geo_ci_secure_files_registry` | Gauge | 15.3 | Number of secure files in the registry | `url` |
-| `geo_ci_secure_files_verification_total` | Gauge | 15.3 | Number of secure files verifications tried on secondary | `url` |
-| `geo_ci_secure_files_verified` | Gauge | 15.3 | Number of secure files verified on secondary | `url` |
-| `geo_ci_secure_files_verification_failed` | Gauge | 15.3 | Number of secure files verifications failed on secondary | `url` |
+| `geo_ci_secure_files_verification_total` | Gauge | 15.3 | Number of secure files to attempt to verify on secondary | `url` |
+| `geo_ci_secure_files_verified` | Gauge | 15.3 | Number of secure files successfully verified on secondary | `url` |
+| `geo_ci_secure_files_verification_failed` | Gauge | 15.3 | Number of secure files that failed verification on secondary | `url` |
| `geo_dependency_proxy_blob` | Gauge | 15.6 | Number of dependency proxy blobs on primary | |
-| `geo_dependency_proxy_blob_checksum_total` | Gauge | 15.6 | Number of dependency proxy blobs tried to checksum on primary | |
-| `geo_dependency_proxy_blob_checksummed` | Gauge | 15.6 | Number of dependency proxy blobs successfully checksummed on primary | |
+| `geo_dependency_proxy_blob_checksum_total` | Gauge | 15.6 | Number of dependency proxy blobs to checksum on primary | |
+| `geo_dependency_proxy_blob_checksummed` | Gauge | 15.6 | Number of dependency proxy blobs that successfully calculated the checksum on primary | |
| `geo_dependency_proxy_blob_checksum_failed` | Gauge | 15.6 | Number of dependency proxy blobs failed to calculate the checksum on primary | |
| `geo_dependency_proxy_blob_synced` | Gauge | 15.6 | Number of dependency proxy blobs synced on secondary | |
| `geo_dependency_proxy_blob_failed` | Gauge | 15.6 | Number of dependency proxy blobs failed to sync on secondary | |
| `geo_dependency_proxy_blob_registry` | Gauge | 15.6 | Number of dependency proxy blobs in the registry | |
-| `geo_dependency_proxy_blob_verification_total` | Gauge | 15.6 | Number of dependency proxy blobs verifications tried on secondary | |
-| `geo_dependency_proxy_blob_verified` | Gauge | 15.6 | Number of dependency proxy blobs verified on secondary | |
-| `geo_dependency_proxy_blob_verification_failed` | Gauge | 15.6 | Number of dependency proxy blobs verifications failed on secondary | |
+| `geo_dependency_proxy_blob_verification_total` | Gauge | 15.6 | Number of dependency proxy blobs to attempt to verify on secondary | |
+| `geo_dependency_proxy_blob_verified` | Gauge | 15.6 | Number of dependency proxy blobs successfully verified on secondary | |
+| `geo_dependency_proxy_blob_verification_failed` | Gauge | 15.6 | Number of dependency proxy blobs that failed verification on secondary | |
| `geo_dependency_proxy_manifests` | Gauge | 15.6 | Number of dependency proxy manifests on primary | `url` |
-| `geo_dependency_proxy_manifests_checksum_total` | Gauge | 15.6 | Number of dependency proxy manifests tried to checksum on primary | `url` |
-| `geo_dependency_proxy_manifests_checksummed` | Gauge | 15.6 | Number of dependency proxy manifests successfully checksummed on primary | `url` |
+| `geo_dependency_proxy_manifests_checksum_total` | Gauge | 15.6 | Number of dependency proxy manifests to checksum on primary | `url` |
+| `geo_dependency_proxy_manifests_checksummed` | Gauge | 15.6 | Number of dependency proxy manifests that successfully calculated the checksum on primary | `url` |
| `geo_dependency_proxy_manifests_checksum_failed` | Gauge | 15.6 | Number of dependency proxy manifests failed to calculate the checksum on primary | `url` |
| `geo_dependency_proxy_manifests_synced` | Gauge | 15.6 | Number of syncable dependency proxy manifests synced on secondary | `url` |
| `geo_dependency_proxy_manifests_failed` | Gauge | 15.6 | Number of syncable dependency proxy manifests failed to sync on secondary | `url` |
| `geo_dependency_proxy_manifests_registry` | Gauge | 15.6 | Number of dependency proxy manifests in the registry | `url` |
-| `geo_dependency_proxy_manifests_verification_total` | Gauge | 15.6 | Number of dependency proxy manifests verifications tried on secondary | `url` |
-| `geo_dependency_proxy_manifests_verified` | Gauge | 15.6 | Number of dependency proxy manifests verified on secondary | `url` |
-| `geo_dependency_proxy_manifests_verification_failed` | Gauge | 15.6 | Number of dependency proxy manifests verifications failed on secondary | `url` |
+| `geo_dependency_proxy_manifests_verification_total` | Gauge | 15.6 | Number of dependency proxy manifests to attempt to verify on secondary | `url` |
+| `geo_dependency_proxy_manifests_verified` | Gauge | 15.6 | Number of dependency proxy manifests successfully verified on secondary | `url` |
+| `geo_dependency_proxy_manifests_verification_failed` | Gauge | 15.6 | Number of dependency proxy manifests that failed verification on secondary | `url` |
+| `geo_project_wiki_repositories` | Gauge | 15.10 | Number of Project Wiki Repositories on primary | `url` |
+| `geo_project_wiki_repositories_checksum_total` | Gauge | 15.10 | Number of Project Wiki Repositories to checksum on primary | `url` |
+| `geo_project_wiki_repositories_checksummed` | Gauge | 15.10 | Number of Project Wiki Repositories that successfully calculated the checksum on primary | `url` |
+| `geo_project_wiki_repositories_checksum_failed` | Gauge | 15.10 | Number of Project Wiki Repositories that failed to calculate the checksum on primary | `url` |
+| `geo_project_wiki_repositories_synced` | Gauge | 15.10 | Number of syncable Project Wiki Repositories synced on secondary | `url` |
+| `geo_project_wiki_repositories_failed` | Gauge | 15.10 | Number of syncable Project Wiki Repositories failed to sync on secondary | `url` |
+| `geo_project_wiki_repositories_registry` | Gauge | 15.10 | Number of Project Wiki Repositories in the registry | `url` |
+| `geo_project_wiki_repositories_verification_total` | Gauge | 15.10 | Number of Project Wiki Repositories to attempt to verify on secondary | `url` |
+| `geo_project_wiki_repositories_verified` | Gauge | 15.10 | Number of Project Wiki Repositories successfully verified on secondary | `url` |
+| `geo_project_wiki_repositories_verification_failed` | Gauge | 15.10 | Number of Project Wiki Repositories that failed verification on secondary | `url` |
| `gitlab_memwd_violations_total` | Counter | 15.9 | Total number of times a Sidekiq process violated a memory threshold | |
| `gitlab_memwd_violations_handled_total` | Counter | 15.9 | Total number of times Sidekiq process memory violations were handled | |
| `sidekiq_watchdog_running_jobs_total` | Counter | 15.9 | Current running jobs when RSS limit was reached | `worker_class` |
+| `gitlab_maintenance_mode` | Gauge | 15.11 | Is GitLab Maintenance Mode enabled? | |
## Database load balancing metrics **(PREMIUM SELF)**
@@ -445,6 +465,7 @@ instance. For example, `cache` or `shared_state`.
| `gitlab_redis_client_exceptions_total` | Counter | 13.2 | Number of Redis client exceptions, broken down by exception class |
| `gitlab_redis_client_requests_total` | Counter | 13.2 | Number of Redis client requests |
| `gitlab_redis_client_requests_duration_seconds` | Histogram | 13.2 | Redis request latency, excluding blocking commands |
+| `gitlab_redis_client_redirections_total` | Counter | 15.10 | Number of Redis Cluster MOVED/ASK redirections, broken down by redirection type |
## Metrics shared directory
diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md
index 56583deca89..013c4515268 100644
--- a/doc/administration/monitoring/prometheus/index.md
+++ b/doc/administration/monitoring/prometheus/index.md
@@ -22,7 +22,7 @@ Prometheus services are on by default.
Prometheus and its exporters don't authenticate users, and are available to anyone who can access
them.
-## Overview
+## How Prometheus works
Prometheus works by periodically connecting to data sources and collecting their
performance metrics through the [various exporters](#bundled-software-metrics). To view
@@ -201,7 +201,10 @@ To use an external Prometheus server:
postgres_exporter['listen_address'] = '0.0.0.0:9187'
# Gitaly nodes
- gitaly['prometheus_listen_addr'] = "0.0.0.0:9236"
+ gitaly['configuration'] = {
+ # ...
+ prometheus_listen_addr: '0.0.0.0:9236',
+ }
```
1. Install and set up a dedicated Prometheus instance, if necessary, using the [official installation instructions](https://prometheus.io/docs/prometheus/latest/installation/).
diff --git a/doc/administration/monitoring/prometheus/node_exporter.md b/doc/administration/monitoring/prometheus/node_exporter.md
index c2f84773b3a..337ab9becaa 100644
--- a/doc/administration/monitoring/prometheus/node_exporter.md
+++ b/doc/administration/monitoring/prometheus/node_exporter.md
@@ -1,6 +1,6 @@
---
-stage: Monitor
-group: Respond
+stage: Shared responsibility based on functional area
+group: Shared responsibility based on functional area
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
diff --git a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md
index 56da3155fd9..f8d48832288 100644
--- a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md
+++ b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md
@@ -1,6 +1,6 @@
---
-stage: Monitor
-group: Respond
+stage: Data Stores
+group: Database
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
diff --git a/doc/administration/monitoring/prometheus/postgres_exporter.md b/doc/administration/monitoring/prometheus/postgres_exporter.md
index 0458a4a5bae..1cae28a069f 100644
--- a/doc/administration/monitoring/prometheus/postgres_exporter.md
+++ b/doc/administration/monitoring/prometheus/postgres_exporter.md
@@ -1,6 +1,6 @@
---
-stage: Monitor
-group: Respond
+stage: Data Stores
+group: Database
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
diff --git a/doc/administration/monitoring/prometheus/redis_exporter.md b/doc/administration/monitoring/prometheus/redis_exporter.md
index b50a2a4aebf..b17bf9787a7 100644
--- a/doc/administration/monitoring/prometheus/redis_exporter.md
+++ b/doc/administration/monitoring/prometheus/redis_exporter.md
@@ -1,6 +1,6 @@
---
-stage: Monitor
-group: Respond
+stage: Shared responsibility based on functional area
+group: Shared responsibility based on functional area
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
diff --git a/doc/administration/monitoring/prometheus/registry_exporter.md b/doc/administration/monitoring/prometheus/registry_exporter.md
index b79e97bd937..3f4d6cfb14d 100644
--- a/doc/administration/monitoring/prometheus/registry_exporter.md
+++ b/doc/administration/monitoring/prometheus/registry_exporter.md
@@ -1,6 +1,6 @@
---
-stage: Monitor
-group: Respond
+stage: Package
+group: Package Registry
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md
index 7349e8ad9ba..3e3712c9645 100644
--- a/doc/administration/nfs.md
+++ b/doc/administration/nfs.md
@@ -12,7 +12,7 @@ recommended for performance reasons.
For data objects such as LFS, Uploads, and Artifacts, an [Object Storage service](object_storage.md)
is recommended over NFS where possible, due to better performance.
-When eliminating the usage of NFS, there are [additional steps you need to take](object_storage.md#other-alternatives-to-file-system-storage)
+When eliminating the usage of NFS, there are [additional steps you need to take](object_storage.md#alternatives-to-file-system-storage)
in addition to moving to Object Storage.
File system performance can impact overall GitLab performance, especially for
diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md
index 4b3e251d407..1e6605e41a8 100644
--- a/doc/administration/object_storage.md
+++ b/doc/administration/object_storage.md
@@ -11,101 +11,428 @@ It's recommended over NFS and
in general it's better in larger setups as object storage is
typically much more performant, reliable, and scalable.
-## Options
+To configure the object storage, you have two options:
-GitLab is tightly integrated with `Fog`, so you can refer to its
-[documentation](https://fog.io/about/provider_documentation.html) to check
-which storage services can be integrated with GitLab.
+- Recommended. [Configure a single storage connection for all object types](#configure-a-single-storage-connection-for-all-object-types-consolidated-form):
+ A single credential is shared by all supported object types. This is called the consolidated form.
+- [Configure each object type to define its own storage connection](#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form):
+ Every object defines its own object storage connection and configuration. This is called the storage-specific form.
+
+ If you already use the storage-specific form, see how to
+ [transition to the consolidated form](#transition-to-consolidated-form).
+
+If you store data locally, see how to
+[migrate to object storage](#migrate-to-object-storage).
+
+## Supported object storage providers
+
+GitLab is tightly integrated with the Fog library, so you can see which
+[providers](https://fog.io/about/provider_documentation.html) can be used
+with GitLab.
Specifically, GitLab has been tested by vendors and customers on a number of object storage providers:
-- [Amazon S3](https://aws.amazon.com/s3/)
+- [Amazon S3](https://aws.amazon.com/s3/) ([Object Lock](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html)
+ is not supported, see [issue #335775](https://gitlab.com/gitlab-org/gitlab/-/issues/335775)
+ for more information)
- [Google Cloud Storage](https://cloud.google.com/storage)
-- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces)
+- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces) (S3 compatible)
- [Oracle Cloud Infrastructure](https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm)
- [OpenStack Swift (S3 compatible mode)](https://docs.openstack.org/swift/latest/s3_compat.html)
- [Azure Blob storage](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction)
+- [MinIO](https://min.io/) (S3 compatible)
- On-premises hardware and appliances from various storage vendors, whose list is not officially established.
-- MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation.
-### Known compatibility issues
+## Configure a single storage connection for all object types (consolidated form)
-- Dell EMC ECS: Prior to GitLab 13.3, there is a
- [known bug in GitLab Workhorse that prevents HTTP Range Requests from working with CI job artifacts](https://gitlab.com/gitlab-org/gitlab/-/issues/223806).
- Be sure to upgrade to GitLab 13.3.0 or above if you use S3 storage with this hardware.
+> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4368) in GitLab 13.2.
-- Ceph S3 prior to [Kraken 11.0.2](https://ceph.com/releases/kraken-11-0-2-released/) does not support the [Upload Copy Part API](https://gitlab.com/gitlab-org/gitlab/-/issues/300604). You may need to [disable multi-threaded copying](#multi-threaded-copying).
+Most types of objects, such as CI artifacts, LFS files, and upload attachments
+can be saved in object storage by specifying a single credential for object
+storage with multiple buckets.
-- Amazon S3 [Object Lock](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html)
- is not supported. Follow [issue #335775](https://gitlab.com/gitlab-org/gitlab/-/issues/335775)
- for progress on enabling this option.
+Configuring the object storage using the consolidated form has a number of advantages:
-## Configuration guides
+- It can simplify your GitLab configuration since the connection details are shared
+ across object types.
+- It enables the use of [encrypted S3 buckets](#encrypted-s3-buckets).
+- It [uploads files to S3 with proper `Content-MD5` headers](https://gitlab.com/gitlab-org/gitlab-workhorse/-/issues/222).
-There are two ways of specifying object storage configuration in GitLab:
+When the consolidated form is used,
+[direct upload](../development/uploads/index.md#direct-upload) is enabled
+automatically. Thus, only the following providers can be used:
-- [Consolidated form](#consolidated-object-storage-configuration): A single credential is
- shared by all supported object types.
-- [Storage-specific form](#storage-specific-configuration): Every object defines its
- own object storage [connection and configuration](#connection-settings).
+- [Amazon S3-compatible providers](#amazon-s3)
+- [Google Cloud Storage](#google-cloud-storage-gcs)
+- [Azure Blob storage](#azure-blob-storage)
-For more information on the differences and to transition from one form to another, see
-[Transition to consolidated form](#transition-to-consolidated-form).
+The consolidated form configuration can't be used for backups or
+Mattermost. Backups can be configured with
+[server side encryption](../raketasks/backup_gitlab.md#s3-encrypted-buckets)
+separately. See the
+[table for a complete list](#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form)
+of supported object storage types.
-If you are currently storing data locally, see
-[Migrate to object storage](#migrate-to-object-storage) for migration details.
+Enabling the consolidated form enables object storage for all object
+types. If not all buckets are specified, you may see an error like:
-### Consolidated object storage configuration
+```plaintext
+Object storage for <object type> must have a bucket specified
+```
-> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4368) in GitLab 13.2.
+If you want to use local storage for specific object types, you can
+[disable object storage for specific features](#disable-object-storage-for-specific-features).
-Using the consolidated object storage configuration has a number of advantages:
+### Configure the common parameters
-- It can simplify your GitLab configuration since the connection details are shared
- across object types.
-- It enables the use of [encrypted S3 buckets](#encrypted-s3-buckets).
-- It [uploads files to S3 with proper `Content-MD5` headers](https://gitlab.com/gitlab-org/gitlab-workhorse/-/issues/222).
+In the consolidated form, the `object_store` section defines a
+common set of parameters.
+
+| Setting | Description |
+|-------------------|-----------------------------------|
+| `enabled` | Enable or disable object storage. |
+| `proxy_download` | Set to `true` to [enable proxying all files served](#proxy-download). Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data. |
+| `connection` | Various [connection options](#configure-the-connection-settings) described below. |
+| `storage_options` | Options to use when saving new objects, such as [server side encryption](#server-side-encryption-headers). Introduced in GitLab 13.3. |
+| `objects` | [Object-specific configuration](#configure-the-parameters-of-each-object). |
-Because [direct upload mode](../development/uploads/index.md#direct-upload)
-must be enabled, only the following providers can be used:
+For an example, see how to [use the consolidated form and Amazon S3](#full-example-using-the-consolidated-form-and-amazon-s3).
-- [Amazon S3-compatible providers](#s3-compatible-connection-settings)
-- [Google Cloud Storage](#google-cloud-storage-gcs)
-- [Azure Blob storage](#azure-blob-storage)
+### Configure the parameters of each object
+
+Each object type must at least define the bucket name where it will be stored.
+
+The following table lists the valid `objects` that can be used:
+
+| Type | Description |
+|--------------------|----------------------------------------------------------------------------|
+| `artifacts` | [CI artifacts](job_artifacts.md) |
+| `external_diffs` | [Merge request diffs](merge_request_diffs.md) |
+| `uploads` | [User uploads](uploads.md) |
+| `lfs` | [Git Large File Storage objects](lfs/index.md) |
+| `packages` | [Project packages (for example, PyPI, Maven, or NuGet)](packages/index.md) |
+| `dependency_proxy` | [Dependency Proxy](packages/dependency_proxy.md) |
+| `terraform_state` | [Terraform state files](terraform_state.md) |
+| `pages` | [Pages](pages/index.md) |
+
+Within each object type, three parameters can be defined:
+
+| Setting | Required? | Description |
+|------------------|------------------------|-------------------------------------|
+| `bucket` | **{check-circle}** Yes\* | Bucket name for the object type. Not required if `enabled` is set to `false`. |
+| `enabled` | **{dotted-circle}** No | Overrides the [common parameter](#configure-the-common-parameters). |
+| `proxy_download` | **{dotted-circle}** No | Overrides the [common parameter](#configure-the-common-parameters). |
+
+For an example, see how to [use the consolidated form and Amazon S3](#full-example-using-the-consolidated-form-and-amazon-s3).
+
+#### Disable object storage for specific features
+
+As seen above, object storage can be disabled for specific types by
+setting the `enabled` flag to `false`. For example, to disable object
+storage for CI artifacts:
+
+```ruby
+gitlab_rails['object_store']['objects']['artifacts']['enabled'] = false
+```
+
+A bucket is not needed if the feature is disabled entirely. For example,
+no bucket is needed if CI artifacts are disabled with this setting:
+
+```ruby
+gitlab_rails['artifacts_enabled'] = false
+```
-When consolidated object storage is used, direct upload is enabled
-automatically. For storage-specific
-configuration, [direct upload may become the default](https://gitlab.com/gitlab-org/gitlab/-/issues/27331)
+## Configure each object type to define its own storage connection (storage-specific form)
+
+With the storage-specific form, every object defines its own object
+storage connection and configuration. If you're using GitLab 13.2 and later,
+you should [transition to the consolidated form](#transition-to-consolidated-form).
+
+The use of [encrypted S3 buckets](#encrypted-s3-buckets) with non-consolidated form is not supported.
+You may get [ETag mismatch errors](#etag-mismatch) if you use it.
+
+NOTE:
+For the storage-specific form,
+[direct upload may become the default](https://gitlab.com/gitlab-org/gitlab/-/issues/27331)
because it does not require a shared folder.
-Consolidated object storage configuration can't be used for backups or
-Mattermost. See the [full table for a complete list](#storage-specific-configuration).
-However, backups can be configured with [server side encryption](../raketasks/backup_gitlab.md#s3-encrypted-buckets) separately.
+For configuring object storage in GitLab 13.1 and earlier, or for storage types not
+supported by consolidated form, refer to the following guides:
+
+| Object storage type | Supported by consolidated form? |
+|---------------------|------------------------------------------|
+| [Backups](../raketasks/backup_gitlab.md#upload-backups-to-a-remote-cloud-storage) | **{dotted-circle}** No |
+| [Container Registry](packages/container_registry.md#use-object-storage) (optional feature) | **{dotted-circle}** No |
+| [Mattermost](https://docs.mattermost.com/configure/file-storage-configuration-settings.html)| **{dotted-circle}** No |
+| [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | **{dotted-circle}** No |
+| [Job artifacts](job_artifacts.md#using-object-storage) including archived job logs | **{check-circle}** Yes |
+| [LFS objects](lfs/index.md#storing-lfs-objects-in-remote-object-storage) | **{check-circle}** Yes |
+| [Uploads](uploads.md#using-object-storage) | **{check-circle}** Yes |
+| [Merge request diffs](merge_request_diffs.md#using-object-storage) | **{check-circle}** Yes |
+| [Packages](packages/index.md#use-object-storage) (optional feature) | **{check-circle}** Yes |
+| [Dependency Proxy](packages/dependency_proxy.md#using-object-storage) (optional feature) | **{check-circle}** Yes |
+| [Terraform state files](terraform_state.md#using-object-storage) | **{check-circle}** Yes |
+| [Pages content](pages/index.md#using-object-storage) | **{check-circle}** Yes |
+
+## Configure the connection settings
+
+Both consolidated and storage-specific form must configure a connection. The following sections describe parameters that can be used
+in the `connection` setting.
+
+### Amazon S3
+
+The connection settings match those provided by [fog-aws](https://github.com/fog/fog-aws):
+
+| Setting | Description | Default |
+|---------------------------------------------|------------------------------------|---------|
+| `provider` | Always `AWS` for compatible hosts. | `AWS` |
+| `aws_access_key_id` | AWS credentials, or compatible. | |
+| `aws_secret_access_key` | AWS credentials, or compatible. | |
+| `aws_signature_version` | AWS signature version to use. `2` or `4` are valid options. Digital Ocean Spaces and other providers may need `2`. | `4` |
+| `enable_signature_v4_streaming` | Set to `true` to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be `false`. | `true` |
+| `region` | AWS region. | |
+| `host` | DEPRECATED: Use `endpoint` instead. S3 compatible host for when not using AWS. For example, `localhost` or `storage.example.com`. HTTPS and port 443 is assumed. | `s3.amazonaws.com` |
+| `endpoint` | Can be used when configuring an S3 compatible service such as [MinIO](https://min.io), by entering a URL such as `http://127.0.0.1:9000`. This takes precedence over `host`. Always use `endpoint` for consolidated form. | (optional) |
+| `path_style` | Set to `true` to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Set to `true` for using [MinIO](https://min.io). Leave as `false` for AWS S3. | `false`. |
+| `use_iam_profile` | Set to `true` to use IAM profile instead of access keys. | `false` |
+| `aws_credentials_refresh_threshold_seconds` | Sets the [automatic refresh threshold](https://github.com/fog/fog-aws#controlling-credential-refresh-time-with-iam-authentication) in seconds when using temporary credentials in IAM. | `15` |
+
+#### Use Amazon instance profiles
+
+Instead of supplying AWS access and secret keys in object storage
+configuration, you can configure GitLab to use Amazon Identity Access and Management (IAM) roles to set up an
+[Amazon instance profile](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html).
+When this is used, GitLab fetches temporary credentials each time an
+S3 bucket is accessed, so no hard-coded values are needed in the
+configuration.
+
+Prerequisites:
+
+- GitLab must be able to connect to the
+ [instance metadata endpoint](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html).
+- If GitLab is [configured to use an internet proxy](https://docs.gitlab.com/omnibus/settings/environment-variables.html), the endpoint IP
+ address must be added to the `no_proxy` list.
+
+To set up an instance profile:
+
+1. Create an IAM role with the necessary permissions. The
+ following example is a role for an S3 bucket named `test-bucket`:
+
+ ```json
+ {
+ "Version": "2012-10-17",
+ "Statement": [
+ {
+ "Sid": "VisualEditor0",
+ "Effect": "Allow",
+ "Action": [
+ "s3:PutObject",
+ "s3:GetObject",
+ "s3:DeleteObject"
+ ],
+ "Resource": "arn:aws:s3:::test-bucket/*"
+ }
+ ]
+ }
+ ```
+
+1. [Attach this role](https://aws.amazon.com/premiumsupport/knowledge-center/attach-replace-ec2-instance-profile/)
+ to the EC2 instance hosting your GitLab instance.
+1. Set the `use_iam_profile` GitLab configuration option to `true`.
+
+#### Encrypted S3 buckets
+
+> - [Introduced](https://gitlab.com/gitlab-org/gitlab-workhorse/-/merge_requests/466) in GitLab 13.1 for instance profiles only and [S3 default encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/bucket-encryption.html).
+> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/34460) in GitLab 13.2 for static credentials when the [consolidated form](#configure-a-single-storage-connection-for-all-object-types-consolidated-form) and [S3 default encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/bucket-encryption.html) is used.
+
+When configured either with an instance profile or with the consolidated
+form, GitLab Workhorse properly uploads files to S3
+buckets that have [SSE-S3 or SSE-KMS encryption enabled by default](https://docs.aws.amazon.com/kms/latest/developerguide/services-s3.html).
+AWS KMS keys and SSE-C encryption are
+[not supported since this requires sending the encryption keys in every request](https://gitlab.com/gitlab-org/gitlab/-/issues/226006).
+
+#### Server-side encryption headers
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38240) in GitLab 13.3.
-Enabling consolidated object storage enables object storage for all object
-types. If not all buckets are specified, `sudo gitlab-ctl reconfigure` may fail with the error like:
+Setting a default encryption on an S3 bucket is the easiest way to
+enable encryption, but you may want to
+[set a bucket policy to ensure only encrypted objects are uploaded](https://repost.aws/knowledge-center/s3-bucket-store-kms-encrypted-objects).
+To do this, you must configure GitLab to send the proper encryption headers
+in the `storage_options` configuration section:
+
+| Setting | Description |
+|-------------------------------------|------------------------------------------|
+| `server_side_encryption` | Encryption mode (`AES256` or `aws:kms`). |
+| `server_side_encryption_kms_key_id` | Amazon Resource Name. Only needed when `aws:kms` is used in `server_side_encryption`. See the [Amazon documentation on using KMS encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html). |
+
+As with the case for default encryption, these options only work when
+the Workhorse S3 client is enabled. One of the following two conditions
+must be fulfilled:
+
+- `use_iam_profile` is `true` in the connection settings.
+- Consolidated form is in use.
+
+[ETag mismatch errors](#etag-mismatch) occur if server side
+encryption headers are used without enabling the Workhorse S3 client.
+
+### Oracle Cloud S3
+
+Oracle Cloud S3 must be sure to use the following settings:
+
+| Setting | Value |
+|---------------------------------|---------|
+| `enable_signature_v4_streaming` | `false` |
+| `path_style` | `true` |
+
+If `enable_signature_v4_streaming` is set to `true`, you may see the
+following error in `production.log`:
```plaintext
-Object storage for <object type> must have a bucket specified
+STREAMING-AWS4-HMAC-SHA256-PAYLOAD is not supported
```
-If you want to use local storage for specific object types, you can
-[selectively disable object storages](#selectively-disabling-object-storage).
+### Google Cloud Storage (GCS)
-Most types of objects, such as CI artifacts, LFS files, and upload
-attachments can be saved in object storage by specifying a single
-credential for object storage with multiple buckets.
+Here are the valid connection parameters for GCS:
-When the consolidated form is:
+| Setting | Description | Example |
+|------------------------------|-------------------|---------|
+| `provider` | Provider name. | `Google` |
+| `google_project` | GCP project name. | `gcp-project-12345` |
+| `google_json_key_location` | JSON key path. | `/path/to/gcp-project-12345-abcde.json` |
+| `google_json_key_string` | JSON key string. | `{ "type": "service_account", "project_id": "example-project-382839", ... }` |
+| `google_application_default` | Set to `true` to use [Google Cloud Application Default Credentials](https://cloud.google.com/docs/authentication#adc) to locate service account credentials. | |
-- Used with an S3-compatible object storage, Workhorse uses its internal S3 client to
- upload files.
-- Not used with an S3-compatible object storage, Workhorse falls back to using
- pre-signed URLs.
+GitLab reads the value of `google_json_key_location`, then `google_json_key_string`, and finally, `google_application_default`.
+It uses the first of these settings that has a value.
+
+The service account must have permission to access the bucket. For more information,
+see the [Cloud Storage authentication documentation](https://cloud.google.com/storage/docs/authentication).
+
+NOTE:
+Bucket encryption with the [Cloud Key Management Service (KMS)](https://cloud.google.com/kms/docs) is not supported and results in [ETag mismatch errors](#etag-mismatch).
+
+#### GCS example
+
+For Omnibus installations, this is an example of the `connection` setting
+in the consolidated form:
+
+```ruby
+gitlab_rails['object_store']['connection'] = {
+ 'provider' => 'Google',
+ 'google_project' => '<GOOGLE PROJECT>',
+ 'google_json_key_location' => '<FILENAME>'
+}
+```
+
+#### GCS example with ADC
-See the section on [ETag mismatch errors](#etag-mismatch) for more details.
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/275979) in GitLab 13.6.
+
+Google Cloud Application Default Credentials (ADC) are typically
+used with GitLab to use the default service account. This eliminates the
+need to supply credentials for the instance. For example, in the consolidated form:
+
+```ruby
+gitlab_rails['object_store']['connection'] = {
+ 'provider' => 'Google',
+ 'google_project' => '<GOOGLE PROJECT>',
+ 'google_application_default' => true
+}
+```
+
+If you use ADC, be sure that:
+
+- The service account that you use has the
+[`iam.serviceAccounts.signBlob` permission](https://cloud.google.com/iam/docs/reference/credentials/rest/v1/projects.serviceAccounts/signBlob).
+ Typically this is done by granting the `Service Account Token Creator` role to the service account.
+- Your virtual machines have the [correct access scopes to access Google Cloud APIs](https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances#changeserviceaccountandscopes). If the machines do not have the right scope, the error logs may show:
+
+ ```markdown
+ Google::Apis::ClientError (insufficientPermissions: Request had insufficient authentication scopes.)
+ ```
+
+### Azure Blob storage
-#### Use AWS S3
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25877) in GitLab 13.4.
+
+Although Azure uses the word `container` to denote a collection of
+blobs, GitLab standardizes on the term `bucket`. Be sure to configure
+Azure container names in the `bucket` settings.
+
+Azure Blob storage can only be used with the [consolidated form](#configure-a-single-storage-connection-for-all-object-types-consolidated-form)
+because a single set of credentials are used to access multiple
+containers. The [storage-specific form](#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form)
+is not supported. For more details, see [how to transition to consolidated form](#transition-to-consolidated-form).
+
+The following are the valid connection parameters for Azure. For more information, see the
+[Azure Blob Storage documentation](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction).
+
+| Setting | Description | Example |
+|------------------------------|----------------|-----------|
+| `provider` | Provider name. | `AzureRM` |
+| `azure_storage_account_name` | Name of the Azure Blob Storage account used to access the storage. | `azuretest` |
+| `azure_storage_access_key` | Storage account access key used to access the container. This is typically a secret, 512-bit encryption key encoded in base64. | `czV2OHkvQj9FKEgrTWJRZVRoV21ZcTN0Nnc5eiRDJkYpSkBOY1JmVWpYbjJy\nNHU3eCFBJUQqRy1LYVBkU2dWaw==\n` |
+| `azure_storage_domain` | Domain name used to contact the Azure Blob Storage API (optional). Defaults to `blob.core.windows.net`. Set this if you are using Azure China, Azure Germany, Azure US Government, or some other custom Azure domain. | `blob.core.windows.net` |
+
+- For Omnibus installations, this is an example of the `connection` setting
+ in the consolidated form:
+
+ ```ruby
+ gitlab_rails['object_store']['connection'] = {
+ 'provider' => 'AzureRM',
+ 'azure_storage_account_name' => '<AZURE STORAGE ACCOUNT NAME>',
+ 'azure_storage_access_key' => '<AZURE STORAGE ACCESS KEY>',
+ 'azure_storage_domain' => '<AZURE STORAGE DOMAIN>'
+ }
+ ```
+
+- For source installations, Workhorse also needs to be configured with Azure
+ credentials. This isn't needed in Omnibus installs, because the Workhorse
+ settings are populated from the previous settings.
+
+ 1. Edit `/home/git/gitlab-workhorse/config.toml` and add or amend the following lines:
+
+ ```toml
+ [object_storage]
+ provider = "AzureRM"
+
+ [object_storage.azurerm]
+ azure_storage_account_name = "<AZURE STORAGE ACCOUNT NAME>"
+ azure_storage_access_key = "<AZURE STORAGE ACCESS KEY>"
+ ```
+
+ If you are using a custom Azure storage domain,
+ `azure_storage_domain` does **not** have to be set in the Workhorse
+ configuration. This information is exchanged in an API call between
+ GitLab Rails and Workhorse.
+
+### Storj Gateway (SJ)
+
+NOTE:
+The Storj Gateway [does not support](https://github.com/storj/gateway-st/blob/4b74c3b92c63b5de7409378b0d1ebd029db9337d/docs/s3-compatibility.md) multi-threaded copying (see `UploadPartCopy` in the table).
+While an implementation [is planned](https://github.com/storj/roadmap/issues/40), you must [disable multi-threaded copying](#multi-threaded-copying) until completion.
+
+The [Storj Network](https://www.storj.io/) provides an S3-compatible API gateway. Use the following configuration example:
+
+```ruby
+gitlab_rails['object_store']['connection'] = {
+ 'provider' => 'AWS',
+ 'endpoint' => 'https://gateway.storjshare.io',
+ 'path_style' => true,
+ 'region' => 'eu1',
+ 'aws_access_key_id' => 'ACCESS_KEY',
+ 'aws_secret_access_key' => 'SECRET_KEY',
+ 'aws_signature_version' => 2,
+ 'enable_signature_v4_streaming' => false
+}
+```
+
+The signature version must be `2`. Using v4 results in a HTTP 411 Length Required error.
+For more information, see [issue #4419](https://gitlab.com/gitlab-org/gitlab/-/issues/4419).
+
+## Full example using the consolidated form and Amazon S3
The following example uses AWS S3 to enable object storage for all supported services:
@@ -116,42 +443,42 @@ The following example uses AWS S3 to enable object storage for all supported ser
1. Edit `/etc/gitlab/gitlab.rb` and add the following lines, substituting
the values you want:
- ```ruby
- # Consolidated object storage configuration
- gitlab_rails['object_store']['enabled'] = true
- gitlab_rails['object_store']['proxy_download'] = true
- gitlab_rails['object_store']['connection'] = {
- 'provider' => 'AWS',
- 'region' => 'eu-central-1',
- 'aws_access_key_id' => '<AWS_ACCESS_KEY_ID>',
- 'aws_secret_access_key' => '<AWS_SECRET_ACCESS_KEY>'
- }
- # OPTIONAL: The following lines are only needed if server side encryption is required
- gitlab_rails['object_store']['storage_options'] = {
- 'server_side_encryption' => '<AES256 or aws:kms>',
- 'server_side_encryption_kms_key_id' => '<arn:aws:kms:xxx>'
- }
- gitlab_rails['object_store']['objects']['artifacts']['bucket'] = 'gitlab-artifacts'
- gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = 'gitlab-mr-diffs'
- gitlab_rails['object_store']['objects']['lfs']['bucket'] = 'gitlab-lfs'
- gitlab_rails['object_store']['objects']['uploads']['bucket'] = 'gitlab-uploads'
- gitlab_rails['object_store']['objects']['packages']['bucket'] = 'gitlab-packages'
- gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = 'gitlab-dependency-proxy'
- gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = 'gitlab-terraform-state'
- gitlab_rails['object_store']['objects']['ci_secure_files']['bucket'] = 'gitlab-ci-secure-files'
- gitlab_rails['object_store']['objects']['pages']['bucket'] = 'gitlab-pages'
- ```
-
- If you’re using [AWS IAM profiles](#using-amazon-instance-profiles), omit
- the AWS access key and secret access key/value pairs. For example:
-
- ```ruby
- gitlab_rails['object_store']['connection'] = {
- 'provider' => 'AWS',
- 'region' => 'eu-central-1',
- 'use_iam_profile' => true
- }
- ```
+ ```ruby
+ # Consolidated object storage configuration
+ gitlab_rails['object_store']['enabled'] = true
+ gitlab_rails['object_store']['proxy_download'] = true
+ gitlab_rails['object_store']['connection'] = {
+ 'provider' => 'AWS',
+ 'region' => 'eu-central-1',
+ 'aws_access_key_id' => '<AWS_ACCESS_KEY_ID>',
+ 'aws_secret_access_key' => '<AWS_SECRET_ACCESS_KEY>'
+ }
+ # OPTIONAL: The following lines are only needed if server side encryption is required
+ gitlab_rails['object_store']['storage_options'] = {
+ 'server_side_encryption' => '<AES256 or aws:kms>',
+ 'server_side_encryption_kms_key_id' => '<arn:aws:kms:xxx>'
+ }
+ gitlab_rails['object_store']['objects']['artifacts']['bucket'] = 'gitlab-artifacts'
+ gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = 'gitlab-mr-diffs'
+ gitlab_rails['object_store']['objects']['lfs']['bucket'] = 'gitlab-lfs'
+ gitlab_rails['object_store']['objects']['uploads']['bucket'] = 'gitlab-uploads'
+ gitlab_rails['object_store']['objects']['packages']['bucket'] = 'gitlab-packages'
+ gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = 'gitlab-dependency-proxy'
+ gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = 'gitlab-terraform-state'
+ gitlab_rails['object_store']['objects']['ci_secure_files']['bucket'] = 'gitlab-ci-secure-files'
+ gitlab_rails['object_store']['objects']['pages']['bucket'] = 'gitlab-pages'
+ ```
+
+ If you're using [AWS IAM profiles](#use-amazon-instance-profiles), omit
+ the AWS access key and secret access key/value pairs. For example:
+
+ ```ruby
+ gitlab_rails['object_store']['connection'] = {
+ 'provider' => 'AWS',
+ 'region' => 'eu-central-1',
+ 'use_iam_profile' => true
+ }
+ ```
1. Save the file and reconfigure GitLab:
@@ -171,7 +498,7 @@ The following example uses AWS S3 to enable object storage for all supported ser
aws_secret_access_key: <AWS_SECRET_ACCESS_KEY>
```
- If you’re using [AWS IAM profiles](#using-amazon-instance-profiles), omit
+ If you're using [AWS IAM profiles](#use-amazon-instance-profiles), omit
the AWS access key and secret access key/value pairs. For example:
```yaml
@@ -293,7 +620,7 @@ The following example uses AWS S3 to enable object storage for all supported ser
gitlab_rails['object_store']['objects']['pages']['bucket'] = 'gitlab-pages'
```
- If you’re using [AWS IAM profiles](#using-amazon-instance-profiles), omit
+ If you're using [AWS IAM profiles](#use-amazon-instance-profiles), omit
the AWS access key and secret access key/value pairs. For example:
```ruby
@@ -348,7 +675,7 @@ The following example uses AWS S3 to enable object storage for all supported ser
bucket: gitlab-pages
```
- If you’re using [AWS IAM profiles](#using-amazon-instance-profiles), omit
+ If you're using [AWS IAM profiles](#use-amazon-instance-profiles), omit
the AWS access key and secret access key/value pairs. For example:
```yaml
@@ -369,7 +696,7 @@ The following example uses AWS S3 to enable object storage for all supported ser
aws_secret_access_key = "<AWS_SECRET_ACCESS_KEY>"
```
- If you’re using [AWS IAM profiles](#using-amazon-instance-profiles), omit
+ If you're using [AWS IAM profiles](#use-amazon-instance-profiles), omit
the AWS access key and secret access key/value pairs. For example:
```yaml
@@ -389,311 +716,7 @@ The following example uses AWS S3 to enable object storage for all supported ser
::EndTabs
-#### Common parameters
-
-In the consolidated configuration, the `object_store` section defines a
-common set of parameters. Here we use the YAML from the source
-installation because it's easier to see the inheritance:
-
-```yaml
- object_store:
- enabled: true
- proxy_download: true
- connection:
- provider: AWS
- aws_access_key_id: <AWS_ACCESS_KEY_ID>
- aws_secret_access_key: <AWS_SECRET_ACCESS_KEY>
- objects:
- ...
-```
-
-The Omnibus configuration maps directly to this:
-
-```ruby
-gitlab_rails['object_store']['enabled'] = true
-gitlab_rails['object_store']['proxy_download'] = true
-gitlab_rails['object_store']['connection'] = {
- 'provider' => 'AWS',
- 'aws_access_key_id' => '<AWS_ACCESS_KEY_ID',
- 'aws_secret_access_key' => '<AWS_SECRET_ACCESS_KEY>'
-}
-```
-
-| Setting | Description |
-|-------------------|-----------------------------------|
-| `enabled` | Enable or disable object storage. |
-| `proxy_download` | Set to `true` to [enable proxying all files served](#proxy-download). Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data. |
-| `connection` | Various [connection options](#connection-settings) described below. |
-| `storage_options` | Options to use when saving new objects, such as [server side encryption](#server-side-encryption-headers). Introduced in GitLab 13.3. |
-| `objects` | [Object-specific configuration](#object-specific-configuration). |
-
-### Connection settings
-
-Both consolidated configuration form and storage-specific configuration form must configure a connection. The following sections describe parameters that can be used
-in the `connection` setting.
-
-#### S3-compatible connection settings
-
-The connection settings match those provided by [fog-aws](https://github.com/fog/fog-aws):
-
-| Setting | Description | Default |
-|---------------------------------------------|------------------------------------|---------|
-| `provider` | Always `AWS` for compatible hosts. | `AWS` |
-| `aws_access_key_id` | AWS credentials, or compatible. | |
-| `aws_secret_access_key` | AWS credentials, or compatible. | |
-| `aws_signature_version` | AWS signature version to use. `2` or `4` are valid options. Digital Ocean Spaces and other providers may need `2`. | `4` |
-| `enable_signature_v4_streaming` | Set to `true` to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be `false`. | `true` |
-| `region` | AWS region. | |
-| `host` | DEPRECATED: Use `endpoint` instead. S3 compatible host for when not using AWS. For example, `localhost` or `storage.example.com`. HTTPS and port 443 is assumed. | `s3.amazonaws.com` |
-| `endpoint` | Can be used when configuring an S3 compatible service such as [MinIO](https://min.io), by entering a URL such as `http://127.0.0.1:9000`. This takes precedence over `host`. Always use `endpoint` for consolidated form. | (optional) |
-| `path_style` | Set to `true` to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Set to `true` for using [MinIO](https://min.io). Leave as `false` for AWS S3. | `false`. |
-| `use_iam_profile` | Set to `true` to use IAM profile instead of access keys. | `false` |
-| `aws_credentials_refresh_threshold_seconds` | Sets the [automatic refresh threshold](https://github.com/fog/fog-aws#controlling-credential-refresh-time-with-iam-authentication) when using temporary credentials in IAM. | `15` |
-
-#### Oracle Cloud S3 connection settings
-
-Oracle Cloud S3 must be sure to use the following settings:
-
-| Setting | Value |
-|---------------------------------|---------|
-| `enable_signature_v4_streaming` | `false` |
-| `path_style` | `true` |
-
-If `enable_signature_v4_streaming` is set to `true`, you may see the
-following error in `production.log`:
-
-```plaintext
-STREAMING-AWS4-HMAC-SHA256-PAYLOAD is not supported
-```
-
-#### Google Cloud Storage (GCS)
-
-Here are the valid connection parameters for GCS:
-
-| Setting | Description | Example |
-|------------------------------|-------------------|---------|
-| `provider` | Provider name. | `Google` |
-| `google_project` | GCP project name. | `gcp-project-12345` |
-| `google_json_key_location` | JSON key path. | `/path/to/gcp-project-12345-abcde.json` |
-| `google_json_key_string` | JSON key string. | `{ "type": "service_account", "project_id": "example-project-382839", ... }` |
-| `google_application_default` | Set to `true` to use [Google Cloud Application Default Credentials](https://cloud.google.com/docs/authentication#adc) to locate service account credentials. | |
-
-GitLab reads the value of `google_json_key_location`, then `google_json_key_string`, and finally, `google_application_default`.
-It uses the first of these settings that has a value.
-
-The service account must have permission to access the bucket. For more information,
-see the [Cloud Storage authentication documentation](https://cloud.google.com/storage/docs/authentication).
-
-NOTE:
-Bucket encryption with the [Cloud Key Management Service (KMS)](https://cloud.google.com/kms/docs) is not supported and results in [ETag mismatch errors](#etag-mismatch).
-
-##### Google example (consolidated form)
-
-For Omnibus installations, this is an example of the `connection` setting:
-
-```ruby
-gitlab_rails['object_store']['connection'] = {
- 'provider' => 'Google',
- 'google_project' => '<GOOGLE PROJECT>',
- 'google_json_key_location' => '<FILENAME>'
-}
-```
-
-##### Google example with ADC (consolidated form)
-
-> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/275979) in GitLab 13.6.
-
-Google Cloud Application Default Credentials (ADC) are typically
-used with GitLab to use the default service account. This eliminates the
-need to supply credentials for the instance. For example:
-
-```ruby
-gitlab_rails['object_store']['connection'] = {
- 'provider' => 'Google',
- 'google_project' => '<GOOGLE PROJECT>',
- 'google_application_default' => true
-}
-```
-
-If you use ADC, be sure that:
-
-- The service account that you use has the
-[`iam.serviceAccounts.signBlob` permission](https://cloud.google.com/iam/docs/reference/credentials/rest/v1/projects.serviceAccounts/signBlob).
- Typically this is done by granting the `Service Account Token Creator` role to the service account.
-- Your virtual machines have the [correct access scopes to access Google Cloud APIs](https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances#changeserviceaccountandscopes). If the machines do not have the right scope, the error logs may show:
-
- ```markdown
- Google::Apis::ClientError (insufficientPermissions: Request had insufficient authentication scopes.)
- ```
-
-#### Azure Blob storage
-
-> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25877) in GitLab 13.4.
-
-Although Azure uses the word `container` to denote a collection of
-blobs, GitLab standardizes on the term `bucket`. Be sure to configure
-Azure container names in the `bucket` settings.
-
-Azure Blob storage can only be used with the [consolidated form](#consolidated-object-storage-configuration)
-because a single set of credentials are used to access multiple
-containers. The [storage-specific form](#storage-specific-configuration)
-is not supported. For more details, see [how to transition to consolidated form](#transition-to-consolidated-form).
-
-The following are the valid connection parameters for Azure. For more information, see the
-[Azure Blob Storage documentation](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction).
-
-| Setting | Description | Example |
-|------------------------------|----------------|-----------|
-| `provider` | Provider name. | `AzureRM` |
-| `azure_storage_account_name` | Name of the Azure Blob Storage account used to access the storage. | `azuretest` |
-| `azure_storage_access_key` | Storage account access key used to access the container. This is typically a secret, 512-bit encryption key encoded in base64. | `czV2OHkvQj9FKEgrTWJRZVRoV21ZcTN0Nnc5eiRDJkYpSkBOY1JmVWpYbjJy\nNHU3eCFBJUQqRy1LYVBkU2dWaw==\n` |
-| `azure_storage_domain` | Domain name used to contact the Azure Blob Storage API (optional). Defaults to `blob.core.windows.net`. Set this if you are using Azure China, Azure Germany, Azure US Government, or some other custom Azure domain. | `blob.core.windows.net` |
-
-##### Azure example (consolidated form)
-
-For Omnibus installations, this is an example of the `connection` setting:
-
-```ruby
-gitlab_rails['object_store']['connection'] = {
- 'provider' => 'AzureRM',
- 'azure_storage_account_name' => '<AZURE STORAGE ACCOUNT NAME>',
- 'azure_storage_access_key' => '<AZURE STORAGE ACCESS KEY>',
- 'azure_storage_domain' => '<AZURE STORAGE DOMAIN>'
-}
-```
-
-###### Azure Workhorse settings (source installs only)
-
-For source installations, Workhorse also needs to be configured with Azure
-credentials. This isn't needed in Omnibus installs, because the Workhorse
-settings are populated from the previous settings.
-
-1. Edit `/home/git/gitlab-workhorse/config.toml` and add or amend the following lines:
-
- ```toml
- [object_storage]
- provider = "AzureRM"
-
- [object_storage.azurerm]
- azure_storage_account_name = "<AZURE STORAGE ACCOUNT NAME>"
- azure_storage_access_key = "<AZURE STORAGE ACCESS KEY>"
- ```
-
-If you are using a custom Azure storage domain,
-`azure_storage_domain` does **not** have to be set in the Workhorse
-configuration. This information is exchanged in an API call between
-GitLab Rails and Workhorse.
-
-#### Storj Gateway Configuration (SJ)
-
-NOTE:
-The Storj Gateway [does not support](https://github.com/storj/gateway-st/blob/4b74c3b92c63b5de7409378b0d1ebd029db9337d/docs/s3-compatibility.md) multi-threaded copying (see `UploadPartCopy` in the table).
-While an implementation [is planned](https://github.com/storj/roadmap/issues/40), you must [disable multi-threaded copying](#multi-threaded-copying) until completion.
-
-The [Storj Network](https://www.storj.io/) provides an S3-compatible API gateway. Use the following configuration example:
-
-```ruby
-gitlab_rails['object_store']['connection'] = {
- 'provider' => 'AWS',
- 'endpoint' => 'https://gateway.storjshare.io',
- 'path_style' => true,
- 'region' => 'eu1',
- 'aws_access_key_id' => 'ACCESS_KEY',
- 'aws_secret_access_key' => 'SECRET_KEY',
- 'aws_signature_version' => 2,
- 'enable_signature_v4_streaming' => false
-}
-```
-
-The signature version must be `2`. Using v4 results in a HTTP 411 Length Required error.
-For more information, see [issue #4419](https://gitlab.com/gitlab-org/gitlab/-/issues/4419).
-
-### Object-specific configuration
-
-The following YAML shows how the `object_store` section defines
-object-specific configuration block and how the `enabled` and
-`proxy_download` flags can be overridden. The `bucket` is the only
-required parameter within each type:
-
-```yaml
- object_store:
- connection:
- ...
- objects:
- artifacts:
- bucket: artifacts
- proxy_download: false
- external_diffs:
- bucket: external-diffs
- lfs:
- bucket: lfs-objects
- uploads:
- bucket: uploads
- packages:
- bucket: packages
- dependency_proxy:
- enabled: false
- bucket: dependency_proxy
- terraform_state:
- bucket: terraform
- pages:
- bucket: pages
-```
-
-This maps to this Omnibus GitLab configuration:
-
-```ruby
-gitlab_rails['object_store']['objects']['artifacts']['bucket'] = 'artifacts'
-gitlab_rails['object_store']['objects']['artifacts']['proxy_download'] = false
-gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = 'external-diffs'
-gitlab_rails['object_store']['objects']['lfs']['bucket'] = 'lfs-objects'
-gitlab_rails['object_store']['objects']['uploads']['bucket'] = 'uploads'
-gitlab_rails['object_store']['objects']['packages']['bucket'] = 'packages'
-gitlab_rails['object_store']['objects']['dependency_proxy']['enabled'] = false
-gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = 'dependency-proxy'
-gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = 'terraform-state'
-gitlab_rails['object_store']['objects']['pages']['bucket'] = 'pages'
-```
-
-This is the list of valid `objects` that can be used:
-
-| Type | Description |
-|--------------------|----------------------------------------------------------------------------|
-| `artifacts` | [CI artifacts](job_artifacts.md) |
-| `external_diffs` | [Merge request diffs](merge_request_diffs.md) |
-| `uploads` | [User uploads](uploads.md) |
-| `lfs` | [Git Large File Storage objects](lfs/index.md) |
-| `packages` | [Project packages (for example, PyPI, Maven, or NuGet)](packages/index.md) |
-| `dependency_proxy` | [Dependency Proxy](packages/dependency_proxy.md) |
-| `terraform_state` | [Terraform state files](terraform_state.md) |
-| `pages` | [Pages](pages/index.md) |
-
-Within each object type, three parameters can be defined:
-
-| Setting | Required? | Description |
-|------------------|------------------------|-------------------------------------|
-| `bucket` | **{check-circle}** Yes | Bucket name for the object storage. |
-| `enabled` | **{dotted-circle}** No | Overrides the common parameter. |
-| `proxy_download` | **{dotted-circle}** No | Overrides the common parameter. |
-
-#### Selectively disabling object storage
-
-As seen above, object storage can be disabled for specific types by
-setting the `enabled` flag to `false`. For example, to disable object
-storage for CI artifacts:
-
-```ruby
-gitlab_rails['object_store']['objects']['artifacts']['enabled'] = false
-```
-
-A bucket is not needed if the feature is disabled entirely. For example,
-no bucket is needed if CI artifacts are disabled with this setting:
-
-```ruby
-gitlab_rails['artifacts_enabled'] = false
-```
-
-### Migrate to object storage
+## Migrate to object storage
To migrate existing local data to object storage see the following guides:
@@ -706,7 +729,7 @@ To migrate existing local data to object storage see the following guides:
- [Terraform state files](terraform_state.md#migrate-to-object-storage)
- [Pages content](pages/index.md#migrate-pages-deployments-to-object-storage)
-### Transition to consolidated form
+## Transition to consolidated form
Prior to GitLab 13.2:
@@ -737,36 +760,55 @@ additional complexity and unnecessary redundancy. Since both GitLab
Rails and Workhorse components need access to object storage, the
consolidated form avoids excessive duplication of credentials.
-The consolidated object storage configuration is used _only_ if all lines from
+The consolidated form is used _only_ if all lines from
the original form is omitted. To move to the consolidated form, remove the
original configuration (for example, `artifacts_object_store_enabled`, or
`uploads_object_store_connection`)
-### Storage-specific configuration
+## Migrate objects to a different object storage provider
-For configuring object storage in GitLab 13.1 and earlier, or for storage types not
-supported by consolidated configuration form, refer to the following guides:
+You may need to migrate GitLab data in object storage to a different object storage provider. The following steps show you how do this using [Rclone](https://rclone.org/).
-| Object storage type | Supported by consolidated configuration? |
-|---------------------|------------------------------------------|
-| [Backups](../raketasks/backup_gitlab.md#upload-backups-to-a-remote-cloud-storage) | **{dotted-circle}** No |
-| [Job artifacts](job_artifacts.md#using-object-storage) including archived job logs | **{check-circle}** Yes |
-| [LFS objects](lfs/index.md#storing-lfs-objects-in-remote-object-storage) | **{check-circle}** Yes |
-| [Uploads](uploads.md#using-object-storage) | **{check-circle}** Yes |
-| [Container Registry](packages/container_registry.md#use-object-storage) (optional feature) | **{dotted-circle}** No |
-| [Merge request diffs](merge_request_diffs.md#using-object-storage) | **{check-circle}** Yes |
-| [Mattermost](https://docs.mattermost.com/configure/file-storage-configuration-settings.html)| **{dotted-circle}** No |
-| [Packages](packages/index.md#use-object-storage) (optional feature) | **{check-circle}** Yes |
-| [Dependency Proxy](packages/dependency_proxy.md#using-object-storage) (optional feature) | **{check-circle}** Yes |
-| [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | **{dotted-circle}** No |
-| [Terraform state files](terraform_state.md#using-object-storage) | **{check-circle}** Yes |
-| [Pages content](pages/index.md#using-object-storage) | **{check-circle}** Yes |
+The steps assume you are moving the `uploads` bucket, but the same process works for other buckets.
+
+Prerequisites:
+
+- Choose the computer to run Rclone on. Depending on how much data you are migrating, Rclone may have to run for a long time so you should avoid using a laptop or desktop computer that can go into power saving. You can use your GitLab server to run Rclone.
+
+1. [Install](https://rclone.org/downloads/) Rclone.
+1. Configure Rclone by running the following:
+
+ ```shell
+ rclone config
+ ```
+
+ The configuration process is interactive. Add at least two "remotes": one for the object storage provider your data is currently on (`old`), and one for the provider you are moving to (`new`).
+
+1. Verify that you can read the old data. The following example refers to the `uploads` bucket , but your bucket may have a different name:
+
+ ```shell
+ rclone ls old:uploads | head
+ ```
+
+ This should print a partial list of the objects currently stored in your `uploads` bucket. If you get an error, or if
+ the list is empty, go back and update your Rclone configuration using `rclone config`.
+
+1. Perform an initial copy. You do not need to take your GitLab server offline for this step.
-WARNING:
-The use of [encrypted S3 buckets](#encrypted-s3-buckets) with non-consolidated configuration is not supported.
-You may start getting [ETag mismatch errors](#etag-mismatch) if you use it.
+ ```shell
+ rclone sync -P old:uploads new:uploads
+ ```
-### Other alternatives to file system storage
+1. After the first sync completes, use the web UI or command-line interface of your new object storage provider to
+ verify that there are objects in the new bucket. If there are none, or if you encounter an error while running `rclone
+ sync`, check your Rclone configuration and try again.
+
+After you have done at least one successful Rclone copy from the old location to the new location, schedule maintenance and take your GitLab server offline. During your maintenance window you must do two things:
+
+1. Perform a final `rclone sync` run, knowing that your users cannot add new objects so you do not leave any behind in the old bucket.
+1. Update the object storage configuration of your GitLab server to use the new provider for `uploads`.
+
+## Alternatives to file system storage
If you're working to [scale out](reference_architectures/index.md) your GitLab implementation,
or add fault tolerance and redundancy, you may be
@@ -779,11 +821,11 @@ See the following additional guides:
1. [Prevent local disk usage for job logs](job_logs.md#prevent-local-disk-usage).
1. [Disable Pages local storage](pages/index.md#disable-pages-local-storage).
-## Warnings, limitations, and known issues
+## Troubleshooting
### Objects are not included in GitLab backups
-As noted in [our backup documentation](../raketasks/backup_restore.md),
+As noted in [the backup documentation](../raketasks/backup_restore.md),
objects are not included in GitLab backups. You can enable backups with
your object storage provider instead.
@@ -882,14 +924,14 @@ If you are seeing this ETag mismatch error with Amazon Web Services S3,
it's likely this is due to [encryption settings on your bucket](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html).
To fix this issue, you have two options:
-- [Use the consolidated object configuration](#consolidated-object-storage-configuration).
-- [Use Amazon instance profiles](#using-amazon-instance-profiles).
+- [Use the consolidated form](#configure-a-single-storage-connection-for-all-object-types-consolidated-form).
+- [Use Amazon instance profiles](#use-amazon-instance-profiles).
The first option is recommended for MinIO. Otherwise, the
[workaround for MinIO](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/1564#note_244497658)
is to use the `--compat` parameter on the server.
-Without consolidated object store configuration or instance profiles enabled,
+Without the consolidated form or instance profiles enabled,
GitLab Workhorse uploads files to S3 using pre-signed URLs that do
not have a `Content-MD5` HTTP header computed for them. To ensure data
is not corrupted, Workhorse checks that the MD5 hash of the data sent
@@ -897,90 +939,16 @@ equals the ETag header returned from the S3 server. When encryption is
enabled, this is not the case, which causes Workhorse to report an `ETag
mismatch` error during an upload.
-With the consolidated object configuration and instance profile, Workhorse has
-S3 credentials so that it can compute the `Content-MD5` header. This
-eliminates the need to compare ETag headers returned from the S3 server.
-
-Encrypting buckets with the GCS [Cloud Key Management Service (KMS)](https://cloud.google.com/kms/docs) is not supported and results in ETag mismatch errors.
-
-### Using Amazon instance profiles
-
-Instead of supplying AWS access and secret keys in object storage
-configuration, GitLab can be configured to use IAM roles to set up an
-[Amazon instance profile](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html).
-When this is used, GitLab fetches temporary credentials each time an
-S3 bucket is accessed, so no hard-coded values are needed in the
-configuration.
-
-To use an Amazon instance profile, GitLab must be able to connect to the
-[instance metadata endpoint](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html).
-If GitLab is [configured to use an Internet proxy](https://docs.gitlab.com/omnibus/settings/environment-variables.html), the endpoint IP
-address must be added to the `no_proxy` list.
-
-#### Encrypted S3 buckets
-
-> - [Introduced](https://gitlab.com/gitlab-org/gitlab-workhorse/-/merge_requests/466) in GitLab 13.1 for instance profiles only and [S3 default encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/bucket-encryption.html).
-> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/34460) in GitLab 13.2 for static credentials when [consolidated object storage configuration](#consolidated-object-storage-configuration) and [S3 default encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/bucket-encryption.html) are used.
-
-When configured either with an instance profile or with the consolidated
-object configuration, GitLab Workhorse properly uploads files to S3
-buckets that have [SSE-S3 or SSE-KMS encryption enabled by default](https://docs.aws.amazon.com/kms/latest/developerguide/services-s3.html).
-Customer master keys (CMKs) and SSE-C encryption are
-[not supported since this requires sending the encryption keys in every request](https://gitlab.com/gitlab-org/gitlab/-/issues/226006).
-
-##### Server-side encryption headers
-
-> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38240) in GitLab 13.3.
-
-Setting a default encryption on an S3 bucket is the easiest way to
-enable encryption, but you may want to
-[set a bucket policy to ensure only encrypted objects are uploaded](https://repost.aws/knowledge-center/s3-bucket-store-kms-encrypted-objects).
-To do this, you must configure GitLab to send the proper encryption headers
-in the `storage_options` configuration section:
-
-| Setting | Description |
-|-------------------------------------|------------------------------------------|
-| `server_side_encryption` | Encryption mode (`AES256` or `aws:kms`). |
-| `server_side_encryption_kms_key_id` | Amazon Resource Name. Only needed when `aws:kms` is used in `server_side_encryption`. See the [Amazon documentation on using KMS encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html). |
-
-As with the case for default encryption, these options only work when
-the Workhorse S3 client is enabled. One of the following two conditions
-must be fulfilled:
-
-- `use_iam_profile` is `true` in the connection settings.
-- Consolidated object storage settings are in use.
-
-[ETag mismatch errors](#etag-mismatch) occur if server side
-encryption headers are used without enabling the Workhorse S3 client.
-
-#### IAM Permissions
-
-To set up an instance profile:
-
-1. Create an Amazon Identity Access and Management (IAM) role with the necessary permissions. The
- following example is a role for an S3 bucket named `test-bucket`:
+When the consolidated form is:
- ```json
- {
- "Version": "2012-10-17",
- "Statement": [
- {
- "Sid": "VisualEditor0",
- "Effect": "Allow",
- "Action": [
- "s3:PutObject",
- "s3:GetObject",
- "s3:DeleteObject"
- ],
- "Resource": "arn:aws:s3:::test-bucket/*"
- }
- ]
- }
- ```
+- Used with an S3-compatible object storage or an istance profile, Workhorse
+ uses its internal S3 client which has S3 credentials so that it can compute
+ the `Content-MD5` header. This eliminates the need to compare ETag headers
+ returned from the S3 server.
+- Not used with an S3-compatible object storage, Workhorse falls back to using
+ pre-signed URLs.
-1. [Attach this role](https://aws.amazon.com/premiumsupport/knowledge-center/attach-replace-ec2-instance-profile/)
- to the EC2 instance hosting your GitLab instance.
-1. Configure GitLab to use it via the `use_iam_profile` configuration option.
+Encrypting buckets with the GCS [Cloud Key Management Service (KMS)](https://cloud.google.com/kms/docs) is not supported and results in ETag mismatch errors.
### Multi-threaded copying
@@ -996,46 +964,3 @@ to run the following command:
```ruby
Feature.disable(:s3_multithreaded_uploads)
```
-
-## Migrate objects to a different object storage provider
-
-You may need to migrate GitLab data in object storage to a different object storage provider. The following steps show you how do this using [Rclone](https://rclone.org/).
-
-The steps assume you are moving the `uploads` bucket, but the same process works for other buckets.
-
-Prerequisites:
-
-- Choose the computer to run Rclone on. Depending on how much data you are migrating, Rclone may have to run for a long time so you should avoid using a laptop or desktop computer that can go into power saving. You can use your GitLab server to run Rclone.
-
-1. [Install](https://rclone.org/downloads/) Rclone.
-1. Configure Rclone by running the following:
-
- ```shell
- rclone config
- ```
-
- The configuration process is interactive. Add at least two "remotes": one for the object storage provider your data is currently on (`old`), and one for the provider you are moving to (`new`).
-
-1. Verify that you can read the old data. The following example refers to the `uploads` bucket , but your bucket may have a different name:
-
- ```shell
- rclone ls old:uploads | head
- ```
-
- This should print a partial list of the objects currently stored in your `uploads` bucket. If you get an error, or if
- the list is empty, go back and update your Rclone configuration using `rclone config`.
-
-1. Perform an initial copy. You do not need to take your GitLab server offline for this step.
-
- ```shell
- rclone sync -P old:uploads new:uploads
- ```
-
-1. After the first sync completes, use the web UI or command-line interface of your new object storage provider to
- verify that there are objects in the new bucket. If there are none, or if you encounter an error while running `rclone
- sync`, check your Rclone configuration and try again.
-
-After you have done at least one successful Rclone copy from the old location to the new location, schedule maintenance and take your GitLab server offline. During your maintenance window you must do two things:
-
-1. Perform a final `rclone sync` run, knowing that your users cannot add new objects so you do not leave any behind in the old bucket.
-1. Update the object storage configuration of your GitLab server to use the new provider for `uploads`.
diff --git a/doc/administration/operations/gitlab_sshd.md b/doc/administration/operations/gitlab_sshd.md
index 7b61631fe3a..249d6232616 100644
--- a/doc/administration/operations/gitlab_sshd.md
+++ b/doc/administration/operations/gitlab_sshd.md
@@ -6,13 +6,13 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# `gitlab-sshd` **(FREE SELF)**
-> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299109) in GitLab 14.5 as an **Alpha** release for self-managed customers.
+> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299109) in GitLab 14.5 as an Experiment for self-managed customers.
> - Ready for production use with [Cloud Native GitLab in GitLab 15.1](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/2540) and [Omnibus GitLab in GitLab 15.9](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5937).
`gitlab-sshd` is [a standalone SSH server](https://gitlab.com/gitlab-org/gitlab-shell/-/tree/main/internal/sshd)
written in Go. It is provided as a part of the `gitlab-shell` package. It has a lower memory
use as a OpenSSH alternative, and supports
-[group access restriction by IP address](../../user/group/index.md) for applications
+[group access restriction by IP address](../../user/group/access_and_permissions.md#restrict-group-access-by-ip-address) for applications
running behind the proxy.
`gitlab-sshd` is a lightweight alternative to OpenSSH for providing
@@ -27,8 +27,9 @@ If you are considering switching from OpenSSH to `gitlab-sshd`, consider these c
- `gitlab-sshd` supports the PROXY protocol. It can run behind proxy servers that rely
on it, such as HAProxy. The PROXY protocol is not enabled by default, but [it can be enabled](#proxy-protocol-support).
-- `gitlab-sshd` **does not** support SSH certificates. For more details, read
- [issue #495](https://gitlab.com/gitlab-org/gitlab-shell/-/issues/495).
+- `gitlab-sshd` **does not** support SSH certificates. For more details, see the
+ [confidential issue](../../user/project/issues/confidential_issues.md)
+ `https://gitlab.com/gitlab-org/gitlab-shell/-/issues/495`.
## Enable `gitlab-sshd`
@@ -110,11 +111,11 @@ To enable the PROXY protocol:
1. Edit `/etc/gitlab/gitlab.rb`:
- ```ruby
- gitlab_sshd['proxy_protocol'] = true
- # # Proxy protocol policy ("use", "require", "reject", "ignore"), "use" is the default value
- gitlab_sshd['proxy_policy'] = "use"
- ```
+ ```ruby
+ gitlab_sshd['proxy_protocol'] = true
+ # # Proxy protocol policy ("use", "require", "reject", "ignore"), "use" is the default value
+ gitlab_sshd['proxy_policy'] = "use"
+ ```
1. Save the file and reconfigure GitLab:
diff --git a/doc/administration/operations/index.md b/doc/administration/operations/index.md
index f6ab46b9fbf..867ef3236ee 100644
--- a/doc/administration/operations/index.md
+++ b/doc/administration/operations/index.md
@@ -4,29 +4,21 @@ group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
-# Performing operations in GitLab **(FREE SELF)**
+# Maintain your GitLab installation **(FREE SELF)**
-Keep your GitLab instance up and running smoothly.
+Keep your GitLab instance up and running.
-- [Rake tasks](../../raketasks/index.md): Tasks for common administration and operational processes such as
- [cleaning up unneeded items from GitLab instance](../../raketasks/cleanup.md), integrity checks,
- and more.
-- [Moving repositories](moving_repositories.md): Moving all repositories managed
- by GitLab to another file system or another server.
-- [Sidekiq MemoryKiller](../sidekiq/sidekiq_memory_killer.md): Configure Sidekiq MemoryKiller
- to restart Sidekiq.
-- [Multiple Sidekiq processes](../sidekiq/extra_sidekiq_processes.md): Configure multiple Sidekiq processes to ensure certain queues always have dedicated workers, no matter the number of jobs that must be processed. **(FREE SELF)**
-- [Puma](puma.md): Understand Puma and puma-worker-killer.
-- [`gitlab-sshd`](gitlab_sshd.md): Use GitLab SSH daemon instead of OpenSSH.
-- Speed up SSH operations by
- [Authorizing SSH users via a fast, indexed lookup to the GitLab database](fast_ssh_key_lookup.md), and/or
- by [doing away with user SSH keys stored on GitLab entirely in favor of SSH certificates](ssh_certificates.md).
-- [File System Performance Benchmarking](filesystem_benchmarking.md): File system
- performance can have a big impact on GitLab performance, especially for actions
- that read or write Git repositories. This information helps benchmark
- file system performance against known good and bad real-world systems.
-- [The Rails Console](rails_console.md): Provides a way to interact with your GitLab instance from the command line.
- Used for troubleshooting a problem or retrieving some data that can only be done through direct access to GitLab.
-- [ChatOps Scripts](https://gitlab.com/gitlab-com/chatops): The GitLab.com Infrastructure team uses this repository to house
- common ChatOps scripts they use to troubleshoot and maintain the production instance of GitLab.com.
- These scripts can be used by administrators of GitLab instances of all sizes.
+- [Housekeeping](../../administration/housekeeping.md)
+- [Activate GitLab EE with license](../../user/admin_area/license_file.md)
+- [Fast SSH key lookup](../../administration/operations/fast_ssh_key_lookup.md)
+- [File system benchmarking](../../administration/operations/filesystem_benchmarking.md)
+- [`gitlab-sshd`](../../administration/operations/gitlab_sshd.md)
+- [Rails console](../../administration/operations/rails_console.md)
+- [Use SSH certificates](../../administration/operations/ssh_certificates.md)
+- [Enable encrypted configuration](../../administration/encrypted_configuration.md)
+- [Rake tasks](../../raketasks/index.md)
+- [Backup and restore](../../raketasks/backup_restore.md)
+- [Inactive project deletion](../../administration/inactive_project_deletion.md)
+- [Move repositories](../../administration/operations/moving_repositories.md)
+- [Read-only state](../../administration/read_only_gitlab.md)
+- [Restart GitLab](../../administration/restart_gitlab.md)
diff --git a/doc/administration/operations/moving_repositories.md b/doc/administration/operations/moving_repositories.md
index aa0477be788..e9d829f3f08 100644
--- a/doc/administration/operations/moving_repositories.md
+++ b/doc/administration/operations/moving_repositories.md
@@ -139,14 +139,14 @@ To move all snippets by using the API:
To move all groups by using the API:
1. [Schedule repository storage moves for all groups on a storage shard](../../api/group_repository_storage_moves.md#schedule-repository-storage-moves-for-all-groups-on-a-storage-shard).
- For example:
-
- ```shell
- curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \
- --header "Content-Type: application/json" \
- --data '{"source_storage_name":"<original_storage_name>","destination_storage_name":"<cluster_storage_name>"}' \
- "https://gitlab.example.com/api/v4/group_repository_storage_moves"
- ```
+ For example:
+
+ ```shell
+ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \
+ --header "Content-Type: application/json" \
+ --data '{"source_storage_name":"<original_storage_name>","destination_storage_name":"<cluster_storage_name>"}' \
+ "https://gitlab.example.com/api/v4/group_repository_storage_moves"
+ ```
1. [Query the most recent repository moves](../../api/group_repository_storage_moves.md#retrieve-all-group-repository-storage-moves).
The response indicates either:
diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md
index f2f9f1cdcda..efc55a5fbc3 100644
--- a/doc/administration/operations/puma.md
+++ b/doc/administration/operations/puma.md
@@ -99,7 +99,7 @@ To change the worker timeout to 600 seconds:
## Disable Puma clustered mode in memory-constrained environments
WARNING:
-This is an experimental [Alpha feature](../../policy/alpha-beta-support.md#alpha-features) and subject to change without notice. The feature
+This feature is an [Experiment](../../policy/alpha-beta-support.md#experiment) and subject to change without notice. The feature
is not ready for production use. If you want to use this feature, you should test
outside of production first. See the [known issues](#puma-single-mode-known-issues)
for additional details.
@@ -182,7 +182,7 @@ steps below:
NOTE:
If using a self-signed certificate from a custom Certificate Authority (CA),
- follow [the documentation](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates)
+ follow [the documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html#install-custom-public-certificates)
to make them trusted by other GitLab components.
1. Edit `/etc/gitlab/gitlab.rb`:
diff --git a/doc/administration/operations/rails_console.md b/doc/administration/operations/rails_console.md
index 652a4fa5497..ac550d30566 100644
--- a/doc/administration/operations/rails_console.md
+++ b/doc/administration/operations/rails_console.md
@@ -569,7 +569,7 @@ def disable_two_factor!
end
def two_factor_enabled?
- two_factor_otp_enabled? || two_factor_u2f_enabled?
+ two_factor_otp_enabled? || two_factor_webauthn_enabled?
end
```
diff --git a/doc/administration/package_information/defaults.md b/doc/administration/package_information/defaults.md
index 3b52b6bca82..96b56388ea9 100644
--- a/doc/administration/package_information/defaults.md
+++ b/doc/administration/package_information/defaults.md
@@ -30,7 +30,7 @@ by default:
| PgBouncer exporter | No | Port | X | 9188 |
| GitLab Exporter | Yes | Port | X | 9168 |
| Sidekiq exporter | Yes | Port | X | 8082 |
-| Sidekiq health check | No | Port | X | 8092[^Sidekiq-health] |
+| Sidekiq health check | Yes | Port | X | 8092[^Sidekiq-health] |
| Web exporter | No | Port | X | 8083 |
| Geo PostgreSQL | No | Socket | Port (5431) | X |
| Redis Sentinel | No | Port | X | 26379 |
@@ -49,8 +49,10 @@ by default:
| PgBouncer | No | Port | X | 6432 |
| Consul | No | Port | X | 8300, 8301(UDP), 8500, 8600[^Consul-notes] |
| Patroni | No | Port | X | 8008 |
-| GitLab KAS | Yes | Port | X | 8150 |
-| Gitaly | No | Port | X | 8075 |
+| GitLab KAS | Yes | Port | X | 8150 |
+| Gitaly | Yes | Socket | Port (8075) | 8075 or 9999 (TLS) |
+| Gitaly exporter | Yes | Port | X | 9236 |
+| Praefect | No | Port | X | 2305 or 3305 (TLS) |
Legend:
diff --git a/doc/administration/package_information/supported_os.md b/doc/administration/package_information/supported_os.md
index 9d1c8dcde5a..a1f589015cb 100644
--- a/doc/administration/package_information/supported_os.md
+++ b/doc/administration/package_information/supported_os.md
@@ -15,13 +15,17 @@ operating systems supported by GitLab are listed in the
The following lists the currently supported OSs and their possible EOL dates.
+NOTE:
+`amd64` and `x86_64` refer to the same 64-bit architecture.
+The names `arm64` and `aarch64` are also interchangeable and refer to the same
+architecture.
+
| OS Version | First supported GitLab version | Arch | Install Documentation | OS EOL | Details |
| ------------------------------------------------------------ | ------------------------------ | --------------- | :----------------------------------------------------------: | ---------- | ------------------------------------------------------------ |
| AlmaLinux 8 | GitLab CE / GitLab EE 14.5.0 | x86_64, aarch64 | [AlmaLinux Install Documentation](https://about.gitlab.com/install/#almalinux-8) | 2029 | <https://almalinux.org/> |
| CentOS 7 | GitLab CE / GitLab EE 7.10.0 | x86_64 | [CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | June 2024 | <https://wiki.centos.org/About/Product> |
| Debian 10 | GitLab CE / GitLab EE 12.2.0 | amd64, arm64 | [Debian Install Documentation](https://about.gitlab.com/install/#debian) | 2024 | <https://wiki.debian.org/LTS> |
| Debian 11 | GitLab CE / GitLab EE 14.6.0 | amd64, arm64 | [Debian Install Documentation](https://about.gitlab.com/install/#debian) | 2026 | <https://wiki.debian.org/LTS> |
-| OpenSUSE 15.3 | GitLab CE / GitLab EE 14.5.0 | x86_64, aarch64 | [OpenSUSE Install Documentation](https://about.gitlab.com/install/#opensuse-leap) | Dec 2022 | <https://en.opensuse.org/Lifetime> |
| OpenSUSE 15.4 | GitLab CE / GitLab EE 15.7.0 | x86_64, aarch64 | [OpenSUSE Install Documentation](https://about.gitlab.com/install/#opensuse-leap) | Nov 2023 | <https://en.opensuse.org/Lifetime> |
| RHEL 8 | GitLab CE / GitLab EE 12.8.1 | x86_64, arm64 | [Use CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | May 2029 | [RHEL Details](https://access.redhat.com/support/policy/updates/errata/#Life_Cycle_Dates) |
| SLES 12 | GitLab EE 9.0.0 | x86_64 | [Use OpenSUSE Install Documentation](https://about.gitlab.com/install/#opensuse-leap) | Oct 2027 | <https://www.suse.com/lifecycle/> |
@@ -31,7 +35,8 @@ The following lists the currently supported OSs and their possible EOL dates.
| Ubuntu 18.04 | GitLab CE / GitLab EE 10.7.0 | amd64 | [Ubuntu Install Documentation](https://about.gitlab.com/install/#ubuntu) | April 2023 | <https://wiki.ubuntu.com/Releases> |
| Ubuntu 20.04 | GitLab CE / GitLab EE 13.2.0 | amd64, arm64 | [Ubuntu Install Documentation](https://about.gitlab.com/install/#ubuntu) | April 2025 | <https://wiki.ubuntu.com/Releases> |
| Ubuntu 22.04 | GitLab CE / GitLab EE 15.5.0 | amd64, arm64 | [Ubuntu Install Documentation](https://about.gitlab.com/install/#ubuntu) | April 2027 | <https://wiki.ubuntu.com/Releases> |
-| Amazon Linux 2 | GitLab CE / GitLab EE 14.9.0 | amd64, arm64 | [Amazon Linux 2 Install Documentation](https://about.gitlab.com/install/#amazonlinux-2) | June 2023 | <https://aws.amazon.com/amazon-linux-2/faqs/> |
+| Amazon Linux 2 | GitLab CE / GitLab EE 14.9.0 | amd64, arm64 | [Amazon Linux 2 Install Documentation](https://about.gitlab.com/install/#amazonlinux-2) | June 2025 | <https://aws.amazon.com/amazon-linux-2/faqs/> |
+| Amazon Linux 2022 | GitLab CE / GitLab EE 15.9.0 | amd64, arm64 | [Amazon Linux 2022 Install Documentation](https://about.gitlab.com/install/#amazonlinux-2022) | October 2027 | <https://aws.amazon.com/linux/amazon-linux-2022/faqs/> |
| Raspberry Pi OS (Buster) (formerly known as Raspbian Buster) | GitLab CE 12.2.0 | armhf | [Raspberry Pi Install Documentation](https://about.gitlab.com/install/#raspberry-pi-os) | 2024 | [Raspberry Pi Details](https://www.raspberrypi.com/news/new-old-functionality-with-raspberry-pi-os-legacy/) |
| Raspberry Pi OS (Bullseye) | GitLab CE 15.5.0 | armhf | [Raspberry Pi Install Documentation](https://about.gitlab.com/install/#raspberry-pi-os) | 2026 | [Raspberry Pi Details](https://www.raspberrypi.com/news/raspberry-pi-os-debian-bullseye/) |
@@ -55,6 +60,14 @@ although [new versions have been released](https://about.gitlab.com/releases/cat
of the [Linux package install guide](https://about.gitlab.com/install/#content).
Future GitLab upgrades are fetched according to your upgraded OS.
+## Update both GitLab and the operating system
+
+To upgrade both the operating system (OS) and GitLab:
+
+1. Upgrade the OS.
+1. Check if it's necessary to [update the GitLab package sources](#update-gitlab-package-sources-after-upgrading-the-os).
+1. [Upgrade GitLab](../../update/index.md).
+
## Packages for ARM64
> [Introduced](https://gitlab.com/gitlab-org/gitlab-omnibus-builder/-/issues/27) in GitLab 13.4.
@@ -94,6 +107,7 @@ release for them can be found below:
| Ubuntu 16.04 | [April 2021](https://ubuntu.com/info/release-end-of-life) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_13.12&dist=ubuntu%2Fxenial) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_13.12&dist=ubuntu%2Fxenial) 13.12 |
| OpenSUSE 15.2 | [December 2021](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-14.7&dist=opensuse%2F15.2) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-14.7&dist=opensuse%2F15.2) 14.7 |
| Debian 9 "Stretch" | [June 2022](https://lists.debian.org/debian-lts-announce/2022/07/msg00002.html) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_15.2&dist=debian%2Fstretch) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_15.2&dist=debian%2Fstretch) 15.2 |
+| OpenSUSE 15.3 | [December 2022](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-15.10&dist=opensuse%2F15.3) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-15.10&dist=opensuse%2F15.3) 15.10 |
NOTE:
An exception to this deprecation policy is when we are unable to provide
diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md
index 34acf57ce70..fd3cbb2ad05 100644
--- a/doc/administration/packages/container_registry.md
+++ b/doc/administration/packages/container_registry.md
@@ -21,10 +21,8 @@ Registry, see the [user documentation](../../user/packages/container_registry/in
If you installed GitLab by using the Omnibus installation package, the Container Registry
may or may not be available by default.
-The Container Registry is automatically enabled and available on your GitLab domain, port 5050 if:
-
-- You're using the built-in [Let's Encrypt integration](https://docs.gitlab.com/omnibus/settings/ssl.html#enable-the-lets-encrypt-integration), and
-- You're using GitLab 12.5 or later.
+The Container Registry is automatically enabled and available on your GitLab domain, port 5050 if
+you're using the built-in [Let's Encrypt integration](https://docs.gitlab.com/omnibus/settings/ssl/index.html#enable-the-lets-encrypt-integration).
Otherwise, the Container Registry is not enabled. To enable it:
@@ -96,7 +94,7 @@ If `auth` is not set up, users can pull Docker images without authentication.
## Container Registry domain configuration
-There are two ways you can configure the Registry's external domain. Either:
+You can configure the Registry's external domain in either of these ways:
- [Use the existing GitLab domain](#configure-container-registry-under-an-existing-gitlab-domain).
The Registry listens on a port and reuses the TLS certificate from GitLab.
@@ -110,13 +108,15 @@ for the first time.
### Configure Container Registry under an existing GitLab domain
-If the Registry is configured to use the existing GitLab domain, you can
-expose the Registry on a port. This way you can reuse the existing GitLab TLS
+If the Container Registry is configured to use the existing GitLab domain, you can
+expose the Container Registry on a port. This way you can reuse the existing GitLab TLS
certificate.
-If the GitLab domain is `https://gitlab.example.com` and the port to the outside world is `5050`, here is what you need to set
-in `gitlab.rb` or `gitlab.yml` if you are using Omnibus GitLab or installed
-GitLab from source respectively.
+If the GitLab domain is `https://gitlab.example.com` and the port to the outside world is `5050`,
+to configure the Container Registry:
+
+- Edit `gitlab.rb` if you are using Omnibus GitLab.
+- Edit `gitlab.yml` if you installed GitLab from source.
Ensure you choose a port different than the one that Registry listens to (`5000` by default),
otherwise conflicts occur.
@@ -202,7 +202,7 @@ domain. For example, `*.gitlab.example.com`, is a wildcard that matches `registr
and is distinct from `*.example.com`.
As well as manually generated SSL certificates (explained here), certificates automatically
-generated by Let's Encrypt are also [supported in Omnibus installs](https://docs.gitlab.com/omnibus/settings/ssl.html).
+generated by Let's Encrypt are also [supported in Omnibus installs](https://docs.gitlab.com/omnibus/settings/ssl/index.html).
Let's assume that you want the container Registry to be accessible at
`https://registry.gitlab.example.com`.
@@ -261,7 +261,7 @@ docker login registry.gitlab.example.com
## Disable Container Registry site-wide
When you disable the Registry by following these steps, you do not
-remove any existing Docker images. This is handled by the
+remove any existing Docker images. Docker image removal is handled by the
Registry application itself.
**Omnibus GitLab**
@@ -557,7 +557,7 @@ you can pull from the Container Registry, but you cannot push.
1. To perform the final data sync,
[put the Container Registry in `read-only` mode](#performing-garbage-collection-without-downtime) and
[reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
-1. Sync any changes since the initial data load to your S3 bucket and delete files that exist in the destination bucket but not in the source:
+1. Sync any changes dating from after the initial data load to your S3 bucket, and delete files that exist in the destination bucket but not in the source:
```shell
sudo aws --endpoint-url https://your-object-storage-backend.com s3 sync registry s3://mybucket --delete --dryrun
@@ -590,15 +590,15 @@ you can pull from the Container Registry, but you cannot push.
#### Moving to Azure Object Storage
-> The default configuration for the storage driver will be [changed](https://gitlab.com/gitlab-org/container-registry/-/issues/854) in GitLab 16.0.
+> The default configuration for the storage driver is scheduled to be [changed](https://gitlab.com/gitlab-org/container-registry/-/issues/854) in GitLab 16.0.
<!--- start_remove The following content will be removed on remove_date: '2023-10-22' -->
WARNING:
-The default configuration for the storage driver will be [changed](https://gitlab.com/gitlab-org/container-registry/-/issues/854) in GitLab 16.0. The storage driver will use `/` as the default root directory. You can add `trimlegacyrootprefix: false` to your current configuration now to avoid any disruptions. For more information, see the [Container Registry configuration](https://gitlab.com/gitlab-org/container-registry/-/tree/master/docs-gitlab#azure-storage-driver) documentation.
+The default configuration for the storage driver is scheduled to be [changed](https://gitlab.com/gitlab-org/container-registry/-/issues/854) in GitLab 16.0. The storage driver will use `/` as the default root directory. You can add `trimlegacyrootprefix: false` to your current configuration now to avoid any disruptions. For more information, see the [Container Registry configuration](https://gitlab.com/gitlab-org/container-registry/-/tree/master/docs-gitlab#azure-storage-driver) documentation.
<!--- end_remove -->
When moving from an existing file system or another object storage provider to Azure Object Storage, you must configure the registry to use the standard root directory.
-This configuration is done by setting [`trimlegacyrootprefix: true]`](https://gitlab.com/gitlab-org/container-registry/-/blob/a3f64464c3ec1c5a599c0a2daa99ebcbc0100b9a/docs-gitlab/README.md#azure-storage-driver) in the Azure storage driver section of the registry configuration.
+Configure it by setting [`trimlegacyrootprefix: true]`](https://gitlab.com/gitlab-org/container-registry/-/blob/a3f64464c3ec1c5a599c0a2daa99ebcbc0100b9a/docs-gitlab/README.md#azure-storage-driver) in the Azure storage driver section of the registry configuration.
Without this configuration, the Azure storage driver uses `//` instead of `/` as the first section of the root path, rendering the migrated images inaccessible.
**Omnibus GitLab installations**
@@ -607,10 +607,10 @@ Without this configuration, the Azure storage driver uses `//` instead of `/` as
registry['storage'] = {
'azure' => {
'accountname' => 'accountname',
- 'accesskey' => 'base64encodedaccountkey',
+ 'accountkey' => 'base64encodedaccountkey',
'container' => 'containername',
'rootdirectory' => '/azure/virtual/container',
- 'trimlegacyrootprefix' => 'true'
+ 'trimlegacyrootprefix' => true
}
}
```
@@ -627,6 +627,8 @@ storage:
trimlegacyrootprefix: true
```
+By default, Azure Storage Driver uses the `core.windows.net` realm. You can set another value for `realm` in the `azure` section (for example, `core.usgovcloudapi.net` for Azure Government Cloud). For more information, see the [Docker documentation](https://docs.docker.com/registry/storage-drivers/azure/).
+
### Disable redirect for storage driver
By default, users accessing a registry configured with a remote backend are redirected to the default backend for the storage driver. For example, registries can be configured using the `s3` storage driver, which redirects requests to a remote S3 bucket to alleviate load on the GitLab server.
@@ -637,20 +639,20 @@ However, this behavior is undesirable for registries used by internal hosts that
1. Edit `/etc/gitlab/gitlab.rb`:
- ```ruby
- registry['storage'] = {
- 's3' => {
- 'accesskey' => 's3-access-key',
- 'secretkey' => 's3-secret-key-for-access-key',
- 'bucket' => 'your-s3-bucket',
- 'region' => 'your-s3-region',
- 'regionendpoint' => 'your-s3-regionendpoint'
- },
- 'redirect' => {
- 'disable' => true
- }
- }
- ```
+ ```ruby
+ registry['storage'] = {
+ 's3' => {
+ 'accesskey' => 's3-access-key',
+ 'secretkey' => 's3-secret-key-for-access-key',
+ 'bucket' => 'your-s3-bucket',
+ 'region' => 'your-s3-region',
+ 'regionendpoint' => 'your-s3-regionendpoint'
+ },
+ 'redirect' => {
+ 'disable' => true
+ }
+ }
+ ```
1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
@@ -658,21 +660,21 @@ However, this behavior is undesirable for registries used by internal hosts that
1. Add the `redirect` flag to your registry configuration YAML file:
- ```yaml
- storage:
- s3:
- accesskey: 'AKIAKIAKI'
- secretkey: 'secret123'
- bucket: 'gitlab-registry-bucket-AKIAKIAKI'
- region: 'your-s3-region'
- regionendpoint: 'your-s3-regionendpoint'
- redirect:
- disable: true
- cache:
- blobdescriptor: inmemory
- delete:
- enabled: true
- ```
+ ```yaml
+ storage:
+ s3:
+ accesskey: 'AKIAKIAKI'
+ secretkey: 'secret123'
+ bucket: 'gitlab-registry-bucket-AKIAKIAKI'
+ region: 'your-s3-region'
+ regionendpoint: 'your-s3-regionendpoint'
+ redirect:
+ disable: true
+ cache:
+ blobdescriptor: inmemory
+ delete:
+ enabled: true
+ ```
1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect.
@@ -680,7 +682,7 @@ However, this behavior is undesirable for registries used by internal hosts that
You can use server-side encryption with AWS KMS for S3 buckets that have
[SSE-S3 or SSE-KMS encryption enabled by default](https://docs.aws.amazon.com/kms/latest/developerguide/services-s3.html).
-Customer master keys (CMKs) and SSE-C encryption aren't supported since this requires sending the
+Customer master keys (CMKs) and SSE-C encryption aren't supported because this requires sending the
encryption keys in every request.
For SSE-S3, you must enable the `encrypt` option in the registry settings. How you do this depends
@@ -690,18 +692,18 @@ For Omnibus GitLab installations:
1. Edit `/etc/gitlab/gitlab.rb`:
- ```ruby
- registry['storage'] = {
- 's3' => {
- 'accesskey' => 's3-access-key',
- 'secretkey' => 's3-secret-key-for-access-key',
- 'bucket' => 'your-s3-bucket',
- 'region' => 'your-s3-region',
- 'regionendpoint' => 'your-s3-regionendpoint',
- 'encrypt' => true
- }
- }
- ```
+ ```ruby
+ registry['storage'] = {
+ 's3' => {
+ 'accesskey' => 's3-access-key',
+ 'secretkey' => 's3-secret-key-for-access-key',
+ 'bucket' => 'your-s3-bucket',
+ 'region' => 'your-s3-region',
+ 'regionendpoint' => 'your-s3-regionendpoint',
+ 'encrypt' => true
+ }
+ }
+ ```
1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure)
for the changes to take effect.
@@ -710,23 +712,23 @@ For installations from source:
1. Edit your registry configuration YAML file:
- ```yaml
- storage:
- s3:
- accesskey: 'AKIAKIAKI'
- secretkey: 'secret123'
- bucket: 'gitlab-registry-bucket-AKIAKIAKI'
- region: 'your-s3-region'
- regionendpoint: 'your-s3-regionendpoint'
- encrypt: true
- ```
+ ```yaml
+ storage:
+ s3:
+ accesskey: 'AKIAKIAKI'
+ secretkey: 'secret123'
+ bucket: 'gitlab-registry-bucket-AKIAKIAKI'
+ region: 'your-s3-region'
+ regionendpoint: 'your-s3-regionendpoint'
+ encrypt: true
+ ```
1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source)
for the changes to take effect.
### Storage limitations
-Currently, there is no storage limitation, which means a user can upload an
+There is no storage limitation, which means a user can upload an
infinite amount of Docker images with arbitrary sizes. This setting should be
configurable in future releases.
@@ -765,7 +767,11 @@ project, you can [disable it from your project's settings](../../user/project/se
## Use an external container registry with GitLab as an auth endpoint
-> Support for external container registries in GitLab is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/376217) in GitLab 15.8 and will be removed in GitLab 16.0.
+WARNING:
+Using external container registries in GitLab is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/376217)
+in GitLab 15.8 and the end of support is scheduled for GitLab 16.0.
+If you need to use external container registries instead of the GitLab Container Registry,
+tell us about your use cases in [feedback issue 958](https://gitlab.com/gitlab-org/container-registry/-/issues/958).
If you use an external container registry, some features associated with the
container registry may be unavailable or have [inherent risks](../../user/packages/container_registry/reduce_container_registry_storage.md#use-with-external-container-registries).
@@ -871,7 +877,7 @@ You can use GitLab as an auth endpoint with an external container registry.
## Configure Container Registry notifications
You can configure the Container Registry to send webhook notifications in
-response to events happening within the registry.
+response to events happening in the registry.
Read more about the Container Registry notifications configuration options in the
[Docker Registry notifications documentation](https://docs.docker.com/registry/notifications/).
@@ -924,13 +930,17 @@ notifications:
WARNING:
If you're using a distributed architecture and Sidekiq is running on a different node, the cleanup
-policies don't work. To fix this, you must configure the `gitlab.rb` file on the Sidekiq nodes to
-point to the correct registry URL and copy the `registry.key` file to each Sidekiq node. For more
-information, see the [Sidekiq configuration](../sidekiq/index.md)
+policies don't work. To fix this:
+
+1. Configure the `gitlab.rb` file on the Sidekiq nodes to
+ point to the correct registry URL.
+1. Copy the `registry.key` file to each Sidekiq node.
+
+For more information, see the [Sidekiq configuration](../sidekiq/index.md)
page.
To reduce the amount of [Container Registry disk space used by a given project](#registry-disk-space-usage-by-project),
-administrators can clean up image tags
+administrators can setup cleanup policies
and [run garbage collection](#container-registry-garbage-collection).
### Registry Disk Space Usage by Project
@@ -993,7 +1003,7 @@ You can also [run cleanup on a schedule](../../user/packages/container_registry/
## Container Registry garbage collection
NOTE:
-Retention policies within your object storage provider, such as Amazon S3 Lifecycle, may prevent
+Retention policies in your object storage provider, such as Amazon S3 Lifecycle, may prevent
objects from being properly deleted.
Container Registry can use considerable amounts of disk space. To clear up
@@ -1037,9 +1047,9 @@ docker build -t my.registry.com/my.group/my.project:latest .
docker push my.registry.com/my.group/my.project:latest
```
-Now, the `:latest` tag points to manifest of `sha256:222222`. However, due to
-the architecture of registry, this data is still accessible when pulling the
-image `my.registry.com/my.group/my.project@sha256:111111`, even though it is
+Now, the `:latest` tag points to manifest of `sha256:222222`.
+Due to the architecture of registry, this data is still accessible when pulling the
+image `my.registry.com/my.group/my.project@sha256:111111`, though it is
no longer directly accessible via the `:latest` tag.
### Recycling unused tags
diff --git a/doc/administration/packages/dependency_proxy.md b/doc/administration/packages/dependency_proxy.md
index 78efd75bbe3..a29d6d93051 100644
--- a/doc/administration/packages/dependency_proxy.md
+++ b/doc/administration/packages/dependency_proxy.md
@@ -122,7 +122,7 @@ To change the local storage path:
Instead of relying on the local storage, you can use an object storage to
store the blobs of the Dependency Proxy. In GitLab 13.2 and later, you should use the
-[consolidated object storage settings](../object_storage.md#consolidated-object-storage-configuration).
+[consolidated object storage settings](../object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form).
This section describes the earlier configuration format. [Migration steps still apply](#migrate-local-dependency-proxy-blobs-and-manifests-to-object-storage).
[Read more about using object storage with GitLab](../object_storage.md).
diff --git a/doc/administration/packages/index.md b/doc/administration/packages/index.md
index 6e53d77109f..f0f238aa5ba 100644
--- a/doc/administration/packages/index.md
+++ b/doc/administration/packages/index.md
@@ -124,7 +124,7 @@ Instead of relying on the local storage, you can use an object storage to store
packages.
For more information, see how to use the
-[consolidated object storage settings](../object_storage.md#consolidated-object-storage-configuration).
+[consolidated object storage settings](../object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form).
### Migrate local packages to object storage
diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md
index ed08b10fe97..1626a4fd41a 100644
--- a/doc/administration/pages/index.md
+++ b/doc/administration/pages/index.md
@@ -1,8 +1,7 @@
---
-stage: Create
-group: Editor
+stage: Plan
+group: Knowledge
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
-description: 'Learn how to administer GitLab Pages.'
---
# GitLab Pages administration **(FREE SELF)**
@@ -15,7 +14,7 @@ This guide is for Omnibus GitLab installations. If you have installed
GitLab from source, see
[GitLab Pages administration for source installations](source.md).
-## Overview
+## The GitLab Pages daemon
GitLab Pages makes use of the [GitLab Pages daemon](https://gitlab.com/gitlab-org/gitlab-pages), a basic HTTP server
written in Go that can listen on an external IP address and provide support for
@@ -247,20 +246,20 @@ control over how the Pages daemon runs and serves content in your environment.
| `enable` | Enable or disable GitLab Pages on the current system. |
| `external_http` | Configure Pages to bind to one or more secondary IP addresses, serving HTTP requests. Multiple addresses can be given as an array, along with exact ports, for example `['1.2.3.4', '1.2.3.5:8063']`. Sets value for `listen_http`. |
| `external_https` | Configure Pages to bind to one or more secondary IP addresses, serving HTTPS requests. Multiple addresses can be given as an array, along with exact ports, for example `['1.2.3.4', '1.2.3.5:8063']`. Sets value for `listen_https`. |
-| `server_shutdown_timeout` | GitLab Pages server shutdown timeout in seconds (default: 30 s). |
-| `gitlab_client_http_timeout` | GitLab API HTTP client connection timeout in seconds (default: 10 s). |
-| `gitlab_client_jwt_expiry` | JWT Token expiry time in seconds (default: 30 s). |
-| `gitlab_cache_expiry` | The maximum time a domain's configuration is stored in the cache (default: 600 s). |
-| `gitlab_cache_refresh` | The interval at which a domain's configuration is set to be due to refresh (default: 60 s). |
-| `gitlab_cache_cleanup` | The interval at which expired items are removed from the cache (default: 60 s). |
-| `gitlab_retrieval_timeout` | The maximum time to wait for a response from the GitLab API per request (default: 30 s). |
-| `gitlab_retrieval_interval` | The interval to wait before retrying to resolve a domain's configuration via the GitLab API (default: 1 s). |
+| `server_shutdown_timeout` | GitLab Pages server shutdown timeout in seconds (default: `30s`). |
+| `gitlab_client_http_timeout` | GitLab API HTTP client connection timeout in seconds (default: `10s`). |
+| `gitlab_client_jwt_expiry` | JWT Token expiry time in seconds (default: `30s`). |
+| `gitlab_cache_expiry` | The maximum time a domain's configuration is stored in the cache (default: `600s`). |
+| `gitlab_cache_refresh` | The interval at which a domain's configuration is set to be due to refresh (default: `60s`). |
+| `gitlab_cache_cleanup` | The interval at which expired items are removed from the cache (default: `60s`). |
+| `gitlab_retrieval_timeout` | The maximum time to wait for a response from the GitLab API per request (default: `30s`). |
+| `gitlab_retrieval_interval` | The interval to wait before retrying to resolve a domain's configuration via the GitLab API (default: `1s`). |
| `gitlab_retrieval_retries` | The maximum number of times to retry to resolve a domain's configuration via the API (default: 3). |
| `domain_config_source` | This parameter was removed in 14.0, on earlier versions it can be used to enable and test API domain configuration source |
| `gitlab_id` | The OAuth application public ID. Leave blank to automatically fill when Pages authenticates with GitLab. |
| `gitlab_secret` | The OAuth application secret. Leave blank to automatically fill when Pages authenticates with GitLab. |
| `auth_scope` | The OAuth application scope to use for authentication. Must match GitLab Pages OAuth application settings. Leave blank to use `api` scope by default. |
-| `auth_cookie_session_timeout` | Authentication cookie session timeout in seconds (default: 600 s). A value of `0` means the cookie is deleted after the browser session ends. |
+| `auth_cookie_session_timeout` | Authentication cookie session timeout in seconds (default: `10m`). A value of `0` means the cookie is deleted after the browser session ends. |
| `gitlab_server` | Server to use for authentication when access control is enabled; defaults to GitLab `external_url`. |
| `headers` | Specify any additional http headers that should be sent to the client with each response. Multiple headers can be given as an array, header and value as one string, for example `['my-header: myvalue', 'my-other-header: my-other-value']` |
| `enable_disk` | Allows the GitLab Pages daemon to serve content from disk. Shall be disabled if shared disk storage isn't available. |
@@ -300,6 +299,10 @@ control over how the Pages daemon runs and serves content in your environment.
| `rate_limit_source_ip_burst` | Rate limit per source IP maximum burst allowed per second. |
| `rate_limit_domain` | Rate limit per domain in number of requests per second. Set to `0` to disable this feature. |
| `rate_limit_domain_burst` | Rate limit per domain maximum burst allowed per second. |
+| `rate_limit_tls_source_ip` | Rate limit per source IP in number of TLS connections per second. Set to `0` to disable this feature. |
+| `rate_limit_tls_source_ip_burst` | Rate limit per source IP maximum TLS connections burst allowed per second. |
+| `rate_limit_tls_domain` | Rate limit per domain in number of TLS connections per second. Set to `0` to disable this feature. |
+| `rate_limit_tls_domain_burst` | Rate limit per domain maximum TLS connections burst allowed per second. |
| `server_read_timeout` | Maximum duration to read the request headers and body. For no timeout, set to `0` or a negative value. Default: `5s` |
| `server_read_header_timeout` | Maximum duration to read the request headers. For no timeout, set to `0` or a negative value. Default: `1s` |
| `server_write_timeout` | Maximum duration to write all files in the response. Larger files require more time. For no timeout, set to `0` or a negative value. Default: `0` |
@@ -514,7 +517,7 @@ internet connectivity is gated by a proxy. To use a proxy for GitLab Pages:
### Using a custom Certificate Authority (CA)
When using certificates issued by a custom CA, [Access Control](../../user/project/pages/pages_access_control.md) and
-the [online view of HTML job artifacts](../../ci/pipelines/job_artifacts.md#download-job-artifacts)
+the [online view of HTML job artifacts](../../ci/jobs/job_artifacts.md#download-job-artifacts)
fails to work if the custom CA is not recognized.
This usually results in this error:
@@ -523,7 +526,7 @@ This usually results in this error:
For installation from source, this can be fixed by installing the custom Certificate
Authority (CA) in the system certificate store.
-For Omnibus, this is fixed by [installing a custom CA in Omnibus GitLab](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates).
+For Omnibus, this is fixed by [installing a custom CA in Omnibus GitLab](https://docs.gitlab.com/omnibus/settings/ssl/index.html#install-custom-public-certificates).
### ZIP serving and cache configuration
@@ -540,22 +543,22 @@ archive. You can modify the cache behavior by changing the following configurati
| Setting | Description |
| ------- | ----------- |
-| `zip_cache_expiration` | The cache expiration interval of ZIP archives. Must be greater than zero to avoid serving stale content. Default is 60 s. |
-| `zip_cache_cleanup` | The interval at which archives are cleaned from memory if they have already expired. Default is 30 s. |
-| `zip_cache_refresh` | The time interval in which an archive is extended in memory if accessed before `zip_cache_expiration`. This works together with `zip_cache_expiration` to determine if an archive is extended in memory. See the [example below](#zip-cache-refresh-example) for important details. Default is 30 s. |
+| `zip_cache_expiration` | The cache expiration interval of ZIP archives. Must be greater than zero to avoid serving stale content. Default is `60s`. |
+| `zip_cache_cleanup` | The interval at which archives are cleaned from memory if they have already expired. Default is `30s`. |
+| `zip_cache_refresh` | The time interval in which an archive is extended in memory if accessed before `zip_cache_expiration`. This works together with `zip_cache_expiration` to determine if an archive is extended in memory. See the [example below](#zip-cache-refresh-example) for important details. Default is `30s`. |
| `zip_open_timeout` | The maximum time allowed to open a ZIP archive. Increase this time for big archives or slow network connections, as doing so may affect the latency of serving Pages. Default is 30 s. |
-| `zip_http_client_timeout` | The maximum time for the ZIP HTTP client. Default is 30 m. |
+| `zip_http_client_timeout` | The maximum time for the ZIP HTTP client. Default is `30m`. |
#### ZIP cache refresh example
Archives are refreshed in the cache (extending the time they are held in memory) if they're accessed
before `zip_cache_expiration`, and the time left before expiring is less than or equal to
-`zip_cache_refresh`. For example, if `archive.zip` is accessed at time 0 s, it expires in 60 s (the
-default for `zip_cache_expiration`). In the example below, if the archive is opened again after 15 s
-it is **not** refreshed because the time left for expiry (45 s) is greater than `zip_cache_refresh`
-(default 30 s). However, if the archive is accessed again after 45 s (from the first time it was
+`zip_cache_refresh`. For example, if `archive.zip` is accessed at time `0s`, it expires in `60s` (the
+default for `zip_cache_expiration`). In the example below, if the archive is opened again after `15s`
+it is **not** refreshed because the time left for expiry (`45s`) is greater than `zip_cache_refresh`
+(default `30s`). However, if the archive is accessed again after `45s` (from the first time it was
opened) it's refreshed. This extends the time the archive remains in memory from
-`45s + zip_cache_expiration (60s)`, for a total of 105 s.
+`45s + zip_cache_expiration (60s)`, for a total of `105s`.
After an archive reaches `zip_cache_expiration`, it's marked as expired and removed on the next
`zip_cache_cleanup` interval.
@@ -631,34 +634,6 @@ are stored.
1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
-Alternatively, if you have existing Pages deployed you can follow
-the below steps to do a no downtime transfer to a new storage location.
-
-1. Pause Pages deployments by setting the following in `/etc/gitlab/gitlab.rb`:
-
- ```ruby
- sidekiq['queue_selector'] = true
- sidekiq['queue_groups'] = [
- "feature_category!=pages"
- ]
- ```
-
-1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
-1. `rsync` contents from the current storage location to the new storage location: `sudo rsync -avzh --progress /var/opt/gitlab/gitlab-rails/shared/pages/ /mnt/storage/pages`
-1. Set the new storage location in `/etc/gitlab/gitlab.rb`:
-
- ```ruby
- gitlab_rails['pages_path'] = "/mnt/storage/pages"
- ```
-
-1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
-1. Verify Pages are still being served up as expected.
-1. Resume Pages deployments by removing from `/etc/gitlab/gitlab.rb` the `sidekiq` setting set above.
-1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
-1. Trigger a new Pages deployment and verify it's working as expected.
-1. Remove the old Pages storage location: `sudo rm -rf /var/opt/gitlab/gitlab-rails/shared/pages`
-1. Verify Pages are still being served up as expected.
-
## Configure listener for reverse proxy requests
Follow the steps below to configure the proxy listener of GitLab Pages.
@@ -683,23 +658,23 @@ Follow the steps below to configure the proxy listener of GitLab Pages.
## Set global maximum size of each GitLab Pages site **(FREE SELF)**
-Prerequisites:
+Prerequisite:
-- Only GitLab administrators can edit this setting.
+- You must have administrator access to the instance.
To set the global maximum pages size for a project:
1. On the top bar, select **Main menu > Admin**.
1. On the left sidebar, select **Settings > Preferences**.
1. Expand **Pages**.
-1. Enter a value under **Maximum size of pages**.
+1. In **Maximum size of pages**, enter a value. The default is `100`.
1. Select **Save changes**.
## Set maximum size of each GitLab Pages site in a group **(PREMIUM SELF)**
-Prerequisites:
+Prerequisite:
-- You must have at least the Maintainer role for the group.
+- You must have administrator access to the instance.
To set the maximum size of each GitLab Pages site in a group, overriding the inherited setting:
@@ -711,9 +686,9 @@ To set the maximum size of each GitLab Pages site in a group, overriding the inh
## Set maximum size of GitLab Pages site in a project **(PREMIUM SELF)**
-Prerequisites:
+Prerequisite:
-- You must have at least the Maintainer role for the project.
+- You must have administrator access to the instance.
To set the maximum size of GitLab Pages site in a project, overriding the inherited setting:
@@ -729,7 +704,7 @@ To set the maximum size of GitLab Pages site in a project, overriding the inheri
Prerequisite:
-- You must be an administrator of a self-managed GitLab instance.
+- You must have administrator access to the instance.
To set the maximum number of GitLab Pages custom domains for a project:
@@ -738,6 +713,15 @@ To set the maximum number of GitLab Pages custom domains for a project:
1. Enter a value for **Maximum number of custom domains per project**. Use `0` for unlimited domains.
1. Select **Save changes**.
+## Set maximum number of files per GitLab Pages website
+
+The total number of file entries (including directories and symlinks) is limited to `200,000` per GitLab Pages website.
+
+You can update the limit in your self-managed instance using the
+[GitLab Rails console](../operations/rails_console.md#starting-a-rails-console-session).
+
+For more information, see [GitLab application limits](../instance_limits.md#number-of-files-per-gitlab-pages-website).
+
## Running GitLab Pages on a separate server
You can run the GitLab Pages daemon on a separate server to decrease the load on
@@ -846,7 +830,7 @@ From [GitLab 13.3 to GitLab 13.12](#domain-source-configuration-before-140) GitL
Starting from [GitLab 14.0](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5993) GitLab Pages uses API
by default and fails to start if it can't connect to it.
-For common issues, see the [troubleshooting section](#failed-to-connect-to-the-internal-gitlab-api).
+For common issues, see [troubleshooting](troubleshooting.md#failed-to-connect-to-the-internal-gitlab-api).
For more details see this [blog post](https://about.gitlab.com/blog/2020/08/03/how-gitlab-pages-uses-the-gitlab-api-to-serve-content/).
@@ -891,7 +875,7 @@ Incorrect configuration of these values may result in intermittent
or persistent errors, or the Pages Daemon serving old content.
NOTE:
-Expiry, interval and timeout flags use [Golang's duration formatting](https://pkg.go.dev/time#ParseDuration).
+Expiry, interval and timeout flags use [Go duration formatting](https://pkg.go.dev/time#ParseDuration).
A duration string is a possibly signed sequence of decimal numbers,
each with optional fraction and a unit suffix, such as `300ms`, `1.5h` or `2h45m`.
Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`.
@@ -923,7 +907,7 @@ configuration is tried to be resolved automatically before reporting an error.
> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5577) in GitLab 13.6.
-[Read more about using object storage with GitLab](../object_storage.md).
+For more information, see [object storage](../object_storage.md).
### Object storage settings
@@ -945,10 +929,10 @@ If you want to stop using and disconnect the NFS server, you need to
#### S3-compatible connection settings
In GitLab 13.2 and later, you should use the
-[consolidated object storage settings](../object_storage.md#consolidated-object-storage-configuration).
+[consolidated object storage settings](../object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form).
This section describes the earlier configuration format.
-See [the available connection settings for different providers](../object_storage.md#connection-settings).
+See [the available connection settings for different providers](../object_storage.md#configure-the-connection-settings).
In Omnibus installations:
@@ -1128,8 +1112,8 @@ In GitLab 14.0 a number of breaking changes were introduced which may require so
The steps below describe the best way to migrate without causing any downtime for your GitLab instance.
A GitLab instance running on a single server typically upgrades to 14.0 smoothly, and there should be minimal issues after the upgrade is complete.
-Regardless, we recommend everyone follow the migration steps to ensure a successful upgrade.
-If at any point you run into issues, consult the [troubleshooting section](#troubleshooting).
+Regardless, you should follow the migration steps to ensure a successful upgrade.
+For common issues, see [troubleshooting](troubleshooting.md).
If your current GitLab version is lower than 13.12, then you must first update to 13.12.
Updating directly to 14.0 is [not supported](../../update/index.md#upgrade-paths)
@@ -1158,14 +1142,14 @@ than GitLab to prevent XSS attacks.
You can enforce rate limits to help minimize the risk of a Denial of Service (DoS) attack. GitLab Pages
uses a [token bucket algorithm](https://en.wikipedia.org/wiki/Token_bucket) to enforce rate limiting. By default,
-requests that exceed the specified limits are reported but not rejected.
+requests or TLS connections that exceed the specified limits are reported but not rejected.
GitLab Pages supports the following types of rate limiting:
-- Per `source_ip`. It limits how many requests are allowed from the single client IP address.
-- Per `domain`. It limits how many requests are allowed per domain hosted on GitLab Pages. It can be a custom domain like `example.com`, or group domain like `group.gitlab.io`.
+- Per `source_ip`. It limits how many requests or TLS connections are allowed from the single client IP address.
+- Per `domain`. It limits how many requests or TLS connections are allowed per domain hosted on GitLab Pages. It can be a custom domain like `example.com`, or group domain like `group.gitlab.io`.
-Rate limits are enforced using the following:
+HTTP request-based rate limits are enforced using the following:
- `rate_limit_source_ip`: Set the maximum threshold in number of requests per client IP per second. Set to 0 to disable this feature.
- `rate_limit_source_ip_burst`: Sets the maximum threshold of number of requests allowed in an initial outburst of requests per client IP.
@@ -1173,7 +1157,15 @@ Rate limits are enforced using the following:
- `rate_limit_domain`: Set the maximum threshold in number of requests per hosted pages domain per second. Set to 0 to disable this feature.
- `rate_limit_domain_burst`: Sets the maximum threshold of number of requests allowed in an initial outburst of requests per hosted pages domain.
-#### Enable source-IP rate limits
+TLS connection-based rate limits are enforced using the following:
+
+- `rate_limit_tls_source_ip`: Set the maximum threshold in number of TLS connections per client IP per second. Set to 0 to disable this feature.
+- `rate_limit_tls_source_ip_burst`: Sets the maximum threshold of number of TLS connections allowed in an initial outburst of TLS connections per client IP.
+ For example, when you load a web page from different web browsers at the same time.
+- `rate_limit_tls_domain`: Set the maximum threshold in number of TLS connections per hosted pages domain per second. Set to 0 to disable this feature.
+- `rate_limit_tls_domain_burst`: Sets the maximum threshold of number of TLS connections allowed in an initial outburst of TLS connections per hosted pages domain.
+
+#### Enable HTTP requests rate limits by source-IP
> [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/631) in GitLab 14.5.
@@ -1184,16 +1176,9 @@ Rate limits are enforced using the following:
gitlab_pages['rate_limit_source_ip_burst'] = 600
```
-1. To reject requests that exceed the specified limits, enable the `FF_ENFORCE_IP_RATE_LIMITS` feature flag in
- `/etc/gitlab/gitlab.rb`:
-
- ```ruby
- gitlab_pages['env'] = {'FF_ENFORCE_IP_RATE_LIMITS' => 'true'}
- ```
-
1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
-#### Enable domain rate limits
+#### Enable HTTP requests rate limits by domain
> [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/630) in GitLab 14.7.
@@ -1204,321 +1189,34 @@ Rate limits are enforced using the following:
gitlab_pages['rate_limit_domain_burst'] = 5000
```
-1. To reject requests that exceed the specified limits, enable the `FF_ENFORCE_DOMAIN_RATE_LIMITS` feature flag in
- `/etc/gitlab/gitlab.rb`:
-
- ```ruby
- gitlab_pages['env'] = {'FF_ENFORCE_DOMAIN_RATE_LIMITS' => 'true'}
- ```
-
1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
-<!-- ## Troubleshooting
-
-Include any troubleshooting steps that you can foresee. If you know beforehand what issues
-one might have when setting this up, or when something is changed, or on upgrading, it's
-important to describe those, too. Think of things that may go wrong and include them in
-the section below.
-This is important to minimize requests for support, and to avoid doc comments with
-questions that you know someone might ask.
-
-Each scenario can be a third-level heading, for example `### Getting error message X`.
-If you have none to add when creating a doc, leave this section in place
-but commented out to help encourage others to add to it in the future. -->
-
-## Troubleshooting
-
-### How to see GitLab Pages logs
-
-You can see Pages daemon logs by running:
-
-```shell
-sudo gitlab-ctl tail gitlab-pages
-```
-
-You can also find the log file in `/var/log/gitlab/gitlab-pages/current`.
-
-### `unsupported protocol scheme \"\""`
-
-If you see the following error:
-
-```plaintext
-{"error":"failed to connect to internal Pages API: Get \"/api/v4/internal/pages/status\": unsupported protocol scheme \"\"","level":"warning","msg":"attempted to connect to the API","time":"2021-06-23T20:03:30Z"}
-```
-
-It means you didn't set the HTTP(S) protocol scheme in the Pages server settings.
-To fix it:
-
-1. Edit `/etc/gitlab/gitlab.rb`:
-
- ```ruby
- gitlab_pages['gitlab_server'] = "https://<your_pages_domain_name>"
- gitlab_pages['internal_gitlab_server'] = "https://<your_pages_domain_name>"
- ```
-
-1. Reconfigure GitLab:
-
- ```shell
- sudo gitlab-ctl reconfigure
- ```
-
-### 502 error when connecting to GitLab Pages proxy when server does not listen over IPv6
-
-In some cases, NGINX might default to using IPv6 to connect to the GitLab Pages
-service even when the server does not listen over IPv6. You can identify when
-this is happening if you see something similar to the log entry below in the
-`gitlab_pages_error.log`:
-
-```plaintext
-2020/02/24 16:32:05 [error] 112654#0: *4982804 connect() failed (111: Connection refused) while connecting to upstream, client: 123.123.123.123, server: ~^(?<group>.*)\.pages\.example\.com$, request: "GET /-/group/project/-/jobs/1234/artifacts/artifact.txt HTTP/1.1", upstream: "http://[::1]:8090//-/group/project/-/jobs/1234/artifacts/artifact.txt", host: "group.example.com"
-```
-
-To resolve this, set an explicit IP and port for the GitLab Pages `listen_proxy` setting
-to define the explicit address that the GitLab Pages daemon should listen on:
-
-```ruby
-gitlab_pages['listen_proxy'] = '127.0.0.1:8090'
-```
-
-### Intermittent 502 errors or after a few days
-
-If you run Pages on a system that uses `systemd` and
-[`tmpfiles.d`](https://www.freedesktop.org/software/systemd/man/tmpfiles.d.html),
-you may encounter intermittent 502 errors trying to serve Pages with an error similar to:
-
-```plaintext
-dial tcp: lookup gitlab.example.com on [::1]:53: dial udp [::1]:53: connect: no route to host"
-```
-
-GitLab Pages creates a [bind mount](https://man7.org/linux/man-pages/man8/mount.8.html)
-inside `/tmp/gitlab-pages-*` that includes files like `/etc/hosts`.
-However, `systemd` may clean the `/tmp/` directory on a regular basis so the DNS
-configuration may be lost.
-
-To stop `systemd` from cleaning the Pages related content:
-
-1. Tell `tmpfiles.d` to not remove the Pages `/tmp` directory:
-
- ```shell
- echo 'x /tmp/gitlab-pages-*' >> /etc/tmpfiles.d/gitlab-pages-jail.conf
- ```
-
-1. Restart GitLab Pages:
-
- ```shell
- sudo gitlab-ctl restart gitlab-pages
- ```
-
-### Unable to access GitLab Pages
-
-If you can't access your GitLab Pages (such as receiving `502 Bad Gateway` errors, or a login loop)
-and in your Pages log shows this error:
-
-```plaintext
-"error":"retrieval context done: context deadline exceeded","host":"root.docs-cit.otenet.gr","level":"error","msg":"could not fetch domain information from a source"
-```
-
-1. Add the following to `/etc/gitlab/gitlab.rb`:
-
- ```ruby
- gitlab_pages['internal_gitlab_server'] = 'http://localhost:8080'
- ```
-
-1. Restart GitLab Pages:
-
- ```shell
- sudo gitlab-ctl restart gitlab-pages
- ```
-
-### Failed to connect to the internal GitLab API
-
-If you see the following error:
+#### Enable TLS connections rate limits by source-IP
-```plaintext
-ERRO[0010] Failed to connect to the internal GitLab API after 0.50s error="failed to connect to internal Pages API: HTTP status: 401"
-```
-
-If you are [Running GitLab Pages on a separate server](#running-gitlab-pages-on-a-separate-server)
-you must copy the `/etc/gitlab/gitlab-secrets.json` file
-from the **GitLab server** to the **Pages server** after upgrading to GitLab 13.3,
-as described in that section.
-
-Other reasons may include network connectivity issues between your
-**GitLab server** and your **Pages server** such as firewall configurations or closed ports.
-For example, if there is a connection timeout:
-
-```plaintext
-error="failed to connect to internal Pages API: Get \"https://gitlab.example.com:3000/api/v4/internal/pages/status\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
-```
-
-### Pages cannot communicate with an instance of the GitLab API
-
-If you use the default value for `domain_config_source=auto` and run multiple instances of GitLab
-Pages, you may see intermittent 502 error responses while serving Pages content. You may also see
-the following warning in the Pages logs:
-
-```plaintext
-WARN[0010] Pages cannot communicate with an instance of the GitLab API. Please sync your gitlab-secrets.json file https://gitlab.com/gitlab-org/gitlab-pages/-/issues/535#workaround. error="pages endpoint unauthorized"
-```
-
-This can happen if your `gitlab-secrets.json` file is out of date between GitLab Rails and GitLab
-Pages. Follow steps 8-10 of [Running GitLab Pages on a separate server](#running-gitlab-pages-on-a-separate-server),
-in all of your GitLab Pages instances.
-
-### Intermittent 502 errors when using an AWS Network Load Balancer and GitLab Pages
-
-Connections will time out when using a Network Load Balancer with client IP preservation enabled and [the request is looped back to the source server](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-troubleshooting.html#loopback-timeout).
-This can happen to GitLab instances with multiple servers
-running both the core GitLab application and GitLab Pages. This can also happen when a single
-container is running both the core GitLab application and GitLab Pages.
-
-AWS [recommends using an IP target type](https://aws.amazon.com/premiumsupport/knowledge-center/target-connection-fails-load-balancer/)
-to resolve this issue.
-
-Turning off [client IP preservation](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html#client-ip-preservation)
-may resolve this issue when the core GitLab application and GitLab Pages run on the same host or
-container.
-
-### 500 error with `securecookie: failed to generate random iv` and `Failed to save the session`
-
-This problem most likely results from an [out-dated operating system](../package_information/supported_os.md#os-versions-that-are-no-longer-supported).
-The [Pages daemon uses the `securecookie` library](https://gitlab.com/search?group_id=9970&project_id=734943&repository_ref=master&scope=blobs&search=securecookie&snippets=false) to get random strings via [`crypto/rand` in Go](https://pkg.go.dev/crypto/rand#pkg-variables).
-This requires the `getrandom` system call or `/dev/urandom` to be available on the host OS.
-Upgrading to an [officially supported operating system](https://about.gitlab.com/install/) is recommended.
-
-### The requested scope is invalid, malformed, or unknown
-
-This problem comes from the permissions of the GitLab Pages OAuth application. To fix it:
-
-1. On the top bar, select **Main menu > Admin**.
-1. On the left sidebar, select **Applications > GitLab Pages**.
-1. Edit the application.
-1. Under **Scopes**, ensure that the `api` scope is selected.
-1. Save your changes.
-
-When running a [separate Pages server](#running-gitlab-pages-on-a-separate-server),
-this setting needs to be configured on the main GitLab server.
-
-### Workaround in case no wildcard DNS entry can be set
-
-If the wildcard DNS [prerequisite](#prerequisites) can't be met, you can still use GitLab Pages in a limited fashion:
-
-1. [Move](../../user/project/settings/index.md#transfer-a-project-to-another-namespace)
- all projects you need to use Pages with into a single group namespace, for example `pages`.
-1. Configure a [DNS entry](#dns-configuration) without the `*.`-wildcard, for example `pages.example.io`.
-1. Configure `pages_external_url http://example.io/` in your `gitlab.rb` file.
- Omit the group namespace here, because it automatically is prepended by GitLab.
-
-### Pages daemon fails with permission denied errors
-
-If `/tmp` is mounted with `noexec`, the Pages daemon fails to start with an error like:
-
-```plaintext
-{"error":"fork/exec /gitlab-pages: permission denied","level":"fatal","msg":"could not create pages daemon","time":"2021-02-02T21:54:34Z"}
-```
+> [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/632) in GitLab 14.9.
-In this case, change `TMPDIR` to a location that is not mounted with `noexec`. Add the following to
-`/etc/gitlab/gitlab.rb`:
-
-```ruby
-gitlab_pages['env'] = {'TMPDIR' => '<new_tmp_path>'}
-```
-
-Once added, reconfigure with `sudo gitlab-ctl reconfigure` and restart GitLab with
-`sudo gitlab-ctl restart`.
-
-### `The redirect URI included is not valid.` when using Pages Access Control
-
-You may see this error if `pages_external_url` was updated at some point of time. Verify the following:
-
-1. The **Callback URL**/Redirect URI in the GitLab Pages [System OAuth application](../../integration/oauth_provider.md#create-an-instance-wide-application)
-is using the protocol (HTTP or HTTPS) that `pages_external_url` is configured to use.
-1. The domain and path components of `Redirect URI` are valid: they should look like `projects.<pages_external_url>/auth`.
-
-### 500 error `cannot serve from disk`
-
-If you get a 500 response from Pages and encounter an error similar to:
-
-```plaintext
-ERRO[0145] cannot serve from disk error="gitlab: disk access is disabled via enable-disk=false" project_id=27 source_path="file:///shared/pages/@hashed/67/06/670671cd97404156226e507973f2ab8330d3022ca96e0c93bdbdb320c41adcaf/pages_deployments/14/artifacts.zip" source_type=zip
-```
-
-It means that GitLab Rails is telling GitLab Pages to serve content from a location on disk,
-however, GitLab Pages was configured to disable disk access.
-
-To enable disk access:
-
-1. Enable disk access for GitLab Pages in `/etc/gitlab/gitlab.rb`:
+1. Set rate limits in `/etc/gitlab/gitlab.rb`:
```ruby
- gitlab_pages['enable_disk'] = true
+ gitlab_pages['rate_limit_tls_source_ip'] = 20.0
+ gitlab_pages['rate_limit_tls_source_ip_burst'] = 600
```
1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
-### `httprange: new resource 403`
-
-If you see an error similar to:
-
-```plaintext
-{"error":"httprange: new resource 403: \"403 Forbidden\"","host":"root.pages.example.com","level":"error","msg":"vfs.Root","path":"/pages1/","time":"2021-06-10T08:45:19Z"}
-```
-
-And you run pages on the separate server syncing files via NFS, it may mean that
-the shared pages directory is mounted on a different path on the main GitLab server and the
-GitLab Pages server.
+#### Enable TLS connections rate limits by domain
-In that case, it's highly recommended you to configure
-[object storage and migrate any existing pages data to it](#using-object-storage).
+> [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/632) in GitLab 14.9.
-Alternatively, you can mount the GitLab Pages shared directory to the same path on
-both servers.
-
-### GitLab Pages doesn't work after upgrading to GitLab 14.0 or above
-
-GitLab 14.0 introduces a number of changes to GitLab Pages which may require manual intervention.
-
-1. Firstly [follow the migration guide](#prepare-gitlab-pages-for-140).
-1. Try to upgrade to GitLab 14.3 or above. Some of the issues were fixed in GitLab 14.1, 14.2 and 14.3.
-1. If it doesn't work, see [GitLab Pages logs](#how-to-see-gitlab-pages-logs), and if you see any errors there then search them on this page.
-
-WARNING:
-In GitLab 14.0-14.2 you can temporarily enable legacy storage and configuration mechanisms.
-
-To do that:
-
-1. Describe the issue you're seeing in the [migration feedback issue](https://gitlab.com/gitlab-org/gitlab/-/issues/331699).
-
-1. Edit `/etc/gitlab/gitlab.rb`:
+1. Set rate limits in `/etc/gitlab/gitlab.rb`:
```ruby
- gitlab_pages['use_legacy_storage'] = true
+ gitlab_pages['rate_limit_tls_domain'] = 1000
+ gitlab_pages['rate_limit_tls_domain_burst'] = 5000
```
1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
-### GitLab Pages deploy job fails with error "is not a recognized provider"
-
-If the **pages** job succeeds but the **deploy** job gives the error "is not a recognized provider":
-
-![Pages Deploy Failure](img/pages_deploy_failure_v14_8.png)
-
-The error message `is not a recognized provider` could be coming from the `fog` gem that GitLab uses to connect to cloud providers for object storage.
-
-To fix that:
-
-1. Check your `gitlab.rb` file. If you have `gitlab_rails['pages_object_store_enabled']` enabled, but no bucket details have been configured, either:
-
- - Configure object storage for your Pages deployments, following the [S3-compatible connection settings](#s3-compatible-connection-settings) guide.
- - Store your deployments locally, by commenting out that line.
-
-1. Save the changes you made to your `gitlab.rb` file, then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
-
-### 404 error `The page you're looking for could not be found`
-
-If you get a `404 Page Not Found` response from GitLab Pages:
-
-1. Check `.gitlab-ci.yml` contains the job `pages:`.
-1. Check the current project's pipeline to confirm the job `pages:deploy` is being run.
+## Related topics
-Without the `pages:deploy` job, the updates to your GitLab Pages site are never published.
+- [Troubleshooting GitLab Pages administration](troubleshooting.md)
diff --git a/doc/administration/pages/source.md b/doc/administration/pages/source.md
index db76d15ec58..b36b87f3b1d 100644
--- a/doc/administration/pages/source.md
+++ b/doc/administration/pages/source.md
@@ -1,6 +1,6 @@
---
-stage: Create
-group: Editor
+stage: Plan
+group: Knowledge
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
@@ -10,8 +10,8 @@ NOTE:
Before attempting to enable GitLab Pages, first make sure you have
[installed GitLab](../../install/installation.md) successfully.
-This is the documentation for configuring a GitLab Pages when you have installed
-GitLab from source and not using the Omnibus packages.
+This document explains how to configure GitLab Pages when you have installed
+GitLab from source and not the Omnibus packages.
You are encouraged to read the [Omnibus documentation](index.md) as it provides
invaluable information about the configuration of GitLab Pages.
@@ -20,12 +20,11 @@ We also highly recommend that you use the Omnibus GitLab packages. We
optimize them specifically for GitLab, and we take care of upgrading GitLab
Pages to the latest supported version.
-## Overview
+## How GitLab Pages works
-GitLab Pages makes use of the [GitLab Pages daemon](https://gitlab.com/gitlab-org/gitlab-pages), a simple HTTP server
-written in Go that can listen on an external IP address and provide support for
-custom domains and custom certificates. It supports dynamic certificates through
-SNI and exposes pages using HTTP2 by default.
+GitLab Pages makes use of the [GitLab Pages daemon](https://gitlab.com/gitlab-org/gitlab-pages), a lightweight HTTP server that listens on an external IP address and provides support for
+custom domains and certificates. It supports dynamic certificates through
+`SNI` and exposes pages using HTTP2 by default.
You are encouraged to read its [README](https://gitlab.com/gitlab-org/gitlab-pages/blob/master/README.md)
to fully understand how it works.
@@ -57,7 +56,7 @@ Before proceeding with the Pages configuration, make sure that:
assume that to be `example.io`.
- You have configured a **wildcard DNS record** for that domain.
- You have installed the `zip` and `unzip` packages in the same server that
- GitLab is installed since they are needed to compress and decompress the
+ GitLab is installed because they are needed to compress and decompress the
Pages artifacts.
- Optional. You have a **wildcard certificate** for the Pages domain if you
decide to serve Pages (`*.example.io`) under HTTPS.
@@ -86,7 +85,7 @@ see the [security section](#security).
Depending on your needs, you can set up GitLab Pages in 4 different ways.
The following options are listed from the easiest setup to the most
advanced one. The absolute minimum requirement is to set up the wildcard DNS
-since that is needed in all configurations.
+because that is needed in all configurations.
### Wildcard domains
@@ -96,7 +95,7 @@ since that is needed in all configurations.
URL scheme: `http://<namespace>.example.io/<project_slug>`
-This is the minimum setup that you can use Pages with. It is the base for all
+This setup is the minimum you can use Pages with. It is the base for all
other setups as described below. NGINX proxies all requests to the daemon.
The Pages daemon doesn't listen to the outside world.
@@ -150,8 +149,8 @@ The Pages daemon doesn't listen to the outside world.
You may use an `http` address, when running GitLab Pages and GitLab on the
same host. If you use `https` and use a self-signed certificate, be sure to
- make your custom CA available to GitLab Pages, for example by setting the
- `SSL_CERT_DIR` environment variable.
+ make your custom CA available to GitLab Pages. For example, you can do this
+ by setting the `SSL_CERT_DIR` environment variable.
1. Add the secret API key:
@@ -264,8 +263,8 @@ that without TLS certificates.
URL scheme: `http://<namespace>.example.io/<project_slug>` and `http://custom-domain.com`
-In that case, the pages daemon is running, NGINX still proxies requests to
-the daemon but the daemon is also able to receive requests from the outside
+In that case, the pages daemon is running. NGINX still proxies requests to
+the daemon, but the daemon is also able to receive requests from the outside
world. Custom domains are supported, but no TLS.
1. Install the Pages daemon:
@@ -296,10 +295,9 @@ world. Custom domains are supported, but no TLS.
external_http: 192.0.2.2:80
```
-1. Edit `/etc/default/gitlab` and set `gitlab_pages_enabled` to `true` in
- order to enable the pages daemon. In `gitlab_pages_options` the
- `-pages-domain` and `-listen-http` must match the `host` and `external_http`
- settings that you set above respectively:
+1. To enable the daemon, edit `/etc/default/gitlab` and set `gitlab_pages_enabled` to `true`.
+ In `gitlab_pages_options`, the value for `-pages-domain` must match the `host` and `-listen-http` must match
+ the `external_http`:
```ini
gitlab_pages_enabled=true
@@ -329,8 +327,8 @@ world. Custom domains are supported, but no TLS.
URL scheme: `https://<namespace>.example.io/<project_slug>` and `https://custom-domain.com`
-In that case, the pages daemon is running, NGINX still proxies requests to
-the daemon but the daemon is also able to receive requests from the outside
+In that case, the pages daemon is running. NGINX still proxies requests to
+the daemon, but the daemon is also able to receive requests from the outside
world. Custom domains and TLS are supported.
1. Install the Pages daemon:
@@ -416,8 +414,8 @@ server_name ~^.*\.pages\.example\.io$;
## Access control
-GitLab Pages access control can be configured per-project, and allows access to a Pages
-site to be controlled based on a user's membership to that project.
+GitLab Pages access control can be configured per project. Access to a Pages
+site can be controlled based on a user's membership to that project.
Access control works by registering the Pages daemon as an OAuth application
with GitLab. Whenever a request to access a private Pages site is made by an
diff --git a/doc/administration/pages/troubleshooting.md b/doc/administration/pages/troubleshooting.md
new file mode 100644
index 00000000000..6f00ae074f5
--- /dev/null
+++ b/doc/administration/pages/troubleshooting.md
@@ -0,0 +1,304 @@
+---
+stage: Plan
+group: Knowledge
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Troubleshooting GitLab Pages administration **(FREE SELF)**
+
+This page contains a list of issues you might encounter when administering GitLab Pages.
+
+## How to see GitLab Pages logs
+
+You can see Pages daemon logs by running:
+
+```shell
+sudo gitlab-ctl tail gitlab-pages
+```
+
+You can also find the log file in `/var/log/gitlab/gitlab-pages/current`.
+
+## `unsupported protocol scheme \"\""`
+
+If you see the following error:
+
+```plaintext
+{"error":"failed to connect to internal Pages API: Get \"/api/v4/internal/pages/status\": unsupported protocol scheme \"\"","level":"warning","msg":"attempted to connect to the API","time":"2021-06-23T20:03:30Z"}
+```
+
+It means you didn't set the HTTP(S) protocol scheme in the Pages server settings.
+To fix it:
+
+1. Edit `/etc/gitlab/gitlab.rb`:
+
+ ```ruby
+ gitlab_pages['gitlab_server'] = "https://<your_gitlab_server_public_host_and_port>"
+ gitlab_pages['internal_gitlab_server'] = "https://<your_gitlab_server_private_host_and_port>" # optional, gitlab_pages['gitlab_server'] is used as default
+ ```
+
+1. Reconfigure GitLab:
+
+ ```shell
+ sudo gitlab-ctl reconfigure
+ ```
+
+## 502 error when connecting to GitLab Pages proxy when server does not listen over IPv6
+
+In some cases, NGINX might default to using IPv6 to connect to the GitLab Pages
+service even when the server does not listen over IPv6. You can identify when
+this is happening if you see something similar to the log entry below in the
+`gitlab_pages_error.log`:
+
+```plaintext
+2020/02/24 16:32:05 [error] 112654#0: *4982804 connect() failed (111: Connection refused) while connecting to upstream, client: 123.123.123.123, server: ~^(?<group>.*)\.pages\.example\.com$, request: "GET /-/group/project/-/jobs/1234/artifacts/artifact.txt HTTP/1.1", upstream: "http://[::1]:8090//-/group/project/-/jobs/1234/artifacts/artifact.txt", host: "group.example.com"
+```
+
+To resolve this, set an explicit IP and port for the GitLab Pages `listen_proxy` setting
+to define the explicit address that the GitLab Pages daemon should listen on:
+
+```ruby
+gitlab_pages['listen_proxy'] = '127.0.0.1:8090'
+```
+
+## Intermittent 502 errors or after a few days
+
+If you run Pages on a system that uses `systemd` and
+[`tmpfiles.d`](https://www.freedesktop.org/software/systemd/man/tmpfiles.d.html),
+you may encounter intermittent 502 errors trying to serve Pages with an error similar to:
+
+```plaintext
+dial tcp: lookup gitlab.example.com on [::1]:53: dial udp [::1]:53: connect: no route to host"
+```
+
+GitLab Pages creates a [bind mount](https://man7.org/linux/man-pages/man8/mount.8.html)
+inside `/tmp/gitlab-pages-*` that includes files like `/etc/hosts`.
+However, `systemd` may clean the `/tmp/` directory on a regular basis so the DNS
+configuration may be lost.
+
+To stop `systemd` from cleaning the Pages related content:
+
+1. Tell `tmpfiles.d` to not remove the Pages `/tmp` directory:
+
+ ```shell
+ echo 'x /tmp/gitlab-pages-*' >> /etc/tmpfiles.d/gitlab-pages-jail.conf
+ ```
+
+1. Restart GitLab Pages:
+
+ ```shell
+ sudo gitlab-ctl restart gitlab-pages
+ ```
+
+## Unable to access GitLab Pages
+
+If you can't access your GitLab Pages (such as receiving `502 Bad Gateway` errors, or a login loop)
+and in your Pages log shows this error:
+
+```plaintext
+"error":"retrieval context done: context deadline exceeded","host":"root.docs-cit.otenet.gr","level":"error","msg":"could not fetch domain information from a source"
+```
+
+1. Add the following to `/etc/gitlab/gitlab.rb`:
+
+ ```ruby
+ gitlab_pages['internal_gitlab_server'] = 'http://localhost:8080'
+ ```
+
+1. Restart GitLab Pages:
+
+ ```shell
+ sudo gitlab-ctl restart gitlab-pages
+ ```
+
+## Failed to connect to the internal GitLab API
+
+If you see the following error:
+
+```plaintext
+ERRO[0010] Failed to connect to the internal GitLab API after 0.50s error="failed to connect to internal Pages API: HTTP status: 401"
+```
+
+If you are [Running GitLab Pages on a separate server](index.md#running-gitlab-pages-on-a-separate-server)
+you must copy the `/etc/gitlab/gitlab-secrets.json` file
+from the **GitLab server** to the **Pages server** after upgrading to GitLab 13.3,
+as described in that section.
+
+Other reasons may include network connectivity issues between your
+**GitLab server** and your **Pages server** such as firewall configurations or closed ports.
+For example, if there is a connection timeout:
+
+```plaintext
+error="failed to connect to internal Pages API: Get \"https://gitlab.example.com:3000/api/v4/internal/pages/status\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
+```
+
+## Pages cannot communicate with an instance of the GitLab API
+
+If you use the default value for `domain_config_source=auto` and run multiple instances of GitLab
+Pages, you may see intermittent 502 error responses while serving Pages content. You may also see
+the following warning in the Pages logs:
+
+```plaintext
+WARN[0010] Pages cannot communicate with an instance of the GitLab API. Please sync your gitlab-secrets.json file https://gitlab.com/gitlab-org/gitlab-pages/-/issues/535#workaround. error="pages endpoint unauthorized"
+```
+
+This can happen if your `gitlab-secrets.json` file is out of date between GitLab Rails and GitLab
+Pages. Follow steps 8-10 of [Running GitLab Pages on a separate server](index.md#running-gitlab-pages-on-a-separate-server),
+in all of your GitLab Pages instances.
+
+## Intermittent 502 errors when using an AWS Network Load Balancer and GitLab Pages
+
+Connections will time out when using a Network Load Balancer with client IP preservation enabled and [the request is looped back to the source server](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-troubleshooting.html#loopback-timeout).
+This can happen to GitLab instances with multiple servers
+running both the core GitLab application and GitLab Pages. This can also happen when a single
+container is running both the core GitLab application and GitLab Pages.
+
+AWS [recommends using an IP target type](https://aws.amazon.com/premiumsupport/knowledge-center/target-connection-fails-load-balancer/)
+to resolve this issue.
+
+Turning off [client IP preservation](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html#client-ip-preservation)
+may resolve this issue when the core GitLab application and GitLab Pages run on the same host or
+container.
+
+## 500 error with `securecookie: failed to generate random iv` and `Failed to save the session`
+
+This problem most likely results from an [out-dated operating system](../package_information/supported_os.md#os-versions-that-are-no-longer-supported).
+The [Pages daemon uses the `securecookie` library](https://gitlab.com/search?group_id=9970&project_id=734943&repository_ref=master&scope=blobs&search=securecookie&snippets=false) to get random strings via [`crypto/rand` in Go](https://pkg.go.dev/crypto/rand#pkg-variables).
+This requires the `getrandom` system call or `/dev/urandom` to be available on the host OS.
+Upgrading to an [officially supported operating system](https://about.gitlab.com/install/) is recommended.
+
+## The requested scope is invalid, malformed, or unknown
+
+This problem comes from the permissions of the GitLab Pages OAuth application. To fix it:
+
+1. On the top bar, select **Main menu > Admin**.
+1. On the left sidebar, select **Applications > GitLab Pages**.
+1. Edit the application.
+1. Under **Scopes**, ensure that the `api` scope is selected.
+1. Save your changes.
+
+When running a [separate Pages server](index.md#running-gitlab-pages-on-a-separate-server),
+this setting needs to be configured on the main GitLab server.
+
+## Workaround in case no wildcard DNS entry can be set
+
+If the wildcard DNS [prerequisite](index.md#prerequisites) can't be met, you can still use GitLab Pages in a limited fashion:
+
+1. [Move](../../user/project/settings/index.md#transfer-a-project-to-another-namespace)
+ all projects you need to use Pages with into a single group namespace, for example `pages`.
+1. Configure a [DNS entry](index.md#dns-configuration) without the `*.`-wildcard, for example `pages.example.io`.
+1. Configure `pages_external_url http://example.io/` in your `gitlab.rb` file.
+ Omit the group namespace here, because it automatically is prepended by GitLab.
+
+## Pages daemon fails with permission denied errors
+
+If `/tmp` is mounted with `noexec`, the Pages daemon fails to start with an error like:
+
+```plaintext
+{"error":"fork/exec /gitlab-pages: permission denied","level":"fatal","msg":"could not create pages daemon","time":"2021-02-02T21:54:34Z"}
+```
+
+In this case, change `TMPDIR` to a location that is not mounted with `noexec`. Add the following to
+`/etc/gitlab/gitlab.rb`:
+
+```ruby
+gitlab_pages['env'] = {'TMPDIR' => '<new_tmp_path>'}
+```
+
+Once added, reconfigure with `sudo gitlab-ctl reconfigure` and restart GitLab with
+`sudo gitlab-ctl restart`.
+
+## `The redirect URI included is not valid.` when using Pages Access Control
+
+You may see this error if `pages_external_url` was updated at some point of time. Verify the following:
+
+1. The **Callback URL**/Redirect URI in the GitLab Pages [System OAuth application](../../integration/oauth_provider.md#create-an-instance-wide-application)
+is using the protocol (HTTP or HTTPS) that `pages_external_url` is configured to use.
+1. The domain and path components of `Redirect URI` are valid: they should look like `projects.<pages_external_url>/auth`.
+
+## 500 error `cannot serve from disk`
+
+If you get a 500 response from Pages and encounter an error similar to:
+
+```plaintext
+ERRO[0145] cannot serve from disk error="gitlab: disk access is disabled via enable-disk=false" project_id=27 source_path="file:///shared/pages/@hashed/67/06/670671cd97404156226e507973f2ab8330d3022ca96e0c93bdbdb320c41adcaf/pages_deployments/14/artifacts.zip" source_type=zip
+```
+
+It means that GitLab Rails is telling GitLab Pages to serve content from a location on disk,
+however, GitLab Pages was configured to disable disk access.
+
+To enable disk access:
+
+1. Enable disk access for GitLab Pages in `/etc/gitlab/gitlab.rb`:
+
+ ```ruby
+ gitlab_pages['enable_disk'] = true
+ ```
+
+1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+
+## `httprange: new resource 403`
+
+If you see an error similar to:
+
+```plaintext
+{"error":"httprange: new resource 403: \"403 Forbidden\"","host":"root.pages.example.com","level":"error","msg":"vfs.Root","path":"/pages1/","time":"2021-06-10T08:45:19Z"}
+```
+
+And you run pages on the separate server syncing files via NFS, it may mean that
+the shared pages directory is mounted on a different path on the main GitLab server and the
+GitLab Pages server.
+
+In that case, it's highly recommended you to configure
+[object storage and migrate any existing pages data to it](index.md#using-object-storage).
+
+Alternatively, you can mount the GitLab Pages shared directory to the same path on
+both servers.
+
+## GitLab Pages doesn't work after upgrading to GitLab 14.0 or above
+
+GitLab 14.0 introduces a number of changes to GitLab Pages which may require manual intervention.
+
+1. Firstly [follow the migration guide](index.md#prepare-gitlab-pages-for-140).
+1. Try to upgrade to GitLab 14.3 or above. Some of the issues were fixed in GitLab 14.1, 14.2 and 14.3.
+1. If it doesn't work, see [GitLab Pages logs](#how-to-see-gitlab-pages-logs), and if you see any errors there then search them on this page.
+
+WARNING:
+In GitLab 14.0-14.2 you can temporarily enable legacy storage and configuration mechanisms.
+
+To do that:
+
+1. Describe the issue you're seeing in the [migration feedback issue](https://gitlab.com/gitlab-org/gitlab/-/issues/331699).
+
+1. Edit `/etc/gitlab/gitlab.rb`:
+
+ ```ruby
+ gitlab_pages['use_legacy_storage'] = true
+ ```
+
+1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+
+## GitLab Pages deploy job fails with error "is not a recognized provider"
+
+If the **pages** job succeeds but the **deploy** job gives the error "is not a recognized provider":
+
+![Pages Deploy Failure](img/pages_deploy_failure_v14_8.png)
+
+The error message `is not a recognized provider` could be coming from the `fog` gem that GitLab uses to connect to cloud providers for object storage.
+
+To fix that:
+
+1. Check your `gitlab.rb` file. If you have `gitlab_rails['pages_object_store_enabled']` enabled, but no bucket details have been configured, either:
+
+ - Configure object storage for your Pages deployments, following the [S3-compatible connection settings](index.md#s3-compatible-connection-settings) guide.
+ - Store your deployments locally, by commenting out that line.
+
+1. Save the changes you made to your `gitlab.rb` file, then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+
+## 404 error `The page you're looking for could not be found`
+
+If you get a `404 Page Not Found` response from GitLab Pages:
+
+1. Check `.gitlab-ci.yml` contains the job `pages:`.
+1. Check the current project's pipeline to confirm the job `pages:deploy` is being run.
+
+Without the `pages:deploy` job, the updates to your GitLab Pages site are never published.
diff --git a/doc/administration/postgresql/database_load_balancing.md b/doc/administration/postgresql/database_load_balancing.md
index 15129770888..d5cf93a135a 100644
--- a/doc/administration/postgresql/database_load_balancing.md
+++ b/doc/administration/postgresql/database_load_balancing.md
@@ -235,3 +235,8 @@ operation retries up to 3 times using an exponential back-off.
When using load balancing, you should be able to safely restart a database server
without it immediately leading to errors being presented to the users.
+
+### Development guide
+
+For detailed development guide on database load balancing,
+see [the development documentation](../../development/database/load_balancing.md).
diff --git a/doc/administration/postgresql/external.md b/doc/administration/postgresql/external.md
index 8605ee94255..8f664f9809e 100644
--- a/doc/administration/postgresql/external.md
+++ b/doc/administration/postgresql/external.md
@@ -17,7 +17,7 @@ If you use a cloud-managed service, or provide your own PostgreSQL instance:
1. Set up PostgreSQL according to the
[database requirements document](../../install/requirements.md#database).
-1. Set up a `gitlab` user with a password of your choice, create the `gitlabhq_production` database, and make the user an owner of the database. You can see an example of this setup in the [installation from source documentation](../../install/installation.md#6-database).
+1. Set up a `gitlab` user with a password of your choice, create the `gitlabhq_production` database, and make the user an owner of the database. You can see an example of this setup in the [installation from source documentation](../../install/installation.md#7-database).
1. If you are using a cloud-managed service, you may need to grant additional
roles to your `gitlab` user:
- Amazon RDS requires the [`rds_superuser`](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.html#Appendix.PostgreSQL.CommonDBATasks.Roles) role.
@@ -30,18 +30,18 @@ If you use a cloud-managed service, or provide your own PostgreSQL instance:
1. Configure the GitLab application servers with the appropriate connection details
for your external PostgreSQL service in your `/etc/gitlab/gitlab.rb` file:
- ```ruby
- # Disable the bundled Omnibus provided PostgreSQL
- postgresql['enable'] = false
+ ```ruby
+ # Disable the bundled Omnibus provided PostgreSQL
+ postgresql['enable'] = false
- # PostgreSQL connection details
- gitlab_rails['db_adapter'] = 'postgresql'
- gitlab_rails['db_encoding'] = 'unicode'
- gitlab_rails['db_host'] = '10.1.0.5' # IP/hostname of database server
- gitlab_rails['db_password'] = 'DB password'
- ```
+ # PostgreSQL connection details
+ gitlab_rails['db_adapter'] = 'postgresql'
+ gitlab_rails['db_encoding'] = 'unicode'
+ gitlab_rails['db_host'] = '10.1.0.5' # IP/hostname of database server
+ gitlab_rails['db_password'] = 'DB password'
+ ```
- For more information on GitLab multi-node setups, refer to the [reference architectures](../reference_architectures/index.md).
+ For more information on GitLab multi-node setups, refer to the [reference architectures](../reference_architectures/index.md).
1. Reconfigure for the changes to take effect:
diff --git a/doc/administration/postgresql/multiple_databases.md b/doc/administration/postgresql/multiple_databases.md
index 836736fb73f..857fd4fc9c5 100644
--- a/doc/administration/postgresql/multiple_databases.md
+++ b/doc/administration/postgresql/multiple_databases.md
@@ -1,6 +1,6 @@
---
stage: Data Stores
-group: Pods
+group: Tenant Scale
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
@@ -15,23 +15,83 @@ By default, GitLab uses a single application database, referred to as the `main`
To scale GitLab, you can configure GitLab to use multiple application databases.
-Due to [known issues](#known-issues), configuring GitLab with multiple databases is in [**Alpha**](../../policy/alpha-beta-support.md#alpha-features).
+Due to [known issues](#known-issues), configuring GitLab with multiple databases is an [Experiment](../../policy/alpha-beta-support.md#experiment).
+
+After you have set up multiple databases, GitLab uses a second application database for
+[CI/CD features](../../ci/index.md), referred to as the `ci` database.
+
+All tables have exactly the same structure in both the `main`, and `ci`
+databases. Some examples:
+
+- When multiple databases are configured, the `ci_pipelines` table exists in
+ both the `main` and `ci` databases, but GitLab reads and writes only to the
+ `ci_pipelines` table in the `ci` database.
+- Similarly, the `projects` table exists in
+ both the `main` and `ci` databases, but GitLab reads and writes only to the
+ `projects` table in the `main` database.
+- For some tables (such as `loose_foreign_keys_deleted_records`) GitLab reads and writes to both the `main` and `ci` databases. See the
+ [development documentation](../../development/database/multiple_databases.md#gitlab-schema)
## Known issues
-- Migrating data from the `main` database to the `ci` database is not supported or documented yet.
- Once data is migrated to the `ci` database, you cannot migrate it back.
-## Set up multiple databases
+## Migrate existing installations
-Use the following content to set up multiple databases with a new GitLab installation.
+To migrate existing data from the `main` database to the `ci` database, you can
+copy the database across.
-There is no documentation for existing GitLab installations yet.
+### Existing source installation
-After you have set up multiple databases, GitLab uses a second application database for
-[CI/CD features](../../ci/index.md), referred to as the `ci` database. For
-example, GitLab reads and writes to the `ci_pipelines` table in the `ci`
-database.
+1. Stop GitLab, except for PostgreSQL:
+
+ ```shell
+ sudo service gitlab stop
+ sudo service postgresql start
+ ```
+
+1. Dump the `main` database:
+
+ ```shell
+ sudo -u git pg_dump -f gitlabhq_production.sql gitlabhq_production
+ ```
+
+1. Create the `ci` database, and copy the data from the previous dump:
+
+ ```shell
+ sudo -u postgres psql -d template1 -c "CREATE DATABASE gitlabhq_production_ci OWNER git;"
+ sudo -u git psql -f gitlabhq_production.sql gitlabhq_production_ci
+ ```
+
+1. Configure GitLab to [use multiple databases](#set-up-multiple-databases).
+
+### Existing Omnibus installation
+
+1. Stop GitLab, except for PostgreSQL:
+
+ ```shell
+ sudo gitlab-ctl stop
+ sudo gitlab-ctl start postgresql
+ ```
+
+1. Dump the `main` database:
+
+ ```shell
+ sudo -u gitlab-psql /opt/gitlab/embedded/bin/pg_dump -h /var/opt/gitlab/postgresql -f gitlabhq_production.sql gitlabhq_production
+ ```
+
+1. Create the `ci` database, and copy the data from the previous dump:
+
+ ```shell
+ sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql -d template1 -c "CREATE DATABASE gitlabhq_production_ci OWNER gitlab;"
+ sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql -f gitlabhq_production.sql gitlabhq_production_ci
+ ```
+
+1. Configure GitLab to [use multiple databases](#set-up-multiple-databases).
+
+## Set up multiple databases
+
+To configure GitLab to use multiple application databases, follow the instructions below for your installation type.
WARNING:
You must stop GitLab before setting up multiple databases. This prevents
@@ -40,6 +100,9 @@ the other way around.
### Installations from source
+1. For existing installations,
+ [migrate the data](#migrate-existing-installations) first.
+
1. [Back up GitLab](../../raketasks/backup_restore.md)
in case of unforeseen issues.
@@ -70,7 +133,7 @@ the other way around.
1. Update the service files to set the `GITLAB_ALLOW_SEPARATE_CI_DATABASE`
environment variable to `true`.
-1. Create the `gitlabhq_production_ci` database:
+1. For new installations only. Create the `gitlabhq_production_ci` database:
```shell
sudo -u postgres psql -d template1 -c "CREATE DATABASE gitlabhq_production OWNER git;"
@@ -91,6 +154,9 @@ the other way around.
### Omnibus GitLab installations
+1. For existing installations,
+ [migrate the data](#migrate-existing-installations) first.
+
1. [Back up GitLab](../../raketasks/backup_restore.md)
in case of unforeseen issues.
@@ -116,7 +182,8 @@ the other way around.
sudo gitlab-ctl reconfigure
```
-1. Optional. Reconfiguring GitLab should create the `gitlabhq_production_ci`. If it did not, manually create the `gitlabhq_production_ci`:
+1. Optional, for new installations only. Reconfiguring GitLab should create the
+ `gitlabhq_production_ci` database if it does not exist. If the database is not created automatically, create it manually:
```shell
sudo gitlab-ctl start postgresql
diff --git a/doc/administration/postgresql/pgbouncer.md b/doc/administration/postgresql/pgbouncer.md
index 5dd0aad7162..59aac9141a4 100644
--- a/doc/administration/postgresql/pgbouncer.md
+++ b/doc/administration/postgresql/pgbouncer.md
@@ -174,10 +174,12 @@ ote_pid | tls
## Procedure for bypassing PgBouncer
+### Omnibus installations
+
Some database changes have to be done directly, and not through PgBouncer.
-Read more about the affected tasks: [database restores](../../raketasks/backup_restore.md#back-up-and-restore-for-installations-using-pgbouncer)
-and [GitLab upgrades](../../update/zero_downtime.md#postgresql).
+The main affected tasks are [database restores](../../raketasks/backup_restore.md#back-up-and-restore-for-installations-using-pgbouncer)
+and [GitLab upgrades with database migrations](../../update/zero_downtime.md#postgresql).
1. To find the primary node, run the following on a database node:
@@ -195,7 +197,7 @@ and [GitLab upgrades](../../update/zero_downtime.md#postgresql).
sudo gitlab-ctl reconfigure
```
-Once you've performed the tasks or procedure, switch back to using PgBouncer:
+After you've performed the tasks or procedure, switch back to using PgBouncer:
1. Change back `/etc/gitlab/gitlab.rb` to point to PgBouncer.
1. Run reconfigure:
@@ -204,6 +206,19 @@ Once you've performed the tasks or procedure, switch back to using PgBouncer:
sudo gitlab-ctl reconfigure
```
+### Helm chart installations
+
+High-availability deployments also need to bypass PgBouncer for the same reasons as Omnibus-based ones.
+For this type of installation:
+
+- Database backup and restore tasks are performed by the toolbox container.
+- Migration tasks are performed by the migrations container.
+
+You should override the PostgreSQL port on each subchart, so these tasks can execute and connect to PostgreSQL directly:
+
+- [Toolbox](https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/charts/gitlab/charts/toolbox/values.yaml#L40)
+- [Migrations](https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/charts/gitlab/charts/migrations/values.yaml#L46)
+
## Fine tuning
PgBouncer's default settings suit the majority of installations.
diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md
index 46890b0b2ca..8ac3ac40a75 100644
--- a/doc/administration/postgresql/replication_and_failover.md
+++ b/doc/administration/postgresql/replication_and_failover.md
@@ -1255,8 +1255,6 @@ To do the switch on **all** PgBouncer nodes:
```
1. Run `gitlab-ctl reconfigure`.
-1. You must also run `rm /var/opt/gitlab/consul/config.d/watcher_postgresql.json`.
- This is a [known issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7293).
#### Clean up
diff --git a/doc/administration/raketasks/github_import.md b/doc/administration/raketasks/github_import.md
index d089682f78e..1babafc902e 100644
--- a/doc/administration/raketasks/github_import.md
+++ b/doc/administration/raketasks/github_import.md
@@ -6,7 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# GitHub import Rake task **(FREE SELF)**
-> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/390690) in GitLab 15.9, Rake task no longer automatically creates namespaces or groups that don't exist.
+> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/390690) in GitLab 15.9, Rake task no longer automatically creates namespaces or groups that don't exist.
+> - Requirement for Maintainer role instead of Developer role introduced in GitLab 16.0 and backported to GitLab 15.11.1 and GitLab 15.10.5.
To retrieve and import GitHub repositories, you need a [GitHub personal access token](https://github.com/settings/tokens).
A username should be passed as the second argument to the Rake task,
@@ -19,10 +20,9 @@ before/after the brackets. Also, some shells (for example, Zsh) can interpret th
Prerequisite:
-- At least the Maintainer role on the destination group to import to. Using the Developer role for this purpose was
- [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/387891) in GitLab 15.8 and will be removed in GitLab 16.0.
+- At least the Maintainer role on the destination group to import to.
-## Caveats
+## Rate limit
If the GitHub [rate limit](https://docs.github.com/en/rest/rate-limit) is reached while
importing, the importing process waits (`sleep()`) until it can continue importing.
diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md
index 5c258d73fdb..06f7203f695 100644
--- a/doc/administration/raketasks/maintenance.md
+++ b/doc/administration/raketasks/maintenance.md
@@ -11,7 +11,7 @@ GitLab provides Rake tasks for general maintenance.
## Gather GitLab and system information
This command gathers information about your GitLab installation and the system it runs on.
-These may be useful when asking for help or reporting issues.
+These may be useful when asking for help or reporting issues. In a multi-node environment, run this command on nodes running GitLab Rails to avoid PostgreSQL socket errors.
**Omnibus Installation**
@@ -117,7 +117,7 @@ If you're running Geo, see also the [Geo Health check Rake task](../geo/replicat
You may also have a look at our troubleshooting guides for:
-- [GitLab](../index.md#troubleshooting)
+- [GitLab](../troubleshooting/index.md)
- [Omnibus GitLab](https://docs.gitlab.com/omnibus/index.html#troubleshooting)
Additionally you should also [verify database values can be decrypted using the current secrets](check.md#verify-database-values-can-be-decrypted-using-the-current-secrets).
@@ -386,7 +386,7 @@ You can also [enable reindexing as a regular cron job](https://docs.gitlab.com/o
Sometimes you may need to re-import the common metrics that power the Metrics dashboards.
-This could be as a result of [updating existing metrics](../../development/prometheus_metrics.md#update-existing-metrics), or as a [troubleshooting measure](../../operations/metrics/dashboards/index.md#troubleshooting).
+This could be as a result of [updating existing metrics](../../development/prometheus_metrics.md#update-existing-metrics).
To re-import the metrics you can run:
diff --git a/doc/administration/raketasks/project_import_export.md b/doc/administration/raketasks/project_import_export.md
index 4694af18af2..17a0eb46a30 100644
--- a/doc/administration/raketasks/project_import_export.md
+++ b/doc/administration/raketasks/project_import_export.md
@@ -4,61 +4,42 @@ group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
-# Project import/export administration **(FREE SELF)**
+# Project import and export Rake tasks **(FREE SELF)**
-GitLab provides Rake tasks relating to project import and export. For more information, see:
+GitLab provides Rake tasks for [project import and export](../../user/project/settings/import_export.md).
-- [Project import/export documentation](../../user/project/settings/import_export.md).
-- [Project import/export API](../../api/project_import_export.md).
-- [Developer documentation: project import/export](../../development/import_export.md)
+You can only import from a [compatible](../../user/project/settings/import_export.md#compatibility) GitLab instance.
-## Project import status
+## Import large projects
-You can query an import through the [Project import/export API](../../api/project_import_export.md#import-status).
-As described in the API documentation, the query may return an import error or exceptions.
+> The [Rake task](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/tasks/gitlab/import_export/import.rake) was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20724) in GitLab 12.6, replacing a GitLab.com Ruby script.
-## Import/export Rake tasks
+This script was introduced in GitLab 12.6 for importing large GitLab project exports.
-The GitLab import/export version can be checked by using the following command:
+As part of this script, we also disable direct upload. This avoids uploading a huge archive to GCS, which can cause idle transaction timeouts.
-```shell
-# Omnibus installations
-sudo gitlab-rake gitlab:import_export:version
+We can run this script from the terminal:
-# Installations from source
-bundle exec rake gitlab:import_export:version RAILS_ENV=production
-```
+Parameters:
-The current list of DB tables to export can be listed by using the following command:
+| Attribute | Type | Required | Description |
+| --------- | ---- | -------- | ----------- |
+| `username` | string | yes | User name |
+| `namespace_path` | string | yes | Namespace path |
+| `project_path` | string | yes | Project path |
+| `archive_path` | string | yes | Path to the exported project tarball you want to import |
```shell
-# Omnibus installations
-sudo gitlab-rake gitlab:import_export:data
-
-# Installations from source
-bundle exec rake gitlab:import_export:data RAILS_ENV=production
+bundle exec rake "gitlab:import_export:import[root, group/subgroup, testingprojectimport, /path/to/file.tar.gz]"
```
-Note the following:
-
-- Importing is only possible if the version of the import and export GitLab instances are
- [compatible](../../user/project/settings/import_export.md#compatibility).
-- The project import option must be enabled:
-
- 1. On the top bar, select **Main menu > Admin**.
- 1. On the left sidebar, select **Settings > General**.
- 1. Expand **Visibility and access controls**.
- 1. Under **Import sources**, check the "Project export enabled" option.
- 1. Select **Save changes**.
-
-- The exports are stored in a temporary directory and are deleted every
- 24 hours by a specific worker.
-
-### Import large projects using a Rake task
+If you're running Omnibus, run the following Rake task:
-If you have a larger project, consider using a Rake task as described in our [developer documentation](../../development/import_project.md#importing-via-a-rake-task).
+```shell
+gitlab-rake "gitlab:import_export:import[root, group/subgroup, testingprojectimport, /path/to/file.tar.gz]"
+```
-### Export using a Rake task
+## Export large projects
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25598) in GitLab 12.9.
@@ -88,3 +69,86 @@ IMPORT_DEBUG=true gitlab-rake "gitlab:import_export:import[root, group/subgroup,
# Export
EXPORT_DEBUG=true gitlab-rake "gitlab:import_export:export[root, group/subgroup, projectnametoexport, /tmp/export_file.tar.gz]"
```
+
+Check the common errors listed below, what they mean, and how to fix them.
+
+### `Exception: undefined method 'name' for nil:NilClass`
+
+The `username` is not valid.
+
+### `Exception: undefined method 'full_path' for nil:NilClass`
+
+The `namespace_path` does not exist.
+For example, one of the groups or subgroups is mistyped or missing,
+or you've specified the project name in the path.
+
+The task only creates the project.
+If you want to import it to a new group or subgroup, create it first.
+
+### `Exception: No such file or directory @ rb_sysopen - (filename)`
+
+The specified project export file in `archive_path` is missing.
+
+### `Exception: Permission denied @ rb_sysopen - (filename)`
+
+The specified project export file cannot be accessed by the `git` user.
+
+To fix the issue:
+
+1. Set the file owner to `git:git`.
+1. Change the file permissions to `0400`.
+1. Move the file to a public folder (for example `/tmp/`).
+
+### `Name can contain only letters, digits, emojis ...`
+
+```plaintext
+Name can contain only letters, digits, emojis, '_', '.', '+', dashes, or spaces. It must start with a letter,
+digit, emoji, or '_', and Path can contain only letters, digits, '_', '-', or '.'. It cannot start
+with '-', end in '.git', or end in '.atom'.
+```
+
+The project name specified in `project_path` is not valid for one of the specified reasons.
+
+Only put the project name in `project_path`. For example, if you provide a path of subgroups
+it fails with this error as `/` is not a valid character in a project name.
+
+### `Name has already been taken and Path has already been taken`
+
+A project with that name already exists.
+
+### `Exception: Error importing repository into (namespace) - No space left on device`
+
+The disk has insufficient space to complete the import.
+
+During import, the tarball is cached in your configured `shared_path` directory. Verify the
+disk has enough free space to accommodate both the cached tarball and the unpacked
+project files on disk.
+
+### Import is successful, but with a `Total number of not imported relations: XX` message, and issues are not created during the import
+
+If you receive a `Total number of not imported relations: XX` message, and issues
+aren't created during the import, check [exceptions_json.log](../logs/index.md#exceptions_jsonlog).
+You might see an error like `N is out of range for ActiveModel::Type::Integer with limit 4 bytes`,
+where `N` is the integer exceeding the 4-byte integer limit. If that's the case, you
+are likely hitting the issue with rebalancing of `relative_position` field of the issues.
+
+```ruby
+# Check the current maximum value of relative_position
+Issue.where(project_id: Project.find(ID).root_namespace.all_projects).maximum(:relative_position)
+
+# Run the rebalancing process and check if the maximum value of relative_position has changed
+Issues::RelativePositionRebalancingService.new(Project.find(ID).root_namespace.all_projects).execute
+Issue.where(project_id: Project.find(ID).root_namespace.all_projects).maximum(:relative_position)
+```
+
+Repeat the import attempt and check if the issues are imported successfully.
+
+### Gitaly calls error when importing
+
+If you're attempting to import a large project into a development environment, Gitaly might throw an error about too many calls or invocations. For example:
+
+```plaintext
+Error importing repository into qa-perf-testing/gitlabhq - GitalyClient#call called 31 times from single request. Potential n+1?
+```
+
+This error is due to a [n+1 calls limit for development setups](../../development/gitaly.md#toomanyinvocationserror-errors). To resolve this error, set `GITALY_DISABLE_REQUEST_LIMITS=1` as an environment variable. Then restart your development environment and import again.
diff --git a/doc/administration/raketasks/service_desk_email.md b/doc/administration/raketasks/service_desk_email.md
index 10de379b1cd..1cbdec35171 100644
--- a/doc/administration/raketasks/service_desk_email.md
+++ b/doc/administration/raketasks/service_desk_email.md
@@ -12,7 +12,7 @@ The following are Service Desk email-related Rake tasks.
## Secrets
-GitLab can use [Service Desk email](../../user/project/service_desk.md#configuring-a-custom-mailbox) secrets read from an encrypted file instead of storing them in plaintext in the file system. The following Rake tasks are provided for updating the contents of the encrypted file.
+GitLab can use [Service Desk email](../../user/project/service_desk.md#configure-a-custom-mailbox) secrets read from an encrypted file instead of storing them in plaintext in the file system. The following Rake tasks are provided for updating the contents of the encrypted file.
### Show secret
diff --git a/doc/administration/raketasks/uploads/sanitize.md b/doc/administration/raketasks/uploads/sanitize.md
index 831abee9739..567a20a37f3 100644
--- a/doc/administration/raketasks/uploads/sanitize.md
+++ b/doc/administration/raketasks/uploads/sanitize.md
@@ -11,7 +11,7 @@ In GitLab 11.9 and later, EXIF data is automatically stripped from JPG or TIFF i
EXIF data may contain sensitive information (for example, GPS location), so you
can remove EXIF data from existing images that were uploaded to an earlier version of GitLab.
-## Requirements
+## Prerequisite
To run this Rake task, you need `exiftool` installed on your system. If you installed GitLab:
diff --git a/doc/administration/read_only_gitlab.md b/doc/administration/read_only_gitlab.md
index 3718741e2e9..3842cf0846b 100644
--- a/doc/administration/read_only_gitlab.md
+++ b/doc/administration/read_only_gitlab.md
@@ -70,28 +70,28 @@ the database is read-only:
in case things don't go as expected.
1. Enter PostgreSQL on the console as an administrator user:
- ```shell
- sudo \
- -u gitlab-psql /opt/gitlab/embedded/bin/psql \
- -h /var/opt/gitlab/postgresql gitlabhq_production
- ```
+ ```shell
+ sudo \
+ -u gitlab-psql /opt/gitlab/embedded/bin/psql \
+ -h /var/opt/gitlab/postgresql gitlabhq_production
+ ```
1. Create the `gitlab_read_only` user. The password is set to `mypassword`,
change that to your liking:
- ```sql
- -- NOTE: Use the password defined earlier
- CREATE USER gitlab_read_only WITH password 'mypassword';
- GRANT CONNECT ON DATABASE gitlabhq_production to gitlab_read_only;
- GRANT USAGE ON SCHEMA public TO gitlab_read_only;
- GRANT SELECT ON ALL TABLES IN SCHEMA public TO gitlab_read_only;
- GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO gitlab_read_only;
-
- -- Tables created by "gitlab" should be made read-only for "gitlab_read_only"
- -- automatically.
- ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON TABLES TO gitlab_read_only;
- ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON SEQUENCES TO gitlab_read_only;
- ```
+ ```sql
+ -- NOTE: Use the password defined earlier
+ CREATE USER gitlab_read_only WITH password 'mypassword';
+ GRANT CONNECT ON DATABASE gitlabhq_production to gitlab_read_only;
+ GRANT USAGE ON SCHEMA public TO gitlab_read_only;
+ GRANT SELECT ON ALL TABLES IN SCHEMA public TO gitlab_read_only;
+ GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO gitlab_read_only;
+
+ -- Tables created by "gitlab" should be made read-only for "gitlab_read_only"
+ -- automatically.
+ ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON TABLES TO gitlab_read_only;
+ ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON SEQUENCES TO gitlab_read_only;
+ ```
1. Get the hashed password of the `gitlab_read_only` user and copy the result:
@@ -101,10 +101,10 @@ the database is read-only:
1. Edit `/etc/gitlab/gitlab.rb` and add the password from the previous step:
- ```ruby
- postgresql['sql_user_password'] = 'a2e20f823772650f039284619ab6f239'
- postgresql['sql_user'] = "gitlab_read_only"
- ```
+ ```ruby
+ postgresql['sql_user_password'] = 'a2e20f823772650f039284619ab6f239'
+ postgresql['sql_user'] = "gitlab_read_only"
+ ```
1. Reconfigure GitLab and restart PostgreSQL:
diff --git a/doc/administration/redis/replication_and_failover.md b/doc/administration/redis/replication_and_failover.md
index 2e56884e309..a37794ec44b 100644
--- a/doc/administration/redis/replication_and_failover.md
+++ b/doc/administration/redis/replication_and_failover.md
@@ -766,6 +766,49 @@ redis['master'] = false
You can find the relevant attributes defined in [`gitlab_rails.rb`](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/libraries/gitlab_rails.rb).
+### Control startup behavior
+
+> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/6646) in GitLab 15.10.
+
+To prevent the bundled Redis service from starting at boot or restarting after changing its configuration:
+
+1. Edit `/etc/gitlab/gitlab.rb`:
+
+ ```ruby
+ redis['start_down'] = true
+ ```
+
+1. Reconfigure GitLab:
+
+ ```shell
+ sudo gitlab-ctl reconfigure
+ ```
+
+If you need to test a new replica node, you may set `start_down` to
+`true` and manually start the node. After the new replica node is confirmed
+working in the Redis cluster, set `start_down` to `false` and reconfigure GitLab
+to ensure the node starts and restarts as expected during operation.
+
+### Control replica configuration
+
+> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/6646) in GitLab 15.10.
+
+To prevent the `replicaof` line from rendering in the Redis configuration file:
+
+1. Edit `/etc/gitlab/gitlab.rb`:
+
+ ```ruby
+ redis['set_replicaof'] = false
+ ```
+
+1. Reconfigure GitLab:
+
+ ```shell
+ sudo gitlab-ctl reconfigure
+ ```
+
+This setting can be used to prevent replication of a Redis node independently of other Redis settings.
+
## Troubleshooting
See the [Redis troubleshooting guide](troubleshooting.md).
diff --git a/doc/administration/redis/replication_and_failover_external.md b/doc/administration/redis/replication_and_failover_external.md
index 23c9ce33c2d..14378b478da 100644
--- a/doc/administration/redis/replication_and_failover_external.md
+++ b/doc/administration/redis/replication_and_failover_external.md
@@ -32,15 +32,15 @@ Note the Redis node's IP address or hostname, port, and password (if required).
1. Configure the GitLab application servers with the appropriate connection details
for your external Redis service in your `/etc/gitlab/gitlab.rb` file:
- ```ruby
- redis['enable'] = false
+ ```ruby
+ redis['enable'] = false
- gitlab_rails['redis_host'] = 'redis.example.com'
- gitlab_rails['redis_port'] = 6379
+ gitlab_rails['redis_host'] = 'redis.example.com'
+ gitlab_rails['redis_port'] = 6379
- # Required if Redis authentication is configured on the Redis node
- gitlab_rails['redis_password'] = 'Redis Password'
- ```
+ # Required if Redis authentication is configured on the Redis node
+ gitlab_rails['redis_password'] = 'Redis Password'
+ ```
1. Reconfigure for the changes to take effect:
@@ -87,7 +87,7 @@ valuable information for the general setup.
Assuming that the Redis primary instance IP is `10.0.0.1`:
-1. [Install Redis](../../install/installation.md#7-redis).
+1. [Install Redis](../../install/installation.md#8-redis).
1. Edit `/etc/redis/redis.conf`:
```conf
@@ -113,7 +113,7 @@ Assuming that the Redis primary instance IP is `10.0.0.1`:
Assuming that the Redis replica instance IP is `10.0.0.2`:
-1. [Install Redis](../../install/installation.md#7-redis).
+1. [Install Redis](../../install/installation.md#8-redis).
1. Edit `/etc/redis/redis.conf`:
```conf
@@ -250,8 +250,8 @@ unauthorized access from other machines, and block traffic from the
outside ([Internet](https://gitlab.com/gitlab-org/gitlab-foss/uploads/c4cc8cd353604bd80315f9384035ff9e/The_Internet_IT_Crowd.png)).
For this example, **Sentinel 1** is configured in the same machine as the
-**Redis Primary**, **Sentinel 2** and **Sentinel 3** in the same machines as the
-**Replica 1** and **Replica 2** respectively.
+**Redis Primary**, **Sentinel 2** in the same machine as **Replica 1**, and
+**Sentinel 3** in the same machine as **Replica 2**.
Here is a list and description of each **machine** and the assigned **IP**:
diff --git a/doc/administration/redis/troubleshooting.md b/doc/administration/redis/troubleshooting.md
index 29407f65efd..947db2c1d4e 100644
--- a/doc/administration/redis/troubleshooting.md
+++ b/doc/administration/redis/troubleshooting.md
@@ -113,15 +113,19 @@ To make sure your configuration is correct:
redis.info
```
- Keep this screen open and try to simulate a failover below.
+ Keep this screen open, and proceed to trigger a failover as described below.
-1. To simulate a failover on primary Redis, SSH into the Redis server and run:
+1. To trigger a failover on the primary Redis, SSH into the Redis server and run:
```shell
# port must match your primary redis port, and the sleep time must be a few seconds bigger than defined one
redis-cli -h localhost -p 6379 DEBUG sleep 20
```
+ WARNING:
+ This action affects services, and takes the instance down for up to 20 seconds. If successful,
+ it should recover after that.
+
1. Then back in the Rails console from the first step, run:
```ruby
diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md
index 529621813aa..ba4c5fd9046 100644
--- a/doc/administration/reference_architectures/10k_users.md
+++ b/doc/administration/reference_architectures/10k_users.md
@@ -31,8 +31,8 @@ full list of reference architectures, see
| Gitaly<sup>5 6</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` |
| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` |
| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` |
-| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` |
-| GitLab Rails | 3 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` |
+| Sidekiq<sup>7</sup> | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` |
+| GitLab Rails<sup>7</sup> | 3 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` |
| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` |
| Object storage<sup>4</sup> | - | - | - | - |
@@ -53,6 +53,8 @@ full list of reference architectures, see
6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large
repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to
[Large repositories](index.md#large-repositories) for more information.
+7. Can be placed in Auto Scaling Groups (ASGs) as the component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes).
+ However, for GitLab Rails certain processes like [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md) should be run on only one node.
<!-- markdownlint-enable MD029 -->
NOTE:
@@ -170,7 +172,7 @@ To set up GitLab and its components to accommodate up to 10,000 users:
environment.
1. [Configure the object storage](#configure-the-object-storage)
used for shared data objects.
-1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster,
+1. [Configure advanced search](#configure-advanced-search) (optional) for faster,
more advanced code search across your entire GitLab instance.
The servers start on the same 10.6.0.0/24 private network range, and can
@@ -309,7 +311,7 @@ Configure your load balancer to pass connections on port 443 as `TCP` rather
than `HTTP(S)` protocol. This will pass the connection to the application node's
NGINX service untouched. NGINX will have the SSL certificate and listen on port 443.
-See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html)
+See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html)
for details on managing SSL certificates and configuring NGINX.
#### Load balancer terminates SSL without backend SSL
@@ -320,7 +322,7 @@ terminating SSL.
Since communication between the load balancer and GitLab will not be secure,
there is some additional configuration needed. See the
-[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination)
+[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination)
for details.
#### Load balancer terminates SSL with backend SSL
@@ -333,7 +335,7 @@ Traffic will also be secure between the load balancers and NGINX in this
scenario. There is no need to add configuration for proxied SSL since the
connection will be secure all the way. However, configuration will need to be
added to GitLab to configure SSL certificates. See
-the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html)
+the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html)
for details on managing SSL certificates and configuring NGINX.
<div align="right">
@@ -417,6 +419,11 @@ spread connections equally in practice.
## Configure Consul
+Next, we set up the Consul servers.
+
+NOTE:
+Consul must be deployed in an odd number of 3 nodes or more. This is to ensure the nodes can take votes as part of a quorum.
+
The following IPs will be used as an example:
- `10.6.0.11`: Consul 1
@@ -801,6 +808,9 @@ Using [Redis](https://redis.io/) in scalable environment is possible using a **P
topology with a [Redis Sentinel](https://redis.io/docs/manual/sentinel/) service to watch and automatically
start the failover procedure.
+NOTE:
+Redis clusters must each be deployed in an odd number of 3 nodes or more. This is to ensure Redis Sentinel can take votes as part of a quorum. This does not apply when configuring Redis externally, such as a cloud provider service.
+
Redis requires authentication if used with Sentinel. See
[Redis Security](https://redis.io/docs/manual/security/) documentation for more
information. We recommend using a combination of a Redis password and tight
@@ -834,9 +844,9 @@ to be used with GitLab. The following IPs will be used as an example:
Managed Redis from cloud providers (such as AWS ElastiCache) will work. If these
services support high availability, be sure it _isn't_ of the Redis Cluster type.
-Redis version 5.0 or higher is required, which is included with Omnibus GitLab
-packages starting with GitLab 13.0. Older Redis versions don't support an
-optional count argument to SPOP, which is required for [Merge Trains](../../ci/pipelines/merge_trains.md).
+
+Because Omnibus GitLab packages ship with Redis 6.0 or later, Redis 6.0 or later is required. Older Redis versions have reached end-of-life.
+
Note the Redis node's IP address or hostname, port, and password (if required).
These will be necessary later when configuring the [GitLab application servers](#configure-gitlab-rails).
@@ -1189,7 +1199,7 @@ For more advanced setups refer to the [standalone Gitaly Cluster documentation](
Praefect, the routing and transaction manager for Gitaly Cluster, requires its own database server to store data on Gitaly Cluster status.
If you want to have a highly available setup, Praefect requires a third-party PostgreSQL database.
-A built-in solution is being [worked on](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919).
+A built-in solution is being [worked on](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7292).
#### Praefect non-HA PostgreSQL standalone using Omnibus GitLab
@@ -1336,6 +1346,9 @@ This is how this would work with a Omnibus GitLab PostgreSQL setup:
Praefect is the router and transaction manager for Gitaly Cluster and all connections to Gitaly go through
it. This section details how to configure it.
+NOTE:
+Consul must be deployed in an odd number of 3 nodes or more. This is to ensure the nodes can take votes as part of a quorum.
+
Praefect requires several secret tokens to secure communications across the Cluster:
- `<praefect_external_token>`: Used for repositories hosted on your Gitaly cluster and can only be accessed by Gitaly clients that carry this token.
@@ -1389,7 +1402,6 @@ Updates to example must be made at:
# Praefect Configuration
praefect['enable'] = true
- praefect['listen_addr'] = '0.0.0.0:2305'
# Prevent database migrations from running on upgrade automatically
praefect['auto_migrate'] = false
@@ -1404,45 +1416,63 @@ Updates to example must be made at:
# Please set the real values as explained in Required Information section
#
- # Praefect External Token
- # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster
- praefect['auth_token'] = '<praefect_external_token>'
-
- # Praefect Database Settings
- praefect['database_host'] = '10.6.0.141'
- praefect['database_port'] = 5432
- # `no_proxy` settings must always be a direct connection for caching
- praefect['database_direct_host'] = '10.6.0.141'
- praefect['database_direct_port'] = 5432
- praefect['database_dbname'] = 'praefect_production'
- praefect['database_user'] = 'praefect'
- praefect['database_password'] = '<praefect_postgresql_password>'
-
- # Praefect Virtual Storage config
- # Name of storage hash must match storage name in git_data_dirs on GitLab
- # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1')
- praefect['virtual_storages'] = {
- 'default' => {
- 'nodes' => {
- 'gitaly-1' => {
- 'address' => 'tcp://10.6.0.91:8075',
- 'token' => '<praefect_internal_token>'
- },
- 'gitaly-2' => {
- 'address' => 'tcp://10.6.0.92:8075',
- 'token' => '<praefect_internal_token>'
- },
- 'gitaly-3' => {
- 'address' => 'tcp://10.6.0.93:8075',
- 'token' => '<praefect_internal_token>'
+ praefect['configuration'] = {
+ # ...
+ listen_addr: '0.0.0.0:2305',
+ auth: {
+ # ...
+ #
+ # Praefect External Token
+ # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster
+ token: '<praefect_external_token>',
+ },
+ # Praefect Database Settings
+ database: {
+ # ...
+ host: '10.6.0.141',
+ port: 5432,
+ # `no_proxy` settings must always be a direct connection for caching
+ session_pooled: {
+ # ...
+ host: '10.6.0.141',
+ port: 5432,
+ dbname: 'praefect_production',
+ user: 'praefect',
+ password: '<praefect_postgresql_password>',
+ },
+ },
+ # Praefect Virtual Storage config
+ # Name of storage hash must match storage name in git_data_dirs on GitLab
+ # server ('praefect') and in gitaly['configuration'][:storage] on Gitaly nodes ('gitaly-1')
+ virtual_storage: [
+ {
+ # ...
+ name: 'default',
+ node: [
+ {
+ storage: 'gitaly-1',
+ address: 'tcp://10.6.0.91:8075',
+ token: '<praefect_internal_token>'
+ },
+ {
+ storage: 'gitaly-2',
+ address: 'tcp://10.6.0.92:8075',
+ token: '<praefect_internal_token>'
+ },
+ {
+ storage: 'gitaly-3',
+ address: 'tcp://10.6.0.93:8075',
+ token: '<praefect_internal_token>'
+ },
+ ],
},
- }
- }
+ ],
+ # Set the network address Praefect will listen on for monitoring
+ prometheus_listen_addr: '0.0.0.0:9652',
}
- # Set the network addresses that the exporters will listen on for monitoring
+ # Set the network address the node exporter will listen on for monitoring
node_exporter['listen_address'] = '0.0.0.0:9100'
- praefect['prometheus_listen_addr'] = '0.0.0.0:9652'
## The IPs of the Consul server nodes
## You can also use FQDNs and intermix them with IPs
@@ -1493,9 +1523,7 @@ Due to Gitaly having notable input and output requirements, we strongly
recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs
should have a throughput of at least 8,000
input/output operations per second (IOPS) for read operations and 2,000 IOPS for
-write operations. These IOPS values are initial recommendations, and may be
-adjusted to greater or lesser values depending on the scale of your
-environment's workload. If you're running the environment on a Cloud provider,
+write operations. If you're running the environment on a Cloud provider,
refer to their documentation about how to configure IOPS correctly.
Gitaly servers must not be exposed to the public internet, as Gitaly's network
@@ -1505,7 +1533,7 @@ to restrict access to the Gitaly server. Another option is to
For configuring Gitaly you should note the following:
-- `git_data_dirs` should be configured to reflect the storage path for the specific Gitaly node
+- `gitaly['configuration'][:storage]` should be configured to reflect the storage path for the specific Gitaly node
- `auth_token` should be the same as `praefect_internal_token`
The following IPs will be used as an example:
@@ -1554,20 +1582,6 @@ Updates to example must be made at:
# Gitaly
gitaly['enable'] = true
- # Make Gitaly accept connections on all network interfaces. You must use
- # firewalls to restrict access to this address/port.
- # Comment out following line if you only want to support TLS connections
- gitaly['listen_addr'] = "0.0.0.0:8075"
-
- # Gitaly Auth Token
- # Should be the same as praefect_internal_token
- gitaly['auth_token'] = '<praefect_internal_token>'
-
- # Gitaly Pack-objects cache
- # Recommended to be enabled for improved performance but can notably increase disk I/O
- # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info
- gitaly['pack_objects_cache_enabled'] = true
-
# Configure the Consul agent
consul['enable'] = true
## Enable service discovery for Prometheus
@@ -1582,9 +1596,29 @@ Updates to example must be made at:
retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13),
}
- # Set the network addresses that the exporters will listen on for monitoring
+ # Set the network address that the node exporter will listen on for monitoring
node_exporter['listen_address'] = '0.0.0.0:9100'
- gitaly['prometheus_listen_addr'] = '0.0.0.0:9236'
+
+ gitaly['configuration'] = {
+ # Make Gitaly accept connections on all network interfaces. You must use
+ # firewalls to restrict access to this address/port.
+ # Comment out following line if you only want to support TLS connections
+ listen_addr: '0.0.0.0:8075',
+ # Set the network address that Gitaly will listen on for monitoring
+ prometheus_listen_addr: '0.0.0.0:9236',
+ auth: {
+ # Gitaly Auth Token
+ # Should be the same as praefect_internal_token
+ token: '<praefect_internal_token>',
+ },
+ pack_objects_cache: {
+ # Gitaly Pack-objects cache
+ # Recommended to be enabled for improved performance but can notably increase disk I/O
+ # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info
+ enabled: true,
+ },
+ }
+
#
# END user configuration
```
@@ -1593,31 +1627,43 @@ Updates to example must be made at:
- On Gitaly node 1:
```ruby
- git_data_dirs({
- "gitaly-1" => {
- "path" => "/var/opt/gitlab/git-data"
- }
- })
+ gitaly['configuration'] = {
+ # ...
+ storage: [
+ {
+ name: 'gitaly-1',
+ path: '/var/opt/gitlab/git-data',
+ },
+ ],
+ }
```
- On Gitaly node 2:
```ruby
- git_data_dirs({
- "gitaly-2" => {
- "path" => "/var/opt/gitlab/git-data"
- }
- })
+ gitaly['configuration'] = {
+ # ...
+ storage: [
+ {
+ name: 'gitaly-2',
+ path: '/var/opt/gitlab/git-data',
+ },
+ ],
+ }
```
- On Gitaly node 3:
```ruby
- git_data_dirs({
- "gitaly-3" => {
- "path" => "/var/opt/gitlab/git-data"
- }
- })
+ gitaly['configuration'] = {
+ # ...
+ storage: [
+ {
+ name: 'gitaly-3',
+ path: '/var/opt/gitlab/git-data',
+ },
+ ],
+ }
```
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
@@ -1637,7 +1683,7 @@ for secure connections, you must:
Additionally the certificate, or its certificate authority, must be installed on all Gitaly servers
and on all Praefect clients that communicate with it following the procedure described in
-[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) (and repeated below).
+[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl/index.html#install-custom-public-certificates) (and repeated below).
Note the following:
@@ -1646,7 +1692,7 @@ Note the following:
- You can configure Praefect servers with both an unencrypted listening address
`listen_addr` and an encrypted listening address `tls_listen_addr` at the same time.
This allows you to do a gradual transition from unencrypted to encrypted traffic, if
- necessary. To disable the unencrypted listener, set `praefect['listen_addr'] = nil`.
+ necessary. To disable the unencrypted listener, set `praefect['configuration'][:listen_addr] = nil`.
- The Internal Load Balancer will also access to the certificates and need to be configured
to allow for TLS pass-through.
Refer to the load balancers documentation on how to configure this.
@@ -1668,9 +1714,15 @@ To configure Praefect with TLS:
1. Edit `/etc/gitlab/gitlab.rb` and add:
```ruby
- praefect['tls_listen_addr'] = "0.0.0.0:3305"
- praefect['certificate_path'] = "/etc/gitlab/ssl/cert.pem"
- praefect['key_path'] = "/etc/gitlab/ssl/key.pem"
+ praefect['configuration'] = {
+ # ...
+ tls_listen_addr: '0.0.0.0:3305',
+ tls: {
+ # ...
+ certificate_path: '/etc/gitlab/ssl/cert.pem',
+ key_path: '/etc/gitlab/ssl/key.pem',
+ },
+ }
```
1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure).
@@ -1751,7 +1803,7 @@ Updates to example must be made at:
# Redis
## Redis connection details
- ## First cluster that will host the cache
+ ## First cluster that will host the cache data
gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache'
gitlab_rails['redis_cache_sentinels'] = [
@@ -1760,22 +1812,11 @@ Updates to example must be made at:
{host: '10.6.0.53', port: 26379},
]
- ## Second cluster that will host the persistent queues, shared state, and actioncable
- gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
- gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
- gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
+ ## Second cluster that hosts all other persistent data
+ redis['master_name'] = 'gitlab-redis-persistent'
+ redis['master_password'] = '<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>'
- gitlab_rails['redis_queues_sentinels'] = [
- {host: '10.6.0.61', port: 26379},
- {host: '10.6.0.62', port: 26379},
- {host: '10.6.0.63', port: 26379},
- ]
- gitlab_rails['redis_shared_state_sentinels'] = [
- {host: '10.6.0.61', port: 26379},
- {host: '10.6.0.62', port: 26379},
- {host: '10.6.0.63', port: 26379},
- ]
- gitlab_rails['redis_actioncable_sentinels'] = [
+ gitlab_rails['redis_sentinels'] = [
{host: '10.6.0.61', port: 26379},
{host: '10.6.0.62', port: 26379},
{host: '10.6.0.63', port: 26379},
@@ -1936,7 +1977,7 @@ On each node perform the following:
gitlab_rails['auto_migrate'] = false
## Redis connection details
- ## First cluster that will host the cache
+ ## First cluster that will host the cache data
gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache'
gitlab_rails['redis_cache_sentinels'] = [
@@ -1945,22 +1986,11 @@ On each node perform the following:
{host: '10.6.0.53', port: 26379},
]
- ## Second cluster that will host the persistent queues, shared state, and actioncable
- gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
- gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
- gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
+ ## Second cluster that hosts all other persistent data
+ redis['master_name'] = 'gitlab-redis-persistent'
+ redis['master_password'] = '<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>'
- gitlab_rails['redis_queues_sentinels'] = [
- {host: '10.6.0.61', port: 26379},
- {host: '10.6.0.62', port: 26379},
- {host: '10.6.0.63', port: 26379},
- ]
- gitlab_rails['redis_shared_state_sentinels'] = [
- {host: '10.6.0.61', port: 26379},
- {host: '10.6.0.62', port: 26379},
- {host: '10.6.0.63', port: 26379},
- ]
- gitlab_rails['redis_actioncable_sentinels'] = [
+ gitlab_rails['redis_sentinels'] = [
{host: '10.6.0.61', port: 26379},
{host: '10.6.0.62', port: 26379},
{host: '10.6.0.63', port: 26379},
@@ -2058,7 +2088,7 @@ On each node perform the following:
When you specify `https` in the `external_url`, as in the previous example,
GitLab expects that the SSL certificates are in `/etc/gitlab/ssl/`. If the
certificates aren't present, NGINX will fail to start. For more information, see
-the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html).
+the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html).
### GitLab Rails post-configuration
@@ -2070,11 +2100,9 @@ the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html).
sudo gitlab-rake gitlab:db:configure
```
- If you encounter a `rake aborted!` error message stating that PgBouncer is
- failing to connect to PostgreSQL, it may be that your PgBouncer node's IP
- address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb`
- on your database nodes. Before proceeding, see
- [PgBouncer error `ERROR: pgbouncer cannot connect to server`](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server).
+ Note that this requires the Rails node to be configured to connect to the primary database
+ directly, [bypassing PgBouncer](../postgresql/pgbouncer.md#procedure-for-bypassing-pgbouncer).
+ After migrations have completed, you must configure the node to pass through PgBouncer again.
1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md).
@@ -2170,15 +2198,15 @@ and scalable.
There are two ways of specifying object storage configuration in GitLab:
-- [Consolidated form](../object_storage.md#consolidated-object-storage-configuration): A single credential is
+- [Consolidated form](../object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form): A single credential is
shared by all supported object types.
-- [Storage-specific form](../object_storage.md#storage-specific-configuration): Every object defines its
- own object storage [connection and configuration](../object_storage.md#connection-settings).
+- [Storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form): Every object defines its
+ own object storage [connection and configuration](../object_storage.md#configure-the-connection-settings).
The consolidated form is used in the following examples when available.
NOTE:
-When using the [storage-specific form](../object_storage.md#storage-specific-configuration)
+When using the [storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form)
in GitLab 14.x and earlier, you should enable [direct upload mode](../../development/uploads/index.md#direct-upload).
The previous [background upload](../../development/uploads/index.md#direct-upload) mode,
which was deprecated in 14.9, requires shared storage such as NFS.
@@ -2200,9 +2228,9 @@ GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily
While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs.
-## Configure Advanced Search
+## Configure advanced search
-You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md)
+You can leverage Elasticsearch and [enable advanced search](../../integration/advanced_search/elasticsearch.md)
for faster, more advanced code search across your entire GitLab instance.
Elasticsearch cluster design and requirements are dependent on your specific
diff --git a/doc/administration/reference_architectures/1k_users.md b/doc/administration/reference_architectures/1k_users.md
index d3aa6fef51e..adcf8eeb4c6 100644
--- a/doc/administration/reference_architectures/1k_users.md
+++ b/doc/administration/reference_architectures/1k_users.md
@@ -78,9 +78,9 @@ You can also optionally configure GitLab to use an [external PostgreSQL service]
or an [external object storage service](../object_storage.md) for added
performance and reliability at an increased complexity cost.
-## Configure Advanced Search **(PREMIUM SELF)**
+## Configure advanced search **(PREMIUM SELF)**
-You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md)
+You can leverage Elasticsearch and [enable advanced search](../../integration/advanced_search/elasticsearch.md)
for faster, more advanced code search across your entire GitLab instance.
Elasticsearch cluster design and requirements are dependent on your specific
diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md
index 71fe8b301e2..7984b9dd6c7 100644
--- a/doc/administration/reference_architectures/25k_users.md
+++ b/doc/administration/reference_architectures/25k_users.md
@@ -31,8 +31,8 @@ full list of reference architectures, see
| Gitaly<sup>5 6</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` |
| Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` |
| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` |
-| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` |
-| GitLab Rails | 5 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` |
+| Sidekiq<sup>7</sup> | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` |
+| GitLab Rails<sup>7</sup> | 5 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` |
| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` |
| Object storage<sup>4</sup> | - | - | - | - |
@@ -53,6 +53,8 @@ full list of reference architectures, see
6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large
repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to
[Large repositories](index.md#large-repositories) for more information.
+7. Can be placed in Auto Scaling Groups (ASGs) as the component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes).
+ However, for GitLab Rails certain processes like [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md) should be run on only one node.
<!-- markdownlint-enable MD029 -->
NOTE:
@@ -170,7 +172,7 @@ To set up GitLab and its components to accommodate up to 25,000 users:
environment.
1. [Configure the object storage](#configure-the-object-storage)
used for shared data objects.
-1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster,
+1. [Configure advanced search](#configure-advanced-search) (optional) for faster,
more advanced code search across your entire GitLab instance.
The servers start on the same 10.6.0.0/24 private network range, and can
@@ -320,7 +322,7 @@ Configure your load balancer to pass connections on port 443 as `TCP` rather
than `HTTP(S)` protocol. This will pass the connection to the application node's
NGINX service untouched. NGINX will have the SSL certificate and listen on port 443.
-See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html)
+See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html)
for details on managing SSL certificates and configuring NGINX.
#### Load balancer terminates SSL without backend SSL
@@ -331,7 +333,7 @@ terminating SSL.
Since communication between the load balancer and GitLab will not be secure,
there is some additional configuration needed. See the
-[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination)
+[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination)
for details.
#### Load balancer terminates SSL with backend SSL
@@ -344,7 +346,7 @@ Traffic will also be secure between the load balancers and NGINX in this
scenario. There is no need to add configuration for proxied SSL since the
connection will be secure all the way. However, configuration will need to be
added to GitLab to configure SSL certificates. See
-the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html)
+the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html)
for details on managing SSL certificates and configuring NGINX.
<div align="right">
@@ -434,6 +436,11 @@ spread connections equally in practice.
## Configure Consul
+Next, we set up the Consul servers.
+
+NOTE:
+Consul must be deployed in an odd number of 3 nodes or more. This is to ensure the nodes can take votes as part of a quorum.
+
The following IPs will be used as an example:
- `10.6.0.11`: Consul 1
@@ -818,6 +825,9 @@ Using [Redis](https://redis.io/) in scalable environment is possible using a **P
topology with a [Redis Sentinel](https://redis.io/docs/manual/sentinel/) service to watch and automatically
start the failover procedure.
+NOTE:
+Redis clusters must each be deployed in an odd number of 3 nodes or more. This is to ensure Redis Sentinel can take votes as part of a quorum. This does not apply when configuring Redis externally, such as a cloud provider service.
+
Redis requires authentication if used with Sentinel. See
[Redis Security](https://redis.io/docs/manual/security/) documentation for more
information. We recommend using a combination of a Redis password and tight
@@ -851,9 +861,9 @@ to be used with GitLab. The following IPs will be used as an example:
Managed Redis from cloud providers (such as AWS ElastiCache) will work. If these
services support high availability, be sure it _isn't_ of the Redis Cluster type.
-Redis version 5.0 or higher is required, which is included with Omnibus GitLab
-packages starting with GitLab 13.0. Older Redis versions don't support an
-optional count argument to SPOP, which is required for [Merge Trains](../../ci/pipelines/merge_trains.md).
+
+Because Omnibus GitLab packages ship with Redis 6.0 or later, Redis 6.0 or later is required. Older Redis versions have reached end-of-life.
+
Note the Redis node's IP address or hostname, port, and password (if required).
These will be necessary later when configuring the [GitLab application servers](#configure-gitlab-rails).
@@ -1208,7 +1218,7 @@ For more advanced setups refer to the [standalone Gitaly Cluster documentation](
Praefect, the routing and transaction manager for Gitaly Cluster, requires its own database server to store data on Gitaly Cluster status.
If you want to have a highly available setup, Praefect requires a third-party PostgreSQL database.
-A built-in solution is being [worked on](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919).
+A built-in solution is being [worked on](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7292).
#### Praefect non-HA PostgreSQL standalone using Omnibus GitLab
@@ -1353,6 +1363,9 @@ This is how this would work with a Omnibus GitLab PostgreSQL setup:
Praefect is the router and transaction manager for Gitaly Cluster and all connections to Gitaly go through
it. This section details how to configure it.
+NOTE:
+Praefect must be deployed in an odd number of 3 nodes or more. This is to ensure the nodes can take votes as part of a quorum.
+
Praefect requires several secret tokens to secure communications across the Cluster:
- `<praefect_external_token>`: Used for repositories hosted on your Gitaly cluster and can only be accessed by Gitaly clients that carry this token.
@@ -1406,7 +1419,6 @@ Updates to example must be made at:
# Praefect Configuration
praefect['enable'] = true
- praefect['listen_addr'] = '0.0.0.0:2305'
# Prevent database migrations from running on upgrade automatically
praefect['auto_migrate'] = false
@@ -1415,51 +1427,69 @@ Updates to example must be made at:
# Configure the Consul agent
consul['enable'] = true
## Enable service discovery for Prometheus
- consul['monitoring_service_discovery'] = true
+ consul['monitoring_service_discovery'] = true
# START user configuration
# Please set the real values as explained in Required Information section
#
- # Praefect External Token
- # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster
- praefect['auth_token'] = '<praefect_external_token>'
-
- # Praefect Database Settings
- praefect['database_host'] = '10.6.0.141'
- praefect['database_port'] = 5432
- # `no_proxy` settings must always be a direct connection for caching
- praefect['database_direct_host'] = '10.6.0.141'
- praefect['database_direct_port'] = 5432
- praefect['database_dbname'] = 'praefect_production'
- praefect['database_user'] = 'praefect'
- praefect['database_password'] = '<praefect_postgresql_password>'
-
- # Praefect Virtual Storage config
- # Name of storage hash must match storage name in git_data_dirs on GitLab
- # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1')
- praefect['virtual_storages'] = {
- 'default' => {
- 'nodes' => {
- 'gitaly-1' => {
- 'address' => 'tcp://10.6.0.91:8075',
- 'token' => '<praefect_internal_token>'
- },
- 'gitaly-2' => {
- 'address' => 'tcp://10.6.0.92:8075',
- 'token' => '<praefect_internal_token>'
- },
- 'gitaly-3' => {
- 'address' => 'tcp://10.6.0.93:8075',
- 'token' => '<praefect_internal_token>'
+ praefect['configuration'] = {
+ # ...
+ listen_addr: '0.0.0.0:2305',
+ auth: {
+ # ...
+ #
+ # Praefect External Token
+ # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster
+ token: '<praefect_external_token>',
+ },
+ # Praefect Database Settings
+ database: {
+ # ...
+ host: '10.6.0.141',
+ port: 5432,
+ # `no_proxy` settings must always be a direct connection for caching
+ session_pooled: {
+ # ...
+ host: '10.6.0.141',
+ port: 5432,
+ dbname: 'praefect_production',
+ user: 'praefect',
+ password: '<praefect_postgresql_password>',
+ },
+ },
+ # Praefect Virtual Storage config
+ # Name of storage hash must match storage name in git_data_dirs on GitLab
+ # server ('praefect') and in gitaly['configuration'][:storage] on Gitaly nodes ('gitaly-1')
+ virtual_storage: [
+ {
+ # ...
+ name: 'default',
+ node: [
+ {
+ storage: 'gitaly-1',
+ address: 'tcp://10.6.0.91:8075',
+ token: '<praefect_internal_token>'
+ },
+ {
+ storage: 'gitaly-2',
+ address: 'tcp://10.6.0.92:8075',
+ token: '<praefect_internal_token>'
+ },
+ {
+ storage: 'gitaly-3',
+ address: 'tcp://10.6.0.93:8075',
+ token: '<praefect_internal_token>'
+ },
+ ],
},
- }
- }
+ ],
+ # Set the network address Praefect will listen on for monitoring
+ prometheus_listen_addr: '0.0.0.0:9652',
}
- # Set the network addresses that the exporters will listen on for monitoring
+ # Set the network address the node exporter will listen on for monitoring
node_exporter['listen_address'] = '0.0.0.0:9100'
- praefect['prometheus_listen_addr'] = '0.0.0.0:9652'
## The IPs of the Consul server nodes
## You can also use FQDNs and intermix them with IPs
@@ -1510,9 +1540,7 @@ Due to Gitaly having notable input and output requirements, we strongly
recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs
should have a throughput of at least 8,000
input/output operations per second (IOPS) for read operations and 2,000 IOPS for
-write operations. These IOPS values are initial recommendations, and may be
-adjusted to greater or lesser values depending on the scale of your
-environment's workload. If you're running the environment on a Cloud provider,
+write operations. If you're running the environment on a Cloud provider,
refer to their documentation about how to configure IOPS correctly.
Gitaly servers must not be exposed to the public internet, as Gitaly's network
@@ -1522,7 +1550,7 @@ to restrict access to the Gitaly server. Another option is to
For configuring Gitaly you should note the following:
-- `git_data_dirs` should be configured to reflect the storage path for the specific Gitaly node
+- `gitaly['configuration'][:storage]` should be configured to reflect the storage path for the specific Gitaly node
- `auth_token` should be the same as `praefect_internal_token`
The following IPs will be used as an example:
@@ -1571,20 +1599,6 @@ Updates to example must be made at:
# Gitaly
gitaly['enable'] = true
- # Make Gitaly accept connections on all network interfaces. You must use
- # firewalls to restrict access to this address/port.
- # Comment out following line if you only want to support TLS connections
- gitaly['listen_addr'] = "0.0.0.0:8075"
-
- # Gitaly Auth Token
- # Should be the same as praefect_internal_token
- gitaly['auth_token'] = '<praefect_internal_token>'
-
- # Gitaly Pack-objects cache
- # Recommended to be enabled for improved performance but can notably increase disk I/O
- # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info
- gitaly['pack_objects_cache_enabled'] = true
-
# Configure the Consul agent
consul['enable'] = true
## Enable service discovery for Prometheus
@@ -1599,9 +1613,29 @@ Updates to example must be made at:
retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13),
}
- # Set the network addresses that the exporters will listen on for monitoring
+ # Set the network address that the node exporter will listen on for monitoring
node_exporter['listen_address'] = '0.0.0.0:9100'
- gitaly['prometheus_listen_addr'] = '0.0.0.0:9236'
+
+ gitaly['configuration'] = {
+ # Make Gitaly accept connections on all network interfaces. You must use
+ # firewalls to restrict access to this address/port.
+ # Comment out following line if you only want to support TLS connections
+ listen_addr: '0.0.0.0:8075',
+ # Set the network address that Gitaly will listen on for monitoring
+ prometheus_listen_addr: '0.0.0.0:9236',
+ auth: {
+ # Gitaly Auth Token
+ # Should be the same as praefect_internal_token
+ token: '<praefect_internal_token>',
+ },
+ pack_objects_cache: {
+ # Gitaly Pack-objects cache
+ # Recommended to be enabled for improved performance but can notably increase disk I/O
+ # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info
+ enabled: true,
+ },
+ }
+
#
# END user configuration
```
@@ -1610,31 +1644,43 @@ Updates to example must be made at:
- On Gitaly node 1:
```ruby
- git_data_dirs({
- "gitaly-1" => {
- "path" => "/var/opt/gitlab/git-data"
- }
- })
+ gitaly['configuration'] = {
+ # ...
+ storage: [
+ {
+ name: 'gitaly-1',
+ path: '/var/opt/gitlab/git-data',
+ },
+ ],
+ }
```
- On Gitaly node 2:
```ruby
- git_data_dirs({
- "gitaly-2" => {
- "path" => "/var/opt/gitlab/git-data"
- }
- })
+ gitaly['configuration'] = {
+ # ...
+ storage: [
+ {
+ name: 'gitaly-2',
+ path: '/var/opt/gitlab/git-data',
+ },
+ ],
+ }
```
- On Gitaly node 3:
```ruby
- git_data_dirs({
- "gitaly-3" => {
- "path" => "/var/opt/gitlab/git-data"
- }
- })
+ gitaly['configuration'] = {
+ # ...
+ storage: [
+ {
+ name: 'gitaly-3',
+ path: '/var/opt/gitlab/git-data',
+ },
+ ],
+ }
```
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
@@ -1654,7 +1700,7 @@ for secure connections, you must:
Additionally the certificate, or its certificate authority, must be installed on all Gitaly servers
and on all Praefect clients that communicate with it following the procedure described in
-[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) (and repeated below).
+[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl/index.html#install-custom-public-certificates) (and repeated below).
Note the following:
@@ -1663,7 +1709,7 @@ Note the following:
- You can configure Praefect servers with both an unencrypted listening address
`listen_addr` and an encrypted listening address `tls_listen_addr` at the same time.
This allows you to do a gradual transition from unencrypted to encrypted traffic, if
- necessary. To disable the unencrypted listener, set `praefect['listen_addr'] = nil`.
+ necessary. To disable the unencrypted listener, set `praefect['configuration'][:listen_addr] = nil`.
- The Internal Load Balancer will also access to the certificates and need to be configured
to allow for TLS passthrough.
Refer to the load balancers documentation on how to configure this.
@@ -1685,9 +1731,15 @@ To configure Praefect with TLS:
1. Edit `/etc/gitlab/gitlab.rb` and add:
```ruby
- praefect['tls_listen_addr'] = "0.0.0.0:3305"
- praefect['certificate_path'] = "/etc/gitlab/ssl/cert.pem"
- praefect['key_path'] = "/etc/gitlab/ssl/key.pem"
+ praefect['configuration'] = {
+ # ...
+ tls_listen_addr: '0.0.0.0:3305',
+ tls: {
+ # ...
+ certificate_path: '/etc/gitlab/ssl/cert.pem',
+ key_path: '/etc/gitlab/ssl/key.pem',
+ },
+ }
```
1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure).
@@ -1768,7 +1820,7 @@ Updates to example must be made at:
# Redis
## Redis connection details
- ## First cluster that will host the cache
+ ## First cluster that will host the cache data
gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache'
gitlab_rails['redis_cache_sentinels'] = [
@@ -1777,22 +1829,11 @@ Updates to example must be made at:
{host: '10.6.0.53', port: 26379},
]
- ## Second cluster that will host the persistent queues, shared state, and actioncable
- gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
- gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
- gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
+ ## Second cluster that hosts all other persistent data
+ redis['master_name'] = 'gitlab-redis-persistent'
+ redis['master_password'] = '<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>'
- gitlab_rails['redis_queues_sentinels'] = [
- {host: '10.6.0.61', port: 26379},
- {host: '10.6.0.62', port: 26379},
- {host: '10.6.0.63', port: 26379},
- ]
- gitlab_rails['redis_shared_state_sentinels'] = [
- {host: '10.6.0.61', port: 26379},
- {host: '10.6.0.62', port: 26379},
- {host: '10.6.0.63', port: 26379},
- ]
- gitlab_rails['redis_actioncable_sentinels'] = [
+ gitlab_rails['redis_sentinels'] = [
{host: '10.6.0.61', port: 26379},
{host: '10.6.0.62', port: 26379},
{host: '10.6.0.63', port: 26379},
@@ -1955,7 +1996,7 @@ On each node perform the following:
gitlab_rails['auto_migrate'] = false
## Redis connection details
- ## First cluster that will host the cache
+ ## First cluster that will host the cache data
gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache'
gitlab_rails['redis_cache_sentinels'] = [
@@ -1964,22 +2005,11 @@ On each node perform the following:
{host: '10.6.0.53', port: 26379},
]
- ## Second cluster that will host the queues, shared state, and actionable
- gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
- gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
- gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
+ ## Second cluster that hosts all other persistent data
+ redis['master_name'] = 'gitlab-redis-persistent'
+ redis['master_password'] = '<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>'
- gitlab_rails['redis_queues_sentinels'] = [
- {host: '10.6.0.61', port: 26379},
- {host: '10.6.0.62', port: 26379},
- {host: '10.6.0.63', port: 26379},
- ]
- gitlab_rails['redis_shared_state_sentinels'] = [
- {host: '10.6.0.61', port: 26379},
- {host: '10.6.0.62', port: 26379},
- {host: '10.6.0.63', port: 26379},
- ]
- gitlab_rails['redis_actioncable_sentinels'] = [
+ gitlab_rails['redis_sentinels'] = [
{host: '10.6.0.61', port: 26379},
{host: '10.6.0.62', port: 26379},
{host: '10.6.0.63', port: 26379},
@@ -2077,7 +2107,7 @@ On each node perform the following:
When you specify `https` in the `external_url`, as in the previous example,
GitLab expects that the SSL certificates are in `/etc/gitlab/ssl/`. If the
certificates aren't present, NGINX will fail to start. For more information, see
-the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html).
+the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html).
### GitLab Rails post-configuration
@@ -2089,11 +2119,9 @@ the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html).
sudo gitlab-rake gitlab:db:configure
```
- If you encounter a `rake aborted!` error message stating that PgBouncer is
- failing to connect to PostgreSQL, it may be that your PgBouncer node's IP
- address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb`
- on your database nodes. Before proceeding, see
- [PgBouncer error `ERROR: pgbouncer cannot connect to server`](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server).
+ Note that this requires the Rails node to be configured to connect to the primary database
+ directly, [bypassing PgBouncer](../postgresql/pgbouncer.md#procedure-for-bypassing-pgbouncer).
+ After migrations have completed, you must configure the node to pass through PgBouncer again.
1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md).
@@ -2188,15 +2216,15 @@ and scalable.
There are two ways of specifying object storage configuration in GitLab:
-- [Consolidated form](../object_storage.md#consolidated-object-storage-configuration): A single credential is
+- [Consolidated form](../object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form): A single credential is
shared by all supported object types.
-- [Storage-specific form](../object_storage.md#storage-specific-configuration): Every object defines its
- own object storage [connection and configuration](../object_storage.md#connection-settings).
+- [Storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form): Every object defines its
+ own object storage [connection and configuration](../object_storage.md#configure-the-connection-settings).
The consolidated form is used in the following examples when available.
NOTE:
-When using the [storage-specific form](../object_storage.md#storage-specific-configuration)
+When using the [storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form)
in GitLab 14.x and earlier, you should enable [direct upload mode](../../development/uploads/index.md#direct-upload).
The previous [background upload](../../development/uploads/index.md#direct-upload) mode,
which was deprecated in 14.9, requires shared storage such as NFS.
@@ -2218,9 +2246,9 @@ GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily
While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs.
-## Configure Advanced Search
+## Configure advanced search
-You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md)
+You can leverage Elasticsearch and [enable advanced search](../../integration/advanced_search/elasticsearch.md)
for faster, more advanced code search across your entire GitLab instance.
Elasticsearch cluster design and requirements are dependent on your specific
diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md
index ee26504902c..14dc9d26293 100644
--- a/doc/administration/reference_architectures/2k_users.md
+++ b/doc/administration/reference_architectures/2k_users.md
@@ -26,7 +26,7 @@ For a full list of reference architectures, see
| PostgreSQL<sup>1</sup> | 1 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` |
| Redis<sup>2</sup> | 1 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `m5.large` | `D2s v3` |
| Gitaly<sup>5</sup> | 1 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` |
-| GitLab Rails | 2 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` |
+| GitLab Rails<sup>6</sup> | 2 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` |
| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` |
| Object storage<sup>4</sup> | - | - | - | - | - |
@@ -45,6 +45,8 @@ For a full list of reference architectures, see
5. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large
repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to
[Large repositories](index.md#large-repositories) for more information.
+6. Can be placed in Auto Scaling Groups (ASGs) as the component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes).
+ However, for GitLab Rails certain processes like [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md) should be run on only one node.
<!-- markdownlint-enable MD029 -->
NOTE:
@@ -101,7 +103,7 @@ To set up GitLab and its components to accommodate up to 2,000 users:
environment.
1. [Configure the object storage](#configure-the-object-storage) used for
shared data objects.
-1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster,
+1. [Configure advanced search](#configure-advanced-search) (optional) for faster,
more advanced code search across your entire GitLab instance.
## Configure the external load balancer
@@ -211,7 +213,7 @@ Configure your load balancer to pass connections on port 443 as `TCP` rather
than `HTTP(S)` protocol. This will pass the connection to the application node's
NGINX service untouched. NGINX will have the SSL certificate and listen on port 443.
-See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html)
+See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html)
for details on managing SSL certificates and configuring NGINX.
#### Load balancer terminates SSL without backend SSL
@@ -222,7 +224,7 @@ terminating SSL.
Since communication between the load balancer and GitLab will not be secure,
there is some additional configuration needed. See the
-[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination)
+[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination)
for details.
#### Load balancer terminates SSL with backend SSL
@@ -235,7 +237,7 @@ Traffic will also be secure between the load balancers and NGINX in this
scenario. There is no need to add configuration for proxied SSL since the
connection will be secure all the way. However, configuration will need to be
added to GitLab to configure SSL certificates. See
-the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html)
+the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html)
for details on managing SSL certificates and configuring NGINX.
<div align="right">
@@ -339,13 +341,7 @@ to be used with GitLab.
### Provide your own Redis instance
-Redis version 5.0 or higher is required, as this is what ships with
-Omnibus GitLab packages starting with GitLab 13.0. Older Redis versions
-do not support an optional count argument to SPOP which is now required for
-[Merge Trains](../../ci/pipelines/merge_trains.md).
-
-In addition, GitLab makes use of certain commands like `UNLINK` and `USAGE` which
-were introduced only in Redis 4.
+Because Omnibus GitLab packages ship with Redis 6.0 or later, Redis 6.0 or later is required. Older Redis versions have reached end-of-life.
Managed Redis from cloud providers such as AWS ElastiCache will work. If these
services support high availability, be sure it is not the Redis Cluster type.
@@ -423,9 +419,7 @@ Due to Gitaly having notable input and output requirements, we strongly
recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs
should have a throughput of at least 8,000
input/output operations per second (IOPS) for read operations and 2,000 IOPS
-for write operations. These IOPS values are initial recommendations, and may be
-adjusted to greater or lesser values depending on the scale of your
-environment's workload. If you're running the environment on a Cloud provider,
+for write operations. If you're running the environment on a Cloud provider,
refer to their documentation about how to configure IOPS correctly.
Be sure to note the following items:
@@ -459,7 +453,7 @@ To configure the Gitaly server, on the server node you want to use for Gitaly:
storage paths, enable the network listener, and to configure the token:
NOTE:
- You can't remove the `default` entry from `git_data_dirs` because [GitLab requires it](../gitaly/configure_gitaly.md#gitlab-requires-a-default-repository-storage).
+ You can't remove the `default` entry from `gitaly['configuration'][:storage]` because [GitLab requires it](../gitaly/configure_gitaly.md#gitlab-requires-a-default-repository-storage).
<!--
Updates to example must be made at:
@@ -493,30 +487,48 @@ Updates to example must be made at:
# Gitaly
gitaly['enable'] = true
- # Make Gitaly accept connections on all network interfaces. You must use
- # firewalls to restrict access to this address/port.
- # Comment out following line if you only want to support TLS connections
- gitaly['listen_addr'] = "0.0.0.0:8075"
- gitaly['prometheus_listen_addr'] = "0.0.0.0:9236"
-
- # Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests
- # to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API.
- # The following two values must be the same as their respective values
- # of the GitLab Rails application setup
- gitaly['auth_token'] = 'gitalysecret'
+ # The secret token is used for authentication callbacks from Gitaly to the GitLab internal API.
+ # This must match the respective value in GitLab Rails application setup.
gitlab_shell['secret_token'] = 'shellsecret'
# Set the network addresses that the exporters used for monitoring will listen on
node_exporter['listen_address'] = '0.0.0.0:9100'
- git_data_dirs({
- 'default' => {
- 'path' => '/var/opt/gitlab/git-data'
- },
- 'storage1' => {
- 'path' => '/mnt/gitlab/git-data'
- },
- })
+ gitaly['configuration'] = {
+ # ...
+ #
+ # Make Gitaly accept connections on all network interfaces. You must use
+ # firewalls to restrict access to this address/port.
+ # Comment out following line if you only want to support TLS connections
+ listen_addr: '0.0.0.0:8075',
+ prometheus_listen_addr: '0.0.0.0:9236',
+ # Gitaly Auth Token
+ # Should be the same as praefect_internal_token
+ auth: {
+ # ...
+ #
+ # Gitaly's authentication token is used to authenticate gRPC requests to Gitaly. This must match
+ # the respective value in GitLab Rails application setup.
+ token: 'gitalysecret',
+ },
+ # Gitaly Pack-objects cache
+ # Recommended to be enabled for improved performance but can notably increase disk I/O
+ # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info
+ pack_objects_cache: {
+ # ...
+ enabled: true,
+ },
+ storage: [
+ {
+ name: 'default',
+ path: '/var/opt/gitlab/git-data',
+ },
+ {
+ name: 'storage1',
+ path: '/mnt/gitlab/git-data',
+ },
+ ],
+ }
```
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
@@ -538,7 +550,7 @@ You will need to bring your own certificates as this isn't provided automaticall
The certificate, or its certificate authority, must be installed on all Gitaly
nodes (including the Gitaly node using the certificate) and on all client nodes
that communicate with it following the procedure described in
-[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates).
+[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl/index.html#install-custom-public-certificates).
NOTE:
The self-signed certificate must specify the address you use to access the
@@ -574,9 +586,14 @@ To configure Gitaly with TLS:
<!-- Updates to following example must also be made at https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab -->
```ruby
- gitaly['tls_listen_addr'] = "0.0.0.0:9999"
- gitaly['certificate_path'] = "/etc/gitlab/ssl/cert.pem"
- gitaly['key_path'] = "/etc/gitlab/ssl/key.pem"
+ gitaly['configuration'] = {
+ # ...
+ tls_listen_addr: '0.0.0.0:9999',
+ tls: {
+ certificate_path: '/etc/gitlab/ssl/cert.pem',
+ key_path: '/etc/gitlab/ssl/key.pem',
+ },
+ }
```
1. Delete `gitaly['listen_addr']` to allow only encrypted connections.
@@ -739,7 +756,7 @@ On each node perform the following:
When you specify `https` in the `external_url`, as in the previous example,
GitLab expects that the SSL certificates are in `/etc/gitlab/ssl/`. If the
certificates aren't present, NGINX will fail to start. For more information, see
-the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html).
+the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html).
### GitLab Rails post-configuration
@@ -751,11 +768,9 @@ the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html).
sudo gitlab-rake gitlab:db:configure
```
- If you encounter a `rake aborted!` error message stating that PgBouncer is
- failing to connect to PostgreSQL, it may be that your PgBouncer node's IP
- address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb`
- on your database nodes. Before proceeding, see
- [PgBouncer error `ERROR: pgbouncer cannot connect to server`](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server).
+ Note that this requires the Rails node to be configured to connect to the primary database
+ directly, [bypassing PgBouncer](../postgresql/pgbouncer.md#procedure-for-bypassing-pgbouncer).
+ After migrations have completed, you must configure the node to pass through PgBouncer again.
1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md).
@@ -881,15 +896,15 @@ and scalable.
There are two ways of specifying object storage configuration in GitLab:
-- [Consolidated form](../object_storage.md#consolidated-object-storage-configuration): A single credential is
+- [Consolidated form](../object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form): A single credential is
shared by all supported object types.
-- [Storage-specific form](../object_storage.md#storage-specific-configuration): Every object defines its
- own object storage [connection and configuration](../object_storage.md#connection-settings).
+- [Storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form): Every object defines its
+ own object storage [connection and configuration](../object_storage.md#configure-the-connection-settings).
The consolidated form is used in the following examples when available.
NOTE:
-When using the [storage-specific form](../object_storage.md#storage-specific-configuration)
+When using the [storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form)
in GitLab 14.x and earlier, you should enable [direct upload mode](../../development/uploads/index.md#direct-upload).
The previous [background upload](../../development/uploads/index.md#direct-upload) mode,
which was deprecated in 14.9, requires shared storage such as NFS.
@@ -911,9 +926,9 @@ GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily
While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs.
-## Configure Advanced Search **(PREMIUM SELF)**
+## Configure advanced search **(PREMIUM SELF)**
-You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md)
+You can leverage Elasticsearch and [enable advanced search](../../integration/advanced_search/elasticsearch.md)
for faster, more advanced code search across your entire GitLab instance.
Elasticsearch cluster design and requirements are dependent on your specific
diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md
index 1d5dad02b18..191e5218c43 100644
--- a/doc/administration/reference_architectures/3k_users.md
+++ b/doc/administration/reference_architectures/3k_users.md
@@ -40,8 +40,8 @@ For a full list of reference architectures, see
| Gitaly<sup>5 6</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` |
| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` |
| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` |
-| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` |
-| GitLab Rails | 3 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` |
+| Sidekiq<sup>7</sup> | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` |
+| GitLab Rails<sup>7</sup> | 3 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` |
| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` |
| Object storage<sup>4</sup> | - | - | - | - |
@@ -62,6 +62,8 @@ For a full list of reference architectures, see
6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large
repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to
[Large repositories](index.md#large-repositories) for more information.
+7. Can be placed in Auto Scaling Groups (ASGs) as the component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes).
+ However, for GitLab Rails certain processes like [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md) should be run on only one node.
<!-- markdownlint-enable MD029 -->
NOTE:
@@ -176,7 +178,7 @@ To set up GitLab and its components to accommodate up to 3,000 users:
environment.
1. [Configure the object storage](#configure-the-object-storage)
used for shared data objects.
-1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster,
+1. [Configure advanced search](#configure-advanced-search) (optional) for faster,
more advanced code search across your entire GitLab instance.
The servers start on the same 10.6.0.0/24 private network range, and can
@@ -321,7 +323,7 @@ Configure your load balancer to pass connections on port 443 as `TCP` rather
than `HTTP(S)` protocol. This will pass the connection to the application node's
NGINX service untouched. NGINX will have the SSL certificate and listen on port 443.
-See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html)
+See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html)
for details on managing SSL certificates and configuring NGINX.
#### Load balancer terminates SSL without backend SSL
@@ -332,7 +334,7 @@ terminating SSL.
Since communication between the load balancer and GitLab will not be secure,
there is some additional configuration needed. See the
-[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination)
+[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination)
for details.
#### Load balancer terminates SSL with backend SSL
@@ -345,7 +347,7 @@ Traffic will also be secure between the load balancers and NGINX in this
scenario. There is no need to add configuration for proxied SSL since the
connection will be secure all the way. However, configuration will need to be
added to GitLab to configure SSL certificates. See
-the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html)
+the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html)
for details on managing SSL certificates and configuring NGINX.
<div align="right">
@@ -453,10 +455,7 @@ to be used with GitLab. The following IPs will be used as an example:
Managed Redis from cloud providers such as AWS ElastiCache will work. If these
services support high availability, be sure it is **not** the Redis Cluster type.
-Redis version 5.0 or higher is required, as this is what ships with
-Omnibus GitLab packages starting with GitLab 13.0. Older Redis versions
-do not support an optional count argument to SPOP which is now required for
-[Merge Trains](../../ci/pipelines/merge_trains.md).
+Because Omnibus GitLab packages ship with Redis 6.0 or later, Redis 6.0 or later is required. Older Redis versions have reached end-of-life.
Note the Redis node's IP address or hostname, port, and password (if required).
These will be necessary when configuring the
@@ -640,8 +639,13 @@ are supported and can be added if needed.
## Configure Consul and Sentinel
-Now that the Redis servers are all set up, let's configure the Sentinel
-servers. The following IPs will be used as an example:
+Now that the Redis servers are all set up, let's configure the Consul + Sentinel
+servers.
+
+NOTE:
+Consul and Redis Sentinel must be deployed in an odd number of 3 nodes or more. This is to ensure the nodes can take votes as part of a quorum.
+
+The following IPs will be used as an example:
- `10.6.0.11`: Consul/Sentinel 1
- `10.6.0.12`: Consul/Sentinel 2
@@ -1144,7 +1148,7 @@ For more advanced setups refer to the [standalone Gitaly Cluster documentation](
Praefect, the routing and transaction manager for Gitaly Cluster, requires its own database server to store data on Gitaly Cluster status.
If you want to have a highly available setup, Praefect requires a third-party PostgreSQL database.
-A built-in solution is being [worked on](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919).
+A built-in solution is being [worked on](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7292).
#### Praefect non-HA PostgreSQL standalone using Omnibus GitLab
@@ -1288,6 +1292,9 @@ This is how this would work with a Omnibus GitLab PostgreSQL setup:
Praefect is the router and transaction manager for Gitaly Cluster and all connections to Gitaly go through
it. This section details how to configure it.
+NOTE:
+Praefect must be deployed in an odd number of 3 nodes or more. This is to ensure the nodes can take votes as part of a quorum.
+
Praefect requires several secret tokens to secure communications across the Cluster:
- `<praefect_external_token>`: Used for repositories hosted on your Gitaly cluster and can only be accessed by Gitaly clients that carry this token.
@@ -1341,7 +1348,6 @@ Updates to example must be made at:
# Praefect Configuration
praefect['enable'] = true
- praefect['listen_addr'] = '0.0.0.0:2305'
# Prevent database migrations from running on upgrade automatically
praefect['auto_migrate'] = false
@@ -1350,51 +1356,69 @@ Updates to example must be made at:
# Configure the Consul agent
consul['enable'] = true
## Enable service discovery for Prometheus
- consul['monitoring_service_discovery'] = true
+ consul['monitoring_service_discovery'] = true
# START user configuration
# Please set the real values as explained in Required Information section
#
- # Praefect External Token
- # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster
- praefect['auth_token'] = '<praefect_external_token>'
-
- # Praefect Database Settings
- praefect['database_host'] = '10.6.0.141'
- praefect['database_port'] = 5432
- # `no_proxy` settings must always be a direct connection for caching
- praefect['database_direct_host'] = '10.6.0.141'
- praefect['database_direct_port'] = 5432
- praefect['database_dbname'] = 'praefect_production'
- praefect['database_user'] = 'praefect'
- praefect['database_password'] = '<praefect_postgresql_password>'
-
- # Praefect Virtual Storage config
- # Name of storage hash must match storage name in git_data_dirs on GitLab
- # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1')
- praefect['virtual_storages'] = {
- 'default' => {
- 'nodes' => {
- 'gitaly-1' => {
- 'address' => 'tcp://10.6.0.91:8075',
- 'token' => '<praefect_internal_token>'
- },
- 'gitaly-2' => {
- 'address' => 'tcp://10.6.0.92:8075',
- 'token' => '<praefect_internal_token>'
- },
- 'gitaly-3' => {
- 'address' => 'tcp://10.6.0.93:8075',
- 'token' => '<praefect_internal_token>'
+ praefect['configuration'] = {
+ # ...
+ listen_addr: '0.0.0.0:2305',
+ auth: {
+ # ...
+ #
+ # Praefect External Token
+ # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster
+ token: '<praefect_external_token>',
+ },
+ # Praefect Database Settings
+ database: {
+ # ...
+ host: '10.6.0.141',
+ port: 5432,
+ # `no_proxy` settings must always be a direct connection for caching
+ session_pooled: {
+ # ...
+ host: '10.6.0.141',
+ port: 5432,
+ dbname: 'praefect_production',
+ user: 'praefect',
+ password: '<praefect_postgresql_password>',
+ },
+ },
+ # Praefect Virtual Storage config
+ # Name of storage hash must match storage name in git_data_dirs on GitLab
+ # server ('praefect') and in gitaly['configuration'][:storage] on Gitaly nodes ('gitaly-1')
+ virtual_storage: [
+ {
+ # ...
+ name: 'default',
+ node: [
+ {
+ storage: 'gitaly-1',
+ address: 'tcp://10.6.0.91:8075',
+ token: '<praefect_internal_token>'
+ },
+ {
+ storage: 'gitaly-2',
+ address: 'tcp://10.6.0.92:8075',
+ token: '<praefect_internal_token>'
+ },
+ {
+ storage: 'gitaly-3',
+ address: 'tcp://10.6.0.93:8075',
+ token: '<praefect_internal_token>'
+ },
+ ],
},
- }
- }
+ ],
+ # Set the network address Praefect will listen on for monitoring
+ prometheus_listen_addr: '0.0.0.0:9652',
}
- # Set the network addresses that the exporters will listen on for monitoring
+ # Set the network address the node exporter will listen on for monitoring
node_exporter['listen_address'] = '0.0.0.0:9100'
- praefect['prometheus_listen_addr'] = '0.0.0.0:9652'
## The IPs of the Consul server nodes
## You can also use FQDNs and intermix them with IPs
@@ -1445,9 +1469,7 @@ Due to Gitaly having notable input and output requirements, we strongly
recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs
should have a throughput of at least 8,000
input/output operations per second (IOPS) for read operations and 2,000 IOPS for
-write operations. These IOPS values are initial recommendations, and may be
-adjusted to greater or lesser values depending on the scale of your
-environment's workload. If you're running the environment on a Cloud provider,
+write operations. If you're running the environment on a Cloud provider,
refer to their documentation about how to configure IOPS correctly.
Gitaly servers must not be exposed to the public internet, as Gitaly's network
@@ -1457,7 +1479,7 @@ to restrict access to the Gitaly server. Another option is to
For configuring Gitaly you should note the following:
-- `git_data_dirs` should be configured to reflect the storage path for the specific Gitaly node
+- `gitaly['configuration'][:storage]` should be configured to reflect the storage path for the specific Gitaly node
- `auth_token` should be the same as `praefect_internal_token`
The following IPs will be used as an example:
@@ -1506,20 +1528,6 @@ Updates to example must be made at:
# balancer.
gitlab_rails['internal_api_url'] = 'https://gitlab.example.com'
- # Make Gitaly accept connections on all network interfaces. You must use
- # firewalls to restrict access to this address/port.
- # Comment out following line if you only want to support TLS connections
- gitaly['listen_addr'] = "0.0.0.0:8075"
-
- # Gitaly Auth Token
- # Should be the same as praefect_internal_token
- gitaly['auth_token'] = '<praefect_internal_token>'
-
- # Gitaly Pack-objects cache
- # Recommended to be enabled for improved performance but can notably increase disk I/O
- # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info
- gitaly['pack_objects_cache_enabled'] = true
-
# Configure the Consul agent
consul['enable'] = true
## Enable service discovery for Prometheus
@@ -1534,9 +1542,33 @@ Updates to example must be made at:
retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13),
}
- # Set the network addresses that the exporters will listen on for monitoring
+ # Set the network address that the node exporter will listen on for monitoring
node_exporter['listen_address'] = '0.0.0.0:9100'
- gitaly['prometheus_listen_addr'] = '0.0.0.0:9236'
+
+ gitaly['configuration'] = {
+ # ...
+ #
+ # Make Gitaly accept connections on all network interfaces. You must use
+ # firewalls to restrict access to this address/port.
+ # Comment out following line if you only want to support TLS connections
+ listen_addr: '0.0.0.0:8075',
+ # Set the network address that Gitaly will listen on for monitoring
+ prometheus_listen_addr: '0.0.0.0:9236',
+ # Gitaly Auth Token
+ # Should be the same as praefect_internal_token
+ auth: {
+ # ...
+ token: '<praefect_internal_token>',
+ },
+ # Gitaly Pack-objects cache
+ # Recommended to be enabled for improved performance but can notably increase disk I/O
+ # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info
+ pack_objects_cache: {
+ # ...
+ enabled: true,
+ },
+ }
+
#
# END user configuration
```
@@ -1545,31 +1577,43 @@ Updates to example must be made at:
- On Gitaly node 1:
```ruby
- git_data_dirs({
- "gitaly-1" => {
- "path" => "/var/opt/gitlab/git-data"
- }
- })
+ gitaly['configuration'] = {
+ # ...
+ storage: [
+ {
+ name: 'gitaly-1',
+ path: '/var/opt/gitlab/git-data',
+ },
+ ],
+ }
```
- On Gitaly node 2:
```ruby
- git_data_dirs({
- "gitaly-2" => {
- "path" => "/var/opt/gitlab/git-data"
- }
- })
+ gitaly['configuration'] = {
+ # ...
+ storage: [
+ {
+ name: 'gitaly-2',
+ path: '/var/opt/gitlab/git-data',
+ },
+ ],
+ }
```
- On Gitaly node 3:
```ruby
- git_data_dirs({
- "gitaly-3" => {
- "path" => "/var/opt/gitlab/git-data"
- }
- })
+ gitaly['configuration'] = {
+ # ...
+ storage: [
+ {
+ name: 'gitaly-3',
+ path: '/var/opt/gitlab/git-data',
+ },
+ ],
+ }
```
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
@@ -1589,7 +1633,7 @@ for secure connections, you must:
Additionally the certificate, or its certificate authority, must be installed on all Gitaly servers
and on all Praefect clients that communicate with it following the procedure described in
-[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) (and repeated below).
+[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl/index.html#install-custom-public-certificates) (and repeated below).
Note the following:
@@ -1598,7 +1642,7 @@ Note the following:
- You can configure Praefect servers with both an unencrypted listening address
`listen_addr` and an encrypted listening address `tls_listen_addr` at the same time.
This allows you to do a gradual transition from unencrypted to encrypted traffic, if
- necessary. To disable the unencrypted listener, set `praefect['listen_addr'] = nil`.
+ necessary. To disable the unencrypted listener, set `praefect['configuration'][:listen_addr] = nil`.
- The Internal Load Balancer will also access to the certificates and need to be configured
to allow for TLS passthrough.
Refer to the load balancers documentation on how to configure this.
@@ -1620,9 +1664,15 @@ To configure Praefect with TLS:
1. Edit `/etc/gitlab/gitlab.rb` and add:
```ruby
- praefect['tls_listen_addr'] = "0.0.0.0:3305"
- praefect['certificate_path'] = "/etc/gitlab/ssl/cert.pem"
- praefect['key_path'] = "/etc/gitlab/ssl/key.pem"
+ praefect['configuration'] = {
+ # ...
+ tls_listen_addr: '0.0.0.0:3305',
+ tls: {
+ # ...
+ certificate_path: '/etc/gitlab/ssl/cert.pem',
+ key_path: '/etc/gitlab/ssl/key.pem',
+ },
+ }
```
1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure).
@@ -2018,7 +2068,7 @@ On each node perform the following:
When you specify `https` in the `external_url`, as in the previous example,
GitLab expects that the SSL certificates are in `/etc/gitlab/ssl/`. If the
certificates aren't present, NGINX will fail to start. For more information, see
-the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html).
+the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html).
### GitLab Rails post-configuration
@@ -2028,11 +2078,9 @@ the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html).
gitlab-rake gitlab:db:configure
```
- If you encounter a `rake aborted!` error message stating that PgBouncer is
- failing to connect to PostgreSQL, it may be that your PgBouncer node's IP
- address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb`
- on your database nodes. Before proceeding, see
- [PgBouncer error `ERROR: pgbouncer cannot connect to server`](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server).
+ Note that this requires the Rails node to be configured to connect to the primary database
+ directly, [bypassing PgBouncer](../postgresql/pgbouncer.md#procedure-for-bypassing-pgbouncer).
+ After migrations have completed, you must configure the node to pass through PgBouncer again.
1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md).
@@ -2135,15 +2183,15 @@ and scalable.
There are two ways of specifying object storage configuration in GitLab:
-- [Consolidated form](../object_storage.md#consolidated-object-storage-configuration): A single credential is
+- [Consolidated form](../object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form): A single credential is
shared by all supported object types.
-- [Storage-specific form](../object_storage.md#storage-specific-configuration): Every object defines its
- own object storage [connection and configuration](../object_storage.md#connection-settings).
+- [Storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form): Every object defines its
+ own object storage [connection and configuration](../object_storage.md#configure-the-connection-settings).
The consolidated form is used in the following examples when available.
NOTE:
-When using the [storage-specific form](../object_storage.md#storage-specific-configuration)
+When using the [storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form)
in GitLab 14.x and earlier, you should enable [direct upload mode](../../development/uploads/index.md#direct-upload).
The previous [background upload](../../development/uploads/index.md#direct-upload) mode,
which was deprecated in 14.9, requires shared storage such as NFS.
@@ -2165,9 +2213,9 @@ GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily
While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs.
-## Configure Advanced Search
+## Configure advanced search
-You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md)
+You can leverage Elasticsearch and [enable advanced search](../../integration/advanced_search/elasticsearch.md)
for faster, more advanced code search across your entire GitLab instance.
Elasticsearch cluster design and requirements are dependent on your specific
diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md
index 3bcffa8f606..7d3ddf7578c 100644
--- a/doc/administration/reference_architectures/50k_users.md
+++ b/doc/administration/reference_architectures/50k_users.md
@@ -31,8 +31,8 @@ full list of reference architectures, see
| Gitaly<sup>5 6</sup> | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | `m5.16xlarge` |
| Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` |
| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` |
-| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` |
-| GitLab Rails | 12 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` |
+| Sidekiq<sup>7</sup> | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` |
+| GitLab Rails<sup>7</sup> | 12 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` |
| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` |
| Object storage<sup>4</sup> | - | - | - | - |
@@ -53,6 +53,8 @@ full list of reference architectures, see
6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large
repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to
[Large repositories](index.md#large-repositories) for more information.
+7. Can be placed in Auto Scaling Groups (ASGs) as the component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes).
+ However, for GitLab Rails certain processes like [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md) should be run on only one node.
<!-- markdownlint-enable MD029 -->
NOTE:
@@ -170,7 +172,7 @@ To set up GitLab and its components to accommodate up to 50,000 users:
environment.
1. [Configure the object storage](#configure-the-object-storage)
used for shared data objects.
-1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster,
+1. [Configure advanced search](#configure-advanced-search) (optional) for faster,
more advanced code search across your entire GitLab instance.
The servers start on the same 10.6.0.0/24 private network range, and can
@@ -318,7 +320,7 @@ Configure your load balancer to pass connections on port 443 as `TCP` rather
than `HTTP(S)` protocol. This will pass the connection to the application node's
NGINX service untouched. NGINX will have the SSL certificate and listen on port 443.
-See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html)
+See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html)
for details on managing SSL certificates and configuring NGINX.
#### Load balancer terminates SSL without backend SSL
@@ -329,7 +331,7 @@ terminating SSL.
Since communication between the load balancer and GitLab will not be secure,
there is some additional configuration needed. See the
-[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination)
+[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination)
for details.
#### Load balancer terminates SSL with backend SSL
@@ -342,7 +344,7 @@ Traffic will also be secure between the load balancers and NGINX in this
scenario. There is no need to add configuration for proxied SSL since the
connection will be secure all the way. However, configuration will need to be
added to GitLab to configure SSL certificates. See
-the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html)
+the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html)
for details on managing SSL certificates and configuring NGINX.
<div align="right">
@@ -426,6 +428,11 @@ spread connections equally in practice.
## Configure Consul
+Next, we set up the Consul servers.
+
+NOTE:
+Consul must be deployed in an odd number of 3 nodes or more. This is to ensure the nodes can take votes as part of a quorum.
+
The following IPs will be used as an example:
- `10.6.0.11`: Consul 1
@@ -811,6 +818,9 @@ Using [Redis](https://redis.io/) in scalable environment is possible using a **P
topology with a [Redis Sentinel](https://redis.io/docs/manual/sentinel/) service to watch and automatically
start the failover procedure.
+NOTE:
+Redis clusters must each be deployed in an odd number of 3 nodes or more. This is to ensure Redis Sentinel can take votes as part of a quorum. This does not apply when configuring Redis externally, such as a cloud provider service.
+
Redis requires authentication if used with Sentinel. See
[Redis Security](https://redis.io/docs/manual/security/) documentation for more
information. We recommend using a combination of a Redis password and tight
@@ -844,9 +854,9 @@ to be used with GitLab. The following IPs will be used as an example:
Managed Redis from cloud providers (such as AWS ElastiCache) will work. If these
services support high availability, be sure it _isn't_ of the Redis Cluster type.
-Redis version 5.0 or higher is required, which is included with Omnibus GitLab
-packages starting with GitLab 13.0. Older Redis versions don't support an
-optional count argument to SPOP, which is required for [Merge Trains](../../ci/pipelines/merge_trains.md).
+
+Because Omnibus GitLab packages ship with Redis 6.0 or later, Redis 6.0 or later is required. Older Redis versions have reached end-of-life.
+
Note the Redis node's IP address or hostname, port, and password (if required).
These will be necessary later when configuring the [GitLab application servers](#configure-gitlab-rails).
@@ -1202,7 +1212,7 @@ For more advanced setups refer to the [standalone Gitaly Cluster documentation](
Praefect, the routing and transaction manager for Gitaly Cluster, requires its own database server to store data on Gitaly Cluster status.
If you want to have a highly available setup, Praefect requires a third-party PostgreSQL database.
-A built-in solution is being [worked on](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919).
+A built-in solution is being [worked on](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7292).
#### Praefect non-HA PostgreSQL standalone using Omnibus GitLab
@@ -1349,6 +1359,9 @@ This is how this would work with a Omnibus GitLab PostgreSQL setup:
Praefect is the router and transaction manager for Gitaly Cluster and all connections to Gitaly go through
it. This section details how to configure it.
+NOTE:
+Praefect must be deployed in an odd number of 3 nodes or later. This is to ensure the nodes can take votes as part of a quorum.
+
Praefect requires several secret tokens to secure communications across the Cluster:
- `<praefect_external_token>`: Used for repositories hosted on your Gitaly cluster and can only be accessed by Gitaly clients that carry this token.
@@ -1402,7 +1415,6 @@ Updates to example must be made at:
# Praefect Configuration
praefect['enable'] = true
- praefect['listen_addr'] = '0.0.0.0:2305'
# Prevent database migrations from running on upgrade automatically
praefect['auto_migrate'] = false
@@ -1411,51 +1423,69 @@ Updates to example must be made at:
# Configure the Consul agent
consul['enable'] = true
## Enable service discovery for Prometheus
- consul['monitoring_service_discovery'] = true
+ consul['monitoring_service_discovery'] = true
# START user configuration
# Please set the real values as explained in Required Information section
#
- # Praefect External Token
- # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster
- praefect['auth_token'] = '<praefect_external_token>'
-
- # Praefect Database Settings
- praefect['database_host'] = '10.6.0.141'
- praefect['database_port'] = 5432
- # `no_proxy` settings must always be a direct connection for caching
- praefect['database_direct_host'] = '10.6.0.141'
- praefect['database_direct_port'] = 5432
- praefect['database_dbname'] = 'praefect_production'
- praefect['database_user'] = 'praefect'
- praefect['database_password'] = '<praefect_postgresql_password>'
-
- # Praefect Virtual Storage config
- # Name of storage hash must match storage name in git_data_dirs on GitLab
- # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1')
- praefect['virtual_storages'] = {
- 'default' => {
- 'nodes' => {
- 'gitaly-1' => {
- 'address' => 'tcp://10.6.0.91:8075',
- 'token' => '<praefect_internal_token>'
- },
- 'gitaly-2' => {
- 'address' => 'tcp://10.6.0.92:8075',
- 'token' => '<praefect_internal_token>'
- },
- 'gitaly-3' => {
- 'address' => 'tcp://10.6.0.93:8075',
- 'token' => '<praefect_internal_token>'
+ praefect['configuration'] = {
+ # ...
+ listen_addr: '0.0.0.0:2305',
+ auth: {
+ # ...
+ #
+ # Praefect External Token
+ # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster
+ token: '<praefect_external_token>',
+ },
+ # Praefect Database Settings
+ database: {
+ # ...
+ host: '10.6.0.141',
+ port: 5432,
+ # `no_proxy` settings must always be a direct connection for caching
+ session_pooled: {
+ # ...
+ host: '10.6.0.141',
+ port: 5432,
+ dbname: 'praefect_production',
+ user: 'praefect',
+ password: '<praefect_postgresql_password>',
+ },
+ },
+ # Praefect Virtual Storage config
+ # Name of storage hash must match storage name in git_data_dirs on GitLab
+ # server ('praefect') and in gitaly['configuration'][:storage] on Gitaly nodes ('gitaly-1')
+ virtual_storage: [
+ {
+ # ...
+ name: 'default',
+ node: [
+ {
+ storage: 'gitaly-1',
+ address: 'tcp://10.6.0.91:8075',
+ token: '<praefect_internal_token>'
+ },
+ {
+ storage: 'gitaly-2',
+ address: 'tcp://10.6.0.92:8075',
+ token: '<praefect_internal_token>'
+ },
+ {
+ storage: 'gitaly-3',
+ address: 'tcp://10.6.0.93:8075',
+ token: '<praefect_internal_token>'
+ },
+ ],
},
- }
- }
+ ],
+ # Set the network address Praefect will listen on for monitoring
+ prometheus_listen_addr: '0.0.0.0:9652',
}
- # Set the network addresses that the exporters will listen on for monitoring
+ # Set the network address the node exporter will listen on for monitoring
node_exporter['listen_address'] = '0.0.0.0:9100'
- praefect['prometheus_listen_addr'] = '0.0.0.0:9652'
## The IPs of the Consul server nodes
## You can also use FQDNs and intermix them with IPs
@@ -1506,9 +1536,7 @@ Due to Gitaly having notable input and output requirements, we strongly
recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs
should have a throughput of at least 8,000
input/output operations per second (IOPS) for read operations and 2,000 IOPS for
-write operations. These IOPS values are initial recommendations, and may be
-adjusted to greater or lesser values depending on the scale of your
-environment's workload. If you're running the environment on a Cloud provider,
+write operations. If you're running the environment on a Cloud provider,
refer to their documentation about how to configure IOPS correctly.
Gitaly servers must not be exposed to the public internet, as Gitaly's network
@@ -1518,7 +1546,7 @@ to restrict access to the Gitaly server. Another option is to
For configuring Gitaly you should note the following:
-- `git_data_dirs` should be configured to reflect the storage path for the specific Gitaly node
+- `gitaly['configuration'][:storage]` should be configured to reflect the storage path for the specific Gitaly node
- `auth_token` should be the same as `praefect_internal_token`
The following IPs will be used as an example:
@@ -1567,20 +1595,6 @@ Updates to example must be made at:
# Gitaly
gitaly['enable'] = true
- # Make Gitaly accept connections on all network interfaces. You must use
- # firewalls to restrict access to this address/port.
- # Comment out following line if you only want to support TLS connections
- gitaly['listen_addr'] = "0.0.0.0:8075"
-
- # Gitaly Auth Token
- # Should be the same as praefect_internal_token
- gitaly['auth_token'] = '<praefect_internal_token>'
-
- # Gitaly Pack-objects cache
- # Recommended to be enabled for improved performance but can notably increase disk I/O
- # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info
- gitaly['pack_objects_cache_enabled'] = true
-
# Configure the Consul agent
consul['enable'] = true
## Enable service discovery for Prometheus
@@ -1595,9 +1609,29 @@ Updates to example must be made at:
retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13),
}
- # Set the network addresses that the exporters will listen on for monitoring
+ # Set the network address that the node exporter will listen on for monitoring
node_exporter['listen_address'] = '0.0.0.0:9100'
- gitaly['prometheus_listen_addr'] = '0.0.0.0:9236'
+
+ gitaly['configuration'] = {
+ # Make Gitaly accept connections on all network interfaces. You must use
+ # firewalls to restrict access to this address/port.
+ # Comment out following line if you only want to support TLS connections
+ listen_addr: '0.0.0.0:8075',
+ # Set the network address that Gitaly will listen on for monitoring
+ prometheus_listen_addr: '0.0.0.0:9236',
+ auth: {
+ # Gitaly Auth Token
+ # Should be the same as praefect_internal_token
+ token: '<praefect_internal_token>',
+ },
+ pack_objects_cache: {
+ # Gitaly Pack-objects cache
+ # Recommended to be enabled for improved performance but can notably increase disk I/O
+ # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info
+ enabled: true,
+ },
+ }
+
#
# END user configuration
```
@@ -1606,31 +1640,43 @@ Updates to example must be made at:
- On Gitaly node 1:
```ruby
- git_data_dirs({
- "gitaly-1" => {
- "path" => "/var/opt/gitlab/git-data"
- }
- })
+ gitaly['configuration'] = {
+ # ...
+ storage: [
+ {
+ name: 'gitaly-1',
+ path: '/var/opt/gitlab/git-data',
+ },
+ ],
+ }
```
- On Gitaly node 2:
```ruby
- git_data_dirs({
- "gitaly-2" => {
- "path" => "/var/opt/gitlab/git-data"
- }
- })
+ gitaly['configuration'] = {
+ # ...
+ storage: [
+ {
+ name: 'gitaly-2',
+ path: '/var/opt/gitlab/git-data',
+ },
+ ],
+ }
```
- On Gitaly node 3:
```ruby
- git_data_dirs({
- "gitaly-3" => {
- "path" => "/var/opt/gitlab/git-data"
- }
- })
+ gitaly['configuration'] = {
+ # ...
+ storage: [
+ {
+ name: 'gitaly-3',
+ path: '/var/opt/gitlab/git-data',
+ },
+ ],
+ }
```
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
@@ -1650,7 +1696,7 @@ for secure connections, you must:
Additionally the certificate, or its certificate authority, must be installed on all Gitaly servers
and on all Praefect clients that communicate with it following the procedure described in
-[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) (and repeated below).
+[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl/index.html#install-custom-public-certificates) (and repeated below).
Note the following:
@@ -1659,7 +1705,7 @@ Note the following:
- You can configure Praefect servers with both an unencrypted listening address
`listen_addr` and an encrypted listening address `tls_listen_addr` at the same time.
This allows you to do a gradual transition from unencrypted to encrypted traffic, if
- necessary. To disable the unencrypted listener, set `praefect['listen_addr'] = nil`.
+ necessary. To disable the unencrypted listener, set `praefect['configuration'][:listen_addr] = nil`.
- The Internal Load Balancer will also access to the certificates and need to be configured
to allow for TLS passthrough.
Refer to the load balancers documentation on how to configure this.
@@ -1681,9 +1727,15 @@ To configure Praefect with TLS:
1. Edit `/etc/gitlab/gitlab.rb` and add:
```ruby
- praefect['tls_listen_addr'] = "0.0.0.0:3305"
- praefect['certificate_path'] = "/etc/gitlab/ssl/cert.pem"
- praefect['key_path'] = "/etc/gitlab/ssl/key.pem"
+ praefect['configuration'] = {
+ # ...
+ tls_listen_addr: '0.0.0.0:3305',
+ tls: {
+ # ...
+ certificate_path: '/etc/gitlab/ssl/cert.pem',
+ key_path: '/etc/gitlab/ssl/key.pem',
+ },
+ }
```
1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure).
@@ -1764,7 +1816,7 @@ Updates to example must be made at:
# Redis
## Redis connection details
- ## First cluster that will host the cache
+ ## First cluster that will host the cache data
gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache'
gitlab_rails['redis_cache_sentinels'] = [
@@ -1773,22 +1825,11 @@ Updates to example must be made at:
{host: '10.6.0.53', port: 26379},
]
- ## Second cluster that will host the persistent queues, shared state, and actioncable
- gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
- gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
- gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
+ ## Second cluster that hosts all other persistent data
+ redis['master_name'] = 'gitlab-redis-persistent'
+ redis['master_password'] = '<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>'
- gitlab_rails['redis_queues_sentinels'] = [
- {host: '10.6.0.61', port: 26379},
- {host: '10.6.0.62', port: 26379},
- {host: '10.6.0.63', port: 26379},
- ]
- gitlab_rails['redis_shared_state_sentinels'] = [
- {host: '10.6.0.61', port: 26379},
- {host: '10.6.0.62', port: 26379},
- {host: '10.6.0.63', port: 26379},
- ]
- gitlab_rails['redis_actioncable_sentinels'] = [
+ gitlab_rails['redis_sentinels'] = [
{host: '10.6.0.61', port: 26379},
{host: '10.6.0.62', port: 26379},
{host: '10.6.0.63', port: 26379},
@@ -1958,7 +1999,7 @@ On each node perform the following:
gitlab_rails['auto_migrate'] = false
## Redis connection details
- ## First cluster that will host the cache
+ ## First cluster that will host the cache data
gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache'
gitlab_rails['redis_cache_sentinels'] = [
@@ -1967,22 +2008,11 @@ On each node perform the following:
{host: '10.6.0.53', port: 26379},
]
- ## Second cluster that will host the persistent queues, shared state, and actionable
- gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
- gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
- gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
+ ## Second cluster that hosts all other persistent data
+ redis['master_name'] = 'gitlab-redis-persistent'
+ redis['master_password'] = '<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>'
- gitlab_rails['redis_queues_sentinels'] = [
- {host: '10.6.0.61', port: 26379},
- {host: '10.6.0.62', port: 26379},
- {host: '10.6.0.63', port: 26379},
- ]
- gitlab_rails['redis_shared_state_sentinels'] = [
- {host: '10.6.0.61', port: 26379},
- {host: '10.6.0.62', port: 26379},
- {host: '10.6.0.63', port: 26379},
- ]
- gitlab_rails['redis_actioncable_sentinels'] = [
+ gitlab_rails['redis_sentinels'] = [
{host: '10.6.0.61', port: 26379},
{host: '10.6.0.62', port: 26379},
{host: '10.6.0.63', port: 26379},
@@ -2076,7 +2106,7 @@ On each node perform the following:
When you specify `https` in the `external_url`, as in the previous example,
GitLab expects that the SSL certificates are in `/etc/gitlab/ssl/`. If the
certificates aren't present, NGINX will fail to start. For more information, see
-the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html).
+the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html).
### GitLab Rails post-configuration
@@ -2088,11 +2118,9 @@ the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html).
sudo gitlab-rake gitlab:db:configure
```
- If you encounter a `rake aborted!` error message stating that PgBouncer is
- failing to connect to PostgreSQL, it may be that your PgBouncer node's IP
- address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb`
- on your database nodes. Before proceeding, see
- [PgBouncer error `ERROR: pgbouncer cannot connect to server`](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server).
+ Note that this requires the Rails node to be configured to connect to the primary database
+ directly, [bypassing PgBouncer](../postgresql/pgbouncer.md#procedure-for-bypassing-pgbouncer).
+ After migrations have completed, you must configure the node to pass through PgBouncer again.
1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md).
@@ -2187,15 +2215,15 @@ and scalable.
There are two ways of specifying object storage configuration in GitLab:
-- [Consolidated form](../object_storage.md#consolidated-object-storage-configuration): A single credential is
+- [Consolidated form](../object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form): A single credential is
shared by all supported object types.
-- [Storage-specific form](../object_storage.md#storage-specific-configuration): Every object defines its
- own object storage [connection and configuration](../object_storage.md#connection-settings).
+- [Storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form): Every object defines its
+ own object storage [connection and configuration](../object_storage.md#configure-the-connection-settings).
The consolidated form is used in the following examples when available.
NOTE:
-When using the [storage-specific form](../object_storage.md#storage-specific-configuration)
+When using the [storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form)
in GitLab 14.x and earlier, you should enable [direct upload mode](../../development/uploads/index.md#direct-upload).
The previous [background upload](../../development/uploads/index.md#direct-upload) mode,
which was deprecated in 14.9, requires shared storage such as NFS.
@@ -2217,9 +2245,9 @@ GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily
While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs.
-## Configure Advanced Search
+## Configure advanced search
-You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md)
+You can leverage Elasticsearch and [enable advanced search](../../integration/advanced_search/elasticsearch.md)
for faster, more advanced code search across your entire GitLab instance.
Elasticsearch cluster design and requirements are dependent on your specific
diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md
index 691f71289c3..82087c91910 100644
--- a/doc/administration/reference_architectures/5k_users.md
+++ b/doc/administration/reference_architectures/5k_users.md
@@ -37,8 +37,8 @@ costly-to-operate environment by using the
| Gitaly<sup>5 6</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` |
| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` |
| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` |
-| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` |
-| GitLab Rails | 3 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | `c5.4xlarge` |
+| Sidekiq<sup>7</sup> | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` |
+| GitLab Rails<sup>7</sup> | 3 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | `c5.4xlarge` |
| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` |
| Object storage<sup>4</sup> | - | - | - | - |
@@ -59,6 +59,8 @@ costly-to-operate environment by using the
6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large
repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to
[Large repositories](index.md#large-repositories) for more information.
+7. Can be placed in Auto Scaling Groups (ASGs) as the component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes).
+ However, for GitLab Rails certain processes like [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md) should be run on only one node.
<!-- markdownlint-enable MD029 -->
NOTE:
@@ -173,7 +175,7 @@ To set up GitLab and its components to accommodate up to 5,000 users:
environment.
1. [Configure the object storage](#configure-the-object-storage)
used for shared data objects.
-1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster,
+1. [Configure advanced search](#configure-advanced-search) (optional) for faster,
more advanced code search across your entire GitLab instance.
The servers start on the same 10.6.0.0/24 private network range, and can
@@ -318,7 +320,7 @@ Configure your load balancer to pass connections on port 443 as `TCP` rather
than `HTTP(S)` protocol. This will pass the connection to the application node's
NGINX service untouched. NGINX will have the SSL certificate and listen on port 443.
-See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html)
+See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html)
for details on managing SSL certificates and configuring NGINX.
#### Load balancer terminates SSL without backend SSL
@@ -329,7 +331,7 @@ terminating SSL.
Since communication between the load balancer and GitLab will not be secure,
there is some additional configuration needed. See the
-[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination)
+[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination)
for details.
#### Load balancer terminates SSL with backend SSL
@@ -342,7 +344,7 @@ Traffic will also be secure between the load balancers and NGINX in this
scenario. There is no need to add configuration for proxied SSL since the
connection will be secure all the way. However, configuration will need to be
added to GitLab to configure SSL certificates. See
-the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html)
+the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html)
for details on managing SSL certificates and configuring NGINX.
<div align="right">
@@ -450,10 +452,7 @@ to be used with GitLab. The following IPs are used as an example:
Managed Redis from cloud providers such as AWS ElastiCache works. If these
services support high availability, be sure it is **not** the Redis Cluster type.
-Redis version 5.0 or higher is required, as this is what ships with
-Omnibus GitLab packages starting with GitLab 13.0. Older Redis versions
-do not support an optional count argument to SPOP which is now required for
-[Merge Trains](../../ci/pipelines/merge_trains.md).
+Because Omnibus GitLab packages ship with Redis 6.0 or later, Redis 6.0 or later is required. Older Redis versions have reached end-of-life.
Note the Redis node's IP address or hostname, port, and password (if required).
These are necessary when configuring the
@@ -637,8 +636,13 @@ are supported and can be added if needed.
## Configure Consul and Sentinel
-Now that the Redis servers are all set up, let's configure the Sentinel
-servers. The following IPs are used as an example:
+Now that the Redis servers are all set up, let's configure the Consul + Sentinel
+servers.
+
+NOTE:
+Consul and Redis Sentinel must be deployed in an odd number of 3 nodes or later. This is to ensure the nodes can take votes as part of a quorum.
+
+The following IPs will be used as an example:
- `10.6.0.11`: Consul/Sentinel 1
- `10.6.0.12`: Consul/Sentinel 2
@@ -1140,7 +1144,7 @@ For more advanced setups refer to the [standalone Gitaly Cluster documentation](
Praefect, the routing and transaction manager for Gitaly Cluster, requires its own database server to store data on Gitaly Cluster status.
If you want to have a highly available setup, Praefect requires a third-party PostgreSQL database.
-A built-in solution is being [worked on](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919).
+A built-in solution is being [worked on](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7292).
#### Praefect non-HA PostgreSQL standalone using Omnibus GitLab
@@ -1285,6 +1289,9 @@ This is how this would work with a Omnibus GitLab PostgreSQL setup:
Praefect is the router and transaction manager for Gitaly Cluster and all connections to Gitaly go through
it. This section details how to configure it.
+NOTE:
+Praefect must be deployed in an odd number of 3 nodes or later. This is to ensure the nodes can take votes as part of a quorum.
+
Praefect requires several secret tokens to secure communications across the Cluster:
- `<praefect_external_token>`: Used for repositories hosted on your Gitaly cluster and can only be accessed by Gitaly clients that carry this token.
@@ -1338,7 +1345,6 @@ Updates to example must be made at:
# Praefect Configuration
praefect['enable'] = true
- praefect['listen_addr'] = '0.0.0.0:2305'
# Prevent database migrations from running on upgrade automatically
praefect['auto_migrate'] = false
@@ -1347,51 +1353,69 @@ Updates to example must be made at:
# Configure the Consul agent
consul['enable'] = true
## Enable service discovery for Prometheus
- consul['monitoring_service_discovery'] = true
+ consul['monitoring_service_discovery'] = true
# START user configuration
# Please set the real values as explained in Required Information section
#
- # Praefect External Token
- # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster
- praefect['auth_token'] = '<praefect_external_token>'
-
- # Praefect Database Settings
- praefect['database_host'] = '10.6.0.141'
- praefect['database_port'] = 5432
- # `no_proxy` settings must always be a direct connection for caching
- praefect['database_direct_host'] = '10.6.0.141'
- praefect['database_direct_port'] = 5432
- praefect['database_dbname'] = 'praefect_production'
- praefect['database_user'] = 'praefect'
- praefect['database_password'] = '<praefect_postgresql_password>'
-
- # Praefect Virtual Storage config
- # Name of storage hash must match storage name in git_data_dirs on GitLab
- # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1')
- praefect['virtual_storages'] = {
- 'default' => {
- 'nodes' => {
- 'gitaly-1' => {
- 'address' => 'tcp://10.6.0.91:8075',
- 'token' => '<praefect_internal_token>'
- },
- 'gitaly-2' => {
- 'address' => 'tcp://10.6.0.92:8075',
- 'token' => '<praefect_internal_token>'
- },
- 'gitaly-3' => {
- 'address' => 'tcp://10.6.0.93:8075',
- 'token' => '<praefect_internal_token>'
+ praefect['configuration'] = {
+ # ...
+ listen_addr: '0.0.0.0:2305',
+ auth: {
+ # ...
+ #
+ # Praefect External Token
+ # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster
+ token: '<praefect_external_token>',
+ },
+ # Praefect Database Settings
+ database: {
+ # ...
+ host: '10.6.0.141',
+ port: 5432,
+ # `no_proxy` settings must always be a direct connection for caching
+ session_pooled: {
+ # ...
+ host: '10.6.0.141',
+ port: 5432,
+ dbname: 'praefect_production',
+ user: 'praefect',
+ password: '<praefect_postgresql_password>',
+ },
+ },
+ # Praefect Virtual Storage config
+ # Name of storage hash must match storage name in git_data_dirs on GitLab
+ # server ('praefect') and in gitaly['configuration'][:storage] on Gitaly nodes ('gitaly-1')
+ virtual_storage: [
+ {
+ # ...
+ name: 'default',
+ node: [
+ {
+ storage: 'gitaly-1',
+ address: 'tcp://10.6.0.91:8075',
+ token: '<praefect_internal_token>'
+ },
+ {
+ storage: 'gitaly-2',
+ address: 'tcp://10.6.0.92:8075',
+ token: '<praefect_internal_token>'
+ },
+ {
+ storage: 'gitaly-3',
+ address: 'tcp://10.6.0.93:8075',
+ token: '<praefect_internal_token>'
+ },
+ ],
},
- }
- }
+ ],
+ # Set the network address Praefect will listen on for monitoring
+ prometheus_listen_addr: '0.0.0.0:9652',
}
- # Set the network addresses that the exporters will listen on for monitoring
+ # Set the network address the node exporter will listen on for monitoring
node_exporter['listen_address'] = '0.0.0.0:9100'
- praefect['prometheus_listen_addr'] = '0.0.0.0:9652'
## The IPs of the Consul server nodes
## You can also use FQDNs and intermix them with IPs
@@ -1442,9 +1466,7 @@ Due to Gitaly having notable input and output requirements, we strongly
recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs
should have a throughput of at least 8,000
input/output operations per second (IOPS) for read operations and 2,000 IOPS for
-write operations. These IOPS values are initial recommendations, and may be
-adjusted to greater or lesser values depending on the scale of your
-environment's workload. If you're running the environment on a Cloud provider,
+write operations. If you're running the environment on a Cloud provider,
refer to their documentation about how to configure IOPS correctly.
Gitaly servers must not be exposed to the public internet, as Gitaly's network
@@ -1454,7 +1476,7 @@ to restrict access to the Gitaly server. Another option is to
For configuring Gitaly you should note the following:
-- `git_data_dirs` should be configured to reflect the storage path for the specific Gitaly node
+- `gitaly['configuration'][:storage]` should be configured to reflect the storage path for the specific Gitaly node
- `auth_token` should be the same as `praefect_internal_token`
The following IPs are used as an example:
@@ -1503,20 +1525,6 @@ Updates to example must be made at:
# Gitaly
gitaly['enable'] = true
- # Make Gitaly accept connections on all network interfaces. You must use
- # firewalls to restrict access to this address/port.
- # Comment out following line if you only want to support TLS connections
- gitaly['listen_addr'] = "0.0.0.0:8075"
-
- # Gitaly Auth Token
- # Should be the same as praefect_internal_token
- gitaly['auth_token'] = '<praefect_internal_token>'
-
- # Gitaly Pack-objects cache
- # Recommended to be enabled for improved performance but can notably increase disk I/O
- # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info
- gitaly['pack_objects_cache_enabled'] = true
-
# Configure the Consul agent
consul['enable'] = true
## Enable service discovery for Prometheus
@@ -1531,9 +1539,29 @@ Updates to example must be made at:
retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13),
}
- # Set the network addresses that the exporters will listen on for monitoring
+ # Set the network address that the node exporter will listen on for monitoring
node_exporter['listen_address'] = '0.0.0.0:9100'
- gitaly['prometheus_listen_addr'] = '0.0.0.0:9236'
+
+ gitaly['configuration'] = {
+ # Make Gitaly accept connections on all network interfaces. You must use
+ # firewalls to restrict access to this address/port.
+ # Comment out following line if you only want to support TLS connections
+ listen_addr: '0.0.0.0:8075',
+ # Set the network address that Gitaly will listen on for monitoring
+ prometheus_listen_addr: '0.0.0.0:9236',
+ auth: {
+ # Gitaly Auth Token
+ # Should be the same as praefect_internal_token
+ token: '<praefect_internal_token>',
+ },
+ pack_objects_cache: {
+ # Gitaly Pack-objects cache
+ # Recommended to be enabled for improved performance but can notably increase disk I/O
+ # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info
+ enabled: true,
+ },
+ }
+
#
# END user configuration
```
@@ -1542,31 +1570,43 @@ Updates to example must be made at:
- On Gitaly node 1:
```ruby
- git_data_dirs({
- "gitaly-1" => {
- "path" => "/var/opt/gitlab/git-data"
- }
- })
+ gitaly['configuration'] = {
+ # ...
+ storage: [
+ {
+ name: 'gitaly-1',
+ path: '/var/opt/gitlab/git-data',
+ },
+ ],
+ }
```
- On Gitaly node 2:
```ruby
- git_data_dirs({
- "gitaly-2" => {
- "path" => "/var/opt/gitlab/git-data"
- }
- })
+ gitaly['configuration'] = {
+ # ...
+ storage: [
+ {
+ name: 'gitaly-2',
+ path: '/var/opt/gitlab/git-data',
+ },
+ ],
+ }
```
- On Gitaly node 3:
```ruby
- git_data_dirs({
- "gitaly-3" => {
- "path" => "/var/opt/gitlab/git-data"
- }
- })
+ gitaly['configuration'] = {
+ # ...
+ storage: [
+ {
+ name: 'gitaly-3',
+ path: '/var/opt/gitlab/git-data',
+ },
+ ],
+ }
```
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
@@ -1586,7 +1626,7 @@ for secure connections, you must:
Additionally the certificate, or its certificate authority, must be installed on all Gitaly servers
and on all Praefect clients that communicate with it following the procedure described in
-[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) (and repeated below).
+[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl/index.html#install-custom-public-certificates) (and repeated below).
Note the following:
@@ -1595,7 +1635,7 @@ Note the following:
- You can configure Praefect servers with both an unencrypted listening address
`listen_addr` and an encrypted listening address `tls_listen_addr` at the same time.
This allows you to do a gradual transition from unencrypted to encrypted traffic, if
- necessary. To disable the unencrypted listener, set `praefect['listen_addr'] = nil`.
+ necessary. To disable the unencrypted listener, set `praefect['configuration'][:listen_addr] = nil`.
- The Internal Load Balancer will also access to the certificates and need to be configured
to allow for TLS passthrough.
Refer to the load balancers documentation on how to configure this.
@@ -1617,9 +1657,15 @@ To configure Praefect with TLS:
1. Edit `/etc/gitlab/gitlab.rb` and add:
```ruby
- praefect['tls_listen_addr'] = "0.0.0.0:3305"
- praefect['certificate_path'] = "/etc/gitlab/ssl/cert.pem"
- praefect['key_path'] = "/etc/gitlab/ssl/key.pem"
+ praefect['configuration'] = {
+ # ...
+ tls_listen_addr: '0.0.0.0:3305',
+ tls: {
+ # ...
+ certificate_path: '/etc/gitlab/ssl/cert.pem',
+ key_path: '/etc/gitlab/ssl/key.pem',
+ },
+ }
```
1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure).
@@ -2017,7 +2063,7 @@ On each node perform the following:
When you specify `https` in the `external_url`, as in the previous example,
GitLab expects that the SSL certificates are in `/etc/gitlab/ssl/`. If the
certificates aren't present, NGINX fails to start. For more information, see
-the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html).
+the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html).
### GitLab Rails post-configuration
@@ -2027,11 +2073,9 @@ the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html).
gitlab-rake gitlab:db:configure
```
- If you encounter a `rake aborted!` error message stating that PgBouncer is
- failing to connect to PostgreSQL, it may be that your PgBouncer node's IP
- address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb`
- on your database nodes. Before proceeding, see
- [PgBouncer error `ERROR: pgbouncer cannot connect to server`](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server).
+ Note that this requires the Rails node to be configured to connect to the primary database
+ directly, [bypassing PgBouncer](../postgresql/pgbouncer.md#procedure-for-bypassing-pgbouncer).
+ After migrations have completed, you must configure the node to pass through PgBouncer again.
1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md).
@@ -2134,15 +2178,15 @@ and scalable.
There are two ways of specifying object storage configuration in GitLab:
-- [Consolidated form](../object_storage.md#consolidated-object-storage-configuration): A single credential is
+- [Consolidated form](../object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form): A single credential is
shared by all supported object types.
-- [Storage-specific form](../object_storage.md#storage-specific-configuration): Every object defines its
- own object storage [connection and configuration](../object_storage.md#connection-settings).
+- [Storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form): Every object defines its
+ own object storage [connection and configuration](../object_storage.md#configure-the-connection-settings).
The consolidated form is used in the following examples when available.
NOTE:
-When using the [storage-specific form](../object_storage.md#storage-specific-configuration)
+When using the [storage-specific form](../object_storage.md#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form)
in GitLab 14.x and earlier, you should enable [direct upload mode](../../development/uploads/index.md#direct-upload).
The previous [background upload](../../development/uploads/index.md#direct-upload) mode,
which was deprecated in 14.9, requires shared storage such as NFS.
@@ -2164,9 +2208,9 @@ GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily
While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs.
-## Configure Advanced Search
+## Configure advanced search
-You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md)
+You can leverage Elasticsearch and [enable advanced search](../../integration/advanced_search/elasticsearch.md)
for faster, more advanced code search across your entire GitLab instance.
Elasticsearch cluster design and requirements are dependent on your specific
diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md
index 7b01efa183b..4b9c26e8626 100644
--- a/doc/administration/reference_architectures/index.md
+++ b/doc/administration/reference_architectures/index.md
@@ -47,9 +47,9 @@ The following Cloud Native Hybrid reference architectures, where select recommen
The first choice to consider is whether a Self Managed approach is correct for you and your requirements.
-Running any application in production is complex, and the same applies for GitLab. While we aim to make this as smooth as possible, there are still the general complexities. This depends on the design chosen, but typically you'll need to manage all aspects such as hardware, operating systems, networking, storage, security, GitLab itself, and more.
+Running any application in production is complex, and the same applies for GitLab. While we aim to make this as smooth as possible, there are still the general complexities. This depends on the design chosen, but typically you'll need to manage all aspects such as hardware, operating systems, networking, storage, security, GitLab itself, and more. This includes both the initial setup of the environment and the longer term maintenance.
-As such, it's recommended that you have a working knowledge of running applications in production when deciding on going down this route. For users who want a more managed solution it's recommended to instead explore our other offerings such as [GitLab SaaS](../../subscriptions/gitlab_com/index.md) or [GitLab Dedicated](../../subscriptions/gitlab_dedicated/index.md).
+As such, it's recommended that you have a working knowledge of running and maintaining applications in production when deciding on going down this route. If you aren't in this position, our [Professional Services](https://about.gitlab.com/services/#implementation-services) team offers implementation services, but for those who want a more managed solution long term, it's recommended to instead explore our other offerings such as [GitLab SaaS](../../subscriptions/gitlab_com/index.md) or [GitLab Dedicated](../../subscriptions/gitlab_dedicated/index.md).
## Deciding which architecture to use
@@ -72,15 +72,13 @@ With standalone setups, especially single node environments, there are [various
### High Availability (HA)
-High Availability ensures every component in the GitLab setup can handle failures through various mechanisms. To achieve this however is complex, and the environments required can be sizable.
+High Availability ensures every component in the GitLab setup can handle failures through various mechanisms. However, to achieve this is complex and the environments required can be sizable.
For environments serving 3,000 or more users we generally recommend that a HA strategy is used as at this level outages have a bigger impact against more users. All the architectures in this range have HA built in by design for this reason.
-For users who still need to have HA for a lower number of users this can also be achieved with an adjusted [3K architecture](3k_users.md#supported-modifications-for-lower-user-counts-ha).
-
#### Do you need High Availability (HA)?
-As mentioned above, achieving HA does come at a cost. The environment's required are sizable as each component needs to be multiplied, which comes with additional actual and maintenance costs.
+As mentioned above, achieving HA does come at a cost. The environment requirements are sizable as each component needs to be multiplied, which comes with additional actual and maintenance costs.
For a lot of our customers with fewer than 3,000 users, we've found a backup strategy is sufficient and even preferable. While this does have a slower recovery time, it also means you have a much smaller architecture and less maintenance costs as a result.
@@ -89,13 +87,17 @@ In general then, we'd only recommend you employ HA in the following scenarios:
- When you have 3,000 or more users.
- When GitLab being down would critically impact your workflow.
+#### Scaled-down High Availability (HA) approaches
+
+If you still need to have HA for a lower number of users, this can be achieved with an adjusted [3K architecture](3k_users.md#supported-modifications-for-lower-user-counts-ha).
+
#### Zero Downtime Upgrades
[Zero Downtime Upgrades](../../update/zero_downtime.md) are available for standard Reference Architecture environments with HA (Cloud Native Hybrid is not supported at this time). This allows for an environment to stay up during an upgrade, but the process is more complex as a result and has some limitations as detailed in the documentation.
-When going through this process it's worth noting that there may still be brief moments of downtime when the HA mechanisms tale effect.
+When going through this process it's worth noting that there may still be brief moments of downtime when the HA mechanisms take effect.
-In most cases the downtime required for doing an upgrade in general shouldn't be substantial, so this is only recommended if it's a key requirement for you.
+In most cases the downtime required for doing an upgrade shouldn't be substantial, so this is only recommended if it's a key requirement for you.
### Cloud Native Hybrid (Kubernetes HA)
@@ -163,7 +165,7 @@ Before implementing a reference architecture, refer to the following requirement
These reference architectures were built and tested on Google Cloud Platform (GCP) using the
[Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms)
-CPU platform as a baseline ([Sysbench benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks)).
+CPU platform as a lowest common denominator baseline ([Sysbench benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks)).
Newer, similarly-sized CPUs are supported and may have improved performance as a result. For Omnibus GitLab environments,
ARM-based equivalents are also supported.
@@ -237,10 +239,10 @@ for more information and guidance.
[Praefect requires its own database server](../gitaly/praefect.md#postgresql) and
that to achieve full High Availability, a third-party PostgreSQL database solution is required.
-We hope to offer a built in solutions for these restrictions in the future but, in the meantime, a non HA PostgreSQL server
-can be set up using Omnibus GitLab, the specifications reflect. Refer to the following issues for more information:
+We hope to offer a built-in solution for these restrictions in the future. In the meantime, a non-HA PostgreSQL server
+can be set up using Omnibus GitLab as the specifications reflect. Refer to the following issues for more information:
-- [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919).
+- [`omnibus-gitlab#7292`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7292).
- [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398).
## Recommended cloud providers and services
@@ -324,10 +326,13 @@ When selecting a database service, it should run a standard, performant, and [su
- Read Replicas for [Database Load Balancing](../postgresql/database_load_balancing.md).
- Cross Region replication for [GitLab Geo](../geo/index.md).
-Several cloud provider services are known not to support the above or have been found to have other issues and aren't recommended:
+#### Unsupported database services
+
+Several database cloud provider services are known not to support the above or have been found to have other issues and aren't recommended:
- [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is incompatible and not supported. See [14.4.0](../../update/index.md#1440) for more details.
-- [Azure Database for PostgreSQL Single Server](https://azure.microsoft.com/en-gb/products/postgresql/#overview) (Single / Flexible) is **strongly not recommended** for use due to notable performance / stability issues or missing functionality. See [Recommendation Notes for Azure](#recommendation-notes-for-azure) for more details.
+- [Azure Database for PostgreSQL Single Server](https://azure.microsoft.com/en-gb/products/postgresql/#overview) (Single / Flexible) is not supported for use due to notable performance / stability issues or missing functionality. See [Recommendation Notes for Azure](#recommendation-notes-for-azure) for more details.
+- Azure Database for PostgreSQL Flexible Server uses Microsoft Azure Active Directory (Azure AD) as authentication mechanism, which is incompatible with GitLab database integration.
- [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo.
- [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported.
@@ -337,10 +342,55 @@ Due to performance issues that we found with several key Azure services, we only
In addition to the above, you should be aware of the additional specific guidance for Azure:
-- **We outright strongly do not recommend [Azure Database for PostgreSQL Single Server](https://learn.microsoft.com/en-us/azure/postgresql/single-server/overview-single-server)** specifically due to significant performance and stability issues found. **For GitLab 14.0 and higher the service is not supported** due to it only supporting up to PostgreSQL 11.
- - A new service, [Azure Database for PostgreSQL Flexible Server](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/) has been released. [Internal testing](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/91) has shown that it does look to perform as expected, but this hasn't been validated in production, so generally isn't recommended at this time. Additionally, as it's a new service, you may find that it's missing some functionality depending on your requirements.
+- [Azure Database for PostgreSQL Single Server](https://azure.microsoft.com/en-gb/products/postgresql/#overview) (Single / Flexible) is not supported for use due to notable performance / stability issues or missing functionality.
+- A new service, [Azure Database for PostgreSQL Flexible Server](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/) has been released. [Internal testing](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/91) has shown that it does look to perform as expected, but this hasn't been validated in production, so generally isn't recommended at this time. Additionally, as it's a new service, you may find that it's missing some functionality depending on your requirements.
- [Azure Blob Storage](https://azure.microsoft.com/en-gb/products/storage/blobs/) has been found to have [performance limits that can impact production use at certain times](https://gitlab.com/gitlab-org/gitlab/-/issues/344861). However, this has only been seen in our largest architectures (25k+) so far.
+## Deviating from the suggested reference architectures
+
+As a general guideline, the further away you move from the reference architectures,
+the harder it is to get support for it. With any deviation, you're introducing
+a layer of complexity that adds challenges to finding out where potential
+issues might lie.
+
+The reference architectures use the official GitLab Linux packages (Omnibus
+GitLab) or [Helm Charts](https://docs.gitlab.com/charts/) to install and configure the various components. The components are
+installed on separate machines (virtualized or bare metal), with machine hardware
+requirements listed in the "Configuration" column and equivalent VM standard sizes listed
+in GCP/AWS/Azure columns of each [available reference architecture](#available-reference-architectures).
+
+Running components on Docker (including Docker Compose) with the same specs should be fine, as Docker is well known in terms of support.
+However, it is still an additional layer and may still add some support complexities, such as not being able to run `strace` easily in containers.
+
+### Unsupported designs
+
+While we endeavour to try and have a good range of support for GitLab environment designs, there are certain approaches we know definitively not to work, and as a result are not supported. Those approaches are detailed in the following sections.
+
+#### Stateful components in Kubernetes
+
+[Running stateful components in Kubernetes, such as Gitaly Cluster, is not supported](https://docs.gitlab.com/charts/installation/#configure-the-helm-chart-to-use-external-stateful-data).
+
+Gitaly Cluster is only supported to be run on VMs as Git itself doesn't match well with the Kubernetes design and attempting to run it can lead to significant and complex issues.
+[Refer to epic 6127 for more information](https://gitlab.com/groups/gitlab-org/-/epics/6127).
+
+This also applies to other third-party stateful components such as Postgres and Redis, but you can explore other third-party solutions for those components if desired such as supported Cloud Provider services unless called out specifically as unsupported.
+
+#### Autoscaling of stateful nodes
+
+As a general guidance, only _stateless_ components of GitLab can be run in Autoscaling groups, namely GitLab Rails
+and Sidekiq.
+
+Other components that have state, such as Gitaly, are not supported in this fashion (for more information, see [issue 2997](https://gitlab.com/gitlab-org/gitaly/-/issues/2997)).
+
+This also applies to other third-party stateful components such as Postgres and Redis, but you can explore other third-party solutions for those components if desired such as supported Cloud Provider services unless called out specifically as unsupported.
+
+#### Spreading one environment over multiple data centers
+
+Deploying one GitLab environment over multiple data centers is not supported due to potential split brain edge cases
+if a data center were to go down. For example, several components of the GitLab setup, namely Consul, Redis Sentinel and Praefect require an odd number quorum to function correctly and splitting over multiple data centers can impact this notably.
+
+For deploying GitLab over multiple data centers or regions we offer [GitLab Geo](https://about.gitlab.com/solutions/geo/) as a comprehensive solution.
+
## Validation and test results
The [Quality Engineering team](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/)
@@ -427,34 +477,34 @@ table.test-coverage th {
<tr>
<th scope="row">1k</th>
<td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/1k">Weekly</a></td>
- <td></td>
- <td></td>
- <td></td>
- <td></td>
+ <td style="background-color:lightgrey"></td>
+ <td style="background-color:lightgrey"></td>
+ <td style="background-color:lightgrey"></td>
+ <td style="background-color:lightgrey"></td>
</tr>
<tr>
<th scope="row">2k</th>
<td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/2k">Weekly</a></td>
- <td></td>
- <td></td>
- <td></td>
- <td></td>
+ <td style="background-color:lightgrey"></td>
+ <td style="background-color:lightgrey"></td>
+ <td style="background-color:lightgrey"></td>
+ <td><i>Planned</i></td>
</tr>
<tr>
<th scope="row">3k</th>
<td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/3k">Weekly</a></td>
- <td></td>
- <td></td>
+ <td style="background-color:lightgrey"></td>
+ <td style="background-color:lightgrey"></td>
<td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/3k_hybrid_aws_services">Weekly</a></td>
- <td></td>
+ <td style="background-color:lightgrey"></td>
</tr>
<tr>
<th scope="row">5k</th>
<td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/5k">Weekly</a></td>
- <td></td>
- <td></td>
- <td></td>
- <td></td>
+ <td style="background-color:lightgrey"></td>
+ <td style="background-color:lightgrey"></td>
+ <td style="background-color:lightgrey"></td>
+ <td style="background-color:lightgrey"></td>
</tr>
<tr>
<th scope="row">10k</th>
@@ -462,29 +512,33 @@ table.test-coverage th {
<td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k_hybrid">Weekly</a></td>
<td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k_aws">Weekly</a></td>
<td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k_hybrid_aws_services">Weekly</a></td>
- <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/10k">Ad-Hoc</a></td>
+ <td style="background-color:lightgrey"></td>
</tr>
<tr>
<th scope="row">25k</th>
<td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/25k">Weekly</a></td>
- <td></td>
- <td></td>
- <td></td>
- <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/25k">Ad-Hoc</a></td>
+ <td style="background-color:lightgrey"></td>
+ <td style="background-color:lightgrey"></td>
+ <td style="background-color:lightgrey"></td>
+ <td style="background-color:lightgrey"></td>
</tr>
<tr>
<th scope="row">50k</th>
<td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/50k">Weekly</a></td>
- <td></td>
- <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/50k">Ad-Hoc (inc Cloud Services)</a></td>
- <td></td>
- <td></td>
+ <td style="background-color:lightgrey"></td>
+ <td style="background-color:lightgrey"></td>
+ <td style="background-color:lightgrey"></td>
+ <td style="background-color:lightgrey"></td>
</tr>
</table>
## Cost to run
-The following table details the cost to run the different reference architectures across GCP, AWS, and Azure. Bare-metal costs are not included here as it varies widely depending on each customer configuration.
+As a starting point, the following table details initial costs for the different reference architectures across GCP, AWS, and Azure via Omnibus.
+
+NOTE:
+Due to the nature of Cloud Native Hybrid, it's not possible to give a static cost calculation.
+Bare-metal costs are also not included here as it varies widely depending on each configuration.
<table class="test-coverage">
<col>
@@ -492,91 +546,93 @@ The following table details the cost to run the different reference architecture
<colgroup span="2"></colgroup>
<tr>
<th rowspan="2">Reference<br/>Architecture</th>
- <th style="text-align: center" colspan="2" scope="colgroup">GCP</th>
- <th style="text-align: center" colspan="2" scope="colgroup">AWS</th>
- <th style="text-align: center" colspan="2" scope="colgroup">Azure</th>
+ <th style="text-align: center" scope="colgroup">GCP</th>
+ <th style="text-align: center" scope="colgroup">AWS</th>
+ <th style="text-align: center" scope="colgroup">Azure</th>
</tr>
<tr>
<th scope="col">Omnibus</th>
- <th scope="col">Cloud Native Hybrid</th>
<th scope="col">Omnibus</th>
- <th scope="col">Cloud Native Hybrid</th>
<th scope="col">Omnibus</th>
</tr>
<tr>
<th scope="row">1k</th>
<td><a href="https://cloud.google.com/products/calculator#id=a6d6a94a-c7dc-4c22-85c4-7c5747f272ed">Calculated cost</a></td>
- <td></td>
<td><a href="https://calculator.aws/#/estimate?id=b51f178f4403b69a63f6eb33ea425f82de3bf249">Calculated cost</a></td>
- <td></td>
<td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=1adf30bef7e34ceba9efa97c4470417b">Calculated cost</a></td>
</tr>
<tr>
<th scope="row">2k</th>
- <td><a href="https://cloud.google.com/products/calculator#id=84d11491-d72a-493c-a16e-650931faa658">Calculated cost</a></td>
- <td></td>
- <td><a href="https://calculator.aws/#/estimate?id=dce36b5cb6ab25211f74e47233d77f58fefb54e2">Calculated cost</a></td>
- <td></td>
- <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=72764902f3854f798407fb03c3de4b6f">Calculated cost</a></td>
+ <td><a href="https://cloud.google.com/products/calculator#id=0d3aff1f-ea3d-43f9-aa59-df49d27c35ca">Calculated cost</a></td>
+ <td><a href="https://calculator.aws/#/estimate?id=3b3e3b81953737132789591d3a5179521943f1c0">Calculated cost</a></td>
+ <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=25f66c35ba454bb98fb4034a8a50bb8c">Calculated cost</a></td>
</tr>
<tr>
<th scope="row">3k</th>
- <td><a href="https://cloud.google.com/products/calculator/#id=ac4838e6-9c40-4a36-ac43-6d1bc1843e08">Calculated cost</a></td>
- <td></td>
- <td><a href="https://calculator.aws/#/estimate?id=b1c5b4e32e990eaeb035a148255132bd28988760">Calculated cost</a></td>
- <td></td>
- <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=0dbfc575051943b9970e5d8ace03680d">Calculated cost</a></td>
+ <td><a href="https://cloud.google.com/products/calculator/#id=15fc2bd9-5b1c-479d-bc46-d5ce096b8107">Calculated cost</a></td>
+ <td><a href="https://calculator.aws/#/estimate?id=7e94eb8712f6845fdeb05e61f459598a91dac3cb">Calculated cost</a></td>
+ <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=24ac11fd947a4985ae9c9a5142649ad3">Calculated cost</a></td>
</tr>
<tr>
<th scope="row">5k</th>
- <td><a href="https://cloud.google.com/products/calculator/#id=8742e8ea-c08f-4e0a-b058-02f3a1c38a2f">Calculated cost</a></td>
- <td></td>
- <td><a href="https://calculator.aws/#/estimate?id=2bf1af883096e6f4c6efddb4f3c35febead7fec2">Calculated cost</a></td>
- <td></td>
- <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=8f618711ffec4b039f1581871ca6a7c9">Calculated cost</a></td>
+ <td><a href="https://cloud.google.com/products/calculator/#id=9a798136-53f2-4c35-be43-8e1e975a6663">Calculated cost</a></td>
+ <td><a href="https://calculator.aws/#/estimate?id=ad4c9db623a214c92d780cd9dff33f444d62cf02">Calculated cost</a></td>
+ <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=bcf23017ddfd40649fdc885cacd08d0c">Calculated cost</a></td>
</tr>
<tr>
<th scope="row">10k</th>
- <td><a href="https://cloud.google.com/products/calculator#id=e77713f6-dc0b-4bb3-bcef-cea904ac8efd">Calculated cost</a></td>
- <td></td>
- <td><a href="https://calculator.aws/#/estimate?id=1d374df13c0f2088d332ab0134f5b1d0f717259e">Calculated cost</a></td>
- <td></td>
- <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=de3da8286dda4d4db1362932bc75410b">Calculated cost</a></td>
+ <td><a href="https://cloud.google.com/products/calculator#id=cbe61840-31a1-487f-88fa-631251c2fde5">Calculated cost</a></td>
+ <td><a href="https://calculator.aws/#/estimate?id=3e2970f919915a6337acea76a9f07655a1ecda4a">Calculated cost</a></td>
+ <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=5748068be4864af6a34efb1cde685fa1">Calculated cost</a></td>
</tr>
<tr>
<th scope="row">25k</th>
- <td><a href="https://cloud.google.com/products/calculator#id=925386e1-c01c-4c0a-8d7d-ebde1824b7b0">Calculated cost</a></td>
- <td></td>
- <td><a href="https://calculator.aws/#/estimate?id=46fe6a6e9256d9b7779fae59fbbfa7e836942b7d">Calculated cost</a></td>
- <td></td>
- <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=69724ebd82914a60857da6a3ace05a64">Calculate cost</a></td>
+ <td><a href="https://cloud.google.com/products/calculator#id=b4b8b587-508a-4433-adc8-dc506bbe924f">Calculated cost</a></td>
+ <td><a href="https://calculator.aws/#/estimate?id=32acaeaa93366110cd5fbf98a66a8a141db7adcb">Calculated cost</a></td>
+ <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=24f878f20ee64b5cb64de459d34c8128">Calculate cost</a></td>
</tr>
<tr>
<th scope="row">50k</th>
- <td><a href="https://cloud.google.com/products/calculator/#id=8006396b-88ee-40cd-a1c8-77cdefa4d3c8">Calculated cost</a></td>
- <td></td>
- <td><a href="https://calculator.aws/#/estimate?id=e15926b1a3c7139e4faf390a3875ff807d2ab91c">Calculated cost</a></td>
- <td></td>
- <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=3f973040ebc14023933d35f576c89846">Calculated cost</a></td>
+ <td><a href="https://cloud.google.com/products/calculator/#id=48b4d817-d6cd-44b8-b069-0ba9a5d123ea">Calculated cost</a></td>
+ <td><a href="https://calculator.aws/#/estimate?id=5a0bba1338e3577d627ec97833dbc80ac9615562">Calculated cost</a></td>
+ <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=4dd065eea2194d70b44d6d897e81f460">Calculated cost</a></td>
</tr>
</table>
-## Deviating from the suggested reference architectures
+## Maintaining a Reference Architecture environment
-As a general guideline, the further away you move from the Reference Architectures,
-the harder it is to get support for it. With any deviation, you're introducing
-a layer of complexity that adds challenges to finding out where potential
-issues might lie.
+Maintaining a Reference Architecture environment is generally the same as any other GitLab environment is generally covered in other sections of this documentation.
-The reference architectures use the official GitLab Linux packages (Omnibus
-GitLab) or [Helm Charts](https://docs.gitlab.com/charts/) to install and configure the various components. The components are
-installed on separate machines (virtualized or bare metal), with machine hardware
-requirements listed in the "Configuration" column and equivalent VM standard sizes listed
-in GCP/AWS/Azure columns of each [available reference architecture](#available-reference-architectures).
+In this section you'll find links to documentation for relevant areas as well as any specific Reference Architecture notes.
-Running components on Docker (including Compose) with the same specs should be fine, as Docker is well known in terms of support.
-However, it is still an additional layer and may still add some support complexities, such as not being able to run `strace` easily in containers.
+### Upgrades
+
+Upgrades for a Reference Architecture environment is the same as any other GitLab environment.
+The main [Upgrade GitLab](../../update/index.md) section has detailed steps on how to approach this.
+
+[Zero-downtime upgrades](#zero-downtime-upgrades) are also available.
+
+NOTE:
+You should upgrade a Reference Architecture in the same order as you created it.
+
+### Scaling an environment
+
+Scaling a GitLab environment is designed to be as seamless as possible.
+
+In terms of the Reference Architectures, you would look to the next size and adjust accordingly.
+Most setups would only need vertical scaling, but there are some specific areas that can be adjusted depending on the setup:
+
+- If you're scaling from a non-HA environment to an HA environment, various components are recommended to be deployed in their HA forms:
+ - Redis to multi-node Redis w/ Redis Sentinel
+ - Postgres to multi-node Postgres w/ Consul + PgBouncer
+ - Gitaly to Gitaly Cluster w/ Praefect
+- From 10k users and higher, Redis is recommended to be split into multiple HA servers as it's single threaded.
+
+Conversely, if you have robust metrics in place that show the environment is over-provisioned you can apply the same process for
+scaling downloads. It's recommended to take an iterative approach when scaling downwards however to ensure there are no issues.
+
+### How to monitor your environment
-Other technologies, like [Docker swarm](https://docs.docker.com/engine/swarm/)
-are not officially supported, but can be implemented at your own risk. In that
-case, GitLab Support is not able to help you.
+To monitor your GitLab environment, you can use the tools
+[bundled with GitLab](../monitoring/index.md), but it's also possible to use third-party
+options if desired.
diff --git a/doc/administration/reply_by_email.md b/doc/administration/reply_by_email.md
index 6e97ffc3b47..b632108b103 100644
--- a/doc/administration/reply_by_email.md
+++ b/doc/administration/reply_by_email.md
@@ -1,6 +1,6 @@
---
-stage: Plan
-group: Certify
+stage: Monitor
+group: Respond
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
@@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w
GitLab can be set up to allow users to comment on issues and merge requests by
replying to notification emails.
-## Requirement
+## Prerequisite
Make sure [incoming email](incoming_email.md) is set up.
diff --git a/doc/administration/repository_checks.md b/doc/administration/repository_checks.md
index 08c1df3d5eb..3f715e451a8 100644
--- a/doc/administration/repository_checks.md
+++ b/doc/administration/repository_checks.md
@@ -9,9 +9,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w
You can use [`git fsck`](https://git-scm.com/docs/git-fsck) to verify the integrity of all data
committed to a repository. GitLab administrators can:
-- Manually trigger this check for a project, using the GitLab UI.
-- Schedule this check to run automatically for all projects.
-- Run this check from the command line.
+- [Manually trigger this check for a project](#check-a-projects-repository-using-gitlab-ui).
+- [Schedule this check](#enable-repository-checks-for-all-projects) to run automatically for all projects.
+- [Run this check from the command line](#run-a-check-using-the-command-line).
- Run a [Rake task](raketasks/check.md#repository-integrity) for checking Git repositories, which can be used to run
`git fsck` against all repositories and generate repository checksums, as a way to compare repositories on different
servers.
@@ -68,9 +68,13 @@ You can run [`git fsck`](https://git-scm.com/docs/git-fsck) using the command li
1. Run the check. For example:
```shell
- sudo -u git /opt/gitlab/embedded/bin/git -C /var/opt/gitlab/git-data/repositories/@hashed/0b/91/0b91...f9.git fsck
+ sudo -u git /opt/gitlab/embedded/bin/git \
+ -C /var/opt/gitlab/git-data/repositories/@hashed/0b/91/0b91...f9.git fsck --no-dangling
```
+ The error `fatal: detected dubious ownership in repository` means you're running the command
+ using the wrong account. For example, `root`.
+
## What to do if a check failed
If a repository check fails, locate the error in the [`repocheck.log` file](logs/index.md#repochecklog) on disk at:
@@ -93,3 +97,26 @@ of date. The `commit-graph` cache is an auxiliary cache and is not required for
While the message can be safely ignored, see the issue [error: Could not read from object database for commit-graph](https://gitlab.com/gitlab-org/gitaly/-/issues/2359)
for more details.
+
+### Dangling commit, tag, or blob messages
+
+Repository check output often includes tags, blobs, and commits that must be pruned:
+
+```plaintext
+dangling tag 5c6886c774b713a43158aae35c4effdb03a3ceca
+dangling blob 3e268c23fcd736db92e89b31d9f267dd4a50ac4b
+dangling commit 919ff61d8d78c2e3ea9a32701dff70ecbefdd1d7
+```
+
+This is common in Git repositories. They're generated by operations like
+force pushing to branches, because this generates a commit in the repository
+that is not longer referred to by a ref or by another commit.
+
+If a repository check fails, the output is likely to include these warnings.
+
+Ignore these messages, and identify the root cause of the repository check failure
+from the other output.
+
+[GitLab 15.8 and later](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5230) no
+longer includes these messages in the check output. Use the `--no-dangling` option
+to suppress then when run from the command line.
diff --git a/doc/administration/restart_gitlab.md b/doc/administration/restart_gitlab.md
index 7996db3d1e1..e86541b7ced 100644
--- a/doc/administration/restart_gitlab.md
+++ b/doc/administration/restart_gitlab.md
@@ -126,3 +126,17 @@ kubectl delete pods -l release=<helm release name>,app=<component name>
```
The release name can be obtained from the output of the `helm list` command.
+
+## Docker installation
+
+If you change the configuration on your [Docker installation](../install/docker.md), for that change to take effect you must restart:
+
+- The main `gitlab` container.
+- Any separate component containers.
+
+For example, if you deployed Sidekiq on a separate container, to restart the containers, run:
+
+```shell
+sudo docker restart gitlab
+sudo docker restart sidekiq
+```
diff --git a/doc/administration/server_hooks.md b/doc/administration/server_hooks.md
index 3d4f39b5ff0..b167412075b 100644
--- a/doc/administration/server_hooks.md
+++ b/doc/administration/server_hooks.md
@@ -2,7 +2,6 @@
stage: Systems
group: Gitaly
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
-disqus_identifier: 'https://docs.gitlab.com/ee/administration/custom_hooks.html'
---
# Git server hooks **(FREE SELF)**
@@ -28,7 +27,45 @@ alternatives to server hooks include:
[Geo](geo/index.md) doesn't replicate server hooks to secondary nodes.
-## Create server hooks for a repository
+## Set server hooks for a repository
+
+::Tabs
+
+:::TabTitle GitLab 15.11 and later
+
+> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4629) in GitLab 15.11, `hooks set` command replaces direct file system access.
+
+Prerequisites:
+
+- The [storage name](gitaly/configure_gitaly.md#gitlab-requires-a-default-repository-storage), path to the Gitaly configuration file
+ (default is `/var/opt/gitlab/gitaly/config.toml` on Omnibus GitLab instances), and the
+ [repository relative path](repository_storage_types.md#from-project-name-to-hashed-path) for the repository.
+
+To set server hooks for a repository:
+
+1. Create tarball containing custom hooks:
+ 1. Write the code to make the server hook function as expected. Git server hooks can be in any programming language.
+ Ensure the [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) at the top reflects the language type. For
+ example, if the script is in Ruby the shebang is probably `#!/usr/bin/env ruby`.
+
+ - To create a single server hook, create a file with a name that matches the hook type. For example, for a
+ `pre-receive` server hook, the filename should be `pre-receive` with no extension.
+ - To create many server hooks, create a directory for the hooks that matches the hook type. For example, for a
+ `pre-receive` server hook, the directory name should be `pre-receive.d`. Put the files for the hook in that
+ directory.
+
+ 1. Ensure the server hook files are executable and do not match the backup file pattern (`*~`). The server hooks be
+ in a `custom_hooks` directory that is at the root of the tarball.
+ 1. Create the custom hooks archive with the tar command. For example, `tar -cf custom_hooks.tar custom_hooks`.
+1. Run the `hooks set` command with required options to set the Git hooks for the repository. For example,
+ `cat hooks.tar | gitaly hooks set --storage <storage> --repository <relative path> --config <config path>`.
+
+ - A path to a valid Gitaly configuration for the node is required to connect to the node and provided to the `--config` flag.
+ - Custom hooks tarball must be passed via `stdin`. For example, `cat hooks.tar | gitaly hooks set --storage <storage> --repository <relative path> --config <config path>`.
+
+If you implemented the server hook code correctly, it should execute when the Git hook is next triggered.
+
+:::TabTitle GitLab 15.10 and earlier
To create server hooks for a repository:
@@ -56,6 +93,8 @@ To create server hooks for a repository:
If the server hook code is properly implemented, it should execute when the Git hook is next triggered.
+::EndTabs
+
### Gitaly Cluster
If you use [Gitaly Cluster](gitaly/index.md), the scripts must be copied to every Gitaly node that has a replica of the repository. Every Gitaly node
@@ -84,7 +123,7 @@ To create a Git hook that applies to all repositories, set a global server hook.
Before creating a global server hook, you must choose a directory for it.
-For Omnibus GitLab installations, the directory is set in `gitlab.rb` under `gitaly['custom_hooks_dir']`. You can either:
+For Omnibus GitLab installations, the directory is set in `gitlab.rb` under `gitaly['configuration'][:hooks][:custom_hooks_dir]`. You can either:
- Use the default suggestion of the `/var/opt/gitlab/gitaly/custom_hooks` directory by uncommenting it.
- Add your own setting.
@@ -113,6 +152,33 @@ To create a global server hook for all repositories:
If the server hook code is properly implemented, it should execute when the Git hook is next triggered. Hooks are executed in alphabetical order by filename in the hook type
subdirectories.
+## Remove server hooks for a repository
+
+::Tabs
+
+:::TabTitle GitLab 15.11 and later
+
+> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4629) in GitLab 15.11, `hooks set` command replaces direct file system access.
+
+Prerequisites:
+
+- The [storage name and relative path](repository_storage_types.md#from-project-name-to-hashed-path) for the repository.
+
+To remove server hooks, pass an empty tarball to `hook set` to indicate that the repository should contain no hooks. For example:
+
+```shell
+cat empty_hooks.tar | gitaly hooks set --storage <storage> --repository <relative path> --config <config path>`.
+```
+
+:::TabTitle GitLab 15.10 and earlier
+
+To remove server hooks:
+
+1. Go to the location of the repository on disk.
+1. Delete the server hooks in the `custom_hooks` directory.
+
+::EndTabs
+
## Chained server hooks
GitLab can execute server hooks in a chain. GitLab searches for and executes server hooks in the following order:
diff --git a/doc/administration/sidekiq/extra_sidekiq_routing.md b/doc/administration/sidekiq/extra_sidekiq_routing.md
deleted file mode 100644
index d1d65498fcc..00000000000
--- a/doc/administration/sidekiq/extra_sidekiq_routing.md
+++ /dev/null
@@ -1,11 +0,0 @@
----
-redirect_to: 'processing_specific_job_classes.md#routing-rules'
-remove_date: '2023-02-01'
----
-
-This document was moved to [another location](processing_specific_job_classes.md#routing-rules).
-
-<!-- This redirect file can be deleted after <2023-02-01>. -->
-<!-- Redirects that point to other docs in the same project expire in three months. -->
-<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. -->
-<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
diff --git a/doc/administration/sidekiq/index.md b/doc/administration/sidekiq/index.md
index bf858476c0c..6fb12aa6ef9 100644
--- a/doc/administration/sidekiq/index.md
+++ b/doc/administration/sidekiq/index.md
@@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w
You can configure an external Sidekiq instance by using the Sidekiq that's bundled in the GitLab package. Sidekiq requires connection to the Redis,
PostgreSQL, and Gitaly instances.
-## Configure TCP access for PostgreSQL, Gitaly, and Redis
+## Configure TCP access for PostgreSQL, Gitaly, and Redis on the GitLab instance
By default, GitLab uses UNIX sockets and is not set up to communicate via TCP. To change this:
@@ -32,12 +32,20 @@ By default, GitLab uses UNIX sockets and is not set up to communicate via TCP. T
## Gitaly
- # Make Gitaly accept connections on all network interfaces
- gitaly['listen_addr'] = "0.0.0.0:8075"
- ## Set up the Gitaly token as a form of authentication since you are accessing Gitaly over the network
- ## https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#about-the-gitaly-token
- gitaly['auth_token'] = 'abc123secret'
- praefect['auth_token'] = 'abc123secret'
+ gitaly['configuration'] = {
+ # ...
+ #
+ # Make Gitaly accept connections on all network interfaces
+ listen_addr: '0.0.0.0:8075',
+ auth: {
+ ## Set up the Gitaly token as a form of authentication since you are accessing Gitaly over the network
+ ## https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#about-the-gitaly-token
+ token: 'abc123secret',
+ },
+ }
+
+ gitaly['auth_token'] = ''
+ praefect['configuration'][:auth][:token] = 'abc123secret'
gitlab_rails['gitaly_token'] = 'abc123secret'
## Redis configuration
@@ -48,7 +56,6 @@ By default, GitLab uses UNIX sockets and is not set up to communicate via TCP. T
redis['password'] = 'redis-password-goes-here'
gitlab_rails['redis_password'] = 'redis-password-goes-here'
- gitlab_rails['auto_migrate'] = false
```
1. Run `reconfigure`:
@@ -63,18 +70,6 @@ By default, GitLab uses UNIX sockets and is not set up to communicate via TCP. T
sudo gitlab-ctl restart postgresql
```
-1. After the restart, set `auto_migrate` to `true` or comment to use the default settings:
-
- ```ruby
- gitlab_rails['auto_migrate'] = true
- ```
-
-1. Run `reconfigure` again:
-
- ```shell
- sudo gitlab-ctl reconfigure
- ```
-
## Set up Sidekiq instance
1. SSH into the Sidekiq server.
@@ -170,7 +165,7 @@ Updates to example must be made at:
# Replace <database_host> and <database_password>
gitlab_rails['db_host'] = '<database_host>'
- gitlab_rails['db_port'] = '5432'
+ gitlab_rails['db_port'] = 5432
gitlab_rails['db_password'] = '<database_password>'
## Prevent database migrations from running on upgrade automatically
gitlab_rails['auto_migrate'] = false
@@ -257,7 +252,7 @@ To configure the metrics server:
```ruby
sidekiq['metrics_enabled'] = true
sidekiq['listen_address'] = "localhost"
- sidekiq['listen_port'] = "8082"
+ sidekiq['listen_port'] = 8082
# Optionally log all the metrics server logs to log/sidekiq_exporter.log
sidekiq['exporter_log_enabled'] = true
@@ -299,7 +294,7 @@ To make health checks available from `localhost:8092`:
```ruby
sidekiq['health_checks_enabled'] = true
sidekiq['health_checks_listen_address'] = "localhost"
- sidekiq['health_checks_listen_port'] = "8092"
+ sidekiq['health_checks_listen_port'] = 8092
```
1. Reconfigure GitLab:
@@ -325,7 +320,7 @@ To enable LDAP with the synchronization worker for Sidekiq:
1. Edit `/etc/gitlab/gitlab.rb`:
- ```ruby
+ ```ruby
gitlab_rails['ldap_enabled'] = true
gitlab_rails['prevent_ldap_sign_in'] = false
gitlab_rails['ldap_servers'] = {
diff --git a/doc/administration/sidekiq/processing_specific_job_classes.md b/doc/administration/sidekiq/processing_specific_job_classes.md
index 61a154c8db2..26c2804f130 100644
--- a/doc/administration/sidekiq/processing_specific_job_classes.md
+++ b/doc/administration/sidekiq/processing_specific_job_classes.md
@@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w
WARNING:
These are advanced settings. While they are used on GitLab.com, most GitLab
-instances should add more processes that all listen to all queues. This is the
+instances should only add more processes that listen to all queues. This is the
same approach we take in our [Reference Architectures](../reference_architectures/index.md).
GitLab has two options for creating Sidekiq processes that only handle specific
@@ -17,7 +17,7 @@ job classes:
1. [Routing rules](#routing-rules) are used on GitLab.com. They direct jobs
inside the application to queue names configured by administrators. This
lowers the load on Redis, which is important on very large-scale deployments.
-1. [Queue selectors](#queue-selectors) perform the job selection outside the
+1. [Queue selectors](#queue-selectors-deprecated) perform the job selection outside the
application, when starting the Sidekiq process. This was used on GitLab.com
until September 2021, and is retained for compatibility reasons.
@@ -106,11 +106,19 @@ not a recommendation.
sudo gitlab-ctl reconfigure
```
-## Queue selectors
+<!--- start_remove The following content will be removed on remove_date: '2024-08-22' -->
+
+## Queue selectors (deprecated)
> - [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/45) in GitLab 12.8.
> - [Sidekiq cluster, including queue selector, moved](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181) to GitLab Free in 12.10.
> - [Renamed from `experimental_queue_selector` to `queue_selector`](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/147) in GitLab 13.6.
+> - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/390787) in GitLab 15.9.
+
+WARNING:
+This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/390787) in GitLab 15.9
+and is planned for removal in 17.0. Most instances should have [all processes to listen to all queues](extra_sidekiq_processes.md#start-multiple-processes).
+Another alternative is to use [routing rules](#routing-rules) (be warned this is an advanced setting). This change is a breaking change.
The `queue_selector` option allows queue groups to be selected in a more general
way using a [worker matching query](#worker-matching-query). After
@@ -141,7 +149,12 @@ syntax.
sudo gitlab-ctl reconfigure
```
-### Negate settings
+### Negate settings (deprecated)
+
+WARNING:
+This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/390787) in GitLab 15.9
+and is planned for removal in 17.0. Most instances should have [all processes to listen to all queues](extra_sidekiq_processes.md#start-multiple-processes).
+Another alternative is to use [routing rules](#routing-rules) (be warned this is an advanced setting). This change is a breaking change.
This allows you to have the Sidekiq process work on every queue **except** the
ones you list. This is generally only used when there are multiple Sidekiq
@@ -168,7 +181,7 @@ nodes. In this example, we exclude all import-related jobs from a Sidekiq node.
We recommend GitLab deployments add more Sidekiq processes listening to all queues, as in the
[Reference Architectures](../reference_architectures/index.md). For very large-scale deployments, we recommend
-[routing rules](#routing-rules) instead of [queue selectors](#queue-selectors). We use routing rules on GitLab.com as
+[routing rules](#routing-rules) instead of [queue selectors](#queue-selectors-deprecated). We use routing rules on GitLab.com as
it helps to lower the load on Redis.
To migrate from queue selectors to routing rules:
@@ -255,11 +268,13 @@ in a queue group entry is 1, while `min_concurrency` is set to `0`, and `max_con
concurrency is set to `2` instead. A concurrency of `2` might be too low in most cases, except for very highly-CPU
bound tasks.
+<!--- end_remove -->
+
## Worker matching query
GitLab provides a query syntax to match a worker based on its attributes. This
query syntax is employed by both [routing rules](#routing-rules) and
-[queue selectors](#queue-selectors). A query includes two components:
+[queue selectors](#queue-selectors-deprecated). A query includes two components:
- Attributes that can be selected.
- Operators used to construct a query.
diff --git a/doc/administration/sidekiq/sidekiq_memory_killer.md b/doc/administration/sidekiq/sidekiq_memory_killer.md
index cb27d44a2e6..63d93919129 100644
--- a/doc/administration/sidekiq/sidekiq_memory_killer.md
+++ b/doc/administration/sidekiq/sidekiq_memory_killer.md
@@ -56,9 +56,7 @@ Sidekiq memory limits are controlled using environment variables.
If jobs do not finish during that time, all currently running jobs are interrupted with a `SIGTERM` signal
sent to the Sidekiq process.
-- `GITLAB_MEMORY_WATCHDOG_ENABLED`: enabled by default. Set the `GITLAB_MEMORY_WATCHDOG_ENABLED` to false, to use legacy
- Daemon Sidekiq Memory Killer implementation used prior GitLab 15.9. Support for setting `GITLAB_MEMORY_WATCHDOG_ENABLED`
- will be removed in GitLab 16.0.
+- `GITLAB_MEMORY_WATCHDOG_ENABLED`: enabled by default. Set the `GITLAB_MEMORY_WATCHDOG_ENABLED` to false, to disable Watchdog from running.
### Monitor worker restarts
diff --git a/doc/administration/silent_mode/index.md b/doc/administration/silent_mode/index.md
new file mode 100644
index 00000000000..e98aaaf4e0a
--- /dev/null
+++ b/doc/administration/silent_mode/index.md
@@ -0,0 +1,64 @@
+---
+stage: Systems
+group: Geo
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# GitLab Silent Mode (Experiment) **(FREE SELF)**
+
+> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/9826) in GitLab 15.11. This feature is an [Experiment](../../policy/alpha-beta-support.md#experiment).
+
+Silent Mode allows you to suppress outbound communication, such as emails, from GitLab. Silent Mode is not intended to be used on environments which are in-use. Two use-cases are:
+
+- Validating Geo site promotion. You have a secondary Geo site as part of your [disaster recovery](../geo/disaster_recovery/index.md) solution. You want to regularly test promoting it to become a primary Geo site, as a best practice to ensure your disaster recovery plan actually works. But you don't want to actually perform an entire failover, since the primary site lives in a region which provides the lowest latency to your users. And you don't want to take downtime during every regular test. So, you let the primary site remain up, while you promote the secondary site. You start smoke testing the promoted site. But, the promoted site starts emailing users, the push mirrors push changes to external Git repositories, etc. This is where Silent Mode comes in. You can enable it as part of site promotion, to avoid this issue.
+- Validating GitLab backups. You set up a testing instance to test that your backups restore successfully. As part of the restore, you enable Silent Mode, for example to avoid sending invalid emails to users.
+
+## Enable Silent Mode
+
+Prerequisites:
+
+- You must have administrator access.
+
+There are two ways to enable Silent Mode:
+
+- [**API**](../../api/settings.md):
+
+ ```shell
+ curl --request PUT --header "PRIVATE-TOKEN:$ADMIN_TOKEN" "<gitlab-url>/api/v4/application/settings?silent_mode_enabled=true"
+ ```
+
+- [**Rails console**](../operations/rails_console.md#starting-a-rails-console-session):
+
+ ```ruby
+ ::Gitlab::CurrentSettings.update!(silent_mode_enabled: true)
+ ```
+
+## Disable Silent Mode
+
+Prerequisites:
+
+- You must have administrator access.
+
+There are two ways to disable Silent Mode:
+
+- [**API**](../../api/settings.md):
+
+ ```shell
+ curl --request PUT --header "PRIVATE-TOKEN:$ADMIN_TOKEN" "<gitlab-url>/api/v4/application/settings?silent_mode_enabled=false"
+ ```
+
+- [**Rails console**](../operations/rails_console.md#starting-a-rails-console-session):
+
+ ```ruby
+ ::Gitlab::CurrentSettings.update!(silent_mode_enabled: false)
+ ```
+
+## Behavior of GitLab features in Silent Mode
+
+### Service Desk
+
+Incoming emails still raise issues, but the users who sent the emails to [Service Desk](../../user/project/service_desk.md) are not notified of issue creation or comments on their issues.
+
+### Outbound emails
+
+Outbound emails are suppressed. It may take up to a minute to take effect after enabling Silent Mode. [Issue 405433](https://gitlab.com/gitlab-org/gitlab/-/issues/405433) proposes removing this delay.
diff --git a/doc/administration/static_objects_external_storage.md b/doc/administration/static_objects_external_storage.md
index ef89e38be6f..73f44ed3889 100644
--- a/doc/administration/static_objects_external_storage.md
+++ b/doc/administration/static_objects_external_storage.md
@@ -1,6 +1,6 @@
---
stage: Create
-group: Editor
+group: IDE
info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments"
type: reference
---
diff --git a/doc/administration/system_hooks.md b/doc/administration/system_hooks.md
index ddfa9fe9860..8d343e7c541 100644
--- a/doc/administration/system_hooks.md
+++ b/doc/administration/system_hooks.md
@@ -1,6 +1,6 @@
---
stage: Manage
-group: Integrations
+group: Import and Integrate
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
type: reference
---
diff --git a/doc/administration/terraform_state.md b/doc/administration/terraform_state.md
index d3b941bd129..3ac9a5fdce8 100644
--- a/doc/administration/terraform_state.md
+++ b/doc/administration/terraform_state.md
@@ -1,6 +1,6 @@
---
-stage: Configure
-group: Configure
+stage: Deploy
+group: Environments
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
@@ -22,9 +22,23 @@ Use [external object storage](https://docs.gitlab.com/charts/advanced/external-o
## Disabling Terraform state
-To disable terraform state site-wide, follow the steps below.
-A GitLab administrator may want to disable Terraform state to reduce disk space or if Terraform is not used in your instance.
-To do so, follow the steps below according to your installation's type.
+You can disable Terraform state across the entire instance. You might want to disable Terraform to reduce disk space,
+or because your instance doesn't use Terraform.
+
+When Terraform state administration is disabled:
+
+- On the left sidebar, you cannot select **Infrastructure > Terraform states**.
+- Any CI/CD jobs that access the Terraform state fail with this error:
+
+ ```shell
+ Error refreshing state: HTTP remote state endpoint invalid auth
+ ```
+
+To disable Terraform administration, follow the steps below according to your installation.
+
+Prerequisite:
+
+- You must be an administrator.
**In Omnibus installations:**
@@ -79,7 +93,7 @@ Terraform state files are stored locally, follow the steps below.
## Using object storage **(FREE SELF)**
Instead of storing Terraform state files on disk, we recommend the use of
-[one of the supported object storage options](object_storage.md#options).
+[one of the supported object storage options](object_storage.md#supported-object-storage-providers).
This configuration relies on valid credentials to be configured already.
[Read more about using object storage with GitLab](object_storage.md).
@@ -161,10 +175,10 @@ sudo find /var/opt/gitlab/gitlab-rails/shared/terraform_state -type f | grep -v
### S3-compatible connection settings
In GitLab 13.2 and later, you should use the
-[consolidated object storage settings](object_storage.md#consolidated-object-storage-configuration).
+[consolidated object storage settings](object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form).
This section describes the earlier configuration format.
-See [the available connection settings for different providers](object_storage.md#connection-settings).
+See [the available connection settings for different providers](object_storage.md#configure-the-connection-settings).
**In Omnibus installations:**
diff --git a/doc/administration/troubleshooting/index.md b/doc/administration/troubleshooting/index.md
index db572f202ea..8be6abc180d 100644
--- a/doc/administration/troubleshooting/index.md
+++ b/doc/administration/troubleshooting/index.md
@@ -27,3 +27,28 @@ that you can check for feature-specific help, including helpful Rails commands.
If you need a testing environment to troubleshoot, see the
[apps for a testing environment](test_environments.md).
+
+## Support team troubleshooting info
+
+The GitLab Support Team has collected a lot of information about troubleshooting GitLab.
+The following documents are used by the Support Team or by customers
+with direct guidance from a Support Team member. GitLab administrators may find the
+information useful for troubleshooting. However, if you are experiencing trouble with your
+GitLab instance, you should check your [support options](https://about.gitlab.com/support/)
+before referring to these documents.
+
+WARNING:
+The commands in the following documentation might result in data loss or
+other damage to a GitLab instance. They should be used only by experienced administrators
+who are aware of the risks.
+
+- [Diagnostics tools](diagnostics_tools.md)
+- [Linux commands](linux_cheat_sheet.md)
+- [Troubleshooting Kubernetes](https://docs.gitlab.com/charts/troubleshooting/kubernetes_cheat_sheet.html)
+- [Troubleshooting PostgreSQL](postgresql.md)
+- [Guide to test environments](test_environments.md) (for Support Engineers)
+- [Troubleshooting SSL](ssl.md)
+- Related links:
+ - [Repairing and recovering broken Git repositories](https://git.seveas.net/repairing-and-recovering-broken-git-repositories.html)
+ - [Testing with OpenSSL](https://www.feistyduck.com/library/openssl-cookbook/online/testing-with-openssl/index.html)
+ - [`strace` zine](https://wizardzines.com/zines/strace/)
diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md
index ff0b8ecf178..ab900d7ea9f 100644
--- a/doc/administration/uploads.md
+++ b/doc/administration/uploads.md
@@ -8,6 +8,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w
Uploads represent all user data that may be sent to GitLab as a single file. For example, avatars and note attachments are uploads. Uploads are integral to GitLab functionality and therefore cannot be disabled.
+NOTE:
+Attachments added to comments or descriptions are deleted **only** when the parent project or group
+is deleted. Attachments remain in file storage even when the comment or resource (like issue, merge
+request, epic) where they were uploaded is deleted.
+
## Using local storage
This is the default configuration. To change the location where the uploads are
@@ -60,7 +65,7 @@ This configuration relies on valid AWS credentials to be configured already.
### Object Storage Settings
In GitLab 13.2 and later, you should use the
-[consolidated object storage settings](object_storage.md#consolidated-object-storage-configuration).
+[consolidated object storage settings](object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form).
This section describes the earlier configuration format.
For source installations the following settings are nested under `uploads:` and then `object_store:`. On Omnibus GitLab installs they are prefixed by `uploads_object_store_`.
@@ -74,7 +79,7 @@ For source installations the following settings are nested under `uploads:` and
#### Connection settings
-See [the available connection settings for different providers](object_storage.md#connection-settings).
+See [the available connection settings for different providers](object_storage.md#configure-the-connection-settings).
**In Omnibus installations:**
@@ -114,7 +119,7 @@ _The uploads are stored by default in
`/home/git/gitlab/public/uploads`._
1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following
- lines, making sure to use the [appropriate ones for your provider](object_storage.md#connection-settings):
+ lines, making sure to use the [appropriate ones for your provider](object_storage.md#configure-the-connection-settings):
```yaml
uploads:
diff --git a/doc/administration/wikis/index.md b/doc/administration/wikis/index.md
index f3442e23160..330e41ee880 100644
--- a/doc/administration/wikis/index.md
+++ b/doc/administration/wikis/index.md
@@ -1,7 +1,7 @@
---
type: reference, howto
-stage: Create
-group: Editor
+stage: Plan
+group: Knowledge
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---