diff options
Diffstat (limited to 'doc/administration')
138 files changed, 3784 insertions, 2434 deletions
diff --git a/doc/administration/audit_event_streaming.md b/doc/administration/audit_event_streaming.md index afc4926fec0..d6a6725210d 100644 --- a/doc/administration/audit_event_streaming.md +++ b/doc/administration/audit_event_streaming.md @@ -1,979 +1,11 @@ --- -stage: Govern -group: Compliance -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: 'audit_event_streaming/index.md' +remove_date: '2023-09-12' --- -# Audit event streaming **(ULTIMATE)** +This document was moved to [another location](audit_event_streaming/index.md). -> - API [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332747) in GitLab 14.5 [with a flag](../administration/feature_flags.md) named `ff_external_audit_events_namespace`. Disabled by default. -> - API [Enabled on GitLab.com and by default on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/338939) in GitLab 14.7. -> - API [Feature flag `ff_external_audit_events_namespace`](https://gitlab.com/gitlab-org/gitlab/-/issues/349588) removed in GitLab 14.8. -> - UI [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336411) in GitLab 14.9. -> - [Subgroup events recording](https://gitlab.com/gitlab-org/gitlab/-/issues/366878) fixed in GitLab 15.2. -> - Custom HTTP headers API [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/361216) in GitLab 15.1 [with a flag](feature_flags.md) named `streaming_audit_event_headers`. Disabled by default. -> - Custom HTTP headers API [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/362941) in GitLab 15.2. -> - Custom HTTP headers API [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/366524) in GitLab 15.3. [Feature flag `streaming_audit_event_headers`](https://gitlab.com/gitlab-org/gitlab/-/issues/362941) removed. -> - Custom HTTP headers UI [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/361630) in GitLab 15.2 [with a flag](feature_flags.md) named `custom_headers_streaming_audit_events_ui`. Disabled by default. -> - Custom HTTP headers UI [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/365259) in GitLab 15.3. [Feature flag `custom_headers_streaming_audit_events_ui`](https://gitlab.com/gitlab-org/gitlab/-/issues/365259) removed. -> - [Improved user experience](https://gitlab.com/gitlab-org/gitlab/-/issues/367963) in GitLab 15.3. -> - User-specified verification token API support [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/360813) in GitLab 15.4. -> - Event type filters API [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/344845) in GitLab 15.7. - -Users can set a streaming destination for a top-level group to receive all audit events about the group, its subgroups, and -projects as structured JSON. - -Top-level group owners can manage their audit logs in third-party systems. Any service that can receive -structured JSON data can be used as the streaming destination. - -Each streaming destination can have up to 20 custom HTTP headers included with each streamed event. - -NOTE: -GitLab can stream a single event more than once to the same destination. Use the `id` key in the payload to deduplicate incoming data. - -## Add a new streaming destination - -WARNING: -Streaming destinations receive **all** audit event data, which could include sensitive information. Make sure you trust the streaming destination. - -### Use the GitLab UI - -Users with the Owner role for a group can add streaming destinations for it: - -1. On the top bar, select **Main menu > Groups** and find your group. -1. On the left sidebar, select **Security and Compliance > Audit events**. -1. On the main area, select **Streams** tab. -1. Select **Add streaming destination** to show the section for adding destinations. -1. Enter the destination URL to add. -1. Optional. Locate the **Custom HTTP headers** table. -1. Ignore the **Active** checkbox because it isn't functional. To track progress on adding functionality to the - **Active** checkbox, see [issue 367509](https://gitlab.com/gitlab-org/gitlab/-/issues/367509). -1. Select **Add header** to create a new name and value pair. Enter as many name and value pairs as required. You can add up to - 20 headers per streaming destination. -1. After all headers have been filled out, select **Add** to add the new streaming destination. - -### Use the API - -To enable streaming and add a destination, users with the Owner role for a group must use the -`externalAuditEventDestinationCreate` mutation in the GraphQL API. - -```graphql -mutation { - externalAuditEventDestinationCreate(input: { destinationUrl: "https://mydomain.io/endpoint/ingest", groupPath: "my-group" } ) { - errors - externalAuditEventDestination { - id - destinationUrl - verificationToken - group { - name - } - } - } -} -``` - -Group owners can also optionally specify their own verification token (instead of the default GitLab-generated one) using the GraphQL `auditEventsStreamingHeadersCreate` -mutation. Verification token length must be within 16 to 24 characters and trailing whitespace are not trimmed. GitLab recommends setting a cryptographically random and unique value. For example: - -```graphql -mutation { - externalAuditEventDestinationCreate(input: { destinationUrl: "https://mydomain.io/endpoint/ingest", groupPath: "my-group", verificationToken: "unique-random-verification-token-here" } ) { - errors - externalAuditEventDestination { - id - destinationUrl - verificationToken - group { - name - } - } - } -} -``` - -Event streaming is enabled if: - -- The returned `errors` object is empty. -- The API responds with `200 OK`. - -Group owners can add an HTTP header using the GraphQL `auditEventsStreamingHeadersCreate` mutation. You can retrieve the destination ID -by [listing all the streaming destinations](#use-the-api-1) for the group or from the mutation above. - -```graphql -mutation { - auditEventsStreamingHeadersCreate(input: { destinationId: "gid://gitlab/AuditEvents::ExternalAuditEventDestination/24601", key: "foo", value: "bar" }) { - errors - } -} -``` - -The header is created if the returned `errors` object is empty. - -## List streaming destinations - -Users with the Owner role for a group can list streaming destinations. - -### Use the GitLab UI - -To list the streaming destinations: - -1. On the top bar, select **Main menu > Groups** and find your group. -1. On the left sidebar, select **Security and Compliance > Audit events**. -1. On the main area, select **Streams** tab. -1. To the right of the item, select **Edit** (**{pencil}**) to see all the custom HTTP headers. - -### Use the API - -Users with the Owner role for a group can view a list of streaming destinations at any time using the -`externalAuditEventDestinations` query type. - -```graphql -query { - group(fullPath: "my-group") { - id - externalAuditEventDestinations { - nodes { - destinationUrl - verificationToken - id - headers { - nodes { - key - value - id - } - } - eventTypeFilters - } - } - } -} -``` - -If the resulting list is empty, then audit streaming is not enabled for that group. - -You need the ID values returned by this query for the update and delete mutations. - -## Update streaming destinations - -Users with the Owner role for a group can update streaming destinations. - -### Use the GitLab UI - -To update a streaming destinations custom HTTP headers: - -1. On the top bar, select **Main menu > Groups** and find your group. -1. On the left sidebar, select **Security and Compliance > Audit events**. -1. On the main area, select **Streams** tab. -1. To the right of the item, select **Edit** (**{pencil}**). -1. Locate the **Custom HTTP headers** table. -1. Locate the header that you wish to update. -1. Ignore the **Active** checkbox because it isn't functional. To track progress on adding functionality to the - **Active** checkbox, see [issue 367509](https://gitlab.com/gitlab-org/gitlab/-/issues/367509). -1. Select **Add header** to create a new name and value pair. Enter as many name and value pairs as required. You can add up to - 20 headers per streaming destination. -1. Select **Save** to update the streaming destination. - -### Use the API - -Users with the Owner role for a group can update streaming destinations custom HTTP headers using the -`auditEventsStreamingHeadersUpdate` mutation type. You can retrieve the custom HTTP headers ID -by [listing all the custom HTTP headers](#use-the-api-1) for the group. - -```graphql -mutation { - externalAuditEventDestinationDestroy(input: { id: destination }) { - errors - } -} -``` - -Streaming destination is updated if: - -- The returned `errors` object is empty. -- The API responds with `200 OK`. - -Group owners can remove an HTTP header using the GraphQL `auditEventsStreamingHeadersDestroy` mutation. You can retrieve the header ID -by [listing all the custom HTTP headers](#use-the-api-1) for the group. - -```graphql -mutation { - auditEventsStreamingHeadersDestroy(input: { headerId: "gid://gitlab/AuditEvents::Streaming::Header/1" }) { - errors - } -} -``` - -The header is deleted if the returned `errors` object is empty. - -## Delete streaming destinations - -Users with the Owner role for a group can delete streaming destinations. - -When the last destination is successfully deleted, streaming is disabled for the group. - -### Use the GitLab UI - -To delete a streaming destination: - -1. On the top bar, select **Main menu > Groups** and find your group. -1. On the left sidebar, select **Security and Compliance > Audit events**. -1. On the main area, select the **Streams** tab. -1. To the right of the item, select **Delete** (**{remove}**). - -To delete only the custom HTTP headers for a streaming destination: - -1. On the top bar, select **Main menu > Groups** and find your group. -1. On the left sidebar, select **Security and Compliance > Audit events**. -1. On the main area, select the **Streams** tab. -1. To the right of the item, **Edit** (**{pencil}**). -1. Locate the **Custom HTTP headers** table. -1. Locate the header that you wish to remove. -1. To the right of the header, select **Delete** (**{remove}**). -1. Select **Save** to update the streaming destination. - -### Use the API - -Users with the Owner role for a group can delete streaming destinations using the -`externalAuditEventDestinationDestroy` mutation type. You can retrieve the destinations ID -by [listing all the streaming destinations](#use-the-api-1) for the group. - -```graphql -mutation { - externalAuditEventDestinationDestroy(input: { id: destination }) { - errors - } -} -``` - -Streaming destination is deleted if: - -- The returned `errors` object is empty. -- The API responds with `200 OK`. - -Group owners can remove an HTTP header using the GraphQL `auditEventsStreamingHeadersDestroy` mutation. You can retrieve the header ID -by [listing all the custom HTTP headers](#use-the-api-1) for the group. - -```graphql -mutation { - auditEventsStreamingHeadersDestroy(input: { headerId: "gid://gitlab/AuditEvents::Streaming::Header/1" }) { - errors - } -} -``` - -The header is deleted if the returned `errors` object is empty. - -## Verify event authenticity - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345424) in GitLab 14.8. - -Each streaming destination has a unique verification token (`verificationToken`) that can be used to verify the authenticity of the event. This -token is either specified by the Owner or generated automatically when the event destination is created and cannot be changed. - -Each streamed event contains the verification token in the `X-Gitlab-Event-Streaming-Token` HTTP header that can be verified against -the destination's value when [listing streaming destinations](#list-streaming-destinations). - -### Use the GitLab UI - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/360814) in GitLab 15.2. - -Users with the Owner role for a group can list streaming destinations and see the verification tokens: - -1. On the top bar, select **Main menu > Groups** and find your group. -1. On the left sidebar, select **Security and Compliance > Audit events**. -1. On the main area, select the **Streams**. -1. View the verification token on the right side of each item. - -## Event type filters - -> Event type filters API [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/344845) in GitLab 15.7. - -When this feature is enabled for a group, you can use an API to permit users to filter streamed audit events per destination. -If the feature is enabled with no filters, the destination receives all audit events. - -A streaming destination that has an event type filter set has a **filtered** (**{filter}**) label. - -### Use the API to add an event type filter - -Prerequisites: - -- You must have the Owner role for the group. - -You can add a list of event type filters using the `auditEventsStreamingDestinationEventsAdd` query type: - -```graphql -mutation { - auditEventsStreamingDestinationEventsAdd(input: { - destinationId: "gid://gitlab/AuditEvents::ExternalAuditEventDestination/1", - eventTypeFilters: ["list of event type filters"]}){ - errors - eventTypeFilters - } -} -``` - -Event type filters are added if: - -- The returned `errors` object is empty. -- The API responds with `200 OK`. - -### Use the API to remove an event type filter - -Prerequisites: - -- You must have the Owner role for the group. - -You can remove a list of event type filters using the `auditEventsStreamingDestinationEventsRemove` query type: - -```graphql -mutation { - auditEventsStreamingDestinationEventsRemove(input: { - destinationId: "gid://gitlab/AuditEvents::ExternalAuditEventDestination/1", - eventTypeFilters: ["list of event type filters"] - }){ - errors - } -} -``` - -Event type filters are removed if: - -- The returned `errors` object is empty. -- The API responds with `200 OK`. - -## Payload schema - -> Documentation for an audit event streaming schema was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/358149) in GitLab 15.3. - -Streamed audit events have a predictable schema in the body of the response. - -| Field | Description | Notes | -|------------------|------------------------------------------------------------|-----------------------------------------------------------------------------------| -| `author_id` | User ID of the user who triggered the event | | -| `author_name` | Human-readable name of the author that triggered the event | Helpful when the author no longer exists | -| `created_at` | Timestamp when event was triggered | | -| `details` | JSON object containing additional metadata | Has no defined schema but often contains additional information about an event | -| `entity_id` | ID of the audit event's entity | | -| `entity_path` | Full path of the entity affected by the auditable event | | -| `entity_type` | String representation of the type of entity | Acceptable values include `User`, `Group`, and `Key`. This list is not exhaustive | -| `event_type` | String representation of the type of audit event | | -| `id` | Unique identifier for the audit event | Can be used for deduplication if required | -| `ip_address` | IP address of the host used to trigger the event | | -| `target_details` | Additional details about the target | | -| `target_id` | ID of the audit event's target | | -| `target_type` | String representation of the target's type | | - -### JSON payload schema - -```json -{ - "properties": { - "id": { - "type": "string" - }, - "author_id": { - "type": "integer" - }, - "author_name": { - "type": "string" - }, - "details": {}, - "ip_address": { - "type": "string" - }, - "entity_id": { - "type": "integer" - }, - "entity_path": { - "type": "string" - }, - "entity_type": { - "type": "string" - }, - "event_type": { - "type": "string" - }, - "target_id": { - "type": "integer" - }, - "target_type": { - "type": "string" - }, - "target_details": { - "type": "string" - }, - }, - "type": "object" -} -``` - -## Audit event streaming on Git operations - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332747) in GitLab 14.9 [with a flag](../administration/feature_flags.md) named `audit_event_streaming_git_operations`. Disabled by default. -> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/357211) in GitLab 15.0. -> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/357211) in GitLab 15.1 by default. -> - [Added `details.author_class` field](https://gitlab.com/gitlab-org/gitlab/-/issues/363876) in GitLab 15.3. -> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/101583) in GitLab 15.6. Feature flag `audit_event_streaming_git_operations` removed. - -Streaming audit events can be sent when authenticated users push, pull, or clone a project's remote Git repositories: - -- [Using SSH](../user/ssh.md). -- Using HTTP or HTTPS. -- Using the **Download** button (**{download}**) in GitLab UI. - -Audit events are not captured for users that are not signed in. For example, when downloading a public project. - -To configure streaming audit events for Git operations, see [Add a new streaming destination](#add-a-new-streaming-destination). - -### Headers - -> `X-Gitlab-Audit-Event-Type` [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/86881) in GitLab 15.0. - -Headers are formatted as follows: - -```plaintext -POST /logs HTTP/1.1 -Host: <DESTINATION_HOST> -Content-Type: application/x-www-form-urlencoded -X-Gitlab-Event-Streaming-Token: <DESTINATION_TOKEN> -X-Gitlab-Audit-Event-Type: repository_git_operation -``` - -### Example payloads for SSH events - -Fetch: - -```json -{ - "id": 1, - "author_id": 1, - "entity_id": 29, - "entity_type": "Project", - "details": { - "author_name": "Administrator", - "author_class": "User", - "target_id": 29, - "target_type": "Project", - "target_details": "example-project", - "custom_message": { - "protocol": "ssh", - "action": "git-upload-pack" - }, - "ip_address": "127.0.0.1", - "entity_path": "example-group/example-project" - }, - "ip_address": "127.0.0.1", - "author_name": "Administrator", - "entity_path": "example-group/example-project", - "target_details": "example-project", - "created_at": "2022-02-23T06:21:05.283Z", - "target_type": "Project", - "target_id": 29, - "event_type": "repository_git_operation" -} -``` - -Push: - -```json -{ - "id": 1, - "author_id": 1, - "entity_id": 29, - "entity_type": "Project", - "details": { - "author_name": "Administrator", - "author_class": "User", - "target_id": 29, - "target_type": "Project", - "target_details": "example-project", - "custom_message": { - "protocol": "ssh", - "action": "git-receive-pack" - }, - "ip_address": "127.0.0.1", - "entity_path": "example-group/example-project" - }, - "ip_address": "127.0.0.1", - "author_name": "Administrator", - "entity_path": "example-group/example-project", - "target_details": "example-project", - "created_at": "2022-02-23T06:23:08.746Z", - "target_type": "Project", - "target_id": 29, - "event_type": "repository_git_operation" -} -``` - -### Example payloads for SSH events with Deploy Key - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/363876) in GitLab 15.3. - -Fetch: - -```json -{ - "id": 1, - "author_id": -3, - "entity_id": 29, - "entity_type": "Project", - "details": { - "author_name": "deploy-key-name", - "author_class": "DeployKey", - "target_id": 29, - "target_type": "Project", - "target_details": "example-project", - "custom_message": { - "protocol": "ssh", - "action": "git-upload-pack" - }, - "ip_address": "127.0.0.1", - "entity_path": "example-group/example-project" - }, - "ip_address": "127.0.0.1", - "author_name": "deploy-key-name", - "entity_path": "example-group/example-project", - "target_details": "example-project", - "created_at": "2022-07-26T05:43:53.662Z", - "target_type": "Project", - "target_id": 29, - "event_type": "repository_git_operation" -} -``` - -### Example payloads for HTTP and HTTPS events - -Fetch: - -```json -{ - "id": 1, - "author_id": 1, - "entity_id": 29, - "entity_type": "Project", - "details": { - "author_name": "Administrator", - "author_class": "User", - "target_id": 29, - "target_type": "Project", - "target_details": "example-project", - "custom_message": { - "protocol": "http", - "action": "git-upload-pack" - }, - "ip_address": "127.0.0.1", - "entity_path": "example-group/example-project" - }, - "ip_address": "127.0.0.1", - "author_name": "Administrator", - "entity_path": "example-group/example-project", - "target_details": "example-project", - "created_at": "2022-02-23T06:25:43.938Z", - "target_type": "Project", - "target_id": 29, - "event_type": "repository_git_operation" -} -``` - -Push: - -```json -{ - "id": 1, - "author_id": 1, - "entity_id": 29, - "entity_type": "Project", - "details": { - "author_name": "Administrator", - "author_class": "User", - "target_id": 29, - "target_type": "Project", - "target_details": "example-project", - "custom_message": { - "protocol": "http", - "action": "git-receive-pack" - }, - "ip_address": "127.0.0.1", - "entity_path": "example-group/example-project" - }, - "ip_address": "127.0.0.1", - "author_name": "Administrator", - "entity_path": "example-group/example-project", - "target_details": "example-project", - "created_at": "2022-02-23T06:26:29.294Z", - "target_type": "Project", - "target_id": 29, - "event_type": "repository_git_operation" -} -``` - -### Example payloads for HTTP and HTTPS events with Deploy Token - -Fetch: - -```json -{ - "id": 1, - "author_id": -2, - "entity_id": 22, - "entity_type": "Project", - "details": { - "author_name": "deploy-token-name", - "author_class": "DeployToken", - "target_id": 22, - "target_type": "Project", - "target_details": "example-project", - "custom_message": { - "protocol": "http", - "action": "git-upload-pack" - }, - "ip_address": "127.0.0.1", - "entity_path": "example-group/example-project" - }, - "ip_address": "127.0.0.1", - "author_name": "deploy-token-name", - "entity_path": "example-group/example-project", - "target_details": "example-project", - "created_at": "2022-07-26T05:46:25.850Z", - "target_type": "Project", - "target_id": 22, - "event_type": "repository_git_operation" -} -``` - -### Example payloads for events from GitLab UI download button - -Fetch: - -```json -{ - "id": 1, - "author_id": 99, - "entity_id": 29, - "entity_type": "Project", - "details": { - "custom_message": "Repository Download Started", - "author_name": "example_username", - "author_class": "User", - "target_id": 29, - "target_type": "Project", - "target_details": "example-group/example-project", - "ip_address": "127.0.0.1", - "entity_path": "example-group/example-project" - }, - "ip_address": "127.0.0.1", - "author_name": "example_username", - "entity_path": "example-group/example-project", - "target_details": "example-group/example-project", - "created_at": "2022-02-23T06:27:17.873Z", - "target_type": "Project", - "target_id": 29, - "event_type": "repository_git_operation" -} -``` - -## Audit event streaming on merge request approval actions - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/271162) in GitLab 14.9. - -Stream audit events that relate to merge approval actions performed within a project. - -### Headers - -Headers are formatted as follows: - -```plaintext -POST /logs HTTP/1.1 -Host: <DESTINATION_HOST> -Content-Type: application/x-www-form-urlencoded -X-Gitlab-Event-Streaming-Token: <DESTINATION_TOKEN> -X-Gitlab-Audit-Event-Type: audit_operation -``` - -### Example payload - -```json -{ - "id": 1, - "author_id": 1, - "entity_id": 6, - "entity_type": "Project", - "details": { - "author_name": "example_username", - "target_id": 20, - "target_type": "MergeRequest", - "target_details": "merge request title", - "custom_message": "Approved merge request", - "ip_address": "127.0.0.1", - "entity_path": "example-group/example-project" - }, - "ip_address": "127.0.0.1", - "author_name": "example_username", - "entity_path": "example-group/example-project", - "target_details": "merge request title", - "created_at": "2022-03-09T06:53:11.181Z", - "target_type": "MergeRequest", - "target_id": 20, - "event_type": "audit_operation" -} -``` - -## Audit event streaming on merge request create actions - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/90911) in GitLab 15.2. - -Stream audit events that relate to merge request create actions using the `/logs` endpoint. - -Send API requests that contain the `X-Gitlab-Audit-Event-Type` header with value `merge_request_create`. GitLab responds with JSON payloads with an -`event_type` field set to `merge_request_create`. - -### Headers - -Headers are formatted as follows: - -```plaintext -POST /logs HTTP/1.1 -Host: <DESTINATION_HOST> -Content-Type: application/x-www-form-urlencoded -X-Gitlab-Audit-Event-Type: merge_request_create -X-Gitlab-Event-Streaming-Token: <DESTINATION_TOKEN> -``` - -### Example payload - -```json -{ - "id": 1, - "author_id": 1, - "entity_id": 24, - "entity_type": "Project", - "details": { - "author_name": "example_user", - "target_id": 132, - "target_type": "MergeRequest", - "target_details": "Update test.md", - "custom_message": "Added merge request", - "ip_address": "127.0.0.1", - "entity_path": "example-group/example-project" - }, - "ip_address": "127.0.0.1", - "author_name": "Administrator", - "entity_path": "example-group/example-project", - "target_details": "Update test.md", - "created_at": "2022-07-04T00:19:22.675Z", - "target_type": "MergeRequest", - "target_id": 132, - "event_type": "merge_request_create" -} -``` - -## Audit event streaming on project fork actions - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/90916) in GitLab 15.2. - -Stream audit events that relate to project fork actions using the `/logs` endpoint. - -Send API requests that contain the `X-Gitlab-Audit-Event-Type` header with value `project_fork_operation`. GitLab responds with JSON payloads with an -`event_type` field set to `project_fork_operation`. - -### Headers - -Headers are formatted as follows: - -```plaintext -POST /logs HTTP/1.1 -Host: <DESTINATION_HOST> -Content-Type: application/x-www-form-urlencoded -X-Gitlab-Audit-Event-Type: project_fork_operation -X-Gitlab-Event-Streaming-Token: <DESTINATION_TOKEN> -``` - -### Example payload - -```json -{ - "id": 1, - "author_id": 1, - "entity_id": 24, - "entity_type": "Project", - "details": { - "author_name": "example_username", - "target_id": 24, - "target_type": "Project", - "target_details": "example-project", - "custom_message": "Forked project to another-group/example-project-forked", - "ip_address": "127.0.0.1", - "entity_path": "example-group/example-project" - }, - "ip_address": "127.0.0.1", - "author_name": "example_username", - "entity_path": "example-group/example-project", - "target_details": "example-project", - "created_at": "2022-06-30T03:43:35.384Z", - "target_type": "Project", - "target_id": 24, - "event_type": "project_fork_operation" -} -``` - -## Audit event streaming on project group link actions - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/90955) in GitLab 15.2. - -Stream audit events that relate to project group link creation, updates, and deletion using the `/logs` endpoint. - -Send API requests that contain the `X-Gitlab-Audit-Event-Type` header with value of either: - -- `project_group_link_create`. -- `project_group_link_update`. -- `project_group_link_destroy`. - -GitLab responds with JSON payloads with an `event_type` field set to either: - -- `project_group_link_create`. -- `project_group_link_update`. -- `project_group_link_destroy`. - -### Example Headers - -Headers are formatted as follows: - -```plaintext -POST /logs HTTP/1.1 -Host: <DESTINATION_HOST> -Content-Type: application/x-www-form-urlencoded -X-Gitlab-Audit-Event-Type: project_group_link_create -X-Gitlab-Event-Streaming-Token: <DESTINATION_TOKEN> -``` - -### Example payload for project group link create - -```json -{ - "id": 1, - "author_id": 1, - "entity_id": 24, - "entity_type": "Project", - "details": { - "author_name": "example-user", - "target_id": 31, - "target_type": "Group", - "target_details": "another-group", - "custom_message": "Added project group link", - "ip_address": "127.0.0.1", - "entity_path": "example-group/example-project" - }, - "ip_address": "127.0.0.1", - "author_name": "example-user", - "entity_path": "example-group/example-project", - "target_details": "another-group", - "created_at": "2022-07-04T00:43:09.318Z", - "target_type": "Group", - "target_id": 31, - "event_type": "project_group_link_create" -} -``` - -### Example payload for project group link update - -```json -{ - "id": 1, - "author_id": 1, - "entity_id": 24, - "entity_type": "Project", - "details": { - "author_name": "example-user", - "target_id": 31, - "target_type": "Group", - "target_details": "another-group", - "custom_message": "Changed project group link profile group_access from Developer to Guest", - "ip_address": "127.0.0.1", - "entity_path": "example-group/example-project" - }, - "ip_address": "127.0.0.1", - "author_name": "example-user", - "entity_path": "example-group/example-project", - "target_details": "another-group", - "created_at": "2022-07-04T00:43:28.328Z", - "target_type": "Group", - "target_id": 31, - "event_type": "project_group_link_update" -} -``` - -### Example payload for project group link delete - -```json -{ - "id": 1, - "author_id": 1, - "entity_id": 24, - "entity_type": "Project", - "details": { - "author_name": "example-user", - "target_id": 31, - "target_type": "Group", - "target_details": "another-group", - "custom_message": "Removed project group link", - "ip_address": "127.0.0.1", - "entity_path": "example-group/example-project" - }, - "ip_address": "127.0.0.1", - "author_name": "example-user", - "entity_path": "example-group/example-project", - "target_details": "another-group", - "created_at": "2022-07-04T00:42:56.279Z", - "target_type": "Group", - "target_id": 31, - "event_type": "project_group_link_destroy" -} -``` - -## Audit event streaming on invalid merge request approver state - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/374566) in GitLab 15.5. - -Stream audit events that relate to invalid merge request approver states within a project. - -### Headers - -Headers are formatted as follows: - -```plaintext -POST /logs HTTP/1.1 -Host: <DESTINATION_HOST> -Content-Type: application/x-www-form-urlencoded -X-Gitlab-Event-Streaming-Token: <DESTINATION_TOKEN> -X-Gitlab-Audit-Event-Type: audit_operation -``` - -### Example payload - -```json -{ - "id": 1, - "author_id": 1, - "entity_id": 6, - "entity_type": "Project", - "details": { - "author_name": "example_username", - "target_id": 20, - "target_type": "MergeRequest", - "target_details": { title: "Merge request title", iid: "Merge request iid", id: "Merge request id" }, - "custom_message": "Invalid merge request approver rules", - "ip_address": "127.0.0.1", - "entity_path": "example-group/example-project" - }, - "ip_address": "127.0.0.1", - "author_name": "example_username", - "entity_path": "example-group/example-project", - "target_details": "merge request title", - "created_at": "2022-03-09T06:53:11.181Z", - "target_type": "MergeRequest", - "target_id": 20, - "event_type": "audit_operation" -} -``` +<!-- This redirect file can be deleted after <2023-09-12>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/audit_event_streaming/examples.md b/doc/administration/audit_event_streaming/examples.md new file mode 100644 index 00000000000..d741e5fd60d --- /dev/null +++ b/doc/administration/audit_event_streaming/examples.md @@ -0,0 +1,578 @@ +--- +stage: Govern +group: Compliance +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Audit event streaming examples + +The following sections provide examples of audit event streaming. + +## Audit event streaming on Git operations + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332747) in GitLab 14.9 [with a flag](../feature_flags.md) named `audit_event_streaming_git_operations`. Disabled by default. +> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/357211) in GitLab 15.0. +> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/357211) in GitLab 15.1 by default. +> - [Added `details.author_class` field](https://gitlab.com/gitlab-org/gitlab/-/issues/363876) in GitLab 15.3. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/101583) in GitLab 15.6. Feature flag `audit_event_streaming_git_operations` removed. + +Streaming audit events can be sent when authenticated users push, pull, or clone a project's remote Git repositories: + +- [Using SSH](../../user/ssh.md). +- Using HTTP or HTTPS. +- Using **Download** (**{download}**) in GitLab UI. + +Audit events are not captured for users that are not signed in. For example, when downloading a public project. + +To configure streaming audit events for Git operations, see [Add a new streaming destination](index.md#add-a-new-streaming-destination). + +### Headers + +> `X-Gitlab-Audit-Event-Type` [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/86881) in GitLab 15.0. + +Headers are formatted as follows: + +```plaintext +POST /logs HTTP/1.1 +Host: <DESTINATION_HOST> +Content-Type: application/x-www-form-urlencoded +X-Gitlab-Event-Streaming-Token: <DESTINATION_TOKEN> +X-Gitlab-Audit-Event-Type: repository_git_operation +``` + +### Example payloads for SSH events + +Fetch: + +```json +{ + "id": 1, + "author_id": 1, + "entity_id": 29, + "entity_type": "Project", + "details": { + "author_name": "Administrator", + "author_class": "User", + "target_id": 29, + "target_type": "Project", + "target_details": "example-project", + "custom_message": { + "protocol": "ssh", + "action": "git-upload-pack" + }, + "ip_address": "127.0.0.1", + "entity_path": "example-group/example-project" + }, + "ip_address": "127.0.0.1", + "author_name": "Administrator", + "entity_path": "example-group/example-project", + "target_details": "example-project", + "created_at": "2022-02-23T06:21:05.283Z", + "target_type": "Project", + "target_id": 29, + "event_type": "repository_git_operation" +} +``` + +Push: + +```json +{ + "id": 1, + "author_id": 1, + "entity_id": 29, + "entity_type": "Project", + "details": { + "author_name": "Administrator", + "author_class": "User", + "target_id": 29, + "target_type": "Project", + "target_details": "example-project", + "custom_message": { + "protocol": "ssh", + "action": "git-receive-pack" + }, + "ip_address": "127.0.0.1", + "entity_path": "example-group/example-project" + }, + "ip_address": "127.0.0.1", + "author_name": "Administrator", + "entity_path": "example-group/example-project", + "target_details": "example-project", + "created_at": "2022-02-23T06:23:08.746Z", + "target_type": "Project", + "target_id": 29, + "event_type": "repository_git_operation" +} +``` + +### Example payloads for SSH events with Deploy Key + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/363876) in GitLab 15.3. + +Fetch: + +```json +{ + "id": 1, + "author_id": -3, + "entity_id": 29, + "entity_type": "Project", + "details": { + "author_name": "deploy-key-name", + "author_class": "DeployKey", + "target_id": 29, + "target_type": "Project", + "target_details": "example-project", + "custom_message": { + "protocol": "ssh", + "action": "git-upload-pack" + }, + "ip_address": "127.0.0.1", + "entity_path": "example-group/example-project" + }, + "ip_address": "127.0.0.1", + "author_name": "deploy-key-name", + "entity_path": "example-group/example-project", + "target_details": "example-project", + "created_at": "2022-07-26T05:43:53.662Z", + "target_type": "Project", + "target_id": 29, + "event_type": "repository_git_operation" +} +``` + +### Example payloads for HTTP and HTTPS events + +Fetch: + +```json +{ + "id": 1, + "author_id": 1, + "entity_id": 29, + "entity_type": "Project", + "details": { + "author_name": "Administrator", + "author_class": "User", + "target_id": 29, + "target_type": "Project", + "target_details": "example-project", + "custom_message": { + "protocol": "http", + "action": "git-upload-pack" + }, + "ip_address": "127.0.0.1", + "entity_path": "example-group/example-project" + }, + "ip_address": "127.0.0.1", + "author_name": "Administrator", + "entity_path": "example-group/example-project", + "target_details": "example-project", + "created_at": "2022-02-23T06:25:43.938Z", + "target_type": "Project", + "target_id": 29, + "event_type": "repository_git_operation" +} +``` + +Push: + +```json +{ + "id": 1, + "author_id": 1, + "entity_id": 29, + "entity_type": "Project", + "details": { + "author_name": "Administrator", + "author_class": "User", + "target_id": 29, + "target_type": "Project", + "target_details": "example-project", + "custom_message": { + "protocol": "http", + "action": "git-receive-pack" + }, + "ip_address": "127.0.0.1", + "entity_path": "example-group/example-project" + }, + "ip_address": "127.0.0.1", + "author_name": "Administrator", + "entity_path": "example-group/example-project", + "target_details": "example-project", + "created_at": "2022-02-23T06:26:29.294Z", + "target_type": "Project", + "target_id": 29, + "event_type": "repository_git_operation" +} +``` + +### Example payloads for HTTP and HTTPS events with Deploy Token + +Fetch: + +```json +{ + "id": 1, + "author_id": -2, + "entity_id": 22, + "entity_type": "Project", + "details": { + "author_name": "deploy-token-name", + "author_class": "DeployToken", + "target_id": 22, + "target_type": "Project", + "target_details": "example-project", + "custom_message": { + "protocol": "http", + "action": "git-upload-pack" + }, + "ip_address": "127.0.0.1", + "entity_path": "example-group/example-project" + }, + "ip_address": "127.0.0.1", + "author_name": "deploy-token-name", + "entity_path": "example-group/example-project", + "target_details": "example-project", + "created_at": "2022-07-26T05:46:25.850Z", + "target_type": "Project", + "target_id": 22, + "event_type": "repository_git_operation" +} +``` + +### Example payloads for events from GitLab UI download button + +Fetch: + +```json +{ + "id": 1, + "author_id": 99, + "entity_id": 29, + "entity_type": "Project", + "details": { + "custom_message": "Repository Download Started", + "author_name": "example_username", + "author_class": "User", + "target_id": 29, + "target_type": "Project", + "target_details": "example-group/example-project", + "ip_address": "127.0.0.1", + "entity_path": "example-group/example-project" + }, + "ip_address": "127.0.0.1", + "author_name": "example_username", + "entity_path": "example-group/example-project", + "target_details": "example-group/example-project", + "created_at": "2022-02-23T06:27:17.873Z", + "target_type": "Project", + "target_id": 29, + "event_type": "repository_git_operation" +} +``` + +## Audit event streaming on merge request approval actions + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/271162) in GitLab 14.9. + +Stream audit events that relate to merge approval actions performed in a project. + +### Headers + +Headers are formatted as follows: + +```plaintext +POST /logs HTTP/1.1 +Host: <DESTINATION_HOST> +Content-Type: application/x-www-form-urlencoded +X-Gitlab-Event-Streaming-Token: <DESTINATION_TOKEN> +X-Gitlab-Audit-Event-Type: audit_operation +``` + +### Example payload + +```json +{ + "id": 1, + "author_id": 1, + "entity_id": 6, + "entity_type": "Project", + "details": { + "author_name": "example_username", + "target_id": 20, + "target_type": "MergeRequest", + "target_details": "merge request title", + "custom_message": "Approved merge request", + "ip_address": "127.0.0.1", + "entity_path": "example-group/example-project" + }, + "ip_address": "127.0.0.1", + "author_name": "example_username", + "entity_path": "example-group/example-project", + "target_details": "merge request title", + "created_at": "2022-03-09T06:53:11.181Z", + "target_type": "MergeRequest", + "target_id": 20, + "event_type": "audit_operation" +} +``` + +## Audit event streaming on merge request create actions + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/90911) in GitLab 15.2. + +Stream audit events that relate to merge request create actions using the `/logs` endpoint. + +Send API requests that contain the `X-Gitlab-Audit-Event-Type` header with value `merge_request_create`. GitLab responds with JSON payloads with an +`event_type` field set to `merge_request_create`. + +### Headers + +Headers are formatted as follows: + +```plaintext +POST /logs HTTP/1.1 +Host: <DESTINATION_HOST> +Content-Type: application/x-www-form-urlencoded +X-Gitlab-Audit-Event-Type: merge_request_create +X-Gitlab-Event-Streaming-Token: <DESTINATION_TOKEN> +``` + +### Example payload + +```json +{ + "id": 1, + "author_id": 1, + "entity_id": 24, + "entity_type": "Project", + "details": { + "author_name": "example_user", + "target_id": 132, + "target_type": "MergeRequest", + "target_details": "Update test.md", + "custom_message": "Added merge request", + "ip_address": "127.0.0.1", + "entity_path": "example-group/example-project" + }, + "ip_address": "127.0.0.1", + "author_name": "Administrator", + "entity_path": "example-group/example-project", + "target_details": "Update test.md", + "created_at": "2022-07-04T00:19:22.675Z", + "target_type": "MergeRequest", + "target_id": 132, + "event_type": "merge_request_create" +} +``` + +## Audit event streaming on project fork actions + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/90916) in GitLab 15.2. + +Stream audit events that relate to project fork actions using the `/logs` endpoint. + +Send API requests that contain the `X-Gitlab-Audit-Event-Type` header with value `project_fork_operation`. GitLab responds with JSON payloads with an +`event_type` field set to `project_fork_operation`. + +### Headers + +Headers are formatted as follows: + +```plaintext +POST /logs HTTP/1.1 +Host: <DESTINATION_HOST> +Content-Type: application/x-www-form-urlencoded +X-Gitlab-Audit-Event-Type: project_fork_operation +X-Gitlab-Event-Streaming-Token: <DESTINATION_TOKEN> +``` + +### Example payload + +```json +{ + "id": 1, + "author_id": 1, + "entity_id": 24, + "entity_type": "Project", + "details": { + "author_name": "example_username", + "target_id": 24, + "target_type": "Project", + "target_details": "example-project", + "custom_message": "Forked project to another-group/example-project-forked", + "ip_address": "127.0.0.1", + "entity_path": "example-group/example-project" + }, + "ip_address": "127.0.0.1", + "author_name": "example_username", + "entity_path": "example-group/example-project", + "target_details": "example-project", + "created_at": "2022-06-30T03:43:35.384Z", + "target_type": "Project", + "target_id": 24, + "event_type": "project_fork_operation" +} +``` + +## Audit event streaming on project group link actions + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/90955) in GitLab 15.2. + +Stream audit events that relate to project group link creation, updates, and deletion using the `/logs` endpoint. + +Send API requests that contain the `X-Gitlab-Audit-Event-Type` header with value of either: + +- `project_group_link_create`. +- `project_group_link_update`. +- `project_group_link_destroy`. + +GitLab responds with JSON payloads with an `event_type` field set to either: + +- `project_group_link_create`. +- `project_group_link_update`. +- `project_group_link_destroy`. + +### Example Headers + +Headers are formatted as follows: + +```plaintext +POST /logs HTTP/1.1 +Host: <DESTINATION_HOST> +Content-Type: application/x-www-form-urlencoded +X-Gitlab-Audit-Event-Type: project_group_link_create +X-Gitlab-Event-Streaming-Token: <DESTINATION_TOKEN> +``` + +### Example payload for project group link create + +```json +{ + "id": 1, + "author_id": 1, + "entity_id": 24, + "entity_type": "Project", + "details": { + "author_name": "example-user", + "target_id": 31, + "target_type": "Group", + "target_details": "another-group", + "custom_message": "Added project group link", + "ip_address": "127.0.0.1", + "entity_path": "example-group/example-project" + }, + "ip_address": "127.0.0.1", + "author_name": "example-user", + "entity_path": "example-group/example-project", + "target_details": "another-group", + "created_at": "2022-07-04T00:43:09.318Z", + "target_type": "Group", + "target_id": 31, + "event_type": "project_group_link_create" +} +``` + +### Example payload for project group link update + +```json +{ + "id": 1, + "author_id": 1, + "entity_id": 24, + "entity_type": "Project", + "details": { + "author_name": "example-user", + "target_id": 31, + "target_type": "Group", + "target_details": "another-group", + "custom_message": "Changed project group link profile group_access from Developer to Guest", + "ip_address": "127.0.0.1", + "entity_path": "example-group/example-project" + }, + "ip_address": "127.0.0.1", + "author_name": "example-user", + "entity_path": "example-group/example-project", + "target_details": "another-group", + "created_at": "2022-07-04T00:43:28.328Z", + "target_type": "Group", + "target_id": 31, + "event_type": "project_group_link_update" +} +``` + +### Example payload for project group link delete + +```json +{ + "id": 1, + "author_id": 1, + "entity_id": 24, + "entity_type": "Project", + "details": { + "author_name": "example-user", + "target_id": 31, + "target_type": "Group", + "target_details": "another-group", + "custom_message": "Removed project group link", + "ip_address": "127.0.0.1", + "entity_path": "example-group/example-project" + }, + "ip_address": "127.0.0.1", + "author_name": "example-user", + "entity_path": "example-group/example-project", + "target_details": "another-group", + "created_at": "2022-07-04T00:42:56.279Z", + "target_type": "Group", + "target_id": 31, + "event_type": "project_group_link_destroy" +} +``` + +## Audit event streaming on invalid merge request approver state + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/374566) in GitLab 15.5. + +Stream audit events that relate to invalid merge request approver states in a project. + +### Headers + +Headers are formatted as follows: + +```plaintext +POST /logs HTTP/1.1 +Host: <DESTINATION_HOST> +Content-Type: application/x-www-form-urlencoded +X-Gitlab-Event-Streaming-Token: <DESTINATION_TOKEN> +X-Gitlab-Audit-Event-Type: audit_operation +``` + +### Example payload + +```json +{ + "id": 1, + "author_id": 1, + "entity_id": 6, + "entity_type": "Project", + "details": { + "author_name": "example_username", + "target_id": 20, + "target_type": "MergeRequest", + "target_details": { title: "Merge request title", iid: "Merge request iid", id: "Merge request id" }, + "custom_message": "Invalid merge request approver rules", + "ip_address": "127.0.0.1", + "entity_path": "example-group/example-project" + }, + "ip_address": "127.0.0.1", + "author_name": "example_username", + "entity_path": "example-group/example-project", + "target_details": "merge request title", + "created_at": "2022-03-09T06:53:11.181Z", + "target_type": "MergeRequest", + "target_id": 20, + "event_type": "audit_operation" +} +``` diff --git a/doc/administration/audit_event_streaming/graphql_api.md b/doc/administration/audit_event_streaming/graphql_api.md new file mode 100644 index 00000000000..f5a31f073dc --- /dev/null +++ b/doc/administration/audit_event_streaming/graphql_api.md @@ -0,0 +1,442 @@ +--- +stage: Govern +group: Compliance +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Audit event streaming GraphQL API **(ULTIMATE)** + +> - API [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332747) in GitLab 14.5 [with a flag](../feature_flags.md) named `ff_external_audit_events_namespace`. Disabled by default. +> - API [Enabled on GitLab.com and by default on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/338939) in GitLab 14.7. +> - API [Feature flag `ff_external_audit_events_namespace`](https://gitlab.com/gitlab-org/gitlab/-/issues/349588) removed in GitLab 14.8. +> - Custom HTTP headers API [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/361216) in GitLab 15.1 [with a flag](../feature_flags.md) named `streaming_audit_event_headers`. Disabled by default. +> - Custom HTTP headers API [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/362941) in GitLab 15.2. +> - Custom HTTP headers API [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/366524) in GitLab 15.3. [Feature flag `streaming_audit_event_headers`](https://gitlab.com/gitlab-org/gitlab/-/issues/362941) removed. +> - User-specified verification token API support [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/360813) in GitLab 15.4. +> - APIs for custom HTTP headers for instance level streaming destinations [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/404560) in GitLab 16.1 [with a flag](../feature_flags.md) named `ff_external_audit_events`. Disabled by default. + +Audit event streaming destinations can be maintained using a GraphQL API. + +## Add a new streaming destination + +Add a new streaming destination to top-level groups or an entire instance. + +WARNING: +Streaming destinations receive **all** audit event data, which could include sensitive information. Make sure you trust the streaming destination. + +### Top-level group streaming destinations + +Prerequisites: + +- Owner role for a top-level group. + +To enable streaming and add a destination to a top-level group, use the `externalAuditEventDestinationCreate` mutation. + +```graphql +mutation { + externalAuditEventDestinationCreate(input: { destinationUrl: "https://mydomain.io/endpoint/ingest", groupPath: "my-group" } ) { + errors + externalAuditEventDestination { + id + destinationUrl + verificationToken + group { + name + } + } + } +} +``` + +You can optionally specify your own verification token (instead of the default GitLab-generated one) using the GraphQL +`externalAuditEventDestinationCreate` +mutation. Verification token length must be within 16 to 24 characters and trailing whitespace are not trimmed. You +should set a cryptographically random and unique value. For example: + +```graphql +mutation { + externalAuditEventDestinationCreate(input: { destinationUrl: "https://mydomain.io/endpoint/ingest", groupPath: "my-group", verificationToken: "unique-random-verification-token-here" } ) { + errors + externalAuditEventDestination { + id + destinationUrl + verificationToken + group { + name + } + } + } +} +``` + +Event streaming is enabled if: + +- The returned `errors` object is empty. +- The API responds with `200 OK`. + +You can add an HTTP header using the GraphQL `auditEventsStreamingHeadersCreate` mutation. You can retrieve the +destination ID by [listing all the streaming destinations](#list-streaming-destinations) for the group or from the +mutation above. + +```graphql +mutation { + auditEventsStreamingHeadersCreate(input: { destinationId: "gid://gitlab/AuditEvents::ExternalAuditEventDestination/24601", key: "foo", value: "bar" }) { + errors + } +} +``` + +The header is created if the returned `errors` object is empty. + +### Instance streaming destinations + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/335175) in GitLab 16.0 [with a flag](../feature_flags.md) named `ff_external_audit_events`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../feature_flags.md) named +`ff_external_audit_events`. On GitLab.com, this feature is not available. The feature is not ready for production use. + +Prerequisites: + +- Administrator access on the instance. + +To enable streaming and add a destination, use the +`instanceExternalAuditEventDestinationCreate` mutation in the GraphQL API. + +```graphql +mutation { + instanceExternalAuditEventDestinationCreate(input: { destinationUrl: "https://mydomain.io/endpoint/ingest"}) { + errors + instanceExternalAuditEventDestination { + destinationUrl + id + verificationToken + } + } +} +``` + +Event streaming is enabled if: + +- The returned `errors` object is empty. +- The API responds with `200 OK`. + +Instance administrators can add an HTTP header using the GraphQL `auditEventsStreamingInstanceHeadersCreate` mutation. You can retrieve the destination ID +by [listing all the streaming destinations](#list-streaming-destinations) for the instance or from the mutation above. + +```graphql +mutation { + auditEventsStreamingInstanceHeadersCreate(input: + { + destinationId: "gid://gitlab/AuditEvents::InstanceExternalAuditEventDestination/42", + key: "foo", + value: "bar" + }) { + errors + header { + id + key + value + } + } +} +``` + +The header is created if the returned `errors` object is empty. + +## List streaming destinations + +List new streaming destinations for top-level groups or an entire instance. + +### Top-level group streaming destinations + +Prerequisites: + +- Owner role for a top-level group. + +You can view a list of streaming destinations for a top-level group using the `externalAuditEventDestinations` query +type. + +```graphql +query { + group(fullPath: "my-group") { + id + externalAuditEventDestinations { + nodes { + destinationUrl + verificationToken + id + headers { + nodes { + key + value + id + } + } + eventTypeFilters + } + } + } +} +``` + +If the resulting list is empty, then audit streaming is not enabled for that group. + +### Instance streaming destinations + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/335175) in GitLab 16.0 [with a flag](../feature_flags.md) named `ff_external_audit_events`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../feature_flags.md) named +`ff_external_audit_events`. On GitLab.com, this feature is not available. The feature is not ready for production use. + +Prerequisites: + +- Administrator access on the instance. + +To view a list of streaming destinations for an instance, use the +`instanceExternalAuditEventDestinations` query type. + +```graphql +query { + instanceExternalAuditEventDestinations { + nodes { + id + destinationUrl + verificationToken + headers { + nodes { + id + key + value + } + } + } + } +} +``` + +If the resulting list is empty, then audit streaming is not enabled for the instance. + +You need the ID values returned by this query for the update and delete mutations. + +## Update streaming destinations + +Update streaming destinations for a top-level group or an entire instance. + +### Top-level group streaming destinations + +Prerequisites: + +- Owner role for a top-level group. + +Users with the Owner role for a group can update streaming destinations' custom HTTP headers using the +`auditEventsStreamingHeadersUpdate` mutation type. You can retrieve the custom HTTP headers ID +by [listing all the custom HTTP headers](#list-streaming-destinations) for the group. + +```graphql +mutation { + externalAuditEventDestinationDestroy(input: { id: destination }) { + errors + } +} +``` + +Streaming destination is updated if: + +- The returned `errors` object is empty. +- The API responds with `200 OK`. + +Group owners can remove an HTTP header using the GraphQL `auditEventsStreamingHeadersDestroy` mutation. You can retrieve the header ID +by [listing all the custom HTTP headers](#list-streaming-destinations) for the group. + +```graphql +mutation { + auditEventsStreamingHeadersDestroy(input: { headerId: "gid://gitlab/AuditEvents::Streaming::Header/1" }) { + errors + } +} +``` + +The header is deleted if the returned `errors` object is empty. + +### Instance streaming destinations + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/335175) in GitLab 16.0 [with a flag](../feature_flags.md) named `ff_external_audit_events`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../feature_flags.md) named +`ff_external_audit_events`. On GitLab.com, this feature is not available. The feature is not ready for production use. + +Prerequisites: + +- Administrator access on the instance. + +To update streaming destinations for an instance, use the +`instanceExternalAuditEventDestinationUpdate` mutation type. You can retrieve the destination ID +by [listing all the external destinations](#list-streaming-destinations) for the instance. + +```graphql +mutation { + instanceExternalAuditEventDestinationUpdate(input: { id: "gid://gitlab/AuditEvents::InstanceExternalAuditEventDestination/1", destinationUrl: "https://www.new-domain.com/webhook"}) { + errors + instanceExternalAuditEventDestination { + destinationUrl + id + verificationToken + } + } +} +``` + +Streaming destination is updated if: + +- The returned `errors` object is empty. +- The API responds with `200 OK`. + +Instance administrators can update streaming destinations custom HTTP headers using the +`auditEventsStreamingInstanceHeadersUpdate` mutation type. You can retrieve the custom HTTP headers ID +by [listing all the custom HTTP headers](#list-streaming-destinations) for the instance. + +```graphql +mutation { + auditEventsStreamingInstanceHeadersUpdate(input: { headerId: "gid://gitlab/AuditEvents::Streaming::InstanceHeader/2", key: "new-key", value: "new-value" }) { + errors + header { + id + key + value + } + } +} +``` + +The header is updated if the returned `errors` object is empty. + +## Delete streaming destinations + +Delete streaming destinations for a top-level group or an entire instance. + +When the last destination is successfully deleted, streaming is disabled for the group or the instance. + +### Top-level group streaming destinations + +Prerequisites: + +- Owner role for a top-level group. + +Users with the Owner role for a group can delete streaming destinations using the +`externalAuditEventDestinationDestroy` mutation type. You can retrieve the destinations ID +by [listing all the streaming destinations](#list-streaming-destinations) for the group. + +```graphql +mutation { + externalAuditEventDestinationDestroy(input: { id: destination }) { + errors + } +} +``` + +Streaming destination is deleted if: + +- The returned `errors` object is empty. +- The API responds with `200 OK`. + +Group owners can remove an HTTP header using the GraphQL `auditEventsStreamingHeadersDestroy` mutation. You can retrieve the header ID +by [listing all the custom HTTP headers](#list-streaming-destinations) for the group. + +```graphql +mutation { + auditEventsStreamingHeadersDestroy(input: { headerId: "gid://gitlab/AuditEvents::Streaming::Header/1" }) { + errors + } +} +``` + +The header is deleted if the returned `errors` object is empty. + +### Instance streaming destinations + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/335175) in GitLab 16.0 [with a flag](../feature_flags.md) named `ff_external_audit_events`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../feature_flags.md) named +`ff_external_audit_events`. On GitLab.com, this feature is not available. The feature is not ready for production use. + +Prerequisites: + +- Administrator access on the instance. + +To delete streaming destinations, use the +`instanceExternalAuditEventDestinationDestroy` mutation type. You can retrieve the destinations ID +by [listing all the streaming destinations](#list-streaming-destinations) for the instance. + +```graphql +mutation { + instanceExternalAuditEventDestinationDestroy(input: { id: "gid://gitlab/AuditEvents::InstanceExternalAuditEventDestination/1" }) { + errors + } +} +``` + +Streaming destination is deleted if: + +- The returned `errors` object is empty. +- The API responds with `200 OK`. + +## Event type filters + +> Event type filters API [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/344845) in GitLab 15.7. + +When this feature is enabled for a group, you can use an API to permit users to filter streamed audit events per destination. +If the feature is enabled with no filters, the destination receives all audit events. + +A streaming destination that has an event type filter set has a **filtered** (**{filter}**) label. + +### Use the API to add an event type filter + +Prerequisites: + +- You must have the Owner role for the group. + +You can add a list of event type filters using the `auditEventsStreamingDestinationEventsAdd` query type: + +```graphql +mutation { + auditEventsStreamingDestinationEventsAdd(input: { + destinationId: "gid://gitlab/AuditEvents::ExternalAuditEventDestination/1", + eventTypeFilters: ["list of event type filters"]}){ + errors + eventTypeFilters + } +} +``` + +Event type filters are added if: + +- The returned `errors` object is empty. +- The API responds with `200 OK`. + +### Use the API to remove an event type filter + +Prerequisites: + +- You must have the Owner role for the group. + +You can remove a list of event type filters using the `auditEventsStreamingDestinationEventsRemove` query type: + +```graphql +mutation { + auditEventsStreamingDestinationEventsRemove(input: { + destinationId: "gid://gitlab/AuditEvents::ExternalAuditEventDestination/1", + eventTypeFilters: ["list of event type filters"] + }){ + errors + } +} +``` + +Event type filters are removed if: + +- The returned `errors` object is empty. +- The API responds with `200 OK`. diff --git a/doc/administration/audit_event_streaming/index.md b/doc/administration/audit_event_streaming/index.md new file mode 100644 index 00000000000..44c6cff7455 --- /dev/null +++ b/doc/administration/audit_event_streaming/index.md @@ -0,0 +1,302 @@ +--- +stage: Govern +group: Compliance +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Audit event streaming **(ULTIMATE)** + +> - UI [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336411) in GitLab 14.9. +> - [Subgroup events recording](https://gitlab.com/gitlab-org/gitlab/-/issues/366878) fixed in GitLab 15.2. +> - Custom HTTP headers UI [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/361630) in GitLab 15.2 [with a flag](../feature_flags.md) named `custom_headers_streaming_audit_events_ui`. Disabled by default. +> - Custom HTTP headers UI [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/365259) in GitLab 15.3. [Feature flag `custom_headers_streaming_audit_events_ui`](https://gitlab.com/gitlab-org/gitlab/-/issues/365259) removed. +> - [Improved user experience](https://gitlab.com/gitlab-org/gitlab/-/issues/367963) in GitLab 15.3. + +Users can set a streaming destination for a top-level group or instance to receive all audit events about the group, subgroups, and +projects as structured JSON. + +Top-level group owners and instance administrators can manage their audit logs in third-party systems. Any service that can receive +structured JSON data can be used as the streaming destination. + +Each streaming destination can have up to 20 custom HTTP headers included with each streamed event. + +NOTE: +GitLab can stream a single event more than once to the same destination. Use the `id` key in the payload to deduplicate incoming data. + +## Add a new streaming destination + +Add a new streaming destination to top-level groups or an entire instance. + +WARNING: +Streaming destinations receive **all** audit event data, which could include sensitive information. Make sure you trust the streaming destination. + +### Top-level group streaming destinations + +Prerequisites: + +- Owner role for a top-level group. + +To add streaming destinations to a top-level group: + +1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. Select **Secure > Audit events**. +1. On the main area, select **Streams** tab. +1. Select **Add streaming destination** to show the section for adding destinations. +1. Enter the destination URL to add. +1. Optional. Locate the **Custom HTTP headers** table. +1. Ignore the **Active** checkbox because it isn't functional. To track progress on adding functionality to the + **Active** checkbox, see [issue 367509](https://gitlab.com/gitlab-org/gitlab/-/issues/367509). +1. Select **Add header** to create a new name and value pair. Enter as many name and value pairs as required. You can add up to + 20 headers per streaming destination. +1. After all headers have been filled out, select **Add** to add the new streaming destination. + +### Instance streaming destinations + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/398107) in GitLab 16.1 [with a flag](../feature_flags.md) named `instance_streaming_audit_events`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../feature_flags.md) named +`instance_streaming_audit_events`. On GitLab.com, this feature is not available. The feature is not ready for production use. + +Prerequisites: + +- Administrator access on the instance. + +To add a streaming destination for an instance: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. On the left sidebar, select **Monitoring > Audit Events**. +1. On the main area, select **Streams** tab. +1. Select **Add streaming destination** to show the section for adding destinations. +1. Enter the destination URL to add. +1. Select **Add** to add the new streaming destination. + +## List streaming destinations + +List new streaming destinations for top-level groups or an entire instance. + +### For top-level group streaming destinations + +Prerequisites: + +- Owner role for a group. + +To list the streaming destinations for a top-level group: + +1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. Select **Secure > Audit events**. +1. On the main area, select **Streams** tab. +1. To the right of the item, select **Edit** (**{pencil}**) to see all the custom HTTP headers. + +### For instance streaming destinations + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/398107) in GitLab 16.1 [with a flag](../feature_flags.md) named `instance_streaming_audit_events`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../feature_flags.md) named +`instance_streaming_audit_events`. On GitLab.com, this feature is not available. The feature is not ready for production use. + +Prerequisites: + +- Administrator access on the instance. + +To list the streaming destinations for an instance: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. On the left sidebar, select **Monitoring > Audit Events**. +1. On the main area, select **Streams** tab. + +## Update streaming destinations + +Update streaming destinations for a top-level group or an entire instance. + +### Top-level group streaming destinations + +Prerequisites: + +- Owner role for a group. + +To update a streaming destination's custom HTTP headers: + +1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. Select **Secure > Audit events**. +1. On the main area, select **Streams** tab. +1. To the right of the item, select **Edit** (**{pencil}**). +1. Locate the **Custom HTTP headers** table. +1. Locate the header that you wish to update. +1. Ignore the **Active** checkbox because it isn't functional. To track progress on adding functionality to the + **Active** checkbox, see [issue 367509](https://gitlab.com/gitlab-org/gitlab/-/issues/367509). +1. Select **Add header** to create a new name and value pair. Enter as many name and value pairs as required. You can add up to + 20 headers per streaming destination. +1. Select **Save** to update the streaming destination. + +## Delete streaming destinations + +Delete streaming destinations for a top-level group or an entire instance. When the last destination is successfully +deleted, streaming is disabled for the group or the instance. + +### Top-level group streaming destinations + +Prerequisites: + +- Owner role for a group. + +To delete a streaming destination: + +1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. Select **Secure > Audit events**. +1. On the main area, select the **Streams** tab. +1. To the right of the item, select **Delete** (**{remove}**). + +To delete only the custom HTTP headers for a streaming destination: + +1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. Select **Secure > Audit events**. +1. On the main area, select the **Streams** tab. +1. To the right of the item, **Edit** (**{pencil}**). +1. Locate the **Custom HTTP headers** table. +1. Locate the header that you wish to remove. +1. To the right of the header, select **Delete** (**{remove}**). +1. Select **Save** to update the streaming destination. + +### Instance streaming destinations + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/398107) in GitLab 16.1 [with a flag](../feature_flags.md) named `instance_streaming_audit_events`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../feature_flags.md) named +`instance_streaming_audit_events`. On GitLab.com, this feature is not available. The feature is not ready for production use. + +Prerequisites: + +- Administrator access on the instance. + +To delete the streaming destinations for an instance: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. On the left sidebar, select **Monitoring > Audit Events**. +1. On the main area, select the **Streams** tab. +1. To the right of the item, select **Delete** (**{remove}**). + +## Verify event authenticity + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345424) in GitLab 14.8. + +Each streaming destination has a unique verification token (`verificationToken`) that can be used to verify the authenticity of the event. This +token is either specified by the Owner or generated automatically when the event destination is created and cannot be changed. + +Each streamed event contains the verification token in the `X-Gitlab-Event-Streaming-Token` HTTP header that can be verified against +the destination's value when [listing streaming destinations](#list-streaming-destinations). + +### Top-level group streaming destinations + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/360814) in GitLab 15.2. + +Prerequisites: + +- Owner role for a group. + +To list streaming destinations and see the verification tokens: + +1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. Select **Secure > Audit events**. +1. On the main area, select the **Streams**. +1. View the verification token on the right side of each item. + +### Instance streaming destinations + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/398107) in GitLab 16.1 [with a flag](../feature_flags.md) named `instance_streaming_audit_events`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../feature_flags.md) named +`instance_streaming_audit_events`. On GitLab.com, this feature is not available. The feature is not ready for production use. + +Prerequisites: + +- Administrator access on the instance. + +To list streaming destinations for an instance and see the verification tokens: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. On the left sidebar, select **Monitoring > Audit Events**. +1. On the main area, select the **Streams**. +1. View the verification token on the right side of each item. + +## Override default content type header + +By default, streaming destinations use a `content-type` header of `application/x-www-form-urlencoded`. However, you +might want to set the `content-type` header to something else. For example ,`application/json`. + +To override the `content-type` header default value for either a top-level group streaming destination or an instance +streaming destination, use either the [GitLab UI](#update-streaming-destinations) or using the +[GraphQL API](graphql_api.md#update-streaming-destinations). + +## Payload schema + +> Documentation for an audit event streaming schema was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/358149) in GitLab 15.3. + +Streamed audit events have a predictable schema in the body of the response. + +| Field | Description | Notes | +|------------------|------------------------------------------------------------|-----------------------------------------------------------------------------------| +| `author_id` | User ID of the user who triggered the event | | +| `author_name` | Human-readable name of the author that triggered the event | Helpful when the author no longer exists | +| `created_at` | Timestamp when event was triggered | | +| `details` | JSON object containing additional metadata | Has no defined schema but often contains additional information about an event | +| `entity_id` | ID of the audit event's entity | | +| `entity_path` | Full path of the entity affected by the auditable event | | +| `entity_type` | String representation of the type of entity | Acceptable values include `User`, `Group`, and `Key`. This list is not exhaustive | +| `event_type` | String representation of the type of audit event | | +| `id` | Unique identifier for the audit event | Can be used for deduplication if required | +| `ip_address` | IP address of the host used to trigger the event | | +| `target_details` | Additional details about the target | | +| `target_id` | ID of the audit event's target | | +| `target_type` | String representation of the target's type | | + +### JSON payload schema + +```json +{ + "properties": { + "id": { + "type": "string" + }, + "author_id": { + "type": "integer" + }, + "author_name": { + "type": "string" + }, + "details": {}, + "ip_address": { + "type": "string" + }, + "entity_id": { + "type": "integer" + }, + "entity_path": { + "type": "string" + }, + "entity_type": { + "type": "string" + }, + "event_type": { + "type": "string" + }, + "target_id": { + "type": "integer" + }, + "target_type": { + "type": "string" + }, + "target_details": { + "type": "string" + }, + }, + "type": "object" +} +``` diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index 50bd943b8e4..e924da39145 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -69,7 +69,8 @@ You can view audit events from user actions across an entire GitLab instance. To view instance audit events: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Audit Events**. ### Export to CSV @@ -80,7 +81,8 @@ To view instance audit events: You can export the current view (including filters) of your instance audit events as a CSV file. To export the instance audit events to CSV: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Audit Events**. 1. Select the available search [filters](#filter-audit-events). 1. Select **Export as CSV**. @@ -393,6 +395,8 @@ The following user actions on a GitLab instance generate instance audit events: - User was unblocked using the Admin Area or API. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/115727) in GitLab 15.11. - User was banned using the Admin Area or API. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/116103) in GitLab 15.11. - User was unbanned using the Admin Area or API. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/116221) in GitLab 15.11. +- User was deactivated using the Admin Area or API. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/117776) in GitLab 16.0. +- User was activated using the Admin Area or API. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121708) in GitLab 16.1. Instance events can also be accessed using the [Instance Audit Events API](../api/audit_events.md#instance-audit-events). diff --git a/doc/administration/auditor_users.md b/doc/administration/auditor_users.md index 38c3a089756..3b6992c92e0 100644 --- a/doc/administration/auditor_users.md +++ b/doc/administration/auditor_users.md @@ -31,7 +31,8 @@ To create a new user account with auditor access (or change an existing user): To create a user account with auditor access: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Overview > Users**. 1. Create a new user or edit an existing one. Set **Access Level** to **Auditor**. 1. If you created a user, select **Create user**. For an existing user, select **Save changes**. diff --git a/doc/administration/auth/atlassian.md b/doc/administration/auth/atlassian.md index 45617b9965c..8525b3e9b98 100644 --- a/doc/administration/auth/atlassian.md +++ b/doc/administration/auth/atlassian.md @@ -29,13 +29,13 @@ To enable the Atlassian OmniAuth provider for passwordless authentication you mu 1. On your GitLab server, open the configuration file: - For Omnibus GitLab installations: + For Linux package installations: ```shell sudo editor /etc/gitlab/gitlab.rb ``` - For installations from source: + For self-compiled installations: ```shell sudo -u git -H editor /home/git/gitlab/config/gitlab.yml @@ -47,7 +47,7 @@ To enable the Atlassian OmniAuth provider for passwordless authentication you mu GitLab account. 1. Add the provider configuration for Atlassian: - For Omnibus GitLab installations: + For Linux package installations: ```ruby gitlab_rails['omniauth_providers'] = [ @@ -61,7 +61,7 @@ To enable the Atlassian OmniAuth provider for passwordless authentication you mu ] ``` - For installations from source: + For self-compiled installations: ```yaml - { name: "atlassian_oauth2", @@ -76,8 +76,8 @@ To enable the Atlassian OmniAuth provider for passwordless authentication you mu 1. Save the configuration file. 1. For the changes to take effect: - - If you installed via Omnibus, [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - - If you installed from source, [restart GitLab](../restart_gitlab.md#installations-from-source). + - If you installed using the Linux package, [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). + - If you self-compiled your installation, [restart GitLab](../restart_gitlab.md#installations-from-source). On the sign-in page there should now be an Atlassian icon below the regular sign in form. Select the icon to begin the authentication process. diff --git a/doc/administration/auth/cognito.md b/doc/administration/auth/cognito.md index cfac958e297..8c8abf1524f 100644 --- a/doc/administration/auth/cognito.md +++ b/doc/administration/auth/cognito.md @@ -37,16 +37,14 @@ To enable AWS Cognito as an authentication provider, complete the following step 1. Save changes for the app client settings. 1. Under **Domain name**, include the AWS domain name for your AWS Cognito application. -1. Under **App Clients**, find your app client ID. Select **Show details* to display the app client secret. These values correspond to the OAuth 2.0 Client ID and Client Secret. Save these values. +1. Under **App Clients**, find your app client ID. Select **Show details** to display the app client secret. These values correspond to the OAuth 2.0 Client ID and Client Secret. Save these values. ## Configure GitLab 1. Configure the [common settings](../../integration/omniauth.md#configure-common-settings) to add `cognito` as a single sign-on provider. This enables Just-In-Time account provisioning for users who do not have an existing GitLab account. -1. On your GitLab server, open the configuration file. - - **For Omnibus installations** +1. On your GitLab server, open the configuration file. For Linux package installations: ```shell sudo editor /etc/gitlab/gitlab.rb @@ -90,7 +88,7 @@ To enable AWS Cognito as an authentication provider, complete the following step ``` 1. Save the configuration file. -1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. +1. Save the file and [reconfigure](../restart_gitlab.md#reconfigure-a-linux-package-installation) GitLab for the changes to take effect. Your sign-in page should now display a Cognito option below the regular sign-in form. Select this option to begin the authentication process. diff --git a/doc/administration/auth/crowd.md b/doc/administration/auth/crowd.md index f89e1a00928..08c1f5e7513 100644 --- a/doc/administration/auth/crowd.md +++ b/doc/administration/auth/crowd.md @@ -26,19 +26,19 @@ this provider also allows Crowd authentication for Git-over-https requests. 1. On your GitLab server, open the configuration file. - **Omnibus:** + - Linux package installations: - ```shell - sudo editor /etc/gitlab/gitlab.rb - ``` + ```shell + sudo editor /etc/gitlab/gitlab.rb + ``` - **Source:** + - Self-compiled installations: - ```shell - cd /home/git/gitlab + ```shell + cd /home/git/gitlab - sudo -u git -H editor config/gitlab.yml - ``` + sudo -u git -H editor config/gitlab.yml + ``` 1. Configure the [common settings](../../integration/omniauth.md#configure-common-settings) to add `crowd` as a single sign-on provider. This enables Just-In-Time @@ -46,39 +46,39 @@ this provider also allows Crowd authentication for Git-over-https requests. 1. Add the provider configuration: - **Omnibus:** - - ```ruby - gitlab_rails['omniauth_providers'] = [ - { - name: "crowd", - # label: "Provider name", # optional label for login button, defaults to "Crowd" - args: { - crowd_server_url: "CROWD_SERVER_URL", - application_name: "YOUR_APP_NAME", - application_password: "YOUR_APP_PASSWORD" + - Linux package installations: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + name: "crowd", + # label: "Provider name", # optional label for login button, defaults to "Crowd" + args: { + crowd_server_url: "CROWD_SERVER_URL", + application_name: "YOUR_APP_NAME", + application_password: "YOUR_APP_PASSWORD" + } } - } - ] - ``` + ] + ``` - **Source:** + - Self-compiled installations: - ```yaml - - { name: 'crowd', - # label: 'Provider name', # optional label for login button, defaults to "Crowd" - args: { - crowd_server_url: 'CROWD_SERVER_URL', - application_name: 'YOUR_APP_NAME', - application_password: 'YOUR_APP_PASSWORD' } } - ``` + ```yaml + - { name: 'crowd', + # label: 'Provider name', # optional label for login button, defaults to "Crowd" + args: { + crowd_server_url: 'CROWD_SERVER_URL', + application_name: 'YOUR_APP_NAME', + application_password: 'YOUR_APP_PASSWORD' } } + ``` 1. Change `CROWD_SERVER_URL` to the [base URL of your Crowd server](https://confluence.atlassian.com/crowdkb/how-to-change-the-crowd-base-url-245827278.html). 1. Change `YOUR_APP_NAME` to the application name from Crowd applications page. 1. Change `YOUR_APP_PASSWORD` to the application password you've set. 1. Save the configuration file. -1. [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) (Omnibus GitLab) or [restart](../restart_gitlab.md#installations-from-source) (source installations) for - the changes to take effect. +1. [Reconfigure](../restart_gitlab.md#reconfigure-a-linux-package-installation) (Linux package installations) or + [restart](../restart_gitlab.md#installations-from-source) (self-compiled installations) for the changes to take effect. On the sign in page there should now be a Crowd tab in the sign in form. diff --git a/doc/administration/auth/jwt.md b/doc/administration/auth/jwt.md index 9994b374038..9a74064136a 100644 --- a/doc/administration/auth/jwt.md +++ b/doc/administration/auth/jwt.md @@ -12,13 +12,13 @@ JWT provides you with a secret key for you to use. 1. On your GitLab server, open the configuration file. - For Omnibus GitLab: + For Linux package installations: ```shell sudo editor /etc/gitlab/gitlab.rb ``` - For installations from source: + For self-compiled installations: ```shell cd /home/git/gitlab @@ -30,7 +30,7 @@ JWT provides you with a secret key for you to use. account provisioning for users who do not have an existing GitLab account. 1. Add the provider configuration. - For Omnibus GitLab: + For Linux package installations: ```ruby gitlab_rails['omniauth_providers'] = [ @@ -49,7 +49,7 @@ JWT provides you with a secret key for you to use. ] ``` - For installation from source: + For self-compiled installations: ```yaml - { name: 'jwt', @@ -70,11 +70,14 @@ JWT provides you with a secret key for you to use. For more information on each configuration option refer to the [OmniAuth JWT usage documentation](https://github.com/mbleigh/omniauth-jwt#usage). + WARNING: + Incorrectly configuring these settings can result in an insecure instance. + 1. Change `YOUR_APP_SECRET` to the client secret and set `auth_url` to your redirect URL. 1. Save the configuration file. -1. For the changes to take effect: - - If you installed via Omnibus, [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - - If you installed from source, [restart GitLab](../restart_gitlab.md#installations-from-source). +1. For changes to take effect, if you: + - Used the Linux package to install GitLab, [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). + - Self-compiled your GitLab installation, [restart GitLab](../restart_gitlab.md#installations-from-source). On the sign in page there should now be a JWT icon below the regular sign in form. Select the icon to begin the authentication process. JWT asks the user to diff --git a/doc/administration/auth/ldap/google_secure_ldap.md b/doc/administration/auth/ldap/google_secure_ldap.md index 042a65be500..d484059c79f 100644 --- a/doc/administration/auth/ldap/google_secure_ldap.md +++ b/doc/administration/auth/ldap/google_secure_ldap.md @@ -72,7 +72,7 @@ values obtained during the LDAP client configuration earlier: - `cert`: The `.crt` file text from the downloaded certificate bundle - `key`: The `.key` file text from the downloaded certificate bundle -**For Omnibus installations** +For Linux package installations: 1. Edit `/etc/gitlab/gitlab.rb`: @@ -140,11 +140,9 @@ values obtained during the LDAP client configuration earlier: EOS ``` -1. Save the file and [reconfigure](../../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. +1. Save the file and [reconfigure](../../restart_gitlab.md#reconfigure-a-linux-package-installation) GitLab for the changes to take effect. ---- - -**For installations from source** +For self-compiled installations: 1. Edit `config/gitlab.yml`: diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index 7687f7c9340..a4484da5940 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -233,6 +233,7 @@ These configuration settings are available: | `user_filter` | Filter LDAP users. Format: [RFC 4515](https://www.rfc-editor.org/rfc/rfc4515.html) Note: GitLab does not support `omniauth-ldap`'s custom filter syntax. | **{dotted-circle}** No | Some examples of the `user_filter` field syntax:<br/><br/>- `'(employeeType=developer)'`<br/>- `'(&(objectclass=user)(|(samaccountname=momo)(samaccountname=toto)))'` | | `lowercase_usernames` | If enabled, GitLab converts the name to lower case. | **{dotted-circle}** No | boolean | | `retry_empty_result_with_codes` | An array of LDAP query response code that attempt to retry the operation if the result/content is empty. For Google Secure LDAP, set this value to `[80]`. | **{dotted-circle}** No | `[80]` | +| `attributes` | A hash of attribute mappings to LDAP for GitLab to use ([see attributes section](#attribute-configuration-settings)). | **{dotted-circle}** No | `'attributes' => { 'username' => ['uid'], 'email' => ['mail', 'email'] },` | ### SSL configuration settings @@ -256,6 +257,8 @@ attribute can be either: The user's LDAP sign in is the LDAP attribute [specified as `uid`](#basic-configuration-settings). +You must define the following attributes in an `attributes` hash. + | Setting | Description | Required | Examples | |--------------|-------------|----------|----------| | `username` | Used in paths for the user's own projects (for example, `gitlab.example.com/username/project`) and when mentioning them in issues, merge request and comments (for example, `@username`). If the attribute specified for `username` contains an email address, the GitLab username is part of the email address before the `@`. | **{dotted-circle}** No | `['uid', 'userid', 'sAMAccountName']` | @@ -1034,8 +1037,8 @@ For more information on synchronizing users and groups between LDAP and GitLab, ## Move from LDAP to SAML 1. [Configure SAML](../../../integration/saml.md). Add `auto_link_ldap_user` to: - - [`gitlab.rb` for Omnibus](../../../integration/saml.html?tab=Linux+package+%28Omnibus%29). - - [`values.yml` for Kubernetes](../../../integration/saml.html?tab=Helm+chart+%28Kubernetes%29). + - [`gitlab.rb` for Linux package installations](../../../integration/saml.html?tab=Linux+package+%28Omnibus%29). + - [`values.yml` for Helm chart installations](../../../integration/saml.html?tab=Helm+chart+%28Kubernetes%29). For more information, see the [initial settings for all providers](../../../integration/omniauth.md#configure-initial-settings). 1. Optional. [Disable the LDAP auth from the sign-in page](#disable-ldap-web-sign-in). @@ -1047,7 +1050,7 @@ For more information on synchronizing users and groups between LDAP and GitLab, 1. In the configuration file, change: - `omniauth_auto_link_user` to `saml` only. - `omniauth_auto_link_ldap_user` to false. - - `ldap_enabled` to `false`. + - `ldap_enabled` to `false`. You can also comment out the LDAP provider settings. ## Troubleshooting diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md index 909ab5245ca..9fb3888b995 100644 --- a/doc/administration/auth/ldap/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap/ldap-troubleshooting.md @@ -167,7 +167,8 @@ may see the following message: `Access denied for your LDAP account`. We have a workaround, based on toggling the access level of affected users: -1. As an administrator, on the top bar, select **Main menu > Admin**. +1. As an administrator, on the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Overview > Users**. 1. Select the name of the affected user. 1. In the upper-right corner, select **Edit**. @@ -224,7 +225,8 @@ field contains no data: To resolve this: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, go to **Settings > General**. 1. Expand both of the following: - **Account and limit**. @@ -367,7 +369,8 @@ things to debug the situation. - Ensure the correct [LDAP group link is added to the GitLab group](ldap_synchronization.md#add-group-links). - Check that the user has an LDAP identity: 1. Sign in to GitLab as an administrator user. - 1. On the top bar, select **Main menu > Admin**. + 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. Select **Admin Area**. 1. On the left sidebar, select **Overview > Users**. 1. Search for the user. 1. Open the user by selecting their name. Do not select **Edit**. @@ -733,10 +736,17 @@ cause: To resolve this error, you must apply a new license to the GitLab instance without the web interface: 1. Remove or comment out the GitLab configuration lines for all non-primary LDAP servers. -1. [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so that it temporarily uses only one LDAP server. +1. [Reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) so that it temporarily uses only one LDAP server. 1. Enter the [Rails console and add the license key](../../../user/admin_area/license_file.md#add-a-license-through-the-console). 1. Re-enable the additional LDAP servers in the GitLab configuration and reconfigure GitLab again. +## Users are being removed from group and re-added again + +If a user has been added to a group during group sync, and removed at the next sync, and this has happened repeatedly, make sure that the user doesn't have +multiple or redundant LDAP identities. + +If one of those identities was added for an older LDAP provider that is no longer in use, [remove the `identity` records that relate to the removed LDAP server](#remove-the-identity-records-that-relate-to-the-removed-ldap-server). + ## Debugging Tools ### LDAP check diff --git a/doc/administration/auth/ldap/ldap_synchronization.md b/doc/administration/auth/ldap/ldap_synchronization.md index f32a4af9e27..e4b43feacc2 100644 --- a/doc/administration/auth/ldap/ldap_synchronization.md +++ b/doc/administration/auth/ldap/ldap_synchronization.md @@ -15,7 +15,7 @@ You can change when synchronization occurs. ## User sync -> Preventing LDAP username synchronization [introduced](<https://gitlab.com/gitlab-org/gitlab/-/issues/11336>) in GitLab 15.11. +> Preventing LDAP user's profile name synchronization [introduced](<https://gitlab.com/gitlab-org/gitlab/-/issues/11336>) in GitLab 15.11. Once per day, GitLab runs a worker to check and update GitLab users against LDAP. @@ -45,9 +45,9 @@ The process also updates the following user information: - SSH public keys if `sync_ssh_keys` is set. - Kerberos identity if Kerberos is enabled. -### Synchronize LDAP username +### Synchronize LDAP user's profile name -By default, GitLab synchronizes the LDAP username field. +By default, GitLab synchronizes the LDAP user's profile name field. To prevent this synchronization, you can set `sync_name` to `false`. @@ -501,7 +501,8 @@ When global group memberships lock is enabled: To enable global group memberships lock: 1. [Configure LDAP](index.md#configure-ldap). -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Settings > General**. 1. Expand the **Visibility and access controls** section. 1. Ensure the **Lock memberships to LDAP synchronization** checkbox is selected. @@ -513,7 +514,8 @@ By default, group members with the Owner role can manage [LDAP group synchroniza GitLab administrators can remove this permission from group Owners: 1. [Configure LDAP](index.md#configure-ldap). -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Settings > General**. 1. Expand **Visibility and access controls**. 1. Ensure the **Allow group owners to manage LDAP-related settings** checkbox is not checked. diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md index 106cc6c23eb..23d2ab512db 100644 --- a/doc/administration/auth/oidc.md +++ b/doc/administration/auth/oidc.md @@ -16,7 +16,7 @@ The OpenID Connect provides you with a client's details and secret for you to us 1. On your GitLab server, open the configuration file. - For Omnibus GitLab: + For Linux package installations: ```shell sudo editor /etc/gitlab/gitlab.rb @@ -35,7 +35,7 @@ The OpenID Connect provides you with a client's details and secret for you to us 1. Add the provider configuration. - For Omnibus GitLab: + For Linux package installations: ```ruby gitlab_rails['omniauth_providers'] = [ @@ -63,7 +63,7 @@ The OpenID Connect provides you with a client's details and secret for you to us ] ``` - For Omnibus GitLab with multiple identity providers: + For Linux package installations with multiple identity providers: ```ruby { 'name' => 'openid_connect', @@ -108,7 +108,7 @@ The OpenID Connect provides you with a client's details and secret for you to us NOTE: For more information on using multiple identity providers with OIDC, see [issue 5992](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5992). - For installation from source: + For self-compiled installations: ```yaml - { name: 'openid_connect', # do not change this parameter @@ -184,10 +184,10 @@ The OpenID Connect provides you with a client's details and secret for you to us - `jwks_uri` is the URL to the endpoint where the Token signer publishes its keys. 1. Save the configuration file. -1. For changes to take effect, if you installed GitLab: +1. For changes to take effect, if you: - - With Omnibus, [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - - From source, [restart GitLab](../restart_gitlab.md#installations-from-source). + - Used the Linux package to install GitLab, [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). + - Self-compiled your GitLab installation, [restart GitLab](../restart_gitlab.md#installations-from-source). On the sign in page, you have an OpenID Connect option below the regular sign in form. Select this option to begin the authentication process. The OpenID Connect provider @@ -197,7 +197,7 @@ by the client. You are redirected to GitLab and signed in. ## Example configurations The following configurations illustrate how to set up OpenID with -different providers with Omnibus GitLab. +different providers when using the Linux package installation. ### Configure Google @@ -240,7 +240,7 @@ you need the following information: [Microsoft Quickstart Register an Application](https://learn.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app) documentation to obtain the tenant ID, client ID, and client secret for your app. -Example Omnibus configuration block: +Example configuration block for Linux package installations: ```ruby gitlab_rails['omniauth_providers'] = [ @@ -372,7 +372,7 @@ but `LocalAccounts` authenticates against local Active Directory accounts. Befor ``` 1. Configure the issuer URL with the custom policy used for `signup_signin`. For example, this is - the Omnibus configuration with a custom policy for `b2c_1a_signup_signin`: + the configuration with a custom policy for `b2c_1a_signup_signin` for Linux package installations: ```ruby gitlab_rails['omniauth_providers'] = [ @@ -432,7 +432,7 @@ HS256 or HS358) to sign tokens. Public key encryption algorithms are: 1. Select **Realm Settings > Tokens > Default Signature Algorithm**. 1. Configure the signature algorithm. -Example Omnibus configuration block: +Example configuration block for Linux package installations: ```ruby gitlab_rails['omniauth_providers'] = [ @@ -556,7 +556,7 @@ For your app, complete the following steps on Casdoor: See the [Casdoor documentation](https://casdoor.org/docs/integration/ruby/gitlab) for more details. -Example Omnibus GitLab configuration (file path: `/etc/gitlab/gitlab.rb`): +Example configuration for Linux package installations (file path: `/etc/gitlab/gitlab.rb`): ```ruby gitlab_rails['omniauth_providers'] = [ @@ -617,7 +617,7 @@ This is not compatible with [configuring users based on OIDC group membership](# The following example configurations show how to offer different levels of authentication, one option with 2FA and one without 2FA. -For Omnibus GitLab: +For Linux package installations: ```ruby gitlab_rails['omniauth_providers'] = [ @@ -668,7 +668,7 @@ gitlab_rails['omniauth_providers'] = [ ] ``` -For installation from source: +For self-compiled installations: ```yaml - { name: 'openid_connect', @@ -774,7 +774,7 @@ response to require users to be members of a certain group, configure GitLab to If you do not set `required_groups` or leave the setting empty, any user authenticated by the IdP through OIDC can use GitLab. -For Omnibus GitLab: +For Linux package installations: 1. Edit `/etc/gitlab/gitlab.rb`: @@ -805,10 +805,10 @@ For Omnibus GitLab: ] ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. -For installation from source: +For self-compiled installations: 1. Edit `/home/git/gitlab/config/gitlab.yml`: @@ -853,7 +853,7 @@ based on group membership, configure GitLab to identify: [external user](../../user/admin_area/external_users.md), using the `external_groups` setting. -For Omnibus GitLab: +For Linux package installations: 1. Edit `/etc/gitlab/gitlab.rb`: @@ -884,10 +884,10 @@ For Omnibus GitLab: ] ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. -For installation from source: +For self-compiled installations: 1. Edit `/home/git/gitlab/config/gitlab.yml`: @@ -930,7 +930,7 @@ response to assign users as administrator based on group membership, configure G - Which group memberships grant the user administrator access, using the `admin_groups` setting. -For Omnibus GitLab: +For Linux package installations: 1. Edit `/etc/gitlab/gitlab.rb`: @@ -961,10 +961,10 @@ For Omnibus GitLab: ] ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. -For installation from source: +For self-compiled installations: 1. Edit `/home/git/gitlab/config/gitlab.yml`: diff --git a/doc/administration/auth/smartcard.md b/doc/administration/auth/smartcard.md index 5b6d299f171..9432a627ed7 100644 --- a/doc/administration/auth/smartcard.md +++ b/doc/administration/auth/smartcard.md @@ -115,7 +115,7 @@ more information, see [the relevant issue](https://gitlab.com/gitlab-org/gitlab/ ## Configure GitLab for smartcard authentication -**For Omnibus installations** +For Linux package installations: 1. Edit `/etc/gitlab/gitlab.rb`: @@ -137,12 +137,10 @@ more information, see [the relevant issue](https://gitlab.com/gitlab-org/gitlab/ `gitlab_rails['smartcard_client_certificate_required_host']` or `gitlab_rails['smartcard_client_certificate_required_port']`. -1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) +1. Save the file and [reconfigure](../restart_gitlab.md#reconfigure-a-linux-package-installation) GitLab for the changes to take effect. ---- - -**For installations from source** +For self-compiled installations: 1. Configure NGINX to request a client side certificate @@ -237,7 +235,7 @@ more information, see [the relevant issue](https://gitlab.com/gitlab-org/gitlab/ ### Additional steps when using SAN extensions -**For Omnibus installations** +For Linux package installations: 1. Add to `/etc/gitlab/gitlab.rb`: @@ -245,10 +243,10 @@ more information, see [the relevant issue](https://gitlab.com/gitlab-org/gitlab/ gitlab_rails['smartcard_san_extensions'] = true ``` -1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) +1. Save the file and [reconfigure](../restart_gitlab.md#reconfigure-a-linux-package-installation) GitLab for the changes to take effect. -**For installations from source** +For self-compiled installations: 1. Add the `san_extensions` line to `config/gitlab.yml` within the smartcard section: @@ -267,7 +265,7 @@ more information, see [the relevant issue](https://gitlab.com/gitlab-org/gitlab/ ### Additional steps when authenticating against an LDAP server -**For Omnibus installations** +For Linux package installations: 1. Edit `/etc/gitlab/gitlab.rb`: @@ -281,10 +279,10 @@ more information, see [the relevant issue](https://gitlab.com/gitlab-org/gitlab/ EOS ``` -1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) +1. Save the file and [reconfigure](../restart_gitlab.md#reconfigure-a-linux-package-installation) GitLab for the changes to take effect. -**For installations from source** +For self-compiled installations: 1. Edit `config/gitlab.yml`: @@ -304,7 +302,7 @@ more information, see [the relevant issue](https://gitlab.com/gitlab-org/gitlab/ ### Require browser session with smartcard sign-in for Git access -**For Omnibus installations** +For Linux package installations: 1. Edit `/etc/gitlab/gitlab.rb`: @@ -312,10 +310,10 @@ more information, see [the relevant issue](https://gitlab.com/gitlab-org/gitlab/ gitlab_rails['smartcard_required_for_git_access'] = true ``` -1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) +1. Save the file and [reconfigure](../restart_gitlab.md#reconfigure-a-linux-package-installation) GitLab for the changes to take effect. -**For installations from source** +For self-compiled installations: 1. Edit `config/gitlab.yml`: diff --git a/doc/administration/cicd.md b/doc/administration/cicd.md index ad0671d4c13..a3d9a853aaf 100644 --- a/doc/administration/cicd.md +++ b/doc/administration/cicd.md @@ -15,7 +15,7 @@ GitLab CI/CD is enabled by default in all new projects on an instance. You can s CI/CD to be disabled by default in new projects by modifying the settings in: - `gitlab.yml` for source installations. -- `gitlab.rb` for Omnibus GitLab installations. +- `gitlab.rb` for Linux package installations. Existing projects that already had CI/CD enabled are unchanged. Also, this setting only changes the project default, so project owners [can still enable CI/CD in the project settings](../ci/enable_or_disable_ci.md#enable-cicd-in-a-project). @@ -42,7 +42,7 @@ For installations from source: sudo service gitlab restart ``` -For Omnibus GitLab installations: +For Linux package installations: 1. Edit `/etc/gitlab/gitlab.rb` and add this line: @@ -88,7 +88,7 @@ is listed in the [GitLab.com settings](../user/gitlab_com/index.md#gitlab-cicd). To change the frequency of the pipeline schedule worker: 1. Edit the `gitlab_rails['pipeline_schedule_worker_cron']` value in your instance's `gitlab.rb` file. -1. [Reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. For example, to set the maximum frequency of pipelines to twice a day, set `pipeline_schedule_worker_cron` to a cron value of `0 */12 * * *` (`00:00` and `12:00` every day). diff --git a/doc/administration/clusters/kas.md b/doc/administration/clusters/kas.md index 6d6e8e5513c..b44c9571715 100644 --- a/doc/administration/clusters/kas.md +++ b/doc/administration/clusters/kas.md @@ -21,12 +21,12 @@ If you use self-managed GitLab, you must install an agent server or specify an e As a GitLab administrator, you can install the agent server: -- For [Omnibus installations](#for-omnibus). -- For [GitLab Helm Chart installations](#for-gitlab-helm-chart). +- For [Linux package installations](#for-linux-package-installations). +- For [GitLab Helm chart installations](#for-gitlab-helm-chart). -### For Omnibus +### For Linux package installations -You can enable the agent server for [Omnibus](https://docs.gitlab.com/omnibus/) package installations on a single node, or on multiple nodes at once. +You can enable the agent server for Linux package installations on a single node, or on multiple nodes at once. #### Enable on a single node @@ -38,7 +38,7 @@ To enable the agent server on a single node: gitlab_kas['enable'] = true ``` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). For additional configuration options, see the **Enable GitLab KAS** section of the [`gitlab.rb.template`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/files/gitlab-config-template/gitlab.rb.template). @@ -68,7 +68,7 @@ To configure KAS to listen on a UNIX socket: } ``` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). #### Enable on multiple nodes @@ -94,7 +94,7 @@ To enable the agent server on multiple nodes: gitlab_rails['gitlab_kas_external_k8s_proxy_url'] = 'https://gitlab.example.com/-/kubernetes-agent/k8s-proxy/' ``` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). ##### Agent server node settings @@ -167,7 +167,7 @@ service logs by running the following command: kubectl logs -f -l=app=kas -n <YOUR-GITLAB-NAMESPACE> ``` -In Omnibus GitLab, find the logs in `/var/log/gitlab/gitlab-kas/`. +In Linux package installations, find the logs in `/var/log/gitlab/gitlab-kas/`. You can also [troubleshoot issues with individual agents](../../user/clusters/agent/troubleshooting.md). @@ -212,7 +212,7 @@ When the agent server tries to connect to the GitLab API, the following error mi {"level":"error","time":"2021-08-16T14:56:47.289Z","msg":"GetAgentInfo()","correlation_id":"01FD7QE35RXXXX8R47WZFBAXTN","grpc_service":"gitlab.agent.reverse_tunnel.rpc.ReverseTunnel","grpc_method":"Connect","error":"Get \"https://gitlab.example.com/api/v4/internal/kubernetes/agent_info\": dial tcp 172.17.0.4:443: connect: connection refused"} ``` -To fix this issue for [Omnibus](https://docs.gitlab.com/omnibus/) package installations, +To fix this issue for Linux package installations, set the following parameter in `/etc/gitlab/gitlab.rb`. Replace `gitlab.example.com` with your GitLab instance's hostname: ```ruby diff --git a/doc/administration/compliance.md b/doc/administration/compliance.md index 9ae0e9d7790..79133d5a6c7 100644 --- a/doc/administration/compliance.md +++ b/doc/administration/compliance.md @@ -73,5 +73,5 @@ These features can also help with compliance requirements: | [Lock project membership to group](../user/group/access_and_permissions.md#prevent-members-from-being-added-to-projects-in-a-group) | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Group owners can prevent new members from being added to projects in a group. | | [LDAP group sync](auth/ldap/ldap_synchronization.md#group-sync) | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Automatically synchronize groups and manage SSH keys, permissions, and authentication, so you can focus on building your product, not configuring your tools. | | [LDAP group sync filters](auth/ldap/ldap_synchronization.md#group-sync) | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Gives more flexibility to synchronize with LDAP based on filters, meaning you can leverage LDAP attributes to map GitLab permissions. | -| [Omnibus GitLab package supports<br/>log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-forwarding) | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Forward your logs to a central system. | +| [Linux package installations support<br/>log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-forwarding) | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Forward your logs to a central system. | | [Restrict SSH Keys](../security/ssh_keys_restrictions.md) | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Control the technology and key length of SSH keys used to access GitLab. | diff --git a/doc/administration/consul.md b/doc/administration/consul.md index 965231db440..5a8946dbcf7 100644 --- a/doc/administration/consul.md +++ b/doc/administration/consul.md @@ -49,7 +49,7 @@ On _each_ Consul server node: gitlab_rails['auto_migrate'] = false ``` -1. [Reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes +1. [Reconfigure GitLab](restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Run the following command to ensure Consul is both configured correctly and to verify that all server nodes are communicating: @@ -76,7 +76,7 @@ To upgrade your Consul nodes, upgrade the GitLab package. Nodes should be: -- Members of a healthy cluster prior to upgrading the Omnibus GitLab package. +- Members of a healthy cluster prior to upgrading the Linux package. - Upgraded one node at a time. Identify any existing health issues in the cluster by running the following command diff --git a/doc/administration/docs_self_host.md b/doc/administration/docs_self_host.md index d1ad36880dd..54f8c15a922 100644 --- a/doc/administration/docs_self_host.md +++ b/doc/administration/docs_self_host.md @@ -84,7 +84,7 @@ You can use GitLab Pages to host the GitLab product documentation. Prerequisite: -- Ensure the Pages site URL does not use a subfolder. Because of the way the +- Ensure the Pages site URL does not use a subfolder. Because of the way the site is pre-compiled, the CSS and JavaScript files are relative to the main domain or subdomain. For example, URLs like `https://example.com/docs/` are not supported. @@ -179,7 +179,7 @@ documentation URL requests as needed. For example, if your GitLab version is To test the setting, in GitLab, select a **Learn more** link. For example: -1. On the top bar, in the upper-right corner, select your avatar. +1. On the left sidebar, select your avatar. 1. Select **Preferences**. 1. In the **Syntax highlighting theme** section, select **Learn more**. diff --git a/doc/administration/encrypted_configuration.md b/doc/administration/encrypted_configuration.md index 1ddf2951f70..808967778fa 100644 --- a/doc/administration/encrypted_configuration.md +++ b/doc/administration/encrypted_configuration.md @@ -19,22 +19,14 @@ GitLab can read settings for certain features from encrypted settings files. The To enable the encrypted configuration settings, a new base key must be generated for `encrypted_settings_key_base`. The secret can be generated in the following ways: -**Omnibus** +- For Linux package installations, the new secret is automatically generated for you, but you must ensure your + `/etc/gitlab/gitlab-secrets.json` contains the same values on all nodes. +- For Helm chart installations, the new secret is automatically generated if you have the `shared-secrets` chart enabled. + Otherwise, you need to follow the [secrets guide for adding the secret](https://docs.gitlab.com/charts/installation/secrets.html#gitlab-rails-secret). +- For self-compiled installations, the new secret can be generated by running: -Starting with 13.7 the new secret is automatically generated for you, but you must ensure your -`/etc/gitlab/gitlab-secrets.json` contains the same values on all nodes. + ```shell + bundle exec rake gitlab:env:info RAILS_ENV=production GITLAB_GENERATE_ENCRYPTED_SETTINGS_KEY_BASE=true + ``` -**Helm chart** - -Starting with GitLab 13.7, the new secret is automatically generated if you have the `shared-secrets` chart enabled. Otherwise, you need -to follow the [secrets guide for adding the secret](https://docs.gitlab.com/charts/installation/secrets.html#gitlab-rails-secret). - -**Source** - -The new secret can be generated by running: - -```shell -bundle exec rake gitlab:env:info RAILS_ENV=production GITLAB_GENERATE_ENCRYPTED_SETTINGS_KEY_BASE=true -``` - -This prints general information on the GitLab instance, but also causes the key to be generated in `<path-to-gitlab-rails>/config/secrets.yml`. + This prints general information on the GitLab instance and generates the key in `<path-to-gitlab-rails>/config/secrets.yml`. diff --git a/doc/administration/environment_variables.md b/doc/administration/environment_variables.md index a453d703a18..260095d12ac 100644 --- a/doc/administration/environment_variables.md +++ b/doc/administration/environment_variables.md @@ -10,8 +10,10 @@ type: reference GitLab exposes certain environment variables which can be used to override their defaults values. -People usually configure GitLab with `/etc/gitlab/gitlab.rb` for Omnibus -installations, or `gitlab.yml` for installations from source. +People usually configure GitLab with: + +- `/etc/gitlab/gitlab.rb` for Linux package installations. +- `gitlab.yml` for self-compiled installations. You can use the following environment variables to override certain values: @@ -44,11 +46,10 @@ We welcome merge requests to make more settings configurable by using variables. Make changes to the `config/initializers/1_settings.rb` file, and use the naming scheme `GITLAB_#{name in 1_settings.rb in upper case}`. -## Omnibus configuration +## Linux package installation configuration To set environment variables, follow [these instructions](https://docs.gitlab.com/omnibus/settings/environment-variables.html). It's possible to preconfigure the GitLab Docker image by adding the environment variable `GITLAB_OMNIBUS_CONFIG` to the `docker run` command. -For more information, see the [Pre-configure Docker container](../install/docker.md#pre-configure-docker-container) -section of the Omnibus GitLab documentation. +For more information, see [Pre-configure Docker container](../install/docker.md#pre-configure-docker-container). diff --git a/doc/administration/feature_flags.md b/doc/administration/feature_flags.md index 26deff0ca84..ea79a2b69b4 100644 --- a/doc/administration/feature_flags.md +++ b/doc/administration/feature_flags.md @@ -71,7 +71,7 @@ the status of the flag and the command to enable or disable it. The first thing you must do to enable or disable a feature behind a flag is to start a session on GitLab Rails console. -For Omnibus installations: +For Linux package installations: ```shell sudo gitlab-rails console diff --git a/doc/administration/file_hooks.md b/doc/administration/file_hooks.md index dfddf38475c..904da47caff 100644 --- a/doc/administration/file_hooks.md +++ b/doc/administration/file_hooks.md @@ -34,13 +34,12 @@ where you can find some basic examples. Follow the steps below to set up a custom hook: -1. On the GitLab server, navigate to the plugin directory. - For an installation from source the path is usually - `/home/git/gitlab/file_hooks/`. For Omnibus installs the path is - usually `/opt/gitlab/embedded/service/gitlab-rails/file_hooks`. +1. On the GitLab server, locate the plugin directory. For self-compiled installations, the path is usually + `/home/git/gitlab/file_hooks/`. For Linux package installations, the path is usually + `/opt/gitlab/embedded/service/gitlab-rails/file_hooks`. - For [configurations with multiple servers](reference_architectures/index.md), - your hook file should exist on each application server. + For [configurations with multiple servers](reference_architectures/index.md), your hook file should exist on each + application server. 1. Inside the `file_hooks` directory, create a file with a name of your choice, without spaces or special characters. @@ -59,11 +58,8 @@ need to restart GitLab to apply a new file hook. If a file hook executes with non-zero exit code or GitLab fails to execute it, a message is logged to: -- `gitlab-rails/file_hook.log` in an Omnibus installation. -- `log/file_hook.log` in a source installation. - -NOTE: -In GitLab 13.12 and earlier, the filename was `plugin.log` +- `gitlab-rails/file_hook.log` in a Linux package installation. +- `log/file_hook.log` in a self-compiled installation. ## Creating file hooks diff --git a/doc/administration/geo/disaster_recovery/background_verification.md b/doc/administration/geo/disaster_recovery/background_verification.md index ea4523c66e6..6d7240a9f92 100644 --- a/doc/administration/geo/disaster_recovery/background_verification.md +++ b/doc/administration/geo/disaster_recovery/background_verification.md @@ -49,7 +49,8 @@ Feature.enable('geo_repository_verification') On the **primary** site: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Geo > Sites**. 1. Expand **Verification information** tab for that site to view automatic checksumming status for repositories and wikis. Successes are shown in green, pending work @@ -59,7 +60,8 @@ On the **primary** site: On the **secondary** site: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Geo > Sites**. 1. Expand **Verification information** tab for that site to view automatic checksumming status for repositories and wikis. Successes are shown in green, pending work @@ -87,7 +89,8 @@ increase load and vice versa. On the **primary** site: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Geo > Sites**. 1. Select **Edit** for the **primary** site to customize the minimum re-verification interval: @@ -135,7 +138,8 @@ sudo gitlab-rake geo:verification:wiki:reset If the **primary** and **secondary** sites have a checksum verification mismatch, the cause may not be apparent. To find the cause of a checksum mismatch: 1. On the **primary** site: - 1. On the top bar, select **Main menu > Admin**. + 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. Select **Admin Area**. 1. On the left sidebar, select **Overview > Projects**. 1. Find the project that you want to check the checksum differences and select its name. diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index 24a91a7a9c5..396162fe9ef 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -236,7 +236,7 @@ do this manually. roles ['geo_secondary_role'] ``` - After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) + After making these changes, [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) on each machine so the changes take effect. 1. Promote the **secondary** to **primary**. SSH into a single application @@ -444,7 +444,7 @@ required: roles ['geo_secondary_role'] ``` - After making these changes [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) + After making these changes [Reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) on each node so the changes take effect. 1. Promote the **secondary** to **primary**. SSH into a single secondary application @@ -542,19 +542,19 @@ Geo on the new **primary** site. To bring a new **secondary** site online, follow the [Geo setup instructions](../index.md#setup-instructions). -### Step 6. (Optional) Removing the secondary's tracking database +### Step 6. Removing the secondary's tracking database Every **secondary** has a special tracking database that is used to save the status of the synchronization of all the items from the **primary**. Because the **secondary** is already promoted, that data in the tracking database is no longer required. -The data can be removed with the following command: +You can remove the data with the following command: ```shell sudo rm -rf /var/opt/gitlab/geo-postgresql ``` If you have any `geo_secondary[]` configuration options enabled in your `gitlab.rb` -file, these can be safely commented out or removed, and then [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) +file, comment them out or remove them, and then [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. ## Promoting secondary Geo replica in multi-secondary configurations @@ -681,7 +681,8 @@ Data that was created on the primary while the secondary was paused is lost. If you are running GitLab 14.5 and later: -1. For each node outside of the **secondary** Kubernetes cluster using Omnibus such as PostgreSQL or Gitaly, SSH into the node and run one of the following commands: +1. For each node (such as PostgreSQL or Gitaly) outside of the **secondary** Kubernetes cluster using the Linux + package, SSH into the node and run one of the following commands: - To promote the **secondary** site node external to the Kubernetes cluster to primary: @@ -731,7 +732,7 @@ If you are running GitLab 14.4 and earlier: roles ['geo_secondary_role'] ``` - After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) on the database node. + After making these changes, [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) on the database node. 1. Find the task runner pod: diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index a52bdc759a2..13e0938fa59 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -149,7 +149,8 @@ ensure these processes are close to 100% as possible during active use. On the **secondary** site: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Geo > Sites**. Replicated objects (shown in green) should be close to 100%, and there should be no failures (shown in red). If a large proportion of @@ -177,7 +178,8 @@ This [content was moved to another location](background_verification.md). On the **primary** site: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Messages**. 1. Add a message notifying users on the maintenance window. You can check under **Geo > Sites** to estimate how long it @@ -190,7 +192,8 @@ To ensure that all data is replicated to a secondary site, updates (write reques be disabled on the **primary** site: 1. Enable [maintenance mode](../../maintenance_mode/index.md) on the **primary** site. -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Cron**. 1. Select `Disable All` to disable non-Geo periodic background jobs. @@ -206,7 +209,8 @@ GitLab 13.9 through GitLab 14.3 are affected by a bug in which the Geo secondary 1. If you are manually replicating any data not managed by Geo, trigger the final replication process now. 1. On the **primary** site: - 1. On the top bar, select **Main menu > Admin**. + 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Queues**, and wait for all queues except those with `geo` in the name to drop to 0. @@ -221,7 +225,8 @@ GitLab 13.9 through GitLab 14.3 are affected by a bug in which the Geo secondary - The Geo log cursor is up to date (0 events behind). 1. On the **secondary** site: - 1. On the top bar, select **Main menu > Admin**. + 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Queues**, and wait for all the `geo` queues to drop to 0 queued and 0 running jobs. diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md index feb5e030b80..a96963cfea1 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md @@ -6,18 +6,18 @@ type: howto --- WARNING: -This runbook is an [Experiment](../../../../policy/alpha-beta-support.md#experiment). For complete, production-ready documentation, see the +This runbook is an [Experiment](../../../../policy/experiment-beta-support.md#experiment). For complete, production-ready documentation, see the [disaster recovery documentation](../index.md). # Disaster Recovery (Geo) promotion runbooks **(PREMIUM SELF)** ## Geo planned failover for a multi-node configuration -| Component | Configuration | -|-------------|-----------------| -| PostgreSQL | Omnibus-managed | -| Geo site | Multi-node | -| Secondaries | One | +| Component | Configuration | +|:------------|:-----------------------------| +| PostgreSQL | Managed by the Linux package | +| Geo site | Multi-node | +| Secondaries | One | This runbook guides you through a planned failover of a multi-node Geo site with one secondary. The following [2000 user reference architecture](../../../../administration/reference_architectures/2k_users.md) is assumed: @@ -68,7 +68,8 @@ GitLab 13.9 through GitLab 14.3 are affected by a bug in which the Geo secondary On the **secondary** site: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Geo > Sites** to see its status. Replicated objects (shown in green) should be close to 100%, and there should be no failures (shown in red). If a large proportion of @@ -95,52 +96,8 @@ ensure these processes are close to 100% as possible during active use. If the **secondary** site is still replicating data from the **primary** site, follow these steps to avoid unnecessary data loss: -1. Until a [read-only mode](https://gitlab.com/gitlab-org/gitlab/-/issues/14609) - is implemented, updates must be prevented from happening manually to the - **primary**. Your **secondary** site still needs read-only - access to the **primary** site during the maintenance window: - - 1. At the scheduled time, using your cloud provider or your site's firewall, block - all HTTP, HTTPS and SSH traffic to/from the **primary** site, **except** for your IP and - the **secondary** site's IP. - - For instance, you can run the following commands on the **primary** site: - - ```shell - sudo iptables -A INPUT -p tcp -s <secondary_site_ip> --destination-port 22 -j ACCEPT - sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 22 -j ACCEPT - sudo iptables -A INPUT --destination-port 22 -j REJECT - - sudo iptables -A INPUT -p tcp -s <secondary_site_ip> --destination-port 80 -j ACCEPT - sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 80 -j ACCEPT - sudo iptables -A INPUT --tcp-dport 80 -j REJECT - - sudo iptables -A INPUT -p tcp -s <secondary_site_ip> --destination-port 443 -j ACCEPT - sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 443 -j ACCEPT - sudo iptables -A INPUT --tcp-dport 443 -j REJECT - ``` - - From this point, users are unable to view their data or make changes on the - **primary** site. They are also unable to sign in to the **secondary** site. - However, existing sessions must work for the remainder of the maintenance period, and - so public data is accessible throughout. - - 1. Verify the **primary** site is blocked to HTTP traffic by visiting it in browser via - another IP. The server should refuse connection. - - 1. Verify the **primary** site is blocked to Git over SSH traffic by attempting to pull an - existing Git repository with an SSH remote URL. The server should refuse - connection. - - 1. On the **primary** site: - 1. On the top bar, select **Main menu > Admin**. - 1. On the left sidebar, select **Monitoring > Background Jobs**. - 1. On the Sidekiq dashboard, select **Cron**. - 1. Select `Disable All` to disable any non-Geo periodic background jobs. - 1. Select `Enable` for the `geo_sidekiq_cron_config_worker` cron job. - This job re-enables several other cron jobs that are essential for planned - failover to complete successfully. - +1. Enable [maintenance mode](../../../maintenance_mode/index.md) on the **primary** site, + and make sure to stop any [background jobs](../../../maintenance_mode/index.md#background-jobs). 1. Finish replicating and verifying all data: WARNING: @@ -151,7 +108,8 @@ follow these steps to avoid unnecessary data loss: [data not managed by Geo](../../replication/datatypes.md#limitations-on-replicationverification), trigger the final replication process now. 1. On the **primary** site: - 1. On the top bar, select **Main menu > Admin**. + 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Queues**, and wait for all queues except those with `geo` in the name to drop to 0. @@ -166,7 +124,8 @@ follow these steps to avoid unnecessary data loss: - The Geo log cursor is up to date (0 events behind). 1. On the **secondary** site: - 1. On the top bar, select **Main menu > Admin**. + 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Queues**, and wait for all the `geo` queues to drop to 0 queued and 0 running jobs. @@ -205,7 +164,7 @@ follow these steps to avoid unnecessary data loss: NOTE: (**CentOS only**) In CentOS 6 or older, it is challenging to prevent GitLab from being - started if the machine reboots isn't available (see [Omnibus GitLab issue #3058](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3058)). + started if the machine reboots isn't available (see [issue 3058](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3058)). It may be safest to uninstall the GitLab package completely with `sudo yum remove gitlab-ee`. NOTE: @@ -312,7 +271,7 @@ Data that was created on the primary while the secondary was paused is lost. roles ['geo_secondary_role'] ``` - After making these changes [Reconfigure GitLab](../../../restart_gitlab.md#omnibus-gitlab-reconfigure) each + After making these changes, [reconfigure GitLab](../../../restart_gitlab.md#reconfigure-a-linux-package-installation) each machine so the changes take effect. 1. Promote the **secondary** to **primary**. SSH into a single Rails node diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md index 19150027cca..7f94d6e4c1a 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md @@ -6,18 +6,18 @@ type: howto --- WARNING: -This runbook is an [Experiment](../../../../policy/alpha-beta-support.md#experiment). For complete, production-ready documentation, see the +This runbook is an [Experiment](../../../../policy/experiment-beta-support.md#experiment). For complete, production-ready documentation, see the [disaster recovery documentation](../index.md). # Disaster Recovery (Geo) promotion runbooks **(PREMIUM SELF)** ## Geo planned failover for a single-node configuration -| Component | Configuration | -|-------------|-----------------| -| PostgreSQL | Omnibus-managed | -| Geo site | Single-node | -| Secondaries | One | +| Component | Configuration | +|:------------|:-----------------------------| +| PostgreSQL | Managed by the Linux package | +| Geo site | Single-node | +| Secondaries | One | This runbook guides you through a planned failover of a single-node Geo site with one secondary. The following general architecture is assumed: @@ -118,7 +118,8 @@ follow these steps to avoid unnecessary data loss: connection. 1. On the **primary** site: - 1. On the top bar, select **Main menu > Admin**. + 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Cron**. 1. Select `Disable All` to disable any non-Geo periodic background jobs. @@ -136,7 +137,8 @@ follow these steps to avoid unnecessary data loss: [data not managed by Geo](../../replication/datatypes.md#limitations-on-replicationverification), trigger the final replication process now. 1. On the **primary** site: - 1. On the top bar, select **Main menu > Admin**. + 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Queues**, and wait for all queues except those with `geo` in the name to drop to 0. @@ -151,7 +153,8 @@ follow these steps to avoid unnecessary data loss: - The Geo log cursor is up to date (0 events behind). 1. On the **secondary** site: - 1. On the top bar, select **Main menu > Admin**. + 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Queues**, and wait for all the `geo` queues to drop to 0 queued and 0 running jobs. @@ -190,7 +193,7 @@ follow these steps to avoid unnecessary data loss: NOTE: (**CentOS only**) In CentOS 6 or older, there is no easy way to prevent GitLab from being - started if the machine reboots isn't available (see [Omnibus GitLab issue #3058](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3058)). + started if the machine reboots isn't available (see [issue 3058](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3058)). It may be safest to uninstall the GitLab package completely with `sudo yum remove gitlab-ee`. NOTE: diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index 31de7f5c62f..ae2cc262160 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -160,7 +160,9 @@ public URL of the primary site is used. To update the internal URL of the primary Geo site: -1. On the top bar, select **Main menu > Admin > Geo > Sites**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. On the left sidebar, select **Geo > Sites**. 1. Select **Edit** on the primary site. 1. Change the **Internal URL**, then select **Save changes**. @@ -199,7 +201,8 @@ This list of limitations only reflects the latest version of GitLab. If you are - [Pages access control](../../user/project/pages/pages_access_control.md) doesn't work on secondaries. See [GitLab issue #9336](https://gitlab.com/gitlab-org/gitlab/-/issues/9336) for details. - [GitLab chart with Geo](https://docs.gitlab.com/charts/advanced/geo/) does not support [Unified URLs](secondary_proxy/index.md#set-up-a-unified-url-for-geo-sites). See [GitLab issue #3522](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/3522) for more details. - [Disaster recovery](disaster_recovery/index.md) for multi-secondary sites causes downtime due to the complete re-synchronization and re-configuration of all non-promoted secondaries. -- For Git over SSH, secondary sites must use the same port as the primary. [GitLab issue #339262](https://gitlab.com/gitlab-org/gitlab/-/issues/339262) proposes to remove this limitation. +- For Git over SSH, to make the project clone URL display correctly regardless of which site you are browsing, secondary sites must use the same port as the primary. [GitLab issue #339262](https://gitlab.com/gitlab-org/gitlab/-/issues/339262) proposes to remove this limitation. +- Git push over SSH against a secondary site does not work for pushes over 1.86 GB. [GitLab issue #413109](https://gitlab.com/gitlab-org/gitlab/-/issues/413109) tracks this bug. ### Limitations on replication/verification @@ -209,14 +212,11 @@ There is a complete list of all GitLab [data types](replication/datatypes.md) an If you try to view replication data on the primary site, you receive a warning that this may be inconsistent: -> Viewing projects and designs data from a primary site is not possible when using a unified URL. Visit the secondary site directly. +> Viewing projects data from a primary site is not possible when using a unified URL. Visit the secondary site directly. The only way to view projects replication data for a particular secondary site is to visit that secondary site directly. For example, `https://<IP of your secondary site>/admin/geo/replication/projects`. An [epic exists](https://gitlab.com/groups/gitlab-org/-/epics/4623) to fix this limitation. -The only way to view designs replication data for a particular secondary site is to visit that secondary site directly. For example, `https://<IP of your secondary site>/admin/geo/replication/designs`. -An [epic exists](https://gitlab.com/groups/gitlab-org/-/epics/4624) to fix this limitation. - Keep in mind that mentioned URLs don't work when [Admin Mode](../../user/admin_area/settings/sign_in_restrictions.md#admin-mode) is enabled. When using Unified URL, visiting the secondary site directly means you must route your requests to the secondary site. Exactly how this might be done depends on your networking configuration. If using DNS to route requests to the appropriate site, then you can, for example, edit your local machine's `/etc/hosts` file to route your requests to the desired secondary site. If the Geo sites are all behind a load balancer, then depending on the load balancer, you might be able to configure all requests from your IP to go to a particular secondary site. @@ -248,10 +248,11 @@ secondary. If the site is paused, be sure to resume before promoting. This issue has been fixed in GitLab 13.4 and later. WARNING: -Pausing and resuming of replication is only supported for Geo installations using an -Omnibus GitLab-managed database. External databases are not supported. +Pausing and resuming of replication is only supported for Geo installations using a +Linux package-managed database. External databases are not supported. -In some circumstances, like during [upgrades](replication/upgrading_the_geo_sites.md) or a [planned failover](disaster_recovery/planned_failover.md), it is desirable to pause replication between the primary and secondary. +In some circumstances, like during [upgrades](replication/upgrading_the_geo_sites.md) or a +[planned failover](disaster_recovery/planned_failover.md), it is desirable to pause replication between the primary and secondary. Pausing and resuming replication is done via a command line tool from the node in the secondary site where the `postgresql` service is enabled. @@ -275,7 +276,7 @@ For information on configuring Geo for multiple nodes, see [Geo for multiple ser ### Configuring Geo with Object Storage -For information on configuring Geo with object storage, see [Geo with Object storage](replication/object_storage.md). +For information on configuring Geo with Object storage, see [Geo with Object storage](replication/object_storage.md). ### Disaster Recovery @@ -333,7 +334,7 @@ For answers to common questions, see the [Geo FAQ](replication/faq.md). ## Log files -Geo stores structured log messages in a `geo.log` file. For Omnibus GitLab +Geo stores structured log messages in a `geo.log` file. For Linux package installations, this file is at `/var/log/gitlab/gitlab-rails/geo.log`. This file contains information about when Geo attempts to sync repositories and files. Each line in the file contains a separate JSON entry that can be ingested into. For example, Elasticsearch or Splunk. diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index 5fa6df393b9..18d0440965e 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -12,7 +12,7 @@ type: howto NOTE: This is the final step in setting up a **secondary** Geo site. Stages of the setup process must be completed in the documented order. -If not, [complete all prior stages](../setup/index.md#using-omnibus-gitlab) before proceeding. +If not, [complete all prior stages](../setup/index.md#using-linux-package-installations) before proceeding. Make sure you [set up the database replication](../setup/database.md), and [configured fast lookup of authorized SSH keys](../../operations/fast_ssh_key_lookup.md) in **both primary and secondary sites**. @@ -202,7 +202,8 @@ keys must be manually replicated to the **secondary** site. ``` 1. Navigate to the Primary Node GitLab Instance: - 1. On the top bar, select **Main menu > Admin**. + 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. Select **Admin Area**. 1. On the left sidebar, select **Geo > Sites**. 1. Select **Add site**. ![Add secondary site](img/adding_a_secondary_v15_8.png) @@ -311,12 +312,13 @@ method to be enabled. This is enabled by default, but if converting an existing On the **primary** site: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Settings > General**. 1. Expand **Visibility and access controls**. 1. If using Git over SSH, then: 1. Ensure "Enabled Git access protocols" is set to "Both SSH and HTTP(S)". - 1. Follow [Fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md) on both primary and secondary sites. + 1. Follow [Fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md) on **all primary and secondary** sites. 1. If not using Git over SSH, then set "Enabled Git access protocols" to "Only HTTP(S)". ### Step 6. Verify proper functioning of the **secondary** site @@ -324,7 +326,8 @@ On the **primary** site: You can sign in to the **secondary** site with the same credentials you used with the **primary** site. After you sign in: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Geo > Sites**. 1. Verify that it's correctly identified as a **secondary** Geo site, and that Geo is enabled. diff --git a/doc/administration/geo/replication/container_registry.md b/doc/administration/geo/replication/container_registry.md index 66c67f29c1c..6dde611a20d 100644 --- a/doc/administration/geo/replication/container_registry.md +++ b/doc/administration/geo/replication/container_registry.md @@ -73,12 +73,11 @@ To configure Container Registry replication: Make sure that you have Container Registry set up and working on the **primary** site before following the next steps. -We need to make Container Registry send notification events to the -**primary** site. +To be able to replicate new container images, the Container Registry must send notification events to the +**primary** site for every push. The token shared between the Container Registry and the web nodes on the +**primary** is used to make communication more secure. -For each application and Sidekiq node on the **primary** site: - -1. SSH into the node and login as the `root` user: +1. SSH into your GitLab **primary** server and login as root (for GitLab HA, you only need a Registry node): ```shell sudo -i @@ -107,15 +106,17 @@ For each application and Sidekiq node on the **primary** site: that starts with a letter. You can generate one with `< /dev/urandom tr -dc _A-Z-a-z-0-9 | head -c 32 | sed "s/^[0-9]*//"; echo` NOTE: - If you use an external Registry (not the one integrated with GitLab), you also have to specify + If you use an external Registry (not the one integrated with GitLab), you only need to specify the notification secret (`registry['notification_secret']`) in the `/etc/gitlab/gitlab.rb` file. - NOTE: - If you use GitLab HA, you also have to specify the notification secret (`registry['notification_secret']`) in - `/etc/gitlab/gitlab.rb` file for every web node. +1. For GitLab HA only. Edit `/etc/gitlab/gitlab.rb` on every web node: + + ```ruby + registry['notification_secret'] = '<replace_with_a_secret_token_generated_above>' + ``` -1. Reconfigure each node: +1. Reconfigure each node you just updated: ```shell gitlab-ctl reconfigure @@ -165,7 +166,8 @@ For each application and Sidekiq node on the **secondary** site: To verify Container Registry replication is working, on the **secondary** site: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Geo > Nodes**. The initial replication, or "backfill", is probably still in progress. diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index d6ce5ef5067..f038bfd705c 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -203,7 +203,7 @@ successfully, you must replicate their data using some other means. |[CI Secure Files](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/secure_file.rb) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [No](object_storage.md#verification-of-files-in-object-storage) | Verification is behind the feature flag `geo_ci_secure_file_replication`, enabled by default in 15.3. | |[Container Registry](../../packages/container_registry.md) | **Yes** (12.3)<sup>1</sup> | **Yes** (15.10) | **Yes** (12.3)<sup>1</sup> | **Yes** (15.10) | See [instructions](container_registry.md) to set up the Container Registry replication. | |[Terraform Module Registry](../../../user/packages/terraform_module_registry/index.md) | **Yes** (14.0) | **Yes** (14.0) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Behind feature flag `geo_package_file_replication`, enabled by default. | -|[Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | N/A | N/A | Designs also require replication of LFS objects and Uploads. | +|[Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | **Yes** (16.1) | N/A | N/A | Designs also require replication of LFS objects and Uploads. Replication is behind the feature flag geo_design_management_repository_replication, enabled by default.| |[Package Registry](../../../user/packages/package_registry/index.md) | **Yes** (13.2) | **Yes** (13.10) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Behind feature flag `geo_package_file_replication`, enabled by default. | |[Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | **Yes** (13.12) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Replication is behind the feature flag `geo_terraform_state_version_replication`, enabled by default. Verification was behind the feature flag `geo_terraform_state_version_verification`, which was removed in 14.0. | |[External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Replication is behind the feature flag `geo_merge_request_diff_replication`, enabled by default. Verification was behind the feature flag `geo_merge_request_diff_verification`, removed in 14.7.| diff --git a/doc/administration/geo/replication/disable_geo.md b/doc/administration/geo/replication/disable_geo.md index c42130a62a7..bedcca7e311 100644 --- a/doc/administration/geo/replication/disable_geo.md +++ b/doc/administration/geo/replication/disable_geo.md @@ -7,7 +7,7 @@ type: howto # Disabling Geo **(PREMIUM SELF)** -If you want to revert to a regular Omnibus setup after a test, or you have encountered a Disaster Recovery +If you want to revert to a regular Linux package installation setup after a test, or you have encountered a Disaster Recovery situation and you want to disable Geo momentarily, you can use these instructions to disable your Geo setup. @@ -36,7 +36,8 @@ to do that. To remove the **primary** site: 1. [Remove all secondary Geo sites](remove_geo_site.md) -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Geo > Nodes**. 1. Select **Remove** for the **primary** node. 1. Confirm by selecting **Remove** when the prompt appears. @@ -80,7 +81,7 @@ Geo node in a PostgreSQL console (`sudo gitlab-psql`): roles ['geo_primary_role'] ``` -1. After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) +1. After making these changes, [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. ## (Optional) Revert PostgreSQL settings to use a password and listen on an IP diff --git a/doc/administration/geo/replication/geo_validation_tests.md b/doc/administration/geo/replication/geo_validation_tests.md index cad3a396bfc..7897635e78f 100644 --- a/doc/administration/geo/replication/geo_validation_tests.md +++ b/doc/administration/geo/replication/geo_validation_tests.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w The Geo team performs manual testing and validation on common deployment configurations to ensure that Geo works when upgrading between minor GitLab versions and major PostgreSQL database versions. -This section contains a journal of recent validation tests and links to the relevant issues. +This section contains a journal of validation tests and links to the relevant issues. ## GitLab upgrades @@ -143,7 +143,7 @@ The following are PostgreSQL upgrade validation tests we performed. we tested fresh installations of GitLab 13.3 with PostgreSQL 12 enabled and Geo installed. - Outcome: Setting up a Geo secondary required manual intervention because the `recovery.conf` file is no longer supported in PostgreSQL 12. We do not recommend deploying Geo with PostgreSQL 12 until - the appropriate changes have been made to Omnibus and verified. + the appropriate changes have been made to the Linux package and verified. - Follow up issues: - [Update `replicate-geo-database` to support PostgreSQL 12](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5575) - [Remove PostgreSQL 12 check in `replicate-geo-database` for 14.0](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5576) @@ -184,14 +184,17 @@ The following are PostgreSQL upgrade validation tests we performed. The following are additional validation tests we performed. -### May 2021 +### April 2022 -[Test failover with object storage replication enabled](https://gitlab.com/gitlab-org/gitlab/-/issues/330362): +[Validate Object storage replication using AWS based object storage](https://gitlab.com/gitlab-org/gitlab/-/issues/351463): -- Description: At the time of testing, Geo's object storage replication functionality was in beta. We tested that object storage replication works as intended and that the data was present on the new primary after a failover. -- Outcome: The test was successful. Data in object storage was replicated and present after a failover. -- Follow up issues: - - [Geo: Failing to replicate initial Monitoring project](https://gitlab.com/gitlab-org/gitlab/-/issues/330485) +- Description: Tested the average time it takes for a single image to replicate from the primary object storage location to the secondary when using AWS based object storage replication and [GitLab based object storage replication](object_storage.md#enabling-gitlab-managed-object-storage-replication). This was tested by uploading a 1 MB image to a project on the primary site every second for 60 seconds. The time was then measured until a image was available on the secondary site. This was achieved using a [Ruby Script](https://gitlab.com/gitlab-org/quality/geo-replication-tester). +- Outcome: When using AWS managed replication the average time for an image to replicate between sites is about 49 seconds, this is true for when sites are located in the same region and when they are further apart (Europe to America). When using Geo managed replication in the same region the average time for replication took just 5 seconds, however when replicating cross region the average time rose to 33 seconds. + +[Validate Object storage replication using GCP based object storage](https://gitlab.com/gitlab-org/gitlab/-/issues/351464): + +- Description: Tested the average time it takes for a single image to replicate from the primary object storage location to the secondary when using GCP based object storage replication and [GitLab based object storage replication](object_storage.md#enabling-gitlab-managed-object-storage-replication). This was tested by uploading a 1 MB image to a project on the primary site every second for 60 seconds. The time was then measured until a image was available on the secondary site. This was achieved using a [Ruby Script](https://gitlab.com/gitlab-org/quality/geo-replication-tester). +- Outcome: GCP handles replication differently than other Cloud Providers. In GCP, the process is to a create single bucket that is either multi, dual, or single region based. This means that the bucket automatically stores replicas in a region based on the option chosen. Even when using multi region, this only replicates in a single continent, the options being America, Europe, or Asia. At current there doesn't seem to be any way to replicate objects between continents using GCP based replication. For Geo managed replication the average time when replicating in the same region was 6 seconds, and when replicating cross region this rose to just 9 seconds. ### January 2022 @@ -202,17 +205,14 @@ The following are additional validation tests we performed. - Follow up issue: - [Validate Cross Region Object storage replication using Azure based object storage](https://gitlab.com/gitlab-org/gitlab/-/issues/358154) -### April 2022 - -[Validate Object storage replication using AWS based object storage](https://gitlab.com/gitlab-org/gitlab/-/issues/351463): - -- Description: Tested the average time it takes for a single image to replicate from the primary object storage location to the secondary when using AWS based object storage replication and [GitLab based object storage replication](object_storage.md#enabling-gitlab-managed-object-storage-replication). This was tested by uploading a 1 MB image to a project on the primary site every second for 60 seconds. The time was then measured until a image was available on the secondary site. This was achieved using a [Ruby Script](https://gitlab.com/gitlab-org/quality/geo-replication-tester). -- Outcome: When using AWS managed replication the average time for an image to replicate between sites is about 49 seconds, this is true for when sites are located in the same region and when they are further apart (Europe to America). When using Geo managed replication in the same region the average time for replication took just 5 seconds, however when replicating cross region the average time rose to 33 seconds. +### May 2021 -[Validate Object storage replication using GCP based object storage](https://gitlab.com/gitlab-org/gitlab/-/issues/351464): +[Test failover with object storage replication enabled](https://gitlab.com/gitlab-org/gitlab/-/issues/330362): -- Description: Tested the average time it takes for a single image to replicate from the primary object storage location to the secondary when using GCP based object storage replication and [GitLab based object storage replication](object_storage.md#enabling-gitlab-managed-object-storage-replication). This was tested by uploading a 1 MB image to a project on the primary site every second for 60 seconds. The time was then measured until a image was available on the secondary site. This was achieved using a [Ruby Script](https://gitlab.com/gitlab-org/quality/geo-replication-tester). -- Outcome: GCP handles replication differently than other Cloud Providers. In GCP, the process is to a create single bucket that is either multi, dual, or single region based. This means that the bucket automatically stores replicas in a region based on the option chosen. Even when using multi region, this only replicates in a single continent, the options being America, Europe, or Asia. At current there doesn't seem to be any way to replicate objects between continents using GCP based replication. For Geo managed replication the average time when replicating in the same region was 6 seconds, and when replicating cross region this rose to just 9 seconds. +- Description: At the time of testing, Geo's object storage replication functionality was in beta. We tested that object storage replication works as intended and that the data was present on the new primary after a failover. +- Outcome: The test was successful. Data in object storage was replicated and present after a failover. +- Follow up issues: + - [Geo: Failing to replicate initial Monitoring project](https://gitlab.com/gitlab-org/gitlab/-/issues/330485) ## Other tests diff --git a/doc/administration/geo/replication/multiple_servers.md b/doc/administration/geo/replication/multiple_servers.md index 155fefb2dbf..4e597a21922 100644 --- a/doc/administration/geo/replication/multiple_servers.md +++ b/doc/administration/geo/replication/multiple_servers.md @@ -35,7 +35,7 @@ The **primary** and **secondary** Geo sites must be able to communicate to each Because of the additional complexity involved in setting up this configuration for PostgreSQL and Redis, it is not covered by this Geo multi-node documentation. -For more information on setting up a multi-node PostgreSQL cluster and Redis cluster using the Omnibus GitLab package, see: +For more information on setting up a multi-node PostgreSQL cluster and Redis cluster using the Linux package, see: - [Geo multi-node database replication](../setup/database.md#multi-node-database-replication) - [Redis multi-node documentation](../../redis/replication_and_failover.md) @@ -77,7 +77,15 @@ The following steps enable a GitLab site to serve as the Geo **primary** site. gitlab_rails['auto_migrate'] = false ``` -After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. +After making these changes, [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) so the changes take effect. + +### Step 2: Define the site as the **primary** site + +1. Execute the following command on one of the frontend nodes: + + ```shell + sudo gitlab-ctl set-geo-primary-node + ``` NOTE: PostgreSQL and Redis should have already been disabled on the @@ -171,7 +179,7 @@ the instructions below. gitlab_rails['auto_migrate'] = false ``` -After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. +After making these changes, [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) so the changes take effect. If using an external PostgreSQL instance, refer also to [Geo with external PostgreSQL instances](../setup/external_database.md). @@ -252,7 +260,7 @@ then make the following modifications: ``` NOTE: -If you had set up PostgreSQL cluster using the omnibus package and had set +If you had set up PostgreSQL cluster using the Linux package and had set `postgresql['sql_user_password'] = 'md5 digest of secret'`, keep in mind that `gitlab_rails['db_password']` and `geo_secondary['db_password']` contains the plaintext passwords. This is used to let the Rails @@ -263,7 +271,7 @@ Make sure that current node's IP is listed in `postgresql['md5_auth_cidr_addresses']` setting of the read-replica database to allow Rails on this node to connect to PostgreSQL. -After making these changes [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. +After making these changes, [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) so the changes take effect. In the [architecture overview](#architecture-overview) topology, the following GitLab services are enabled on the "frontend" nodes: diff --git a/doc/administration/geo/replication/object_storage.md b/doc/administration/geo/replication/object_storage.md index 8128eaf5310..86db8186a13 100644 --- a/doc/administration/geo/replication/object_storage.md +++ b/doc/administration/geo/replication/object_storage.md @@ -9,7 +9,7 @@ type: howto Geo can be used in combination with Object Storage (AWS S3, or other compatible object storage). -Currently, **secondary** sites can use either: +**Secondary** sites can use one of the following: - The same storage bucket as the **primary** site. - A replicated storage bucket. @@ -41,7 +41,8 @@ whether they are stored on the local file system or in object storage. To enable GitLab replication: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Geo > Nodes**. 1. Select **Edit** on the **secondary** site. 1. In the **Synchronization Settings** section, find the **Allow this secondary node to replicate content on Object Storage** diff --git a/doc/administration/geo/replication/remove_geo_site.md b/doc/administration/geo/replication/remove_geo_site.md index 9d92652daf4..de0ad3fe2fb 100644 --- a/doc/administration/geo/replication/remove_geo_site.md +++ b/doc/administration/geo/replication/remove_geo_site.md @@ -9,7 +9,8 @@ type: howto **Secondary** sites can be removed from the Geo cluster using the Geo administration page of the **primary** site. To remove a **secondary** site: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Geo > Nodes**. 1. For the **secondary** site you want to remove, select **Remove**. 1. Confirm by selecting **Remove** when the prompt appears. diff --git a/doc/administration/geo/replication/security_review.md b/doc/administration/geo/replication/security_review.md index 92eff2faf60..e099c2b2e9d 100644 --- a/doc/administration/geo/replication/security_review.md +++ b/doc/administration/geo/replication/security_review.md @@ -127,9 +127,9 @@ from [owasp.org](https://owasp.org/). ### What details regarding required OS components and lock‐down needs have been defined? -- The supported installation method (Omnibus) packages most components itself. +- The supported Linux package installation method packages most components itself. - There are significant dependencies on the system-installed OpenSSH daemon (Geo - requires users to set up custom authentication methods) and the omnibus or + requires users to set up custom authentication methods) and the Linux package-provided or system-provided PostgreSQL daemon (it must be configured to listen on TCP, additional users and replication slots must be added, etc). - The process for dealing with security updates (for example, if there is a @@ -237,7 +237,7 @@ from [owasp.org](https://owasp.org/). - In transit, data should be encrypted, although the application does permit communication to proceed unencrypted. The two main transits are the **secondary** site's replication process for PostgreSQL, and for Git repositories/files. Both should - be protected using TLS, with the keys for that managed via Omnibus per existing + be protected using TLS, with the keys for that managed by the Linux package per existing configuration for end-user access to GitLab. ### What capabilities exist to detect the leakage of sensitive data? diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index a8736b3ed1d..4047167e4af 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -27,7 +27,8 @@ Before attempting more advanced troubleshooting: On the **primary** site: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Geo > Sites**. We perform the following health checks on each **secondary** site @@ -104,9 +105,7 @@ Checking Geo ... Finished You can also specify a custom NTP server using environment variables. For example: ```shell -export NTP_HOST="ntp.ubuntu.com" -export NTP_TIMEOUT="30" -sudo gitlab-rake gitlab:geo:check +sudo gitlab-rake gitlab:geo:check NTP_HOST="ntp.ubuntu.com" NTP_TIMEOUT="30" ``` The following environment variables are supported. @@ -120,7 +119,9 @@ The following environment variables are supported. #### Sync status Rake task Current sync information can be found manually by running this Rake task on any -node running Rails (Puma, Sidekiq, or Geo Log Cursor) on the Geo **secondary** site: +node running Rails (Puma, Sidekiq, or Geo Log Cursor) on the Geo **secondary** site. + +GitLab does **not** verify objects that are stored in Object Storage. If you are using Object Storage, you will see all of the "verified" checks showing 0 successes. This is expected and not a cause for concern. ```shell sudo gitlab-rake geo:status @@ -208,8 +209,8 @@ Geo finds the current Puma or Sidekiq node's Geo [site](../glossary.md) name in 1. Get the "Geo node name" (there is [an issue to rename the settings to "Geo site name"](https://gitlab.com/gitlab-org/gitlab/-/issues/335944)): - - Omnibus GitLab: Get the `gitlab_rails['geo_node_name']` setting. - - GitLab Helm Charts: Get the `global.geo.nodeName` setting (see [Charts with GitLab Geo](https://docs.gitlab.com/charts/advanced/geo/index.html)). + - Linux package: get the `gitlab_rails['geo_node_name']` setting. + - GitLab Helm charts: get the `global.geo.nodeName` setting (see [Charts with GitLab Geo](https://docs.gitlab.com/charts/advanced/geo/index.html)). 1. If that is not defined, then get the `external_url` setting. This name is used to look up the Geo site with the same **Name** in the **Geo Sites** @@ -598,7 +599,7 @@ To help us resolve this problem, consider commenting on ### Message: `FATAL: could not connect to the primary server: server certificate for "PostgreSQL" does not match host name` -This happens because the PostgreSQL certificate that the Omnibus GitLab package automatically creates contains +This happens because the PostgreSQL certificate that the Linux package automatically creates contains the Common Name `PostgreSQL`, but the replication is connecting to a different host and GitLab attempts to use the `verify-full` SSL mode by default. @@ -852,8 +853,11 @@ to start again from scratch, there are a few steps that can help you: ### Design repository failures on mirrored projects and project imports -On the top bar, under **Main menu > Admin > Geo > Sites**, -if the Design repositories progress bar shows +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. On the left sidebar, select **Geo > Sites**. + +If the Design repositories progress bar shows `Synced` and `Failed` greater than 100%, and negative `Queued`, the instance is likely affected by [a bug in GitLab 13.2 and 13.3](https://gitlab.com/gitlab-org/gitlab/-/issues/241668). @@ -1156,7 +1160,7 @@ get_ctl_options': invalid option: --force (OptionParser::InvalidOption) ``` This can happen with XFS or file systems that list files in lexical order, because the -load order of the Omnibus GitLab command files can be different than expected, and a global function would get redefined. +load order of the Linux package command files can be different than expected, and a global function would get redefined. More details can be found in [the related issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6076). The workaround is to manually run the preflight checks and promote the database, by running @@ -1188,7 +1192,8 @@ site's URL matches its external URL. On the **primary** site: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Geo > Sites**. 1. Find the affected **secondary** site and select **Edit**. 1. Ensure the **URL** field matches the value found in `/etc/gitlab/gitlab.rb` @@ -1206,7 +1211,7 @@ This section documents common error messages reported in the Admin Area on the w GitLab cannot find or doesn't have permission to access the `database_geo.yml` configuration file. -In an Omnibus GitLab installation, the file should be in `/var/opt/gitlab/gitlab-rails/etc`. +In a Linux package installation, the file should be in `/var/opt/gitlab/gitlab-rails/etc`. If it doesn't exist or inadvertent changes have been made to it, run `sudo gitlab-ctl reconfigure` to restore it to its correct state. If this path is mounted on a remote volume, ensure your volume configuration @@ -1239,7 +1244,7 @@ This error message indicates that the replica database in the **secondary** site To restore the database and resume replication, you can do one of the following: - [Reset the Geo secondary site replication](#resetting-geo-secondary-site-replication). -- [Set up a new secondary Geo Omnibus instance](../setup/index.md#using-omnibus-gitlab). +- [Set up a new Geo secondary using the Linux package](../setup/index.md#using-linux-package-installations). If you set up a new secondary from scratch, you must also [remove the old site from the Geo cluster](remove_geo_site.md#removing-secondary-geo-sites). @@ -1258,7 +1263,7 @@ Make sure you follow the [Geo database replication](../setup/database.md) instru ### Geo database version (...) does not match latest migration (...) -If you are using Omnibus GitLab installation, something might have failed during upgrade. You can: +If you are using the Linux package installation, something might have failed during upgrade. You can: - Run `sudo gitlab-ctl reconfigure`. - Manually trigger the database migration by running: `sudo gitlab-rake db:migrate:geo` as root on the **secondary** site. @@ -1359,7 +1364,9 @@ If you have installed GitLab using the Linux package (Omnibus) and have configur - `15.6.0`-`15.6.3` - `15.7.0`-`15.7.1` -This is due to [a bug introduced in the included version of cURL](https://github.com/curl/curl/issues/10122) shipped with Omnibus GitLab 15.4.6 and later. You are encouraged to upgrade to a later version where this has been [fixed](https://about.gitlab.com/releases/2023/01/09/security-release-gitlab-15-7-2-released/). +This is due to [a bug introduced in the included version of cURL](https://github.com/curl/curl/issues/10122) shipped with +the Linux package 15.4.6 and later. You should upgrade to a later version where this has been +[fixed](https://about.gitlab.com/releases/2023/01/09/security-release-gitlab-15-7-2-released/). The bug causes all wildcard domains (`.example.com`) to be ignored except for the last on in the `no_proxy` environment variable list. Therefore, if for any reason you cannot upgrade to a newer version, you can work around the issue by moving your wildcard domain to the end of the list: @@ -1392,8 +1399,9 @@ If you have updated the value of `external_url` in `/etc/gitlab/gitlab.rb` for t In this case, make sure to update the changed URL on all your sites: -1. On the top bar, select **Main menu > Admin**. -1. On the left sidebar, select **Admin > Geo > Sites**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. On the left sidebar, select **Geo > Sites**. 1. Change the URL and save the change. ## Fixing non-PostgreSQL replication failures @@ -1474,7 +1482,31 @@ Geo::PackageFileRegistry.where(last_sync_failure: 'The file is missing on the Ge This iterates over all package files on the secondary, looking at the `verification_checksum` stored in the database (which came from the primary) and then calculate this value on the secondary to check if they match. This -does not change anything in the UI: +does not change anything in the UI. + +For GitLab 14.4 and later: + +```ruby +# Run on secondary +status = {} + +Packages::PackageFile.find_each do |package_file| + primary_checksum = package_file.verification_checksum + secondary_checksum = Packages::PackageFile.sha256_hexdigest(package_file.file.path) + verification_status = (primary_checksum == secondary_checksum) + + status[verification_status.to_s] ||= [] + status[verification_status.to_s] << package_file.id +end + +# Count how many of each value we get +status.keys.each {|key| puts "#{key} count: #{status[key].count}"} + +# See the output in its entirety +status +``` + +For GitLab 14.3 and earlier: ```ruby # Run on secondary @@ -1557,7 +1589,7 @@ registry.replicator.send(:sync_repository) #### Find repository verification failures [Start a Rails console session](../../../administration/operations/rails_console.md#starting-a-rails-console-session) -to gather the following, basic troubleshooting information. +**on the secondary Geo site** to gather more information. WARNING: Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. @@ -1583,14 +1615,14 @@ Geo::ProjectRegistry.sync_failed('repository') #### Resync project and project wiki repositories [Start a Rails console session](../../../administration/operations/rails_console.md#starting-a-rails-console-session) -to enact the following, basic troubleshooting steps. +**on the secondary Geo site** to perform the following changes. WARNING: Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. ##### Queue up all repositories for resync -When you run this, Sidekiq handles each sync. +When you run this, the sync is handled in the background by Sidekiq. ```ruby Geo::ProjectRegistry.update_all(resync_repository: true, resync_wiki: true) @@ -1604,6 +1636,27 @@ project = Project.find_by_full_path('<group/project>') Geo::RepositorySyncService.new(project).execute ``` +##### Sync all failed repositories now + +The following script: + +- Loops over all currently failed repositories. +- Displays the project details and the reasons for the last failure. +- Attempts to resync the repository. +- Reports back if a failure occurs, and why. + +```ruby +Geo::ProjectRegistry.sync_failed('repository').find_each do |p| + begin + project = p.project + puts "#{project.full_path} | id: #{p.project_id} | last error: '#{p.last_repository_sync_failure}'" + Geo::RepositorySyncService.new(project).execute + rescue => e + puts "ID: #{p.project_id} failed: '#{e}'", e.backtrace.join("\n") + end +end ; nil +``` + #### Find repository check failures in a Geo secondary site When [enabled for all projects](../../repository_checks.md#enable-repository-checks-for-all-projects), [Repository checks](../../repository_checks.md) are also performed on Geo secondary sites. The metadata is stored in the Geo tracking database. @@ -1721,7 +1774,7 @@ If the above steps are **not successful**, proceed through the next steps: If different operating systems or different operating system versions are deployed across Geo sites, you should perform a locale data compatibility check before setting up Geo. -Geo uses PostgreSQL and Streaming Replication to replicate data across Geo sites. PostgreSQL uses locale data provided by the operating system's C library for sorting text. If the locale data in the C library is incompatible across Geo sites, erroneous query results that lead to [incorrect behavior on secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/360723). +Geo uses PostgreSQL and Streaming Replication to replicate data across Geo sites. PostgreSQL uses locale data provided by the operating system's C library for sorting text. If the locale data in the C library is incompatible across Geo sites, it can cause erroneous query results that lead to [incorrect behavior on secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/360723). For example, Ubuntu 18.04 (and earlier) and RHEL/Centos7 (and earlier) are incompatible with their later releases. See the [PostgreSQL wiki for more details](https://wiki.postgresql.org/wiki/Locale_data_changes). diff --git a/doc/administration/geo/replication/tuning.md b/doc/administration/geo/replication/tuning.md index 4dc3ba93d66..9e1638643cb 100644 --- a/doc/administration/geo/replication/tuning.md +++ b/doc/administration/geo/replication/tuning.md @@ -14,7 +14,8 @@ in the background. On the **primary** site: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Geo > Sites**. 1. Select **Edit** of the secondary site you want to tune. 1. Under **Tuning settings**, there are several variables that can be tuned to diff --git a/doc/administration/geo/replication/version_specific_upgrades.md b/doc/administration/geo/replication/version_specific_upgrades.md index f8e013a8776..c0215c4551c 100644 --- a/doc/administration/geo/replication/version_specific_upgrades.md +++ b/doc/administration/geo/replication/version_specific_upgrades.md @@ -253,16 +253,16 @@ upgrade to GitLab 13.4 or later. ## Upgrading to GitLab 13.0 Upgrading to GitLab 13.0 requires GitLab 12.10 to already be using PostgreSQL -version 11. For the recommended procedure, see the -[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). +version 11. For the recommended procedure, see how +[to upgrade a Geo instance](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). ## Upgrading to GitLab 12.10 GitLab 12.10 doesn't attempt to upgrade the embedded PostgreSQL server when using Geo, because the PostgreSQL upgrade requires downtime for secondaries while reinitializing streaming replication. It must be upgraded manually. For -the recommended procedure, see the -[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). +the recommended procedure, see how +[to upgrade a Geo instance](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). ## Upgrading to GitLab 12.9 @@ -273,8 +273,8 @@ The issue is fixed in GitLab 12.9.4. Upgrade to GitLab 12.9.4 or later. By default, GitLab 12.9 attempts to upgrade the embedded PostgreSQL server version from 9.6 to 10.12, which requires downtime on secondaries while -reinitializing streaming replication. For the recommended procedure, see the -[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). +reinitializing streaming replication. For the recommended procedure, see how +[to upgrade a Geo instance](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). You can temporarily disable this behavior by running the following before upgrading: @@ -287,8 +287,8 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade By default, GitLab 12.8 attempts to upgrade the embedded PostgreSQL server version from 9.6 to 10.12, which requires downtime on secondaries while -reinitializing streaming replication. For the recommended procedure, see the -[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). +reinitializing streaming replication. For the recommended procedure, see how +[to upgrade a Geo instance](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). You can temporarily disable this behavior by running the following before upgrading: @@ -307,8 +307,8 @@ shipped in 12.7.5. By default, GitLab 12.7 attempts to upgrade the embedded PostgreSQL server version from 9.6 to 10.9, which requires downtime on secondaries while -reinitializing streaming replication. For the recommended procedure, see the -[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). +reinitializing streaming replication. For the recommended procedure, see how +[to upgrade a Geo instance](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). You can temporarily disable this behavior by running the following before upgrading: @@ -321,8 +321,8 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade By default, GitLab 12.6 attempts to upgrade the embedded PostgreSQL server version from 9.6 to 10.9, which requires downtime on secondaries while -reinitializing streaming replication. For the recommended procedure, see the -[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). +reinitializing streaming replication. For the recommended procedure, see how +[to upgrade a Geo instance](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). You can temporarily disable this behavior by running the following before upgrading: @@ -335,8 +335,8 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade By default, GitLab 12.5 attempts to upgrade the embedded PostgreSQL server version from 9.6 to 10.9, which requires downtime on secondaries while -reinitializing streaming replication. For the recommended procedure, see the -[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). +reinitializing streaming replication. For the recommended procedure, see how +[to upgrade a Geo instance](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). You can temporarily disable this behavior by running the following before upgrading: @@ -349,8 +349,8 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade By default, GitLab 12.4 attempts to upgrade the embedded PostgreSQL server version from 9.6 to 10.9, which requires downtime on secondaries while -reinitializing streaming replication. For the recommended procedure, see the -[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). +reinitializing streaming replication. For the recommended procedure, see how +[to upgrade a Geo instance](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). You can temporarily disable this behavior by running the following before upgrading: @@ -365,13 +365,13 @@ WARNING: If the existing PostgreSQL server version is 9.6.x, we recommend upgrading to GitLab 12.4 or later. By default, GitLab 12.3 attempts to upgrade the embedded PostgreSQL server version from 9.6 to 10.9. In certain circumstances, it can -fail. For more information, see the -[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). +fail. For more information, see how +[to upgrade a Geo instance](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). Additionally, if the PostgreSQL upgrade doesn't fail, a successful upgrade requires downtime for secondaries while reinitializing streaming replication. -For the recommended procedure, see the -[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). +For the recommended procedure, see how +[to upgrade a Geo instance](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). ## Upgrading to GitLab 12.2 @@ -379,13 +379,13 @@ WARNING: If the existing PostgreSQL server version is 9.6.x, we recommend upgrading to GitLab 12.4 or later. By default, GitLab 12.2 attempts to upgrade the embedded PostgreSQL server version from 9.6 to 10.9. In certain circumstances, it can -fail. For more information, see the -[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). +fail. For more information, see how +[to upgrade a Geo instance](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). Additionally, if the PostgreSQL upgrade doesn't fail, a successful upgrade requires downtime for secondaries while reinitializing streaming replication. -For the recommended procedure, see the -[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). +For the recommended procedure, see how +[to upgrade a Geo instance](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). GitLab 12.2 includes the following minor PostgreSQL upgrades: @@ -410,13 +410,13 @@ WARNING: If the existing PostgreSQL server version is 9.6.x, we recommend upgrading to GitLab 12.4 or later. By default, GitLab 12.1 attempts to upgrade the embedded PostgreSQL server version from 9.6 to 10.9. In certain circumstances, it can -fail. For more information, see the -[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). +fail. For more information, see how +[to upgrade a Geo instance](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). Additionally, if the PostgreSQL upgrade doesn't fail, a successful upgrade requires downtime for secondaries while reinitializing streaming replication. -For the recommended procedure, see the -[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). +For the recommended procedure, see how +[to upgrade a Geo instance](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). ## Upgrading to GitLab 12.0 diff --git a/doc/administration/geo/secondary_proxy/index.md b/doc/administration/geo/secondary_proxy/index.md index 35ab1d8252c..3081d1c2485 100644 --- a/doc/administration/geo/secondary_proxy/index.md +++ b/doc/administration/geo/secondary_proxy/index.md @@ -122,7 +122,7 @@ for details. To use TLS certificates with Let's Encrypt, you can manually point the domain to one of the Geo sites, generate the certificate, then copy it to all other sites. -- [Viewing projects and designs data from a primary site is not possible when using a unified URL](../index.md#view-replication-data-on-the-primary-site). +- [Viewing projects data from a primary site is not possible when using a unified URL](../index.md#view-replication-data-on-the-primary-site). - When secondary proxying is used together with separate URLs, registering [GitLab runners](https://docs.gitlab.com/runner/) to clone from secondary sites is not supported. The runner registration succeeds, but the clone URL defaults to the primary site. The runner @@ -152,6 +152,8 @@ sites for improved latency and bandwidth nearby. All write requests are proxied The following table details the components currently tested through the Geo secondary site Workhorse proxy. It does not cover all data types. +In this context, accelerated reads refer to read requests served from the secondary site, provided that the data is up to date for the component on the secondary site. If the data on the secondary site is determined to be out of date, the request is forwarded to the primary site. Read requests for components not listed in the table below are always automatically forwarded to the primary site. + | Feature / component | Accelerated reads? | |:----------------------------------------------------|:-----------------------| | Project, wiki, design repository (using the web UI) | **{dotted-circle}** No | @@ -165,11 +167,12 @@ It does not cover all data types. | LFS objects (using Git) | **{check-circle}** Yes | | Pages | **{dotted-circle}** No <sup>2</sup> | | Advanced search (using the web UI) | **{dotted-circle}** No | -| Container registry | **{dotted-circle}** No | +| Container registry | **{dotted-circle}** No <sup>3</sup>| 1. Git reads are served from the local secondary while pushes get proxied to the primary. Selective sync or cases where repositories don't exist locally on the Geo secondary throw a "not found" error. 1. Pages can use the same URL (without access control), but must be configured separately and are not proxied. +1. The container registry is only recommended for Disaster Recovery scenarios. If the secondary site's container registry is not up to date, the read request is served with old data as the request is not forwarded to the primary site. ## Disable Geo proxying @@ -179,7 +182,8 @@ Secondary proxying is enabled by default in GitLab 15.1 on a secondary site even Additionally, the `gitlab-workhorse` service polls `/api/v4/geo/proxy` every 10 seconds. In GitLab 15.2 and later, it is only polled once, if Geo is not enabled. Prior to GitLab 15.2, you can stop this polling by disabling secondary proxying. -You can disable the secondary proxying on each Geo site, separately, by following these steps with Omnibus-based packages: +You can disable the secondary proxying on each Geo site separately by following these steps on a Linux package +installation: 1. SSH into each application node (serving user traffic directly) on your secondary Geo site and add the following environment variable: diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index cf0c40dbbc5..31d0fbdffe0 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -12,14 +12,13 @@ GitLab database to a secondary site's database. You may have to change some values, based on attributes including your database's setup and size. NOTE: -If your GitLab installation uses external (not managed by Omnibus GitLab) -PostgreSQL instances, the Omnibus roles cannot perform all necessary -configuration steps. In this case, use the [Geo with external PostgreSQL instances](external_database.md) -process instead. +If your GitLab installation uses external PostgreSQL instances (not managed by a Linux package installation), +the roles cannot perform all necessary configuration steps. In this case, use the +[Geo with external PostgreSQL instances](external_database.md) process instead. NOTE: The stages of the setup process must be completed in the documented order. -If not, [complete all prior stages](../setup/index.md#using-omnibus-gitlab) before proceeding. +If not, [complete all prior stages](../setup/index.md#using-linux-package-installations) before proceeding. Ensure the **secondary** site is running the same version of GitLab Enterprise Edition as the **primary** site. Confirm you have added a license for a [Premium or Ultimate subscription](https://about.gitlab.com/pricing/) to your **primary** site. @@ -51,10 +50,10 @@ recover. See below for more details. The following guide assumes that: -- You are using Omnibus and therefore you are using PostgreSQL 12 or later, +- You are using the Linux package (so are using PostgreSQL 12 or later), which includes the [`pg_basebackup` tool](https://www.postgresql.org/docs/12/app-pgbasebackup.html). - You have a **primary** site already set up (the GitLab server you are - replicating from), running Omnibus' PostgreSQL (or equivalent version), and + replicating from), running PostgreSQL (or equivalent version) managed by your Linux package installation, and you have a new **secondary** site set up with the same [versions of PostgreSQL](../index.md#requirements-for-running-geo), OS, and GitLab on all sites. @@ -140,7 +139,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o postgresql['sql_replication_password'] = '<md5_hash_of_your_password>' ``` - If you are using an external database not managed by Omnibus GitLab, you need + If you are using an external database not managed by your Linux package installation, you need to create the `gitlab_replicator` user and define a password for that user manually: ```sql @@ -205,7 +204,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o NOTE: If you need to use `0.0.0.0` or `*` as the `listen_address`, you also must add `127.0.0.1/32` to the `postgresql['md5_auth_cidr_addresses']` setting, to allow Rails to connect through - `127.0.0.1`. For more information, see [omnibus-5258](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5258). + `127.0.0.1`. For more information, see [issue 5258](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5258). Depending on your network configuration, the suggested addresses may be incorrect. If your **primary** site and **secondary** sites connect over a local @@ -374,7 +373,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o to the private key, which is **only** present on the **primary** site. 1. Test that the `gitlab-psql` user can connect to the **primary** site's database - (the default Omnibus database name is `gitlabhq_production`): + (the default database name is `gitlabhq_production` on a Linux package installation): ```shell sudo \ @@ -456,8 +455,8 @@ Below is a script that connects the database on the **secondary** site to the database on the **primary** site. This script replicates the database and creates the needed files for streaming replication. -The directories used are the defaults that are set up in Omnibus. If you have -changed any defaults, configure the script accordingly, replacing any directories and paths. +The directories used are the defaults that are set up in a Linux package installation. If you have +changed any defaults, configure the script accordingly (replacing any directories and paths). WARNING: Make sure to run this on the **secondary** site as it removes all PostgreSQL's @@ -537,12 +536,12 @@ You should use PgBouncer if you use GitLab in a highly available configuration with a cluster of nodes supporting a Geo **primary** site and two other clusters of nodes supporting a Geo **secondary** site. You need two PgBouncer nodes: one for the main database and the other for the tracking database. For more information, -see [High Availability with Omnibus GitLab](../../postgresql/replication_and_failover.md). +see [the relevant documentation](../../postgresql/replication_and_failover.md). ### Changing the replication password To change the password for the [replication user](https://wiki.postgresql.org/wiki/Streaming_Replication) -when using Omnibus-managed PostgreSQL instances: +when using PostgreSQL instances managed by a Linux package installation: On the GitLab Geo **primary** site: @@ -628,7 +627,7 @@ to `gitlab.rb` where `<slot_name>` is the name of the replication slot for your ### Migrating a single PostgreSQL node to Patroni -Before the introduction of Patroni, Geo had no Omnibus support for HA setups on the **secondary** site. +Before the introduction of Patroni, Geo had no support for Linux package installations for HA setups on the **secondary** site. With Patroni, this support is now possible. To migrate the existing PostgreSQL to Patroni: @@ -639,7 +638,7 @@ With Patroni, this support is now possible. To migrate the existing PostgreSQL t 1. [Configure a Standby Cluster](#step-4-configure-a-standby-cluster-on-the-secondary-site) on that single node machine. -You end up with a “Standby Cluster” with a single node. That allows you to add additional Patroni nodes by following the same instructions above. +You end up with a _Standby Cluster_ with a single node. That allows you to add additional Patroni nodes by following the same instructions above. ### Patroni support @@ -649,7 +648,7 @@ Using Patroni on a **secondary** site is optional and you don't have to use the nodes on each Geo site. For instructions on how to set up Patroni on the primary site, see the -[PostgreSQL replication and failover with Omnibus GitLab](../../postgresql/replication_and_failover.md#patroni) page. +[relevant documentation](../../postgresql/replication_and_failover.md#patroni). #### Configuring Patroni cluster for a Geo secondary site @@ -743,8 +742,8 @@ Leader is elected on the primary site, you should set up a TCP internal load balancer. This load balancer provides a single endpoint for connecting to the Patroni cluster's Leader. -The Omnibus GitLab packages do not include a Load Balancer. Here's how you -could do it with [HAProxy](https://www.haproxy.org/). +Linux packages do not include a Load Balancer. Here's how you could do it with +[HAProxy](https://www.haproxy.org/). The following IPs and names are used as an example: @@ -787,7 +786,7 @@ three Consul nodes and a minimum of one PgBouncer node. However, it is recommend one PgBouncer node per database node. An internal load balancer (TCP) is required when there is more than one PgBouncer service node. The internal load balancer provides a single endpoint for connecting to the PgBouncer cluster. For more information, -see [High Availability with Omnibus GitLab](../../postgresql/replication_and_failover.md). +see [the relevant documentation](../../postgresql/replication_and_failover.md). On each node running a PgBouncer instance on the **secondary** site: @@ -921,7 +920,7 @@ For each node running a Patroni instance on the secondary site: ### Migrating a single tracking database node to Patroni -Before the introduction of Patroni, Geo provided no Omnibus support for HA setups on +Before the introduction of Patroni, Geo provided no support for Linux package installations for HA setups on the secondary site. With Patroni, it's now possible to support HA setups. However, some restrictions in Patroni @@ -935,7 +934,7 @@ synchronization is required. **Secondary** sites use a separate PostgreSQL installation as a tracking database to keep track of replication status and automatically recover from potential replication issues. -Omnibus automatically configures a tracking database when `roles(['geo_secondary_role'])` is set. +The Linux package automatically configures a tracking database when `roles(['geo_secondary_role'])` is set. If you want to run this database in a highly available configuration, don't use the `geo_secondary_role` above. Instead, follow the instructions below. @@ -945,7 +944,7 @@ If you want to run the Geo tracking database on a single node, see [Configure th A production-ready and secure setup for the tracking PostgreSQL DB requires at least three Consul nodes: two Patroni nodes, and one PgBouncer node on the secondary site. -Because of [omnibus-6587](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6587), Consul can't track multiple +Because of [issue 6587](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6587), Consul can't track multiple services, so these must be different than the nodes used for the Standby Cluster database. Be sure to use [password credentials](../../postgresql/replication_and_failover.md#database-authorization-for-patroni) diff --git a/doc/administration/geo/setup/external_database.md b/doc/administration/geo/setup/external_database.md index 65ea2e6e5af..81541d1cb9c 100644 --- a/doc/administration/geo/setup/external_database.md +++ b/doc/administration/geo/setup/external_database.md @@ -6,17 +6,17 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Geo with external PostgreSQL instances **(PREMIUM SELF)** -This document is relevant if you are using a PostgreSQL instance that is *not -managed by Omnibus*. This includes cloud-managed instances like Amazon RDS, or +This document is relevant if you are using a PostgreSQL instance that is not +managed by the Linux package. This includes cloud-managed instances like Amazon RDS, or manually installed and configured PostgreSQL instances. Ensure that you are using one of the PostgreSQL versions that -[Omnibus ships with](../../package_information/postgresql_versions.md) +the [Linux package ships with](../../package_information/postgresql_versions.md) to [avoid version mismatches](../index.md#requirements-for-running-geo) in case a Geo site has to be rebuilt. NOTE: -We strongly recommend running Omnibus-managed instances as they are actively +We strongly recommend running instances installed using the Linux package as they are actively developed and tested. We aim to be compatible with most external (not managed by Omnibus) databases but we do not guarantee compatibility. @@ -62,8 +62,8 @@ developed and tested. We aim to be compatible with most external To set up an external database, you can either: -- Set up [streaming replication](https://www.postgresql.org/docs/12/warm-standby.html#STREAMING-REPLICATION-SLOTS) yourself (for example Amazon RDS, or bare metal not managed by Omnibus). -- Perform the Omnibus configuration manually as follows. +- Set up [streaming replication](https://www.postgresql.org/docs/12/warm-standby.html#STREAMING-REPLICATION-SLOTS) yourself (for example Amazon RDS, or bare metal not managed by the Linux package). +- Manually perform the configuration of your Linux package installations as follows. #### Leverage your cloud provider's tools to replicate the primary database @@ -142,7 +142,7 @@ hot_standby = on ### Configure **secondary** site to use the external read-replica -With Omnibus, the +With Linux package installations, the [`geo_secondary_role`](https://docs.gitlab.com/omnibus/roles/#gitlab-geo-roles) has three main functions: @@ -179,15 +179,15 @@ To configure the connection to the external read-replica database and enable Log postgresql['enable'] = false ``` -1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) +1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) ### Configure the tracking database **Secondary** sites use a separate PostgreSQL installation as a tracking database to keep track of replication status and automatically recover from -potential replication issues. Omnibus automatically configures a tracking database +potential replication issues. The Linux package automatically configures a tracking database when `roles ['geo_secondary_role']` is set. -If you want to run this database external to Omnibus GitLab, use the following instructions. +If you want to run this database external to your Linux package installation, use the following instructions. If you are using a cloud-managed service for the tracking database, you may need to grant additional roles to your tracking database user (by default, this is @@ -244,7 +244,7 @@ the tracking database on port 5432. geo_postgresql['enable'] = false # don't use internal managed instance ``` -1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) +1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) 1. The reconfigure should automatically create the database. If needed, you can perform this task manually. This task (whether run by itself or during reconfigure) requires the database user to be a superuser. diff --git a/doc/administration/geo/setup/index.md b/doc/administration/geo/setup/index.md index da7e55509e7..359f706a8aa 100644 --- a/doc/administration/geo/setup/index.md +++ b/doc/administration/geo/setup/index.md @@ -18,22 +18,41 @@ type: howto - Ensure the **primary** site has a [GitLab Premium or Ultimate](https://about.gitlab.com/pricing/) subscription to unlock Geo. You only need one license for all the sites. - Confirm the [requirements for running Geo](../index.md#requirements-for-running-geo) are met by all sites. For example, sites must use the same GitLab version, and sites must be able to communicate with each other over certain ports. +- Confirm the **primary** and **secondary** site storage configurations match. If the primary Geo site uses object storage, the secondary Geo site must use it too. For more information, see [Geo with Object storage](../replication/object_storage.md). +- Ensure clocks are synchronized between the **primary** site and the **secondary** site. Synchronized clocks are required for Geo to function correctly. For example, if the clock drift between the **primary** and **secondary** sites exceeds 1 minute, replication fails. -## Using Omnibus GitLab +## Using Linux package installations -If you installed GitLab using the Omnibus packages (highly recommended): +If you installed GitLab using the Linux package (highly recommended), the process for setting up Geo depends on whether you need to set up +a single-node Geo site or a multi-node Geo site. -1. [Set up the database replication](database.md) (`primary (read-write) <-> secondary (read-only)` topology). +### Single-node Geo sites + +If both Geo sites are based on the [1K reference architecture](../../reference_architectures/1k_users.md): + +1. Set up the database replication based on your choice of PostgreSQL instances (`primary (read-write) <-> secondary (read-only)` topology): + - [Using Linux package PostgreSQL instances](database.md) . + - [Using external PostgreSQL instances](external_database.md) 1. [Configure GitLab](../replication/configuration.md) to set the **primary** and **secondary** sites. -1. Optional: [Configure Object storage](../../object_storage.md) +1. Recommended: [Configure unified URLs](../secondary_proxy/index.md) to use a single, unified URL for all Geo sites. +1. Optional: [Configure Object storage replication](../../object_storage.md) 1. Optional: [Configure a secondary LDAP server](../../auth/ldap/index.md) for the **secondary** sites. See [notes on LDAP](../index.md#ldap). -1. Optional: [Configure Geo secondary proxying](../secondary_proxy/index.md) to use a single, unified URL for all Geo sites. This step is recommended to accelerate most read requests while transparently proxying writes to the primary Geo site. +1. Optional: [Configure Container Registry for the secondary site](../replication/container_registry.md). 1. Follow the [Using a Geo Site](../replication/usage.md) guide. +### Multi-node Geo sites + +If one or more of your sites is using the [2K reference architecture](../../reference_architectures/2k_users.md) or larger, see +[Configure Geo for multiple nodes](../replication/multiple_servers.md). + ## Using GitLab Charts [Configure the GitLab chart with GitLab Geo](https://docs.gitlab.com/charts/advanced/geo/). +## Geo and self-compiled installations + +Geo is not supported when you use a [self-compiled GitLab installation](../../../install/installation.md). + ## Post-installation documentation After installing GitLab on the **secondary** sites and performing the initial configuration, see the [following documentation for post-installation information](../index.md#post-installation-documentation). diff --git a/doc/administration/get_started.md b/doc/administration/get_started.md index 0b5de38a78b..60291732a20 100644 --- a/doc/administration/get_started.md +++ b/doc/administration/get_started.md @@ -37,8 +37,8 @@ Watch an overview of [groups and projects](https://www.youtube.com/watch?v=cqb2m Get started: - Create a [project](../user/project/index.md#create-a-project). -- Create a [group](../user/group/manage.md#create-a-group). -- [Add members](../user/group/manage.md#add-users-to-a-group) to the group. +- Create a [group](../user/group/index.md#create-a-group). +- [Add members](../user/group/index.md#add-users-to-a-group) to the group. - Create a [subgroup](../user/group/subgroups/index.md#create-a-subgroup). - [Add members](../user/group/subgroups/index.md#subgroup-membership) to the subgroup. - Enable [external authorization control](../user/admin_area/settings/external_authorization.md#configuration). @@ -126,11 +126,11 @@ GitLab provides backup methods to keep your data safe and recoverable. Whether y ### Back up a GitLab self-managed instance -The routine differs, depending on whether you deployed with Omnibus or the Helm chart. +The routine differs, depending on whether you deployed with the Linux package or the Helm chart. -When you backing up an Omnibus (single node) GitLab server, you can use a single Rake task. +When backing up (single node) GitLab server installed using the Linux package, you can use a single Rake task. -Learn about [backing up Omnibus or Helm variations](../raketasks/backup_restore.md). +Learn about [backing up Linux package or Helm variations](../raketasks/backup_restore.md). This process backs up your entire instance, but does not back up the configuration files. Ensure those are backed up separately. Keep your configuration files and backup archives in a separate location to ensure the encryption keys are not kept with the encrypted data. @@ -138,7 +138,7 @@ Keep your configuration files and backup archives in a separate location to ensu You can restore a backup only to **the exact same version and type** (Community Edition/Enterprise Edition) of GitLab on which it was created. -- Review the [Omnibus backup and restore documentation](https://docs.gitlab.com/omnibus/settings/backups). +- Review the [Linux package (Omnibus) backup and restore documentation](https://docs.gitlab.com/omnibus/settings/backups). - Review the [Helm Chart backup and restore documentation](https://docs.gitlab.com/charts/backup-restore/). ### Back up GitLab SaaS @@ -172,7 +172,7 @@ If your GitLab server contains a lot of Git repository data, you may find the Gi Slowness typically starts at a Git repository data size of around 200 GB. In this case, you might consider using file system snapshots as part of your backup strategy. For example, consider a GitLab server with the following components: -- Using Omnibus GitLab +- Using the Linux package. - Hosted on AWS with an EBS drive containing an ext4 file system mounted at `/var/opt/gitlab`. The EC2 instance meets the requirements for an application data backup by taking an EBS snapshot. The backup includes all repositories, uploads, and PostgreSQL data. diff --git a/doc/administration/git_protocol.md b/doc/administration/git_protocol.md index b996b9efae9..1ece7d773ee 100644 --- a/doc/administration/git_protocol.md +++ b/doc/administration/git_protocol.md @@ -29,7 +29,7 @@ and [All-in-one Docker image](../install/docker.md), the SSH service is already configured to accept the `GIT_PROTOCOL` environment. Users need not do anything more. -For Omnibus GitLab and installations from source, update +For installations from the Linux package or self-compiled installations, update the SSH configuration of your server manually by adding this line to the `/etc/ssh/sshd_config` file: ```plaintext diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md index f3eb2de3f1d..5c6fc370052 100644 --- a/doc/administration/gitaly/configure_gitaly.md +++ b/doc/administration/gitaly/configure_gitaly.md @@ -14,7 +14,7 @@ Configure Gitaly in one of two ways: 1. Edit `/etc/gitlab/gitlab.rb` and add or change the [Gitaly settings](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/1dd07197c7e5ae23626aad5a4a070a800b670380/files/gitlab-config-template/gitlab.rb.template#L1622-1676). -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). :::TabTitle Self-compiled (source) @@ -40,8 +40,8 @@ By default, Gitaly is run on the same server as Gitaly clients and is [configured as above](#configure-gitaly). Single-server installations are best served by this default configuration used by: -- [Omnibus GitLab](https://docs.gitlab.com/omnibus/). -- The GitLab [source installation guide](../../install/installation.md). +- [Linux package installations](https://docs.gitlab.com/omnibus/). +- [Self-compiled installations](../../install/installation.md). However, Gitaly can be deployed to its own server, which can benefit GitLab installations that span multiple machines. @@ -112,12 +112,11 @@ You can use as few as one server with one repository storage if desired. ### Install Gitaly -Install Gitaly on each Gitaly server using either Omnibus GitLab or install it from source: +Install Gitaly on each Gitaly server using either: -- For Omnibus GitLab, [download and install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want but **do not** provide the `EXTERNAL_URL=` value. -- To install from source, follow the steps at - [Install Gitaly](../../install/installation.md#install-gitaly). +- A Linux package installation. [Download and install](https://about.gitlab.com/install/) the Linux package you want + but **do not** provide the `EXTERNAL_URL=` value. +- A self-compiled installation. Follow the steps at [Install Gitaly](../../install/installation.md#install-gitaly). ### Configure Gitaly servers @@ -301,7 +300,7 @@ Configure Gitaly server in one of two ways: } ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). 1. Confirm that Gitaly can perform callbacks to the GitLab internal API: - For GitLab 15.3 and later, run `sudo /opt/gitlab/embedded/bin/gitaly check /var/opt/gitlab/gitaly/config.toml`. - For GitLab 15.2 and earlier, run `sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml`. @@ -424,7 +423,7 @@ Configure Gitaly clients in one of two ways: }) ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). 1. Run `sudo gitlab-rake gitlab:gitaly:check` on the Gitaly client (for example, the Rails application) to confirm it can connect to Gitaly servers. 1. Tail the logs to see the requests: @@ -564,7 +563,7 @@ Disable Gitaly on a GitLab server in one of two ways: gitaly['enable'] = false ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). :::TabTitle Self-compiled (source) @@ -633,7 +632,7 @@ Configure Gitaly with TLS in one of two ways: }) ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). 1. On the Gitaly servers, create the `/etc/gitlab/ssl` directory and copy your key and certificate there: @@ -668,14 +667,14 @@ Configure Gitaly with TLS in one of two ways: } ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). 1. Verify Gitaly traffic is being served over TLS by [observing the types of Gitaly connections](#observe-type-of-gitaly-connections). 1. Optional. Improve security by: 1. Disabling non-TLS connections by commenting out or deleting `gitaly['configuration'][:listen_addr]` in `/etc/gitlab/gitlab.rb`. 1. Saving the file. - 1. [Reconfiguring GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + 1. [Reconfiguring GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). :::TabTitle Self-compiled (source) @@ -880,7 +879,7 @@ to `gitaly['configuration'][:cgroups]` in `/etc/gitlab/gitlab.rb`: - `mountpoint` is where the parent cgroup directory is mounted. Defaults to `/sys/fs/cgroup`. - `hierarchy_root` is the parent cgroup under which Gitaly creates groups, and - is expected to be owned by the user and group Gitaly runs as. Omnibus GitLab + is expected to be owned by the user and group Gitaly runs as. A Linux package installation creates the set of directories `mountpoint/<cpu|memory>/hierarchy_root` when Gitaly starts. - `memory_bytes` is the total memory limit that is imposed collectively on all @@ -939,7 +938,7 @@ in `/etc/gitlab/gitlab.rb`: commands to these cgroups. - `cgroups_mountpoint` is where the parent cgroup directory is mounted. Defaults to `/sys/fs/cgroup`. - `cgroups_hierarchy_root` is the parent cgroup under which Gitaly creates groups, and - is expected to be owned by the user and group Gitaly runs as. Omnibus GitLab + is expected to be owned by the user and group Gitaly runs as. A Linux package installation creates the set of directories `mountpoint/<cpu|memory>/hierarchy_root` when Gitaly starts. - `cgroups_memory_enabled` enables or disables the memory limit on cgroups. @@ -1388,7 +1387,7 @@ objects even if the project doesn't have malicious intent. Instance administrators can override consistency checks if they must process repositories that do not pass consistency checks. -For Omnibus GitLab installations, edit `/etc/gitlab/gitlab.rb` and set the +For Linux package installations, edit `/etc/gitlab/gitlab.rb` and set the following keys (in this example, to disable the `hasDotgit` consistency check): - In [GitLab 15.10](https://gitlab.com/gitlab-org/gitaly/-/issues/4754) and later: @@ -1537,7 +1536,7 @@ Configure Gitaly to sign commits made with the GitLab UI in one of two ways: } ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). :::TabTitle Self-compiled (source) diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 359d4ef90dc..2a3c3da24b3 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -12,7 +12,7 @@ It is used by GitLab to read and write Git data. Gitaly is present in every GitLab installation and coordinates Git repository storage and retrieval. Gitaly can be: -- A background service operating on a single instance Omnibus GitLab (all of +- A background service operating on a single instance Linux package installation (all of GitLab on one machine). - Separated onto its own instance and configured in a full cluster configuration, depending on scaling and availability requirements. @@ -164,11 +164,11 @@ E --> F ### Configure Gitaly -Gitaly comes pre-configured with Omnibus GitLab, which is a configuration +Gitaly comes pre-configured with a Linux package installation, which is a configuration [suitable for up to 1000 users](../reference_architectures/1k_users.md). For: -- Omnibus GitLab installations for up to 2000 users, see [specific Gitaly configuration instructions](../reference_architectures/2k_users.md#configure-gitaly). -- Source installations or custom Gitaly installations, see [Configure Gitaly](configure_gitaly.md). +- Linux package installations for up to 2000 users, see [specific Gitaly configuration instructions](../reference_architectures/2k_users.md#configure-gitaly). +- Self-compiled installations or custom Gitaly installations, see [Configure Gitaly](configure_gitaly.md). GitLab installations for more than 2000 active users performing daily Git write operation may be best suited by using Gitaly Cluster. @@ -481,8 +481,7 @@ You can [monitor distribution of reads](monitoring.md#monitor-gitaly-cluster) us #### Strong consistency -> - In GitLab 13.6 to 13.12, strong consistency must be manually configured. Refer to [the 13.12 documentation](https://docs.gitlab.com/13.12/ee/administration/gitaly/praefect.html#strong-consistency). -> - In GitLab 14.0, strong consistency is the primary replication method. +> In GitLab 14.0, strong consistency is the primary replication method. Gitaly Cluster provides strong consistency by writing changes synchronously to all healthy, up-to-date replicas. If a replica is outdated or unhealthy at the time of the transaction, the write is asynchronously replicated to it. @@ -596,7 +595,7 @@ Direct access to Git uses code in GitLab known as the "Rugged patches". Before Gitaly existed, what are now Gitaly clients accessed Git repositories directly, either: -- On a local disk in the case of a single-machine Omnibus GitLab installation. +- On a local disk in the case of a single-machine Linux package installation. - Using NFS in the case of a horizontally-scaled GitLab installation. In addition to running plain `git` commands, GitLab used a Ruby library called diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index 51201ec442f..f80a5192c55 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -90,7 +90,7 @@ GitLab application database. ## Setup Instructions -If you [installed](https://about.gitlab.com/install/) GitLab using the Omnibus GitLab package +If you [installed](https://about.gitlab.com/install/) GitLab using the Linux package (highly recommended), follow the steps below: 1. [Preparation](#preparation) @@ -107,7 +107,7 @@ Before beginning, you should already have a working GitLab instance. [Learn how to install GitLab](https://about.gitlab.com/install/). Provision a PostgreSQL server. You should use the PostgreSQL that is shipped -with Omnibus GitLab and use it to configure the PostgreSQL database. You can use an +with the Linux package and use it to configure the PostgreSQL database. You can use an external PostgreSQL server (version 11 or newer) but you must set it up [manually](#manual-database-setup). Prepare all your new nodes by [installing GitLab](https://about.gitlab.com/install/). You need: @@ -159,7 +159,7 @@ with secure tokens as you complete the setup process. We note in the instructions below where these secrets are required. NOTE: -Omnibus GitLab installations can use `gitlab-secrets.json` for `GITLAB_SHELL_SECRET_TOKEN`. +Linux package installations can use `gitlab-secrets.json` for `GITLAB_SHELL_SECRET_TOKEN`. ### Customize time server setting @@ -178,7 +178,7 @@ The replication state is internal to each instance of GitLab and should not be replicated. These instructions help set up a single PostgreSQL database, which creates a single point of failure. To avoid this, you can configure your own clustered -PostgreSQL. Support for PostgreSQL replication and failover using Omnibus GitLab is proposed in [epic 7814](https://gitlab.com/groups/gitlab-org/-/epics/7814). +PostgreSQL. Support for PostgreSQL replication and failover using the Linux package is proposed in [epic 7814](https://gitlab.com/groups/gitlab-org/-/epics/7814). Clustered database support for other databases (for example, Praefect and Geo databases) is proposed in [issue 7292](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7292). @@ -198,7 +198,7 @@ Setting up PostgreSQL creates empty Praefect tables. For more information, see t #### Running GitLab and Praefect databases on the same server The GitLab application database and the Praefect database can be run on the same server. However, Praefect should have -its own database server when using Omnibus GitLab PostgreSQL. If there is a failover, Praefect isn't aware and starts to +its own database server when using PostgreSQL from the Linux package. If there is a failover, Praefect isn't aware and starts to fail as the database it's trying to use would either: - Be unavailable. @@ -213,10 +213,10 @@ To complete this section you need: - A PostgreSQL user with permissions to manage the database server In this section, we configure the PostgreSQL database. This can be used for both external -and Omnibus-provided PostgreSQL server. +and Linux package-provided PostgreSQL server. To run the following instructions, you can use the Praefect node, where `psql` is installed -by Omnibus GitLab (`/opt/gitlab/embedded/bin/psql`). If you are using the Omnibus-provided +by the Linux package (`/opt/gitlab/embedded/bin/psql`). If you are using the Linux package-provided PostgreSQL you can use `gitlab-psql` on the PostgreSQL node instead: 1. Create a new user `praefect` to be used by Praefect: @@ -233,11 +233,11 @@ PostgreSQL you can use `gitlab-psql` on the PostgreSQL node instead: CREATE DATABASE praefect_production WITH OWNER praefect ENCODING UTF8; ``` -For using Omnibus-provided PgBouncer you need to take the following additional steps. We strongly -recommend using the PostgreSQL that is shipped with Omnibus as the backend. The following -instructions only work on Omnibus-provided PostgreSQL: +When using the Linux package-provided PgBouncer, you need to take the following additional steps. We strongly +recommend using the PostgreSQL that is shipped with the Linux package as the backend. The following +instructions only work on the Linux package-provided PostgreSQL: -1. For Omnibus-provided PgBouncer, you need to use the hash of `praefect` password instead the of the +1. For the Linux package-provided PgBouncer, you need to use the hash of `praefect` password instead the of the actual password: ```sql @@ -247,7 +247,7 @@ instructions only work on Omnibus-provided PostgreSQL: Replace `<PRAEFECT_SQL_PASSWORD_HASH>` with the hash of the password you generated in the preparation step. It is prefixed with `md5` literal. -1. The PgBouncer that is shipped with Omnibus is configured to use [`auth_query`](https://www.pgbouncer.org/config.html#generic-settings) +1. The PgBouncer that is shipped with the Linux package is configured to use [`auth_query`](https://www.pgbouncer.org/config.html#generic-settings) and uses `pg_shadow_lookup` function. You need to create this function in `praefect_production` database: @@ -447,7 +447,7 @@ praefect['configuration'] = { With this configuration, Praefect uses PgBouncer for both connection types. NOTE: -Omnibus GitLab handles the authentication requirements (using `auth_query`), but if you are preparing +Linux package installations handle the authentication requirements (using `auth_query`), but if you are preparing your databases manually and configuring an external PgBouncer, you must include `praefect` user and its password in the file used by PgBouncer. For example, `userlist.txt` if the [`auth_file`](https://www.pgbouncer.org/config.html#auth_file) configuration option is set. For more details, consult the PgBouncer documentation. @@ -670,7 +670,7 @@ Updates to example must be made at: 1. Enable [distribution of reads](index.md#distributed-reads). 1. Save the changes to `/etc/gitlab/gitlab.rb` and - [reconfigure Praefect](../restart_gitlab.md#omnibus-gitlab-reconfigure): + [reconfigure Praefect](../restart_gitlab.md#reconfigure-a-linux-package-installation): ```shell gitlab-ctl reconfigure @@ -694,7 +694,7 @@ Updates to example must be made at: additional configuration changes can be done and then reconfigure can be run manually. 1. Save the changes to `/etc/gitlab/gitlab.rb` and - [reconfigure Praefect](../restart_gitlab.md#omnibus-gitlab-reconfigure): + [reconfigure Praefect](../restart_gitlab.md#reconfigure-a-linux-package-installation): ```shell gitlab-ctl reconfigure @@ -702,7 +702,7 @@ Updates to example must be made at: 1. To ensure that Praefect [has updated its Prometheus listen address](https://gitlab.com/gitlab-org/gitaly/-/issues/2734), - [restart Praefect](../restart_gitlab.md#omnibus-gitlab-restart): + [restart Praefect](../restart_gitlab.md#reconfigure-a-linux-package-installation): ```shell gitlab-ctl restart praefect @@ -758,9 +758,9 @@ Note the following: } ``` -To configure Praefect with TLS: +Configure Praefect with TLS. -**For Omnibus GitLab** +For Linux package installations: 1. Create certificates for Praefect servers. @@ -788,7 +788,7 @@ To configure Praefect with TLS: } ``` -1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file and [reconfigure](../restart_gitlab.md#reconfigure-a-linux-package-installation). 1. On the Praefect clients (including each Gitaly server), copy the certificates, or their certificate authority, into `/etc/gitlab/trusted-certs`: @@ -809,9 +809,9 @@ To configure Praefect with TLS: }) ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). -**For installations from source** +For self-compiled installations: 1. Create certificates for Praefect servers. 1. On the Praefect servers, create the `/etc/gitlab/ssl` directory and copy your key and certificate @@ -950,7 +950,7 @@ You can also appoint an authoritative name server by setting it in this format: praefect['consul_service_name'] = 'praefect' ``` -1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file and [reconfigure](../restart_gitlab.md#reconfigure-a-linux-package-installation). 1. On the Praefect clients (except Gitaly servers), edit `git_data_dirs` in `/etc/gitlab/gitlab.rb` as follows. Replace `PRAEFECT_SERVICE_DISCOVERY_ADDRESS` with Praefect service discovery address, such as `praefect.service.consul`. @@ -964,7 +964,7 @@ with Praefect service discovery address, such as `praefect.service.consul`. }) ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). :::TabTitle Self-compiled (source) @@ -1091,7 +1091,7 @@ For more information on Gitaly server configuration, see our 1. Copy `/etc/gitlab/gitlab-secrets.json` from the Gitaly client to same path on the Gitaly servers and any other Gitaly clients. - 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) on Gitaly servers. + 1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) on Gitaly servers. - Method 2: @@ -1146,7 +1146,7 @@ For more information on Gitaly server configuration, see our ``` 1. Save the changes to `/etc/gitlab/gitlab.rb` and - [reconfigure Gitaly](../restart_gitlab.md#omnibus-gitlab-reconfigure): + [reconfigure Gitaly](../restart_gitlab.md#reconfigure-a-linux-package-installation): ```shell gitlab-ctl reconfigure @@ -1154,7 +1154,7 @@ For more information on Gitaly server configuration, see our 1. To ensure that Gitaly [has updated its Prometheus listen address](https://gitlab.com/gitlab-org/gitaly/-/issues/2734), - [restart Gitaly](../restart_gitlab.md#omnibus-gitlab-restart): + [restart Gitaly](../restart_gitlab.md#reconfigure-a-linux-package-installation): ```shell gitlab-ctl restart gitaly @@ -1273,7 +1273,7 @@ Particular attention should be shown to: 1. Copy `/etc/gitlab/gitlab-secrets.json` from the Gitaly client to same path on the Gitaly servers and any other Gitaly clients. - 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) on Gitaly servers. + 1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) on Gitaly servers. - Method 2: @@ -1317,7 +1317,7 @@ Particular attention should be shown to: ] ``` -1. Save the changes to `/etc/gitlab/gitlab.rb` and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure): +1. Save the changes to `/etc/gitlab/gitlab.rb` and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation): ```shell gitlab-ctl reconfigure @@ -1335,7 +1335,8 @@ Particular attention should be shown to: 1. Check that the Praefect storage is configured to store new repositories: - 1. On the top bar, select **Main menu > Admin**. + 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Repository**. 1. Expand the **Repository storage** section. @@ -1388,7 +1389,7 @@ To get started quickly: ``` 1. Save the changes to `/etc/gitlab/gitlab.rb` and - [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure): + [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation): ```shell gitlab-ctl reconfigure @@ -1424,49 +1425,59 @@ Praefect does not store the actual replication factor, but assigns enough storag so the desired replication factor is met. If a storage node is later removed from the virtual storage, the replication factor of repositories assigned to the storage is decreased accordingly. -You can configure: +You can configure either: -- A default replication factor for each virtual storage that is applied to newly-created repositories. - The configuration is added to the `/etc/gitlab/gitlab.rb` file: +- A default replication factor for each virtual storage that is applied to newly created repositories. +- A replication factor for an existing repository with the `set-replication-factor` subcommand. - ```ruby - praefect['configuration'] = { - # ... - virtual_storage: [ - { - # ... - name: 'default', - default_replication_factor: 1, - }, - ], - } - ``` +### Configure default replication factor -- A replication factor for an existing repository using the `set-replication-factor` sub-command. - `set-replication-factor` automatically assigns or unassigns random storage nodes as - necessary to reach the desired replication factor. The repository's primary node is - always assigned first and is never unassigned. +If `default_replication_factor` is unset, the repositories are always replicated on every node defined in +`virtual_storages`. If a new node is introduced to the virtual storage, both new and existing repositories are +replicated to the node automatically. - ```shell - sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml set-replication-factor -virtual-storage <virtual-storage> -repository <relative-path> -replication-factor <replication-factor> - ``` +For large Gitaly Cluster deployments with many Gitaly nodes, replicating a repository to every storage is often not +sensible and can cause problems. The higher the replication factor, the higher the pressure on the primary repository. +You should explicitly set the default replication factor for large Gitaly Cluster deployments. + +To configure a default replication factor, add configuration to the `/etc/gitlab/gitlab.rb` file: + +```ruby +praefect['configuration'] = { + # ... + virtual_storage: [ + { + # ... + name: 'default', + default_replication_factor: 1, + }, + ], +} +``` - - `-virtual-storage` is the virtual storage the repository is located in. - - `-repository` is the repository's relative path in the storage. - - `-replication-factor` is the desired replication factor of the repository. The minimum value is - `1`, as the primary needs a copy of the repository. The maximum replication factor is the number of - storages in the virtual storage. +### Configure replication factor for existing repositories - On success, the assigned host storages are printed. For example: +The `set-replication-factor` subcommand automatically assigns or unassigns random storage nodes as +necessary to reach the desired replication factor. The repository's primary node is +always assigned first and is never unassigned. - ```shell - $ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml set-replication-factor -virtual-storage default -repository @hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.git -replication-factor 2 +```shell +sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml set-replication-factor -virtual-storage <virtual-storage> -repository <relative-path> -replication-factor <replication-factor> +``` - current assignments: gitaly-1, gitaly-2 - ``` +- `-virtual-storage` is the virtual storage the repository is located in. +- `-repository` is the repository's relative path in the storage. +- `-replication-factor` is the desired replication factor of the repository. The minimum value is + `1`, as the primary needs a copy of the repository. The maximum replication factor is the number of + storages in the virtual storage. + +On success, the assigned host storages are printed. For example: -If `default_replication_factor` is unset, the repositories are always replicated on every node defined in `virtual_storages`. If a new -node is introduced to the virtual storage, both new and existing repositories are replicated to the node automatically. +```shell +$ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml set-replication-factor -virtual-storage default -repository @hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.git -replication-factor 2 + +current assignments: gitaly-1, gitaly-2 +``` ### Repository storage recommendations diff --git a/doc/administration/gitaly/recovery.md b/doc/administration/gitaly/recovery.md index 7d27a633512..dbbed0f60ba 100644 --- a/doc/administration/gitaly/recovery.md +++ b/doc/administration/gitaly/recovery.md @@ -100,7 +100,7 @@ The following parameters are available: some assigned copies that are not available. NOTE: -`dataloss` is still in [Beta](../../policy/alpha-beta-support.md#beta) and the output format is subject to change. +`dataloss` is still in [Beta](../../policy/experiment-beta-support.md#beta) and the output format is subject to change. To check for repositories with outdated primaries or for unavailable repositories, run: diff --git a/doc/administration/gitaly/reference.md b/doc/administration/gitaly/reference.md index 81b3faf859e..aa04c9a92c4 100644 --- a/doc/administration/gitaly/reference.md +++ b/doc/administration/gitaly/reference.md @@ -7,11 +7,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Gitaly reference **(FREE SELF)** Gitaly is configured via a [TOML](https://github.com/toml-lang/toml) -configuration file. Unlike installations from source, in Omnibus GitLab, you -would not edit this file directly. For Omnibus GitLab installations, the default file location is `/var/opt/gitlab/gitaly/config.toml`. +configuration file. Unlike self-compiled installations, in Linux package installations you +would not edit this file directly. For Linux package installations, the default file location is `/var/opt/gitlab/gitaly/config.toml`. -The configuration file is passed as an argument to the `gitaly` executable, which is usually done by either Omnibus -GitLab or your [init](https://en.wikipedia.org/wiki/Init) script. +The configuration file is passed as an argument to the `gitaly` executable, which is usually done by either your Linux +package installation or your [init](https://en.wikipedia.org/wiki/Init) script. An [example configuration file](https://gitlab.com/gitlab-org/gitaly/blob/master/config.toml.example) can be found in the Gitaly project. diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index dbbd348556c..afef787e9c3 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -20,7 +20,8 @@ and our advice on [parsing the `gitaly/current` file](../logs/log_parsing.md#par When using standalone Gitaly servers, you must make sure they are the same version as GitLab to ensure full compatibility: -1. On the top bar, select **Main menu > Admin** on your GitLab instance. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Overview > Gitaly Servers**. 1. Confirm all Gitaly servers indicate that they are up to date. @@ -85,7 +86,7 @@ check for an SSL or TLS problem: ``` Check whether `Verify return code` field indicates a -[known Omnibus GitLab configuration problem](https://docs.gitlab.com/omnibus/settings/ssl/index.html). +[known Linux package installation configuration problem](https://docs.gitlab.com/omnibus/settings/ssl/index.html). If `openssl` succeeds but `gitlab-rake gitlab:gitaly:check` fails, check [certificate requirements](configure_gitaly.md#certificate-requirements) for Gitaly. @@ -93,7 +94,7 @@ check [certificate requirements](configure_gitaly.md#certificate-requirements) f ### Server side gRPC logs gRPC tracing can also be enabled in Gitaly itself with the `GODEBUG=http2debug` -environment variable. To set this in an Omnibus GitLab install: +environment variable. To set this in a Linux package installation: 1. Add the following to your `gitlab.rb` file: @@ -103,7 +104,7 @@ environment variable. To set this in an Omnibus GitLab install: } ``` -1. [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab. +1. [Reconfigure](../restart_gitlab.md#reconfigure-a-linux-package-installation) GitLab. ### Correlating Git processes with RPCs @@ -216,7 +217,7 @@ Confirm the following are all true: To fix this problem, confirm that your [`gitlab-secrets.json` file](configure_gitaly.md#configure-gitaly-servers) on the Gitaly server matches the one on Gitaly client. If it doesn't match, update the secrets file on the Gitaly server to match the Gitaly client, then -[reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). +[reconfigure](../restart_gitlab.md#reconfigure-a-linux-package-installation). If you've confirmed that your `gitlab-secrets.json` file is the same on all Gitaly servers and clients, the application might be fetching this secret from a different file. Your Gitaly server's @@ -400,6 +401,10 @@ To resolve this, remove the `noexec` option from the file system mount. An alter 1. Add `gitaly['runtime_dir'] = '<PATH_WITH_EXEC_PERM>'` to `/etc/gitlab/gitlab.rb` and specify a location without `noexec` set. 1. Run `sudo gitlab-ctl reconfigure`. +### Commit signing fails with `invalid argument: signing key is encrypted` or `invalid data: tag byte does not have MSB set.` + +Because Gitaly commit signing is headless and not associated with a specific user, the GPG signing key must be created without a passphrase, or the passphrase must be removed before export. + ## Troubleshoot Praefect (Gitaly Cluster) The following sections provide possible solutions to Gitaly Cluster errors. diff --git a/doc/administration/housekeeping.md b/doc/administration/housekeeping.md index 3a2b3657145..eed14fe1bf1 100644 --- a/doc/administration/housekeeping.md +++ b/doc/administration/housekeeping.md @@ -76,7 +76,8 @@ frequently. You can change how often Gitaly is asked to optimize a repository. -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Repository**. 1. Expand **Repository maintenance**. 1. In the **Housekeeping** section, configure the housekeeping options. @@ -108,7 +109,7 @@ housekeeping tasks. The manual trigger can be useful when either: To trigger housekeeping tasks manually: -1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. 1. On the left sidebar, select **Settings > General**. 1. Expand **Advanced**. 1. Select **Run housekeeping**. @@ -135,7 +136,7 @@ reduce the likelihood of such race conditions. To trigger a manual prune of unreachable objects: -1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. 1. On the left sidebar, select **Settings > General**. 1. Expand **Advanced**. 1. Select **Run housekeeping**. diff --git a/doc/administration/inactive_project_deletion.md b/doc/administration/inactive_project_deletion.md index aad6a420246..278d585f2d9 100644 --- a/doc/administration/inactive_project_deletion.md +++ b/doc/administration/inactive_project_deletion.md @@ -23,7 +23,8 @@ For the default setting on GitLab.com, see the [GitLab.com settings page](../use To configure deletion of inactive projects: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Repository**. 1. Expand **Repository maintenance**. 1. In the **Inactive project deletion** section, select **Delete inactive projects**. diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index 3b4eaeb161c..0024c42972b 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -156,7 +156,7 @@ If the sender's address is spoofed, the reject notice is delivered to the spoofe `FROM` address, which can cause the mail server's IP or domain to appear on a block list. -### Omnibus package installations +### Linux package installations 1. Find the `incoming_email` section in `/etc/gitlab/gitlab.rb`, enable the feature and fill in the details for your specific IMAP server and email account (see [examples](#configuration-examples) below). @@ -270,7 +270,7 @@ Reply by email should now be working. Example configuration for Postfix mail server. Assumes mailbox `incoming@gitlab.example.com`. -Example for Omnibus installs: +Example for Linux package installations: ```ruby gitlab_rails['incoming_email_enabled'] = true @@ -362,7 +362,7 @@ Example configuration for Gmail/Google Workspace. Assumes mailbox `gitlab-incomi NOTE: `incoming_email_email` cannot be a Gmail alias account. -Example for Omnibus installs: +Example for Linux package installations: ```ruby gitlab_rails['incoming_email_enabled'] = true @@ -459,7 +459,7 @@ Exchange does not support sub-addressing, only two options exist: Assumes the catch-all mailbox `incoming@exchange.example.com`. -Example for Omnibus installs: +Example for Linux package installations: ```ruby gitlab_rails['incoming_email_enabled'] = true @@ -533,7 +533,7 @@ Cannot support [Service Desk](../user/project/service_desk.md). Assumes the dedicated email address `incoming@exchange.example.com`. -Example for Omnibus installs: +Example for Linux package installations: ```ruby gitlab_rails['incoming_email_enabled'] = true @@ -617,7 +617,7 @@ To enable sub-addressing: Disconnect-ExchangeOnline ``` -This example for Omnibus GitLab assumes the mailbox `incoming@office365.example.com`: +This example for Linux package installations assumes the mailbox `incoming@office365.example.com`: ```ruby gitlab_rails['incoming_email_enabled'] = true @@ -678,7 +678,7 @@ incoming_email: ##### Catch-all mailbox -This example for Omnibus installs assumes the catch-all mailbox `incoming@office365.example.com`: +This example for Linux package installations assumes the catch-all mailbox `incoming@office365.example.com`: ```ruby gitlab_rails['incoming_email_enabled'] = true @@ -743,7 +743,7 @@ NOTE: Supports [Reply by Email](reply_by_email.md) only. Cannot support [Service Desk](../user/project/service_desk.md). -This example for Omnibus installs assumes the dedicated email address `incoming@office365.example.com`: +This example for Linux package installations assumes the dedicated email address `incoming@office365.example.com`: ```ruby gitlab_rails['incoming_email_enabled'] = true @@ -821,7 +821,7 @@ To mitigate security concerns, we recommend configuring an application access policy which limits the mailbox access for all accounts, as described in [Microsoft documentation](https://learn.microsoft.com/en-us/graph/auth-limit-mailbox-access). -This example for Omnibus GitLab assumes you're using the following mailbox: `incoming@example.onmicrosoft.com`: +This example for Linux package installations assumes you're using the following mailbox: `incoming@example.onmicrosoft.com`: ##### Configure Microsoft Graph diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index 03c7c51251b..df364a3f737 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -781,7 +781,7 @@ Plan.default.actual_limits.update!(dast_profile_schedules: 50) ### Maximum size and depth of CI/CD configuration YAML files -The default maximum size of a CI/CD configuration YAML file is 1 megabyte and the +The default maximum size of a single CI/CD configuration YAML file is 1 megabyte and the default depth is 100. You can change these limits in the [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): diff --git a/doc/administration/instance_review.md b/doc/administration/instance_review.md index 277595190b3..6a2ead82538 100644 --- a/doc/administration/instance_review.md +++ b/doc/administration/instance_review.md @@ -20,7 +20,7 @@ details and contact you with suggestions to improve your use of GitLab. To request an instance review: 1. Sign in as an administrator. -1. On the top bar, in the upper-right corner, select your avatar. +1. On the left sidebar, select your avatar. 1. Select **Get a free instance review**. ![Instance review](img/instance_review_v14_7.png) diff --git a/doc/administration/integration/diagrams_net.md b/doc/administration/integration/diagrams_net.md new file mode 100644 index 00000000000..a4e8528fb25 --- /dev/null +++ b/doc/administration/integration/diagrams_net.md @@ -0,0 +1,54 @@ +--- +stage: Create +group: Source Code +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" +type: reference, howto +--- + +# Diagrams.net **(FREE)** + +With the [diagrams.net](https://www.diagrams.net/) integration, you can create and embed SVG diagrams in wikis. +The diagram editor is available in both the Markdown editor and the content editor. + +On GitLab.com, this integration is enabled for all SaaS users and does not require any additional configuration. + +On self-managed GitLab, you can choose to integrate with the free [diagrams.net](https://www.diagrams.net/) +website, or use a self-managed diagrams.net site in offline environments. + +To set up the integration on a self-managed instance, you must: + +1. Choose to integrate with the free diagrams.net website or + [configure your diagrams.net server](#configure-your-diagramsnet-server). +1. [Enable the integration](#enable-diagramsnet-integration). + +After completing the integration, the diagrams.net editor opens with the URL you provided. + +## Configure your diagrams.net server + +You can set up your own diagrams.net server to generate the diagrams. + +This is a required step for users on offline (or "air-gapped") self-managed GitLab installations. + +For example, to run a diagrams.net container in Docker, run the following command: + +```shell +docker run -it --rm --name="draw" -p 8080:8080 -p 8443:8443 jgraph/drawio +``` + +Make note of the hostname of the server running the container, to be used as the diagrams.net URL +when you enable the integration. + +For more information, see [Run your own diagrams.net server with Docker](https://www.diagrams.net/blog/diagrams-docker-app). + +## Enable Diagrams.net integration + +1. Sign in to GitLab as an [Administrator](../../user/permissions.md) user. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. On the left sidebar, select **Settings > General**. +1. Expand **Diagrams.net**. +1. Select the **Enable Diagrams.net** checkbox. +1. Enter the Diagrams.net URL. To connect to: + - The free public instance: enter `https://embed.diagrams.net`. + - A self-managed diagrams.net instance: enter the URL you [configured earlier](#configure-your-diagramsnet-server). +1. Select **Save changes**. diff --git a/doc/administration/integration/kroki.md b/doc/administration/integration/kroki.md index f90458200b3..0356212d6dd 100644 --- a/doc/administration/integration/kroki.md +++ b/doc/administration/integration/kroki.md @@ -61,7 +61,8 @@ read the [Kroki installation](https://docs.kroki.io/kroki/setup/install/#_images You need to enable Kroki integration from Settings under Admin Area. To do that, sign in with an administrator account and follow these steps: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. Go to **Settings > General**. 1. Expand the **Kroki** section. 1. Select **Enable Kroki** checkbox. diff --git a/doc/administration/integration/mailgun.md b/doc/administration/integration/mailgun.md index 218b2888033..87428d27c66 100644 --- a/doc/administration/integration/mailgun.md +++ b/doc/administration/integration/mailgun.md @@ -43,7 +43,8 @@ After configuring your Mailgun domain for the webhook endpoints, you're ready to enable the Mailgun integration: 1. Sign in to GitLab as an [Administrator](../../user/permissions.md) user. -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, go to **Settings > General** and expand the **Mailgun** section. 1. Select the **Enable Mailgun** checkbox. 1. Enter the Mailgun HTTP webhook signing key as described in diff --git a/doc/administration/integration/plantuml.md b/doc/administration/integration/plantuml.md index fcfae6cbe70..9c5e07eedaa 100644 --- a/doc/administration/integration/plantuml.md +++ b/doc/administration/integration/plantuml.md @@ -10,12 +10,7 @@ type: reference, howto With the [PlantUML](https://plantuml.com) integration, you can create diagrams in snippets, wikis, and repositories. This integration is enabled on GitLab.com for all SaaS users and does not require any additional configuration. -To set up the integration on a self-managed instance, you must: - -1. [Configure your PlantUML server](#configure-your-plantuml-server). -1. [Configure local PlantUML access](#configure-local-plantuml-access). -1. [Configure PlantUML security](#configure-plantuml-security). -1. [Enable the integration](#enable-plantuml-integration). +To set up the integration on a self-managed instance, you must [configure your PlantUML server](#configure-your-plantuml-server). After completing the integration, PlantUML converts `plantuml` blocks to an HTML image tag, with the source pointing to the PlantUML instance. The PlantUML @@ -123,41 +118,7 @@ services: container_name: plantuml ``` -### Debian/Ubuntu - -You can install and configure a PlantUML server in Debian/Ubuntu distributions -using Tomcat: - -1. Run these commands to create a `plantuml.war` file from the source code: - - ```shell - sudo apt-get install graphviz openjdk-8-jdk git-core maven - git clone https://github.com/plantuml/plantuml-server.git - cd plantuml-server - mvn package - ``` - -1. Deploy the `.war` file from the previous step with these commands: - - ```shell - sudo apt-get install tomcat8 - sudo cp target/plantuml.war /var/lib/tomcat8/webapps/plantuml.war - sudo chown tomcat8:tomcat8 /var/lib/tomcat8/webapps/plantuml.war - sudo service tomcat8 restart - ``` - -The Tomcat service should restart. After the restart is complete, the -PlantUML integration is ready and listening for requests on port 8080: -`http://localhost:8080/plantuml` - -To change these defaults, edit the `/etc/tomcat8/server.xml` file. - -NOTE: -The default URL is different when using this approach. The Docker-based image -makes the service available at the root URL, with no relative path. Adjust -the configuration below accordingly. - -## Configure local PlantUML access +#### Configure local PlantUML access The PlantUML server runs locally on your server, so it can't be accessed externally by default. Your server must catch external PlantUML @@ -180,9 +141,6 @@ To enable this redirection: ```ruby # Docker deployment nginx['custom_gitlab_server_config'] = "location /-/plantuml/ { \n rewrite ^/-/plantuml/(.*) /$1 break;\n proxy_cache off; \n proxy_pass http://plantuml:8080/; \n}\n" - - # Built from source - nginx['custom_gitlab_server_config'] = "location /-/plantuml { \n rewrite ^/-/plantuml/(.*) /$1 break;\n proxy_cache off; \n proxy_pass http://localhost:8080/plantuml; \n}\n" ``` 1. To activate the changes, run the following command: @@ -191,6 +149,146 @@ To enable this redirection: sudo gitlab-ctl reconfigure ``` +### Debian/Ubuntu + +You can install and configure a PlantUML server in Debian/Ubuntu distributions +using Tomcat or Jetty. + +Prerequisites: + +- JRE/JDK version 11 or later. +- Apache Maven version 3.0.2 or later. +- (Recommended) Jetty version 11 or later. +- (Recommended) Tomcat version 10 or later. + +#### Installation + +PlantUML recommends to install Tomcat 10 or above. The scope of this page only +includes setting up a basic Tomcat server. For more production-ready configurations, +see the [Tomcat Documentation](https://tomcat.apache.org/tomcat-10.1-doc/index.html). + +1. Install JDK/JRE 11 and Maven: + + ```shell + sudo apt update + sudo apt-get install graphviz default-jdk git-core maven + ``` + +1. Add a user for Tomcat: + + ```shell + sudo useradd -m -d /opt/tomcat -U -s /bin/false tomcat + ``` + +1. Install and configure Tomcat 10: + + ```shell + cd /tmp & wget https://dlcdn.apache.org/tomcat/tomcat-10/v10.1.9/bin/apache-tomcat-10.1.9.tar.gz + sudo tar xzvf apache-tomcat-10*tar.gz -C /opt/tomcat --strip-components=1 + sudo chown -R tomcat:tomcat /opt/tomcat/ + sudo chmod -R u+x /opt/tomcat/bin + ``` + +1. Create a systemd service. Edit the `/etc/systemd/system/tomcat.service` file and add: + + ```shell + [Unit] + Description=Tomcat + After=network.target + + [Service] + Type=forking + + User=tomcat + Group=tomcat + + Environment="JAVA_HOME=/usr/lib/jvm/java-1.11.0-openjdk-amd64" + Environment="JAVA_OPTS=-Djava.security.egd=file:///dev/urandom" + Environment="CATALINA_BASE=/opt/tomcat" + Environment="CATALINA_HOME=/opt/tomcat" + Environment="CATALINA_PID=/opt/tomcat/temp/tomcat.pid" + Environment="CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC" + + ExecStart=/opt/tomcat/bin/startup.sh + ExecStop=/opt/tomcat/bin/shutdown.sh + + RestartSec=10 + Restart=always + + [Install] + WantedBy=multi-user.target + ``` + + `JAVA_HOME` should be the same path as seen in `sudo update-java-alternatives -l`. + +1. To configure ports, edit your `/opt/tomcat/conf/server.xml` and choose your + ports. Avoid using port `8080`, as [Puma](../operations/puma.md) listens on port `8080` for metrics. + + ```shell + <Server port="8006" shutdown="SHUTDOWN"> + ... + <Connector port="8005" protocol="HTTP/1.1" + ... + ``` + +1. Reload and start Tomcat: + + ```shell + sudo systemctl daemon-reload + sudo systemctl start tomcat + sudo systemctl status tomcat + sudo systemctl enable tomcat + ``` + + The Java process should be listening on these ports: + + ```shell + root@gitlab-omnibus:/plantuml-server# netstat -plnt | grep java + tcp6 0 0 127.0.0.1:8006 :::* LISTEN 14935/java + tcp6 0 0 :::8005 :::* LISTEN 14935/java + ``` + +1. Modify your NGINX configuration. The `proxy_pass` port matches the Connector port in the `server.xml`: + + ```shell + nginx['custom_gitlab_server_config'] = "location /-/plantuml { + rewrite ^/-/(plantuml.*) /$1 break; + proxy_set_header HOST $host; + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_cache off; + proxy_pass http://localhost:8005/plantuml; + }" + ``` + +1. Reconfigure GitLab to read the new changes: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. Install PlantUML and copy the `.war` file: + + ```shell + cd / & git clone https://github.com/plantuml/plantuml-server.git + cd plantuml-server + mvn package + cp /plantuml-server/target/plantuml.war /opt/tomcat/webapps/plantuml.war + chown tomcat:tomcat /opt/tomcat/webapps/plantuml.war + systemctl restart tomcat + ``` + +The Tomcat service should restart. After the restart is complete, the +PlantUML integration is ready and listening for requests on port `8005`: +`http://localhost:8005/plantuml` + +To change the Tomcat defaults, edit the `/opt/tomcat/conf/server.xml` file. + +NOTE: +The default URL is different when using this approach. The Docker-based image +makes the service available at the root URL, with no relative path. Adjust +the configuration below accordingly. + ### Configure PlantUML security PlantUML has features that allow fetching network resources. If you self-host the @@ -210,7 +308,8 @@ stop; After configuring your local PlantUML server, you're ready to enable the PlantUML integration: 1. Sign in to GitLab as an [Administrator](../../user/permissions.md) user. -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, go to **Settings > General** and expand the **PlantUML** section. 1. Select the **Enable PlantUML** checkbox. 1. Set the PlantUML instance as `https://gitlab.example.com/-/plantuml/`, @@ -221,7 +320,7 @@ these steps: - For PlantUML servers running v1.2020.9 and above, such as [plantuml.com](https://plantuml.com), you must set the `PLANTUML_ENCODING` environment variable to enable the `deflate` - compression. In Omnibus GitLab, you can set this value in `/etc/gitlab.rb` with + compression. In Linux package installations, you can set this value in `/etc/gitlab.rb` with this command: ```ruby diff --git a/doc/administration/integration/terminal.md b/doc/administration/integration/terminal.md index df9bb6b6e17..e5920520be7 100644 --- a/doc/administration/integration/terminal.md +++ b/doc/administration/integration/terminal.md @@ -87,16 +87,14 @@ it's safe to enable support for these headers globally. If you prefer a narrower set of rules, you can restrict it to URLs ending with `/terminal.ws`. This approach may still result in a few false positives. -If you installed from source, or have made any configuration changes to your -Omnibus installation before upgrading to 8.15, you may need to make some changes -to your configuration. Read +If you self-compiled your installation, you may need to make some changes to your configuration. Read [Upgrading Community Edition and Enterprise Edition from source](../../update/upgrading_from_source.md#nginx-configuration) for more details. To disable web terminal support in GitLab, stop passing the `Connection` and `Upgrade` hop-by-hop headers in the *first* HTTP reverse proxy in the chain. For most users, this is the NGINX server bundled with -Omnibus GitLab, in which case, you need to: +Linux package installations. In this case, you need to: - Find the `nginx['proxy_set_headers']` section of your `gitlab.rb` file - Ensure the whole block is uncommented, and then comment out or remove the @@ -114,7 +112,7 @@ they receive a `Connection failed` message. By default, terminal sessions do not expire. To limit the terminal session lifetime in your GitLab instance: -1. On the top bar, select **Main menu > Admin**. -1. Select - [**Settings > Web terminal**](../../user/admin_area/settings/index.md#general). +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select [**Settings > Web terminal**](../../user/admin_area/settings/index.md#general). 1. Set a `max session time`. diff --git a/doc/administration/lfs/index.md b/doc/administration/lfs/index.md index ee5cd04f3d6..e4475e877d8 100644 --- a/doc/administration/lfs/index.md +++ b/doc/administration/lfs/index.md @@ -401,7 +401,7 @@ To delete these references: If you configure GitLab to [disable TLS v1.2](https://docs.gitlab.com/omnibus/settings/nginx.html) and only enable TLS v1.3 connections, LFS operations require a -[Git LFS client](https://git-lfs.github.com) version 2.11.0 or later. If you use +[Git LFS client](https://git-lfs.com/) version 2.11.0 or later. If you use a Git LFS client earlier than version 2.11.0, GitLab displays an error: ```plaintext diff --git a/doc/administration/libravatar.md b/doc/administration/libravatar.md index 802a3be46fa..1b7406546ef 100644 --- a/doc/administration/libravatar.md +++ b/doc/administration/libravatar.md @@ -25,7 +25,7 @@ You can use only the MD5 hash in the URL for the Libravatar service. See [issue In the [`gitlab.yml` gravatar section](https://gitlab.com/gitlab-org/gitlab/-/blob/68dac188ec6b1b03d53365e7579422f44cbe7a1c/config/gitlab.yml.example#L469-476), set the configuration options as follows: -**For Omnibus installations** +For Linux package installations: 1. Edit `/etc/gitlab/gitlab.rb`: @@ -39,7 +39,7 @@ the configuration options as follows: 1. To apply the changes, run `sudo gitlab-ctl reconfigure`. -**For installations from source** +For self-compiled installations: 1. Edit `config/gitlab.yml`: @@ -57,12 +57,12 @@ the configuration options as follows: ## Set the Libravatar service to default (Gravatar) -**For Omnibus installations** +For Linux package installations: 1. Delete `gitlab_rails['gravatar_ssl_url']` or `gitlab_rails['gravatar_plain_url']` from `/etc/gitlab/gitlab.rb`. 1. To apply the changes, run `sudo gitlab-ctl reconfigure`. -**For installations from source** +For self-compiled installations: 1. Remove `gravatar:` section from `config/gitlab.yml`. 1. Save the file, then [restart](restart_gitlab.md#installations-from-source) @@ -72,7 +72,7 @@ the configuration options as follows: To disable Gravatar, for example, to prohibit third-party services, complete the following steps: -**For Omnibus installations** +For Linux package installations: 1. Edit `/etc/gitlab/gitlab.rb`: @@ -82,7 +82,7 @@ To disable Gravatar, for example, to prohibit third-party services, complete the 1. To apply the changes, run `sudo gitlab-ctl reconfigure`. -**For installations from source** +For self-compiled installations: 1. Edit `config/gitlab.yml`: diff --git a/doc/administration/logs/index.md b/doc/administration/logs/index.md index 7fab97f76da..8dcb25e22df 100644 --- a/doc/administration/logs/index.md +++ b/doc/administration/logs/index.md @@ -14,7 +14,7 @@ This guide talks about how to read and use these system log files. Read more about the log system and using the logs: -- [Customize logging on Omnibus GitLab installations](https://docs.gitlab.com/omnibus/settings/logs.html) +- [Customize logging on Linux package installations](https://docs.gitlab.com/omnibus/settings/logs.html) including adjusting log retention, log forwarding, switching logs from JSON to plain text logging, and more. - [How to parse and analyze JSON logs](../logs/log_parsing.md). @@ -88,7 +88,7 @@ except those captured by `runit`. | [Alertmanager logs](#alertmanager-logs) | **{dotted-circle}** No | **{check-circle}** Yes | | [crond logs](#crond-logs) | **{dotted-circle}** No | **{check-circle}** Yes | | [Gitaly](#gitaly-logs) | **{check-circle}** Yes | **{check-circle}** Yes | -| [GitLab Exporter for Omnibus](#gitlab-exporter) | **{dotted-circle}** No | **{check-circle}** Yes | +| [GitLab Exporter for Linux package installations](#gitlab-exporter) | **{dotted-circle}** No | **{check-circle}** Yes | | [GitLab Pages logs](#pages-logs) | **{check-circle}** Yes | **{check-circle}** Yes | | GitLab Rails | **{check-circle}** Yes | **{dotted-circle}** No | | [GitLab Shell logs](#gitlab-shelllog) | **{check-circle}** Yes | **{dotted-circle}** No | @@ -107,10 +107,10 @@ except those captured by `runit`. ## `production_json.log` -Depending on your installation method, this file is located at: +This file is located at: -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/production_json.log` -- Installations from source: `/home/git/gitlab/log/production_json.log` +- `/var/log/gitlab/gitlab-rails/production_json.log` on Linux package installations. +- `/home/git/gitlab/log/production_json.log` on self-compiled installations. It contains a structured log for Rails controller requests received from GitLab, thanks to [Lograge](https://github.com/roidrage/lograge/). @@ -264,10 +264,10 @@ Starting with GitLab 12.5, if an error occurs, an ## `production.log` -Depending on your installation method, this file is located at: +This file is located at: -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/production.log` -- Installations from source: `/home/git/gitlab/log/production.log` +- `/var/log/gitlab/gitlab-rails/production.log` on Linux package installations. +- `/home/git/gitlab/log/production.log` on self-compiled installations. It contains information about all performed requests. You can see the URL and type of request, IP address, and what parts of code were @@ -300,10 +300,10 @@ The request was processed by `Projects::TreeController`. ## `api_json.log` -Depending on your installation method, this file is located at: +This file is located at: -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/api_json.log` -- Installations from source: `/home/git/gitlab/log/api_json.log` +- `/var/log/gitlab/gitlab-rails/api_json.log` on Linux package installations. +- `/home/git/gitlab/log/api_json.log` on self-compiled installations. It helps you see requests made directly to the API. For example: @@ -354,10 +354,10 @@ process on Redis or external HTTP, not only the serialization process. > [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/111046) in GitLab 15.10. -Depending on your installation method, this file is located at: +This file is located at: -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/application.log` -- Installations from source: `/home/git/gitlab/log/application.log` +- `/var/log/gitlab/gitlab-rails/application.log` on Linux package installations. +- `/home/git/gitlab/log/application.log` on self-compiled installations. It contains a less structured version of the logs in [`application_json.log`](#application_jsonlog), like this example: @@ -374,10 +374,10 @@ October 07, 2014 11:25: Project "project133" was removed > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22812) in GitLab 12.7. -Depending on your installation method, this file is located at: +This file is located at: -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/application_json.log` -- Installations from source: `/home/git/gitlab/log/application_json.log` +- `/var/log/gitlab/gitlab-rails/application_json.log` on Linux package installations. +- `/home/git/gitlab/log/application_json.log` on self-compiled installations. It helps you discover events happening in your instance such as user creation and project deletion. For example: @@ -399,10 +399,10 @@ and project deletion. For example: ## `integrations_json.log` -Depending on your installation method, this file is located at: +This file is located at: -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/integrations_json.log` -- Installations from source: `/home/git/gitlab/log/integrations_json.log` +- `/var/log/gitlab/gitlab-rails/integrations_json.log` on Linux package installations. +- `/home/git/gitlab/log/integrations_json.log` on self-compiled installations. It contains information about [integration](../../user/project/integrations/index.md) activities, such as Jira, Asana, and irker services. It uses JSON format, @@ -434,19 +434,19 @@ like this example: > [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. -Depending on your installation method, this file is located at: +This file is located at: -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/kubernetes.log` -- Installations from source: `/home/git/gitlab/log/kubernetes.log` +- `/var/log/gitlab/gitlab-rails/kubernetes.log` on Linux package installations. +- `/home/git/gitlab/log/kubernetes.log` on self-compiled installations. It logs information related to [certificate-based clusters](../../user/project/clusters/index.md), such as connectivity errors. Each line contains JSON that can be ingested by services like Elasticsearch and Splunk. ## `git_json.log` -Depending on your installation method, this file is located at: +This file is located at: -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/git_json.log` -- Installations from source: `/home/git/gitlab/log/git_json.log` +- `/var/log/gitlab/gitlab-rails/git_json.log` on Linux package installations. +- `/home/git/gitlab/log/git_json.log` on self-compiled installations. After GitLab version 12.2, this file was renamed from `githost.log` to `git_json.log` and stored in JSON format. @@ -472,10 +472,10 @@ NOTE: GitLab Free tracks a small number of different audit events. GitLab Premium tracks many more. -Depending on your installation method, this file is located at: +This file is located at: -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/audit_json.log` -- Installations from source: `/home/git/gitlab/log/audit_json.log` +- `/var/log/gitlab/gitlab-rails/audit_json.log` on Linux package installations. +- `/home/git/gitlab/log/audit_json.log` on self-compiled installations. Changes to group or project settings and memberships (`target_details`) are logged to this file. For example: @@ -499,18 +499,15 @@ are logged to this file. For example: ## Sidekiq logs -NOTE: -In Omnibus GitLab `12.10` or earlier, the Sidekiq log is at `/var/log/gitlab/gitlab-rails/sidekiq.log`. - -For Omnibus GitLab installations, some Sidekiq logs are in `/var/log/gitlab/sidekiq/current` +For Linux package installations, some Sidekiq logs are in `/var/log/gitlab/sidekiq/current` and as follows. ### `sidekiq.log` -Depending on your installation method, this file is located at: +This file is located at: -- Omnibus GitLab: `/var/log/gitlab/sidekiq/current` -- Installations from source: `/home/git/gitlab/log/sidekiq.log` +- `/var/log/gitlab/sidekiq/current` on Linux package installations. +- `/home/git/gitlab/log/sidekiq.log` on self-compiled installations. GitLab uses background jobs for processing tasks which can take a long time. All information about processing these jobs are written down to @@ -549,7 +546,7 @@ Sidekiq. For example: } ``` -For Omnibus GitLab installations, add the configuration option: +For Linux package installations, add the configuration option: ```ruby sidekiq['log_format'] = 'json' @@ -568,10 +565,10 @@ For installations from source, edit the `gitlab.yml` and set the Sidekiq > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26586) in GitLab 12.9. -Depending on your installation method, this file is located at: +This file is located at: -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/sidekiq_client.log` -- Installations from source: `/home/git/gitlab/log/sidekiq_client.log` +- `/var/log/gitlab/gitlab-rails/sidekiq_client.log` on Linux package installations. +- `/home/git/gitlab/log/sidekiq_client.log` on self-compiled installations. This file contains logging information about jobs before Sidekiq starts processing them, such as before being enqueued. @@ -585,8 +582,6 @@ you've configured this for Sidekiq as mentioned above. GitLab Shell is used by GitLab for executing Git commands and provide SSH access to Git repositories. -### For GitLab versions 12.10 and up - Information containing `git-{upload-pack,receive-pack}` requests is at `/var/log/gitlab/gitlab-shell/gitlab-shell.log`. Information about hooks to GitLab Shell from Gitaly is at `/var/log/gitlab/gitaly/current`. @@ -640,55 +635,18 @@ Example log entries for `/var/log/gitlab/gitaly/current`: } ``` -### For GitLab versions 12.5 through 12.9 - -For GitLab 12.5 to 12.9, depending on your installation method, this -file is located at: - -- Omnibus GitLab: `/var/log/gitlab/gitaly/gitlab-shell.log` -- Installation from source: `/home/git/gitaly/gitlab-shell.log` - -Example log entries: - -```json -{ - "method": "POST", - "url": "http://127.0.0.1:8080/api/v4/internal/post_receive", - "duration": 0.031809164, - "gitaly_embedded": true, - "pid": 27056, - "level": "info", - "msg": "finished HTTP request", - "time": "2020-04-17T16:24:38+00:00" -} -``` - -### For GitLab 12.5 and earlier - -For GitLab 12.5 and earlier, the file is at `/var/log/gitlab/gitlab-shell/gitlab-shell.log`. - -Example log entries: - -```plaintext -I, [2015-02-13T06:17:00.671315 #9291] INFO -- : Adding project root/example.git at </var/opt/gitlab/git-data/repositories/root/dcdcdcdcd.git>. -I, [2015-02-13T06:17:00.679433 #9291] INFO -- : Moving existing hooks directory and symlinking global hooks directory for /var/opt/gitlab/git-data/repositories/root/example.git. -``` - -User clone/fetch activity using SSH transport appears in this log as -`executing git command <gitaly-upload-pack...`. - ## Gitaly logs This file is in `/var/log/gitlab/gitaly/current` and is produced by [runit](http://smarden.org/runit/). -`runit` is packaged with Omnibus GitLab and a brief explanation of its purpose -is available [in the Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/architecture/#runit). +`runit` is packaged with the Linux package and a brief explanation of its purpose +is available [in the Linux package documentation](https://docs.gitlab.com/omnibus/architecture/#runit). [Log files are rotated](http://smarden.org/runit/svlogd.8.html), renamed in Unix timestamp format, and `gzip`-compressed (like `@1584057562.s`). ### `grpc.log` -This file is at `/var/log/gitlab/gitlab-rails/grpc.log` for Omnibus GitLab -packages. Native [gRPC](https://grpc.io/) logging used by Gitaly. +This file is at `/var/log/gitlab/gitlab-rails/grpc.log` for Linux +package installations. Native [gRPC](https://grpc.io/) logging used by Gitaly. ### `gitaly_hooks.log` @@ -700,34 +658,34 @@ failures received during processing of the responses from GitLab API. ### `puma_stdout.log` -Depending on your installation method, this file is located at: +This file is located at: -- Omnibus GitLab: `/var/log/gitlab/puma/puma_stdout.log` -- Installations from source: `/home/git/gitlab/log/puma_stdout.log` +- `/var/log/gitlab/puma/puma_stdout.log` on Linux package installations. +- `/home/git/gitlab/log/puma_stdout.log` on self-compiled installations. ### `puma_stderr.log` -Depending on your installation method, this file is located at: +This file is located at: -- Omnibus GitLab: `/var/log/gitlab/puma/puma_stderr.log` -- Installations from source: `/home/git/gitlab/log/puma_stderr.log` +- `/var/log/gitlab/puma/puma_stderr.log` on Linux package installations. +- `/home/git/gitlab/log/puma_stderr.log` on self-compiled installations. ## `repocheck.log` -Depending on your installation method, this file is located at: +This file is located at: -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/repocheck.log` -- Installations from source: `/home/git/gitlab/log/repocheck.log` +- `/var/log/gitlab/gitlab-rails/repocheck.log` on Linux package installations. +- `/home/git/gitlab/log/repocheck.log` on self-compiled installations. It logs information whenever a [repository check is run](../repository_checks.md) on a project. ## `importer.log` -Depending on your installation method, this file is located at: +This file is located at: -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/importer.log` -- Installations from source: `/home/git/gitlab/log/importer.log` +- `/var/log/gitlab/gitlab-rails/importer.log` on Linux package installations. +- `/home/git/gitlab/log/importer.log` on self-compiled installations. This file logs the progress of [project imports and migrations](../../user/project/import/index.md). @@ -735,10 +693,10 @@ This file logs the progress of [project imports and migrations](../../user/proje > Introduced in GitLab 13.1. -Depending on your installation method, this file is located at: +This file is located at: -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/exporter.log` -- Installations from source: `/home/git/gitlab/log/exporter.log` +- `/var/log/gitlab/gitlab-rails/exporter.log` on Linux package installations. +- `/home/git/gitlab/log/exporter.log` on self-compiled installations. It logs the progress of the export process. @@ -746,10 +704,10 @@ It logs the progress of the export process. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/59587) in GitLab 13.7. -Depending on your installation method, this file is located at: +This file is located at: -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/features_json.log` -- Installations from source: `/home/git/gitlab/log/features_json.log` +- `/var/log/gitlab/gitlab-rails/features_json.log` on Linux package installations. +- `/home/git/gitlab/log/features_json.log` on self-compiled installations. The modification events from [Feature flags in development of GitLab](../../development/feature_flags/index.md) are recorded in this file. For example: @@ -771,10 +729,10 @@ are recorded in this file. For example: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/384180) in GitLab 15.9. -Depending on your installation method, this file is located at: +This file is located at: -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/ci_resource_group_json.log` -- Installations from source: `/home/git/gitlab/log/ci_resource_group_json.log` +- `/var/log/gitlab/gitlab-rails/ci_resource_group_json.log` on Linux package installations. +- `/home/git/gitlab/log/ci_resource_group_json.log` on self-compiled installations. It contains information about [resource group](../../ci/resource_groups/index.md) acquisition. For example: @@ -789,10 +747,10 @@ The examples show the `resource_group_id`, `processable_id`, `message`, and `suc > Introduced in GitLab 12.0. -Depending on your installation method, this file is located at: +This file is located at: -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/auth.log` -- Installations from source: `/home/git/gitlab/log/auth.log` +- `/var/log/gitlab/gitlab-rails/auth.log` on Linux package installations. +- `/home/git/gitlab/log/auth.log` on self-compiled installations. This log records: @@ -803,10 +761,10 @@ This log records: ## `auth_json.log` -Depending on your installation method, this file is located at: +This file is located at: -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/auth_json.log` -- Installations from source: `/home/git/gitlab/log/auth_json.log` +- `/var/log/gitlab/gitlab-rails/auth_json.log` on Linux package installations. +- `/home/git/gitlab/log/auth_json.log` on self-compiled installations. This file contains the JSON version of the logs in `auth.log`, for example: @@ -827,10 +785,10 @@ This file contains the JSON version of the logs in `auth.log`, for example: > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/59587) in GitLab 12.0. -Depending on your installation method, this file is located at: +This file is located at: -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/graphql_json.log` -- Installations from source: `/home/git/gitlab/log/graphql_json.log` +- `/var/log/gitlab/gitlab-rails/graphql_json.log` on Linux package installations. +- `/home/git/gitlab/log/graphql_json.log` on self-compiled installations. GraphQL queries are recorded in the file. For example: @@ -842,10 +800,10 @@ GraphQL queries are recorded in the file. For example: > Introduced in GitLab 12.3. -Depending on your installation method, this file is located at: +This file is located at: -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/migrations.log` -- Installations from source: `/home/git/gitlab/log/migrations.log` +- `/var/log/gitlab/gitlab-rails/migrations.log` on Linux package installations. +- `/home/git/gitlab/log/migrations.log` on self-compiled installations. This file logs the progress of [database migrations](../raketasks/maintenance.md#display-status-of-database-migrations). @@ -853,19 +811,19 @@ This file logs the progress of [database migrations](../raketasks/maintenance.md > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/19186) in GitLab 12.6. -Depending on your installation method, this file is located at: +This file is located at: -- Omnibus GitLab: `/var/log/gitlab/mailroom/current` -- Installations from source: `/home/git/gitlab/log/mail_room_json.log` +- `/var/log/gitlab/mailroom/current` on Linux package installations. +- `/home/git/gitlab/log/mail_room_json.log` on self-compiled installations. This structured log file records internal activity in the `mail_room` gem. Its name and path are configurable, so the name and path may not match the above. ## Reconfigure logs -Reconfigure log files are in `/var/log/gitlab/reconfigure` for Omnibus GitLab -packages. Installations from source don't have reconfigure logs. A reconfigure log -is populated whenever `gitlab-ctl reconfigure` is run manually or as part of an upgrade. +Reconfigure log files are in `/var/log/gitlab/reconfigure` for Linux package installations. Self-compiled installations +don't have reconfigure logs. A reconfigure log is populated whenever `gitlab-ctl reconfigure` is run manually or as part +of an upgrade. Reconfigure logs files are named according to the UNIX timestamp of when the reconfigure was initiated, such as `1509705644.log` @@ -875,34 +833,32 @@ was initiated, such as `1509705644.log` If Prometheus metrics and the Sidekiq Exporter are both enabled, Sidekiq starts a Web server and listen to the defined port (default: `8082`). By default, Sidekiq Exporter access logs are disabled but can -be enabled based on your installation method: +be enabled: -- Omnibus GitLab: Use the `sidekiq['exporter_log_enabled'] = true` - option in `/etc/gitlab/gitlab.rb` -- Installations from source: Use the `sidekiq_exporter.log_enabled` option - in `gitlab.yml` +- Use the `sidekiq['exporter_log_enabled'] = true` option in `/etc/gitlab/gitlab.rb` on Linux package installations. +- Use the `sidekiq_exporter.log_enabled` option in `gitlab.yml` on self-compiled installations. When enabled, depending on your installation method, this file is located at: -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/sidekiq_exporter.log` -- Installations from source: `/home/git/gitlab/log/sidekiq_exporter.log` +- `/var/log/gitlab/gitlab-rails/sidekiq_exporter.log` on Linux package installations. +- `/home/git/gitlab/log/sidekiq_exporter.log` on self-compiled installations. If Prometheus metrics and the Web Exporter are both enabled, Puma starts a Web server and listen to the defined port (default: `8083`), and access logs are generated in a location based on your installation method: -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/web_exporter.log` -- Installations from source: `/home/git/gitlab/log/web_exporter.log` +- `/var/log/gitlab/gitlab-rails/web_exporter.log` on Linux package installations. +- `/home/git/gitlab/log/web_exporter.log` on self-compiled installations. ## `database_load_balancing.log` **(PREMIUM SELF)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/15442) in GitLab 12.3. Contains details of GitLab [Database Load Balancing](../postgresql/database_load_balancing.md). -Depending on your installation method, this file is located at: +This file is located at: -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/database_load_balancing.log` -- Installations from source: `/home/git/gitlab/log/database_load_balancing.log` +- `/var/log/gitlab/gitlab-rails/database_load_balancing.log` on Linux package installations. +- `/home/git/gitlab/log/database_load_balancing.log` on self-compiled installations. ## `zoekt.log` **(PREMIUM SELF)** @@ -912,21 +868,20 @@ This file logs information related to the [Exact code search](../../user/search/exact_code_search.md) feature which is powered by Zoekt. -Depending on your installation method, this file is located at: +This file is located at: -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/zoekt.log` -- Installations from source: `/home/git/gitlab/log/zoekt.log` +- `/var/log/gitlab/gitlab-rails/zoekt.log` on Linux package installations. +- `/home/git/gitlab/log/zoekt.log` on self-compiled installations. ## `elasticsearch.log` **(PREMIUM SELF)** > Introduced in GitLab 12.6. This file logs information related to the Elasticsearch Integration, including -errors during indexing or searching Elasticsearch. Depending on your installation -method, this file is located at: +errors during indexing or searching Elasticsearch. This file is located at: -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/elasticsearch.log` -- Installations from source: `/home/git/gitlab/log/elasticsearch.log` +- `/var/log/gitlab/gitlab-rails/elasticsearch.log` on Linux package installations. +- `/home/git/gitlab/log/elasticsearch.log` on self-compiled installations. Each line contains JSON that can be ingested by services like Elasticsearch and Splunk. Line breaks have been added to the following example line for clarity: @@ -952,10 +907,10 @@ Line breaks have been added to the following example line for clarity: This file logs the information about exceptions being tracked by `Gitlab::ErrorTracking`, which provides a standard and consistent way of [processing rescued exceptions](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/logging.md#exception-handling). -Depending on your installation method, this file is located at: +This file is located at: -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/exceptions_json.log` -- Installations from source: `/home/git/gitlab/log/exceptions_json.log` +- `/var/log/gitlab/gitlab-rails/exceptions_json.log` on Linux package installations. +- `/home/git/gitlab/log/exceptions_json.log` on self-compiled installations. Each line contains JSON that can be ingested by Elasticsearch. For example: @@ -980,10 +935,10 @@ Each line contains JSON that can be ingested by Elasticsearch. For example: > Introduced in GitLab 13.0. -Depending on your installation method, this file is located at: +This file is located at: -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/service_measurement.log` -- Installations from source: `/home/git/gitlab/log/service_measurement.log` +- `/var/log/gitlab/gitlab-rails/service_measurement.log` on Linux package installations. +- `/home/git/gitlab/log/service_measurement.log` on self-compiled installations. It contains only a single structured log with measurements for each service execution. It contains measurements such as the number of SQL calls, `execution_time`, `gc_stats`, and `memory usage`. @@ -1013,10 +968,10 @@ This message shows that Geo detected that a repository update was needed for pro ## `update_mirror_service_json.log` -Depending on your installation method, this file is located at: +This file is located at: -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/update_mirror_service_json.log` -- Installations from source: `/home/git/gitlab/log/update_mirror_service_json.log` +- `/var/log/gitlab/gitlab-rails/update_mirror_service_json.log` on Linux package installations. +- `/home/git/gitlab/log/update_mirror_service_json.log` on self-compiled installations. This file contains information about LFS errors that occurred during project mirroring. While we work to move other project mirroring errors into this log, the [general log](#productionlog) @@ -1034,13 +989,25 @@ can be used. } ``` +## `llm.log` **(ULTIMATE SAAS)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/120506) in GitLab 16.0. + +The `llm.log` file logs information related to +[AI features](../../user/ai_features.md). + +This file is located at: + +- `/var/log/gitlab/gitlab-rails/llm.log` on Linux package installations. +- `/home/git/gitlab/log/llm.log` on self-compiled installations. + ## Registry logs -For Omnibus GitLab installations, Container Registry logs are in `/var/log/gitlab/registry/current`. +For Linux package installations, Container Registry logs are in `/var/log/gitlab/registry/current`. ## NGINX logs -For Omnibus GitLab installations, NGINX logs are in: +For Linux package installations, NGINX logs are in: - `/var/log/gitlab/nginx/gitlab_access.log`: A log of requests made to GitLab - `/var/log/gitlab/nginx/gitlab_error.log`: A log of NGINX errors for GitLab @@ -1063,7 +1030,7 @@ for sensitive query string parameters such as secret tokens. ## Pages logs -For Omnibus GitLab installations, Pages logs are in `/var/log/gitlab/gitlab-pages/current`. +For Linux package installations, Pages logs are in `/var/log/gitlab/gitlab-pages/current`. For example: @@ -1092,67 +1059,67 @@ For example: ## Mattermost logs -For Omnibus GitLab installations, Mattermost logs are in these locations: +For Linux package installations, Mattermost logs are in these locations: - `/var/log/gitlab/mattermost/mattermost.log` - `/var/log/gitlab/mattermost/current` ## Workhorse logs -For Omnibus GitLab installations, Workhorse logs are in `/var/log/gitlab/gitlab-workhorse/current`. +For Linux package installations, Workhorse logs are in `/var/log/gitlab/gitlab-workhorse/current`. ## PgBouncer logs -For Omnibus GitLab installations, PgBouncer logs are in `/var/log/gitlab/pgbouncer/current`. +For Linux package installations, PgBouncer logs are in `/var/log/gitlab/pgbouncer/current`. ## PostgreSQL logs -For Omnibus GitLab installations, PostgreSQL logs are in `/var/log/gitlab/postgresql/current`. +For Linux package installations, PostgreSQL logs are in `/var/log/gitlab/postgresql/current`. ## Prometheus logs -For Omnibus GitLab installations, Prometheus logs are in `/var/log/gitlab/prometheus/current`. +For Linux package installations, Prometheus logs are in `/var/log/gitlab/prometheus/current`. ## Redis logs -For Omnibus GitLab installations, Redis logs are in `/var/log/gitlab/redis/current`. +For Linux package installations, Redis logs are in `/var/log/gitlab/redis/current`. ## Alertmanager logs -For Omnibus GitLab installations, Alertmanager logs are in `/var/log/gitlab/alertmanager/current`. +For Linux package installations, Alertmanager logs are in `/var/log/gitlab/alertmanager/current`. <!-- vale gitlab.Spelling = NO --> ## crond logs -For Omnibus GitLab installations, crond logs are in `/var/log/gitlab/crond/`. +For Linux package installations, crond logs are in `/var/log/gitlab/crond/`. <!-- vale gitlab.Spelling = YES --> ## Grafana logs -For Omnibus GitLab installations, Grafana logs are in `/var/log/gitlab/grafana/current`. +For Linux package installations, Grafana logs are in `/var/log/gitlab/grafana/current`. ## LogRotate logs -For Omnibus GitLab installations, `logrotate` logs are in `/var/log/gitlab/logrotate/current`. +For Linux package installations, `logrotate` logs are in `/var/log/gitlab/logrotate/current`. ## GitLab Monitor logs -For Omnibus GitLab installations, GitLab Monitor logs are in `/var/log/gitlab/gitlab-monitor/`. +For Linux package installations, GitLab Monitor logs are in `/var/log/gitlab/gitlab-monitor/`. ## GitLab Exporter -For Omnibus GitLab installations, GitLab Exporter logs are in `/var/log/gitlab/gitlab-exporter/current`. +For Linux package installations, GitLab Exporter logs are in `/var/log/gitlab/gitlab-exporter/current`. ## GitLab agent server -For Omnibus GitLab installations, GitLab agent server logs are +For Linux package installations, GitLab agent server logs are in `/var/log/gitlab/gitlab-kas/current`. ## Praefect logs -For Omnibus GitLab installations, Praefect logs are in `/var/log/gitlab/praefect/`. +For Linux package installations, Praefect logs are in `/var/log/gitlab/praefect/`. GitLab also tracks [Prometheus metrics for Praefect](../gitaly/monitoring.md#monitor-gitaly-cluster). @@ -1168,10 +1135,10 @@ This log is populated when a [GitLab backup is created](../../raketasks/backup_r > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/48149) in GitLab 13.7. -Depending on your installation method, this file is located at: +This file is located at: -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/performance_bar_json.log` -- Installations from source: `/home/git/gitlab/log/performance_bar_json.log` +- `/var/log/gitlab/gitlab-rails/performance_bar_json.log` on Linux package installations. +- `/home/git/gitlab/log/performance_bar_json.log` on self-compiled installations. Performance bar statistics (currently only duration of SQL queries) are recorded in that file. For example: diff --git a/doc/administration/logs/log_parsing.md b/doc/administration/logs/log_parsing.md index d9520ea9bc0..21ce3d7f17f 100644 --- a/doc/administration/logs/log_parsing.md +++ b/doc/administration/logs/log_parsing.md @@ -24,7 +24,7 @@ include use cases targeted for parsing GitLab log files. ## Parsing Logs The examples listed below address their respective log files by -their relative Omnibus paths and default filenames. +their relative Linux package installation paths and default filenames. Find the respective full paths in the [GitLab logs sections](../logs/index.md#production_jsonlog). ### General Commands diff --git a/doc/administration/maintenance_mode/index.md b/doc/administration/maintenance_mode/index.md index 8347015a8a3..3bbebe7ecce 100644 --- a/doc/administration/maintenance_mode/index.md +++ b/doc/administration/maintenance_mode/index.md @@ -21,7 +21,8 @@ Maintenance Mode allows most external actions that do not change internal state. Enable Maintenance Mode as an administrator in one of these ways: - **Web UI**: - 1. On the top bar, select **Main menu > Admin**. + 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > General**. 1. Expand **Maintenance Mode**, and toggle **Enable Maintenance Mode**. You can optionally add a message for the banner as well. @@ -45,7 +46,8 @@ Enable Maintenance Mode as an administrator in one of these ways: Disable Maintenance Mode in one of three ways: - **Web UI**: - 1. On the top bar, select **Main menu > Admin**. + 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > General**. 1. Expand **Maintenance Mode**, and toggle **Enable Maintenance Mode**. You can optionally add a message for the banner as well. @@ -173,7 +175,8 @@ you should disable all cron jobs except for those related to Geo. To monitor queues and disable jobs: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Background Jobs**. ### Incident management diff --git a/doc/administration/merge_request_diffs.md b/doc/administration/merge_request_diffs.md index 3250a2339e2..00ea1e43600 100644 --- a/doc/administration/merge_request_diffs.md +++ b/doc/administration/merge_request_diffs.md @@ -21,7 +21,7 @@ that only [stores outdated diffs](#alternative-in-database-storage) outside of d ## Using external storage -**In Omnibus installations:** +For Linux package installations: 1. Edit `/etc/gitlab/gitlab.rb` and add the following line: @@ -38,10 +38,10 @@ that only [stores outdated diffs](#alternative-in-database-storage) outside of d gitlab_rails['external_diffs_storage_path'] = "/mnt/storage/external-diffs" ``` -1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and [reconfigure GitLab](restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. GitLab then migrates your existing merge request diffs to external storage. -**In installations from source:** +For self-compiled installations: 1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: @@ -74,7 +74,7 @@ Instead of storing the external diffs on disk, we recommended the use of an obje store like AWS S3 instead. This configuration relies on valid AWS credentials to be configured already. -**In Omnibus installations:** +For Linux package installations: 1. Edit `/etc/gitlab/gitlab.rb` and add the following line: @@ -83,10 +83,10 @@ be configured already. ``` 1. Set [object storage settings](#object-storage-settings). -1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and [reconfigure GitLab](restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. GitLab then migrates your existing merge request diffs to external storage. -**In installations from source:** +For self-compiled installations: 1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: @@ -109,7 +109,7 @@ In GitLab 13.2 and later, you should use the This section describes the earlier configuration format. For source installations, these settings are nested under `external_diffs:` and -then `object_store:`. On Omnibus installations, they are prefixed by +then `object_store:`. On Linux package installations, they are prefixed by `external_diffs_object_store_`. | Setting | Description | Default | @@ -123,7 +123,7 @@ then `object_store:`. On Omnibus installations, they are prefixed by See [the available connection settings for different providers](object_storage.md#configure-the-connection-settings). -**In Omnibus installations:** +For Linux package installations: 1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with the values you want: @@ -151,9 +151,9 @@ See [the available connection settings for different providers](object_storage.m } ``` -1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and [reconfigure GitLab](restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. -**In installations from source:** +For self-compiled installations: 1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: @@ -182,7 +182,7 @@ in the database. To enable this feature, perform the following steps: -**In Omnibus installations:** +For Linux package installations: 1. Edit `/etc/gitlab/gitlab.rb` and add the following line: @@ -190,9 +190,9 @@ To enable this feature, perform the following steps: gitlab_rails['external_diffs_when'] = 'outdated' ``` -1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and [reconfigure GitLab](restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. -**In installations from source:** +For self-compiled installations: 1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: @@ -236,13 +236,13 @@ Then you are affected by this issue. Because it's not possible to safely determi all these conditions automatically, we've provided a Rake task in GitLab v13.2.0 that you can run manually to correct the data: -**In Omnibus installations:** +For Linux package installations: ```shell sudo gitlab-rake gitlab:external_diffs:force_object_storage ``` -**In installations from source:** +For self-compiled installations: ```shell sudo -u git -H bundle exec rake gitlab:external_diffs:force_object_storage RAILS_ENV=production diff --git a/doc/administration/monitoring/ip_allowlist.md b/doc/administration/monitoring/ip_allowlist.md index 94d6876b16f..72640cd6218 100644 --- a/doc/administration/monitoring/ip_allowlist.md +++ b/doc/administration/monitoring/ip_allowlist.md @@ -20,7 +20,7 @@ hosts or use IP ranges: gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '192.168.0.1'] ``` -1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. +1. Save the file and [reconfigure](../restart_gitlab.md#reconfigure-a-linux-package-installation) GitLab for the changes to take effect. --- diff --git a/doc/administration/monitoring/performance/gitlab_configuration.md b/doc/administration/monitoring/performance/gitlab_configuration.md index cb5852a7dac..0d2037f3a92 100644 --- a/doc/administration/monitoring/performance/gitlab_configuration.md +++ b/doc/administration/monitoring/performance/gitlab_configuration.md @@ -9,7 +9,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w GitLab Performance Monitoring is disabled by default. To enable it and change any of its settings: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Metrics and profiling** (`/admin/application_settings/metrics_and_profiling`). 1. Add the necessary configuration changes. diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md index 1113dcfef32..b448ac461c8 100644 --- a/doc/administration/monitoring/performance/grafana_configuration.md +++ b/doc/administration/monitoring/performance/grafana_configuration.md @@ -14,7 +14,7 @@ For more information, see [deprecation notes](#deprecation-of-bundled-grafana). [Grafana](https://grafana.com/) is a tool that enables you to visualize time series metrics through graphs and dashboards. GitLab writes performance data to Prometheus, -and Grafana allows you to query the data to display useful graphs. +and Grafana allows you to query the data to display graphs. ## Deprecation of bundled Grafana @@ -30,29 +30,22 @@ To switch away from bundled Grafana to a newer version of Grafana from Grafana L 1. Set up a version of Grafana from Grafana Labs. 1. [Export the existing dashboards](https://grafana.com/docs/grafana/latest/dashboards/manage-dashboards/#export-a-dashboard) from bundled Grafana. 1. [Import the existing dashboards](https://grafana.com/docs/grafana/latest/dashboards/manage-dashboards/#import-a-dashboard) in the new Grafana instance. -1. [Configure GitLab](#integration-with-gitlab-ui) to use the new Grafana instance. +1. [Configure GitLab](#integrate-with-gitlab-ui) to use the new Grafana instance. ### Temporary workaround In GitLab versions 16.0 to 16.2, you can still force Omnibus GitLab to enable and configure Grafana by setting the following: - `grafana['enable'] = true`. -- `grafana['enable_deprecated_service'] = true`. - -You see a deprecation message when reconfiguring GitLab. +- `grafana['enable_deprecated_service'] = true`. -## Installation +You see a deprecation message when reconfiguring GitLab. -Omnibus GitLab can [help you install Grafana (recommended)](https://docs.gitlab.com/omnibus/settings/grafana.html) -or Grafana supplies package repositories (Yum/Apt) for easy installation. -See [Grafana installation documentation](https://grafana.com/docs/grafana/latest/setup-grafana/installation/) -for detailed steps. +## Configure Grafana -Before starting Grafana for the first time, set the administration user -and password in `/etc/grafana/grafana.ini`. If you don't, the default password -is `admin`. +Prerequisites: -## Configuration +- Grafana installed. 1. Log in to Grafana as the administration user. 1. Select **Data Sources** from the **Configuration** menu. @@ -62,9 +55,9 @@ is `admin`. Grafana should indicate the data source is working. -## Import Dashboards +## Import dashboards -You can now import a set of default dashboards to start displaying useful information. +You can now import a set of default dashboards to start displaying information. GitLab has published a set of default [Grafana dashboards](https://gitlab.com/gitlab-org/grafana-dashboards) to get you started. To use them: @@ -86,14 +79,15 @@ instance. For more information about this process, see the [README of the Grafana dashboards](https://gitlab.com/gitlab-org/grafana-dashboards) repository. -## Integration with GitLab UI +## Integrate with GitLab UI > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/61005) in GitLab 12.1. -After setting up Grafana, you can enable a link to access it easily from the +After setting up Grafana, you can enable a link to access it from the GitLab sidebar: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Metrics and profiling** and expand **Metrics - Grafana**. 1. Select the **Add a link to Grafana** checkbox. @@ -129,7 +123,7 @@ configuration screen: > prior to 13.10, the API scope: > > - Is required to access Grafana through the GitLab OAuth provider. -> - Is set by enabling the Grafana application as shown in [Integration with GitLab UI](#integration-with-gitlab-ui). +> - Is set by enabling the Grafana application as shown in [Integration with GitLab UI](#integrate-with-gitlab-ui). ## Security Update diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index fc6318922f1..3fdd4c24177 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -108,7 +108,8 @@ The performance bar is disabled by default for non-administrators. To enable it for a given group: 1. Sign in as a user with administrator access. -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Metrics and profiling** (`admin/application_settings/metrics_and_profiling`), and expand **Profiling - Performance bar**. diff --git a/doc/administration/monitoring/prometheus/gitlab_exporter.md b/doc/administration/monitoring/prometheus/gitlab_exporter.md index 1f4156349c5..0bd13fe5a87 100644 --- a/doc/administration/monitoring/prometheus/gitlab_exporter.md +++ b/doc/administration/monitoring/prometheus/gitlab_exporter.md @@ -24,7 +24,7 @@ To enable the GitLab exporter in an Omnibus GitLab instance: gitlab_exporter['enable'] = true ``` -1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) +1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. Prometheus automatically begins collecting performance data from @@ -49,7 +49,7 @@ To change the Rack server to Puma: gitlab_exporter['server_name'] = 'puma' ``` -1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) +1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. The supported Rack servers are `webrick` and `puma`. diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index c0a175c76cd..5d6827b79ee 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -9,10 +9,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w To enable the GitLab Prometheus metrics: 1. Log in to GitLab as a user with administrator access. -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Metrics and profiling**. 1. Find the **Metrics - Prometheus** section, and select **Enable GitLab Prometheus metrics endpoint**. -1. [Restart GitLab](../../restart_gitlab.md#omnibus-gitlab-restart) for the changes to take effect. +1. [Restart GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. For installations from source you must configure it yourself. @@ -36,7 +37,7 @@ The following metrics are available: | Metric | Type | Since | Description | Labels | | :--------------------------------------------------------------- | :---------- | ------: | :-------------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------- | -| `gitlab_cache_misses_total` | Counter | 10.2 | Cache read miss | `controller`, `action` | +| `gitlab_cache_misses_total` | Counter | 10.2 | Cache read miss | `controller`, `action`, `store` | | `gitlab_cache_operation_duration_seconds` | Histogram | 10.2 | Cache access time | `operation`, `store` | | `gitlab_cache_operations_total` | Counter | 12.2 | Cache operations by controller or action | `controller`, `action`, `operation`, `store` | | `gitlab_cache_read_multikey_count` | Histogram | 15.7 | Count of keys in multi-key cache read operations | `controller`, `action`, `store` | @@ -52,6 +53,7 @@ The following metrics are available: | `gitlab_ci_active_jobs` | Histogram | 14.2 | Count of active jobs when pipeline is created | | | `gitlab_database_transaction_seconds` | Histogram | 12.1 | Time spent in database transactions, in seconds | | | `gitlab_method_call_duration_seconds` | Histogram | 10.2 | Method calls real duration | `controller`, `action`, `module`, `method` | +| `gitlab_omniauth_login_total` | Counter | 16.1 | Total number of OmniAuth logins attempts | `omniauth_provider`, `status` | | `gitlab_page_out_of_bounds` | Counter | 12.8 | Counter for the PageLimiter pagination limit being hit | `controller`, `action`, `bot` | | `gitlab_rails_boot_time_seconds` | Gauge | 14.8 | Time elapsed for Rails primary process to finish startup | | | `gitlab_rails_queue_duration_seconds` | Histogram | 9.4 | Measures latency between GitLab Workhorse forwarding a request to Rails | | @@ -63,8 +65,8 @@ The following metrics are available: | `gitlab_transaction_cache_<key>_duration_total` | Counter | 10.2 | Counter for total time (seconds) spent in Rails cache calls (per key) | | | `gitlab_transaction_cache_count_total` | Counter | 10.2 | Counter for total Rails cache calls (aggregate) | | | `gitlab_transaction_cache_duration_total` | Counter | 10.2 | Counter for total time (seconds) spent in Rails cache calls (aggregate) | | -| `gitlab_transaction_cache_read_hit_count_total` | Counter | 10.2 | Counter for cache hits for Rails cache calls | `controller`, `action` | -| `gitlab_transaction_cache_read_miss_count_total` | Counter | 10.2 | Counter for cache misses for Rails cache calls | `controller`, `action` | +| `gitlab_transaction_cache_read_hit_count_total` | Counter | 10.2 | Counter for cache hits for Rails cache calls | `controller`, `action`, `store` | +| `gitlab_transaction_cache_read_miss_count_total` | Counter | 10.2 | Counter for cache misses for Rails cache calls | `controller`, `action`, `store` | | `gitlab_transaction_duration_seconds` | Histogram | 10.2 | Duration for successful requests (`gitlab_transaction_*` metrics) | `controller`, `action` | | `gitlab_transaction_event_build_found_total` | Counter | 9.4 | Counter for build found for API /jobs/request | | | `gitlab_transaction_event_build_invalid_total` | Counter | 9.4 | Counter for build invalid due to concurrency conflict for API /jobs/request | | @@ -147,9 +149,9 @@ The following metrics are available: | `service_desk_thank_you_email` | Counter | 14.0 | Total number of email responses to new Service Desk emails | | | `service_desk_new_note_email` | Counter | 14.0 | Total number of email notifications on new Service Desk comment | | | `email_receiver_error` | Counter | 14.1 | Total number of errors when processing incoming emails | | -| `gitlab_snowplow_events_total` | Counter | 14.1 | Total number of GitLab Snowplow product intelligence events emitted | | -| `gitlab_snowplow_failed_events_total` | Counter | 14.1 | Total number of GitLab Snowplow product intelligence events emission failures | | -| `gitlab_snowplow_successful_events_total` | Counter | 14.1 | Total number of GitLab Snowplow product intelligence events emission successes | | +| `gitlab_snowplow_events_total` | Counter | 14.1 | Total number of GitLab Snowplow Analytics Instrumentation events emitted | | +| `gitlab_snowplow_failed_events_total` | Counter | 14.1 | Total number of GitLab Snowplow Analytics Instrumentation events emission failures | | +| `gitlab_snowplow_successful_events_total` | Counter | 14.1 | Total number of GitLab Snowplow Analytics Instrumentation events emission successes | | | `gitlab_ci_build_trace_errors_total` | Counter | 14.4 | Total amount of different error types on a build trace | `error_reason` | | `gitlab_presentable_object_cacheless_render_real_duration_seconds` | Histogram | 15.3 | Duration of real time spent caching and representing specific web request objects | `controller`, `action` | | `cached_object_operations_total` | Counter | 15.3 | Total number of objects cached for specific web requests | `controller`, `action` | @@ -204,6 +206,7 @@ configuration option in `gitlab.yml`. These metrics are served from the | `sidekiq_jobs_dead_total` | Counter | 13.7 | Sidekiq dead jobs (jobs that have run out of retries) | `queue`, `boundary`, `external_dependencies`, `feature_category`, `urgency` | | `sidekiq_redis_requests_total` | Counter | 13.1 | Redis requests during a Sidekiq job execution | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency` | | `sidekiq_elasticsearch_requests_total` | Counter | 13.1 | Elasticsearch requests during a Sidekiq job execution | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency` | +| `sidekiq_jobs_deferred_total` | Counter | 16.1 | Number of jobs being deferred when `defer_sidekiq_jobs` feature flag is enabled | `worker` | | `sidekiq_running_jobs` | Gauge | 12.2 | Number of Sidekiq jobs running | `queue`, `boundary`, `external_dependencies`, `feature_category`, `urgency` | | `sidekiq_concurrency` | Gauge | 12.5 | Maximum number of Sidekiq jobs | | | `sidekiq_mem_total_bytes` | Gauge | 15.3 | Number of bytes allocated for both objects consuming an object slot and objects that required a malloc'| | @@ -221,9 +224,6 @@ configuration option in `gitlab.yml`. These metrics are served from the | `geo_lfs_objects_verified` | Gauge | 14.6 | Number of LFS objects successfully verified on secondary | `url` | | `geo_lfs_objects_verification_failed` | Gauge | 14.6 | Number of LFS objects that failed verifications on secondary | `url` | | `geo_lfs_objects_verification_total` | Gauge | 14.6 | Number of LFS objects to attempt to verify on secondary | `url` | -| `geo_attachments` | Gauge | 10.2 | Total number of file attachments available on primary | `url` | -| `geo_attachments_synced` | Gauge | 10.2 | Number of attachments synced on secondary | `url` | -| `geo_attachments_failed` | Gauge | 10.2 | Number of attachments failed to sync on secondary | `url` | | `geo_last_event_id` | Gauge | 10.2 | Database ID of the latest event log entry on the primary | `url` | | `geo_last_event_timestamp` | Gauge | 10.2 | UNIX timestamp of the latest event log entry on the primary | `url` | | `geo_cursor_last_event_id` | Gauge | 10.2 | Last database ID of the event log processed by the secondary | `url` | @@ -375,6 +375,16 @@ configuration option in `gitlab.yml`. These metrics are served from the | `gitlab_memwd_violations_handled_total` | Counter | 15.9 | Total number of times Sidekiq process memory violations were handled | | | `sidekiq_watchdog_running_jobs_total` | Counter | 15.9 | Current running jobs when RSS limit was reached | `worker_class` | | `gitlab_maintenance_mode` | Gauge | 15.11 | Is GitLab Maintenance Mode enabled? | | +| `geo_design_management_repositories` | Gauge | 16.1 | Number of design repositories on primary | `url` | +| `geo_design_management_repositories_checksum_total` | Gauge | 16.1 | Number of design repositories tried to checksum on primary | `url` | +| `geo_design_management_repositories_checksummed` | Gauge | 16.1 | Number of design repositories successfully checksummed on primary | `url` | +| `geo_design_management_repositories_checksum_failed` | Gauge | 16.1 | Number of design repositories failed to calculate the checksum on primary | `url` | +| `geo_design_management_repositories_synced` | Gauge | 16.1 | Number of syncable design repositories synced on secondary | `url` | +| `geo_design_management_repositories_failed` | Gauge | 16.1 | Number of syncable design repositories failed to sync on secondary | `url` | +| `geo_design_management_repositories_registry` | Gauge | 16.1 | Number of design repositories in the registry | `url` | +| `geo_design_management_repositories_verification_total` | Gauge | 16.1 | Number of design repositories verifications tried on secondary | `url` | +| `geo_design_management_repositories_verified` | Gauge | 16.1 | Number of design repositories verified on secondary | `url` | +| `geo_design_management_repositories_verification_failed` | Gauge | 16.1 | Number of design repositories verifications failed on secondary | `url` | ## Database load balancing metrics **(PREMIUM SELF)** diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index 013c4515268..0e8315e528a 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -49,7 +49,7 @@ To disable Prometheus and all of its exporters, as well as any added in the futu prometheus_monitoring['enable'] = false ``` -1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to +1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. ### Changing the port and address Prometheus listens on @@ -80,7 +80,7 @@ listens on: prometheus['listen_address'] = '0.0.0.0:9090' ``` -1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to +1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect ### Adding custom scrape configurations @@ -158,7 +158,7 @@ The next step is to tell all the other nodes where the monitoring node is: Where `10.0.0.1:9090` is the IP address and port of the Prometheus node. -1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to +1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. After monitoring using Service Discovery is enabled with `consul['monitoring_service_discovery'] = true`, @@ -190,6 +190,7 @@ To use an external Prometheus server: # Rails nodes gitlab_exporter['listen_address'] = '0.0.0.0' gitlab_exporter['listen_port'] = '9168' + registry['debug_addr'] = '0.0.0.0:5001' # Sidekiq nodes sidekiq['listen_address'] = '0.0.0.0' @@ -205,14 +206,12 @@ To use an external Prometheus server: # ... prometheus_listen_addr: '0.0.0.0:9236', } + + # Pgbouncer nodes + pgbouncer_exporter['listen_address'] = '0.0.0.0:9188' ``` 1. Install and set up a dedicated Prometheus instance, if necessary, using the [official installation instructions](https://prometheus.io/docs/prometheus/latest/installation/). -1. Add the Prometheus server IP address to the [monitoring IP allowlist](../ip_allowlist.md). For example: - - ```ruby - gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '192.168.0.1'] - ``` 1. On **all** GitLab Rails(Puma, Sidekiq) servers, set the Prometheus server IP address and listen port. For example: @@ -232,7 +231,27 @@ To use an external Prometheus server: } ``` -1. [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) to apply the changes. + You can also specify more than one IP address if you have multiple Prometheus servers: + + ```ruby + nginx['status']['options'] = { + "server_tokens" => "off", + "access_log" => "off", + "allow" => ["192.168.0.1", "192.168.0.2"] + "deny" => "all", + } + ``` + +1. To allow the Prometheus server to fetch from the [GitLab metrics](#gitlab-metrics) endpoint, add the Prometheus +server IP address to the [monitoring IP allowlist](../ip_allowlist.md): + + ```ruby + gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '192.168.0.1'] + ``` + +1. As we are setting each bundled service's [exporter](#bundled-software-metrics) to listen on a network address, +update the firewall on the instance to only allow traffic from your Prometheus IP for the exporters enabled. A full reference list of exporter services and their respective ports can be found [here](../../package_information/defaults.md#ports). +1. [Reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) to apply the changes. 1. Edit the Prometheus server's configuration file. 1. Add each node's exporters to the Prometheus server's [scrape target configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E). @@ -429,7 +448,7 @@ To disable the monitoring of Kubernetes: prometheus['monitor_kubernetes'] = false ``` -1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to +1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. ## Troubleshooting diff --git a/doc/administration/monitoring/prometheus/node_exporter.md b/doc/administration/monitoring/prometheus/node_exporter.md index 337ab9becaa..f4d5d9942ca 100644 --- a/doc/administration/monitoring/prometheus/node_exporter.md +++ b/doc/administration/monitoring/prometheus/node_exporter.md @@ -21,7 +21,7 @@ To enable the node exporter: node_exporter['enable'] = true ``` -1. Save the file, and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) +1. Save the file, and [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. Prometheus begins collecting performance data from the node exporter diff --git a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md index f8d48832288..a95dc47dc34 100644 --- a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md +++ b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md @@ -21,7 +21,7 @@ To enable the PgBouncer exporter: pgbouncer_exporter['enable'] = true ``` -1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) +1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. Prometheus begins collecting performance data from the PgBouncer exporter diff --git a/doc/administration/monitoring/prometheus/postgres_exporter.md b/doc/administration/monitoring/prometheus/postgres_exporter.md index 1cae28a069f..39f6c81e8d0 100644 --- a/doc/administration/monitoring/prometheus/postgres_exporter.md +++ b/doc/administration/monitoring/prometheus/postgres_exporter.md @@ -23,7 +23,7 @@ To enable the PostgreSQL Server Exporter: address is [listed in `trust_auth_cidr_addresses`](../../postgresql/replication_and_failover.md#network-information) or the exporter can't connect to the database. -1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to +1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. Prometheus begins collecting performance data from @@ -70,5 +70,5 @@ use the following configuration options: postgres_exporter['sslrootcert'] = 'ssl-root.crt' ``` -1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) +1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. diff --git a/doc/administration/monitoring/prometheus/redis_exporter.md b/doc/administration/monitoring/prometheus/redis_exporter.md index b17bf9787a7..5f11034dc15 100644 --- a/doc/administration/monitoring/prometheus/redis_exporter.md +++ b/doc/administration/monitoring/prometheus/redis_exporter.md @@ -22,7 +22,7 @@ To enable the Redis exporter: redis_exporter['enable'] = true ``` -1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) +1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. Prometheus begins collecting performance data from diff --git a/doc/administration/monitoring/prometheus/registry_exporter.md b/doc/administration/monitoring/prometheus/registry_exporter.md index 3f4d6cfb14d..1c46224f7b3 100644 --- a/doc/administration/monitoring/prometheus/registry_exporter.md +++ b/doc/administration/monitoring/prometheus/registry_exporter.md @@ -16,7 +16,7 @@ To enable it: registry['debug_addr'] = "localhost:5001" # localhost:5001/metrics ``` -1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) +1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. Prometheus automatically begins collecting performance data from diff --git a/doc/administration/monitoring/prometheus/web_exporter.md b/doc/administration/monitoring/prometheus/web_exporter.md index 0212091f14c..eb6fe6d4d60 100644 --- a/doc/administration/monitoring/prometheus/web_exporter.md +++ b/doc/administration/monitoring/prometheus/web_exporter.md @@ -47,7 +47,7 @@ To enable the dedicated server: to `localhost:8083/metrics`. Refer to the [Adding custom scrape configurations](index.md#adding-custom-scrape-configurations) page for how to configure scraper targets. For external Prometheus setup, refer to [Using an external Prometheus server](index.md#using-an-external-prometheus-server) instead. -1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) +1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. Metrics can now be served and scraped from `localhost:8083/metrics`. @@ -66,7 +66,7 @@ To serve metrics via HTTPS instead of HTTP, enable TLS in the exporter settings: puma['exporter_tls_key_path'] = "/path/to/private-key.pem" ``` -1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) +1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. When TLS is enabled, the same `port` and `address` is used as described above. diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index 1e6605e41a8..f2b966bd180 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -114,6 +114,7 @@ The following table lists the valid `objects` that can be used: | `dependency_proxy` | [Dependency Proxy](packages/dependency_proxy.md) | | `terraform_state` | [Terraform state files](terraform_state.md) | | `pages` | [Pages](pages/index.md) | +| `ci_secure_files` | [Project-level Secure Files](secure_files.md) | Within each object type, three parameters can be defined: @@ -161,6 +162,7 @@ supported by consolidated form, refer to the following guides: | Object storage type | Supported by consolidated form? | |---------------------|------------------------------------------| +| [Project-level Secure Files](secure_files.md#using-object-storage) | **{dotted-circle}** No | | [Backups](../raketasks/backup_gitlab.md#upload-backups-to-a-remote-cloud-storage) | **{dotted-circle}** No | | [Container Registry](packages/container_registry.md#use-object-storage) (optional feature) | **{dotted-circle}** No | | [Mattermost](https://docs.mattermost.com/configure/file-storage-configuration-settings.html)| **{dotted-circle}** No | @@ -315,8 +317,7 @@ Bucket encryption with the [Cloud Key Management Service (KMS)](https://cloud.go #### GCS example -For Omnibus installations, this is an example of the `connection` setting -in the consolidated form: +For Linux Package installations, this is an example of the `connection` setting in the consolidated form: ```ruby gitlab_rails['object_store']['connection'] = { @@ -376,8 +377,7 @@ The following are the valid connection parameters for Azure. For more informatio | `azure_storage_access_key` | Storage account access key used to access the container. This is typically a secret, 512-bit encryption key encoded in base64. | `czV2OHkvQj9FKEgrTWJRZVRoV21ZcTN0Nnc5eiRDJkYpSkBOY1JmVWpYbjJy\nNHU3eCFBJUQqRy1LYVBkU2dWaw==\n` | | `azure_storage_domain` | Domain name used to contact the Azure Blob Storage API (optional). Defaults to `blob.core.windows.net`. Set this if you are using Azure China, Azure Germany, Azure US Government, or some other custom Azure domain. | `blob.core.windows.net` | -- For Omnibus installations, this is an example of the `connection` setting - in the consolidated form: +- For Linux package installations, this is an example of the `connection` setting in the consolidated form: ```ruby gitlab_rails['object_store']['connection'] = { @@ -388,8 +388,8 @@ The following are the valid connection parameters for Azure. For more informatio } ``` -- For source installations, Workhorse also needs to be configured with Azure - credentials. This isn't needed in Omnibus installs, because the Workhorse +- For self-compiled installations, Workhorse also needs to be configured with Azure + credentials. This isn't needed in Linux package installations because the Workhorse settings are populated from the previous settings. 1. Edit `/home/git/gitlab-workhorse/config.toml` and add or amend the following lines: @@ -465,7 +465,6 @@ The following example uses AWS S3 to enable object storage for all supported ser gitlab_rails['object_store']['objects']['packages']['bucket'] = 'gitlab-packages' gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = 'gitlab-dependency-proxy' gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = 'gitlab-terraform-state' - gitlab_rails['object_store']['objects']['ci_secure_files']['bucket'] = 'gitlab-ci-secure-files' gitlab_rails['object_store']['objects']['pages']['bucket'] = 'gitlab-pages' ``` @@ -728,6 +727,7 @@ To migrate existing local data to object storage see the following guides: - [Dependency Proxy](packages/dependency_proxy.md#migrate-local-dependency-proxy-blobs-and-manifests-to-object-storage) - [Terraform state files](terraform_state.md#migrate-to-object-storage) - [Pages content](pages/index.md#migrate-pages-deployments-to-object-storage) +- [Project-level Secure Files](secure_files.md#migrate-to-object-storage) ## Transition to consolidated form @@ -738,7 +738,7 @@ Prior to GitLab 13.2: - Object store connection parameters such as passwords and endpoint URLs had to be duplicated for each type. -For example, an Omnibus GitLab install might have the following configuration: +For example, a Linux package installation might have the following configuration: ```ruby # Original object storage configuration @@ -833,10 +833,9 @@ your object storage provider instead. Using separate buckets for each data type is the recommended approach for GitLab. This ensures there are no collisions across the various types of data GitLab stores. -There are plans to [enable the use of a single bucket](https://gitlab.com/gitlab-org/gitlab/-/issues/292958) -in the future. +[Issue 292958](https://gitlab.com/gitlab-org/gitlab/-/issues/292958) proposes to enable the use of a single bucket. -With Omnibus and source installations it is possible to split a single +With Linux package and self-compiled installations, it is possible to split a single real bucket into multiple virtual buckets. If your object storage bucket is called `my-gitlab-objects` you can configure uploads to go into `my-gitlab-objects/uploads`, artifacts into @@ -941,7 +940,7 @@ mismatch` error during an upload. When the consolidated form is: -- Used with an S3-compatible object storage or an istance profile, Workhorse +- Used with an S3-compatible object storage or an instance profile, Workhorse uses its internal S3 client which has S3 credentials so that it can compute the `Content-MD5` header. This eliminates the need to compare ETag headers returned from the S3 server. diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index 1e887d8bd67..d54d286c19d 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -121,7 +121,8 @@ users as long as a large file exists. To disable writes to the `authorized_keys` file: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Network**. 1. Expand **Performance optimization**. 1. Clear the **Use authorized_keys file to authenticate SSH keys** checkbox. @@ -140,7 +141,8 @@ This overview is brief. Refer to the above instructions for more context. 1. [Rebuild the `authorized_keys` file](../raketasks/maintenance.md#rebuild-authorized_keys-file). 1. Enable writes to the `authorized_keys` file. - 1. On the top bar, select **Main menu > Admin**. + 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Network**. 1. Expand **Performance optimization**. 1. Select the **Use authorized_keys file to authenticate SSH keys** checkbox. diff --git a/doc/administration/operations/gitlab_sshd.md b/doc/administration/operations/gitlab_sshd.md index 249d6232616..5c4af32fc3d 100644 --- a/doc/administration/operations/gitlab_sshd.md +++ b/doc/administration/operations/gitlab_sshd.md @@ -27,9 +27,8 @@ If you are considering switching from OpenSSH to `gitlab-sshd`, consider these c - `gitlab-sshd` supports the PROXY protocol. It can run behind proxy servers that rely on it, such as HAProxy. The PROXY protocol is not enabled by default, but [it can be enabled](#proxy-protocol-support). -- `gitlab-sshd` **does not** support SSH certificates. For more details, see the - [confidential issue](../../user/project/issues/confidential_issues.md) - `https://gitlab.com/gitlab-org/gitlab-shell/-/issues/495`. +- `gitlab-sshd` does not support SSH certificates. For discussion about adding them, + see [issue 655](https://gitlab.com/gitlab-org/gitlab-shell/-/issues/655). ## Enable `gitlab-sshd` diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md index efc55a5fbc3..d7d6f6228f9 100644 --- a/doc/administration/operations/puma.md +++ b/doc/administration/operations/puma.md @@ -99,7 +99,7 @@ To change the worker timeout to 600 seconds: ## Disable Puma clustered mode in memory-constrained environments WARNING: -This feature is an [Experiment](../../policy/alpha-beta-support.md#experiment) and subject to change without notice. The feature +This feature is an [Experiment](../../policy/experiment-beta-support.md#experiment) and subject to change without notice. The feature is not ready for production use. If you want to use this feature, you should test outside of production first. See the [known issues](#puma-single-mode-known-issues) for additional details. @@ -211,6 +211,71 @@ make Prometheus scrape them over HTTPS, and support for it is being discussed Hence, it is not technically possible to turn off this HTTP listener without losing Prometheus metrics. +### Using an encrypted SSL key + +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7799) in GitLab 16.1. + +Puma supports the use of an encrypted private SSL key, which can be +decrypted at runtime. The following instructions illustrate how to +configure this: + +1. Encrypt the key with a password if it is not already: + + ```shell + openssl rsa -aes256 -in /path/to/ssl-key.pem -out /path/to/encrypted-ssl-key.pem + ``` + + Enter in a password twice to write the encrypted file. In this + example, we use `some-password-here`. + +1. Create a script or executable that prints the password. For + example, create a basic script in + `/var/opt/gitlab/gitlab-rails/etc/puma-ssl-key-password` that echoes + the password: + + ```shell + #!/bin/sh + echo some-password-here + ``` + + Note that in production, you should avoid storing the password on + disk and use a secure mechanism for retrieving a password, such as + Vault. For example, the script might look like: + + ```shell + #!/bin/sh + export VAULT_ADDR=http://vault-password-distribution-point:8200 + export VAULT_TOKEN=<some token> + + echo "$(vault kv get -mount=secret puma-ssl-password)" + ``` + +1. Ensure the Puma process has sufficient permissions to execute the + script and to read the encrypted key: + + ```shell + chown git:git /var/opt/gitlab/gitlab-rails/etc/puma-ssl-key-password + chmod 770 /var/opt/gitlab/gitlab-rails/etc/puma-ssl-key-password + chmod 660 /path/to/encrypted-ssl-key.pem + ``` + +1. Edit `/etc/gitlab/gitlab.rb`, and replace `puma['ssl_certificate_key']` with the encrypted key and specify + `puma['ssl_key_password_command]`: + + ```ruby + puma['ssl_certificate_key'] = '/path/to/encrypted-ssl-key.pem' + puma['ssl_key_password_command'] = '/var/opt/gitlab/gitlab-rails/etc/puma-ssl-key-password' + ``` + +1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. If GitLab comes up successfully, you should be able to remove the + unencrypted SSL key that was stored on the GitLab instance. + ## Switch from Unicorn to Puma NOTE: @@ -333,7 +398,7 @@ gitlab_rails['env'] = { For source installations, set the environment variable. Refer to [Puma Worker timeout](../operations/puma.md#change-the-worker-timeout). -[Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. +[Reconfigure](../restart_gitlab.md#reconfigure-a-linux-package-installation) GitLab for the changes to take effect. #### Troubleshooting without affecting other users diff --git a/doc/administration/package_information/defaults.md b/doc/administration/package_information/defaults.md index 96b56388ea9..ac183afdc2f 100644 --- a/doc/administration/package_information/defaults.md +++ b/doc/administration/package_information/defaults.md @@ -53,6 +53,8 @@ by default: | Gitaly | Yes | Socket | Port (8075) | 8075 or 9999 (TLS) | | Gitaly exporter | Yes | Port | X | 9236 | | Praefect | No | Port | X | 2305 or 3305 (TLS) | +| GitLab Workhorse exporter | Yes | Port | X | 9229 | +| Registry exporter | No | Port | X | 5001 | Legend: diff --git a/doc/administration/package_information/postgresql_versions.md b/doc/administration/package_information/postgresql_versions.md index c1e9f7320ea..44032883eb4 100644 --- a/doc/administration/package_information/postgresql_versions.md +++ b/doc/administration/package_information/postgresql_versions.md @@ -30,6 +30,7 @@ Read more about update policies and warnings in the PostgreSQL | GitLab version | PostgreSQL versions | Default version for fresh installs | Default version for upgrades | Notes | | -------------- | --------------------- | ---------------------------------- | ---------------------------- | ----- | +| 16.0 | 13.11 | 13.11 | 13.11 | | | 15.6 | 12.12, 13.8 | 13.8 | 12.12 | For upgrades, users can manually upgrade to 13.8 following the [upgrade documentation](https://docs.gitlab.com/omnibus/settings/database.html#gitlab-150-and-later). | | 15.0 | 12.10, 13.6 | 13.6 | 12.10 | For upgrades, users can manually upgrade to 13.6 following the [upgrade documentation](https://docs.gitlab.com/omnibus/settings/database.html#gitlab-150-and-later). | | 14.1 | 12.7, 13.3 | 12.7 | 12.7 | PostgreSQL 13 available for fresh installations if not using [Geo](../geo/index.md#requirements-for-running-geo) or [Patroni](../postgresql/index.md#postgresql-replication-and-failover-with-omnibus-gitlab). diff --git a/doc/administration/package_information/supported_os.md b/doc/administration/package_information/supported_os.md index a1f589015cb..8ad3a656e05 100644 --- a/doc/administration/package_information/supported_os.md +++ b/doc/administration/package_information/supported_os.md @@ -28,6 +28,7 @@ architecture. | Debian 11 | GitLab CE / GitLab EE 14.6.0 | amd64, arm64 | [Debian Install Documentation](https://about.gitlab.com/install/#debian) | 2026 | <https://wiki.debian.org/LTS> | | OpenSUSE 15.4 | GitLab CE / GitLab EE 15.7.0 | x86_64, aarch64 | [OpenSUSE Install Documentation](https://about.gitlab.com/install/#opensuse-leap) | Nov 2023 | <https://en.opensuse.org/Lifetime> | | RHEL 8 | GitLab CE / GitLab EE 12.8.1 | x86_64, arm64 | [Use CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | May 2029 | [RHEL Details](https://access.redhat.com/support/policy/updates/errata/#Life_Cycle_Dates) | +| RHEL 9 | GitLab CE / GitLab EE 16.0.0 | x86_64, arm64 | [Use CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | May 2032 | [RHEL Details](https://access.redhat.com/support/policy/updates/errata/#Life_Cycle_Dates) | | SLES 12 | GitLab EE 9.0.0 | x86_64 | [Use OpenSUSE Install Documentation](https://about.gitlab.com/install/#opensuse-leap) | Oct 2027 | <https://www.suse.com/lifecycle/> | | SLES 15 | GitLab EE 14.8.0 | x86_64 | [Use OpenSUSE Install Documentation](https://about.gitlab.com/install/#opensuse-leap) | Dec 2024 | <https://www.suse.com/lifecycle/> | | Oracle Linux | GitLab CE / GitLab EE 8.14.0 | x86_64 | [Use CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | Jul 2024 | <https://www.oracle.com/a/ocom/docs/elsp-lifetime-069338.pdf> | diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index fd3cbb2ad05..e3867a25846 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -147,7 +147,7 @@ under the `registry_external_url` line, rather than the port listed under registry_nginx['ssl_certificate_key'] = "/path/to/certificate.key" ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Validate using: @@ -226,7 +226,7 @@ Let's assume that you want the container Registry to be accessible at The `registry_external_url` is listening on HTTPS. -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. If you have a [wildcard certificate](https://en.wikipedia.org/wiki/Wildcard_certificate), you must specify the path to the certificate in addition to the URL, in this case `/etc/gitlab/gitlab.rb` @@ -272,7 +272,7 @@ Registry application itself. registry['enable'] = false ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. **Installations from source** @@ -300,7 +300,7 @@ the Container Registry by themselves, follow the steps below. gitlab_rails['gitlab_default_projects_features_container_registry'] = false ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. **Installations from source** @@ -325,7 +325,8 @@ the Container Registry by themselves, follow the steps below. In GitLab, tokens for the Container Registry expire every five minutes. To increase the token duration: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Settings > CI/CD**. 1. Expand **Container Registry**. 1. For the **Authorization token duration (minutes)**, update the value. @@ -391,7 +392,7 @@ The default location where images are stored in Omnibus, is gitlab_rails['registry_path'] = "/path/to/registry/storage" ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. **Installations from source** @@ -490,7 +491,7 @@ To configure the `s3` storage driver in Omnibus: } ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. **Installations from source** @@ -556,7 +557,7 @@ you can pull from the Container Registry, but you cannot push. 1. To perform the final data sync, [put the Container Registry in `read-only` mode](#performing-garbage-collection-without-downtime) and - [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). 1. Sync any changes dating from after the initial data load to your S3 bucket, and delete files that exist in the destination bucket but not in the source: ```shell @@ -586,7 +587,7 @@ you can pull from the Container Registry, but you cannot push. The output of these commands should match, except for the content in the `_uploads` directories and sub-directories. 1. Configure your registry to [use the S3 bucket for storage](#use-object-storage). -1. For the changes to take effect, set the Registry back to `read-write` mode and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. For the changes to take effect, set the Registry back to `read-write` mode and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). #### Moving to Azure Object Storage @@ -598,7 +599,7 @@ The default configuration for the storage driver is scheduled to be [changed](ht <!--- end_remove --> When moving from an existing file system or another object storage provider to Azure Object Storage, you must configure the registry to use the standard root directory. -Configure it by setting [`trimlegacyrootprefix: true]`](https://gitlab.com/gitlab-org/container-registry/-/blob/a3f64464c3ec1c5a599c0a2daa99ebcbc0100b9a/docs-gitlab/README.md#azure-storage-driver) in the Azure storage driver section of the registry configuration. +Configure it by setting [`trimlegacyrootprefix: true`](https://gitlab.com/gitlab-org/container-registry/-/blob/a3f64464c3ec1c5a599c0a2daa99ebcbc0100b9a/docs-gitlab/README.md#azure-storage-driver) in the Azure storage driver section of the registry configuration. Without this configuration, the Azure storage driver uses `//` instead of `/` as the first section of the root path, rendering the migrated images inaccessible. **Omnibus GitLab installations** @@ -654,7 +655,7 @@ However, this behavior is undesirable for registries used by internal hosts that } ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. **Installations from source** @@ -705,7 +706,7 @@ For Omnibus GitLab installations: } ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. For installations from source: @@ -746,7 +747,7 @@ In the examples below we set the Registry's port to `5010`. registry['registry_http_addr'] = "localhost:5010" ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. **Installations from source** @@ -850,7 +851,7 @@ You can use GitLab as an auth endpoint with an external container registry. gitlab_rails['registry_port'] = "5005" ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. **Installations from source** @@ -905,7 +906,7 @@ To configure a notification endpoint in Omnibus: ] ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. **Installations from source** @@ -1285,6 +1286,33 @@ specified in its configuration and allows the operation. GitLab background jobs processing (through Sidekiq) also interacts with Registry. These jobs talk directly to Registry to handle image deletion. +## Migrate from a third-party registry + +Using external container registries in GitLab was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/376217) +in GitLab 15.8 and the end of support occurred in GitLab 16.0. See the [deprecation notice](../../update/deprecations.md#use-of-third-party-container-registries-is-deprecated) for more details. + +The integration is not disabled in GitLab 16.0, but support for debugging and fixing issues +is no longer provided. Additionally, the integration is no longer being developed or +enhanced with new features. Third-party registry functionality might be completely removed +after the new GitLab Container Registry version is available for self-managed (see epic [5521](https://gitlab.com/groups/gitlab-org/-/epics/5521)). Only the GitLab Container Registry is planned to be supported. + +This section has guidance for administrators migrating from third-party registries +to the GitLab Container Registry. If the third-party container registry you are using is not listed here, +you can describe your use cases in [the feedback issue](https://gitlab.com/gitlab-org/container-registry/-/issues/958). + +For all of the instructions provided below, you should try them first on a test environment. +Make sure everything continues to work as expected before replicating it in production. + +### Docker Distribution Registry + +The [Docker Distribution Registry](https://docs.docker.com/registry/) was donated to the CNCF +and is now known as the [Distribution Registry](https://github.com/distribution/distribution). +This registry is the open source implementation that the GitLab Container Registry is based on. +The GitLab Container Registry is compatible with the basic functionality provided by the Distribution Registry, +including all the supported storage backends. To migrate to the GitLab Container Registry +you can follow the instructions on this page, and use the same storage backend as the Distribution Registry. +The GitLab Container Registry should accept the same configuration that you are using for the Distribution Registry. + ## Troubleshooting Before diving in to the following sections, here's some basic troubleshooting: @@ -1422,7 +1450,7 @@ Start with a value between `25000000` (25 MB) and `50000000` (50 MB). } ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. **For installations from source** @@ -1456,7 +1484,7 @@ You can add a configuration option for backwards compatibility. registry['compatibility_schema1_enabled'] = true ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. **For installations from source** @@ -1504,7 +1532,7 @@ and a simple solution would be to enable relative URLs in the Registry. } ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. **For installations from source** @@ -1532,7 +1560,7 @@ in your `gitlab.rb` configuration. registry['debug_addr'] = "localhost:5001" ``` -After adding the setting, [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) to apply the change. +After adding the setting, [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) to apply the change. Use curl to request debug output from the debug server: @@ -1824,7 +1852,7 @@ In this case, follow these steps: gitlab_rails['registry_enabled'] = true ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Try the removal again. diff --git a/doc/administration/packages/dependency_proxy.md b/doc/administration/packages/dependency_proxy.md index a29d6d93051..5e770c2464b 100644 --- a/doc/administration/packages/dependency_proxy.md +++ b/doc/administration/packages/dependency_proxy.md @@ -39,7 +39,7 @@ correspond to your GitLab installation: gitlab_rails['dependency_proxy_enabled'] = false ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. ### Helm chart installations @@ -104,7 +104,7 @@ To change the local storage path: gitlab_rails['dependency_proxy_storage_path'] = "/mnt/dependency_proxy" ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. **Installations from source** @@ -156,7 +156,7 @@ This section describes the earlier configuration format. [Migration steps still } ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. **Installations from source** @@ -266,4 +266,4 @@ The default expiration and the expiration on GitLab.com is 15 minutes. "https_proxy" => "http://USERNAME:PASSWORD@example.com:8080" ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. diff --git a/doc/administration/packages/index.md b/doc/administration/packages/index.md index f0f238aa5ba..b735204c323 100644 --- a/doc/administration/packages/index.md +++ b/doc/administration/packages/index.md @@ -57,6 +57,91 @@ When downloading packages as dependencies in downstream projects, many requests Packages API. You may therefore reach enforced user and IP rate limits. To address this issue, you can define specific rate limits for the Packages API. For more details, see [Package Registry Rate Limits](../../user/admin_area/settings/package_registry_rate_limits.md). +## Enable or disable the Package Registry + +The Package Registry is enabled by default. To disable it: + +::Tabs + +:::TabTitle Linux package (Omnibus) + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + # Change to true to enable packages - enabled by default if not defined + gitlab_rails['packages_enabled'] = false + ``` + +1. Save the file and reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +:::TabTitle Helm chart (Kubernetes) + +1. Export the Helm values: + + ```shell + helm get values gitlab > gitlab_values.yaml + ``` + +1. Edit `gitlab_values.yaml`: + + ```yaml + global: + appConfig: + packages: + enabled: false + ``` + +1. Save the file and apply the new values: + + ```shell + helm upgrade -f gitlab_values.yaml gitlab gitlab/gitlab + ``` + +:::TabTitle Docker + +1. Edit `docker-compose.yml`: + + ```yaml + version: "3.6" + services: + gitlab: + environment: + GITLAB_OMNIBUS_CONFIG: | + gitlab_rails['packages_enabled'] = false + ``` + +1. Save the file and restart GitLab: + + ```shell + docker compose up -d + ``` + +:::TabTitle Self-compiled (source) + +1. Edit `/home/git/gitlab/config/gitlab.yml`: + + ```yaml + production: &base + packages: + enabled: false + ``` + +1. Save the file and restart GitLab: + + ```shell + # For systems running systemd + sudo systemctl restart gitlab.target + + # For systems running SysV init + sudo service gitlab restart + ``` + +::EndTabs + ## Change the storage path By default, the packages are stored locally, but you can change the default diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index 1626a4fd41a..e86726524d0 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -146,7 +146,7 @@ The Pages daemon doesn't listen to the outside world. pages_external_url "http://pages.example.com" # not a subdomain of external_url ``` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). Watch the [video tutorial](https://youtu.be/dD8c7WNcc6s) for this configuration. @@ -182,7 +182,7 @@ you must also add the full paths as shown below: pages_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/pages-nginx.key" ``` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). 1. If you're using [Pages Access Control](#access-control), update the redirect URI in the GitLab Pages [System OAuth application](../../integration/oauth_provider.md#create-an-instance-wide-application) to use the HTTPS protocol. @@ -221,13 +221,13 @@ This setup is primarily intended to be used when [installing a GitLab POC on Ama pages_nginx['redirect_http_to_https'] = true ``` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). ### Global settings Below is a table of all configuration settings known to Pages in Omnibus GitLab, and what they do. These options can be adjusted in `/etc/gitlab/gitlab.rb`, -and take effect after you [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +and take effect after you [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). Most of these settings don't have to be configured manually unless you need more granular control over how the Pages daemon runs and serves content in your environment. @@ -343,7 +343,7 @@ world. Custom domains are supported, but no TLS. If you don't have IPv6, you can omit the IPv6 address. -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). ### Custom domains with TLS support @@ -385,7 +385,7 @@ then you need to also add the full paths as shown below: gitlab_pages['cert_key'] = "/etc/gitlab/ssl/example.io.key" ``` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). 1. If you're using [Pages Access Control](#access-control), update the redirect URI in the GitLab Pages [System OAuth application](../../integration/oauth_provider.md#create-an-instance-wide-application) to use the HTTPS protocol. @@ -406,7 +406,8 @@ domain as a custom domain to their project. If your user base is private or otherwise trusted, you can disable the verification requirement: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Preferences**. 1. Expand **Pages**. 1. Clear the **Require users to prove ownership of custom domains** checkbox. @@ -423,7 +424,8 @@ sites served under a custom domain. To enable it: 1. Choose an email address on which you want to receive notifications about expiring domains. -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Preferences**. 1. Expand **Pages**. 1. Enter the email address for receiving notifications and accept Let's Encrypt's Terms of Service. @@ -453,7 +455,7 @@ Pages access control is disabled by default. To enable it: gitlab_pages['access_control'] = true ``` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). 1. Users can now configure it in their [projects' settings](../../user/project/pages/pages_access_control.md). NOTE: @@ -476,8 +478,9 @@ pre-existing applications must modify the GitLab Pages OAuth application. Follow this: 1. Enable [access control](#access-control). -1. On the top bar, select **Main menu > Admin**. -1. On the left sidebar, select **Settings > Applications**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. On the left sidebar, select **Applications**. 1. Expand **GitLab Pages**. 1. Clear the `api` scope's checkbox and select the desired scope's checkbox (for example, `read_api`). @@ -495,7 +498,8 @@ This can be helpful to restrict information published with Pages websites to the of your instance only. To do that: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Preferences**. 1. Expand **Pages**. 1. Select the **Disable public access to Pages sites** checkbox. @@ -512,7 +516,7 @@ internet connectivity is gated by a proxy. To use a proxy for GitLab Pages: gitlab_pages['env']['http_proxy'] = 'http://example:8080' ``` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. ### Using a custom Certificate Authority (CA) @@ -598,7 +602,7 @@ Follow the steps below to configure verbose logging of GitLab Pages daemon. gitlab_pages['log_verbose'] = true ``` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). ## Propagating the correlation ID @@ -617,7 +621,7 @@ To enable the propagation of the correlation ID: gitlab_pages['propagate_correlation_id'] = true ``` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). ## Change storage path @@ -632,7 +636,7 @@ are stored. gitlab_rails['pages_path'] = "/mnt/storage/pages" ``` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). ## Configure listener for reverse proxy requests @@ -654,7 +658,7 @@ Follow the steps below to configure the proxy listener of GitLab Pages. gitlab_pages['listen_proxy'] = "localhost:10080" ``` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). ## Set global maximum size of each GitLab Pages site **(FREE SELF)** @@ -664,7 +668,8 @@ Prerequisite: To set the global maximum pages size for a project: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Preferences**. 1. Expand **Pages**. 1. In **Maximum size of pages**, enter a value. The default is `100`. @@ -678,7 +683,7 @@ Prerequisite: To set the maximum size of each GitLab Pages site in a group, overriding the inherited setting: -1. On the top bar, select **Main menu > Groups** and find your group. +1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. 1. On the left sidebar, select **Settings > General**. 1. Expand **Pages**. 1. Enter a value under **Maximum size** in MB. @@ -692,7 +697,7 @@ Prerequisite: To set the maximum size of GitLab Pages site in a project, overriding the inherited setting: -1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. 1. On the left sidebar, select **Settings > Pages**. If this path is not visible, select **Deployments > Pages**. @@ -708,7 +713,8 @@ Prerequisite: To set the maximum number of GitLab Pages custom domains for a project: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Preferences**, and expand **Pages**. 1. Enter a value for **Maximum number of custom domains per project**. Use `0` for unlimited domains. 1. Select **Save changes**. @@ -754,7 +760,7 @@ database encryption. Proceed with caution. 1. Configure [the object storage and migrate pages data to it](#using-object-storage). -1. [Reconfigure the **GitLab server**](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the +1. [Reconfigure the **GitLab server**](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. The `gitlab-secrets.json` file is now updated with the new configuration. @@ -794,7 +800,7 @@ database encryption. Proceed with caution. mv /var/opt/gitlab/gitlab-rails/shared/pages/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json ``` -1. [Reconfigure the **Pages server**](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure the **Pages server**](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. On the **GitLab server**, make the following changes to `/etc/gitlab/gitlab.rb`: @@ -804,7 +810,7 @@ database encryption. Proceed with caution. pages_nginx['enable'] = false ``` -1. [Reconfigure the **GitLab server**](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure the **GitLab server**](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. It's possible to run GitLab Pages on multiple servers if you wish to distribute the load. You can do this through standard load balancing practices such as @@ -853,7 +859,7 @@ To explicitly enable API source: gitlab_pages['domain_config_source'] = "gitlab" ``` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. Or if you want to use legacy configuration source you can: @@ -863,7 +869,7 @@ Or if you want to use legacy configuration source you can: gitlab_pages['domain_config_source'] = "disk" ``` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. ### GitLab API cache configuration @@ -960,7 +966,7 @@ In Omnibus installations: } ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. [Migrate existing Pages deployments to object storage.](#migrate-pages-deployments-to-object-storage) @@ -1102,7 +1108,7 @@ If you use [object storage](#using-object-storage), you can disable local storag gitlab_rails['pages_local_store_enabled'] = false ``` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. Starting from GitLab 13.12, this setting also disables the [legacy storage](#migrate-legacy-storage-to-zip-storage), so if you were using NFS to serve Pages, you can completely disconnect from it. @@ -1176,7 +1182,7 @@ TLS connection-based rate limits are enforced using the following: gitlab_pages['rate_limit_source_ip_burst'] = 600 ``` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). #### Enable HTTP requests rate limits by domain @@ -1189,7 +1195,7 @@ TLS connection-based rate limits are enforced using the following: gitlab_pages['rate_limit_domain_burst'] = 5000 ``` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). #### Enable TLS connections rate limits by source-IP @@ -1202,7 +1208,7 @@ TLS connection-based rate limits are enforced using the following: gitlab_pages['rate_limit_tls_source_ip_burst'] = 600 ``` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). #### Enable TLS connections rate limits by domain @@ -1215,7 +1221,7 @@ TLS connection-based rate limits are enforced using the following: gitlab_pages['rate_limit_tls_domain_burst'] = 5000 ``` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). ## Related topics diff --git a/doc/administration/pages/source.md b/doc/administration/pages/source.md index b36b87f3b1d..2ee9dd653f0 100644 --- a/doc/administration/pages/source.md +++ b/doc/administration/pages/source.md @@ -483,7 +483,8 @@ The default for the maximum size of unpacked archives per project is 100 MB. To change this value: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Preferences**. 1. Expand **Pages**. 1. Update the value for **Maximum size of pages (MB)**. diff --git a/doc/administration/pages/troubleshooting.md b/doc/administration/pages/troubleshooting.md index 6f00ae074f5..e829943f270 100644 --- a/doc/administration/pages/troubleshooting.md +++ b/doc/administration/pages/troubleshooting.md @@ -170,7 +170,8 @@ Upgrading to an [officially supported operating system](https://about.gitlab.com This problem comes from the permissions of the GitLab Pages OAuth application. To fix it: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Applications > GitLab Pages**. 1. Edit the application. 1. Under **Scopes**, ensure that the `api` scope is selected. @@ -234,7 +235,7 @@ To enable disk access: gitlab_pages['enable_disk'] = true ``` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). ## `httprange: new resource 403` @@ -275,7 +276,7 @@ To do that: gitlab_pages['use_legacy_storage'] = true ``` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). ## GitLab Pages deploy job fails with error "is not a recognized provider" @@ -292,7 +293,7 @@ To fix that: - Configure object storage for your Pages deployments, following the [S3-compatible connection settings](index.md#s3-compatible-connection-settings) guide. - Store your deployments locally, by commenting out that line. -1. Save the changes you made to your `gitlab.rb` file, then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the changes you made to your `gitlab.rb` file, then [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). ## 404 error `The page you're looking for could not be found` diff --git a/doc/administration/polling.md b/doc/administration/polling.md index deb6e89183d..50dbc7467d3 100644 --- a/doc/administration/polling.md +++ b/doc/administration/polling.md @@ -26,7 +26,8 @@ The default value (`1`) is recommended for the majority of GitLab installations. To adjust the polling interval multiplier: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Preferences**. 1. Expand **Polling interval multiplier**. 1. Set a value for the polling interval multiplier. This multiplier is applied to all resources at diff --git a/doc/administration/postgresql/database_load_balancing.md b/doc/administration/postgresql/database_load_balancing.md index d5cf93a135a..cc550dbe389 100644 --- a/doc/administration/postgresql/database_load_balancing.md +++ b/doc/administration/postgresql/database_load_balancing.md @@ -76,22 +76,31 @@ Database Load Balancing can be configured in one of two ways: ### Hosts -To configure a list of hosts, perform these steps on all GitLab Rails (Sidekiq) +<!-- Including the Primary host in Database Load Balancing is now recommended for improved performance - Approved by the Reference Architecture and Database groups. --> + +To configure a list of hosts, perform these steps on all GitLab Rails and Sidekiq nodes for each environment you want to balance: 1. Edit the `/etc/gitlab/gitlab.rb` file. -1. In `gitlab_rails['db_load_balancing']`, create an array of the read-only - replicas you want to balance. Do not add the primary host. For example, on +1. In `gitlab_rails['db_load_balancing']`, create the array of the database + hosts you want to balance. For example, on an environment with PostgreSQL running on the hosts `primary.example.com`, - `host1.example.com`, `host2.example.com`, and `host3.example.com`: + `secondary1.example.com`, `secondary2.example.com`: ```ruby - gitlab_rails['db_load_balancing'] = { 'hosts' => ['host1.example.com', 'host2.example.com', `host3.example.com`] } + gitlab_rails['db_load_balancing'] = { 'hosts' => ['primary.example.com', 'secondary1.example.com', 'secondary2.example.com'] } ``` - These replicas must be reachable on the same port configured with `gitlab_rails['db_port']`. + These hosts must be reachable on the same port configured with `gitlab_rails['db_port']`. + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +NOTE: +Adding the primary to the hosts list is optional, but recommended. +This makes the primary eligible for load-balanced read queries, improving system performance +when the primary has capacity for these queries. +Very high-traffic instances may not have capacity on the primary for it to serve as a read replica. +The primary will be used for write queries whether or not it is present in this list. ### Service Discovery @@ -121,7 +130,7 @@ record. For example: } ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. | Option | Description | Default | |----------------------|---------------------------------------------------------------------------------------------------|-----------| @@ -173,7 +182,7 @@ To configure these options with a hosts list, use the following example: ```ruby gitlab_rails['db_load_balancing'] = { - 'hosts' => ['host1.example.com', 'host2.example.com', `host3.example.com`] + 'hosts' => ['primary.example.com', 'secondary1.example.com', 'secondary2.example.com'] 'max_replication_difference' => 16777216 # 16 MB 'max_replication_lag_time' => 30 'replica_check_interval' => 30 diff --git a/doc/administration/postgresql/multiple_databases.md b/doc/administration/postgresql/multiple_databases.md index 857fd4fc9c5..f03781d0ee2 100644 --- a/doc/administration/postgresql/multiple_databases.md +++ b/doc/administration/postgresql/multiple_databases.md @@ -15,10 +15,10 @@ By default, GitLab uses a single application database, referred to as the `main` To scale GitLab, you can configure GitLab to use multiple application databases. -Due to [known issues](#known-issues), configuring GitLab with multiple databases is an [Experiment](../../policy/alpha-beta-support.md#experiment). +Due to [known issues](#known-issues), configuring GitLab with multiple databases is an [Experiment](../../policy/experiment-beta-support.md#experiment). After you have set up multiple databases, GitLab uses a second application database for -[CI/CD features](../../ci/index.md), referred to as the `ci` database. +[CI/CD features](../../ci/index.md), referred to as the `ci` database. We do not exclude hosting both databases on a single PostgreSQL instance. All tables have exactly the same structure in both the `main`, and `ci` databases. Some examples: diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index 8ac3ac40a75..392f9f2b89c 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -328,10 +328,10 @@ consul['configuration'] = { All database nodes use the same configuration. The leader node is not determined in configuration, and there is no additional or different configuration for either leader or replica nodes. -After the configuration of a node is complete, you must [reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) +After the configuration of a node is complete, you must [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) on each node for the changes to take effect. -Generally, when Consul cluster is ready, the first node that [reconfigures](../restart_gitlab.md#omnibus-gitlab-reconfigure) +Generally, when Consul cluster is ready, the first node that [reconfigures](../restart_gitlab.md#reconfigure-a-linux-package-installation) becomes the leader. You do not need to sequence the nodes reconfiguration. You can run them in parallel or in any order. If you choose an arbitrary order, you do not have any predetermined leader. @@ -552,7 +552,7 @@ attributes set, but the following need to be set. gitlab_rails['db_load_balancing'] = { 'hosts' => ['POSTGRESQL_NODE_1', 'POSTGRESQL_NODE_2', 'POSTGRESQL_NODE_3'] } ``` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. #### Application node post-configuration @@ -625,7 +625,7 @@ consul['configuration'] = { consul['monitoring_service_discovery'] = true ``` -[Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +[Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. #### Example recommended setup for PgBouncer servers @@ -654,7 +654,7 @@ consul['configuration'] = { consul['monitoring_service_discovery'] = true ``` -[Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +[Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. #### Internal load balancer setup @@ -703,7 +703,7 @@ consul['configuration'] = { consul['monitoring_service_discovery'] = true ``` -[Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +[Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. #### Example recommended setup manual steps @@ -979,7 +979,7 @@ You can switch an exiting database cluster to use Patroni instead of repmgr with 1. On the primary node, [configure Patroni](#configuring-patroni-cluster). Remove `repmgr` and any other repmgr-specific configuration. Also remove any configuration that is related to PostgreSQL replication. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) on the primary node. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) on the primary node. It makes it the leader. You can check this with: ```shell @@ -1076,6 +1076,10 @@ Considering these, you should carefully plan your PostgreSQL upgrade: sudo gitlab-ctl pg-upgrade -V 13 ``` +If issues are encountered upgrading the replicas, +[there is a troubleshooting section](#postgresql-major-version-upgrade-fails-on-a-patroni-replica) +that might be the solution. + NOTE: Reverting the PostgreSQL upgrade with `gitlab-ctl revert-pg-upgrade` has the same considerations as `gitlab-ctl pg-upgrade`. You should follow the same procedure by first stopping the replicas, @@ -1157,7 +1161,7 @@ In addition to the common configuration, you must apply the following in `gitlab ``` 1. Make sure that Consul agents don't mix PostgreSQL services offered by the existing and the new Patroni - clusters. For this purpose, you must use an internal attribute that is currently undocumented: + clusters. For this purpose, you must use an internal attribute: ```ruby consul['internal']['postgresql_service_name'] = 'postgresql_new' @@ -1301,7 +1305,7 @@ To fix the problem, add the IP address to `/etc/gitlab/gitlab.rb`. postgresql['trust_auth_cidr_addresses'] = %w(123.123.123.123/32 <other_cidrs>) ``` -[Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +[Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. ### Reinitialize a replica @@ -1329,7 +1333,7 @@ If a replica cannot start or rejoin the cluster, or when it lags behind and cann This can be run on any Patroni node, but be aware that `sudo gitlab-ctl patroni reinitialize-replica` without `--member` restarts the server it is run on. - It is recommended to run it locally on the broken server to reduce the risk of + You should run it locally on the broken server to reduce the risk of unintended data loss. 1. Monitor the logs: @@ -1340,15 +1344,42 @@ If a replica cannot start or rejoin the cluster, or when it lags behind and cann ### Reset the Patroni state in Consul WARNING: -This is a destructive process and may lead the cluster into a bad state. Make sure that you have a healthy backup before running this process. +Resetting the Patroni state in Consul is a potentially destructive process. Make sure that you have a healthy database backup first. + +As a last resort you can reset the Patroni state in Consul completely. + +This may be required if your Patroni cluster is in an unknown or bad state and no node can start: + +```plaintext ++ Cluster: postgresql-ha (6970678148837286213) ------+---------+---------+----+-----------+ +| Member | Host | Role | State | TL | Lag in MB | ++-------------------------------------+--------------+---------+---------+----+-----------+ +| gitlab-database-1.example.com | 172.18.0.111 | Replica | stopped | | unknown | +| gitlab-database-2.example.com | 172.18.0.112 | Replica | stopped | | unknown | +| gitlab-database-3.example.com | 172.18.0.113 | Replica | stopped | | unknown | ++-------------------------------------+--------------+---------+---------+----+-----------+ +``` -As a last resort, if your Patroni cluster is in an unknown or bad state and no node can start, you can -reset the Patroni state in Consul completely, resulting in a reinitialized Patroni cluster when +**Before deleting the Patroni state in Consul**, +[try and resolve the `gitlab-ctl` errors](#errors-running-gitlab-ctl) on the Patroni nodes. + +This process results in a reinitialized Patroni cluster when the first Patroni node starts. To reset the Patroni state in Consul: -1. Take note of the Patroni node that was the leader, or that the application thinks is the current leader, if the current state shows more than one, or none. One way to do this is to look on the PgBouncer nodes in `/var/opt/gitlab/consul/databases.ini`, which contains the hostname of the current leader. +1. Take note of the Patroni node that was the leader, or that the application thinks is the current leader, + if the current state shows more than one, or none: + - Look on the PgBouncer nodes in `/var/opt/gitlab/consul/databases.ini`, + which contains the hostname of the current leader. + - Look in the Patroni logs `/var/log/gitlab/patroni/current` (or the older rotated and + compressed logs `/var/log/gitlab/patroni/@40000*`) on **all** database nodes to see + which server was most recently identified as the leader by the cluster: + + ```plaintext + INFO: no action. I am a secondary (database1.local) and following a leader (database2.local) + ``` + 1. Stop Patroni on all nodes: ```shell @@ -1395,7 +1426,7 @@ To fix the problem, ensure the loopback interface is included in the CIDR addres postgresql['trust_auth_cidr_addresses'] = %w(<other_cidrs> 127.0.0.1/32) ``` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Check that [all the replicas are synchronized](#check-replication-status) ### Errors in Patroni logs: the requested start point is ahead of the Write Ahead Log (WAL) flush position @@ -1435,10 +1466,243 @@ Workarounds: - If set to enforcing, SELinux may also prevent these operations. Verify the issue is fixed by setting SELinux to permissive. -Patroni has been shipping with Omnibus GitLab since 13.1, along with a build of Python 3.7. -Workarounds should stop being required when GitLab 14.x starts shipping with -[a later version of Python](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6164) as -the code which causes this was removed from Python 3.8. +Patroni first shipped in Omnibus GitLab 13.1, along with a build of Python 3.7. +The code which causes this was removed in Python 3.8: this fix shipped in +[Omnibus GitLab 14.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5547) +and later, removing the need for a workaround. + +### Errors running `gitlab-ctl` + +Patroni nodes can get into a state where `gitlab-ctl` commands fail +and `gitlab-ctl reconfigure` cannot fix the node. + +If this co-incides with a version upgrade of PostgreSQL, [follow a different procedure](#postgresql-major-version-upgrade-fails-on-a-patroni-replica) + +One common symptom is that `gitlab-ctl` cannot determine +information it needs about the installation if the database server is failing to start: + +```plaintext +Malformed configuration JSON file found at /opt/gitlab/embedded/nodes/<HOSTNAME>.json. +This usually happens when your last run of `gitlab-ctl reconfigure` didn't complete successfully. +``` + +```plaintext +Error while reinitializing replica on the current node: Attributes not found in +/opt/gitlab/embedded/nodes/<HOSTNAME>.json, has reconfigure been run yet? +``` + +Similarly, the nodes file (`/opt/gitlab/embedded/nodes/<HOSTNAME>.json`) should contain a lot of information, +but might get created with only: + +```json +{ + "name": "<HOSTNAME>" +} +``` + +The following process for fixing this includes reinitializing this replica: +the current state of PostgreSQL on this node is discarded: + +1. Shut down the Patroni and (if present) PostgreSQL services: + + ```shell + sudo gitlab-ctl status + sudo gitlab-ctl stop patroni + sudo gitlab-ctl stop postgresql + ``` + +1. Remove `/var/opt/gitlab/postgresql/data` in case its state prevents + PostgreSQL from starting: + + ```shell + cd /var/opt/gitlab/postgresql + sudo rm -rf data + ``` + + **Take care with this step to avoid data loss**. + This step can be also achieved by renaming `data/`: + make sure there's enough free disk for a new copy of the primary database, + and remove the extra directory when the replica is fixed. + +1. With PostgreSQL not running, the nodes file now gets created successfully: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. Start Patroni: + + ```shell + sudo gitlab-ctl start patroni + ``` + +1. Monitor the logs and check the cluster state: + + ```shell + sudo gitlab-ctl tail patroni + sudo gitlab-ctl patroni members + ``` + +1. Re-run `reconfigure` again: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. Reinitialize the replica if `gitlab-ctl patroni members` indicates this is needed: + + ```shell + sudo gitlab-ctl patroni reinitialize-replica + ``` + +If this procedure doesn't work **and** if the cluster is unable to elect a leader, +[there is a another fix](#reset-the-patroni-state-in-consul) which should only be +used as a last resort. + +### PostgreSQL major version upgrade fails on a Patroni replica + +A Patroni **replica** can get stuck in a loop during `gitlab-ctl pg-upgrade`, and +the upgrade fails. + +An example set of symptoms is as follows: + +1. A `postgresql` service is defined, + which shouldn't usually be present on a Patroni node. It is present because + `gitlab-ctl pg-upgrade` adds it to create a new empty database: + + ```plaintext + run: patroni: (pid 1972) 1919s; run: log: (pid 1971) 1919s + down: postgresql: 1s, normally up, want up; run: log: (pid 1973) 1919s + ``` + +1. PostgreSQL generates `PANIC` log entries in + `/var/log/gitlab/postgresql/current` as Patroni is removing + `/var/opt/gitlab/postgresql/data` as part of reinitializing the replica: + + ```plaintext + DETAIL: Could not open file "pg_xact/0000": No such file or directory. + WARNING: terminating connection because of crash of another server process + LOG: all server processes terminated; reinitializing + PANIC: could not open file "global/pg_control": No such file or directory + ``` + +1. In `/var/log/gitlab/patroni/current`, Patroni logs the following. + The local PostgreSQL version is different from the cluster leader: + + ```plaintext + INFO: trying to bootstrap from leader 'HOSTNAME' + pg_basebackup: incompatible server version 12.6 + pg_basebackup: removing data directory "/var/opt/gitlab/postgresql/data" + ERROR: Error when fetching backup: pg_basebackup exited with code=1 + ``` + +**Important**: This workaround applies when the Patroni cluster is in the following state: + +- The [leader has been successfully upgraded to the new major version](#upgrading-postgresql-major-version-in-a-patroni-cluster). +- The step to upgrade PostgreSQL on replicas is failing. + +This workaround completes the PostgreSQL upgrade on a Patroni replica +by setting the node to use the new PostgreSQL version, and then reinitializing +it as a replica in the new cluster that was created +when the leader was upgraded: + +1. Check the cluster status on all nodes to confirm which is the leader + and what state the replicas are in + + ```shell + sudo gitlab-ctl patroni members + ``` + +1. Replica: check which version of PostgreSQL is active: + + ```shell + sudo ls -al /opt/gitlab/embedded/bin | grep postgres + ``` + +1. Replica: ensure the nodes file is correct and `gitlab-ctl` can run. This resolves + the [errors running `gitlab-ctl`](#errors-running-gitlab-ctl) issue if the replica + has any of those errors as well: + + ```shell + sudo gitlab-ctl stop patroni + sudo gitlab-ctl reconfigure + ``` + +1. Replica: relink the PostgreSQL binaries to the required version + to fix the `incompatible server version` error: + + 1. Edit `/etc/gitlab/gitlab.rb` and specify the required version: + + ```ruby + postgresql['version'] = 13 + ``` + + 1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + + 1. Check the binaries are relinked. The binaries distributed for + PostgreSQL vary between major releases, it's typical to + have a small number of incorrect symbolic links: + + ```shell + sudo ls -al /opt/gitlab/embedded/bin | grep postgres + ``` + +1. Replica: ensure PostgreSQL is fully reinitialized for the specified version: + + ```shell + cd /var/opt/gitlab/postgresql + sudo rm -rf data + sudo gitlab-ctl reconfigure + ``` + +1. Replica: optionally monitor the database in two additional terminal sessions: + + - Disk use increases as `pg_basebackup` runs. Track progress of the + replica initialization with: + + ```shell + cd /var/opt/gitlab/postgresql + watch du -sh data + ``` + + - Monitor the process in the logs: + + ```shell + sudo gitlab-ctl tail patroni + ``` + +1. Replica: Start Patroni to reinitialize the replica: + + ```shell + sudo gitlab-ctl start patroni + ``` + +1. Replica: After it completes, remove the hardcoded version from `/etc/gitlab/gitlab.rb`: + + 1. Edit `/etc/gitlab/gitlab.rb` and remove `postgresql['version']`. + 1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + + 1. Check the correct binaries are linked: + + ```shell + sudo ls -al /opt/gitlab/embedded/bin | grep postgres + ``` + +1. Check the cluster status on all nodes: + + ```shell + sudo gitlab-ctl patroni members + ``` + +Repeat this procedure on the other replica if required. ### Issues with other components diff --git a/doc/administration/postgresql/standalone.md b/doc/administration/postgresql/standalone.md index 77ff9fd2fe1..d00310ecee0 100644 --- a/doc/administration/postgresql/standalone.md +++ b/doc/administration/postgresql/standalone.md @@ -57,7 +57,7 @@ together with Omnibus GitLab. This is recommended as part of our gitlab_rails['auto_migrate'] = false ``` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Note the PostgreSQL node's IP address or hostname, port, and plain text password. These are necessary when configuring the GitLab application servers later. diff --git a/doc/administration/raketasks/check.md b/doc/administration/raketasks/check.md index 2cb664b0859..e55a0f1c8a7 100644 --- a/doc/administration/raketasks/check.md +++ b/doc/administration/raketasks/check.md @@ -121,12 +121,14 @@ Integrity checks are supported for the following types of file: - CI artifacts (introduced in GitLab 10.7.0) - LFS objects (introduced in GitLab 10.6.0) +- Project-level Secure Files (introduced in GitLab 16.1.0) - User uploads (introduced in GitLab 10.6.0) **Omnibus Installation** ```shell sudo gitlab-rake gitlab:artifacts:check +sudo gitlab-rake gitlab:ci_secure_files:check sudo gitlab-rake gitlab:lfs:check sudo gitlab-rake gitlab:uploads:check ``` @@ -135,6 +137,7 @@ sudo gitlab-rake gitlab:uploads:check ```shell sudo -u git -H bundle exec rake gitlab:artifacts:check RAILS_ENV=production +sudo -u git -H bundle exec rake gitlab:ci_secure_files:check RAILS_ENV=production sudo -u git -H bundle exec rake gitlab:lfs:check RAILS_ENV=production sudo -u git -H bundle exec rake gitlab:uploads:check RAILS_ENV=production ``` @@ -151,6 +154,7 @@ Variable | Type | Description ```shell sudo gitlab-rake gitlab:artifacts:check BATCH=100 ID_FROM=50 ID_TO=250 +sudo gitlab-rake gitlab:ci_secure_files:check BATCH=100 ID_FROM=50 ID_TO=250 sudo gitlab-rake gitlab:lfs:check BATCH=100 ID_FROM=50 ID_TO=250 sudo gitlab-rake gitlab:uploads:check BATCH=100 ID_FROM=50 ID_TO=250 ``` @@ -415,3 +419,37 @@ To update these references to point to local storage: ``` The script to [delete references to missing artifacts](check.md#delete-references-to-missing-artifacts) now functions correctly and cleans up the database. + +### Delete references to missing secure files + +`VERBOSE=1 gitlab-rake gitlab:ci_secure_files:check` detects when secure files: + +- Are deleted outside of GitLab. +- Have references still in the GitLab database. + +When this scenario is detected, the Rake task displays an error message. For example: + +```shell +Checking integrity of CI Secure Files +- 1..15: Failures: 2 + - Job SecureFile: 9: #<Errno::ENOENT: No such file or directory @ rb_sysopen - /var/opt/gitlab/gitlab-rails/shared/ci_secure_files/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a/2022_06_30/8/9/distribution.cer> + - Job SecureFile: 15: Remote object does not exist +Done! + +``` + +To delete these references to missing local or remote secure files: + +1. Open the [GitLab Rails Console](../operations/rails_console.md#starting-a-rails-console-session). +1. Run the following Ruby code: + + ```ruby + secure_files_deleted = 0 + ::Ci::SecureFile.find_each do |secure_file| ### Iterate secure files + next if secure_file.file.file.exists? ### Skip if the file reference is valid + secure_files_deleted += 1 + puts "#{secure_file.id} #{secure_file.file.path} is missing." ### Allow verification before destroy + # secure_file.destroy! ### Uncomment to actually destroy + end + puts "Count of identified/destroyed invalid references: #{secure_files_deleted}" + ``` diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md index 06f7203f695..50c4b004f9c 100644 --- a/doc/administration/raketasks/maintenance.md +++ b/doc/administration/raketasks/maintenance.md @@ -350,37 +350,54 @@ status in the output of the `sudo gitlab-rake db:migrate:status` command. sudo gitlab-ctl restart sidekiq ``` -## Rebuild database indexes +## Rebuild database indexes (Experiment) WARNING: -This is an experimental feature that isn't enabled by default. It requires PostgreSQL 12 or later. +This feature is experimental, and isn't enabled by default. Use caution when +running in a production environment, and run during off-peak times. -Database indexes can be rebuilt regularly to reclaim space and maintain healthy levels of index bloat over time. +Database indexes can be rebuilt regularly to reclaim space and maintain healthy +levels of index bloat over time. Reindexing can also be run as a +[regular cron job](https://docs.gitlab.com/omnibus/settings/database.html#automatic-database-reindexing). +A "healthy" level of bloat is highly dependent on the specific index, but generally +should be below 30%. -To rebuild the two indexes with the highest estimated bloat, use the following Rake task: +Prerequisites: -```shell -sudo gitlab-rake gitlab:db:reindex -``` +- This feature requires PostgreSQL 12 or later. +- These index types are not supported: expression indexes, partitioned indexes, + and indexes used for constraint exclusion. -To target a specific index, use the following Rake task: +To manually rebuild a database index: -```shell -sudo gitlab-rake gitlab:db:reindex['public.a_specific_index'] -``` +1. Optional. To send annotations to a Grafana (4.6 or later) endpoint, enable annotations + with these custom environment variables (see [setting custom environment variables](https://docs.gitlab.com/omnibus/settings/environment-variables.html)): + + 1. `GRAFANA_API_URL`: The base URL for Grafana, such as `http://some-host:3000`. + 1. `GRAFANA_API_KEY`: A Grafana API key with at least `Editor role`. + +1. Run the Rake task to rebuild the two indexes with the highest estimated bloat: + + ```shell + sudo gitlab-rake gitlab:db:reindex + ``` -The following index types are not supported: +1. The reindexing task (`gitlab:db:reindex`) rebuilds only the two indexes in each database + with the highest bloat. To rebuild more than two indexes, run the task again + until all desired indexes have been rebuilt. -1. Indexes used for constraint exclusion -1. Partitioned indexes -1. Expression indexes +### Notes -Optionally, this Rake task sends annotations to a Grafana (4.6 or later) endpoint. Use the following custom environment variables to enable annotations: +- Rebuilding database indexes is a disk-intensive task, so you should perform the +task during off-peak hours. Running the task during peak hours can lead to +_increased_ bloat, and can also cause certain queries to perform slowly. -1. `GRAFANA_API_URL` - The base URL for Grafana, for example `http://some-host:3000`. -1. `GRAFANA_API_KEY` - Grafana API key with at least `Editor role`. +- The task requires free disk space for the index being restored. The created +indexes are appended with `_ccnew`. If the reindexing task fails, re-running the +task cleans up the temporary indexes. -You can also [enable reindexing as a regular cron job](https://docs.gitlab.com/omnibus/settings/database.html#automatic-database-reindexing). +- The time it takes for database index rebuilding to complete depends on the size +of the target database. It can take between several hours and several days. ## Import common metrics diff --git a/doc/administration/raketasks/praefect.md b/doc/administration/raketasks/praefect.md index d1dbc83ad86..1f9eb06f567 100644 --- a/doc/administration/raketasks/praefect.md +++ b/doc/administration/raketasks/praefect.md @@ -18,6 +18,8 @@ Rake tasks are available for projects that have been created on Praefect storage - The primary Gitaly node. - Secondary internal Gitaly nodes. +Run this Rake task on the node that GitLab is installed and not on the node that Praefect is installed. + **Omnibus Installation** ```shell diff --git a/doc/administration/raketasks/storage.md b/doc/administration/raketasks/storage.md index d740aaa9c96..3664a79bf43 100644 --- a/doc/administration/raketasks/storage.md +++ b/doc/administration/raketasks/storage.md @@ -109,7 +109,8 @@ sudo gitlab-rake gitlab:storage:migrate_to_hashed ID_FROM=50 ID_TO=100 To monitor the progress in GitLab: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. Watch how long the `hashed_storage:hashed_storage_project_migrate` queue takes to finish. After it reaches zero, you can confirm every project diff --git a/doc/administration/redis/replication_and_failover.md b/doc/administration/redis/replication_and_failover.md index a37794ec44b..4a6f58a8d6a 100644 --- a/doc/administration/redis/replication_and_failover.md +++ b/doc/administration/redis/replication_and_failover.md @@ -275,7 +275,7 @@ If you fail to replicate first, you may loose data (unprocessed background jobs) gitlab_rails['auto_migrate'] = false ``` -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. NOTE: You can specify multiple roles like sentinel and Redis as: @@ -332,7 +332,7 @@ Read more about [roles](https://docs.gitlab.com/omnibus/roles/). gitlab_rails['auto_migrate'] = false ``` -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Go through the steps again for all the other replica nodes. NOTE: @@ -348,11 +348,7 @@ the same Sentinels. ### Step 3. Configuring the Redis Sentinel instances NOTE: -If you are using an external Redis Sentinel instance, be sure -to exclude the `requirepass` parameter from the Sentinel -configuration. This parameter causes clients to report `NOAUTH -Authentication required.`. -[Redis Sentinel 3.2.x does not support password authentication](https://github.com/antirez/redis/issues/3279). +[Support for Sentinel password authentication](https://gitlab.com/gitlab-org/gitlab/-/issues/235938) was introduced in GitLab 16.1. Now that the Redis servers are all set up, let's configure the Sentinel servers. @@ -408,6 +404,9 @@ multiple machines with the Sentinel daemon. ## Configure Sentinel sentinel['bind'] = '10.0.0.1' + ## Optional password for Sentinel authentication. Defaults to no password required. + # sentinel['password'] = 'sentinel-password-goes here' + # Port that Sentinel listens on, uncomment to change to non default. Defaults # to `26379`. # sentinel['port'] = 26379 @@ -458,7 +457,7 @@ multiple machines with the Sentinel daemon. Only the primary GitLab application server should handle migrations. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Go through the steps again for all the other Sentinel nodes. ### Step 4. Configuring the GitLab application @@ -493,9 +492,10 @@ which ideally should not have Redis or Sentinels on it for a HA setup. {'host' => '10.0.0.2', 'port' => 26379}, {'host' => '10.0.0.3', 'port' => 26379} ] + # gitlab_rails['redis_sentinels_password'] = 'sentinel-password-goes-here' # uncomment and set it to the same value as in sentinel['password'] ``` -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. ### Step 5. Enable Monitoring @@ -570,13 +570,14 @@ redis['master_password'] = 'redis-password-goes-here' # the same value defined i redis['master_ip'] = '10.0.0.1' # ip of the initial primary redis instance #redis['master_port'] = 6379 # port of the initial primary redis instance, uncomment to change to non default sentinel['bind'] = '10.0.0.1' +# sentinel['password'] = 'sentinel-password-goes-here' # must be the same in every sentinel node, uncomment to set a password # sentinel['port'] = 26379 # uncomment to change default port sentinel['quorum'] = 2 # sentinel['down_after_milliseconds'] = 10000 # sentinel['failover_timeout'] = 60000 ``` -[Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +[Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. ### Example configuration for Redis replica 1 and Sentinel 2 @@ -592,13 +593,14 @@ redis['master_ip'] = '10.0.0.1' # IP of primary Redis server #redis['master_port'] = 6379 # Port of primary Redis server, uncomment to change to non default redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node sentinel['bind'] = '10.0.0.2' +# sentinel['password'] = 'sentinel-password-goes-here' # must be the same in every sentinel node, uncomment to set a password # sentinel['port'] = 26379 # uncomment to change default port sentinel['quorum'] = 2 # sentinel['down_after_milliseconds'] = 10000 # sentinel['failover_timeout'] = 60000 ``` -[Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +[Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. ### Example configuration for Redis replica 2 and Sentinel 3 @@ -614,13 +616,14 @@ redis['master_ip'] = '10.0.0.1' # IP of primary Redis server #redis['master_port'] = 6379 # Port of primary Redis server, uncomment to change to non default redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node sentinel['bind'] = '10.0.0.3' +# sentinel['password'] = 'sentinel-password-goes-here' # must be the same in every sentinel node, uncomment to set a password # sentinel['port'] = 26379 # uncomment to change default port sentinel['quorum'] = 2 # sentinel['down_after_milliseconds'] = 10000 # sentinel['failover_timeout'] = 60000 ``` -[Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +[Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. ### Example configuration for the GitLab application @@ -634,9 +637,10 @@ gitlab_rails['redis_sentinels'] = [ {'host' => '10.0.0.2', 'port' => 26379}, {'host' => '10.0.0.3', 'port' => 26379} ] +# gitlab_rails['redis_sentinels_password'] = 'sentinel-password-goes-here' # uncomment and set it to the same value as in sentinel['password'] ``` -[Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +[Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. ## Advanced configuration @@ -657,6 +661,7 @@ persistence classes. | `trace_chunks` | Store [CI trace chunks](../job_logs.md#enable-or-disable-incremental-logging) data. | | `rate_limiting` | Store [rate limiting](../../user/admin_area/settings/user_and_ip_rate_limits.md) state. | | `sessions` | Store [sessions](../../../ee/development/session.md#gitlabsession). | +| `repository_cache` | Store cache data specific to repositories. | To make this work with Sentinel: @@ -671,6 +676,7 @@ To make this work with Sentinel: gitlab_rails['redis_trace_chunks_instance'] = REDIS_TRACE_CHUNKS_URL gitlab_rails['redis_rate_limiting_instance'] = REDIS_RATE_LIMITING_URL gitlab_rails['redis_sessions_instance'] = REDIS_SESSIONS_URL + gitlab_rails['redis_repository_cache_instance'] = REDIS_REPOSITORY_CACHE_URL # Configure the Sentinels gitlab_rails['redis_cache_sentinels'] = [ @@ -701,6 +707,19 @@ To make this work with Sentinel: { host: SESSIONS_SENTINEL_HOST, port: 26379 }, { host: SESSIONS_SENTINEL_HOST2, port: 26379 } ] + gitlab_rails['redis_repository_cache_sentinels'] = [ + { host: REPOSITORY_CACHE_SENTINEL_HOST, port: 26379 }, + { host: REPOSITORY_CACHE_SENTINEL_HOST2, port: 26379 } + ] + + # gitlab_rails['redis_cache_sentinels_password'] = 'sentinel-password-goes-here' + # gitlab_rails['redis_queues_sentinels_password'] = 'sentinel-password-goes-here' + # gitlab_rails['redis_shared_state_sentinels_password'] = 'sentinel-password-goes-here' + # gitlab_rails['redis_actioncable_sentinels_password'] = 'sentinel-password-goes-here' + # gitlab_rails['redis_trace_chunks_sentinels_password'] = 'sentinel-password-goes-here' + # gitlab_rails['redis_rate_limiting_sentinels_password'] = 'sentinel-password-goes-here' + # gitlab_rails['redis_sessions_sentinels_password'] = 'sentinel-password-goes-here' + # gitlab_rails['redis_repository_cache_sentinels_password'] = 'sentinel-password-goes-here' ``` - Redis URLs should be in the format: `redis://:PASSWORD@SENTINEL_PRIMARY_NAME`, where: diff --git a/doc/administration/redis/standalone.md b/doc/administration/redis/standalone.md index c36d75a566d..703601cda42 100644 --- a/doc/administration/redis/standalone.md +++ b/doc/administration/redis/standalone.md @@ -42,7 +42,7 @@ Omnibus GitLab: gitlab_rails['auto_migrate'] = false ``` -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Note the Redis node's IP address or hostname, port, and Redis password. These are necessary when [configuring the GitLab application servers](#set-up-the-gitlab-rails-application-instance). @@ -68,7 +68,7 @@ On the instance where GitLab is installed: 1. Save your changes to `/etc/gitlab/gitlab.rb`. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. ## Troubleshooting diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index ba4c5fd9046..c3cf7c599a3 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -38,17 +38,11 @@ full list of reference architectures, see <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. - - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo. - - Note that [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported. - - [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440). - - Consul is primarily used for Omnibus PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. -2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Memorystore](https://cloud.google.com/memorystore) and [Amazon ElastiCache](https://aws.amazon.com/elasticache/) are known to work. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance) and [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Provide your own Redis instance](#provide-your-own-redis-instance) and [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. + - Redis is primarily single threaded. It's strongly recommended separating out the instances as specified into Cache and Persistent data to achieve optimum performance at this scale. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. -4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. +4. Should be run on reputable Cloud Provider or Self Managed solutions. See [Configure the object storage](#configure-the-object-storage) for more information. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. 6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to @@ -387,9 +381,9 @@ backend pgbouncer mode tcp option tcp-check - server pgbouncer1 10.6.0.21:6432 check - server pgbouncer2 10.6.0.22:6432 check - server pgbouncer3 10.6.0.23:6432 check + server pgbouncer1 10.6.0.31:6432 check + server pgbouncer2 10.6.0.32:6432 check + server pgbouncer3 10.6.0.33:6432 check backend praefect mode tcp @@ -462,7 +456,7 @@ To configure Consul: 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Go through the steps again for all the other Consul nodes, and make sure you set up the correct IPs. @@ -504,24 +498,21 @@ cluster to be used with GitLab. ### Provide your own PostgreSQL instance -If you're hosting GitLab on a cloud provider, you can optionally use a -managed service for PostgreSQL. +You can optionally use a [third party external service for PostgreSQL](../../administration/postgresql/external.md). -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default from [14.4.0](../../update/index.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. -If you use a cloud-managed service, or provide your own PostgreSQL: +If you use a third party external service: +1. Note that the HA Omnibus PostgreSQL setup encompasses PostgreSQL, PgBouncer and Consul. All of these components would no longer be required when using a third party external service. 1. Set up PostgreSQL according to the [database requirements document](../../install/requirements.md#database). 1. Set up a `gitlab` username with a password of your choice. The `gitlab` user needs privileges to create the `gitlabhq_production` database. 1. Configure the GitLab application servers with the appropriate details. This step is covered in [Configuring the GitLab Rails application](#configure-gitlab-rails). -1. For improved performance, configuring [Database Load Balancing](../postgresql/database_load_balancing.md) - with multiple read replicas is recommended. - -See [Configure GitLab using an external PostgreSQL service](../postgresql/external.md) for -further configuration steps. +1. The number of nodes required to achieve HA may differ depending on the service compared to Omnibus and doesn't need to match accordingly. +1. However, if [Database Load Balancing](../postgresql/database_load_balancing.md) via Read Replicas is desired for further improved performance it's recommended to follow the node count for the Reference Architecture. ### Standalone PostgreSQL using Omnibus GitLab @@ -665,7 +656,7 @@ For more information, see the various [Patroni replication methods](../postgresq 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/database.html) are supported and can be added if needed. @@ -754,7 +745,7 @@ The following IPs will be used as an example: 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. If an error `execute[generate databases.ini]` occurs, this is due to an existing [known issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4713). @@ -767,7 +758,7 @@ The following IPs will be used as an example: gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul ``` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) once again +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) once again to resolve any potential errors from the previous steps. 1. Ensure each node is talking to the current primary: @@ -811,6 +802,9 @@ start the failover procedure. NOTE: Redis clusters must each be deployed in an odd number of 3 nodes or more. This is to ensure Redis Sentinel can take votes as part of a quorum. This does not apply when configuring Redis externally, such as a cloud provider service. +NOTE: +Redis is primarily single threaded. It's strongly recommended separating out the instances as specified into Cache and Persistent data to achieve optimum performance at this scale. + Redis requires authentication if used with Sentinel. See [Redis Security](https://redis.io/docs/manual/security/) documentation for more information. We recommend using a combination of a Redis password and tight @@ -840,15 +834,11 @@ to be used with GitLab. The following IPs will be used as an example: - `10.6.0.62`: Redis - Persistent Replica 1 - `10.6.0.63`: Redis - Persistent Replica 2 -### Providing your own Redis instance +### Provide your own Redis instance -Managed Redis from cloud providers (such as AWS ElastiCache) will work. If these -services support high availability, be sure it _isn't_ of the Redis Cluster type. +You can optionally use a third party external service for Redis as long as it meets the [requirements](../../install/requirements.md#redis). -Because Omnibus GitLab packages ship with Redis 6.0 or later, Redis 6.0 or later is required. Older Redis versions have reached end-of-life. - -Note the Redis node's IP address or hostname, port, and password (if required). -These will be necessary later when configuring the [GitLab application servers](#configure-gitlab-rails). +A reputable provider or solution should be used for this. [Google Memorystore](https://cloud.google.com/memorystore/docs/redis/redis-overview) and [AWS Elasticache](https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/WhatIs.html) are known to work. However, note that the Redis Cluster mode specifically isn't supported by GitLab. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. ### Configure the Redis Cache cluster @@ -929,7 +919,7 @@ a node and change its status from primary to replica (and vice versa). 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. #### Configure the replica Redis Cache nodes @@ -1002,7 +992,7 @@ a node and change its status from primary to replica (and vice versa). 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. @@ -1055,7 +1045,7 @@ a node and change its status from primary to replica (and vice versa). #redis['master_port'] = 6379 # Set up password authentication for Redis and replicas (use the same password in all nodes). - redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' ## Must be the same in every Redis node @@ -1084,7 +1074,7 @@ a node and change its status from primary to replica (and vice versa). 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. #### Configure the replica Redis Persistent nodes @@ -1147,7 +1137,7 @@ a node and change its status from primary to replica (and vice versa). 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. @@ -1266,7 +1256,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Follow the [post configuration](#praefect-postgresql-post-configuration). @@ -1498,10 +1488,10 @@ Updates to example must be made at: sudo touch /etc/gitlab/skip-auto-reconfigure ``` - 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect and + 1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect and to run the Praefect database migrations. -1. On all other Praefect nodes, [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. On all other Praefect nodes, [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. ### Configure Gitaly @@ -1669,7 +1659,7 @@ Updates to example must be made at: 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). ### Gitaly Cluster TLS support @@ -1725,7 +1715,7 @@ To configure Praefect with TLS: } ``` -1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file and [reconfigure](../restart_gitlab.md#reconfigure-a-linux-package-installation). 1. On the Praefect clients (including each Gitaly server), copy the certificates, or their certificate authority, into `/etc/gitlab/trusted-certs`: @@ -1746,7 +1736,7 @@ To configure Praefect with TLS: }) ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -1903,7 +1893,7 @@ Updates to example must be made at: Only a single designated node should handle migrations as detailed in the [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. NOTE: If you find that the environment's Sidekiq job processing is slow with long queues, @@ -2067,7 +2057,7 @@ On each node perform the following: Only a single designated node should handle migrations as detailed in the [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. [Enable incremental logging](#enable-incremental-logging). 1. Confirm the node can connect to Gitaly: @@ -2178,7 +2168,7 @@ To configure the Monitoring node: nginx['enable'] = true ``` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. In the GitLab UI, set `admin/application_settings/metrics_and_profiling` > Metrics - Grafana to `/-/grafana` to `http[s]://<MONITOR NODE>/-/grafana` @@ -2194,7 +2184,7 @@ To configure the Monitoring node: GitLab supports using an [object storage](../object_storage.md) service for holding numerous types of data. It's recommended over [NFS](../nfs.md) for data objects and in general it's better in larger setups as object storage is typically much more performant, reliable, -and scalable. +and scalable. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. There are two ways of specifying object storage configuration in GitLab: @@ -2309,17 +2299,11 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. - - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo. - - Note that [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported. - - [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440). - - Consul is primarily used for Omnibus PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. -2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Memorystore](https://cloud.google.com/memorystore) and [Amazon ElastiCache](https://aws.amazon.com/elasticache/) are known to work. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance) and [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Provide your own Redis instance](#provide-your-own-redis-instance) and [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. + - Redis is primarily single threaded. It's strongly recommended separating out the instances as specified into Cache and Persistent data to achieve optimum performance at this scale. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. -4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. +4. Should be run on reputable Cloud Provider or Self Managed solutions. See [Configure the object storage](#configure-the-object-storage) for more information. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. 6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 7984b9dd6c7..37571ed5771 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -38,17 +38,11 @@ full list of reference architectures, see <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. - - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo. - - Note that [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported. - - [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440). - - Consul is primarily used for Omnibus PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. -2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Memorystore](https://cloud.google.com/memorystore) and [Amazon ElastiCache](https://aws.amazon.com/elasticache/) are known to work. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance) for more information. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Provide your own Redis instance](#provide-your-own-redis-instance) for more information. + - Redis is primarily single threaded. It's strongly recommended separating out the instances as specified into Cache and Persistent data to achieve optimum performance at this scale. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. -4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. +4. Should be run on reputable Cloud Provider or Self Managed solutions. See [Configure the object storage](#configure-the-object-storage) for more information. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. 6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to @@ -398,9 +392,9 @@ backend pgbouncer mode tcp option tcp-check - server pgbouncer1 10.6.0.21:6432 check - server pgbouncer2 10.6.0.22:6432 check - server pgbouncer3 10.6.0.23:6432 check + server pgbouncer1 10.6.0.31:6432 check + server pgbouncer2 10.6.0.32:6432 check + server pgbouncer3 10.6.0.33:6432 check backend praefect mode tcp @@ -479,7 +473,7 @@ To configure Consul: 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Go through the steps again for all the other Consul nodes, and make sure you set up the correct IPs. @@ -521,24 +515,21 @@ cluster to be used with GitLab. ### Provide your own PostgreSQL instance -If you're hosting GitLab on a cloud provider, you can optionally use a -managed service for PostgreSQL. +You can optionally use a [third party external service for PostgreSQL](../../administration/postgresql/external.md). -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default from [14.4.0](../../update/index.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. -If you use a cloud-managed service, or provide your own PostgreSQL: +If you use a third party external service: +1. Note that the HA Omnibus PostgreSQL setup encompasses PostgreSQL, PgBouncer and Consul. All of these components would no longer be required when using a third party external service. 1. Set up PostgreSQL according to the [database requirements document](../../install/requirements.md#database). 1. Set up a `gitlab` username with a password of your choice. The `gitlab` user needs privileges to create the `gitlabhq_production` database. 1. Configure the GitLab application servers with the appropriate details. This step is covered in [Configuring the GitLab Rails application](#configure-gitlab-rails). -1. For improved performance, configuring [Database Load Balancing](../postgresql/database_load_balancing.md) - with multiple read replicas is recommended. - -See [Configure GitLab using an external PostgreSQL service](../postgresql/external.md) for -further configuration steps. +1. The number of nodes required to achieve HA may differ depending on the service compared to Omnibus and doesn't need to match accordingly. +1. However, if [Database Load Balancing](../postgresql/database_load_balancing.md) via Read Replicas is desired for further improved performance it's recommended to follow the node count for the Reference Architecture. ### Standalone PostgreSQL using Omnibus GitLab @@ -682,7 +673,7 @@ For more information, see the various [Patroni replication methods](../postgresq 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/database.html) are supported and can be added if needed. @@ -771,7 +762,7 @@ The following IPs will be used as an example: 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. If an error `execute[generate databases.ini]` occurs, this is due to an existing [known issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4713). @@ -784,7 +775,7 @@ The following IPs will be used as an example: gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul ``` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) once again +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) once again to resolve any potential errors from the previous steps. 1. Ensure each node is talking to the current primary: @@ -828,6 +819,9 @@ start the failover procedure. NOTE: Redis clusters must each be deployed in an odd number of 3 nodes or more. This is to ensure Redis Sentinel can take votes as part of a quorum. This does not apply when configuring Redis externally, such as a cloud provider service. +NOTE: +Redis is primarily single threaded. It's strongly recommended separating out the instances as specified into Cache and Persistent data to achieve optimum performance at this scale. + Redis requires authentication if used with Sentinel. See [Redis Security](https://redis.io/docs/manual/security/) documentation for more information. We recommend using a combination of a Redis password and tight @@ -857,15 +851,11 @@ to be used with GitLab. The following IPs will be used as an example: - `10.6.0.62`: Redis - Persistent Replica 1 - `10.6.0.63`: Redis - Persistent Replica 2 -### Providing your own Redis instance - -Managed Redis from cloud providers (such as AWS ElastiCache) will work. If these -services support high availability, be sure it _isn't_ of the Redis Cluster type. +### Provide your own Redis instance -Because Omnibus GitLab packages ship with Redis 6.0 or later, Redis 6.0 or later is required. Older Redis versions have reached end-of-life. +You can optionally use a third party external service for Redis as long as it meets the [requirements](../../install/requirements.md#redis). -Note the Redis node's IP address or hostname, port, and password (if required). -These will be necessary later when configuring the [GitLab application servers](#configure-gitlab-rails). +A reputable provider or solution should be used for this. [Google Memorystore](https://cloud.google.com/memorystore/docs/redis/redis-overview) and [AWS Elasticache](https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/WhatIs.html) are known to work. However, note that the Redis Cluster mode specifically isn't supported by GitLab. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. ### Configure the Redis Cache cluster @@ -945,7 +935,7 @@ a node and change its status from primary to replica (and vice versa). 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. #### Configure the replica Redis Cache nodes @@ -1017,7 +1007,7 @@ a node and change its status from primary to replica (and vice versa). 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. @@ -1070,7 +1060,7 @@ a node and change its status from primary to replica (and vice versa). #redis['master_port'] = 6379 # Set up password authentication for Redis and replicas (use the same password in all nodes). - redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' ## Must be the same in every Redis node @@ -1099,7 +1089,7 @@ a node and change its status from primary to replica (and vice versa). 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. #### Configure the replica Redis Persistent nodes @@ -1165,7 +1155,7 @@ a node and change its status from primary to replica (and vice versa). 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. @@ -1285,7 +1275,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Follow the [post configuration](#praefect-postgresql-post-configuration). @@ -1515,10 +1505,10 @@ the file of the same name on this server. If this is the first Omnibus node you sudo touch /etc/gitlab/skip-auto-reconfigure ``` - 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect and + 1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect and to run the Praefect database migrations. -1. On all other Praefect nodes, [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. On all other Praefect nodes, [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. ### Configure Gitaly @@ -1686,7 +1676,7 @@ Updates to example must be made at: 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). ### Gitaly Cluster TLS support @@ -1742,7 +1732,7 @@ To configure Praefect with TLS: } ``` -1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file and [reconfigure](../restart_gitlab.md#reconfigure-a-linux-package-installation). 1. On the Praefect clients (including each Gitaly server), copy the certificates, or their certificate authority, into `/etc/gitlab/trusted-certs`: @@ -1763,7 +1753,7 @@ To configure Praefect with TLS: }) ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -1920,7 +1910,7 @@ Updates to example must be made at: Only a single designated node should handle migrations as detailed in the [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. NOTE: If you find that the environment's Sidekiq job processing is slow with long queues, @@ -2086,7 +2076,7 @@ On each node perform the following: Only a single designated node should handle migrations as detailed in the [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. [Enable incremental logging](#enable-incremental-logging). 1. Confirm the node can connect to Gitaly: @@ -2197,7 +2187,7 @@ To configure the Monitoring node: nginx['enable'] = true ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). 1. In the GitLab UI, set `admin/application_settings/metrics_and_profiling` > Metrics - Grafana to `/-/grafana` to `http[s]://<MONITOR NODE>/-/grafana` @@ -2212,7 +2202,7 @@ To configure the Monitoring node: GitLab supports using an [object storage](../object_storage.md) service for holding numerous types of data. It's recommended over [NFS](../nfs.md) for data objects and in general it's better in larger setups as object storage is typically much more performant, reliable, -and scalable. +and scalable. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. There are two ways of specifying object storage configuration in GitLab: @@ -2327,17 +2317,11 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. - - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo. - - Note that [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported. - - [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440). - - Consul is primarily used for Omnibus PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. -2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Memorystore](https://cloud.google.com/memorystore) and [Amazon ElastiCache](https://aws.amazon.com/elasticache/) are known to work. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance) for more information. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Provide your own Redis instance](#provide-your-own-redis-instance) for more information. + - Redis is primarily single threaded. It's strongly recommended separating out the instances as specified into Cache and Persistent data to achieve optimum performance at this scale. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. -4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. +4. Should be run on reputable Cloud Provider or Self Managed solutions. See [Configure the object storage](#configure-the-object-storage) for more information. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. 6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index 14dc9d26293..455b0fbafd1 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -31,16 +31,10 @@ For a full list of reference architectures, see | Object storage<sup>4</sup> | - | - | - | - | - | <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. - - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo. - - Note that [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported. - - [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and [Azure Database for PostgreSQL](https://azure.microsoft.com/en-gb/products/postgresql/#overview) is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). - - Consul is primarily used for Omnibus PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. -2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Memorystore](https://cloud.google.com/memorystore) and [Amazon ElastiCache](https://aws.amazon.com/elasticache/) are known to work. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance) and [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Provide your own Redis instance](#provide-your-own-redis-instance) and [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. +4. Should be run on reputable Cloud Provider or Self Managed solutions. See [Configure the object storage](#configure-the-object-storage) for more information. 4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. 5. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to @@ -253,23 +247,20 @@ to be used with GitLab. ### Provide your own PostgreSQL instance -If you're hosting GitLab on a cloud provider, you can optionally use a -managed service for PostgreSQL. +You can optionally use a [third party external service for PostgreSQL](../../administration/postgresql/external.md). -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default from [14.4.0](../../update/index.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. -If you use a cloud-managed service, or provide your own PostgreSQL: +If you use a third party external service: +1. Note that the HA Omnibus PostgreSQL setup encompasses PostgreSQL, PgBouncer and Consul. All of these components would no longer be required when using a third party external service. 1. Set up PostgreSQL according to the [database requirements document](../../install/requirements.md#database). -1. Create a `gitlab` username with a password of your choice. The `gitlab` user +1. Set up a `gitlab` username with a password of your choice. The `gitlab` user needs privileges to create the `gitlabhq_production` database. 1. Configure the GitLab application servers with the appropriate details. This step is covered in [Configuring the GitLab Rails application](#configure-gitlab-rails). -See [Configure GitLab using an external PostgreSQL service](../postgresql/external.md) for -further configuration steps. - ### Standalone PostgreSQL using Omnibus GitLab 1. SSH in to the PostgreSQL server. @@ -320,7 +311,7 @@ further configuration steps. 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Note the PostgreSQL node's IP address or hostname, port, and plain text password. These will be necessary when configuring the [GitLab application server](#configure-gitlab-rails) later. @@ -341,14 +332,9 @@ to be used with GitLab. ### Provide your own Redis instance -Because Omnibus GitLab packages ship with Redis 6.0 or later, Redis 6.0 or later is required. Older Redis versions have reached end-of-life. - -Managed Redis from cloud providers such as AWS ElastiCache will work. If these -services support high availability, be sure it is not the Redis Cluster type. +You can optionally use a third party external service for Redis as long as it meets the [requirements](../../install/requirements.md#redis). -Note the Redis node's IP address or hostname, port, and password (if required). -These will be necessary when configuring the -[GitLab application servers](#configure-gitlab-rails) later. +A reputable provider or solution should be used for this. [Google Memorystore](https://cloud.google.com/memorystore/docs/redis/redis-overview) and [AWS Elasticache](https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/WhatIs.html) are known to work. However, note that the Redis Cluster mode specifically isn't supported by GitLab. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. ### Standalone Redis using Omnibus GitLab @@ -384,7 +370,7 @@ Omnibus: 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Note the Redis node's IP address or hostname, port, and Redis password. These will be necessary when @@ -534,7 +520,7 @@ Updates to example must be made at: 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Confirm that Gitaly can perform callbacks to the internal API: - For GitLab 15.3 and later, run `sudo /opt/gitlab/embedded/bin/gitaly check /var/opt/gitlab/gitaly/config.toml`. @@ -597,7 +583,7 @@ To configure Gitaly with TLS: ``` 1. Delete `gitaly['listen_addr']` to allow only encrypted connections. -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -743,7 +729,7 @@ On each node perform the following: Only a single designated node should handle migrations as detailed in the [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. [Enable incremental logging](#enable-incremental-logging). 1. Run `sudo gitlab-rake gitlab:gitaly:check` to confirm the node can connect to Gitaly. @@ -877,7 +863,7 @@ running [Prometheus](../monitoring/prometheus/index.md) and ] ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). 1. In the GitLab UI, set `admin/application_settings/metrics_and_profiling` > Metrics - Grafana to `/-/grafana` to `http[s]://<MONITOR NODE>/-/grafana` @@ -892,7 +878,7 @@ running [Prometheus](../monitoring/prometheus/index.md) and GitLab supports using an [object storage](../object_storage.md) service for holding numerous types of data. It's recommended over [NFS](../nfs.md) for data objects and in general it's better in larger setups as object storage is typically much more performant, reliable, -and scalable. +and scalable. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. There are two ways of specifying object storage configuration in GitLab: @@ -1005,15 +991,9 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. - - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo. - - Note that [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported. - - [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and [Azure Database for PostgreSQL](https://azure.microsoft.com/en-gb/products/postgresql/#overview) is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). - - Consul is primarily used for Omnibus PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. -2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Memorystore](https://cloud.google.com/memorystore) and [Amazon ElastiCache](https://aws.amazon.com/elasticache/) are known to work. -3. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance) and [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Provide your own Redis instance](#provide-your-own-redis-instance) and [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. +3. Should be run on reputable Cloud Provider or Self Managed solutions. See [Configure the object storage](#configure-the-object-storage) for more information. <!-- markdownlint-enable MD029 --> NOTE: diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index 191e5218c43..6a7d9864376 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -47,17 +47,10 @@ For a full list of reference architectures, see <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. - - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo. - - Note that [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported. - - [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440). - - Consul is primarily used for Omnibus PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. -2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Memorystore](https://cloud.google.com/memorystore) and [Amazon ElastiCache](https://aws.amazon.com/elasticache/) are known to work. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance) for more information. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Provide your own Redis instance](#provide-your-own-redis-instance) for more information. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. -4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. +4. Should be run on reputable Cloud Provider or Self Managed solutions. See [Configure the object storage](#configure-the-object-storage) for more information. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. 6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to @@ -187,19 +180,19 @@ connect to each other freely on these addresses. The following list includes descriptions of each server and its assigned IP: - `10.6.0.10`: External Load Balancer -- `10.6.0.61`: Redis Primary -- `10.6.0.62`: Redis Replica 1 -- `10.6.0.63`: Redis Replica 2 - `10.6.0.11`: Consul/Sentinel 1 - `10.6.0.12`: Consul/Sentinel 2 - `10.6.0.13`: Consul/Sentinel 3 -- `10.6.0.31`: PostgreSQL primary -- `10.6.0.32`: PostgreSQL secondary 1 -- `10.6.0.33`: PostgreSQL secondary 2 -- `10.6.0.21`: PgBouncer 1 -- `10.6.0.22`: PgBouncer 2 -- `10.6.0.23`: PgBouncer 3 +- `10.6.0.21`: PostgreSQL primary +- `10.6.0.22`: PostgreSQL secondary 1 +- `10.6.0.23`: PostgreSQL secondary 2 +- `10.6.0.31`: PgBouncer 1 +- `10.6.0.32`: PgBouncer 2 +- `10.6.0.33`: PgBouncer 3 - `10.6.0.20`: Internal Load Balancer +- `10.6.0.61`: Redis Primary +- `10.6.0.62`: Redis Replica 1 +- `10.6.0.63`: Redis Replica 2 - `10.6.0.51`: Gitaly 1 - `10.6.0.52`: Gitaly 2 - `10.6.0.93`: Gitaly 3 @@ -399,9 +392,9 @@ backend pgbouncer mode tcp option tcp-check - server pgbouncer1 10.6.0.21:6432 check - server pgbouncer2 10.6.0.22:6432 check - server pgbouncer3 10.6.0.23:6432 check + server pgbouncer1 10.6.0.31:6432 check + server pgbouncer2 10.6.0.32:6432 check + server pgbouncer3 10.6.0.33:6432 check backend praefect mode tcp @@ -435,6 +428,9 @@ Using [Redis](https://redis.io/) in scalable environment is possible using a **P topology with a [Redis Sentinel](https://redis.io/docs/manual/sentinel/) service to watch and automatically start the failover procedure. +NOTE: +Redis clusters must each be deployed in an odd number of 3 nodes or more. This is to ensure Redis Sentinel can take votes as part of a quorum. This does not apply when configuring Redis externally, such as a cloud provider service. + Redis requires authentication if used with Sentinel. See [Redis Security](https://redis.io/docs/manual/security/) documentation for more information. We recommend using a combination of a Redis password and tight @@ -452,14 +448,9 @@ to be used with GitLab. The following IPs will be used as an example: ### Provide your own Redis instance -Managed Redis from cloud providers such as AWS ElastiCache will work. If these -services support high availability, be sure it is **not** the Redis Cluster type. - -Because Omnibus GitLab packages ship with Redis 6.0 or later, Redis 6.0 or later is required. Older Redis versions have reached end-of-life. +You can optionally use a third party external service for Redis as long as it meets the [requirements](../../install/requirements.md#redis). -Note the Redis node's IP address or hostname, port, and password (if required). -These will be necessary when configuring the -[GitLab application servers](#configure-gitlab-rails) later. +A reputable provider or solution should be used for this. [Google Memorystore](https://cloud.google.com/memorystore/docs/redis/redis-overview) and [AWS Elasticache](https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/WhatIs.html) are known to work. However, note that the Redis Cluster mode specifically isn't supported by GitLab. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. ### Standalone Redis using Omnibus GitLab @@ -530,7 +521,7 @@ a node and change its status from primary to replica (and vice versa). 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. You can specify multiple roles, like sentinel and Redis, as: `roles ['redis_sentinel_role', 'redis_master_role']`. Read more about @@ -615,7 +606,7 @@ run: redis-exporter: (pid 30075) 76861s; run: log: (pid 29674) 76896s 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. @@ -751,7 +742,7 @@ To configure the Sentinel: 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Go through the steps again for all the other Consul/Sentinel nodes, and make sure you set up the correct IPs. @@ -794,24 +785,21 @@ cluster to be used with GitLab. ### Provide your own PostgreSQL instance -If you're hosting GitLab on a cloud provider, you can optionally use a -managed service for PostgreSQL. +You can optionally use a [third party external service for PostgreSQL](../../administration/postgresql/external.md). -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default from [14.4.0](../../update/index.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. -If you use a cloud-managed service, or provide your own PostgreSQL: +If you use a third party external service: +1. Note that the HA Omnibus PostgreSQL setup encompasses PostgreSQL, PgBouncer and Consul. All of these components would no longer be required when using a third party external service. 1. Set up PostgreSQL according to the [database requirements document](../../install/requirements.md#database). 1. Set up a `gitlab` username with a password of your choice. The `gitlab` user needs privileges to create the `gitlabhq_production` database. 1. Configure the GitLab application servers with the appropriate details. This step is covered in [Configuring the GitLab Rails application](#configure-gitlab-rails). -1. For improved performance, configuring [Database Load Balancing](../postgresql/database_load_balancing.md) - with multiple read replicas is recommended. - -See [Configure GitLab using an external PostgreSQL service](../postgresql/external.md) for -further configuration steps. +1. The number of nodes required to achieve HA may differ depending on the service compared to Omnibus and doesn't need to match accordingly. +1. However, if [Database Load Balancing](../postgresql/database_load_balancing.md) via Read Replicas is desired for further improved performance it's recommended to follow the node count for the Reference Architecture. ### Standalone PostgreSQL using Omnibus GitLab @@ -828,9 +816,9 @@ replication and failover requires: The following IPs will be used as an example: -- `10.6.0.31`: PostgreSQL primary -- `10.6.0.32`: PostgreSQL secondary 1 -- `10.6.0.33`: PostgreSQL secondary 2 +- `10.6.0.21`: PostgreSQL primary +- `10.6.0.22`: PostgreSQL secondary 1 +- `10.6.0.23`: PostgreSQL secondary 2 First, make sure to [install](https://about.gitlab.com/install/) the Linux GitLab package **on each node**. Following the steps, @@ -955,7 +943,7 @@ For more information, see the various [Patroni replication methods](../postgresq 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/database.html) are supported and can be added if needed. @@ -981,9 +969,9 @@ SSH in to any of the Patroni nodes on the **primary site**: ```plaintext | Cluster | Member | Host | Role | State | TL | Lag in MB | Pending restart | |---------------|-----------------------------------|-----------|--------|---------|-----|-----------|-----------------| - | postgresql-ha | <PostgreSQL primary hostname> | 10.6.0.31 | Leader | running | 175 | | * | - | postgresql-ha | <PostgreSQL secondary 1 hostname> | 10.6.0.32 | | running | 175 | 0 | * | - | postgresql-ha | <PostgreSQL secondary 2 hostname> | 10.6.0.33 | | running | 175 | 0 | * | + | postgresql-ha | <PostgreSQL primary hostname> | 10.6.0.21 | Leader | running | 175 | | * | + | postgresql-ha | <PostgreSQL secondary 1 hostname> | 10.6.0.22 | | running | 175 | 0 | * | + | postgresql-ha | <PostgreSQL secondary 2 hostname> | 10.6.0.23 | | running | 175 | 0 | * | ``` If the 'State' column for any node doesn't say "running", check the @@ -1003,9 +991,9 @@ for tracking and handling reads/writes to the primary database. The following IPs will be used as an example: -- `10.6.0.21`: PgBouncer 1 -- `10.6.0.22`: PgBouncer 2 -- `10.6.0.23`: PgBouncer 3 +- `10.6.0.31`: PgBouncer 1 +- `10.6.0.32`: PgBouncer 2 +- `10.6.0.33`: PgBouncer 3 1. On each PgBouncer node, edit `/etc/gitlab/gitlab.rb`, and replace `<consul_password_hash>` and `<pgbouncer_password_hash>` with the @@ -1045,7 +1033,7 @@ The following IPs will be used as an example: 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Create a `.pgpass` file so Consul is able to reload PgBouncer. Enter the PgBouncer password twice when asked: @@ -1215,7 +1203,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Follow the [post configuration](#praefect-postgresql-post-configuration). <div align="right"> @@ -1444,10 +1432,10 @@ the file of the same name on this server. If this is the first Omnibus node you sudo touch /etc/gitlab/skip-auto-reconfigure ``` - 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect and + 1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect and to run the Praefect database migrations. -1. On all other Praefect nodes, [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. On all other Praefect nodes, [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. ### Configure Gitaly @@ -1619,7 +1607,7 @@ Updates to example must be made at: 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). ### Gitaly Cluster TLS support @@ -1675,7 +1663,7 @@ To configure Praefect with TLS: } ``` -1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file and [reconfigure](../restart_gitlab.md#reconfigure-a-linux-package-installation). 1. On the Praefect clients (including each Gitaly server), copy the certificates, or their certificate authority, into `/etc/gitlab/trusted-certs`: @@ -1696,7 +1684,7 @@ To configure Praefect with TLS: }) ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -1781,7 +1769,7 @@ Updates to example must be made at: gitlab_rails['db_host'] = '10.6.0.40' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' - gitlab_rails['db_load_balancing'] = { 'hosts' => ['10.6.0.31', '10.6.0.32', '10.6.0.33'] } # PostgreSQL IPs + gitlab_rails['db_load_balancing'] = { 'hosts' => ['10.6.0.21', '10.6.0.22', '10.6.0.23'] } # PostgreSQL IPs ## Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -1848,7 +1836,7 @@ Updates to example must be made at: Only a single designated node should handle migrations as detailed in the [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). 1. Verify the GitLab services are running: @@ -1924,7 +1912,7 @@ On each node perform the following: gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' - gitlab_rails['db_load_balancing'] = { 'hosts' => ['10.6.0.31', '10.6.0.32', '10.6.0.33'] } # PostgreSQL IPs + gitlab_rails['db_load_balancing'] = { 'hosts' => ['10.6.0.21', '10.6.0.22', '10.6.0.23'] } # PostgreSQL IPs # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -2038,7 +2026,7 @@ On each node perform the following: Only a single designated node should handle migrations as detailed in the [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. [Enable incremental logging](#enable-incremental-logging). 1. Run `sudo gitlab-rake gitlab:gitaly:check` to confirm the node can connect to Gitaly. 1. Tail the logs to see the requests: @@ -2127,9 +2115,9 @@ running [Prometheus](../monitoring/prometheus/index.md) and 'job_name': 'pgbouncer', 'static_configs' => [ 'targets' => [ - "10.6.0.21:9188", - "10.6.0.22:9188", - "10.6.0.23:9188", + "10.6.0.31:9188", + "10.6.0.32:9188", + "10.6.0.33:9188", ], ], }, @@ -2149,7 +2137,7 @@ running [Prometheus](../monitoring/prometheus/index.md) and nginx['enable'] = true ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). 1. In the GitLab UI, set `admin/application_settings/metrics_and_profiling` > Metrics - Grafana to `/-/grafana` to `http[s]://<MONITOR NODE>/-/grafana`. 1. Verify the GitLab services are running: @@ -2179,7 +2167,7 @@ running [Prometheus](../monitoring/prometheus/index.md) and GitLab supports using an [object storage](../object_storage.md) service for holding numerous types of data. It's recommended over [NFS](../nfs.md) for data objects and in general it's better in larger setups as object storage is typically much more performant, reliable, -and scalable. +and scalable. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. There are two ways of specifying object storage configuration in GitLab: @@ -2317,17 +2305,10 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. - - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo. - - Note that [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported. - - [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440). - - Consul is primarily used for Omnibus PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. -2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Memorystore](https://cloud.google.com/memorystore) and [Amazon ElastiCache](https://aws.amazon.com/elasticache/) are known to work. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance) for more information. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Provide your own Redis instance](#provide-your-own-redis-instance) for more information. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. -4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. +4. Should be run on reputable Cloud Provider or Self Managed solutions. See [Configure the object storage](#configure-the-object-storage) for more information. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. 6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 7d3ddf7578c..9a2c354f27c 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -38,17 +38,11 @@ full list of reference architectures, see <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. - - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo. - - Note that [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported. - - [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440). - - Consul is primarily used for Omnibus PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. -2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Memorystore](https://cloud.google.com/memorystore) and [Amazon ElastiCache](https://aws.amazon.com/elasticache/) are known to work. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance) for more information. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Provide your own Redis instance](#provide-your-own-redis-instance) for more information. + - Redis is primarily single threaded. It's strongly recommended separating out the instances as specified into Cache and Persistent data to achieve optimum performance at this scale. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. -4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. +4. Should be run on reputable Cloud Provider or Self Managed solutions. See [Configure the object storage](#configure-the-object-storage) for more information. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. 6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to @@ -396,9 +390,9 @@ backend pgbouncer mode tcp option tcp-check - server pgbouncer1 10.6.0.21:6432 check - server pgbouncer2 10.6.0.22:6432 check - server pgbouncer3 10.6.0.23:6432 check + server pgbouncer1 10.6.0.31:6432 check + server pgbouncer2 10.6.0.32:6432 check + server pgbouncer3 10.6.0.33:6432 check backend praefect mode tcp @@ -471,7 +465,7 @@ To configure Consul: 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Go through the steps again for all the other Consul nodes, and make sure you set up the correct IPs. @@ -513,24 +507,21 @@ cluster to be used with GitLab. ### Provide your own PostgreSQL instance -If you're hosting GitLab on a cloud provider, you can optionally use a -managed service for PostgreSQL. +You can optionally use a [third party external service for PostgreSQL](../../administration/postgresql/external.md). -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default from [14.4.0](../../update/index.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. -If you use a cloud-managed service, or provide your own PostgreSQL: +If you use a third party external service: +1. Note that the HA Omnibus PostgreSQL setup encompasses PostgreSQL, PgBouncer and Consul. All of these components would no longer be required when using a third party external service. 1. Set up PostgreSQL according to the [database requirements document](../../install/requirements.md#database). 1. Set up a `gitlab` username with a password of your choice. The `gitlab` user needs privileges to create the `gitlabhq_production` database. 1. Configure the GitLab application servers with the appropriate details. This step is covered in [Configuring the GitLab Rails application](#configure-gitlab-rails). -1. For improved performance, configuring [Database Load Balancing](../postgresql/database_load_balancing.md) - with multiple read replicas is recommended. - -See [Configure GitLab using an external PostgreSQL service](../postgresql/external.md) for -further configuration steps. +1. The number of nodes required to achieve HA may differ depending on the service compared to Omnibus and doesn't need to match accordingly. +1. However, if [Database Load Balancing](../postgresql/database_load_balancing.md) via Read Replicas is desired for further improved performance it's recommended to follow the node count for the Reference Architecture. ### Standalone PostgreSQL using Omnibus GitLab @@ -675,7 +666,7 @@ For more information, see the various [Patroni replication methods](../postgresq 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/database.html) are supported and can be added if needed. @@ -764,7 +755,7 @@ The following IPs will be used as an example: 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. If an error `execute[generate databases.ini]` occurs, this is due to an existing [known issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4713). @@ -777,7 +768,7 @@ The following IPs will be used as an example: gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul ``` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) once again +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) once again to resolve any potential errors from the previous steps. 1. Ensure each node is talking to the current primary: @@ -821,6 +812,9 @@ start the failover procedure. NOTE: Redis clusters must each be deployed in an odd number of 3 nodes or more. This is to ensure Redis Sentinel can take votes as part of a quorum. This does not apply when configuring Redis externally, such as a cloud provider service. +NOTE: +Redis is primarily single threaded. It's strongly recommended separating out the instances as specified into Cache and Persistent data to achieve optimum performance at this scale. + Redis requires authentication if used with Sentinel. See [Redis Security](https://redis.io/docs/manual/security/) documentation for more information. We recommend using a combination of a Redis password and tight @@ -850,15 +844,11 @@ to be used with GitLab. The following IPs will be used as an example: - `10.6.0.62`: Redis - Persistent Replica 1 - `10.6.0.63`: Redis - Persistent Replica 2 -### Providing your own Redis instance +### Provide your own Redis instance -Managed Redis from cloud providers (such as AWS ElastiCache) will work. If these -services support high availability, be sure it _isn't_ of the Redis Cluster type. +You can optionally use a third party external service for Redis as long as it meets the [requirements](../../install/requirements.md#redis). -Because Omnibus GitLab packages ship with Redis 6.0 or later, Redis 6.0 or later is required. Older Redis versions have reached end-of-life. - -Note the Redis node's IP address or hostname, port, and password (if required). -These will be necessary later when configuring the [GitLab application servers](#configure-gitlab-rails). +A reputable provider or solution should be used for this. [Google Memorystore](https://cloud.google.com/memorystore/docs/redis/redis-overview) and [AWS Elasticache](https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/WhatIs.html) are known to work. However, note that the Redis Cluster mode specifically isn't supported by GitLab. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. ### Configure the Redis Cache cluster @@ -939,7 +929,7 @@ a node and change its status from primary to replica (and vice versa). 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. #### Configure the replica Redis Cache nodes @@ -1014,7 +1004,7 @@ a node and change its status from primary to replica (and vice versa). server. If this is the first Omnibus node you are configuring then you can skip this step. - 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes + 1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Go through the steps again for all the other replica nodes, and @@ -1068,7 +1058,7 @@ a node and change its status from primary to replica (and vice versa). #redis['master_port'] = 6379 # Set up password authentication for Redis and replicas (use the same password in all nodes). - redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' ## Must be the same in every Redis node @@ -1097,7 +1087,7 @@ a node and change its status from primary to replica (and vice versa). 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. #### Configure the replica Redis Persistent nodes @@ -1160,7 +1150,7 @@ a node and change its status from primary to replica (and vice versa). 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. @@ -1279,7 +1269,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Follow the [post configuration](#praefect-postgresql-post-configuration). @@ -1511,10 +1501,10 @@ the file of the same name on this server. If this is the first Omnibus node you sudo touch /etc/gitlab/skip-auto-reconfigure ``` - 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect and + 1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect and to run the Praefect database migrations. -1. On all other Praefect nodes, [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. On all other Praefect nodes, [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. ### Configure Gitaly @@ -1682,7 +1672,7 @@ Updates to example must be made at: 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). ### Gitaly Cluster TLS support @@ -1738,7 +1728,7 @@ To configure Praefect with TLS: } ``` -1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file and [reconfigure](../restart_gitlab.md#reconfigure-a-linux-package-installation). 1. On the Praefect clients (including each Gitaly server), copy the certificates, or their certificate authority, into `/etc/gitlab/trusted-certs`: @@ -1759,7 +1749,7 @@ To configure Praefect with TLS: }) ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -1916,7 +1906,7 @@ Updates to example must be made at: Only a single designated node should handle migrations as detailed in the [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. NOTE: If you find that the environment's Sidekiq job processing is slow with long queues, @@ -2085,7 +2075,7 @@ On each node perform the following: Only a single designated node should handle migrations as detailed in the [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. [Enable incremental logging](#enable-incremental-logging). 1. Confirm the node can connect to Gitaly: @@ -2196,7 +2186,7 @@ To configure the Monitoring node: nginx['enable'] = true ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). 1. In the GitLab UI, set `admin/application_settings/metrics_and_profiling` > Metrics - Grafana to `/-/grafana` to `http[s]://<MONITOR NODE>/-/grafana` @@ -2211,7 +2201,7 @@ To configure the Monitoring node: GitLab supports using an [object storage](../object_storage.md) service for holding numerous types of data. It's recommended over [NFS](../nfs.md) for data objects and in general it's better in larger setups as object storage is typically much more performant, reliable, -and scalable. +and scalable. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. There are two ways of specifying object storage configuration in GitLab: @@ -2326,17 +2316,11 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. - - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo. - - Note that [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported. - - [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440). - - Consul is primarily used for Omnibus PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. -2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Memorystore](https://cloud.google.com/memorystore) and [Amazon ElastiCache](https://aws.amazon.com/elasticache/) are known to work. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance) for more information. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Provide your own Redis instance](#provide-your-own-redis-instance) for more information. + - Redis is primarily single threaded. It's strongly recommended separating out the instances as specified into Cache and Persistent data to achieve optimum performance at this scale. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. -4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. +4. Should be run on reputable Cloud Provider or Self Managed solutions. See [Configure the object storage](#configure-the-object-storage) for more information. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. 6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 82087c91910..b0bc70aaf00 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -44,17 +44,10 @@ costly-to-operate environment by using the <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. - - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo. - - Note that [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported. - - [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440). - - Consul is primarily used for Omnibus PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. -2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Memorystore](https://cloud.google.com/memorystore) and [Amazon ElastiCache](https://aws.amazon.com/elasticache/) are known to work. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance) for more information. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Provide your own Redis instance](#provide-your-own-redis-instance) for more information. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. -4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. +4. Should be run on reputable Cloud Provider or Self Managed solutions. See [Configure the object storage](#configure-the-object-storage) for more information. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. 6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to @@ -184,19 +177,19 @@ connect to each other freely on these addresses. The following list includes descriptions of each server and its assigned IP: - `10.6.0.10`: External Load Balancer -- `10.6.0.61`: Redis Primary -- `10.6.0.62`: Redis Replica 1 -- `10.6.0.63`: Redis Replica 2 - `10.6.0.11`: Consul/Sentinel 1 - `10.6.0.12`: Consul/Sentinel 2 - `10.6.0.13`: Consul/Sentinel 3 -- `10.6.0.31`: PostgreSQL primary -- `10.6.0.32`: PostgreSQL secondary 1 -- `10.6.0.33`: PostgreSQL secondary 2 -- `10.6.0.21`: PgBouncer 1 -- `10.6.0.22`: PgBouncer 2 -- `10.6.0.23`: PgBouncer 3 +- `10.6.0.21`: PostgreSQL primary +- `10.6.0.22`: PostgreSQL secondary 1 +- `10.6.0.23`: PostgreSQL secondary 2 +- `10.6.0.31`: PgBouncer 1 +- `10.6.0.32`: PgBouncer 2 +- `10.6.0.33`: PgBouncer 3 - `10.6.0.20`: Internal Load Balancer +- `10.6.0.61`: Redis Primary +- `10.6.0.62`: Redis Replica 1 +- `10.6.0.63`: Redis Replica 2 - `10.6.0.51`: Gitaly 1 - `10.6.0.52`: Gitaly 2 - `10.6.0.93`: Gitaly 3 @@ -396,9 +389,9 @@ backend pgbouncer mode tcp option tcp-check - server pgbouncer1 10.6.0.21:6432 check - server pgbouncer2 10.6.0.22:6432 check - server pgbouncer3 10.6.0.23:6432 check + server pgbouncer1 10.6.0.31:6432 check + server pgbouncer2 10.6.0.32:6432 check + server pgbouncer3 10.6.0.33:6432 check backend praefect mode tcp @@ -449,14 +442,9 @@ to be used with GitLab. The following IPs are used as an example: ### Provide your own Redis instance -Managed Redis from cloud providers such as AWS ElastiCache works. If these -services support high availability, be sure it is **not** the Redis Cluster type. - -Because Omnibus GitLab packages ship with Redis 6.0 or later, Redis 6.0 or later is required. Older Redis versions have reached end-of-life. +You can optionally use a third party external service for Redis as long as it meets the [requirements](../../install/requirements.md#redis). -Note the Redis node's IP address or hostname, port, and password (if required). -These are necessary when configuring the -[GitLab application servers](#configure-gitlab-rails) later. +A reputable provider or solution should be used for this. [Google Memorystore](https://cloud.google.com/memorystore/docs/redis/redis-overview) and [AWS Elasticache](https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/WhatIs.html) are known to work. However, note that the Redis Cluster mode specifically isn't supported by GitLab. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. ### Standalone Redis using Omnibus GitLab @@ -527,7 +515,7 @@ a node and change its status from primary to replica (and vice versa). 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. You can specify multiple roles, like sentinel and Redis, as: `roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about @@ -612,7 +600,7 @@ run: redis-exporter: (pid 30075) 76861s; run: log: (pid 29674) 76896s 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. @@ -748,7 +736,7 @@ To configure the Sentinel: 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Go through the steps again for all the other Consul/Sentinel nodes, and make sure you set up the correct IPs. @@ -790,24 +778,21 @@ cluster to be used with GitLab. ### Provide your own PostgreSQL instance -If you're hosting GitLab on a cloud provider, you can optionally use a -managed service for PostgreSQL. +You can optionally use a [third party external service for PostgreSQL](../../administration/postgresql/external.md). -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default from [14.4.0](../../update/index.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. -If you use a cloud-managed service, or provide your own PostgreSQL: +If you use a third party external service: +1. Note that the HA Omnibus PostgreSQL setup encompasses PostgreSQL, PgBouncer and Consul. All of these components would no longer be required when using a third party external service. 1. Set up PostgreSQL according to the [database requirements document](../../install/requirements.md#database). 1. Set up a `gitlab` username with a password of your choice. The `gitlab` user needs privileges to create the `gitlabhq_production` database. 1. Configure the GitLab application servers with the appropriate details. This step is covered in [Configuring the GitLab Rails application](#configure-gitlab-rails). -1. For improved performance, configuring [Database Load Balancing](../postgresql/database_load_balancing.md) - with multiple read replicas is recommended. - -See [Configure GitLab using an external PostgreSQL service](../postgresql/external.md) for -further configuration steps. +1. The number of nodes required to achieve HA may differ depending on the service compared to Omnibus and doesn't need to match accordingly. +1. However, if [Database Load Balancing](../postgresql/database_load_balancing.md) via Read Replicas is desired for further improved performance it's recommended to follow the node count for the Reference Architecture. ### Standalone PostgreSQL using Omnibus GitLab @@ -824,9 +809,9 @@ replication and failover requires: The following IPs are used as an example: -- `10.6.0.31`: PostgreSQL primary -- `10.6.0.32`: PostgreSQL secondary 1 -- `10.6.0.33`: PostgreSQL secondary 2 +- `10.6.0.21`: PostgreSQL primary +- `10.6.0.22`: PostgreSQL secondary 1 +- `10.6.0.23`: PostgreSQL secondary 2 First, make sure to [install](https://about.gitlab.com/install/) the Linux GitLab package **on each node**. Following the steps, @@ -951,7 +936,7 @@ For more information, see the various [Patroni replication methods](../postgresq 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/database.html) are supported and can be added if needed. @@ -977,9 +962,9 @@ SSH in to any of the Patroni nodes on the **primary site**: ```plaintext | Cluster | Member | Host | Role | State | TL | Lag in MB | Pending restart | |---------------|-----------------------------------|-----------|--------|---------|-----|-----------|-----------------| - | postgresql-ha | <PostgreSQL primary hostname> | 10.6.0.31 | Leader | running | 175 | | * | - | postgresql-ha | <PostgreSQL secondary 1 hostname> | 10.6.0.32 | | running | 175 | 0 | * | - | postgresql-ha | <PostgreSQL secondary 2 hostname> | 10.6.0.33 | | running | 175 | 0 | * | + | postgresql-ha | <PostgreSQL primary hostname> | 10.6.0.21 | Leader | running | 175 | | * | + | postgresql-ha | <PostgreSQL secondary 1 hostname> | 10.6.0.22 | | running | 175 | 0 | * | + | postgresql-ha | <PostgreSQL secondary 2 hostname> | 10.6.0.23 | | running | 175 | 0 | * | ``` If the 'State' column for any node doesn't say "running", check the @@ -999,9 +984,9 @@ for tracking and handling reads/writes to the primary database. The following IPs are used as an example: -- `10.6.0.21`: PgBouncer 1 -- `10.6.0.22`: PgBouncer 2 -- `10.6.0.23`: PgBouncer 3 +- `10.6.0.31`: PgBouncer 1 +- `10.6.0.32`: PgBouncer 2 +- `10.6.0.33`: PgBouncer 3 1. On each PgBouncer node, edit `/etc/gitlab/gitlab.rb`, and replace `<consul_password_hash>` and `<pgbouncer_password_hash>` with the @@ -1041,7 +1026,7 @@ The following IPs are used as an example: 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Create a `.pgpass` file so Consul is able to reload PgBouncer. Enter the PgBouncer password twice when asked: @@ -1211,7 +1196,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Follow the [post configuration](#praefect-postgresql-post-configuration). @@ -1441,10 +1426,10 @@ the file of the same name on this server. If this is the first Omnibus node you sudo touch /etc/gitlab/skip-auto-reconfigure ``` - 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect and + 1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect and to run the Praefect database migrations. -1. On all other Praefect nodes, [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. On all other Praefect nodes, [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. ### Configure Gitaly @@ -1612,7 +1597,7 @@ Updates to example must be made at: 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. -1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). ### Gitaly Cluster TLS support @@ -1668,7 +1653,7 @@ To configure Praefect with TLS: } ``` -1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file and [reconfigure](../restart_gitlab.md#reconfigure-a-linux-package-installation). 1. On the Praefect clients (including each Gitaly server), copy the certificates, or their certificate authority, into `/etc/gitlab/trusted-certs`: @@ -1689,7 +1674,7 @@ To configure Praefect with TLS: }) ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -1773,7 +1758,7 @@ Updates to example must be made at: gitlab_rails['db_host'] = '10.6.0.40' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' - gitlab_rails['db_load_balancing'] = { 'hosts' => ['10.6.0.31', '10.6.0.32', '10.6.0.33'] } # PostgreSQL IPs + gitlab_rails['db_load_balancing'] = { 'hosts' => ['10.6.0.21', '10.6.0.22', '10.6.0.23'] } # PostgreSQL IPs ## Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -1840,7 +1825,7 @@ Updates to example must be made at: Only a single designated node should handle migrations as detailed in the [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Verify the GitLab services are running: @@ -1916,7 +1901,7 @@ On each node perform the following: gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' - gitlab_rails['db_load_balancing'] = { 'hosts' => ['10.6.0.31', '10.6.0.32', '10.6.0.33'] } # PostgreSQL IPs + gitlab_rails['db_load_balancing'] = { 'hosts' => ['10.6.0.21', '10.6.0.22', '10.6.0.23'] } # PostgreSQL IPs # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -2033,7 +2018,7 @@ On each node perform the following: Only a single designated node should handle migrations as detailed in the [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. [Enable incremental logging](#enable-incremental-logging). 1. Run `sudo gitlab-rake gitlab:gitaly:check` to confirm the node can connect to Gitaly. 1. Tail the logs to see the requests: @@ -2144,7 +2129,7 @@ running [Prometheus](../monitoring/prometheus/index.md) and nginx['enable'] = true ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). 1. In the GitLab UI, set `admin/application_settings/metrics_and_profiling` > Metrics - Grafana to `/-/grafana` to `http[s]://<MONITOR NODE>/-/grafana`. 1. Verify the GitLab services are running: @@ -2174,7 +2159,7 @@ running [Prometheus](../monitoring/prometheus/index.md) and GitLab supports using an [object storage](../object_storage.md) service for holding numerous types of data. It's recommended over [NFS](../nfs.md) for data objects and in general it's better in larger setups as object storage is typically much more performant, reliable, -and scalable. +and scalable. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. There are two ways of specifying object storage configuration in GitLab: @@ -2288,17 +2273,10 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. - - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo. - - Note that [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported. - - [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440). - - Consul is primarily used for Omnibus PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. -2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Memorystore](https://cloud.google.com/memorystore) and [Amazon ElastiCache](https://aws.amazon.com/elasticache/) are known to work. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance) for more information. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Provide your own Redis instance](#provide-your-own-redis-instance) for more information. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. -4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. +4. Should be run on reputable Cloud Provider or Self Managed solutions. See [Configure the object storage](#configure-the-object-storage) for more information. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. 6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index 4b9c26e8626..08aeb149454 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -167,7 +167,7 @@ These reference architectures were built and tested on Google Cloud Platform (GC [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) CPU platform as a lowest common denominator baseline ([Sysbench benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks)). -Newer, similarly-sized CPUs are supported and may have improved performance as a result. For Omnibus GitLab environments, +Newer, similarly-sized CPUs are supported and may have improved performance as a result. For Linux package environments, ARM-based equivalents are also supported. NOTE: @@ -240,7 +240,7 @@ for more information and guidance. that to achieve full High Availability, a third-party PostgreSQL database solution is required. We hope to offer a built-in solution for these restrictions in the future. In the meantime, a non-HA PostgreSQL server -can be set up using Omnibus GitLab as the specifications reflect. Refer to the following issues for more information: +can be set up using the Linux package as the specifications reflect. Refer to the following issues for more information: - [`omnibus-gitlab#7292`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7292). - [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398). @@ -321,18 +321,21 @@ Additionally, the following cloud provider services are validated and supported ### Recommendation notes for the database services -When selecting a database service, it should run a standard, performant, and [supported version](../../install/requirements.md#postgresql-requirements) of PostgreSQL with the following features: +[When selecting to use an external database service](../postgresql/external.md), it should run a standard, performant, and [supported version](../../install/requirements.md#postgresql-requirements). -- Read Replicas for [Database Load Balancing](../postgresql/database_load_balancing.md). -- Cross Region replication for [GitLab Geo](../geo/index.md). +If you choose to use a third party external service: + +1. Note that the HA Linux package PostgreSQL setup encompasses PostgreSQL, PgBouncer and Consul. All of these components would no longer be required when using a third party external service. +1. The number of nodes required to achieve HA may differ depending on the service compared to the Linux package and doesn't need to match accordingly. +1. However, if [Database Load Balancing](../postgresql/database_load_balancing.md) via Read Replicas is desired for further improved performance it's recommended to follow the node count for the Reference Architecture. +1. If [GitLab Geo](../geo/index.md) is to be used the service will need to support Cross Region replication #### Unsupported database services Several database cloud provider services are known not to support the above or have been found to have other issues and aren't recommended: - [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is incompatible and not supported. See [14.4.0](../../update/index.md#1440) for more details. -- [Azure Database for PostgreSQL Single Server](https://azure.microsoft.com/en-gb/products/postgresql/#overview) (Single / Flexible) is not supported for use due to notable performance / stability issues or missing functionality. See [Recommendation Notes for Azure](#recommendation-notes-for-azure) for more details. -- Azure Database for PostgreSQL Flexible Server uses Microsoft Azure Active Directory (Azure AD) as authentication mechanism, which is incompatible with GitLab database integration. +- [Azure Database for PostgreSQL Single Server](https://azure.microsoft.com/en-gb/products/postgresql/#overview) is not supported for use due to notable performance / stability issues or missing functionality. See [Recommendation Notes for Azure](#recommendation-notes-for-azure) for more details. - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo. - [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported. @@ -342,8 +345,9 @@ Due to performance issues that we found with several key Azure services, we only In addition to the above, you should be aware of the additional specific guidance for Azure: -- [Azure Database for PostgreSQL Single Server](https://azure.microsoft.com/en-gb/products/postgresql/#overview) (Single / Flexible) is not supported for use due to notable performance / stability issues or missing functionality. +- [Azure Database for PostgreSQL Single Server](https://azure.microsoft.com/en-gb/products/postgresql/#overview) is not supported for use due to notable performance / stability issues or missing functionality. - A new service, [Azure Database for PostgreSQL Flexible Server](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/) has been released. [Internal testing](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/91) has shown that it does look to perform as expected, but this hasn't been validated in production, so generally isn't recommended at this time. Additionally, as it's a new service, you may find that it's missing some functionality depending on your requirements. + - Only standard Postgres authentication is supported at this time with this service. Microsoft Azure Active Directory (Azure AD) is not compatible. - [Azure Blob Storage](https://azure.microsoft.com/en-gb/products/storage/blobs/) has been found to have [performance limits that can impact production use at certain times](https://gitlab.com/gitlab-org/gitlab/-/issues/344861). However, this has only been seen in our largest architectures (25k+) so far. ## Deviating from the suggested reference architectures @@ -353,8 +357,8 @@ the harder it is to get support for it. With any deviation, you're introducing a layer of complexity that adds challenges to finding out where potential issues might lie. -The reference architectures use the official GitLab Linux packages (Omnibus -GitLab) or [Helm Charts](https://docs.gitlab.com/charts/) to install and configure the various components. The components are +The reference architectures use the official Linux packages or [Helm Charts](https://docs.gitlab.com/charts/) to +install and configure the various components. The components are installed on separate machines (virtualized or bare metal), with machine hardware requirements listed in the "Configuration" column and equivalent VM standard sizes listed in GCP/AWS/Azure columns of each [available reference architecture](#available-reference-architectures). @@ -414,9 +418,10 @@ Testing occurs against all reference architectures and cloud providers in an aut Network latency on the test environments between components on all Cloud Providers were measured at <5 ms. This is shared as an observation and not as an implicit recommendation. -We aim to have a "test smart" approach where architectures tested have a good range that can also apply to others. Testing focuses on 10k Omnibus on GCP as the testing has shown this is a good bellwether for the other architectures and cloud providers as well as Cloud Native Hybrids. +We aim to have a "test smart" approach where architectures tested have a good range that can also apply to others. Testing focuses on a 10k Linux package +installation on GCP as the testing has shown this is a good bellwether for the other architectures and cloud providers as well as Cloud Native Hybrids. -The Standard Reference Architectures are designed to be platform-agnostic, with everything being run on VMs via [Omnibus GitLab](https://docs.gitlab.com/omnibus/). While testing occurs primarily on GCP, ad-hoc testing has shown that they perform similarly on hardware with equivalent specs on other Cloud Providers or if run on premises (bare-metal). +The Standard Reference Architectures are designed to be platform-agnostic, with everything being run on VMs through [the Linux package](https://docs.gitlab.com/omnibus/). While testing occurs primarily on GCP, ad-hoc testing has shown that they perform similarly on hardware with equivalent specs on other Cloud Providers or if run on premises (bare-metal). Testing on these reference architectures is performed with the [GitLab Performance Tool](https://gitlab.com/gitlab-org/quality/performance) @@ -468,11 +473,11 @@ table.test-coverage th { <th style="text-align: center" colspan="2" scope="colgroup">Azure</th> </tr> <tr> - <th scope="col">Omnibus</th> + <th scope="col">Linux package</th> <th scope="col">Cloud Native Hybrid</th> - <th scope="col">Omnibus</th> + <th scope="col">Linux package</th> <th scope="col">Cloud Native Hybrid</th> - <th scope="col">Omnibus</th> + <th scope="col">Linux package</th> </tr> <tr> <th scope="row">1k</th> @@ -534,7 +539,7 @@ table.test-coverage th { ## Cost to run -As a starting point, the following table details initial costs for the different reference architectures across GCP, AWS, and Azure via Omnibus. +As a starting point, the following table details initial costs for the different reference architectures across GCP, AWS, and Azure through the Linux package. NOTE: Due to the nature of Cloud Native Hybrid, it's not possible to give a static cost calculation. @@ -551,9 +556,9 @@ Bare-metal costs are also not included here as it varies widely depending on eac <th style="text-align: center" scope="colgroup">Azure</th> </tr> <tr> - <th scope="col">Omnibus</th> - <th scope="col">Omnibus</th> - <th scope="col">Omnibus</th> + <th scope="col">Linux package</th> + <th scope="col">Linux package</th> + <th scope="col">Linux package</th> </tr> <tr> <th scope="row">1k</th> diff --git a/doc/administration/repository_checks.md b/doc/administration/repository_checks.md index 3f715e451a8..8cc635b50fc 100644 --- a/doc/administration/repository_checks.md +++ b/doc/administration/repository_checks.md @@ -20,7 +20,8 @@ committed to a repository. GitLab administrators can: To check a project's repository using GitLab UI: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Overview > Projects**. 1. Select the project to check. 1. In the **Repository check** section, select **Trigger repository check**. @@ -32,7 +33,8 @@ project page in the Admin Area. If the checks fail, see [what to do](#what-to-do Instead of checking repositories manually, GitLab can be configured to run the checks periodically: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Repository** (`/admin/application_settings/repository`). 1. Expand the **Repository maintenance** section. 1. Enable **Enable repository checks**. @@ -41,7 +43,7 @@ When enabled, GitLab periodically runs a repository check on all project reposit repositories to detect possible data corruption. A project is checked no more than once per month. Administrators can configure the frequency of repository checks. To edit the frequency: -- For Omnibus GitLab installations, edit `gitlab_rails['repository_check_worker_cron']` in +- For Linux package installations, edit `gitlab_rails['repository_check_worker_cron']` in `/etc/gitlab/gitlab.rb`. - For source-based installations, edit `[gitlab.cron_jobs.repository_check_worker]` in `/home/git/gitlab/config/gitlab.yml`. @@ -59,7 +61,7 @@ You can run [`git fsck`](https://git-scm.com/docs/git-fsck) using the command li [Gitaly servers](gitaly/index.md). To locate the repositories: 1. Go to the storage location for repositories: - - For Omnibus GitLab installations, repositories are stored in the `/var/opt/gitlab/git-data/repositories` directory + - For Linux package installations, repositories are stored in the `/var/opt/gitlab/git-data/repositories` directory by default. - For GitLab Helm chart installations, repositories are stored in the `/home/git/repositories` directory inside the Gitaly pod by default. @@ -79,13 +81,14 @@ You can run [`git fsck`](https://git-scm.com/docs/git-fsck) using the command li If a repository check fails, locate the error in the [`repocheck.log` file](logs/index.md#repochecklog) on disk at: -- `/var/log/gitlab/gitlab-rails` for Omnibus GitLab installations. -- `/home/git/gitlab/log` for installations from source. +- `/var/log/gitlab/gitlab-rails` for Linux package installations. +- `/home/git/gitlab/log` for self-compiled installations. - `/var/log/gitlab` in the Sidekiq pod for GitLab Helm chart installations. If periodic repository checks cause false alarms, you can clear all repository check states: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Repository** (`/admin/application_settings/repository`). 1. Expand the **Repository maintenance** section. 1. Select **Clear all repository checks**. diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md index 808659132f9..1a83a05c3dd 100644 --- a/doc/administration/repository_storage_paths.md +++ b/doc/administration/repository_storage_paths.md @@ -18,7 +18,7 @@ storage is either: GitLab allows you to define multiple repository storages to distribute the storage load between several mount points. For example: -- When using Gitaly (Omnibus GitLab-style configuration): +- When using Gitaly (Linux package installation-style configuration): ```ruby git_data_dirs({ @@ -27,7 +27,7 @@ several mount points. For example: }) ``` -- When using direct repository storage (source install-style configuration): +- When using direct repository storage (self-compiled installation-style configuration): ```plaintext default: @@ -51,8 +51,8 @@ configuration option is deprecated in favor of using [Gitaly](gitaly/index.md). To configure repository storage paths: 1. Edit the necessary configuration files: - - `/etc/gitlab/gitlab.rb`, for Omnibus GitLab installations. - - `gitlab.yml`, for installations from source. + - `/etc/gitlab/gitlab.rb`, for Linux package installations. + - `gitlab.yml`, for self-compiled installations. 1. Add the required repository storage paths. For repository storage paths: @@ -78,15 +78,15 @@ For [backups](../raketasks/backup_restore.md) to work correctly: - The repository storage path cannot be a mount point. - The GitLab user must have correct permissions for the parent directory of the path. -Omnibus GitLab takes care of these issues for you, but for source installations you should be extra +The Linux package takes care of these issues for you but for self-compiled installations, you should be extra careful. While restoring a backup, the current contents of `/home/git/repositories` are moved to `/home/git/repositories.old`. If `/home/git/repositories` is a mount point, then `mv` would be moving things between mount points, and problems can occur. -Ideally, `/home/git` is the mount point, so things remain inside the same mount point. Omnibus -GitLab installations guarantee this because they don't specify the full repository path but instead +Ideally, `/home/git` is the mount point, so things remain inside the same mount point. Linux package +installations guarantee this because they don't specify the full repository path but instead the parent path, but source installations do not. ### Example configuration @@ -94,14 +94,14 @@ the parent path, but source installations do not. In the examples below, we add two additional repository storage paths configured to two additional mount points. -For compatibility reasons `gitlab.yml` has a different structure than Omnibus GitLab configuration: +For compatibility reasons `gitlab.yml` has a different structure than Linux package installation configuration: -- In `gitlab.yml`, you indicate the path for the repositories, for example `/home/git/repositories` -- In Omnibus GitLab configuration you indicate `git_data_dirs`, which could be `/home/git` for - example. Then Omnibus GitLab creates a `repositories` directory under that path to use with +- In `gitlab.yml`, you indicate the path for the repositories. For example, `/home/git/repositories`. +- In Linux package installation configuration, you indicate `git_data_dirs`, which could be `/home/git` for + example. The Linux package installation then creates a `repositories` directory under that path to use with `gitlab.yml`. -**For installations from source** +For self-compiled installations: 1. Edit `gitlab.yml` and add the storage paths: @@ -122,7 +122,7 @@ For compatibility reasons `gitlab.yml` has a different structure than Omnibus Gi 1. [Configure where new repositories are stored](#configure-where-new-repositories-are-stored). -**For Omnibus installations** +For Linux package installations: 1. Edit `/etc/gitlab/gitlab.rb` by appending the rest of the paths to the default one: @@ -134,19 +134,20 @@ For compatibility reasons `gitlab.yml` has a different structure than Omnibus Gi }) ``` -1. [Restart GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Restart GitLab](restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. [Configure where new repositories are stored](#configure-where-new-repositories-are-stored). NOTE: -Omnibus stores the repositories in a `repositories` subdirectory of the `git-data` directory. +Linux package installations store the repositories in a `repositories` subdirectory of the `git-data` directory. ## Configure where new repositories are stored After you [configure](#configure-repository-storage-paths) multiple repository storage paths, you can choose where new repositories are stored: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Repository** and expand the **Repository storage** section. 1. Enter values in the **Storage nodes for new repositories** fields. diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md index ab95887380d..3bd73b4df94 100644 --- a/doc/administration/repository_storage_types.md +++ b/doc/administration/repository_storage_types.md @@ -19,9 +19,8 @@ GitLab can be configured to use one or multiple repository storages. These stora In GitLab: - Repository storages are configured in: - - `/etc/gitlab/gitlab.rb` by the `git_data_dirs({})` configuration hash for Omnibus GitLab - installations. - - `gitlab.yml` by the `repositories.storages` key for installations from source. + - `/etc/gitlab/gitlab.rb` by the `git_data_dirs({})` configuration hash for Linux package installations. + - `gitlab.yml` by the `repositories.storages` key for self-compiled installations. - The `default` repository storage is available in any installations that haven't customized it. By default, it points to a Gitaly node. @@ -79,7 +78,8 @@ Administrators can look up a project's hashed path from its name or ID using: To look up a project's hash path in the Admin Area: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Overview > Projects** and select the project. The **Gitaly relative path** is displayed there and looks similar to: @@ -115,7 +115,7 @@ To look up a project's name using the Rails console: ``` The quoted string in that command is the directory tree you can find on your GitLab server. For -example, on a default Omnibus installation this would be `/var/opt/gitlab/git-data/repositories/@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git` +example, on a default Linux package installation this would be `/var/opt/gitlab/git-data/repositories/@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git` with `.git` from the end of the directory name removed. The output includes the project ID and the project name. For example: @@ -126,8 +126,8 @@ The output includes the project ID and the project name. For example: To look up a project's name using the `config` file in the `*.git` directory: -1. Navigate to the to the `*.git` directory. This directory is located in `/var/opt/gitlab/git-data/repositories/@hashed/`, where the first four - characters of the hash are the first two directories in the path under `@hashed/`. For example, on a default Omnibus GitLab installation the +1. Locate the `*.git` directory. This directory is located in `/var/opt/gitlab/git-data/repositories/@hashed/`, where the first four + characters of the hash are the first two directories in the path under `@hashed/`. For example, on a default Linux package installation the `*.git` directory of the hash `b17eb17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9` would be `/var/opt/gitlab/git-data/repositories/@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git`. 1. Open the `config` file and locate the `fullpath=` key under `[gitlab]`. diff --git a/doc/administration/restart_gitlab.md b/doc/administration/restart_gitlab.md index e86541b7ced..51a4fe40b16 100644 --- a/doc/administration/restart_gitlab.md +++ b/doc/administration/restart_gitlab.md @@ -12,12 +12,12 @@ its services. NOTE: A short downtime is expected for all methods. -## Omnibus installations +## Linux package installations -If you have used the [Omnibus packages](https://about.gitlab.com/install/) to install GitLab, +If you have used the [Linux package](https://about.gitlab.com/install/) to install GitLab, you should already have `gitlab-ctl` in your `PATH`. -`gitlab-ctl` interacts with the Omnibus packages and can be used to restart the +`gitlab-ctl` interacts with the Linux package installation and can be used to restart the GitLab Rails application (Puma) as well as the other components, like: - GitLab Workhorse @@ -28,10 +28,10 @@ GitLab Rails application (Puma) as well as the other components, like: - [Mailroom](reply_by_email.md) - Logrotate -### Omnibus GitLab restart +### Restart a Linux package installation There may be times in the documentation where you are asked to _restart_ -GitLab. To restart, run the following command: +GitLab. To restart a Linux package installation, run: ```shell sudo gitlab-ctl restart @@ -71,15 +71,14 @@ In that case, you can use `gitlab-ctl kill <service>` to send the `SIGKILL` signal to the service, for example `sidekiq`. After that, a restart should perform fine. -As a last resort, you can try to -[reconfigure GitLab](#omnibus-gitlab-reconfigure) instead. +As a last resort, you can try to reconfigure GitLab instead. -### Omnibus GitLab reconfigure +### Reconfigure a Linux package installation There may be times in the documentation where you are asked to _reconfigure_ -GitLab. Remember that this method applies only for the Omnibus packages. +GitLab. Remember that this method applies only for Linux package installations. -Reconfigure Omnibus GitLab with: +To reconfigure a Linux package installation, run: ```shell sudo gitlab-ctl reconfigure @@ -89,10 +88,10 @@ Reconfiguring GitLab should occur in the event that something in its configuration (`/etc/gitlab/gitlab.rb`) has changed. When you run `gitlab-ctl reconfigure`, [Chef](https://www.chef.io/products/chef-infra), -the underlying configuration management application that powers Omnibus GitLab, runs some checks. +the underlying configuration management application that powers Linux package installations, runs some checks. Chef ensures directories, permissions, and services are in place and working. -Chef also [restarts GitLab components](#how-to-restart-gitlab) if any of their configuration files have changed. +Chef also restarts GitLab components if any of their configuration files have changed. If you manually edit any files in `/var/opt/gitlab` that are managed by Chef, running `reconfigure` reverts the changes and restarts the services that diff --git a/doc/administration/secure_files.md b/doc/administration/secure_files.md new file mode 100644 index 00000000000..3c9d40c3e73 --- /dev/null +++ b/doc/administration/secure_files.md @@ -0,0 +1,201 @@ +--- +stage: Mobile +group: Mobile DevOps +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Secure Files administration **(FREE)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78227) in GitLab 14.8 [with a flag](feature_flags.md) named `ci_secure_files`. Disabled by default. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/350748) in GitLab 15.7. Feature flag `ci_secure_files` removed. + +You can securely store up to 100 files for use in CI/CD pipelines as secure files. +These files are stored securely outside of your project's repository and are not version controlled. +It is safe to store sensitive information in these files. Secure files support both plain text +and binary file types, and must be 5 MB or less. + +The storage location of these files can be configured using the options described below, +but the default locations are: + +- `/var/opt/gitlab/gitlab-rails/shared/ci_secure_files` for installations using the Linux package. +- `/home/git/gitlab/shared/ci_secure_files` for self-compiled installations. + +Use [external object storage](https://docs.gitlab.com/charts/advanced/external-object-storage/#lfs-artifacts-uploads-packages-external-diffs-terraform-state-dependency-proxy) +configuration for [GitLab Helm chart](https://docs.gitlab.com/charts/) installations. + +## Disabling Secure Files + +You can disable Secure Files across the entire GitLab instance. You might want to disable +Secure Files to reduce disk space, or to remove access to the feature. + +To disable Secure Files, follow the steps below according to your installation. + +Prerequisite: + +- You must be an administrator. + +**For Linux package installations** + +1. Edit `/etc/gitlab/gitlab.rb` and add the following line: + + ```ruby + gitlab_rails['ci_secure_files_enabled'] = false + ``` + +1. Save the file and reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +**For self-compiled installations** + +1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: + + ```yaml + ci_secure_files: + enabled: false + ``` + +1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. + +## Using local storage + +The default configuration uses local storage. To change the location where Secure Files +are stored locally, follow the steps below. + +**For Linux package installations** + +1. To change the storage path for example to `/mnt/storage/ci_secure_files`, edit + `/etc/gitlab/gitlab.rb` and add the following line: + + ```ruby + gitlab_rails['ci_secure_files_storage_path'] = "/mnt/storage/ci_secure_files" + ``` + +1. Save the file and [reconfigure GitLab](restart_gitlab.md#reconfigure-a-linux-package-installation) +1. Save the file and reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +**For self-compiled installations** + +1. To change the storage path for example to `/mnt/storage/ci_secure_files`, edit + `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: + + ```yaml + ci_secure_files: + enabled: true + storage_path: /mnt/storage/ci_secure_files + ``` + +1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) + for the changes to take effect. + +## Using object storage **(FREE SELF)** + +Instead of storing Secure Files on disk, you should use [one of the supported object storage options](object_storage.md#supported-object-storage-providers). +This configuration relies on valid credentials to be configured already. + +[Read more about using object storage with GitLab](object_storage.md). + +NOTE: +This feature is not supported by consolidated object storage configuration. +Adding support is proposed in [issue 414673](https://gitlab.com/gitlab-org/gitlab/-/issues/414673). + +### Object storage settings + +The following settings are: + +- Nested under `ci_secure_files:` and then `object_store:` on source installations. +- Prefixed by `ci_secure_files_object_store_` on Linux package installations. + +| Setting | Description | Default | +|---------|-------------|---------| +| `enabled` | Enable/disable object storage | `false` | +| `remote_directory` | The bucket name where Secure Files are stored | | +| `connection` | Various connection options described below | | + +### S3-compatible connection settings + +See [the available connection settings for different providers](object_storage.md#configure-the-connection-settings). + +**For Linux package installations:** + +1. Edit `/etc/gitlab/gitlab.rb` and add the following lines, but using + the values you want: + + ```ruby + gitlab_rails['ci_secure_files_object_store_enabled'] = true + gitlab_rails['ci_secure_files_object_store_remote_directory'] = "terraform" + gitlab_rails['ci_secure_files_object_store_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-central-1', + 'aws_access_key_id' => 'AWS_ACCESS_KEY_ID', + 'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY' + } + ``` + + NOTE: + If you are using AWS IAM profiles, be sure to omit the AWS access key and secret access key/value pairs: + + ```ruby + gitlab_rails['ci_secure_files_object_store_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-central-1', + 'use_iam_profile' => true + } + ``` + +1. Save the file and [reconfigure GitLab](restart_gitlab.md#reconfigure-a-linux-package-installation) +1. Save the file and reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. [Migrate any existing local states to the object storage](#migrate-to-object-storage) + +**For self-compiled installations** + +1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: + + ```yaml + ci_secure_files: + enabled: true + object_store: + enabled: true + remote_directory: "ci_secure_files" # The bucket name + connection: + provider: AWS # Only AWS supported at the moment + aws_access_key_id: AWS_ACCESS_KEY_ID + aws_secret_access_key: AWS_SECRET_ACCESS_KEY + region: eu-central-1 + ``` + +1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. +1. [Migrate any existing local states to the object storage](#migrate-to-object-storage) + +### Migrate to object storage + +> [Introduced](https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/readme/-/issues/125) in GitLab 16.1. + +WARNING: +It's not possible to migrate Secure Files from object storage back to local storage, +so proceed with caution. + +To migrate Secure Files to object storage, follow the instructions below. + +- For Linux package installations: + + ```shell + sudo gitlab-rake gitlab:ci_secure_files:migrate + ``` + +- For self-compiled installations: + + ```shell + sudo -u git -H bundle exec rake gitlab:ci_secure_files:migrate RAILS_ENV=production + ``` diff --git a/doc/administration/server_hooks.md b/doc/administration/server_hooks.md index b167412075b..82360581504 100644 --- a/doc/administration/server_hooks.md +++ b/doc/administration/server_hooks.md @@ -33,12 +33,12 @@ alternatives to server hooks include: :::TabTitle GitLab 15.11 and later -> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4629) in GitLab 15.11, `hooks set` command replaces direct file system access. +> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4629) in GitLab 15.11, `hooks set` command replaces direct file system access. Existing Git hooks don't need migrating for the `hooks set` command. Prerequisites: - The [storage name](gitaly/configure_gitaly.md#gitlab-requires-a-default-repository-storage), path to the Gitaly configuration file - (default is `/var/opt/gitlab/gitaly/config.toml` on Omnibus GitLab instances), and the + (default is `/var/opt/gitlab/gitaly/config.toml` on Linux package instances), and the [repository relative path](repository_storage_types.md#from-project-name-to-hashed-path) for the repository. To set server hooks for a repository: @@ -57,11 +57,13 @@ To set server hooks for a repository: 1. Ensure the server hook files are executable and do not match the backup file pattern (`*~`). The server hooks be in a `custom_hooks` directory that is at the root of the tarball. 1. Create the custom hooks archive with the tar command. For example, `tar -cf custom_hooks.tar custom_hooks`. -1. Run the `hooks set` command with required options to set the Git hooks for the repository. For example, +1. Run the `hooks set` subcommand with required options to set the Git hooks for the repository. For example, `cat hooks.tar | gitaly hooks set --storage <storage> --repository <relative path> --config <config path>`. - A path to a valid Gitaly configuration for the node is required to connect to the node and provided to the `--config` flag. - Custom hooks tarball must be passed via `stdin`. For example, `cat hooks.tar | gitaly hooks set --storage <storage> --repository <relative path> --config <config path>`. +1. If you are using Gitaly Cluster, you must run `hooks set` subcommand on all Gitaly nodes. For more information, see + [Server hooks on a Gitaly Cluster](#server-hooks-on-a-gitaly-cluster). If you implemented the server hook code correctly, it should execute when the Git hook is next triggered. @@ -69,15 +71,16 @@ If you implemented the server hook code correctly, it should execute when the Gi To create server hooks for a repository: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. Go to **Overview > Projects** and select the project you want to add a server hook to. 1. On the page that appears, locate the value of **Gitaly relative path**. This path is where server hooks must be located. - If you are using [hashed storage](repository_storage_types.md#hashed-storage), see [Translate hashed storage paths](repository_storage_types.md#translate-hashed-storage-paths) for information on interpreting the relative path. - If you are not using [hashed storage](repository_storage_types.md#hashed-storage): - - For Omnibus GitLab installations, the path is usually `/var/opt/gitlab/git-data/repositories/<group>/<project>.git`. - - For an installation from source, the path is usually `/home/git/repositories/<group>/<project>.git`. + - For Linux package installations, the path is usually `/var/opt/gitlab/git-data/repositories/<group>/<project>.git`. + - For self-compiled installations, the path is usually `/home/git/repositories/<group>/<project>.git`. 1. On the file system, create a new directory in the correct location called `custom_hooks`. 1. In the new `custom_hooks` directory: - To create a single server hook, create a file with a name that matches the hook type. For example, for a @@ -90,15 +93,18 @@ To create server hooks for a repository: example, if the script is in Ruby the shebang is probably `#!/usr/bin/env ruby`. 1. Ensure the hook file does not match the backup file pattern (`*~`). +1. If you are using Gitaly Cluster, you must repeat this process on all Gitaly nodes. For more information, see + [Server hooks on a Gitaly Cluster](#server-hooks-on-a-gitaly-cluster). If the server hook code is properly implemented, it should execute when the Git hook is next triggered. ::EndTabs -### Gitaly Cluster +### Server hooks on a Gitaly Cluster -If you use [Gitaly Cluster](gitaly/index.md), the scripts must be copied to every Gitaly node that has a replica of the repository. Every Gitaly node -needs a copy because any node can be made a primary at any time. Server hooks only run on primary nodes. +If you use [Gitaly Cluster](gitaly/index.md), an individual repository may be replicated to multiple Gitaly storages in Praefect. +Consequentially, the hook scripts must be copied to every Gitaly node that has a replica of the repository. +To accomplish this, follow the same steps for setting custom repository hooks for the applicable version and repeat for each storage. The location to copy the scripts to depends on where repositories are stored: @@ -123,7 +129,7 @@ To create a Git hook that applies to all repositories, set a global server hook. Before creating a global server hook, you must choose a directory for it. -For Omnibus GitLab installations, the directory is set in `gitlab.rb` under `gitaly['configuration'][:hooks][:custom_hooks_dir]`. You can either: +For Linux package installations, the directory is set in `gitlab.rb` under `gitaly['configuration'][:hooks][:custom_hooks_dir]`. You can either: - Use the default suggestion of the `/var/opt/gitlab/gitaly/custom_hooks` directory by uncommenting it. - Add your own setting. @@ -201,7 +207,7 @@ The following GitLab environment variables are supported for all server hooks: | Environment variable | Description | |:---------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------| -| `GL_ID` | GitLab identifier of user that initiated the push. For example, `user-2234`. | +| `GL_ID` | GitLab identifier of user or SSH key that initiated the push. For example, `user-2234` or `key-4`. | | `GL_PROJECT_PATH` | (GitLab 13.2 and later) GitLab project path. | | `GL_PROTOCOL` | (GitLab 13.2 and later) Protocol used for this change. One of: `http` (Git `push` using HTTP), `ssh` (Git `push` using SSH), or `web` (all other actions). | | `GL_REPOSITORY` | `project-<id>` where `id` is the ID of the project. | diff --git a/doc/administration/sidekiq/extra_sidekiq_processes.md b/doc/administration/sidekiq/extra_sidekiq_processes.md index 7959d1a5ce7..31e713bbf06 100644 --- a/doc/administration/sidekiq/extra_sidekiq_processes.md +++ b/doc/administration/sidekiq/extra_sidekiq_processes.md @@ -48,7 +48,8 @@ to all available queues: To view the Sidekiq processes in GitLab: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Background Jobs**. ## Concurrency diff --git a/doc/administration/sidekiq/index.md b/doc/administration/sidekiq/index.md index 6fb12aa6ef9..bb517e21fd0 100644 --- a/doc/administration/sidekiq/index.md +++ b/doc/administration/sidekiq/index.md @@ -278,7 +278,7 @@ To serve metrics via HTTPS instead of HTTP, enable TLS in the exporter settings: sidekiq['exporter_tls_key_path'] = "/path/to/private-key.pem" ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. When TLS is enabled, the same `port` and `address` are used as described above. diff --git a/doc/administration/sidekiq/processing_specific_job_classes.md b/doc/administration/sidekiq/processing_specific_job_classes.md index 26c2804f130..696b0b9444c 100644 --- a/doc/administration/sidekiq/processing_specific_job_classes.md +++ b/doc/administration/sidekiq/processing_specific_job_classes.md @@ -25,10 +25,6 @@ Both of these use the same [worker matching query](#worker-matching-query) syntax. While they can technically be used together, most deployments should choose one or the other; there is no particular benefit in combining them. -Routing rules must be the same across all GitLab nodes as they are part of the -application configuration. Queue selectors can be different across GitLab nodes -because they only change the arguments to the launched Sidekiq process. - ## Routing rules > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/59604) in GitLab 13.12. @@ -61,6 +57,12 @@ the migration to avoid losing jobs entirely, especially in a system with long queues of jobs. The migration can be done by following the migration steps mentioned in [Sidekiq job migration](sidekiq_job_migration.md). +### Routing rules in a scaled architecture + +Routing rules must be the same across all GitLab nodes (especially GitLab Rails and Sidekiq nodes) as they are part of the +application configuration. Queue selectors can be different across GitLab nodes +because they only change the arguments to the launched Sidekiq process. + ### Detailed example This is a comprehensive example intended to show different possibilities. It is diff --git a/doc/administration/sidekiq/sidekiq_job_migration.md b/doc/administration/sidekiq/sidekiq_job_migration.md index 1332d9b35d8..89da174eb34 100644 --- a/doc/administration/sidekiq/sidekiq_job_migration.md +++ b/doc/administration/sidekiq/sidekiq_job_migration.md @@ -17,7 +17,7 @@ If the Sidekiq routing rules are changed, administrators need to take care with 1. Listen to both the old and new queues. 1. Update the routing rules. -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Run the [Rake tasks for migrating queued and future jobs](#migrate-queued-and-future-jobs). 1. Stop listening to the old queues. diff --git a/doc/administration/sidekiq/sidekiq_troubleshooting.md b/doc/administration/sidekiq/sidekiq_troubleshooting.md index 8b95a9f6f0a..315714cd00b 100644 --- a/doc/administration/sidekiq/sidekiq_troubleshooting.md +++ b/doc/administration/sidekiq/sidekiq_troubleshooting.md @@ -494,6 +494,41 @@ has number of drawbacks, as mentioned in [Why Ruby's Timeout is dangerous (and T > > Nobody writes code to defend against an exception being raised on literally any line. That's not even possible. So Thread.raise is basically like a sneak attack on your code that could result in almost anything. It would probably be okay if it were pure-functional code that did not modify any state. But this is Ruby, so that's unlikely :) +## Manually trigger a cron job + +By visiting `/admin/background_jobs`, you can look into what jobs are scheduled/running/pending on your instance. + +You can trigger a cron job from the UI by selecting the "Enqueue Now" button. To trigger a cron job programmatically first open a [Rails console](../operations/rails_console.md). + +To find the cron job you want to test: + +```irb +job = Sidekiq::Cron::Job.find('job-name') + +# get status of job: +job.status + +# enqueue job right now! +job.enque! +``` + +For example, to trigger the `update_all_mirrors_worker` cron job that updates the repository mirrors: + +```irb +irb(main):001:0> job = Sidekiq::Cron::Job.find('update_all_mirrors_worker') +=> +#<Sidekiq::Cron::Job:0x00007f147f84a1d0 +... +irb(main):002:0> job.status +=> "enabled" +irb(main):003:0> job.enque! +=> 257 +``` + +The list of available jobs can be found in the [workers](https://gitlab.com/gitlab-org/gitlab/-/tree/master/app/workers) directory. + +For more information about Sidekiq jobs, see the [Sidekiq-cron](https://github.com/sidekiq-cron/sidekiq-cron#work-with-job) documentation. + ## Omnibus GitLab 14.0 and later: remove the `sidekiq-cluster` service Omnibus GitLab instances that were configured to run `sidekiq-cluster` prior to GitLab 14.0 @@ -550,3 +585,7 @@ This change might reduce the amount of work Sidekiq can do. Symptoms like delays indicate that additional Sidekiq processes would be beneficial. Consider [adding additional Sidekiq processes](extra_sidekiq_processes.md) to compensate for removing the `sidekiq-cluster` service. + +## Related topics + +- [Elasticsearch workers overload Sidekiq](../../integration/advanced_search/elasticsearch_troubleshooting.md#elasticsearch-workers-overload-sidekiq). diff --git a/doc/administration/silent_mode/index.md b/doc/administration/silent_mode/index.md index e98aaaf4e0a..49c95d75768 100644 --- a/doc/administration/silent_mode/index.md +++ b/doc/administration/silent_mode/index.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GitLab Silent Mode (Experiment) **(FREE SELF)** -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/9826) in GitLab 15.11. This feature is an [Experiment](../../policy/alpha-beta-support.md#experiment). +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/9826) in GitLab 15.11. This feature is an [Experiment](../../policy/experiment-beta-support.md#experiment). Silent Mode allows you to suppress outbound communication, such as emails, from GitLab. Silent Mode is not intended to be used on environments which are in-use. Two use-cases are: @@ -33,6 +33,8 @@ There are two ways to enable Silent Mode: ::Gitlab::CurrentSettings.update!(silent_mode_enabled: true) ``` +It may take up to a minute to take effect. [Issue 405433](https://gitlab.com/gitlab-org/gitlab/-/issues/405433) proposes removing this delay. + ## Disable Silent Mode Prerequisites: @@ -53,12 +55,26 @@ There are two ways to disable Silent Mode: ::Gitlab::CurrentSettings.update!(silent_mode_enabled: false) ``` +It may take up to a minute to take effect. [Issue 405433](https://gitlab.com/gitlab-org/gitlab/-/issues/405433) proposes removing this delay. + ## Behavior of GitLab features in Silent Mode +This section documents the current behavior of GitLab when Silent Mode is enabled. While Silent Mode is an Experiment, the behavior may change without notice. The work for the first iteration of Silent Mode is tracked by [Epic 9826](https://gitlab.com/groups/gitlab-org/-/epics/9826). + ### Service Desk Incoming emails still raise issues, but the users who sent the emails to [Service Desk](../../user/project/service_desk.md) are not notified of issue creation or comments on their issues. +### Webhooks + +[Project and group webhooks](../../user/project/integrations/webhooks.md), and [system hooks](../system_hooks.md) are suppressed. The relevant Sidekiq jobs fail 4 times and then disappear, while Silent Mode is enabled. [Issue 393639](https://gitlab.com/gitlab-org/gitlab/-/issues/393639) discusses preventing the Sidekiq jobs from running in the first place. + +Triggering webhook tests via the UI results in HTTP status 500 responses. + ### Outbound emails -Outbound emails are suppressed. It may take up to a minute to take effect after enabling Silent Mode. [Issue 405433](https://gitlab.com/gitlab-org/gitlab/-/issues/405433) proposes removing this delay. +Outbound emails are suppressed. + +### Outbound HTTP requests + +Many outbound HTTP requests are suppressed. A list of unsuppressed requests does not exist at this time, since more suppression is planned. diff --git a/doc/administration/smime_signing_email.md b/doc/administration/smime_signing_email.md index 4e49eef44df..5748c4d32d4 100644 --- a/doc/administration/smime_signing_email.md +++ b/doc/administration/smime_signing_email.md @@ -30,7 +30,7 @@ WARNING: Be mindful of the access levels for your private keys and visibility to third parties. -**For Omnibus installations:** +For Linux package installations: 1. Edit `/etc/gitlab/gitlab.rb` and adapt the file paths: @@ -42,11 +42,11 @@ third parties. gitlab_rails['gitlab_email_smime_ca_certs_file'] = '/etc/gitlab/ssl/gitlab_smime_cas.crt' ``` -1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and [reconfigure GitLab](restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. The key must be readable by the GitLab system user (`git` by default). -**For installations from source:** +For self-compiled installations: 1. Edit `config/gitlab.yml`: diff --git a/doc/administration/static_objects_external_storage.md b/doc/administration/static_objects_external_storage.md index 73f44ed3889..c7a22da38de 100644 --- a/doc/administration/static_objects_external_storage.md +++ b/doc/administration/static_objects_external_storage.md @@ -16,7 +16,8 @@ storage such as a content delivery network (CDN). To configure external storage for static objects: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Repository**. 1. Expand the **External storage for repository static objects** section. 1. Enter the base URL and an arbitrary token. When you [set up external storage](#set-up-external-storage), diff --git a/doc/administration/system_hooks.md b/doc/administration/system_hooks.md index 8d343e7c541..dcacacaf47b 100644 --- a/doc/administration/system_hooks.md +++ b/doc/administration/system_hooks.md @@ -54,7 +54,8 @@ for Push and Tag events, but we never display commits. To create a system hook: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **System Hooks**. 1. Provide the **URL** and **Secret Token**. 1. Select the checkbox next to each optional **Trigger** you want to enable. diff --git a/doc/administration/terraform_state.md b/doc/administration/terraform_state.md index 3ac9a5fdce8..84f596dacf2 100644 --- a/doc/administration/terraform_state.md +++ b/doc/administration/terraform_state.md @@ -13,7 +13,7 @@ files. The files are encrypted before being stored. This feature is enabled by d The storage location of these files defaults to: -- `/var/opt/gitlab/gitlab-rails/shared/terraform_state` for Omnibus GitLab installations. +- `/var/opt/gitlab/gitlab-rails/shared/terraform_state` for Linux package installations. - `/home/git/gitlab/shared/terraform_state` for source installations. These locations can be configured using the options described below. @@ -40,7 +40,7 @@ Prerequisite: - You must be an administrator. -**In Omnibus installations:** +For Linux package installations: 1. Edit `/etc/gitlab/gitlab.rb` and add the following line: @@ -48,9 +48,9 @@ Prerequisite: gitlab_rails['terraform_state_enabled'] = false ``` -1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and [reconfigure GitLab](restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. -**In installations from source:** +For self-compiled installations: 1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: @@ -66,7 +66,7 @@ Prerequisite: The default configuration uses local storage. To change the location where Terraform state files are stored locally, follow the steps below. -**In Omnibus installations:** +For Linux package installations: 1. To change the storage path for example to `/mnt/storage/terraform_state`, edit `/etc/gitlab/gitlab.rb` and add the following line: @@ -75,9 +75,9 @@ Terraform state files are stored locally, follow the steps below. gitlab_rails['terraform_state_storage_path'] = "/mnt/storage/terraform_state" ``` -1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and [reconfigure GitLab](restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. -**In installations from source:** +For self-compiled installations: 1. To change the storage path for example to `/mnt/storage/terraform_state`, edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: @@ -103,7 +103,7 @@ This configuration relies on valid credentials to be configured already. The following settings are: - Nested under `terraform_state:` and then `object_store:` on source installations. -- Prefixed by `terraform_state_object_store_` on Omnibus GitLab installations. +- Prefixed by `terraform_state_object_store_` on Linux package installations. | Setting | Description | Default | |---------|-------------|---------| @@ -120,15 +120,15 @@ It's not possible to migrate Terraform state files from object storage back to l so proceed with caution. [An issue exists](https://gitlab.com/gitlab-org/gitlab/-/issues/350187) to change this behavior. -To migrate Terraform state files to object storage, follow the instructions below. +To migrate Terraform state files to object storage: -- For Omnibus package installations: +- For Linux package installations: ```shell gitlab-rake gitlab:terraform_states:migrate ``` -- For source installations: +- For self-compiled installations: ```shell sudo -u git -H bundle exec rake gitlab:terraform_states:migrate RAILS_ENV=production @@ -152,9 +152,9 @@ For GitLab 13.8 and earlier versions, you can use a workaround for the Rake task You can optionally track progress and verify that all Terraform state files migrated successfully using the [PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database): -- `sudo gitlab-rails dbconsole` for Omnibus GitLab 14.1 and earlier. -- `sudo gitlab-rails dbconsole --database main` for Omnibus GitLab 14.2 and later. -- `sudo -u git -H psql -d gitlabhq_production` for source-installed instances. +- `sudo gitlab-rails dbconsole` for Linux package installations running GitLab 14.1 and earlier. +- `sudo gitlab-rails dbconsole --database main` for Linux package installations running GitLab 14.2 and later. +- `sudo -u git -H psql -d gitlabhq_production` for self-compiled installations. Verify `objectstg` below (where `file_store=2`) has count of all states: @@ -207,7 +207,7 @@ See [the available connection settings for different providers](object_storage.m } ``` -1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and [reconfigure GitLab](restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. [Migrate any existing local states to the object storage](#migrate-to-object-storage) **In installations from source:** diff --git a/doc/administration/timezone.md b/doc/administration/timezone.md index b0dc995c684..e0f70cecb43 100644 --- a/doc/administration/timezone.md +++ b/doc/administration/timezone.md @@ -18,12 +18,13 @@ Uncomment and customize if you want to change the default time zone of the GitLa To see all available time zones, run `bundle exec rake time:zones:all`. -For Omnibus installations, run `gitlab-rake time:zones:all`. +For Linux package installations, run `gitlab-rake time:zones:all`. NOTE: -This Rake task does not list time zones in TZInfo format required by Omnibus GitLab during a reconfigure: [#27209](https://gitlab.com/gitlab-org/gitlab/-/issues/27209). +This Rake task does not list time zones in TZInfo format required by a Linux package installation during a reconfigure. For more information, +see [issue 27209](https://gitlab.com/gitlab-org/gitlab/-/issues/27209). -## Changing time zone in Omnibus installations +## Changing time zone in Linux package installations GitLab defaults its time zone to UTC. It has a global time zone configuration parameter in `/etc/gitlab/gitlab.rb`. diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index ddee79046f6..fc319fad3e8 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -92,4 +92,4 @@ Moved to [Geo replication troubleshooting](../geo/replication/troubleshooting.md ## Generate Service Ping -This content has been moved to [Service Ping Troubleshooting](../../development/service_ping/troubleshooting.md). +This content has been moved to [Service Ping Troubleshooting](../../development/internal_analytics/service_ping/troubleshooting.md). diff --git a/doc/administration/troubleshooting/postgresql.md b/doc/administration/troubleshooting/postgresql.md index d5288bfead8..120d566a7e7 100644 --- a/doc/administration/troubleshooting/postgresql.md +++ b/doc/administration/troubleshooting/postgresql.md @@ -135,7 +135,7 @@ postgresql['statement_timeout'] = '15s' postgresql['idle_in_transaction_session_timeout'] = '60s' ``` -Once saved, [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +Once saved, [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. NOTE: These are Omnibus GitLab settings. If an external database, such as a customer's PostgreSQL installation or Amazon RDS is being used, these values don't get set, and would have to be set externally. @@ -148,7 +148,7 @@ The following advice does not apply in case because the changed timeout might affect more transactions than intended. In some situations, it may be desirable to set a different statement timeout -without having to [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure), +without having to [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation), which in this case would restart Puma and Sidekiq. For example, a backup may fail with the following errors in the output of the @@ -190,6 +190,12 @@ This error likely means that `autovacuum` is failing to complete its run: ERROR: database is not accepting commands to avoid wraparound data loss in database "gitlabhq_production" ``` +Or + +```plaintext + ERROR: failed to re-find parent key in index "XXX" for deletion target page XXX +``` + To resolve the error, run `VACUUM` manually: 1. Stop GitLab with the command `gitlab-ctl stop`. diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index ab900d7ea9f..99f19821916 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -23,7 +23,7 @@ For historical reasons, uploads for the whole instance (for example the [favicon which by default is `uploads/-/system`. Changing the base directory on an existing GitLab installation is strongly discouraged. -**In Omnibus GitLab installations:** +For Linux package installations: _The uploads are stored by default in `/var/opt/gitlab/gitlab-rails/uploads`._ @@ -36,9 +36,9 @@ _The uploads are stored by default in `/var/opt/gitlab/gitlab-rails/uploads`._ This setting only applies if you haven't changed the `gitlab_rails['uploads_storage_path']` directory. -1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and [reconfigure GitLab](restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. -**In installations from source:** +For self-compiled installations: _The uploads are stored by default in `/home/git/gitlab/public/uploads`._ @@ -68,7 +68,8 @@ In GitLab 13.2 and later, you should use the [consolidated object storage settings](object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form). This section describes the earlier configuration format. -For source installations the following settings are nested under `uploads:` and then `object_store:`. On Omnibus GitLab installs they are prefixed by `uploads_object_store_`. +For self-compiled installations, the following settings are nested under `uploads:` and then `object_store:`. On Linux +package installations, they are prefixed by `uploads_object_store_`. | Setting | Description | Default | |---------|-------------|---------| @@ -81,7 +82,7 @@ For source installations the following settings are nested under `uploads:` and See [the available connection settings for different providers](object_storage.md#configure-the-connection-settings). -**In Omnibus installations:** +For Linux package installations: _The uploads are stored by default in `/var/opt/gitlab/gitlab-rails/uploads`._ @@ -110,10 +111,10 @@ _The uploads are stored by default in } ``` -1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and [reconfigure GitLab](restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. 1. Migrate any existing local uploads to the object storage with the [`gitlab:uploads:migrate:all` Rake task](raketasks/uploads/migrate.md). -**In installations from source:** +For self-compiled installations: _The uploads are stored by default in `/home/git/gitlab/public/uploads`._ diff --git a/doc/administration/user_settings.md b/doc/administration/user_settings.md index b02e1c7244b..c4b00d05f9d 100644 --- a/doc/administration/user_settings.md +++ b/doc/administration/user_settings.md @@ -18,9 +18,9 @@ ability to create top-level groups (does not affect existing users' setting), Gi - The [application setting API](../api/settings.md#change-application-settings). - In GitLab 15.4 and earlier, in a configuration file by following the steps in this section. -To disable new users' ability to create top-level groups using the configuration file: +To disable new users' ability to create top-level groups using the configuration file. -**Omnibus GitLab installations** +For Linux package installations: 1. Edit `/etc/gitlab/gitlab.rb` and add the following line: @@ -28,9 +28,9 @@ To disable new users' ability to create top-level groups using the configuration gitlab_rails['gitlab_default_can_create_group'] = false ``` -1. [Reconfigure and restart GitLab](restart_gitlab.md#omnibus-installations). +1. [Reconfigure and restart GitLab](restart_gitlab.md#reconfigure-a-linux-package-installation). -**Source installations** +For self-compiled installations: 1. Edit `config/gitlab.yml` and uncomment the following line: @@ -50,9 +50,9 @@ Administrators can: ## Prevent users from changing their usernames By default, new users can change their usernames. To disable your users' -ability to change their usernames: +ability to change their usernames. -**Omnibus GitLab installations** +For Linux package installations: 1. Edit `/etc/gitlab/gitlab.rb` and add the following line: @@ -60,9 +60,9 @@ ability to change their usernames: gitlab_rails['gitlab_username_changing_enabled'] = false ``` -1. [Reconfigure and restart GitLab](restart_gitlab.md#omnibus-installations). +1. [Reconfigure and restart GitLab](restart_gitlab.md#reconfigure-a-linux-package-installation). -**Source installations** +For self-compiled installations: 1. Edit `config/gitlab.yml` and uncomment the following line: diff --git a/doc/administration/whats-new.md b/doc/administration/whats-new.md index d9c8a991577..b34b054d87c 100644 --- a/doc/administration/whats-new.md +++ b/doc/administration/whats-new.md @@ -23,7 +23,7 @@ All users can see the feature list, but the entries might differ depending on th To access the **What's new** feature: -1. On the top bar, select the **{question}** icon. +1. On the left sidebar, at the bottom, select **Help** (**{question}**). 1. Select **What's new** from the menu. ## Configure What's new @@ -31,7 +31,8 @@ To access the **What's new** feature: You can configure **What's new** to display features based on the tier, or you can hide it. To configure it: -1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Preferences**. 1. Expand **What's new**, and choose one of the following options: diff --git a/doc/administration/wikis/index.md b/doc/administration/wikis/index.md index 330e41ee880..540e50d5c70 100644 --- a/doc/administration/wikis/index.md +++ b/doc/administration/wikis/index.md @@ -82,6 +82,51 @@ so you should keep your wiki repositories as compact as possible. For more information about tools to compact repositories, read the documentation on [reducing repository size](../../user/project/repository/reducing_the_repo_size_using_git.md). +## Allow URI includes for AsciiDoc + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/348687) in GitLab 16.1. + +Include directives import content from separate pages or external URLs, +and display them as part of the content of the current document. To enable +AsciiDoc includes, enable the feature through the Rails console or the API. + +### Through the Rails console + +To configure this setting through the Rails console: + +1. Start the Rails console: + + ```shell + # For Omnibus installations + sudo gitlab-rails console + + # For installations from source + sudo -u git -H bundle exec rails console -e production + ``` + +1. Update the wiki to allow URI includes for AsciiDoc: + + ```ruby + ApplicationSetting.first.update!(wiki_asciidoc_allow_uri_includes: true) + ``` + +To check if includes are enabled, start the Rails console and run: + + ```ruby + Gitlab::CurrentSettings.wiki_asciidoc_allow_uri_includes + ``` + +### Through the API + +To set the wiki to allow URI includes for AsciiDoc through the +[Application Settings API](../../api/settings.md#change-application-settings), +use a `curl` command: + +```shell +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/application/settings?wiki_asciidoc_allow_uri_includes=true" +``` + ## Related topics - [User documentation for wikis](../../user/project/wiki/index.md) |