diff options
Diffstat (limited to 'doc/administration')
130 files changed, 4583 insertions, 5173 deletions
diff --git a/doc/administration/application_settings_cache.md b/doc/administration/application_settings_cache.md index 6c58e6886c4..30fd9ab85a8 100644 --- a/doc/administration/application_settings_cache.md +++ b/doc/administration/application_settings_cache.md @@ -1,6 +1,6 @@ --- stage: Data Stores -group: Memory +group: Application Performance info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/audit_event_streaming.md b/doc/administration/audit_event_streaming.md index 817f22debbc..3bb0ce41861 100644 --- a/doc/administration/audit_event_streaming.md +++ b/doc/administration/audit_event_streaming.md @@ -6,44 +6,53 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Audit event streaming **(ULTIMATE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332747) in GitLab 14.5 [with a flag](../administration/feature_flags.md) named `ff_external_audit_events_namespace`. Disabled by default. -> - [Enabled on GitLab.com and by default on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/338939) in GitLab 14.7. -> - [Feature flag `ff_external_audit_events_namespace`](https://gitlab.com/gitlab-org/gitlab/-/issues/349588) removed in GitLab 14.8. +> - API [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332747) in GitLab 14.5 [with a flag](../administration/feature_flags.md) named `ff_external_audit_events_namespace`. Disabled by default. +> - API [Enabled on GitLab.com and by default on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/338939) in GitLab 14.7. +> - API [Feature flag `ff_external_audit_events_namespace`](https://gitlab.com/gitlab-org/gitlab/-/issues/349588) removed in GitLab 14.8. +> - UI [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336411) in GitLab 14.9. > - [Subgroup events recording](https://gitlab.com/gitlab-org/gitlab/-/issues/366878) fixed in GitLab 15.2. - -Users can set an HTTP endpoint for a top-level group to receive all audit events about the group, its subgroups, and +> - Custom HTTP headers API [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/361216) in GitLab 15.1 [with a flag](feature_flags.md) named `streaming_audit_event_headers`. Disabled by default. +> - Custom HTTP headers API [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/362941) in GitLab 15.2. +> - Custom HTTP headers API [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/366524) in GitLab 15.3. [Feature flag `streaming_audit_event_headers`](https://gitlab.com/gitlab-org/gitlab/-/issues/362941) removed. +> - Custom HTTP headers UI [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/361630) in GitLab 15.2 [with a flag](feature_flags.md) named `custom_headers_streaming_audit_events_ui`. Disabled by default. +> - Custom HTTP headers UI [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/365259) in GitLab 15.3. [Feature flag `custom_headers_streaming_audit_events_ui`](https://gitlab.com/gitlab-org/gitlab/-/issues/365259) removed. +> - [Improved user experience](https://gitlab.com/gitlab-org/gitlab/-/issues/367963) in GitLab 15.4. + +Users can set a streaming destination for a top-level group to receive all audit events about the group, its subgroups, and projects as structured JSON. Top-level group owners can manage their audit logs in third-party systems. Any service that can receive -structured JSON data can be used as the endpoint. +structured JSON data can be used as the streaming destination. + +Each streaming destination can have up to 20 custom HTTP headers included with each streamed event. NOTE: GitLab can stream a single event more than once to the same destination. Use the `id` key in the payload to deduplicate incoming data. -## Add a new event streaming destination +## Add a new streaming destination WARNING: -Event streaming destinations receive **all** audit event data, which could include sensitive information. Make sure you trust the destination endpoint. +Streaming destinations receive **all** audit event data, which could include sensitive information. Make sure you trust the streaming destination. ### Use the GitLab UI -Users with at least the Owner role for a group can add event streaming destinations for it: +Users with the Owner role for a group can add streaming destinations for it: 1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **Security & Compliance > Audit events**. 1. On the main area, select **Streams** tab. - - When the destination list is empty, select **Add stream** to show the section for adding destinations. - - When the destination list is not empty, select **{plus}** to show the section for adding destinations. -1. Enter the destination URL to add and select **Add**. - -Event streaming is enabled if: - -- No warning is shown. -- The added endpoint is displayed in the UI. +1. Select **Add streaming destination** to show the section for adding destinations. +1. Enter the destination URL to add. +1. Optional. Locate the **Custom HTTP headers** table. +1. Ignore the **Active** checkbox because it isn't functional. To track progress on adding functionality to the **Active** checkbox, see the + [relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/361925). +1. Select **Add header** to create a new name and value pair. Enter as many name and value pairs as required. You can add up to + 20 headers per streaming destination. +1. After all headers have been filled out, select **Add** to add the new streaming destination. ### Use the API -To enable event streaming and add a destination, users with at least the Owner role for a group must use the +To enable streaming and add a destination, users with the Owner role for a group must use the `externalAuditEventDestinationCreate` mutation in the GraphQL API. ```graphql @@ -51,6 +60,7 @@ mutation { externalAuditEventDestinationCreate(input: { destinationUrl: "https://mydomain.io/endpoint/ingest", groupPath: "my-group" } ) { errors externalAuditEventDestination { + id destinationUrl verificationToken group { @@ -66,23 +76,35 @@ Event streaming is enabled if: - The returned `errors` object is empty. - The API responds with `200 OK`. +Group owners can add an HTTP header using the GraphQL `auditEventsStreamingHeadersCreate` mutation. You can retrieve the destination ID +by [listing all the streaming destinations](#use-the-api-1) for the group or from the mutation above. + +```graphql +mutation { + auditEventsStreamingHeadersCreate(input: { destinationId: "gid://gitlab/AuditEvents::ExternalAuditEventDestination/24601", key: "foo", value: "bar" }) { + errors + } +} +``` + +The header is created if the returned `errors` object is empty. + ## List streaming destinations -Users with at least the Owner role for a group can list event streaming destinations. +Users with the Owner role for a group can list streaming destinations. ### Use the GitLab UI -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336411) in GitLab 14.9. - -Users with at least the Owner role for a group can list event streaming destinations: +To list the streaming destinations: 1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **Security & Compliance > Audit events**. 1. On the main area, select **Streams** tab. +1. To the right of the item, select **Edit** (**{pencil}**) to see all the custom HTTP headers. ### Use the API -Users with at least the Owner role for a group can view a list of event streaming destinations at any time using the +Users with the Owner role for a group can view a list of streaming destinations at any time using the `externalAuditEventDestinations` query type. ```graphql @@ -94,42 +116,48 @@ query { destinationUrl verificationToken id + headers { + nodes { + key + value + id + } + } } } } } ``` -If the resulting list is empty, then audit event streaming is not enabled for that group. +If the resulting list is empty, then audit streaming is not enabled for that group. -## Delete streaming destinations +You need the ID values returned by this query for the update and delete mutations. -Users with at least the Owner role for a group can delete event streaming destinations using the -`deleteAuditEventDestinations` mutation type. +## Update streaming destinations -When the last destination is successfully deleted, event streaming is disabled for the group. +Users with the Owner role for a group can update streaming destinations. ### Use the GitLab UI -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336411) in GitLab 14.9. - -Users with at least the Owner role for a group can delete event streaming destinations. +To update a streaming destinations custom HTTP headers: 1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **Security & Compliance > Audit events**. 1. On the main area, select **Streams** tab. -1. Select **{remove}** at the right side of each item. - -The external streaming destination is deleted when: - -- No warning is shown. -- The deleted endpoint is not displayed in the UI. +1. To the right of the item, select **Edit** (**{pencil}**). +1. Locate the **Custom HTTP headers** table. +1. Locate the header that you wish to update. +1. Ignore the **Active** checkbox because it isn't functional. To track progress on adding functionality to the **Active** checkbox, see the + [relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/361925). +1. Select **Add header** to create a new name and value pair. Enter as many name and value pairs as required. You can add up to + 20 headers per streaming destination. +1. Select **Save** to update the streaming destination. ### Use the API -Delete an event streaming destination by specifying an ID. Get the required ID by -[listing the details](audit_event_streaming.md#use-the-api-1) of event -streaming destinations. +Users with the Owner role for a group can update streaming destinations custom HTTP headers using the +`auditEventsStreamingHeadersUpdate` mutation type. You can retrieve the custom HTTP headers ID +by [listing all the custom HTTP headers](#use-the-api-1) for the group. ```graphql mutation { @@ -139,89 +167,71 @@ mutation { } ``` -Destination is deleted if: +Streaming destination is updated if: - The returned `errors` object is empty. - The API responds with `200 OK`. -## Custom HTTP headers - -> - API [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/361216) in GitLab 15.1 [with a flag](feature_flags.md) named `streaming_audit_event_headers`. Disabled by default. -> - API [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/362941) in GitLab 15.2. -> - UI [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/361630) in GitLab 15.2 [with a flag](feature_flags.md) named `custom_headers_streaming_audit_events_ui`. Disabled by default. - -FLAG: -On self-managed GitLab, by default the API for this feature is available. To hide the feature, ask an administrator to [disable the feature flag](../administration/feature_flags.md) named `streaming_audit_event_headers`. -On GitLab.com, the API for this feature is available. - -Each streaming destination can have up to 20 custom HTTP headers included with each streamed event. - -### Adding custom HTTP headers - -Add custom HTTP headers with the API or GitLab UI. - -#### Use the API - -Group owners can add a HTTP header using the GraphQL `auditEventsStreamingHeadersCreate` mutation. You can retrieve the destination ID -by [listing the external audit destinations](#list-streaming-destinations) on the group. +Group owners can remove an HTTP header using the GraphQL `auditEventsStreamingHeadersDestroy` mutation. You can retrieve the header ID +by [listing all the custom HTTP headers](#use-the-api-1) for the group. ```graphql mutation { - auditEventsStreamingHeadersCreate(input: { destinationId: "gid://gitlab/AuditEvents::ExternalAuditEventDestination/24601", key: "foo", value: "bar" }) { + auditEventsStreamingHeadersDestroy(input: { headerId: "gid://gitlab/AuditEvents::Streaming::Header/1" }) { errors } } ``` -The header is created if the returned `errors` object is empty. +The header is deleted if the returned `errors` object is empty. -#### Use the GitLab UI +## Delete streaming destinations -FLAG: -On self-managed GitLab, by default the UI for this feature is not available. To make it available per group, ask an administrator to -[enable the feature flag](../administration/feature_flags.md) named `custom_headers_streaming_audit_events_ui`. On GitLab.com, the UI for this feature is -not available. The UI for this feature is not ready for production use. +Users with the Owner role for a group can delete streaming destinations. -Users with at least the Owner role for a group can add event streaming destinations and custom HTTP headers for it: +When the last destination is successfully deleted, streaming is disabled for the group. + +### Use the GitLab UI + +To delete a streaming destination: 1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **Security & Compliance > Audit events**. -1. On the main area, select **Streams** tab. - - When the destination list is empty, select **Add stream** to show the section for adding destinations. - - When the destination list is not empty, select **{plus}** to show the section for adding destinations. -1. Enter the destination URL to add. -1. Locate the **Custom HTTP headers** table. -1. In the **Header** column, add the header's name. -1. In the **Value** column, add the header's value. -1. Ignore the **Active** checkbox because it isn't functional. To track progress on adding functionality to the **Active** checkbox, see the - [relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/361925). -1. Enter as many name and value pairs as required. When you enter a unique name and a value for a header, a new row in the table automatically appears. You can add up to - 20 headers per endpoint. -1. After all headers have been filled out, select **Add** to add the new endpoint. - -Event streaming is enabled if: +1. On the main area, select the **Streams** tab. +1. To the right of the item, select **Delete** (**{remove}**). -- No warning is shown. -- The added endpoint is displayed in the UI. +To delete only the custom HTTP headers for a streaming destination: -### Updating custom HTTP headers +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Security & Compliance > Audit events**. +1. On the main area, select the **Streams** tab. +1. To the right of the item, **Edit** (**{pencil}**). +1. Locate the **Custom HTTP headers** table. +1. Locate the header that you wish to remove. +1. To the right of the header, select **Delete** (**{remove}**). +1. Select **Save** to update the streaming destination. -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/361964) in GitLab 15.2. +### Use the API -Group owners can update a HTTP header using the GraphQL `auditEventsStreamingHeadersCreate` mutation. +Users with the Owner role for a group can delete streaming destinations using the +`externalAuditEventDestinationDestroy` mutation type. You can retrieve the destinations ID +by [listing all the streaming destinations](#use-the-api-1) for the group. ```graphql mutation { - auditEventsStreamingHeadersUpdate(input: { headerId: "gid://gitlab/AuditEvents::Streaming::Header/24255", key: "new-foo", value: "new-bar" }) { + externalAuditEventDestinationDestroy(input: { id: destination }) { errors } } ``` -### Deleting custom HTTP headers +Streaming destination is deleted if: -Group owners can remove a HTTP header using the GraphQL `auditEventsStreamingHeadersDestroy` mutation. You can retrieve the header ID -by [listing all the custom headers](#list-all-custom-headers) on the group. +- The returned `errors` object is empty. +- The API responds with `200 OK`. + +Group owners can remove an HTTP header using the GraphQL `auditEventsStreamingHeadersDestroy` mutation. You can retrieve the header ID +by [listing all the custom HTTP headers](#use-the-api-1) for the group. ```graphql mutation { @@ -233,52 +243,6 @@ mutation { The header is deleted if the returned `errors` object is empty. -### List all custom headers - -List all custom HTTP headers with the API or GitLab UI. - -#### Use the API - -You can list all custom headers for a top-level group as well as their value and ID using the GraphQL `externalAuditEventDestinations` query. The ID -value returned by this query is what you need to pass to the `deletion` mutation. - -```graphql -query { - group(fullPath: "your-group") { - id - externalAuditEventDestinations { - nodes { - destinationUrl - id - headers { - nodes { - key - value - id - } - } - } - } - } -} -``` - -#### Use the GitLab UI - -FLAG: -On self-managed GitLab, by default the UI for this feature is not available. To make it available per group, ask an administrator to -[enable the feature flag](../administration/feature_flags.md) named `custom_headers_streaming_audit_events_ui`. On GitLab.com, the UI for this feature is -not available. The UI for this feature is not ready for production use. - -Users with at least the Owner role for a group can add event streaming destinations and custom HTTP headers for it: - -1. On the top bar, select **Menu > Groups** and find your group. -1. On the left sidebar, select **Security & Compliance > Audit events**. -1. On the main area, select **Streams** tab. -1. Select **{pencil}** at the right side of an item. -1. A read-only view of the items custom headers is shown. To track progress on adding editing functionality, see the [relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/361925). -1. Select **Cancel** to close the read-only view. - ## Verify event authenticity > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345424) in GitLab 14.8. @@ -293,18 +257,41 @@ the destination's value when [listing streaming destinations](#list-streaming-de > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/360814) in GitLab 15.2. -Users with at least the Owner role for a group can list event streaming destinations and see the verification tokens: +Users with the Owner role for a group can list streaming destinations and see the verification tokens: 1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **Security & Compliance > Audit events**. -1. On the main area, select **Streams**. +1. On the main area, select the **Streams**. 1. View the verification token on the right side of each item. +## Payload schema + +> Documentation for an audit event streaming schema was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/358149) in GitLab 15.3. + +Streamed audit events have a predictable schema in the body of the response. + +| Field | Description | Notes | +|------------------|------------------------------------------------------------|-----------------------------------------------------------------------------------| +| `author_id` | User ID of the user who triggered the event | | +| `author_name` | Human-readable name of the author that triggered the event | Helpful when the author no longer exists | +| `created_at` | Timestamp when event was triggered | | +| `details` | JSON object containing additional metadata | Has no defined schema but often contains additional information about an event | +| `entity_id` | ID of the audit event's entity | | +| `entity_path` | Full path of the entity affected by the auditable event | | +| `entity_type` | String representation of the type of entity | Acceptable values include `User`, `Group`, and `Key`. This list is not exhaustive | +| `event_type` | String representation of the type of audit event | | +| `id` | Unique identifier for the audit event | Can be used for deduplication if required | +| `ip_address` | IP address of the host used to trigger the event | | +| `target_details` | Additional details about the target | | +| `target_id` | ID of the audit event's target | | +| `target_type` | String representation of the target's type | | + ## Audit event streaming on Git operations > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332747) in GitLab 14.9 [with a flag](../administration/feature_flags.md) named `audit_event_streaming_git_operations`. Disabled by default. > - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/357211) in GitLab 15.0. > - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/357211) in GitLab 15.1 by default. +> - [Added `details.author_class` field](https://gitlab.com/gitlab-org/gitlab/-/issues/363876) in GitLab 15.3. FLAG: On self-managed GitLab, by default this feature is available. To hide the @@ -318,7 +305,7 @@ Streaming audit events can be sent when signed-in users push or pull a project's Audit events are not captured for users that are not signed in. For example, when downloading a public project. -To configure streaming audit events for Git operations, see [Add a new event streaming destination](#add-a-new-event-streaming-destination). +To configure streaming audit events for Git operations, see [Add a new streaming destination](#add-a-new-streaming-destination). ### Headers @@ -346,6 +333,7 @@ Fetch: "entity_type": "Project", "details": { "author_name": "Administrator", + "author_class": "User", "target_id": 29, "target_type": "Project", "target_details": "example-project", @@ -377,6 +365,7 @@ Push: "entity_type": "Project", "details": { "author_name": "Administrator", + "author_class": "User", "target_id": 29, "target_type": "Project", "target_details": "example-project", @@ -398,6 +387,42 @@ Push: } ``` +### Example payloads for SSH events with Deploy Key + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/363876) in GitLab 15.3. + +Fetch: + +```json +{ + "id": 1, + "author_id": -3, + "entity_id": 29, + "entity_type": "Project", + "details": { + "author_name": "deploy-key-name", + "author_class": "DeployKey", + "target_id": 29, + "target_type": "Project", + "target_details": "example-project", + "custom_message": { + "protocol": "ssh", + "action": "git-upload-pack" + }, + "ip_address": "127.0.0.1", + "entity_path": "example-group/example-project" + }, + "ip_address": "127.0.0.1", + "author_name": "deploy-key-name", + "entity_path": "example-group/example-project", + "target_details": "example-project", + "created_at": "2022-07-26T05:43:53.662Z", + "target_type": "Project", + "target_id": 29, + "event_type": "repository_git_operation" +} +``` + ### Example payloads for HTTP and HTTPS events Fetch: @@ -410,6 +435,7 @@ Fetch: "entity_type": "Project", "details": { "author_name": "Administrator", + "author_class": "User", "target_id": 29, "target_type": "Project", "target_details": "example-project", @@ -441,6 +467,7 @@ Push: "entity_type": "Project", "details": { "author_name": "Administrator", + "author_class": "User", "target_id": 29, "target_type": "Project", "target_details": "example-project", @@ -462,6 +489,40 @@ Push: } ``` +### Example payloads for HTTP and HTTPS events with Deploy Token + +Fetch: + +```json +{ + "id": 1, + "author_id": -2, + "entity_id": 22, + "entity_type": "Project", + "details": { + "author_name": "deploy-token-name", + "author_class": "DeployToken", + "target_id": 22, + "target_type": "Project", + "target_details": "example-project", + "custom_message": { + "protocol": "http", + "action": "git-upload-pack" + }, + "ip_address": "127.0.0.1", + "entity_path": "example-group/example-project" + }, + "ip_address": "127.0.0.1", + "author_name": "deploy-token-name", + "entity_path": "example-group/example-project", + "target_details": "example-project", + "created_at": "2022-07-26T05:46:25.850Z", + "target_type": "Project", + "target_id": 22, + "event_type": "repository_git_operation" +} +``` + ### Example payloads for events from GitLab UI download button Fetch: @@ -475,6 +536,7 @@ Fetch: "details": { "custom_message": "Repository Download Started", "author_name": "example_username", + "author_class": "User", "target_id": 29, "target_type": "Project", "target_details": "example-group/example-project", diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index a329adbed22..92504c226fb 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w GitLab offers a way to view the changes made within the GitLab server for owners and administrators on a [paid plan](https://about.gitlab.com/pricing/). -GitLab system administrators can also view all audit events by accessing the [`audit_json.log` file](logs.md#audit_jsonlog). +GitLab system administrators can also view all audit events by accessing the [`audit_json.log` file](logs/index.md#audit_jsonlog). The JSON audit log does not include events that are [only streamed](../development/audit_event_guide/index.md#event-streaming). You can: @@ -115,7 +115,7 @@ From there, you can see the following actions: - Instance administrator started or stopped impersonation of a group member. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/300961) in GitLab 14.8. - Group deploy token was successfully created, revoked, or deleted. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/353452) in GitLab 14.9. - Failed attempt to create a group deploy token. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/353452) in GitLab 14.9. -- [IP restrictions](../user/group/index.md#group-access-restriction-by-ip-address) changed. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/358986) in GitLab 15.0. +- [IP restrictions](../user/group/access_and_permissions.md#restrict-group-access-by-ip-address) changed. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/358986) in GitLab 15.0. - Changes to push rules. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/227629) in GitLab 15.0. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/356152) in GitLab 15.1, changes to the following merge request approvals settings: - Prevent approval by author. @@ -123,6 +123,7 @@ From there, you can see the following actions: - Prevent editing approval rules in projects and merge requests. - Require user password to approve. - Remove all approvals when commits are added to the source branch. +- Changes to streaming audit destination custom HTTP headers. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/366350) in GitLab 15.3. Group events can also be accessed via the [Group Audit Events API](../api/audit_events.md#group-audit-events) @@ -151,6 +152,7 @@ From there, you can see the following actions: - Added, removed, or updated protected branches - Release was added to a project - Release was updated +- Release was deleted ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/94793/) in GitLab 13.5) - Release milestone associations changed - Permission to approve merge requests by committers was updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7531) in GitLab 12.9) - Permission to approve merge requests by committers was updated. @@ -271,7 +273,7 @@ Don't see the event you want in any of the epics linked above? You can either: - Use the **Audit Event Proposal** issue template to [create an issue](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Audit%20Event%20Proposal) to request it. -- [Add it yourself](../development/audit_event_guide/). +- [Add it yourself](../development/audit_event_guide/index.md). ### Removed events diff --git a/doc/administration/audit_reports.md b/doc/administration/audit_reports.md index 6fa592b96db..e33b5153c5b 100644 --- a/doc/administration/audit_reports.md +++ b/doc/administration/audit_reports.md @@ -26,4 +26,4 @@ needs. ## Features - [Audit events](audit_events.md) -- [Log system](logs.md) +- [Log system](logs/index.md) diff --git a/doc/administration/auditor_users.md b/doc/administration/auditor_users.md index d82683e1778..561aa5d7b2e 100644 --- a/doc/administration/auditor_users.md +++ b/doc/administration/auditor_users.md @@ -55,3 +55,9 @@ If you are signed in with auditor access, you: you can push commits or comment on issues. - Can access the same resources using the GitLab UI or API. - Can't view the Admin Area, or perform any administration actions. + +## Maintain auditor users using API + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/366404) in GitLab 15.3. + +Administrators can use the GitLab API to [create](../api/users.md#user-creation) and [modify](../api/users.md#user-modification) auditor users. diff --git a/doc/administration/auth/crowd.md b/doc/administration/auth/crowd.md index 8dfa832a103..ced7cdb7119 100644 --- a/doc/administration/auth/crowd.md +++ b/doc/administration/auth/crowd.md @@ -5,7 +5,11 @@ group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Atlassian Crowd OmniAuth Provider **(FREE SELF)** +# Atlassian Crowd OmniAuth provider (deprecated) **(FREE SELF)** + +WARNING: +This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/369117) in GitLab 15.3 and is planned for +removal in 16.0. Authenticate to GitLab using the Atlassian Crowd OmniAuth provider. Enabling this provider also allows Crowd authentication for Git-over-https requests. @@ -68,24 +72,24 @@ this provider also allows Crowd authentication for Git-over-https requests. application_password: 'YOUR_APP_PASSWORD' } } ``` -1. Change `CROWD_SERVER_URL` to the URL of your Crowd server. +1. Change `CROWD_SERVER_URL` to the [base URL of your Crowd server](https://confluence.atlassian.com/crowdkb/how-to-change-the-crowd-base-url-245827278.html). 1. Change `YOUR_APP_NAME` to the application name from Crowd applications page. 1. Change `YOUR_APP_PASSWORD` to the application password you've set. 1. Save the configuration file. -1. [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) or [restart](../restart_gitlab.md#installations-from-source) for the changes to take effect if you - installed GitLab via Omnibus or from source respectively. +1. [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) (Omnibus GitLab) or [restart](../restart_gitlab.md#installations-from-source) (source installations) for + the changes to take effect. On the sign in page there should now be a Crowd tab in the sign in form. ## Troubleshooting -If you see an error message like the one below when you sign in after Crowd authentication is configured, you may want to consult the Crowd administrator for the Crowd log file to know the exact cause: +### Error: "could not authorize you from Crowd because invalid credentials" -```plaintext -could not authorize you from Crowd because invalid credentials -``` +This error sometimes occurs when a user attempts to authenticate with Crowd. The +Crowd administrator should consult the Crowd log file to know the exact cause of +this error message. -Ensure the Crowd users who need to sign in to GitLab are authorized to the +Ensure the Crowd users who must sign in to GitLab are authorized to the [application](#configure-a-new-crowd-application) in the **Authorization** step. This could be verified by trying "Authentication test" for Crowd (as of 2.11). diff --git a/doc/administration/auth/ldap/img/multi_login.png b/doc/administration/auth/ldap/img/multi_login.png Binary files differdeleted file mode 100644 index 512f403a442..00000000000 --- a/doc/administration/auth/ldap/img/multi_login.png +++ /dev/null diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index 05eee338e64..2f0a0db9d6f 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -75,7 +75,7 @@ To configure LDAP integration, add your LDAP server settings in: - `/home/git/gitlab/config/gitlab.yml` for source install instances. After configuring LDAP, to test the configuration, use the -[LDAP check Rake task](../../raketasks/check.md#ldap-check). +[LDAP check Rake task](../../raketasks/ldap.md#check). NOTE: The `encryption` value `simple_tls` corresponds to 'Simple TLS' in the LDAP @@ -95,42 +95,42 @@ This example shows configuration for Omnibus GitLab instances: gitlab_rails['ldap_enabled'] = true gitlab_rails['prevent_ldap_sign_in'] = false gitlab_rails['ldap_servers'] = { -'main' => { - 'label' => 'LDAP', - 'host' => 'ldap.mydomain.com', - 'port' => 389, - 'uid' => 'sAMAccountName', - 'encryption' => 'simple_tls', - 'verify_certificates' => true, - 'bind_dn' => '_the_full_dn_of_the_user_you_will_bind_with', - 'password' => '_the_password_of_the_bind_user', - 'tls_options' => { - 'ca_file' => '', - 'ssl_version' => '', - 'ciphers' => '', - 'cert' => '', - 'key' => '' - }, - 'timeout' => 10, - 'active_directory' => true, - 'allow_username_or_email_login' => false, - 'block_auto_created_users' => false, - 'base' => 'dc=example,dc=com', - 'user_filter' => '', - 'attributes' => { - 'username' => ['uid', 'userid', 'sAMAccountName'], - 'email' => ['mail', 'email', 'userPrincipalName'], - 'name' => 'cn', - 'first_name' => 'givenName', - 'last_name' => 'sn' - }, - 'lowercase_usernames' => false, - - # EE Only - 'group_base' => '', - 'admin_group' => '', - 'external_groups' => [], - 'sync_ssh_keys' => false + 'main' => { + 'label' => 'LDAP', + 'host' => 'ldap.mydomain.com', + 'port' => 389, + 'uid' => 'sAMAccountName', + 'encryption' => 'simple_tls', + 'verify_certificates' => true, + 'bind_dn' => '_the_full_dn_of_the_user_you_will_bind_with', + 'password' => '_the_password_of_the_bind_user', + 'tls_options' => { + 'ca_file' => '', + 'ssl_version' => '', + 'ciphers' => '', + 'cert' => '', + 'key' => '' + }, + 'timeout' => 10, + 'active_directory' => true, + 'allow_username_or_email_login' => false, + 'block_auto_created_users' => false, + 'base' => 'dc=example,dc=com', + 'user_filter' => '', + 'attributes' => { + 'username' => ['uid', 'userid', 'sAMAccountName'], + 'email' => ['mail', 'email', 'userPrincipalName'], + 'name' => 'cn', + 'first_name' => 'givenName', + 'last_name' => 'sn' + }, + 'lowercase_usernames' => false, + + # EE Only + 'group_base' => '', + 'admin_group' => '', + 'external_groups' => [], + 'sync_ssh_keys' => false } } ``` @@ -248,33 +248,34 @@ The following example shows how to configure three LDAP servers in `gitlab.rb`: ```ruby gitlab_rails['ldap_enabled'] = true gitlab_rails['ldap_servers'] = { -'main' => { - 'label' => 'GitLab AD', - 'host' => 'ad.example.org', - 'port' => 636, - ... + 'main' => { + 'label' => 'GitLab AD', + 'host' => 'ad.example.org', + 'port' => 636, + ... }, -'secondary' => { - 'label' => 'GitLab Secondary AD', - 'host' => 'ad-secondary.example.net', - 'port' => 636, - ... + 'secondary' => { + 'label' => 'GitLab Secondary AD', + 'host' => 'ad-secondary.example.net', + 'port' => 636, + ... }, -'tertiary' => { - 'label' => 'GitLab Tertiary AD', - 'host' => 'ad-tertiary.example.net', - 'port' => 636, - ... + 'tertiary' => { + 'label' => 'GitLab Tertiary AD', + 'host' => 'ad-tertiary.example.net', + 'port' => 636, + ... } - } ``` -This example results in the following sign-in page: +This example results in a sign-in page with the following tabs: - +- **GitLab AD**. +- **GitLab Secondary AD**. +- **GitLab Tertiary AD**. ### Set up LDAP user filter @@ -286,9 +287,9 @@ necessary, you can set up an LDAP user filter. The filter must comply with [RFC ```ruby gitlab_rails['ldap_servers'] = { - 'main' => { - # snip... - 'user_filter' => '(employeeType=developer)' + 'main' => { + # snip... + 'user_filter' => '(employeeType=developer)' } } ``` @@ -363,9 +364,9 @@ the configuration option `lowercase_usernames`. By default, this configuration o ```ruby gitlab_rails['ldap_servers'] = { - 'main' => { - # snip... - 'lowercase_usernames' => true + 'main' => { + # snip... + 'lowercase_usernames' => true } } ``` @@ -444,15 +445,15 @@ If initially your LDAP configuration looked like: 1. In `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['ldap_servers'] = { - 'main' => { - # snip... - 'bind_dn' => 'admin', - 'password' => '123' - } - } - ``` + ```ruby + gitlab_rails['ldap_servers'] = { + 'main' => { + # snip... + 'bind_dn' => 'admin', + 'password' => '123' + } + } + ``` 1. Edit the encrypted secret: diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md index c7572ec0a18..64ef27cbf51 100644 --- a/doc/administration/auth/ldap/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap/ldap-troubleshooting.md @@ -181,7 +181,7 @@ The user should now be able to sign in. #### Email has already been taken A user tries to sign in with the correct LDAP credentials, is denied access, -and the [production.log](../../logs.md#productionlog) shows an error that looks like this: +and the [production.log](../../logs/index.md#productionlog) shows an error that looks like this: ```plaintext (LDAP) Error saving user <USER DN> (email@example.com): ["Email has already been taken"] @@ -210,8 +210,8 @@ This shows you which user has this email address. One of two steps must be taken remove this email as a secondary email and make it a primary one so GitLab associates this profile to the LDAP identity. -The user can do either of these steps [in their -profile](../../../user/profile/index.md#access-your-user-profile) or an administrator can do it. +The user can do either of these steps +[in their profile](../../../user/profile/index.md#access-your-user-profile) or an administrator can do it. #### Projects limit errors @@ -426,13 +426,12 @@ Rails.logger.level = Logger::DEBUG LdapAllGroupsSyncWorker.new.perform ``` -Next, [learn how to read the -output](#example-console-output-after-a-group-sync). +Next, [learn how to read the output](#example-console-output-after-a-group-sync). ##### Example console output after a group sync -Like the output from the user sync, the output from the [manual group -sync](#sync-all-groups) is also very verbose. However, it contains lots +Like the output from the user sync, the output from the +[manual group sync](#sync-all-groups) is also very verbose. However, it contains lots of helpful information. Indicates the point where syncing actually begins: @@ -644,10 +643,10 @@ users, [see what to do when no users are found](#no-users-are-found). ### GitLab logs If a user account is blocked or unblocked due to the LDAP configuration, a -message is [logged to `application.log`](../../logs.md#applicationlog). +message is [logged to `application.log`](../../logs/index.md#applicationlog). If there is an unexpected error during an LDAP lookup (configuration error, -timeout), the sign-in is rejected and a message is [logged to `production.log`](../../logs.md#productionlog). +timeout), the sign-in is rejected and a message is [logged to `production.log`](../../logs/index.md#productionlog). ### ldapsearch @@ -743,8 +742,7 @@ For instructions about how to use the rails console, refer to this This provides debug output that shows what GitLab is doing and with what. This value is not persisted, and is only enabled for this session in the Rails console. -To enable debug output in the rails console, [enter the rails -console](#rails-console) and run: +To enable debug output in the rails console, [enter the rails console](#rails-console) and run: ```ruby Rails.logger.level = Logger::DEBUG diff --git a/doc/administration/auth/ldap/ldap_synchronization.md b/doc/administration/auth/ldap/ldap_synchronization.md index b0ada1c11dd..62706a9e3b9 100644 --- a/doc/administration/auth/ldap/ldap_synchronization.md +++ b/doc/administration/auth/ldap/ldap_synchronization.md @@ -128,7 +128,7 @@ To take advantage of group sync, group Owners or users with the [Maintainer role ### Add group links For information on adding group links by using CNs and filters, refer to the -[GitLab groups documentation](../../../user/group/index.md#manage-group-memberships-via-ldap). +[GitLab groups documentation](../../../user/group/access_and_permissions.md#manage-group-memberships-via-ldap). ### Administrator sync diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md index 60a4cc8706f..8c5bf96e99e 100644 --- a/doc/administration/auth/oidc.md +++ b/doc/administration/auth/oidc.md @@ -250,8 +250,8 @@ but `LocalAccounts` works for authenticating against local, Active Directory acc <OutputClaim ClaimTypeReferenceId="signInNames.emailAddress" PartnerClaimType="email" /> ``` -1. For OIDC discovery to work with B2C, the policy must be configured with an issuer compatible with the [OIDC - specification](https://openid.net/specs/openid-connect-discovery-1_0.html#rfc.section.4.3). +1. For OIDC discovery to work with B2C, the policy must be configured with an issuer compatible with the + [OIDC specification](https://openid.net/specs/openid-connect-discovery-1_0.html#rfc.section.4.3). See the [token compatibility settings](https://docs.microsoft.com/en-us/azure/active-directory-b2c/configure-tokens?pivots=b2c-custom-policy#token-compatibility-settings). In `TrustFrameworkBase.xml` under `JwtIssuer`, set `IssuanceClaimPattern` to `AuthorityWithTfp`: @@ -529,8 +529,7 @@ If you're having trouble, here are some tips: 1. Check your system clock to ensure the time is synchronized properly. -1. As mentioned in [the - documentation](https://github.com/m0n9oose/omniauth_openid_connect), +1. As mentioned in [the documentation](https://github.com/m0n9oose/omniauth_openid_connect), make sure `issuer` corresponds to the base URL of the Discovery URL. For example, `https://accounts.google.com` is used for the URL `https://accounts.google.com/.well-known/openid-configuration`. @@ -540,5 +539,4 @@ If you're having trouble, here are some tips: If you are seeing 401 errors upon retrieving the `userinfo` endpoint, you may want to check your OpenID Web server configuration. For example, for [`oauth2-server-php`](https://github.com/bshaffer/oauth2-server-php), you - may need to [add a configuration parameter to - Apache](https://github.com/bshaffer/oauth2-server-php/issues/926#issuecomment-387502778). + may need to [add a configuration parameter to Apache](https://github.com/bshaffer/oauth2-server-php/issues/926#issuecomment-387502778). diff --git a/doc/administration/compliance.md b/doc/administration/compliance.md index 869de929eb8..8c7f8bf766d 100644 --- a/doc/administration/compliance.md +++ b/doc/administration/compliance.md @@ -97,7 +97,7 @@ These features can also help with compliance requirements: projects): Search dependencies for their licenses. This lets you determine if the licenses of your project's dependencies are compatible with your project's license. -- [**Lock project membership to group**](../user/group/index.md#prevent-members-from-being-added-to-projects-in-a-group) +- [**Lock project membership to group**](../user/group/access_and_permissions.md#prevent-members-from-being-added-to-projects-in-a-group) (for groups): Group owners can prevent new members from being added to projects within a group. - [**LDAP group sync**](auth/ldap/ldap_synchronization.md#group-sync) (for diff --git a/doc/administration/external_pipeline_validation.md b/doc/administration/external_pipeline_validation.md index 06eb9ffc84e..99876cdf503 100644 --- a/doc/administration/external_pipeline_validation.md +++ b/doc/administration/external_pipeline_validation.md @@ -46,6 +46,7 @@ required number of seconds. "user", "pipeline", "builds", + "total_builds_count", "namespace" ], "properties" : { @@ -61,7 +62,9 @@ required number of seconds. "properties": { "id": { "type": "integer" }, "path": { "type": "string" }, - "created_at": { "type": ["string", "null"], "format": "date-time" } + "created_at": { "type": ["string", "null"], "format": "date-time" }, + "shared_runners_enabled": { "type": "boolean" }, + "group_runners_enabled": { "type": "boolean" } } }, "user": { @@ -121,6 +124,7 @@ required number of seconds. } } }, + "total_builds_count": { "type": "integer" }, "namespace": { "type": "object", "required": [ diff --git a/doc/administration/geo/disaster_recovery/bring_primary_back.md b/doc/administration/geo/disaster_recovery/bring_primary_back.md index 833b9a877e9..a2d4f35a7c3 100644 --- a/doc/administration/geo/disaster_recovery/bring_primary_back.md +++ b/doc/administration/geo/disaster_recovery/bring_primary_back.md @@ -41,8 +41,8 @@ To bring the former **primary** site up to date: NOTE: If you [changed the DNS records](index.md#step-4-optional-updating-the-primary-domain-dns-record) - for this site during disaster recovery procedure you may need to [block - all the writes to this site](planned_failover.md#prevent-updates-to-the-primary-site) + for this site during disaster recovery procedure you may need to + [block all the writes to this site](planned_failover.md#prevent-updates-to-the-primary-site) during this procedure. 1. [Set up database replication](../setup/database.md). In this case, the **secondary** site diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index f457cb4b0a2..0cae94fcec1 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -572,7 +572,7 @@ and after that you also need two extra steps. postgresql['md5_auth_cidr_addresses'] = ['<primary_site_ip>/32', '<secondary_site_ip>/32'] # Every secondary site needs to have its own slot so specify the number of secondary sites you're going to have - postgresql['max_replication_slots'] = 1 + # postgresql['max_replication_slots'] = 1 # Set this to be the number of Geo secondary nodes if you have more than one ## ## Disable automatic database migrations temporarily @@ -581,7 +581,7 @@ and after that you also need two extra steps. gitlab_rails['auto_migrate'] = false ``` - (For more details about these settings you can read [Configure the primary server](../setup/database.md#step-1-configure-the-primary-server)) + (For more details about these settings you can read [Configure the primary server](../setup/database.md#step-1-configure-the-primary-site)) 1. Save the file and reconfigure GitLab for the database listen changes and the replication slot changes to be applied: diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index 5a5d896c20a..6c3353e7d7e 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -45,7 +45,7 @@ be adapted for use with any other file-based data, such as [GitLab Pages](../../ ### Container registry By default, the container registry is not automatically replicated to secondary -sites and this needs to be manually configured, see [Docker Registry for a secondary site](../replication/docker_registry.md). +sites and this needs to be manually configured, see [Container Registry for a secondary site](../replication/container_registry.md). If you are using local storage on your current primary site for the container registry, you can `rsync` the container registry objects to the secondary diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index cf7d2649142..e3bf5ccdfe7 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -119,8 +119,8 @@ The following are required to run Geo: The following operating systems are known to ship with a current version of OpenSSH: - [CentOS](https://www.centos.org) 7.4 or later - [Ubuntu](https://ubuntu.com) 16.04 or later -- PostgreSQL 12 with [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication) - - PostgreSQL 13 is not supported for Geo, see [epic 3832](https://gitlab.com/groups/gitlab-org/-/epics/3832) +- PostgreSQL 12 or 13 with [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication) + - Note,[PostgreSQL 12 is deprecated](../../update/deprecations.md#postgresql-12-deprecated) and will be removed in GitLab 16.0. - Git 2.9 or later - Git-lfs 2.4.2 or later on the user side when using LFS - All sites must run [the same GitLab and PostgreSQL versions](setup/database.md#postgresql-replication). @@ -283,7 +283,7 @@ For information on using Geo in disaster recovery situations to mitigate data-lo ### Replicating the Container Registry -For more information on how to replicate the Container Registry, see [Docker Registry for a **secondary** site](replication/docker_registry.md). +For more information on how to replicate the Container Registry, see [Container Registry for a **secondary** site](replication/container_registry.md). ### Geo secondary proxy diff --git a/doc/administration/geo/replication/container_registry.md b/doc/administration/geo/replication/container_registry.md new file mode 100644 index 00000000000..b425e5dcc0d --- /dev/null +++ b/doc/administration/geo/replication/container_registry.md @@ -0,0 +1,167 @@ +--- +stage: Systems +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: howto +--- + +# Container Registry for a secondary site **(PREMIUM SELF)** + +You can set up a Container Registry on your **secondary** Geo site that mirrors the one on the **primary** Geo site. + +## Supported container registries + +Geo supports the following types of container registries: + +- [Docker](https://docs.docker.com/registry/) +- [OCI](https://github.com/opencontainers/distribution-spec/blob/main/spec.md) + +## Supported image formats + +The following container image formats are support by Geo: + +- [Docker V2, schema 1](https://docs.docker.com/registry/spec/manifest-v2-1/) +- [Docker V2, schema 2](https://docs.docker.com/registry/spec/manifest-v2-2/) +- [OCI (Open Container Initiative)](https://github.com/opencontainers/image-spec) + +In addition, Geo also supports [BuildKit cache images](https://github.com/moby/buildkit). + +## Supported storage + +### Docker + +For more information on supported registry storage drivers see +[Docker registry storage drivers](https://docs.docker.com/registry/storage-drivers/) + +Read the [Load balancing considerations](https://docs.docker.com/registry/deploying/#load-balancing-considerations) +when deploying the Registry, and how to set up the storage driver for the GitLab integrated +[Container Registry](../../packages/container_registry.md#use-object-storage). + +### Registries that support OCI artifacts + +The following registries support OCI artifacts: + +- CNCF Distribution - local/offline verification +- Azure Container Registry (ACR) +- Amazon Elastic Container Registry (ECR) +- Google Artifact Registry (GAR) +- GitHub Packages container registry (GHCR) +- Bundle Bar + +For more information, see the [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec). + +## Configure Container Registry replication + +You can enable a storage-agnostic replication so it +can be used for cloud or local storage. Whenever a new image is pushed to the +**primary** site, each **secondary** site pulls it to its own container +repository. + +To configure Container Registry replication: + +1. Configure the [**primary** site](#configure-primary-site). +1. Configure the [**secondary** site](#configure-secondary-site). +1. Verify Container Registry [replication](#verify-replication). + +### Configure **primary** site + +Make sure that you have Container Registry set up and working on +the **primary** site before following the next steps. + +We need to make Container Registry send notification events to the +**primary** site. + +1. SSH into your GitLab **primary** server and login as root: + + ```shell + sudo -i + ``` + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + registry['notifications'] = [ + { + 'name' => 'geo_event', + 'url' => 'https://<example.com>/api/v4/container_registry_event/events', + 'timeout' => '500ms', + 'threshold' => 5, + 'backoff' => '1s', + 'headers' => { + 'Authorization' => ['<replace_with_a_secret_token>'] + } + } + ] + ``` + + NOTE: + Replace `<example.com>` with the `external_url` defined in your primary site's `/etc/gitlab/gitlab.rb` file, and + replace `<replace_with_a_secret_token>` with a case sensitive alphanumeric string + that starts with a letter. You can generate one with `< /dev/urandom tr -dc _A-Z-a-z-0-9 | head -c 32 | sed "s/^[0-9]*//"; echo` + + NOTE: + If you use an external Registry (not the one integrated with GitLab), you must add + these settings to its configuration yourself. In this case, you also have to specify + notification secret in `registry.notification_secret` section of + `/etc/gitlab/gitlab.rb` file. + + NOTE: + If you use GitLab HA, you also have to specify + the notification secret in `registry.notification_secret` section of + `/etc/gitlab/gitlab.rb` file for every web node. + +1. Reconfigure the **primary** node for the change to take effect: + + ```shell + gitlab-ctl reconfigure + ``` + +### Configure **secondary** site + +Make sure you have Container Registry set up and working on +the **secondary** site before following the next steps. + +The following steps should be done on each **secondary** site you're +expecting to see the container images replicated. + +Because we need to allow the **secondary** site to communicate securely with +the **primary** site Container Registry, we need to have a single key +pair for all the sites. The **secondary** site uses this key to +generate a short-lived JWT that is pull-only-capable to access the +**primary** site Container Registry. + +For each application and Sidekiq node on the **secondary** site: + +1. SSH into the node and login as the `root` user: + + ```shell + sudo -i + ``` + +1. Copy `/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key` from the **primary** to the node. + +1. Edit `/etc/gitlab/gitlab.rb` and add: + + ```ruby + gitlab_rails['geo_registry_replication_enabled'] = true + + # Primary registry's hostname and port, it will be used by + # the secondary node to directly communicate to primary registry + gitlab_rails['geo_registry_replication_primary_api_url'] = 'https://primary.example.com:5050/' + ``` + +1. Reconfigure the node for the change to take effect: + + ```shell + gitlab-ctl reconfigure + ``` + +### Verify replication + +To verify Container Registry replication is working, on the **secondary** site: + +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Geo > Nodes**. + The initial replication, or "backfill", is probably still in progress. + +You can monitor the synchronization process on each Geo site from the **primary** site's **Geo Nodes** dashboard in your browser. diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index acd27d5feed..40b71684bac 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -57,6 +57,8 @@ verification methods: | Blobs | Pipeline artifacts _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | Pages _(file system)_ | Geo with API | SHA256 checksum | | Blobs | Pages _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | CI Secure Files _(file system)_ | Geo with API | SHA256 checksum | +| Blobs | CI Secure Files _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | - (*1*): Redis replication can be used as part of HA with Redis sentinel. It's not used between Geo sites. - (*2*): Object storage replication can be performed by Geo or by your object storage provider/appliance @@ -185,8 +187,8 @@ successfully, you must replicate their data using some other means. |Feature | Replicated (added in GitLab version) | Verified (added in GitLab version) | GitLab-managed object storage replication (added in GitLab version) | GitLab-managed object storage verification (added in GitLab version) | Notes | |:--------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------|:---------------------------------------------------------------------------|:--------------------------------------------------------------------|:----------------------------------------------------------------|:------| |[Application data in PostgreSQL](../../postgresql/index.md) | **Yes** (10.2) | **Yes** (10.2) | N/A | N/A | | -|[Project repository](../../../user/project/repository/) | **Yes** (10.2) | **Yes** (10.7) | N/A | N/A | | -|[Project wiki repository](../../../user/project/wiki/) | **Yes** (10.2) | **Yes** (10.7) | N/A | N/A | | +|[Project repository](../../../user/project/repository/index.md) | **Yes** (10.2) | **Yes** (10.7) | N/A | N/A | | +|[Project wiki repository](../../../user/project/wiki/index.md) | **Yes** (10.2) | **Yes** (10.7) | N/A | N/A | | |[Group wiki repository](../../../user/project/wiki/group.md) | [**Yes** (13.10)](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | No | N/A | N/A | Behind feature flag `geo_group_wiki_repository_replication`, enabled by default. | |[Uploads](../../uploads.md) | **Yes** (10.2) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Replication is behind the feature flag `geo_upload_replication`, enabled by default. Verification was behind the feature flag `geo_upload_verification`, removed in 14.8. | |[LFS objects](../../lfs/index.md) | **Yes** (10.2) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696).<br /><br />Replication is behind the feature flag `geo_lfs_object_replication`, enabled by default. Verification was behind the feature flag `geo_lfs_object_verification`, removed in 14.7. | @@ -194,7 +196,8 @@ successfully, you must replicate their data using some other means. |[Project snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | N/A | N/A | | |[CI job artifacts](../../../ci/pipelines/job_artifacts.md) | **Yes** (10.4) | **Yes** (14.10) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Verification is behind the feature flag `geo_job_artifact_replication`, enabled by default in 14.10. | |[CI Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/pipeline_artifact.rb) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Persists additional artifacts after a pipeline completes. | -|[Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | No | No | Disabled by default. See [instructions](docker_registry.md) to enable. | +|[CI Secure Files](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/secure_file.rb) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [No](object_storage.md#verification-of-files-in-object-storage) | Verification is behind the feature flag `geo_ci_secure_file_replication`, enabled by default in 15.3. | +|[Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | No | No | Disabled by default. See [instructions](container_registry.md) to enable. | |[Infrastructure Registry](../../../user/packages/infrastructure_registry/index.md) | **Yes** (14.0) | **Yes** (14.0) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Behind feature flag `geo_package_file_replication`, enabled by default. | |[Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | N/A | N/A | Designs also require replication of LFS objects and Uploads. | |[Package Registry](../../../user/packages/package_registry/index.md) | **Yes** (13.2) | **Yes** (13.10) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Behind feature flag `geo_package_file_replication`, enabled by default. | @@ -202,9 +205,10 @@ successfully, you must replicate their data using some other means. |[External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Replication is behind the feature flag `geo_merge_request_diff_replication`, enabled by default. Verification was behind the feature flag `geo_merge_request_diff_verification`, removed in 14.7.| |[Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [**Yes** (14.2)](https://gitlab.com/groups/gitlab-org/-/epics/2810) | N/A | N/A | Verification was implemented behind the feature flag `geo_snippet_repository_verification` in 13.11, and the feature flag was removed in 14.2. | |[GitLab Pages](../../pages/index.md) | [**Yes** (14.3)](https://gitlab.com/groups/gitlab-org/-/epics/589) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Behind feature flag `geo_pages_deployment_replication`, enabled by default. Verification was behind the feature flag `geo_pages_deployment_verification`, removed in 14.7. | +|[Project-level Secure files](../../../ci/secure_files/index.md) | **Yes** (15.3) | **Yes** (15.3) | **Yes** (15.3) | [No](object_storage.md#verification-of-files-in-object-storage) | | |[Incident Metric Images](../../../operations/incident_management/incidents.md#metrics) | [Planned](https://gitlab.com/gitlab-org/gitlab/-/issues/362561) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/362561) | No | No | | |[Alert Metric Images](../../../operations/incident_management/alerts.md#metrics-tab) | [Planned](https://gitlab.com/gitlab-org/gitlab/-/issues/362564) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/362564) | No | No | | |[Server-side Git hooks](../../server_hooks.md) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/1867) | No | N/A | N/A | Not planned because of current implementation complexity, low customer interest, and availability of alternatives to hooks. | |[Elasticsearch integration](../../../integration/elasticsearch.md) | [Not planned](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | No | No | Not planned because further product discovery is required and Elasticsearch (ES) clusters can be rebuilt. Secondaries use the same ES cluster as the primary. | |[Dependency proxy images](../../../user/packages/dependency_proxy/index.md) | [Not planned](https://gitlab.com/gitlab-org/gitlab/-/issues/259694) | No | No | No | Blocked by [Geo: Secondary Mimicry](https://gitlab.com/groups/gitlab-org/-/epics/1528). Replication of this cache is not needed for disaster recovery purposes because it can be recreated from external sources. | -|[Vulnerability Export](../../../user/application_security/vulnerability_report/#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | No | No | Not planned because they are ephemeral and sensitive information. They can be regenerated on demand. | +|[Vulnerability Export](../../../user/application_security/vulnerability_report/index.md#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | No | No | Not planned because they are ephemeral and sensitive information. They can be regenerated on demand. | diff --git a/doc/administration/geo/replication/disable_geo.md b/doc/administration/geo/replication/disable_geo.md index 5d4ae12c990..f0658ae45a2 100644 --- a/doc/administration/geo/replication/disable_geo.md +++ b/doc/administration/geo/replication/disable_geo.md @@ -28,7 +28,7 @@ To disable Geo, you need to first remove all your secondary Geo sites, which mea anymore on these sites. You can follow our docs to [remove your secondary Geo sites](remove_geo_site.md). If the current site that you want to keep using is a secondary site, you need to first promote it to primary. -You can use our steps on [how to promote a secondary site](../disaster_recovery/#step-3-promoting-a-secondary-site) +You can use our steps on [how to promote a secondary site](../disaster_recovery/index.md#step-3-promoting-a-secondary-site) to do that. ## Remove the primary site from the UI diff --git a/doc/administration/geo/replication/docker_registry.md b/doc/administration/geo/replication/docker_registry.md index 855e33d9a51..d0af6f2a66f 100644 --- a/doc/administration/geo/replication/docker_registry.md +++ b/doc/administration/geo/replication/docker_registry.md @@ -1,137 +1,11 @@ --- -stage: Systems -group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: howto +redirect_to: 'container_registry.md' +remove_date: '2022-10-29' --- -# Docker Registry for a secondary site **(PREMIUM SELF)** +This document was moved to [another location](container_registry.md). -You can set up a [Docker Registry](https://docs.docker.com/registry/) on your -**secondary** Geo site that mirrors the one on the **primary** Geo site. - -## Storage support - -Docker Registry currently supports a few types of storage. If you choose a -distributed storage (`azure`, `gcs`, `s3`, `swift`, or `oss`) for your Docker -Registry on the **primary** site, you can use the same storage for a **secondary** -Docker Registry as well. For more information, read the -[Load balancing considerations](https://docs.docker.com/registry/deploying/#load-balancing-considerations) -when deploying the Registry, and how to set up the storage driver for the GitLab -integrated [Container Registry](../../packages/container_registry.md#use-object-storage). - -## Replicating Docker Registry - -You can enable a storage-agnostic replication so it -can be used for cloud or local storage. Whenever a new image is pushed to the -**primary** site, each **secondary** site pulls it to its own container -repository. - -To configure Docker Registry replication: - -1. Configure the [**primary** site](#configure-primary-site). -1. Configure the [**secondary** site](#configure-secondary-site). -1. Verify Docker Registry [replication](#verify-replication). - -### Configure **primary** site - -Make sure that you have Container Registry set up and working on -the **primary** site before following the next steps. - -We need to make Docker Registry send notification events to the -**primary** site. - -1. SSH into your GitLab **primary** server and login as root: - - ```shell - sudo -i - ``` - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - registry['notifications'] = [ - { - 'name' => 'geo_event', - 'url' => 'https://<example.com>/api/v4/container_registry_event/events', - 'timeout' => '500ms', - 'threshold' => 5, - 'backoff' => '1s', - 'headers' => { - 'Authorization' => ['<replace_with_a_secret_token>'] - } - } - ] - ``` - - NOTE: - Replace `<example.com>` with the `external_url` defined in your primary site's `/etc/gitlab/gitlab.rb` file, and - replace `<replace_with_a_secret_token>` with a case sensitive alphanumeric string - that starts with a letter. You can generate one with `< /dev/urandom tr -dc _A-Z-a-z-0-9 | head -c 32 | sed "s/^[0-9]*//"; echo` - - NOTE: - If you use an external Registry (not the one integrated with GitLab), you must add - these settings to its configuration yourself. In this case, you also have to specify - notification secret in `registry.notification_secret` section of - `/etc/gitlab/gitlab.rb` file. - - NOTE: - If you use GitLab HA, you also have to specify - the notification secret in `registry.notification_secret` section of - `/etc/gitlab/gitlab.rb` file for every web node. - -1. Reconfigure the **primary** node for the change to take effect: - - ```shell - gitlab-ctl reconfigure - ``` - -### Configure **secondary** site - -Make sure you have Container Registry set up and working on -the **secondary** site before following the next steps. - -The following steps should be done on each **secondary** site you're -expecting to see the Docker images replicated. - -Because we need to allow the **secondary** site to communicate securely with -the **primary** site Container Registry, we need to have a single key -pair for all the sites. The **secondary** site uses this key to -generate a short-lived JWT that is pull-only-capable to access the -**primary** site Container Registry. - -For each application and Sidekiq node on the **secondary** site: - -1. SSH into the node and login as the `root` user: - - ```shell - sudo -i - ``` - -1. Copy `/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key` from the **primary** to the node. - -1. Edit `/etc/gitlab/gitlab.rb` and add: - - ```ruby - gitlab_rails['geo_registry_replication_enabled'] = true - - # Primary registry's hostname and port, it will be used by - # the secondary node to directly communicate to primary registry - gitlab_rails['geo_registry_replication_primary_api_url'] = 'https://primary.example.com:5050/' - ``` - -1. Reconfigure the node for the change to take effect: - - ```shell - gitlab-ctl reconfigure - ``` - -### Verify replication - -To verify Container Registry replication is working, on the **secondary** site: - -1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **Geo > Nodes**. - The initial replication, or "backfill", is probably still in progress. - -You can monitor the synchronization process on each Geo site from the **primary** site's **Geo Nodes** dashboard in your browser. +<!-- This redirect file can be deleted after <2022-10-29>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
\ No newline at end of file diff --git a/doc/administration/geo/replication/faq.md b/doc/administration/geo/replication/faq.md index bdf1771e8a8..7a67af1cfa2 100644 --- a/doc/administration/geo/replication/faq.md +++ b/doc/administration/geo/replication/faq.md @@ -65,10 +65,10 @@ connectivity between your sites, your hardware, and so on. That's totally fine. We use HTTP(s) to fetch repository changes from the **primary** site to all **secondary** sites. -## Is this possible to set up a Docker Registry for a **secondary** site that mirrors the one on the **primary** site? +## Is this possible to set up a Container Registry for a **secondary** site that mirrors the one on the **primary** site? -Yes. See [Docker Registry for a **secondary** site](docker_registry.md). +Yes. See [Container Registry for a **secondary** site](container_registry.md). -## Can you login to a secondary site? +## Can you log in to a secondary site? Yes, but secondary sites receive all authentication data (like user accounts and logins) from the primary instance. This means you are re-directed to the primary for authentication and then routed back. diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 082ecbbb208..26d192f62cd 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -129,7 +129,7 @@ http://secondary.example.com/ ``` To find more details about failed items, check -[the `gitlab-rails/geo.log` file](../../troubleshooting/log_parsing.md#find-most-common-geo-sync-errors) +[the `gitlab-rails/geo.log` file](../../logs/log_parsing.md#find-most-common-geo-sync-errors) ### Check if PostgreSQL replication is working @@ -191,7 +191,7 @@ If a replication slot is inactive, the `pg_wal` logs corresponding to the slot are reserved forever (or until the slot is active again). This causes continuous disk usage growth and the following messages appear repeatedly in the -[PostgreSQL logs](../../logs.md#postgresql-logs): +[PostgreSQL logs](../../logs/index.md#postgresql-logs): ```plaintext WARNING: oldest xmin is far in the past @@ -331,8 +331,7 @@ Be sure to restart PostgreSQL for this to take effect. See the This occurs when PostgreSQL does not have a replication slot for the **secondary** node by that name. -You may want to rerun the [replication -process](../setup/database.md) on the **secondary** node . +You may want to rerun the [replication process](../setup/database.md) on the **secondary** node . ### Message: "Command exceeded allowed execution time" when setting up replication? @@ -376,7 +375,7 @@ log data to build up in `pg_xlog`. Removing the unused slots can reduce the amou Slots where `active` is `f` are not active. - When this slot should be active, because you have a **secondary** node configured using that slot, - sign in to that **secondary** node and check the [PostgreSQL logs](../../logs.md#postgresql-logs) + sign in to that **secondary** node and check the [PostgreSQL logs](../../logs/index.md#postgresql-logs) to view why the replication is not running. - If you are no longer using the slot (for example, you no longer have Geo enabled), you can remove it with in the @@ -510,7 +509,7 @@ To solve this: 1. Back up [the `.git` folder](../../repository_storage_types.md#translate-hashed-storage-paths). -1. Optional. [Spot-check](../../troubleshooting/log_parsing.md#find-all-projects-affected-by-a-fatal-git-problem) +1. Optional. [Spot-check](../../logs/log_parsing.md#find-all-projects-affected-by-a-fatal-git-problem) a few of those IDs whether they indeed correspond to a project with known Geo replication failures. Use `fatal: 'geo'` as the `grep` term and the following API call: @@ -597,7 +596,7 @@ to start again from scratch, there are a few steps that can help you: gitlab-ctl stop geo-logcursor ``` - You can watch the [Sidekiq logs](../../logs.md#sidekiq-logs) to know when Sidekiq jobs processing has finished: + You can watch the [Sidekiq logs](../../logs/index.md#sidekiq-logs) to know when Sidekiq jobs processing has finished: ```shell gitlab-ctl tail sidekiq @@ -837,7 +836,7 @@ to transfer each affected repository from the primary to the secondary site. The following are possible error messages that might be encountered during failover or when promoting a secondary to a primary node with strategies to resolve them. -### Message: ActiveRecord::RecordInvalid: Validation failed: Name has already been taken +### Message: `ActiveRecord::RecordInvalid: Validation failed: Name has already been taken` When [promoting a **secondary** site](../disaster_recovery/index.md#step-3-promoting-a-secondary-site), you might encounter the following error message: @@ -869,11 +868,10 @@ or `gitlab-ctl promote-to-primary-node`, either: ``` - Upgrade to GitLab 12.6.3 or later if it is safe to do so. For example, - if the failover was just a test. A [caching-related - bug](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22021) was - fixed. + if the failover was just a test. A + [caching-related bug](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22021) was fixed. -### Message: ActiveRecord::RecordInvalid: Validation failed: Enabled Geo primary node cannot be disabled +### Message: `ActiveRecord::RecordInvalid: Validation failed: Enabled Geo primary node cannot be disabled` If you disabled a secondary node, either with the [replication pause task](../index.md#pausing-and-resuming-replication) (GitLab 13.2) or by using the user interface (GitLab 13.1 and earlier), you must first @@ -1127,12 +1125,6 @@ Geo secondary sites continue to replicate and verify data, and the secondary sit This bug was [fixed in GitLab 14.4](https://gitlab.com/gitlab-org/gitlab/-/issues/292983). -### GitLab Pages return 404 errors after promoting - -This is due to [Pages data not being managed by Geo](datatypes.md#limitations-on-replicationverification). -Find advice to resolve those error messages in the -[Pages administration documentation](../../../administration/pages/index.md#404-error-after-promoting-a-geo-secondary-to-a-primary-node). - ### Primary site returns 500 error when accessing `/admin/geo/replication/projects` Navigating to **Admin > Geo > Replication** (or `/admin/geo/replication/projects`) on a primary Geo site, shows a 500 error, while that same link on the secondary works fine. The primary's `production.log` has a similar entry to the following: @@ -1146,7 +1138,28 @@ Geo::TrackingBase::SecondaryNotConfigured: Geo secondary database is not configu On a Geo primary site this error can be ignored. -This happens because GitLab is attempting to display registries from the [Geo tracking database](../../../administration/geo/#geo-tracking-database) which doesn't exist on the primary site (only the original projects exist on the primary; no replicated projects are present, therefore no tracking database exists). +This happens because GitLab is attempting to display registries from the [Geo tracking database](../../../administration/geo/index.md#geo-tracking-database) which doesn't exist on the primary site (only the original projects exist on the primary; no replicated projects are present, therefore no tracking database exists). + +### Secondary site returns 400 error "Request header or cookie too large" + +This error can happen when the internal URL of the primary site is incorrect. + +For example, when you use a unified URL and the primary site's internal URL is also equal to the external URL. This causes a loop when a secondary site proxies requests to the primary site's internal URL. + +To fix this issue, set the primary site's internal URL to a URL that is: + +- Unique to the primary site. +- Accessible from all secondary sites. + +1. Enter the [Rails console](../../operations/rails_console.md) on the primary site. + +1. Run the following, replacing `https://unique.url.for.primary.site` with your specific internal URL. + For example, depending on your network configuration, you could use an IP address, like + `http://1.2.3.4`. + + ```ruby + GeoNode.where(primary: true).first.update!(internal_url: "https://unique.url.for.primary.site") + ``` ## Fixing client errors @@ -1158,6 +1171,23 @@ requests redirected from the secondary to the primary node do not properly send Authorization header. This may result in either an infinite `Authorization <-> Redirect` loop, or Authorization error messages. +### Error: Net::ReadTimeout when pushing through SSH on a Geo secondary + +When you push large repositories through SSH on a Geo secondary site, you may encounter a timeout. +This is because Rails proxies the push to the primary and has a 60 second default timeout, +[as described in this Geo issue](https://gitlab.com/gitlab-org/gitlab/-/issues/7405). + +Current workarounds are: + +- Push through HTTP instead, where Workhorse proxies the request to the primary (or redirects to the primary if Geo proxying is not enabled). +- Push directly to the primary. + +Example log (`gitlab-shell.log`): + +```plaintext +Failed to contact primary https://primary.domain.com/namespace/push_test.git\\nError: Net::ReadTimeout\",\"result\":null}" code=500 method=POST pid=5483 url="http://127.0.0.1:3000/api/v4/geo/proxy_git_push_ssh/push" +``` + ## Recovering from a partial failover The partial failover to a secondary Geo *site* may be the result of a temporary/transient issue. Therefore, first attempt to run the promote command again. diff --git a/doc/administration/geo/replication/tuning.md b/doc/administration/geo/replication/tuning.md index 670459624f3..755ab45a76c 100644 --- a/doc/administration/geo/replication/tuning.md +++ b/doc/administration/geo/replication/tuning.md @@ -30,7 +30,7 @@ However, this may not lead to more downloads in parallel unless the number of available Sidekiq threads is also increased. For example, if repository synchronization concurrency is increased from 25 to 50, you may also want to increase the number of Sidekiq threads from 25 to 50. See the -[Sidekiq concurrency documentation](../../operations/extra_sidekiq_processes.md#number-of-threads) +[Sidekiq concurrency documentation](../../sidekiq/extra_sidekiq_processes.md#number-of-threads) for more details. ## Repository re-verification diff --git a/doc/administration/geo/replication/upgrading_the_geo_sites.md b/doc/administration/geo/replication/upgrading_the_geo_sites.md index 30961de0381..ce1ff4fe6a5 100644 --- a/doc/administration/geo/replication/upgrading_the_geo_sites.md +++ b/doc/administration/geo/replication/upgrading_the_geo_sites.md @@ -28,12 +28,23 @@ and cause downtime. If you want to avoid downtime, consider using To upgrade the Geo sites when a new GitLab version is released, upgrade **primary** and all **secondary** sites: -1. **Optional:** [Pause replication on each **secondary** sites.](../index.md#pausing-and-resuming-replication) +1. Optional. [Pause replication on each **secondary** site](../index.md#pausing-and-resuming-replication) + to protect the disaster recovery (DR) capability of the **secondary** sites. 1. SSH into each node of the **primary** site. 1. [Upgrade GitLab on the **primary** site](../../../update/package/index.md#upgrade-using-the-official-repositories). +1. Perform testing on the **primary** site, particularly if you paused replication in step 1 to protect DR. [There are some suggestions for post-upgrade testing](../../../update/plan_your_upgrade.md#pre-upgrade-and-post-upgrade-checks) in the upgrade documentation. 1. SSH into each node of **secondary** sites. 1. [Upgrade GitLab on each **secondary** site](../../../update/package/index.md#upgrade-using-the-official-repositories). -1. If you paused replication in step 1, [resume replication on each **secondary**](../index.md#pausing-and-resuming-replication) +1. If you paused replication in step 1, [resume replication on each **secondary**](../index.md#pausing-and-resuming-replication). + Then, restart Puma and Sidekiq on each **secondary** site. This is to ensure they + are initialized against the newer database schema that is now replicated from + the previously upgraded **primary** site. + + ```shell + sudo gitlab-ctl restart sidekiq + sudo gitlab-ctl restart puma + ``` + 1. [Test](#check-status-after-upgrading) **primary** and **secondary** sites, and check version in each. ### Check status after upgrading diff --git a/doc/administration/geo/replication/version_specific_upgrades.md b/doc/administration/geo/replication/version_specific_upgrades.md index f0925bdf87e..350310c7076 100644 --- a/doc/administration/geo/replication/version_specific_upgrades.md +++ b/doc/administration/geo/replication/version_specific_upgrades.md @@ -178,11 +178,15 @@ GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab ## Upgrading to GitLab 13.7 -We've detected an issue with the `FetchRemove` call used by Geo secondaries. -This causes performance issues as we execute reference transaction hooks for -each upgraded reference. Delay any upgrade attempts until this is in the -[13.7.5 patch release.](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3002). -More details are available [in this issue](https://gitlab.com/gitlab-org/git/-/issues/79). +- We've detected an issue with the `FetchRemove` call used by Geo secondaries. + This causes performance issues as we execute reference transaction hooks for + each upgraded reference. Delay any upgrade attempts until this is in the + [13.7.5 patch release.](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3002). + More details are available [in this issue](https://gitlab.com/gitlab-org/git/-/issues/79). +- A new secret is generated in `/etc/gitlab/gitlab-secrets.json`. + In an HA GitLab or GitLab Geo environment, secrets need to be the same on all nodes. + Ensure this new secret is also accounted for if you are manually syncing the file across + nodes, or manually specifying secrets in `/etc/gitlab/gitlab.rb`. ## Upgrading to GitLab 13.5 @@ -243,8 +247,8 @@ the recommended procedure, see the ## Upgrading to GitLab 12.9 WARNING: -GitLab 12.9.0 through GitLab 12.9.3 are affected by [a bug that stops -repository verification](https://gitlab.com/gitlab-org/gitlab/-/issues/213523). +GitLab 12.9.0 through GitLab 12.9.3 are affected by +[a bug that stops repository verification](https://gitlab.com/gitlab-org/gitlab/-/issues/213523). The issue is fixed in GitLab 12.9.4. Upgrade to GitLab 12.9.4 or later. By default, GitLab 12.9 attempts to upgrade the embedded PostgreSQL server @@ -397,6 +401,6 @@ For the recommended procedure, see the ## Upgrading to GitLab 12.0 WARNING: -This version is affected by a [bug that results in new LFS objects not being -replicated to Geo secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). +This version is affected by a +[bug that results in new LFS objects not being replicated to Geo secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). The issue is fixed in GitLab 12.1. Be sure to upgrade to GitLab 12.1 or later. diff --git a/doc/administration/geo/secondary_proxy/index.md b/doc/administration/geo/secondary_proxy/index.md index e8c290e197b..6c1812b2754 100644 --- a/doc/administration/geo/secondary_proxy/index.md +++ b/doc/administration/geo/secondary_proxy/index.md @@ -112,8 +112,9 @@ gitlab: Since GitLab 15.1, Geo secondary proxying is enabled by default for separate URLs also. -There are minor known issues linked in the ["Geo secondary proxying with separate URLs" -epic](https://gitlab.com/groups/gitlab-org/-/epics/6865). You can also add feedback in the epic about any use-cases that +There are minor known issues linked in the +["Geo secondary proxying with separate URLs" epic](https://gitlab.com/groups/gitlab-org/-/epics/6865). +You can also add feedback in the epic about any use-cases that are not possible anymore with proxying enabled. If you run into issues, to disable this feature, disable the `geo_secondary_proxy_separate_urls` feature flag. diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index c0ed7829fce..8a919a0a269 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -8,7 +8,7 @@ type: howto # Geo database replication **(PREMIUM SELF)** This document describes the minimal required steps to replicate your primary -GitLab database to a secondary node's database. You may have to change some +GitLab database to a secondary site's database. You may have to change some values, based on attributes including your database's setup and size. NOTE: @@ -41,31 +41,31 @@ instructions on setting up replication with a Patroni cluster. ### PostgreSQL replication -The GitLab **primary** node where the write operations happen connects to -the **primary** database server, and **secondary** nodes +The GitLab **primary** site where the write operations happen connects to +the **primary** database server, and **secondary** sites connect to their own database servers (which are read-only). We recommend using [PostgreSQL replication slots](https://medium.com/@tk512/replication-slots-in-postgresql-b4b03d277c75) -to ensure that the **primary** node retains all the data necessary for the **secondary** nodes to +to ensure that the **primary** site retains all the data necessary for the **secondary** sites to recover. See below for more details. The following guide assumes that: - You are using Omnibus and therefore you are using PostgreSQL 12 or later which includes the [`pg_basebackup` tool](https://www.postgresql.org/docs/12/app-pgbasebackup.html). -- You have a **primary** node already set up (the GitLab server you are +- You have a **primary** site already set up (the GitLab server you are replicating from), running Omnibus' PostgreSQL (or equivalent version), and - you have a new **secondary** server set up with the same + you have a new **secondary** site set up with the same [versions of PostgreSQL](../index.md#requirements-for-running-geo), - OS, and GitLab on all nodes. + OS, and GitLab on all sites. WARNING: Geo works with streaming replication. Logical replication is not supported at this time. There is an [issue where support is being discussed](https://gitlab.com/gitlab-org/gitlab/-/issues/7420). -#### Step 1. Configure the **primary** server +#### Step 1. Configure the **primary** site -1. SSH into your GitLab **primary** server and login as root: +1. SSH into your GitLab **primary** site and login as root: ```shell sudo -i @@ -81,13 +81,13 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o gitlab_rails['geo_node_name'] = '<site_name_here>' ``` -1. Reconfigure the **primary** node for the change to take effect: +1. Reconfigure the **primary** site for the change to take effect: ```shell gitlab-ctl reconfigure ``` -1. Execute the command below to define the node as **primary** node: +1. Execute the command below to define the site as **primary** site: ```shell gitlab-ctl set-geo-primary-node @@ -154,17 +154,17 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o 1. Configure PostgreSQL to listen on network interfaces: For security reasons, PostgreSQL does not listen on any network interfaces - by default. However, Geo requires the **secondary** node to be able to - connect to the **primary** node's database. For this reason, we need the address of - each node. + by default. However, Geo requires the **secondary** site to be able to + connect to the **primary** site's database. For this reason, we need the IP address of + each site. NOTE: For external PostgreSQL instances, see [additional instructions](external_database.md). If you are using a cloud provider, you can lookup the addresses for each - Geo node through your cloud provider's management console. + Geo site through your cloud provider's management console. - To lookup the address of a Geo node, SSH in to the Geo node and execute: + To lookup the address of a Geo site, SSH in to the Geo site and execute: ```shell ## @@ -183,11 +183,11 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o | Configuration | Address | |:----------------------------------------|:----------------------------------------------------------------------| - | `postgresql['listen_address']` | **Primary** node's public or VPC private address. | - | `postgresql['md5_auth_cidr_addresses']` | **Primary** and **Secondary** nodes' public or VPC private addresses. | + | `postgresql['listen_address']` | **Primary** site's public or VPC private address. | + | `postgresql['md5_auth_cidr_addresses']` | **Primary** and **Secondary** sites' public or VPC private addresses. | If you are using Google Cloud Platform, SoftLayer, or any other vendor that - provides a virtual private cloud (VPC) you can use the **primary** and **secondary** nodes + provides a virtual private cloud (VPC) you can use the **primary** and **secondary** sites private addresses (corresponds to "internal address" for Google Cloud Platform) for `postgresql['md5_auth_cidr_addresses']` and `postgresql['listen_address']`. @@ -201,10 +201,10 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o `127.0.0.1`. For more information, see [omnibus-5258](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5258). Depending on your network configuration, the suggested addresses may not - be correct. If your **primary** node and **secondary** nodes connect over a local + be correct. If your **primary** site and **secondary** sites connect over a local area network, or a virtual network connecting availability zones like [Amazon's VPC](https://aws.amazon.com/vpc/) or [Google's VPC](https://cloud.google.com/vpc/) - you should use the **secondary** node's private address for `postgresql['md5_auth_cidr_addresses']`. + you should use the **secondary** site's private address for `postgresql['md5_auth_cidr_addresses']`. Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP addresses with addresses appropriate to your network configuration: @@ -225,19 +225,18 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ## Primary address ## - replace '<primary_node_ip>' with the public or VPC address of your Geo primary node ## - postgresql['listen_address'] = '<primary_node_ip>' + postgresql['listen_address'] = '<primary_site_ip>' ## # Allow PostgreSQL client authentication from the primary and secondary IPs. These IPs may be # public or VPC addresses in CIDR format, for example ['198.51.100.1/32', '198.51.100.2/32'] ## - postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32'] + postgresql['md5_auth_cidr_addresses'] = ['<primary_site_ip>/32', '<secondary_site_ip>/32'] ## ## Replication settings - ## - set this to be the number of Geo secondary nodes you have ## - postgresql['max_replication_slots'] = 1 + # postgresql['max_replication_slots'] = 1 # Set this to be the number of Geo secondary nodes if you have more than one # postgresql['max_wal_senders'] = 10 # postgresql['wal_keep_segments'] = 10 @@ -248,10 +247,10 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o gitlab_rails['auto_migrate'] = false ``` -1. Optional: If you want to add another **secondary** node, the relevant setting would look like: +1. Optional: If you want to add another **secondary** site, the relevant setting would look like: ```ruby - postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32', '<another_secondary_node_ip>/32'] + postgresql['md5_auth_cidr_addresses'] = ['<primary_site_ip>/32', '<secondary_site_ip>/32', '<another_secondary_site_ip>/32'] ``` You may also want to edit the `wal_keep_segments` and `max_wal_senders` to match your @@ -288,20 +287,20 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o 1. Now that the PostgreSQL server is set up to accept remote connections, run `netstat -plnt | grep 5432` to make sure that PostgreSQL is listening on port - `5432` to the **primary** server's private address. + `5432` to the **primary** site's private address. 1. A certificate was automatically generated when GitLab was reconfigured. This is used automatically to protect your PostgreSQL traffic from eavesdroppers, but to protect against active ("man-in-the-middle") attackers, - the **secondary** node needs a copy of the certificate. Make a copy of the PostgreSQL - `server.crt` file on the **primary** node by running this command: + the **secondary** site needs a copy of the certificate. Make a copy of the PostgreSQL + `server.crt` file on the **primary** site by running this command: ```shell cat ~gitlab-psql/data/server.crt ``` Copy the output into a clipboard or into a local file. You - need it when setting up the **secondary** node! The certificate is not sensitive + need it when setting up the **secondary** site! The certificate is not sensitive data. However, this certificate is created with a generic `PostgreSQL` Common Name. For this, @@ -324,7 +323,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o #### Step 2. Configure the **secondary** server -1. SSH into your GitLab **secondary** server and login as root: +1. SSH into your GitLab **secondary** site and login as root: ```shell sudo -i @@ -338,28 +337,28 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ``` NOTE: - This step is important so we don't try to execute anything before the node is fully configured. + This step is important so we don't try to execute anything before the site is fully configured. -1. [Check TCP connectivity](../../raketasks/maintenance.md) to the **primary** node's PostgreSQL server: +1. [Check TCP connectivity](../../raketasks/maintenance.md) to the **primary** site's PostgreSQL server: ```shell - gitlab-rake gitlab:tcp_check[<primary_node_ip>,5432] + gitlab-rake gitlab:tcp_check[<primary_site_ip>,5432] ``` NOTE: If this step fails, you may be using the wrong IP address, or a firewall may - be preventing access to the server. Check the IP address, paying close + be preventing access to the site. Check the IP address, paying close attention to the difference between public and private addresses and ensure - that, if a firewall is present, the **secondary** node is permitted to connect to the - **primary** node on port 5432. + that, if a firewall is present, the **secondary** site is permitted to connect to the + **primary** site on port 5432. -1. Create a file `server.crt` in the **secondary** server, with the content you got on the last step of the **primary** node's setup: +1. Create a file `server.crt` in the **secondary** site, with the content you got on the last step of the **primary** site's setup: ```shell editor server.crt ``` -1. Set up PostgreSQL TLS verification on the **secondary** node: +1. Set up PostgreSQL TLS verification on the **secondary** site: Install the `server.crt` file: @@ -374,9 +373,9 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o PostgreSQL now only recognizes that exact certificate when verifying TLS connections. The certificate can only be replicated by someone with access - to the private key, which is **only** present on the **primary** node. + to the private key, which is **only** present on the **primary** site. -1. Test that the `gitlab-psql` user can connect to the **primary** node's database +1. Test that the `gitlab-psql` user can connect to the **primary** site's database (the default Omnibus database name is `gitlabhq_production`): ```shell @@ -386,7 +385,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o -U gitlab_replicator \ -d "dbname=gitlabhq_production sslmode=verify-ca" \ -W \ - -h <primary_node_ip> + -h <primary_site_ip> ``` NOTE: @@ -397,11 +396,11 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o When prompted enter the _plaintext_ password you set in the first step for the `gitlab_replicator` user. If all worked correctly, you should see - the list of **primary** node's databases. + the list of **primary** site's databases. A failure to connect here indicates that the TLS configuration is incorrect. - Ensure that the contents of `~gitlab-psql/data/server.crt` on the **primary** node - match the contents of `~gitlab-psql/.postgresql/root.crt` on the **secondary** node. + Ensure that the contents of `~gitlab-psql/data/server.crt` on the **primary** site + match the contents of `~gitlab-psql/.postgresql/root.crt` on the **secondary** site. 1. Configure PostgreSQL: @@ -420,14 +419,14 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ## ## Secondary address - ## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node + ## - replace '<secondary_site_ip>' with the public or VPC address of your Geo secondary site ## - postgresql['listen_address'] = '<secondary_node_ip>' - postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32'] + postgresql['listen_address'] = '<secondary_site_ip>' + postgresql['md5_auth_cidr_addresses'] = ['<secondary_site_ip>/32'] ## - ## Database credentials password (defined previously in primary node) - ## - replicate same values here as defined in primary node + ## Database credentials password (defined previously in primary site) + ## - replicate same values here as defined in primary site ## postgresql['sql_replication_password'] = '<md5_hash_of_your_password>' postgresql['sql_user_password'] = '<md5_hash_of_your_password>' @@ -435,7 +434,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ``` For external PostgreSQL instances, see [additional instructions](external_database.md). - If you bring a former **primary** node back online to serve as a **secondary** node, then you also must remove `roles(['geo_primary_role'])` or `geo_primary_role['enable'] = true`. + If you bring a former **primary** site back online to serve as a **secondary** site, then you also must remove `roles(['geo_primary_role'])` or `geo_primary_role['enable'] = true`. 1. Reconfigure GitLab for the changes to take effect: @@ -451,24 +450,24 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o #### Step 3. Initiate the replication process -Below we provide a script that connects the database on the **secondary** node to -the database on the **primary** node, replicates the database, and creates the +Below we provide a script that connects the database on the **secondary** site to +the database on the **primary** site, replicates the database, and creates the needed files for streaming replication. The directories used are the defaults that are set up in Omnibus. If you have changed any defaults, configure it as you see fit replacing the directories and paths. WARNING: -Make sure to run this on the **secondary** server as it removes all PostgreSQL's +Make sure to run this on the **secondary** site as it removes all PostgreSQL's data before running `pg_basebackup`. -1. SSH into your GitLab **secondary** server and login as root: +1. SSH into your GitLab **secondary** site and login as root: ```shell sudo -i ``` -1. Choose a database-friendly name to use for your **secondary** node to +1. Choose a database-friendly name to use for your **secondary** site to use as the replication slot name. For example, if your domain is `secondary.geo.example.com`, you may use `secondary_example` as the slot name as shown in the commands below. @@ -476,13 +475,13 @@ data before running `pg_basebackup`. 1. Execute the command below to start a backup/restore and begin the replication WARNING: - Each Geo **secondary** node must have its own unique replication slot name. + Each Geo **secondary** site must have its own unique replication slot name. Using the same slot name between two secondaries breaks PostgreSQL replication. ```shell gitlab-ctl replicate-geo-database \ - --slot-name=<secondary_node_name> \ - --host=<primary_node_ip> \ + --slot-name=<secondary_site_name> \ + --host=<primary_site_ip> \ --sslmode=verify-ca ``` @@ -516,7 +515,7 @@ data before running `pg_basebackup`. - Change the `--slot-name` to the name of the replication slot to be used on the **primary** database. The script attempts to create the replication slot automatically if it does not exist. - - If you're repurposing an old server into a Geo **secondary** node, you must + - If you're repurposing an old site into a Geo **secondary** site, you must add `--force` to the command line. - When not in a production machine you can disable backup step if you really sure this is what you want by adding `--skip-backup` @@ -540,7 +539,7 @@ see [High Availability with Omnibus GitLab](../../postgresql/replication_and_fai To change the password for the [replication user](https://wiki.postgresql.org/wiki/Streaming_Replication) when using Omnibus-managed PostgreSQL instances: -On the GitLab Geo **primary** server: +On the GitLab Geo **primary** site: 1. The default value for the replication user is `gitlab_replicator`, but if you've set a custom replication user in your `/etc/gitlab/gitlab.rb` under the `postgresql['sql_replication_user']` setting, make sure to @@ -574,19 +573,18 @@ On the GitLab Geo **primary** server: sudo gitlab-ctl restart postgresql ``` -Until the password is updated on any **secondary** servers, the [PostgreSQL log](../../logs.md#postgresql-logs) on +Until the password is updated on any **secondary** sites, the [PostgreSQL log](../../logs/index.md#postgresql-logs) on the secondaries will report the following error message: ```console FATAL: could not connect to the primary server: FATAL: password authentication failed for user "gitlab_replicator" ``` -On all GitLab Geo **secondary** servers: +On all GitLab Geo **secondary** sites: 1. The first step isn't necessary from a configuration perspective, because the hashed `'sql_replication_password'` - is not used on the GitLab Geo **secondary**. However in the event that **secondary** needs to be promoted - to the GitLab Geo **primary**, make sure to match the `'sql_replication_password'` in the secondary - server configuration. + is not used on the GitLab Geo **secondary** sites. However in the event that **secondary** site needs to be promoted + to the GitLab Geo **primary**, make sure to match the `'sql_replication_password'` in the **secondary** site configuration. Edit `/etc/gitlab/gitlab.rb`: @@ -616,6 +614,28 @@ In GitLab 14.0, Patroni replaced `repmgr` as the supported NOTE: If you still haven't [migrated from repmgr to Patroni](#migrating-from-repmgr-to-patroni) you're highly advised to do so. +### Migrating from repmgr to Patroni + +1. Before migrating, we recommend that there is no replication lag between the **primary** and **secondary** sites and that replication is paused. In GitLab 13.2 and later, you can pause and resume replication with `gitlab-ctl geo-replication-pause` and `gitlab-ctl geo-replication-resume` on a Geo secondary database node. +1. Follow the [instructions to migrate repmgr to Patroni](../../postgresql/replication_and_failover.md#switching-from-repmgr-to-patroni). When configuring Patroni on each **primary** site database node, add `patroni['replication_slots'] = { '<slot_name>' => 'physical' }` +to `gitlab.rb` where `<slot_name>` is the name of the replication slot for your **secondary** site. This ensures that Patroni recognizes the replication slot as permanent and not drop it upon restarting. +1. If database replication to the **secondary** site was paused before migration, resume replication once Patroni is confirmed working on the **primary** site. + +### Migrating a single PostgreSQL node to Patroni + +Before the introduction of Patroni, Geo had no Omnibus support for HA setups on the **secondary** site. + +With Patroni it's now possible to support that. To migrate the existing PostgreSQL to Patroni: + +1. Make sure you have a Consul cluster setup on the secondary (similar to how you set it up on the **primary** site). +1. [Configure a permanent replication slot](#step-1-configure-patroni-permanent-replication-slot-on-the-primary-site). +1. [Configure the internal load balancer](#step-2-configure-the-internal-load-balancer-on-the-primary-site). +1. [Configure a PgBouncer node](#step-3-configure-pgbouncer-nodes-on-the-secondary-site) +1. [Configure a Standby Cluster](#step-4-configure-a-standby-cluster-on-the-secondary-site) + on that single node machine. + +You end up with a “Standby Cluster” with a single node. That allows you to later on add additional Patroni nodes by following the same instructions above. + ### Patroni support Patroni is the official replication management solution for Geo. It @@ -648,11 +668,11 @@ and other database best practices. ##### Step 1. Configure Patroni permanent replication slot on the primary site -To set up database replication with Patroni on a secondary node, we must -configure a _permanent replication slot_ on the primary node's Patroni cluster, +To set up database replication with Patroni on a secondary site, we must +configure a _permanent replication slot_ on the primary site's Patroni cluster, and ensure password authentication is used. -For each Patroni instance on the primary site **starting on the Patroni +On each node running a Patroni instance on the primary site **starting on the Patroni Leader instance**: 1. SSH into your Patroni instance and login as root: @@ -755,7 +775,7 @@ backend postgresql Refer to your preferred Load Balancer's documentation for further guidance. -##### Step 3. Configure a PgBouncer node on the secondary site +##### Step 3. Configure PgBouncer nodes on the secondary site A production-ready and highly available configuration requires at least three Consul nodes, a minimum of one PgBouncer node, but it's recommended to have @@ -764,7 +784,7 @@ more than one PgBouncer service nodes. The internal load balancer provides a sin endpoint for connecting to the PgBouncer cluster. For more information, see [High Availability with Omnibus GitLab](../../postgresql/replication_and_failover.md). -Follow the minimal configuration for the PgBouncer node: +On each node running a PgBouncer instance on the **secondary** site: 1. SSH into your PgBouncer node and login as root: @@ -820,11 +840,10 @@ Follow the minimal configuration for the PgBouncer node: ##### Step 4. Configure a Standby cluster on the secondary site NOTE: -If you are converting a secondary site to a Patroni Cluster, you must start -on the PostgreSQL instance. It becomes the Patroni Standby Leader instance, -and then you can switchover to another replica if you need. +If you are converting a secondary site with a single PostgreSQL instance to a Patroni Cluster, you must start on the PostgreSQL instance. It becomes the Patroni Standby Leader instance, +and then you can switch over to another replica if you need. -For each Patroni instance on the secondary site: +For each node running a Patroni instance on the secondary site: 1. SSH into your Patroni node and login as root: @@ -894,39 +913,29 @@ For each Patroni instance on the secondary site: gitlab-ctl start patroni ``` -### Migrating from repmgr to Patroni - -1. Before migrating, we recommend that there is no replication lag between the primary and secondary sites and that replication is paused. In GitLab 13.2 and later, you can pause and resume replication with `gitlab-ctl geo-replication-pause` and `gitlab-ctl geo-replication-resume` on a Geo secondary database node. -1. Follow the [instructions to migrate repmgr to Patroni](../../postgresql/replication_and_failover.md#switching-from-repmgr-to-patroni). When configuring Patroni on each primary site database node, add `patroni['replication_slots'] = { '<slot_name>' => 'physical' }` -to `gitlab.rb` where `<slot_name>` is the name of the replication slot for your Geo secondary. This ensures that Patroni recognizes the replication slot as permanent and not drop it upon restarting. -1. If database replication to the secondary was paused before migration, resume replication once Patroni is confirmed working on the primary. - -### Migrating a single PostgreSQL node to Patroni - -Before the introduction of Patroni, Geo had no Omnibus support for HA setups on the secondary node. +### Migrating a single tracking database node to Patroni -With Patroni it's now possible to support that. To migrate the existing PostgreSQL to Patroni: +Before the introduction of Patroni, Geo had no Omnibus support for HA setups on +the secondary site. -1. Make sure you have a Consul cluster setup on the secondary (similar to how you set it up on the primary). -1. [Configure a permanent replication slot](#step-1-configure-patroni-permanent-replication-slot-on-the-primary-site). -1. [Configure the internal load balancer](#step-2-configure-the-internal-load-balancer-on-the-primary-site). -1. [Configure a PgBouncer node](#step-3-configure-a-pgbouncer-node-on-the-secondary-site) -1. [Configure a Standby Cluster](#step-4-configure-a-standby-cluster-on-the-secondary-site) - on that single node machine. +With Patroni, it's now possible to support that. Due to some restrictions on the +Patroni implementation on Omnibus that do not allow us to manage two different +clusters on the same machine, we recommend setting up a new Patroni cluster for +the tracking database by following the same instructions above. -You end up with a "Standby Cluster" with a single node. That allows you to later on add additional Patroni nodes -by following the same instructions above. +The secondary nodes backfill the new tracking database, and no data +synchronization is required. ### Configuring Patroni cluster for the tracking PostgreSQL database -Secondary sites use a separate PostgreSQL installation as a tracking database to +**Secondary** sites use a separate PostgreSQL installation as a tracking database to keep track of replication status and automatically recover from potential replication issues. Omnibus automatically configures a tracking database when `roles(['geo_secondary_role'])` is set. If you want to run this database in a highly available configuration, don't use the `geo_secondary_role` above. Instead, follow the instructions below. -A production-ready and secure setup requires at least three Consul nodes, two +A production-ready and secure setup for the tracking PostgreSQL DB requires at least three Consul nodes, two Patroni nodes and one PgBouncer node on the secondary site. Because of [omnibus-6587](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6587), Consul can't track multiple @@ -935,9 +944,9 @@ services, so these must be different than the nodes used for the Standby Cluster Be sure to use [password credentials](../../postgresql/replication_and_failover.md#database-authorization-for-patroni) and other database best practices. -#### Step 1. Configure a PgBouncer node on the secondary site +#### Step 1. Configure PgBouncer nodes on the secondary site -Follow the minimal configuration for the PgBouncer node for the tracking database: +On each node running the PgBouncer service for the PostgreSQL tracking database: 1. SSH into your PgBouncer node and login as root: @@ -999,7 +1008,7 @@ Follow the minimal configuration for the PgBouncer node for the tracking databas #### Step 2. Configure a Patroni cluster -For each Patroni instance on the secondary site for the tracking database: +On each node running a Patroni instance on the secondary site for the PostgreSQL tracking database: 1. SSH into your Patroni node and login as root: @@ -1063,7 +1072,7 @@ For each Patroni instance on the secondary site for the tracking database: gitlab-ctl reconfigure ``` -#### Step 3. Configure the tracking database on the secondary nodes +#### Step 3. Configure the tracking database on the secondary sites For each node running the `gitlab-rails`, `sidekiq`, and `geo-logcursor` services: @@ -1100,19 +1109,6 @@ For each node running the `gitlab-rails`, `sidekiq`, and `geo-logcursor` service gitlab-rake db:migrate:geo ``` -### Migrating a single tracking database node to Patroni - -Before the introduction of Patroni, Geo had no Omnibus support for HA setups on -the secondary node. - -With Patroni, it's now possible to support that. Due to some restrictions on the -Patroni implementation on Omnibus that do not allow us to manage two different -clusters on the same machine, we recommend setting up a new Patroni cluster for -the tracking database by following the same instructions above. - -The secondary nodes backfill the new tracking database, and no data -synchronization is required. - ## Troubleshooting Read the [troubleshooting document](../replication/troubleshooting.md). diff --git a/doc/administration/geo/setup/index.md b/doc/administration/geo/setup/index.md index 5ddfee6774e..79b52ef71da 100644 --- a/doc/administration/geo/setup/index.md +++ b/doc/administration/geo/setup/index.md @@ -22,8 +22,10 @@ The steps below should be followed in the order they appear. **Make sure the Git If you installed GitLab using the Omnibus packages (highly recommended): +1. Confirm the [requirements for running Geo](../index.md#requirements-for-running-geo) are met. 1. [Install GitLab Enterprise Edition](https://about.gitlab.com/install/) on the nodes that serve as the **secondary** site. **Do not create an account or log in** to the new **secondary** site. The **GitLab version must match** across primary and secondary sites. 1. [Add the GitLab License](../../../user/admin_area/license.md) on the **primary** site to unlock Geo. The license must be for [GitLab Premium](https://about.gitlab.com/pricing/) or higher. +1. [Confirm network connectivity](../index.md#firewall-rules) between the **primary** and **secondary** site. 1. [Set up the database replication](database.md) (`primary (read-write) <-> secondary (read-only)` topology). 1. [Configure fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md). This step is required and needs to be done on **both** the **primary** and **secondary** sites. 1. [Configure GitLab](../replication/configuration.md) to set the **primary** and **secondary** sites. diff --git a/doc/administration/get_started.md b/doc/administration/get_started.md index b2bdd876499..44b09ef185a 100644 --- a/doc/administration/get_started.md +++ b/doc/administration/get_started.md @@ -37,8 +37,8 @@ Watch an overview of [groups and projects](https://www.youtube.com/watch?v=cqb2m Get started: - Create a [project](../user/project/working_with_projects.md#create-a-project). -- Create a [group](../user/group/index.md#create-a-group). -- [Add members](../user/group/index.md#add-users-to-a-group) to the group. +- Create a [group](../user/group/manage.md#create-a-group). +- [Add members](../user/group/manage.md#add-users-to-a-group) to the group. - Create a [subgroup](../user/group/subgroups/index.md#create-a-subgroup). - [Add members](../user/group/subgroups/index.md#subgroup-membership) to the subgroup. - Enable [external authorization control](../user/admin_area/settings/external_authorization.md#configuration). @@ -144,7 +144,7 @@ You can restore a backup only to **the exact same version and type** (Community ### Back up GitLab SaaS -Backups of GitLab databases and filesystems are taken every 24 hours, and are kept for two weeks on a rolling schedule. All backups are encrypted. +Backups of GitLab databases and file systems are taken every 24 hours, and are kept for two weeks on a rolling schedule. All backups are encrypted. - GitLab SaaS creates backups to ensure your data is secure, but you can't use these methods to export or back up your data yourself. - Issues are stored in the database. They can't be stored in Git itself. diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md index 4b2832bebc0..5b868c274cd 100644 --- a/doc/administration/gitaly/configure_gitaly.md +++ b/doc/administration/gitaly/configure_gitaly.md @@ -280,8 +280,9 @@ Updates to example must be made at: ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -1. Run `sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml` - to confirm that Gitaly can perform callbacks to the GitLab internal API. +1. Confirm that Gitaly can perform callbacks to the GitLab internal API: + - For GitLab 15.3 and later, run `sudo /opt/gitlab/embedded/bin/gitaly check /var/opt/gitlab/gitaly/config.toml`. + - For GitLab 15.2 and earlier, run `sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml`. **For installations from source** @@ -330,8 +331,9 @@ Updates to example must be made at: ``` 1. Save the files and [restart GitLab](../restart_gitlab.md#installations-from-source). -1. Run `sudo -u git /home/git/gitaly/gitaly-hooks check /home/git/gitaly/config.toml` - to confirm that Gitaly can perform callbacks to the GitLab internal API. +1. Confirm that Gitaly can perform callbacks to the GitLab internal API: + - For GitLab 15.3 and later, run `sudo /opt/gitlab/embedded/bin/gitaly check /var/opt/gitlab/gitaly/config.toml`. + - For GitLab 15.2 and earlier, run `sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml`. WARNING: If directly copying repository data from a GitLab server to Gitaly, ensure that the metadata file, @@ -792,7 +794,7 @@ gitaly['concurrency'] = [ - `max_per_repo` is the maximum number of in-flight RPC calls for the given RPC per repository. - `max_queue_time` is the maximum amount of time a request can wait in the concurrency queue to be picked up by Gitaly. -- `max_queue_size` is the maximum size the concurrency queue can grow to before requests are rejected by +- `max_queue_size` is the maximum size the concurrency queue (per RPC method) can grow to before requests are rejected by Gitaly. This limits the number of in-flight RPC calls for the given RPCs. The limit is applied per @@ -811,26 +813,12 @@ information, see the [relevant documentation](monitoring.md#monitor-gitaly-concu ## Control groups FLAG: -On self-managed GitLab, by default cgroups are not available. To make it available, ask an administrator to +On self-managed GitLab, by default repository cgroups are not available. To make it available, ask an administrator to [enable the feature flag](../feature_flags.md) named `gitaly_run_cmds_in_cgroup`. -Gitaly shells out to Git for many of its operations. Git can consume a lot of resources for certain operations, -especially for large repositories. - Control groups (cgroups) in Linux allow limits to be imposed on how much memory and CPU can be consumed. See the [`cgroups` Linux man page](https://man7.org/linux/man-pages/man7/cgroups.7.html) for more information. -cgroups can be useful for protecting the system against resource exhaustion because of overcomsumption of memory and CPU. - -Gitaly has built-in cgroups control. When configured, Gitaly assigns Git -processes to a cgroup based on the repository the Git command is operating in. -Each cgroup has a memory and CPU limit. When a cgroup reaches its: - -- Memory limit, the kernel looks through the processes for a candidate to kill. -- CPU limit, processes are not killed, but the processes are prevented from consuming more CPU than allowed. - -The main reason to configure cgroups for your GitLab installation is that it -protects against system resource starvation due to a few large repositories or -bad actors. +cgroups can be useful for protecting the system against resource exhaustion because of over consumption of memory and CPU. Some Git operations are expensive by nature. `git clone`, for instance, spawns a `git-upload-pack` process on the server that can consume a lot of memory @@ -838,33 +826,33 @@ for large repositories. For example, a client that keeps on cloning a large repository over and over again. This situation could potentially use up all of the memory on a server, causing other operations to fail for other users. -There are many ways someone can create a repository that can consume large amounts of memory when cloned or downloaded. +A repository can consume large amounts of memory for many reasons when cloned or downloaded. Using cgroups allows the kernel to kill these operations before they hog up all system resources. -### Configure cgroups in Gitaly +Gitaly shells out to Git for many of its operations. Git can consume a lot of resources for certain operations, +especially for large repositories. -Two ways of configuring cgroups are available. +Gitaly has built-in cgroups control. When configured, Gitaly assigns Git processes to a cgroup based on the repository +the Git command is operating in. These cgroups are called repository cgroups. Each repository cgroup: -#### Configure cgroups (new method) +- Has a memory and CPU limit. +- Contains the Git processes for a single repository. +- Uses a consistent hash to ensure a Git process for a given repository always ends up in the same cgroup. -> This method of configuring cgroups introduced in GitLab 15.1. +When a repository cgroup reaches its: -Gitaly creates a pool of cgroups that are isolated based on the repository used in the Git command to be placed under one of these cgroups. +- Memory limit, the kernel looks through the processes for a candidate to kill. +- CPU limit, processes are not killed, but the processes are prevented from consuming more CPU than allowed. -To configure cgroups in Gitaly, add `gitaly['cgroups']` to `/etc/gitlab/gitlab.rb`. +You configure repository cgroups for your GitLab installation to protect against system resource starvation from a few +large repositories or bad actors. -For example: +### Configure repository cgroups (new method) -```ruby -# in /etc/gitlab/gitlab.rb -gitaly['cgroups_mountpoint'] = "/sys/fs/cgroup" -gitaly['cgroups_hierarchy_root'] =>"gitaly" -gitaly['cgroups_memory_bytes'] = 64424509440, # 60gb -gitaly['cgroups_cpu_shares'] = 1024 -gitaly['cgroups_repositories_count'] => 1000, -gitaly['cgroups_repositories_memory_bytes'] => 32212254720 # 20gb -gitaly['cgroups_repositories_cpu_shares'] => 512 -``` +> This method of configuring repository cgroups was introduced in GitLab 15.1. + +To configure repository cgroups in Gitaly using the new method, use the following settings for the new configuration method +to `gitaly['cgroups']` in `/etc/gitlab/gitlab.rb`: - `cgroups_mountpoint` is where the parent cgroup directory is mounted. Defaults to `/sys/fs/cgroup`. - `cgroups_hierarchy_root` is the parent cgroup under which Gitaly creates groups, and @@ -873,7 +861,7 @@ gitaly['cgroups_repositories_cpu_shares'] => 512 when Gitaly starts. - `cgroups_memory_bytes` is the total memory limit that is imposed collectively on all Git processes that Gitaly spawns. 0 implies no limit. -- `cgroups_cpu_shares` is the cpu limit that is imposed collectively on all Git +- `cgroups_cpu_shares` is the CPU limit that is imposed collectively on all Git processes that Gitaly spawns. 0 implies no limit. The maximum is 1024 shares, which represents 100% of CPU. - `cgroups_repositories_count` is the number of cgroups in the cgroups pool. Each time a new Git @@ -881,30 +869,30 @@ gitaly['cgroups_repositories_cpu_shares'] => 512 on the repository the command is for. A circular hashing algorithm assigns Git commands to these cgroups, so a Git command for a repository is always assigned to the same cgroup. -- `cgroups_repositories_memory_bytes` is the total memory limit that is imposed collectively on all - Git processes that Gitaly spawns. 0 implies no limit. This value cannot exceed - that of the top level `cgroups_memory_bytes`. -- `cgroups_repositories_cpu_shares` is the CPU limit that is imposed collectively on all Git - processes Gitaly spawns. 0 implies no limit. The maximum is 1024 shares, - which represents 100% of CPU. This value cannot exceed that of the top - level`cgroups_cpu_shares`. - -#### Configure cgroups (legacy method) +- `cgroups_repositories_memory_bytes` is the total memory limit imposed on all Git processes contained in a repository cgroup. + 0 implies no limit. This value cannot exceed that of the top level `cgroups_memory_bytes`. +- `cgroups_repositories_cpu_shares` is the CPU limit that is imposed on all Git processes contained in a repository cgroup. + 0 implies no limit. The maximum is 1024 shares, which represents 100% of CPU. + This value cannot exceed that of the top level`cgroups_cpu_shares`. -To configure cgroups in Gitaly for GitLab versions using the legacy method, add `gitaly['cgroups']` to `/etc/gitlab/gitlab.rb`. For -example: +For example: ```ruby # in /etc/gitlab/gitlab.rb -gitaly['cgroups_count'] = 1000 gitaly['cgroups_mountpoint'] = "/sys/fs/cgroup" -gitaly['cgroups_hierarchy_root'] = "gitaly" -gitaly['cgroups_memory_limit'] = 32212254720 -gitaly['cgroups_memory_enabled'] = true +gitaly['cgroups_hierarchy_root'] => "gitaly" +gitaly['cgroups_memory_bytes'] = 64424509440, # 60gb gitaly['cgroups_cpu_shares'] = 1024 -gitaly['cgroups_cpu_enabled'] = true +gitaly['cgroups_repositories_count'] => 1000, +gitaly['cgroups_repositories_memory_bytes'] => 32212254720 # 20gb +gitaly['cgroups_repositories_cpu_shares'] => 512 ``` +### Configure repository cgroups (legacy method) + +To configure repository cgroups in Gitaly using the legacy method, use the following settings +in `/etc/gitlab/gitlab.rb`: + - `cgroups_count` is the number of cgroups created. Each time a new command is spawned, Gitaly assigns it to one of these cgroups based on the command line arguments of the command. A circular hashing algorithm assigns @@ -917,7 +905,21 @@ gitaly['cgroups_cpu_enabled'] = true - `cgroups_memory_enabled` enables or disables the memory limit on cgroups. - `cgroups_memory_bytes` is the total memory limit each cgroup imposes on the processes added to it. - `cgroups_cpu_enabled` enables or disables the CPU limit on cgroups. -- `cgroups_cpu_shares` is the CPU limit each cgroup imposes on the processes added to it. The maximum is 1024 shares, which represents 100% of CPU. +- `cgroups_cpu_shares` is the CPU limit each cgroup imposes on the processes added to it. The maximum is 1024 shares, + which represents 100% of CPU. + +For example: + +```ruby +# in /etc/gitlab/gitlab.rb +gitaly['cgroups_count'] = 1000 +gitaly['cgroups_mountpoint'] = "/sys/fs/cgroup" +gitaly['cgroups_hierarchy_root'] = "gitaly" +gitaly['cgroups_memory_limit'] = 32212254720 +gitaly['cgroups_memory_enabled'] = true +gitaly['cgroups_cpu_shares'] = 1024 +gitaly['cgroups_cpu_enabled'] = true +``` ### Configuring oversubscription @@ -926,16 +928,15 @@ In the previous example using the new configuration method: - The top level memory limit is capped at 60gb. - Each of the 1000 cgroups in the repositories pool is capped at 20gb. -This is called "oversubscription". Each cgroup in the pool has a much larger capacity than 1/1000th +This configuration leads to "oversubscription". Each cgroup in the pool has a much larger capacity than 1/1000th of the top-level memory limit. This strategy has two main benefits: -- It gives the host protection from overall memory starvation (OOM), because the top-level - cgroup's memory limit can be set to a threshold smaller than the host's - capacity. Processes outside of that cgroup are not at risk of OOM. +- It gives the host protection from overall memory starvation (OOM), because the memory limit of the top-level cgroup + can be set to a threshold smaller than the host's capacity. Processes outside of that cgroup are not at risk of OOM. - It allows each individual cgroup in the pool to burst up to a generous upper - bound (in this example 20 GB) that is smaller than the parent cgroup's limit, + bound (in this example 20 GB) that is smaller than the limit of the parent cgroup, but substantially larger than 1/N of the parent's limit. In this example, up to 3 child cgroups can concurrently burst up to their max. In general, all 1000 cgroups would use much less than the 20 GB. @@ -1141,8 +1142,7 @@ gitaly['pack_objects_cache_enabled'] = true #### `enabled` defaults to `false` The cache is disabled by default. This is because in some cases, it -can create an [extreme -increase](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4010#note_534564684) +can create an [extreme increase](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4010#note_534564684) in the number of bytes written to disk. On GitLab.com, we have verified that our repository storage disks can handle this extra workload, but we felt we cannot assume this is true everywhere. @@ -1302,14 +1302,26 @@ process repositories that do not pass consistency checks. For Omnibus GitLab installations, edit `/etc/gitlab/gitlab.rb` and set the following keys (in this example, to disable the `hasDotgit` consistency check): -```ruby -ignored_git_errors = ["hasDotgit = ignore"] -omnibus_gitconfig['system'] = { - "fsck" => ignored_git_errors, - "fetch.fsck" => ignored_git_errors, - "receive.fsck" => ignored_git_errors, -} -``` +- In [GitLab 15.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6800) and later: + + ```ruby + gitaly['gitconfig'] = [ + { key: "fsck.hasDotgit", value: "ignore" }, + { key: "fetch.fsck.hasDotgit", value: "ignore" }, + { key: "receive.fsck.hasDotgit", value: "ignore "}, + ] + ``` + +- In GitLab 15.2 and earlier (legacy method): + + ```ruby + ignored_git_errors = ["hasDotgit = ignore"] + omnibus_gitconfig['system'] = { + "fsck" => ignored_git_errors, + "fetch.fsck" => ignored_git_errors, + "receive.fsck" => ignored_git_errors, + } + ``` For source installs, edit the Gitaly configuration (`gitaly.toml`) to do the equivalent: diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index c543f62f135..b053da7ac9b 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -70,7 +70,7 @@ the current status of these issues, please refer to the referenced issues and ep | Issue | Summary | How to avoid | |:--------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------| -| Gitaly Cluster + Geo - Issues retrying failed syncs | If Gitaly Cluster is used on a Geo secondary site, repositories that have failed to sync could continue to fail when Geo tries to resync them. Recovering from this state requires assistance from support to run manual steps. Work is in-progress to update Gitaly Cluster to [identify repositories with a unique and persistent identifier](https://gitlab.com/gitlab-org/gitaly/-/issues/3485), which is expected to resolve the issue. | No known solution at this time. | +| Gitaly Cluster + Geo - Issues retrying failed syncs | If Gitaly Cluster is used on a Geo secondary site, repositories that have failed to sync could continue to fail when Geo tries to resync them. Recovering from this state requires assistance from support to run manual steps. | No known solution prior to GitLab 15.0. In GitLab 15.0 to 15.2, enable the [`gitaly_praefect_generated_replica_paths` feature flag](#praefect-generated-replica-paths-gitlab-150-and-later). In GitLab 15.3, the feature flag is enabled by default. | | Praefect unable to insert data into the database due to migrations not being applied after an upgrade | If the database is not kept up to date with completed migrations, then the Praefect node is unable to perform normal operation. | Make sure the Praefect database is up and running with all migrations completed (For example: `/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migrate-status` should show a list of all applied migrations). Consider [requesting upgrade assistance](https://about.gitlab.com/support/scheduling-upgrade-assistance/) so your upgrade plan can be reviewed by support. | | Restoring a Gitaly Cluster node from a snapshot in a running cluster | Because the Gitaly Cluster runs with consistent state, introducing a single node that is behind will result in the cluster not being able to reconcile the nodes data and other nodes data | Don't restore a single Gitaly Cluster node from a backup snapshot. If you must restore from backup, it's best to snapshot all Gitaly Cluster nodes at the same time and take a database dump of the Praefect database. | @@ -80,16 +80,31 @@ Gitaly Cluster does not support snapshot backups. Snapshot backups can cause iss out of sync with the disk storage. Because of how Praefect rebuilds the replication metadata of Gitaly disk information during a restore, we recommend using the [official backup and restore Rake tasks](../../raketasks/backup_restore.md). -If you are unable to use this method, please contact customer support for restoration help. +The [incremental backup method](../../raketasks/backup_gitlab.md#incremental-repository-backups) +can be used to speed up Gitaly Cluster backups. -We are tracking in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/351383) improvements to the -[official backup and restore Rake tasks](../../raketasks/backup_restore.md) to add support for incremental backups. For -more information, see [this epic](https://gitlab.com/groups/gitlab-org/-/epics/2094). +If you are unable to use either method, please contact customer support for restoration help. ### What to do if you are on Gitaly Cluster experiencing an issue or limitation Please contact customer support for immediate help in restoration or recovery. +## Directly accessing repositories + +GitLab doesn't advise directly accessing Gitaly repositories stored on disk with a Git client or any other tool, +because Gitaly is being continuously improved and changed. These improvements may invalidate +your assumptions, resulting in performance degradation, instability, and even data loss. For example: + +- Gitaly has optimizations such as the [`info/refs` advertisement cache](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/design_diskcache.md), + that rely on Gitaly controlling and monitoring access to repositories by using the official gRPC + interface. +- [Gitaly Cluster](#gitaly-cluster) has optimizations, such as fault tolerance and + [distributed reads](#distributed-reads), that depend on the gRPC interface and database + to determine repository state. + +WARNING: +Accessing Git repositories directly is done at your own risk and is not supported. + ## Gitaly The following shows GitLab set up to use direct access to Gitaly: @@ -141,6 +156,11 @@ Gitaly comes pre-configured with Omnibus GitLab, which is a configuration GitLab installations for more than 2000 active users performing daily Git write operation may be best suited by using Gitaly Cluster. +### Backing up repositories + +When backing up or syncing repositories using tools other than GitLab, you must [prevent writes](../../raketasks/backup_restore.md#prevent-writes-and-copy-the-git-repository-data) +while copying repository data. + ## Gitaly Cluster Git storage is provided through the Gitaly service in GitLab, and is essential to the operation of @@ -297,11 +317,12 @@ follow the [hashed storage](../repository_storage_types.md#hashed-storage) schem #### Praefect-generated replica paths (GitLab 15.0 and later) -> Introduced in GitLab 15.0 behind [a feature flag](https://gitlab.com/gitlab-org/gitaly/-/issues/4218) named `gitaly_praefect_generated_replica_paths`. Disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4218) in GitLab 15.0 [with a flag](../feature_flags.md) named `gitaly_praefect_generated_replica_paths`. Disabled by default. +> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitaly/-/issues/4218) in GitLab 15.2. +> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4809) in GitLab 15.3. FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../feature_flags.md) -named `gitaly_praefect_generated_replica_paths`. On GitLab.com, this feature is available but can be configured by GitLab.com administrators only. The feature is not ready for production use. +On self-managed GitLab, by default this feature is available. To hide the feature, ask an administrator to [disable the feature flag](../feature_flags.md) named `gitaly_praefect_generated_replica_paths`. On GitLab.com, this feature is available but can be configured by GitLab.com administrators only. When Gitaly Cluster creates a repository, it assigns the repository a unique and permanent ID called the _repository ID_. The repository ID is internal to Gitaly Cluster and doesn't relate to any IDs elsewhere in GitLab. If a repository is removed from Gitaly Cluster and later moved @@ -525,12 +546,9 @@ To upgrade a Gitaly Cluster, follow the documentation for ### Downgrade Gitaly Cluster to a previous version -If you need to roll back a Gitaly Cluster to an earlier version, some Praefect database migrations may need to be reverted. In a cluster with: +If you need to roll back a Gitaly Cluster to an earlier version, some Praefect database migrations may need to be reverted. -- A single Praefect node, this happens when GitLab itself is downgraded. -- Multiple Praefect nodes, additional steps are required. - -To downgrade a Gitaly Cluster with multiple Praefect nodes: +To downgrade a Gitaly Cluster (assuming multiple Praefect nodes): 1. Stop the Praefect service on all Praefect nodes: @@ -565,7 +583,7 @@ To downgrade a Gitaly Cluster with multiple Praefect nodes: gitlab-ctl start praefect ``` -## Migrate to Gitaly Cluster +### Migrate to Gitaly Cluster WARNING: Some [known issues](#known-issues) exist in Gitaly Cluster. Review the following information before you continue. @@ -587,7 +605,7 @@ To migrate to Gitaly Cluster: Even if you don't use the `default` repository storage, you must ensure it is configured. [Read more about this limitation](configure_gitaly.md#gitlab-requires-a-default-repository-storage). -## Migrate off Gitaly Cluster +### Migrate off Gitaly Cluster If the limitations and tradeoffs of Gitaly Cluster are found to be not suitable for your environment, you can Migrate off Gitaly Cluster to a sharded Gitaly instance: @@ -596,22 +614,6 @@ off Gitaly Cluster to a sharded Gitaly instance: 1. [Move the repositories](../operations/moving_repositories.md#move-repositories) to the newly created storage. You can move them by shard or by group, which gives you the opportunity to spread them over multiple Gitaly servers. -## Do not bypass Gitaly - -GitLab doesn't advise directly accessing Gitaly repositories stored on disk with a Git client, -because Gitaly is being continuously improved and changed. These improvements may invalidate -your assumptions, resulting in performance degradation, instability, and even data loss. For example: - -- Gitaly has optimizations such as the [`info/refs` advertisement cache](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/design_diskcache.md), - that rely on Gitaly controlling and monitoring access to repositories by using the official gRPC - interface. -- [Gitaly Cluster](#gitaly-cluster) has optimizations, such as fault tolerance and - [distributed reads](#distributed-reads), that depend on the gRPC interface and database - to determine repository state. - -WARNING: -Accessing Git repositories directly is done at your own risk and is not supported. - ## Direct access to Git in GitLab Direct access to Git uses code in GitLab known as the "Rugged patches". @@ -683,8 +685,12 @@ To see if GitLab can access the repository file system directly, we use the foll - GitLab Rails tries to read the metadata file directly. If it exists, and if the UUID's match, assume we have direct access. -Direct Git access is enable by default in Omnibus GitLab because it fills in the correct repository -paths in the GitLab configuration file `config/gitlab.yml`. This satisfies the UUID check. +Versions of GitLab 15.3 and later disable direct Git access by default. + +For versions of GitLab prior to 15.3, direct Git access is enabled by +default in Omnibus GitLab because it fills in the correct repository +paths in the GitLab configuration file `config/gitlab.yml`. This +satisfies the UUID check. ### Transition to Gitaly Cluster diff --git a/doc/administration/gitaly/monitoring.md b/doc/administration/gitaly/monitoring.md index 0dd9e0d7128..a435fff74fc 100644 --- a/doc/administration/gitaly/monitoring.md +++ b/doc/administration/gitaly/monitoring.md @@ -34,7 +34,7 @@ of requests dropped due to request limiting. The `reason` label indicates why a You can observe specific behavior of [concurrency-queued requests](configure_gitaly.md#limit-rpc-concurrency) using the Gitaly logs and Prometheus: -- In the [Gitaly logs](../logs.md#gitaly-logs), look for the string (or structured log field) +- In the [Gitaly logs](../logs/index.md#gitaly-logs), look for the string (or structured log field) `acquire_ms`. Messages that have this field are reporting about the concurrency limiter. - In Prometheus, look for the following metrics: - `gitaly_concurrency_limiting_in_progress` indicates how many concurrent requests are @@ -188,7 +188,7 @@ To monitor [repository verification](praefect.md#repository-verification), use t - `gitaly_praefect_stale_verification_leases_released_total`, the number of stale verification leases released. -You can also monitor the [Praefect logs](../logs.md#praefect-logs). +You can also monitor the [Praefect logs](../logs/index.md#praefect-logs). ### Database metrics `/db_metrics` endpoint diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index 5635898293b..488e6a0df5f 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -97,8 +97,8 @@ If you [installed](https://about.gitlab.com/install/) GitLab using the Omnibus G ### Preparation -Before beginning, you should already have a working GitLab instance. [Learn how -to install GitLab](https://about.gitlab.com/install/). +Before beginning, you should already have a working GitLab instance. +[Learn how to install GitLab](https://about.gitlab.com/install/). Provision a PostgreSQL server. We recommend using the PostgreSQL that is shipped with Omnibus GitLab and use it to configure the PostgreSQL database. You can use an @@ -331,8 +331,8 @@ To configure the additional connection, you must either: #### Configure a new PgBouncer database with `pool_mode = session` We recommend using PgBouncer with `session` pool mode. You can use the -[bundled PgBouncer](../postgresql/pgbouncer.md) or use an external PgBouncer and [configure it -manually](https://www.pgbouncer.org/config.html). +[bundled PgBouncer](../postgresql/pgbouncer.md) or use an external PgBouncer and +[configure it manually](https://www.pgbouncer.org/config.html). The following example uses the bundled PgBouncer and sets up two separate connection pools on PostgreSQL host, one in `session` pool mode and the other in `transaction` pool mode. For this example to work, @@ -620,8 +620,8 @@ Updates to example must be made at: gitlab-ctl reconfigure ``` -1. To ensure that Praefect [has updated its Prometheus listen - address](https://gitlab.com/gitlab-org/gitaly/-/issues/2734), +1. To ensure that Praefect + [has updated its Prometheus listen address](https://gitlab.com/gitlab-org/gitaly/-/issues/2734), [restart Praefect](../restart_gitlab.md#omnibus-gitlab-restart): ```shell @@ -928,8 +928,8 @@ For more information on Gitaly server configuration, see our gitlab-ctl reconfigure ``` -1. To ensure that Gitaly [has updated its Prometheus listen - address](https://gitlab.com/gitlab-org/gitaly/-/issues/2734), +1. To ensure that Gitaly + [has updated its Prometheus listen address](https://gitlab.com/gitlab-org/gitaly/-/issues/2734), [restart Gitaly](../restart_gitlab.md#omnibus-gitlab-restart): ```shell @@ -1103,10 +1103,8 @@ Particular attention should be shown to: ``` 1. Verify on each Gitaly node the Git Hooks can reach GitLab. On each Gitaly node run: - - ```shell - /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml - ``` + - For GitLab 15.3 and later, run `sudo /opt/gitlab/embedded/bin/gitaly check /var/opt/gitlab/gitaly/config.toml`. + - For GitLab 15.2 and earlier, run `sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml`. 1. Verify that GitLab can reach Praefect: @@ -1151,8 +1149,7 @@ running multiple Gitaly storages. ### Grafana Grafana is included with GitLab, and can be used to monitor your Praefect -cluster. See [Grafana Dashboard -Service](https://docs.gitlab.com/omnibus/settings/grafana.html) +cluster. See [Grafana Dashboard Service](https://docs.gitlab.com/omnibus/settings/grafana.html) for detailed documentation. To get started quickly: @@ -1320,8 +1317,7 @@ praefect['background_verification_verification_interval'] = '0' WARNING: Deletions are disabled by default due to a race condition with repository renames that can cause incorrect deletions. This is especially prominent in Geo instances as Geo performs more renames than instances without Geo. -See [Handle repository creations, deletions and renames atomically](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4101) -for progress on a fix. We do not recommend enabling the deletions until this is fixed. +You should enable deletions only if the [`gitaly_praefect_generated_replica_paths` feature flag](index.md#praefect-generated-replica-paths-gitlab-150-and-later) is enabled. By default, the worker does not delete invalid metadata records but simply logs them and outputs Prometheus metrics for them. diff --git a/doc/administration/gitaly/recovery.md b/doc/administration/gitaly/recovery.md index 551a7ab289a..bd4846a986d 100644 --- a/doc/administration/gitaly/recovery.md +++ b/doc/administration/gitaly/recovery.md @@ -296,6 +296,7 @@ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.t > - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3767) in GitLab 14.3. > - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4054) in GitLab 14.6, support for dry-run mode. +> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4715) in GitLab 15.3, support for removing repositories from Praefect's database. The `remove-repository` Praefect sub-command removes a repository from a Gitaly Cluster, and all state associated with a given repository including: @@ -310,6 +311,9 @@ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.t - Replace `<virtual-storage>` with the name of the virtual storage containing the repository. - Replace `<repository>` with the relative path of the repository to remove. +- In GitLab 15.3 and later, add `-db-only` to remove the Praefect database entry without removing the on-disk repository. Use this option to remove orphaned database entries and to + protect on-disk repository data from deletion when a valid repository is accidentally specified. If the database entry is accidentally deleted, re-track the repository with the + [`track-repository` command](#manually-track-repositories). - In GitLab 14.6 and later, add `-apply` to run the command outside of dry-run mode and remove the repository. For example: ```shell diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index fb3a31d61b8..e308fa9da43 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -13,7 +13,7 @@ Refer to the information below when troubleshooting Gitaly and Gitaly Cluster. The following sections provide possible solutions to Gitaly errors. See also [Gitaly timeout](../../user/admin_area/settings/gitaly_timeouts.md) settings, -and our advice on [parsing the `gitaly/current` file](../troubleshooting/log_parsing.md#parsing-gitalycurrent). +and our advice on [parsing the `gitaly/current` file](../logs/log_parsing.md#parsing-gitalycurrent). ### Check versions when using standalone Gitaly servers @@ -24,6 +24,17 @@ as GitLab to ensure full compatibility: 1. On the left sidebar, select **Overview > Gitaly Servers**. 1. Confirm all Gitaly servers indicate that they are up to date. +### Find storage resource details + +You can run the following commands in a [Rails console](../operations/rails_console.md#starting-a-rails-console-session) +to determine the available and used space on a Gitaly storage: + +```ruby +Gitlab::GitalyClient::ServerService.new("default").storage_disk_statistics +# For Gitaly Cluster +Gitlab::GitalyClient::ServerService.new("<storage name>").disk_statistics +``` + ### Use `gitaly-debug` The `gitaly-debug` command provides "production debugging" tools for Gitaly and Git @@ -444,15 +455,15 @@ If you receive an error, check `/var/log/gitlab/gitlab-rails/production.log`. Here are common errors and potential causes: - 500 response code - - **ActionView::Template::Error (7:permission denied)** + - `ActionView::Template::Error (7:permission denied)` - `praefect['auth_token']` and `gitlab_rails['gitaly_token']` do not match on the GitLab server. - - **Unable to save project. Error: 7:permission denied** + - `Unable to save project. Error: 7:permission denied` - Secret token in `praefect['storage_nodes']` on GitLab server does not match the value in `gitaly['auth_token']` on one or more Gitaly servers. - 503 response code - - **GRPC::Unavailable (14:failed to connect to all addresses)** + - `GRPC::Unavailable (14:failed to connect to all addresses)` - GitLab was unable to reach Praefect. - - **GRPC::Unavailable (14:all SubCons are in TransientFailure...)** + - `GRPC::Unavailable (14:all SubCons are in TransientFailure...)` - Praefect cannot reach one or more of its child Gitaly nodes. Try running the Praefect connection checker to diagnose. diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index 95301ec026a..9351a101ee4 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -75,11 +75,12 @@ Email is processed correctly when a configured email address is present in one o (sorted in the order they are checked): - `To` -- `References` - `Delivered-To` - `Envelope-To` or `X-Envelope-To` - `Received` +The `References` header is also accepted, however it is used specifically to relate email responses to existing discussion threads. It is not used for creating issues by email. + In GitLab 14.6 and later, [Service Desk](../user/project/service_desk.md) also checks accepted headers. diff --git a/doc/administration/index.md b/doc/administration/index.md index 4565c0a8e00..3f2ae3170ab 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -207,9 +207,8 @@ Learn how to install, configure, update, and maintain your GitLab instance. ## Troubleshooting -- [Log system](logs.md): Where to look for logs. +- [Log system](logs/index.md): Where to look for logs. - [Sidekiq Troubleshooting](troubleshooting/sidekiq.md): Debug when Sidekiq appears hung and is not processing jobs. -- [Troubleshooting Elasticsearch](troubleshooting/elasticsearch.md) - [Navigating GitLab via Rails console](operations/rails_console.md) - [GitLab application limits](instance_limits.md) - [Responding to security incidents](../security/responding_to_security_incidents.md) @@ -241,4 +240,4 @@ who are aware of the risks. - [Testing with OpenSSL](https://www.feistyduck.com/library/openssl-cookbook/online/testing-with-openssl/index.html) - [`strace` zine](https://wizardzines.com/zines/strace/) - GitLab.com-specific resources: - - [Group SAML/SCIM setup](troubleshooting/group_saml_scim.md) + - [Example group SAML and SCIM configurations](../user/group/saml_sso/example_saml_config.md) diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index 5158e2bbdd9..31434a8ee6f 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -639,6 +639,7 @@ setting is used: | `ci_max_artifact_size_secret_detection` | 0 | | `ci_max_artifact_size_terraform` | 5 MB ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37018) in GitLab 13.3) | | `ci_max_artifact_size_trace` | 0 | +| `ci_max_artifact_size_cyclonedx` | 1 MB | For example, to set the `ci_max_artifact_size_junit` limit to 10 MB on a self-managed installation, run the following in the [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): @@ -1047,7 +1048,8 @@ The [secure files API](../api/secure_files.md) enforces the following limits: ## Changelog API limits -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/89032) in GitLab 15.1 [with a flag](../administration/feature_flags.md) named `changelog_commits_limitation`. Disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/89032) in GitLab 15.1 [with a flag](../administration/feature_flags.md) named `changelog_commits_limitation`. Disabled by default. +> - [Enabled on GitLab.com and by default on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/33893) in GitLab 15.3. The [changelog API](../api/repositories.md#add-changelog-data-to-a-changelog-file) enforces the following limits: diff --git a/doc/administration/integration/mailgun.md b/doc/administration/integration/mailgun.md index 6b0cb0466fc..c007116d213 100644 --- a/doc/administration/integration/mailgun.md +++ b/doc/administration/integration/mailgun.md @@ -18,8 +18,8 @@ After completing the integration, Mailgun `temporary_failure` and `permanent_fai ## Configure your Mailgun domain -> [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/359113) the `/-/members/mailgun/permanent_failures` URL in GitLab 15.0. -> [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/359113) the URL to handle both temporary and permanent failures in GitLab 15.0. +> - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/359113) the `/-/members/mailgun/permanent_failures` URL in GitLab 15.0. +> - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/359113) the URL to handle both temporary and permanent failures in GitLab 15.0. Before you can enable Mailgun in GitLab, set up your own Mailgun endpoints to receive the webhooks. @@ -47,6 +47,6 @@ you're ready to enable the Mailgun integration: 1. On the left sidebar, go to **Settings > General** and expand the **Mailgun** section. 1. Select the **Enable Mailgun** check box. 1. Enter the Mailgun HTTP webhook signing key as described in - [the Mailgun documentation](https://documentation.mailgun.com/en/latest/user_manual.html#webhooks) and + [the Mailgun documentation](https://documentation.mailgun.com/en/latest/user_manual.html#webhooks-1) and shown in the [API security](https://app.mailgun.com/app/account/security/api_keys) section for your Mailgun account. 1. Select **Save changes**. diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 2582df1b0c4..a4ab0e07020 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -533,7 +533,7 @@ review: WARNING: Uploading artifacts as "archive" to coordinator... failed id=12345 responseStatus=500 Internal Server Error status=500 token=abcd1234 ``` -- The [workhorse log](logs.md#workhorse-logs) for an error message similar to: +- The [workhorse log](logs/index.md#workhorse-logs) for an error message similar to: ```json {"error":"MissingRegion: could not find region configuration","level":"error","msg":"error uploading S3 session","time":"2021-03-16T22:10:55-04:00"} diff --git a/doc/administration/libravatar.md b/doc/administration/libravatar.md index eac7c6f848b..88a46759924 100644 --- a/doc/administration/libravatar.md +++ b/doc/administration/libravatar.md @@ -11,63 +11,99 @@ GitLab by default supports the [Gravatar](https://gravatar.com) avatar service. Libravatar is another service that delivers your avatar (profile picture) to other websites. The Libravatar API is -[heavily based on gravatar](https://wiki.libravatar.org/api/), so you can +[heavily based on Gravatar](https://wiki.libravatar.org/api/), so you can switch to the Libravatar avatar service or even your own Libravatar server. -## Configuration +You cannot use any Libravatar service including Gravatar in [FIPS mode](../development/fips_compliance.md). -In the [`gitlab.yml` gravatar section](https://gitlab.com/gitlab-org/gitlab/-/blob/672bd3902d86b78d730cea809fce312ec49d39d7/config/gitlab.yml.example#L122), set +## Change the Libravatar service to your own service + +NOTE: +You can use only the MD5 hash in the URL for the Libravatar service. See [issue 370057](https://gitlab.com/gitlab-org/gitlab/-/issues/370057) for adding SHA-256 support. + +In the [`gitlab.yml` gravatar section](https://gitlab.com/gitlab-org/gitlab/-/blob/68dac188ec6b1b03d53365e7579422f44cbe7a1c/config/gitlab.yml.example#L469-476), set the configuration options as follows: -### For HTTP +**For Omnibus installations** -```yaml - gravatar: - enabled: true - # gravatar URLs: possible placeholders: %{hash} %{size} %{email} %{username} - plain_url: "http://cdn.libravatar.org/avatar/%{hash}?s=%{size}&d=identicon" -``` +1. Edit `/etc/gitlab/gitlab.rb`: -### For HTTPS + ```ruby + gitlab_rails['gravatar_enabled'] = true + #### For HTTPS + gitlab_rails['gravatar_ssl_url'] = "https://seccdn.libravatar.org/avatar/%{hash}?s=%{size}&d=identicon" + #### Use this line instead for HTTP + # gitlab_rails['gravatar_plain_url'] = "http://cdn.libravatar.org/avatar/%{hash}?s=%{size}&d=identicon" + ``` -```yaml - gravatar: - enabled: true - # gravatar URLs: possible placeholders: %{hash} %{size} %{email} %{username} - ssl_url: "https://seccdn.libravatar.org/avatar/%{hash}?s=%{size}&d=identicon" -``` +1. To apply the changes, run `sudo gitlab-ctl reconfigure`. -### Your own Libravatar server +**For installations from source** -If you are [running your own Libravatar service](https://wiki.libravatar.org/running_your_own/), -the URL is different in the configuration, but you must provide the same -placeholders so GitLab can parse the URL correctly. +1. Edit `config/gitlab.yml`: -For example, you host a service on `http://libravatar.example.com` and the -`plain_url` you must supply in `gitlab.yml` is + ```yaml + gravatar: + enabled: true + # default: https://www.gravatar.com/avatar/%{hash}?s=%{size}&d=identicon + plain_url: "http://cdn.libravatar.org/avatar/%{hash}?s=%{size}&d=identicon" + # default: https://secure.gravatar.com/avatar/%{hash}?s=%{size}&d=identicon + ssl_url: https://seccdn.libravatar.org/avatar/%{hash}?s=%{size}&d=identicon" + ``` -`http://libravatar.example.com/avatar/%{hash}?s=%{size}&d=identicon` +1. Save the file, and then [restart](restart_gitlab.md#installations-from-source) + GitLab for the changes to take effect. -### Omnibus GitLab example +## Set the Libravatar service to default (Gravatar) -In `/etc/gitlab/gitlab.rb`: +**For Omnibus installations** -#### For HTTP +1. Delete `gitlab_rails['gravatar_ssl_url']` or `gitlab_rails['gravatar_plain_url']` from `/etc/gitlab/gitlab.rb`. +1. To apply the changes, run `sudo gitlab-ctl reconfigure`. -```ruby -gitlab_rails['gravatar_enabled'] = true -gitlab_rails['gravatar_plain_url'] = "http://cdn.libravatar.org/avatar/%{hash}?s=%{size}&d=identicon" -``` +**For installations from source** -#### For HTTPS +1. Remove `gravatar:` section from `config/gitlab.yml`. +1. Save the file, then [restart](restart_gitlab.md#installations-from-source) + GitLab to apply the changes. -```ruby -gitlab_rails['gravatar_enabled'] = true -gitlab_rails['gravatar_ssl_url'] = "https://seccdn.libravatar.org/avatar/%{hash}?s=%{size}&d=identicon" -``` +## Disable Gravatar service + +To disable Gravatar, for example, to prohibit third-party services, complete the following steps: + +**For Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['gravatar_enabled'] = false + ``` + +1. To apply the changes, run `sudo gitlab-ctl reconfigure`. + +**For installations from source** + +1. Edit `config/gitlab.yml`: + + ```yaml + gravatar: + enabled: false + ``` + +1. Save the file, then [restart](restart_gitlab.md#installations-from-source) + GitLab to apply the changes. + +### Your own Libravatar server + +If you are [running your own Libravatar service](https://wiki.libravatar.org/running_your_own/), +the URL is different in the configuration, but you must provide the same +placeholders so GitLab can parse the URL correctly. + +For example, you host a service on `https://libravatar.example.com` and the +`ssl_url` you must supply in `gitlab.yml` is: -Then run `sudo gitlab-ctl reconfigure` for the changes to take effect. +`https://libravatar.example.com/avatar/%{hash}?s=%{size}&d=identicon` ## Default URL for missing images @@ -77,7 +113,7 @@ service. To use a set other than `identicon`, replace the `&d=identicon` portion of the URL with another supported set. For example, you can use the `retro` set, in -which case the URL would look like: `plain_url: "http://cdn.libravatar.org/avatar/%{hash}?s=%{size}&d=retro"` +which case the URL would look like: `ssl_url: "https://seccdn.libravatar.org/avatar/%{hash}?s=%{size}&d=retro"` ## Usage examples for Microsoft Office 365 diff --git a/doc/administration/logs.md b/doc/administration/logs.md index 671c264ed85..7264cf6db98 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -1,1178 +1,11 @@ --- -stage: Monitor -group: Respond -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'logs/index.md' +remove_date: '2022-10-23' --- -# Log system **(FREE SELF)** +This document was moved to [another location](logs/index.md). -GitLab has an advanced log system where everything is logged, so you -can analyze your instance using various system log files. In addition to -system log files, GitLab Enterprise Edition provides [Audit Events](audit_events.md). - -System log files are typically plain text in a standard log file format. -This guide talks about how to read and use these system log files. - -Read more about the log system and using the logs: - -- [Customize logging on Omnibus GitLab installations](https://docs.gitlab.com/omnibus/settings/logs.html) -including adjusting log retention, log forwarding, -switching logs from JSON to plain text logging, and more. -- [How to parse and analyze JSON logs](troubleshooting/log_parsing.md). - -## Log Levels - -Each log message has an assigned log level that indicates its importance and verbosity. -Each logger has an assigned minimum log level. -A logger emits a log message only if its log level is equal to or above the minimum log level. - -The following log levels are supported: - -| Level | Name | -|:------|:----------| -| 0 | `DEBUG` | -| 1 | `INFO` | -| 2 | `WARN` | -| 3 | `ERROR` | -| 4 | `FATAL` | -| 5 | `UNKNOWN` | - -GitLab loggers emit all log messages because they are set to `DEBUG` by default. - -### Override default log level - -You can override the minimum log level for GitLab loggers using the `GITLAB_LOG_LEVEL` environment variable. -Valid values are either a value of `0` to `5`, or the name of the log level. - -Example: - -```shell -GITLAB_LOG_LEVEL=info -``` - -For some services, other log levels are in place that are not affected by this setting. -Some of these services have their own environment variables to override the log level. For example: - -| Service | Log level | Environment variable | -|:---------------------|:----------|:---------------------| -| GitLab API | `INFO` | | -| GitLab Cleanup | `INFO` | `DEBUG` | -| GitLab Doctor | `INFO` | `VERBOSE` | -| GitLab Export | `INFO` | `EXPORT_DEBUG` | -| GitLab Geo | `INFO` | | -| GitLab Import | `INFO` | `IMPORT_DEBUG` | -| GitLab QA Runtime | `INFO` | `QA_LOG_LEVEL` | -| Google APIs | `INFO` | | -| Rack Timeout | `ERROR` | | -| Sidekiq (server) | `INFO` | | -| Snowplow Tracker | `FATAL` | | -| gRPC Client (Gitaly) | `WARN` | `GRPC_LOG_LEVEL` | - -## Log Rotation - -The logs for a given service may be managed and rotated by: - -- `logrotate` -- `svlogd` (`runit`'s service logging daemon) -- `logrotate` and `svlogd` -- Or not at all - -The following table includes information about what's responsible for managing and rotating logs for -the included services. Logs -[managed by `svlogd`](https://docs.gitlab.com/omnibus/settings/logs.html#runit-logs) -are written to a file called `current`. The `logrotate` service built into GitLab -[manages all logs](https://docs.gitlab.com/omnibus/settings/logs.html#logrotate) -except those captured by `runit`. - -| Log type | Managed by logrotate | Managed by svlogd/runit | -|:------------------------------------------------|:------------------------|:------------------------| -| [Alertmanager Logs](#alertmanager-logs) | **{dotted-circle}** No | **{check-circle}** Yes | -| [Crond Logs](#crond-logs) | **{dotted-circle}** No | **{check-circle}** Yes | -| [Gitaly](#gitaly-logs) | **{check-circle}** Yes | **{check-circle}** Yes | -| [GitLab Exporter for Omnibus](#gitlab-exporter) | **{dotted-circle}** No | **{check-circle}** Yes | -| [GitLab Pages Logs](#pages-logs) | **{check-circle}** Yes | **{check-circle}** Yes | -| GitLab Rails | **{check-circle}** Yes | **{dotted-circle}** No | -| [GitLab Shell Logs](#gitlab-shelllog) | **{check-circle}** Yes | **{dotted-circle}** No | -| [Grafana Logs](#grafana-logs) | **{dotted-circle}** No | **{check-circle}** Yes | -| [LogRotate Logs](#logrotate-logs) | **{dotted-circle}** No | **{check-circle}** Yes | -| [Mailroom](#mail_room_jsonlog-default) | **{check-circle}** Yes | **{check-circle}** Yes | -| [NGINX](#nginx-logs) | **{check-circle}** Yes | **{check-circle}** Yes | -| [PostgreSQL Logs](#postgresql-logs) | **{dotted-circle}** No | **{check-circle}** Yes | -| [Praefect Logs](#praefect-logs) | **{dotted-circle}** Yes | **{check-circle}** Yes | -| [Prometheus Logs](#prometheus-logs) | **{dotted-circle}** No | **{check-circle}** Yes | -| [Puma](#puma-logs) | **{check-circle}** Yes | **{check-circle}** Yes | -| [Redis Logs](#redis-logs) | **{dotted-circle}** No | **{check-circle}** Yes | -| [Registry Logs](#registry-logs) | **{dotted-circle}** No | **{check-circle}** Yes | -| [Workhorse Logs](#workhorse-logs) | **{check-circle}** Yes | **{check-circle}** Yes | - -## `production_json.log` - -Depending on your installation method, this file is located at: - -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/production_json.log` -- Installations from source: `/home/git/gitlab/log/production_json.log` - -It contains a structured log for Rails controller requests received from -GitLab, thanks to [Lograge](https://github.com/roidrage/lograge/). -Requests from the API are logged to a separate file in `api_json.log`. - -Each line contains JSON that can be ingested by services like Elasticsearch and Splunk. -Line breaks were added to examples for legibility: - -```json -{ - "method":"GET", - "path":"/gitlab/gitlab-foss/issues/1234", - "format":"html", - "controller":"Projects::IssuesController", - "action":"show", - "status":200, - "time":"2017-08-08T20:15:54.821Z", - "params":[{"key":"param_key","value":"param_value"}], - "remote_ip":"18.245.0.1", - "user_id":1, - "username":"admin", - "queue_duration_s":0.0, - "gitaly_calls":16, - "gitaly_duration_s":0.16, - "redis_calls":115, - "redis_duration_s":0.13, - "redis_read_bytes":1507378, - "redis_write_bytes":2920, - "correlation_id":"O1SdybnnIq7", - "cpu_s":17.50, - "db_duration_s":0.08, - "view_duration_s":2.39, - "duration_s":20.54, - "pid": 81836, - "worker_id":"puma_0" -} -``` - -This example was a GET request for a specific -issue. Each line also contains performance data, with times in -seconds: - -- `duration_s`: Total time to retrieve the request -- `queue_duration_s`: Total time the request was queued inside GitLab Workhorse -- `view_duration_s`: Total time inside the Rails views -- `db_duration_s`: Total time to retrieve data from PostgreSQL -- `cpu_s`: Total time spent on CPU -- `gitaly_duration_s`: Total time by Gitaly calls -- `gitaly_calls`: Total number of calls made to Gitaly -- `redis_calls`: Total number of calls made to Redis -- `redis_duration_s`: Total time to retrieve data from Redis -- `redis_read_bytes`: Total bytes read from Redis -- `redis_write_bytes`: Total bytes written to Redis -- `redis_<instance>_calls`: Total number of calls made to a Redis instance -- `redis_<instance>_duration_s`: Total time to retrieve data from a Redis instance -- `redis_<instance>_read_bytes`: Total bytes read from a Redis instance -- `redis_<instance>_write_bytes`: Total bytes written to a Redis instance -- `pid`: The worker's Linux process ID (changes when workers restart) -- `worker_id`: The worker's logical ID (does not change when workers restart) - -User clone and fetch activity using HTTP transport appears in the log as `action: git_upload_pack`. - -In addition, the log contains the originating IP address, -(`remote_ip`), the user's ID (`user_id`), and username (`username`). - -Some endpoints (such as `/search`) may make requests to Elasticsearch if using -[Advanced Search](../user/search/advanced_search.md). These -additionally log `elasticsearch_calls` and `elasticsearch_call_duration_s`, -which correspond to: - -- `elasticsearch_calls`: Total number of calls to Elasticsearch -- `elasticsearch_duration_s`: Total time taken by Elasticsearch calls -- `elasticsearch_timed_out_count`: Total number of calls to Elasticsearch that - timed out and therefore returned partial results - -ActionCable connection and subscription events are also logged to this file and they follow the -previous format. The `method`, `path`, and `format` fields are not applicable, and are always empty. -The ActionCable connection or channel class is used as the `controller`. - -```json -{ - "method":null, - "path":null, - "format":null, - "controller":"IssuesChannel", - "action":"subscribe", - "status":200, - "time":"2020-05-14T19:46:22.008Z", - "params":[{"key":"project_path","value":"gitlab/gitlab-foss"},{"key":"iid","value":"1"}], - "remote_ip":"127.0.0.1", - "user_id":1, - "username":"admin", - "ua":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:76.0) Gecko/20100101 Firefox/76.0", - "correlation_id":"jSOIEynHCUa", - "duration_s":0.32566 -} -``` - -NOTE: -Starting with GitLab 12.5, if an error occurs, an -`exception` field is included with `class`, `message`, and -`backtrace`. Previous versions included an `error` field instead of -`exception.class` and `exception.message`. For example: - -```json -{ - "method": "GET", - "path": "/admin", - "format": "html", - "controller": "Admin::DashboardController", - "action": "index", - "status": 500, - "time": "2019-11-14T13:12:46.156Z", - "params": [], - "remote_ip": "127.0.0.1", - "user_id": 1, - "username": "root", - "ua": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:70.0) Gecko/20100101 Firefox/70.0", - "queue_duration": 274.35, - "correlation_id": "KjDVUhNvvV3", - "queue_duration_s":0.0, - "gitaly_calls":16, - "gitaly_duration_s":0.16, - "redis_calls":115, - "redis_duration_s":0.13, - "correlation_id":"O1SdybnnIq7", - "cpu_s":17.50, - "db_duration_s":0.08, - "view_duration_s":2.39, - "duration_s":20.54, - "pid": 81836, - "worker_id": "puma_0", - "exception.class": "NameError", - "exception.message": "undefined local variable or method `adsf' for #<Admin::DashboardController:0x00007ff3c9648588>", - "exception.backtrace": [ - "app/controllers/admin/dashboard_controller.rb:11:in `index'", - "ee/app/controllers/ee/admin/dashboard_controller.rb:14:in `index'", - "ee/lib/gitlab/ip_address_state.rb:10:in `with'", - "ee/app/controllers/ee/application_controller.rb:43:in `set_current_ip_address'", - "lib/gitlab/session.rb:11:in `with_session'", - "app/controllers/application_controller.rb:450:in `set_session_storage'", - "app/controllers/application_controller.rb:444:in `set_locale'", - "ee/lib/gitlab/jira/middleware.rb:19:in `call'" - ] -} -``` - -## `production.log` - -Depending on your installation method, this file is located at: - -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/production.log` -- Installations from source: `/home/git/gitlab/log/production.log` - -It contains information about all performed requests. You can see the -URL and type of request, IP address, and what parts of code were -involved to service this particular request. Also, you can see all SQL -requests performed, and how much time each took. This task is -more useful for GitLab contributors and developers. Use part of this log -file when you're reporting bugs. For example: - -```plaintext -Started GET "/gitlabhq/yaml_db/tree/master" for 168.111.56.1 at 2015-02-12 19:34:53 +0200 -Processing by Projects::TreeController#show as HTML - Parameters: {"project_id"=>"gitlabhq/yaml_db", "id"=>"master"} - - ... [CUT OUT] - - Namespaces"."created_at" DESC, "namespaces"."id" DESC LIMIT 1 [["id", 26]] - CACHE (0.0ms) SELECT "members".* FROM "members" WHERE "members"."source_type" = 'Project' AND "members"."type" IN ('ProjectMember') AND "members"."source_id" = $1 AND "members"."source_type" = $2 AND "members"."user_id" = 1 ORDER BY "members"."created_at" DESC, "members"."id" DESC LIMIT 1 [["source_id", 18], ["source_type", "Project"]] - CACHE (0.0ms) SELECT "members".* FROM "members" WHERE "members"."source_type" = 'Project' AND "members". - (1.4ms) SELECT COUNT(*) FROM "merge_requests" WHERE "merge_requests"."target_project_id" = $1 AND ("merge_requests"."state" IN ('opened','reopened')) [["target_project_id", 18]] - Rendered layouts/nav/_project.html.haml (28.0ms) - Rendered layouts/_collapse_button.html.haml (0.2ms) - Rendered layouts/_flash.html.haml (0.1ms) - Rendered layouts/_page.html.haml (32.9ms) -Completed 200 OK in 166ms (Views: 117.4ms | ActiveRecord: 27.2ms) -``` - -In this example, the server processed an HTTP request with URL -`/gitlabhq/yaml_db/tree/master` from IP `168.111.56.1` at `2015-02-12 19:34:53 +0200`. -The request was processed by `Projects::TreeController`. - -## `api_json.log` - -Depending on your installation method, this file is located at: - -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/api_json.log` -- Installations from source: `/home/git/gitlab/log/api_json.log` - -It helps you see requests made directly to the API. For example: - -```json -{ - "time":"2018-10-29T12:49:42.123Z", - "severity":"INFO", - "duration":709.08, - "db":14.59, - "view":694.49, - "status":200, - "method":"GET", - "path":"/api/v4/projects", - "params":[{"key":"action","value":"git-upload-pack"},{"key":"changes","value":"_any"},{"key":"key_id","value":"secret"},{"key":"secret_token","value":"[FILTERED]"}], - "host":"localhost", - "remote_ip":"::1", - "ua":"Ruby", - "route":"/api/:version/projects", - "user_id":1, - "username":"root", - "queue_duration":100.31, - "gitaly_calls":30, - "gitaly_duration":5.36, - "pid": 81836, - "worker_id": "puma_0", - ... -} -``` - -This entry shows an internal endpoint accessed to check whether an -associated SSH key can download the project in question by using a `git fetch` or -`git clone`. In this example, we see: - -- `duration`: Total time in milliseconds to retrieve the request -- `queue_duration`: Total time in milliseconds the request was queued inside GitLab Workhorse -- `method`: The HTTP method used to make the request -- `path`: The relative path of the query -- `params`: Key-value pairs passed in a query string or HTTP body (sensitive parameters, such as passwords and tokens, are filtered out) -- `ua`: The User-Agent of the requester - -## `application.log` - -Depending on your installation method, this file is located at: - -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/application.log` -- Installations from source: `/home/git/gitlab/log/application.log` - -It helps you discover events happening in your instance such as user creation -and project deletion. For example: - -```plaintext -October 06, 2014 11:56: User "Administrator" (admin@example.com) was created -October 06, 2014 11:56: Documentcloud created a new project "Documentcloud / Underscore" -October 06, 2014 11:56: Gitlab Org created a new project "Gitlab Org / Gitlab Ce" -October 07, 2014 11:25: User "Claudie Hodkiewicz" (nasir_stehr@olson.co.uk) was removed -October 07, 2014 11:25: Project "project133" was removed -``` - -## `application_json.log` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22812) in GitLab 12.7. - -Depending on your installation method, this file is located at: - -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/application_json.log` -- Installations from source: `/home/git/gitlab/log/application_json.log` - -It contains the JSON version of the logs in `application.log`, like this example: - -```json -{ - "severity":"INFO", - "time":"2020-01-14T13:35:15.466Z", - "correlation_id":"3823a1550b64417f9c9ed8ee0f48087e", - "message":"User \"Administrator\" (admin@example.com) was created" -} -{ - "severity":"INFO", - "time":"2020-01-14T13:35:15.466Z", - "correlation_id":"78e3df10c9a18745243d524540bd5be4", - "message":"Project \"project133\" was removed" -} -``` - -## `integrations_json.log` - -Depending on your installation method, this file is located at: - -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/integrations_json.log` -- Installations from source: `/home/git/gitlab/log/integrations_json.log` - -It contains information about [integration](../user/project/integrations/index.md) -activities, such as Jira, Asana, and irker services. It uses JSON format, -like this example: - -```json -{ - "severity":"ERROR", - "time":"2018-09-06T14:56:20.439Z", - "service_class":"Integrations::Jira", - "project_id":8, - "project_path":"h5bp/html5-boilerplate", - "message":"Error sending message", - "client_url":"http://jira.gitlap.com:8080", - "error":"execution expired" -} -{ - "severity":"INFO", - "time":"2018-09-06T17:15:16.365Z", - "service_class":"Integrations::Jira", - "project_id":3, - "project_path":"namespace2/project2", - "message":"Successfully posted", - "client_url":"http://jira.example.com" -} -``` - -## `kubernetes.log` - -Depending on your installation method, this file is located at: - -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/kubernetes.log` -- Installations from source: `/home/git/gitlab/log/kubernetes.log` - -It logs information related to the Kubernetes Integration, including errors -during installing cluster applications on your managed Kubernetes -clusters. - -Each line contains JSON that can be ingested by services like Elasticsearch and Splunk. -Line breaks have been added to the following example for clarity: - -```json -{ - "severity":"ERROR", - "time":"2018-11-23T15:14:54.652Z", - "exception":"Kubeclient::HttpError", - "error_code":401, - "service":"Clusters::Applications::CheckInstallationProgressService", - "app_id":14, - "project_ids":[1], - "group_ids":[], - "message":"Unauthorized" -} -{ - "severity":"ERROR", - "time":"2018-11-23T15:42:11.647Z", - "exception":"Kubeclient::HttpError", - "error_code":null, - "service":"Clusters::Applications::InstallService", - "app_id":2, - "project_ids":[19], - "group_ids":[], - "message":"SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)" -} -``` - -## `git_json.log` - -Depending on your installation method, this file is located at: - -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/git_json.log` -- Installations from source: `/home/git/gitlab/log/git_json.log` - -After GitLab version 12.2, this file was renamed from `githost.log` to -`git_json.log` and stored in JSON format. - -GitLab has to interact with Git repositories, but in some rare cases -something can go wrong. If this happens, you need to know exactly what -happened. This log file contains all failed requests from GitLab to Git -repositories. In the majority of cases this file is useful for developers -only. For example: - -```json -{ - "severity":"ERROR", - "time":"2019-07-19T22:16:12.528Z", - "correlation_id":"FeGxww5Hj64", - "message":"Command failed [1]: /usr/bin/git --git-dir=/Users/vsizov/gitlab-development-kit/gitlab/tmp/tests/gitlab-satellites/group184/gitlabhq/.git --work-tree=/Users/vsizov/gitlab-development-kit/gitlab/tmp/tests/gitlab-satellites/group184/gitlabhq merge --no-ff -mMerge branch 'feature_conflict' into 'feature' source/feature_conflict\n\nerror: failed to push some refs to '/Users/vsizov/gitlab-development-kit/repositories/gitlabhq/gitlab_git.git'" -} -``` - -## `audit_json.log` **(FREE)** - -NOTE: -GitLab Free tracks a small number of different audit events. -GitLab Premium tracks many more. - -Depending on your installation method, this file is located at: - -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/audit_json.log` -- Installations from source: `/home/git/gitlab/log/audit_json.log` - -Changes to group or project settings and memberships (`target_details`) -are logged to this file. For example: - -```json -{ - "severity":"INFO", - "time":"2018-10-17T17:38:22.523Z", - "author_id":3, - "entity_id":2, - "entity_type":"Project", - "change":"visibility", - "from":"Private", - "to":"Public", - "author_name":"John Doe4", - "target_id":2, - "target_type":"Project", - "target_details":"namespace2/project2" -} -``` - -## Sidekiq Logs - -NOTE: -In Omnibus GitLab `12.10` or earlier, the Sidekiq log is at `/var/log/gitlab/gitlab-rails/sidekiq.log`. - -For Omnibus GitLab installations, some Sidekiq logs are in `/var/log/gitlab/sidekiq/current` -and as follows. - -### `sidekiq.log` - -Depending on your installation method, this file is located at: - -- Omnibus GitLab: `/var/log/gitlab/sidekiq/current` -- Installations from source: `/home/git/gitlab/log/sidekiq.log` - -GitLab uses background jobs for processing tasks which can take a long -time. All information about processing these jobs are written down to -this file. For example: - -```plaintext -2014-06-10T07:55:20Z 2037 TID-tm504 ERROR: /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/1.9.1/gems/redis-3.0.7/lib/redis/client.rb:228:in `read' -2014-06-10T18:18:26Z 14299 TID-55uqo INFO: Booting Sidekiq 3.0.0 with redis options {:url=>"redis://localhost:6379/0", :namespace=>"sidekiq"} -``` - -Instead of the previous format, you can opt to generate JSON logs for -Sidekiq. For example: - -```json -{ - "severity":"INFO", - "time":"2018-04-03T22:57:22.071Z", - "queue":"cronjob:update_all_mirrors", - "args":[], - "class":"UpdateAllMirrorsWorker", - "retry":false, - "queue_namespace":"cronjob", - "jid":"06aeaa3b0aadacf9981f368e", - "created_at":"2018-04-03T22:57:21.930Z", - "enqueued_at":"2018-04-03T22:57:21.931Z", - "pid":10077, - "worker_id":"sidekiq_0", - "message":"UpdateAllMirrorsWorker JID-06aeaa3b0aadacf9981f368e: done: 0.139 sec", - "job_status":"done", - "duration":0.139, - "completed_at":"2018-04-03T22:57:22.071Z", - "db_duration":0.05, - "db_duration_s":0.0005, - "gitaly_duration":0, - "gitaly_calls":0 -} -``` - -For Omnibus GitLab installations, add the configuration option: - -```ruby -sidekiq['log_format'] = 'json' -``` - -For installations from source, edit the `gitlab.yml` and set the Sidekiq -`log_format` configuration option: - -```yaml - ## Sidekiq - sidekiq: - log_format: json -``` - -### `sidekiq_client.log` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26586) in GitLab 12.9. - -Depending on your installation method, this file is located at: - -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/sidekiq_client.log` -- Installations from source: `/home/git/gitlab/log/sidekiq_client.log` - -This file contains logging information about jobs before Sidekiq starts -processing them, such as before being enqueued. - -This log file follows the same structure as -[`sidekiq.log`](#sidekiqlog), so it is structured as JSON if -you've configured this for Sidekiq as mentioned above. - -## `gitlab-shell.log` - -GitLab Shell is used by GitLab for executing Git commands and provide SSH -access to Git repositories. - -### For GitLab versions 12.10 and up - -Information containing `git-{upload-pack,receive-pack}` requests is at -`/var/log/gitlab/gitlab-shell/gitlab-shell.log`. Information about hooks to -GitLab Shell from Gitaly is at `/var/log/gitlab/gitaly/current`. - -Example log entries for `/var/log/gitlab/gitlab-shell/gitlab-shell.log`: - -```json -{ - "duration_ms": 74.104, - "level": "info", - "method": "POST", - "msg": "Finished HTTP request", - "time": "2020-04-17T20:28:46Z", - "url": "http://127.0.0.1:8080/api/v4/internal/allowed" -} -{ - "command": "git-upload-pack", - "git_protocol": "", - "gl_project_path": "root/example", - "gl_repository": "project-1", - "level": "info", - "msg": "executing git command", - "time": "2020-04-17T20:28:46Z", - "user_id": "user-1", - "username": "root" -} -``` - -Example log entries for `/var/log/gitlab/gitaly/current`: - -```json -{ - "method": "POST", - "url": "http://127.0.0.1:8080/api/v4/internal/allowed", - "duration": 0.058012959, - "gitaly_embedded": true, - "pid": 16636, - "level": "info", - "msg": "finished HTTP request", - "time": "2020-04-17T20:29:08+00:00" -} -{ - "method": "POST", - "url": "http://127.0.0.1:8080/api/v4/internal/pre_receive", - "duration": 0.031022552, - "gitaly_embedded": true, - "pid": 16636, - "level": "info", - "msg": "finished HTTP request", - "time": "2020-04-17T20:29:08+00:00" -} -``` - -### For GitLab versions 12.5 through 12.9 - -For GitLab 12.5 to 12.9, depending on your installation method, this -file is located at: - -- Omnibus GitLab: `/var/log/gitlab/gitaly/gitlab-shell.log` -- Installation from source: `/home/git/gitaly/gitlab-shell.log` - -Example log entries: - -```json -{ - "method": "POST", - "url": "http://127.0.0.1:8080/api/v4/internal/post_receive", - "duration": 0.031809164, - "gitaly_embedded": true, - "pid": 27056, - "level": "info", - "msg": "finished HTTP request", - "time": "2020-04-17T16:24:38+00:00" -} -``` - -### For GitLab 12.5 and earlier - -For GitLab 12.5 and earlier, the file is at `/var/log/gitlab/gitlab-shell/gitlab-shell.log`. - -Example log entries: - -```plaintext -I, [2015-02-13T06:17:00.671315 #9291] INFO -- : Adding project root/example.git at </var/opt/gitlab/git-data/repositories/root/dcdcdcdcd.git>. -I, [2015-02-13T06:17:00.679433 #9291] INFO -- : Moving existing hooks directory and symlinking global hooks directory for /var/opt/gitlab/git-data/repositories/root/example.git. -``` - -User clone/fetch activity using SSH transport appears in this log as -`executing git command <gitaly-upload-pack...`. - -## Gitaly Logs - -This file is in `/var/log/gitlab/gitaly/current` and is produced by [runit](http://smarden.org/runit/). -`runit` is packaged with Omnibus GitLab and a brief explanation of its purpose -is available [in the Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/architecture/#runit). -[Log files are rotated](http://smarden.org/runit/svlogd.8.html), renamed in -Unix timestamp format, and `gzip`-compressed (like `@1584057562.s`). - -### `grpc.log` - -This file is at `/var/log/gitlab/gitlab-rails/grpc.log` for Omnibus GitLab -packages. Native [gRPC](https://grpc.io/) logging used by Gitaly. - -### `gitaly_ruby_json.log` - -> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/2678) in GitLab 13.6. - -This file is at `/var/log/gitlab/gitaly/gitaly_ruby_json.log` and is -produced by [`gitaly-ruby`](gitaly/reference.md#gitaly-ruby). It contains an -access log of gRPC calls made by Gitaly to `gitaly-ruby`. - -### `gitaly_hooks.log` - -This file is at `/var/log/gitlab/gitaly/gitaly_hooks.log` and is -produced by `gitaly-hooks` command. It also contains records about -failures received during processing of the responses from GitLab API. - -## Puma Logs - -### `puma_stdout.log` - -Depending on your installation method, this file is located at: - -- Omnibus GitLab: `/var/log/gitlab/puma/puma_stdout.log` -- Installations from source: `/home/git/gitlab/log/puma_stdout.log` - -### `puma_stderr.log` - -Depending on your installation method, this file is located at: - -- Omnibus GitLab: `/var/log/gitlab/puma/puma_stderr.log` -- Installations from source: `/home/git/gitlab/log/puma_stderr.log` - -## `repocheck.log` - -Depending on your installation method, this file is located at: - -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/repocheck.log` -- Installations from source: `/home/git/gitlab/log/repocheck.log` - -It logs information whenever a [repository check is run](repository_checks.md) -on a project. - -## `importer.log` - -Depending on your installation method, this file is located at: - -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/importer.log` -- Installations from source: `/home/git/gitlab/log/importer.log` - -It logs the progress of the import process. - -## `exporter.log` - -> Introduced in GitLab 13.1. - -Depending on your installation method, this file is located at: - -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/exporter.log` -- Installations from source: `/home/git/gitlab/log/exporter.log` - -It logs the progress of the export process. - -## `features_json.log` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/59587) in GitLab 13.7. - -Depending on your installation method, this file is located at: - -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/features_json.log` -- Installations from source: `/home/git/gitlab/log/features_json.log` - -The modification events from [Feature flags in development of GitLab](../development/feature_flags/index.md) -are recorded in this file. For example: - -```json -{"severity":"INFO","time":"2020-11-24T02:30:59.860Z","correlation_id":null,"key":"cd_auto_rollback","action":"enable","extra.thing":"true"} -{"severity":"INFO","time":"2020-11-24T02:31:29.108Z","correlation_id":null,"key":"cd_auto_rollback","action":"enable","extra.thing":"true"} -{"severity":"INFO","time":"2020-11-24T02:31:29.129Z","correlation_id":null,"key":"cd_auto_rollback","action":"disable","extra.thing":"false"} -{"severity":"INFO","time":"2020-11-24T02:31:29.177Z","correlation_id":null,"key":"cd_auto_rollback","action":"enable","extra.thing":"Project:1"} -{"severity":"INFO","time":"2020-11-24T02:31:29.183Z","correlation_id":null,"key":"cd_auto_rollback","action":"disable","extra.thing":"Project:1"} -{"severity":"INFO","time":"2020-11-24T02:31:29.188Z","correlation_id":null,"key":"cd_auto_rollback","action":"enable_percentage_of_time","extra.percentage":"50"} -{"severity":"INFO","time":"2020-11-24T02:31:29.193Z","correlation_id":null,"key":"cd_auto_rollback","action":"disable_percentage_of_time"} -{"severity":"INFO","time":"2020-11-24T02:31:29.198Z","correlation_id":null,"key":"cd_auto_rollback","action":"enable_percentage_of_actors","extra.percentage":"50"} -{"severity":"INFO","time":"2020-11-24T02:31:29.203Z","correlation_id":null,"key":"cd_auto_rollback","action":"disable_percentage_of_actors"} -{"severity":"INFO","time":"2020-11-24T02:31:29.329Z","correlation_id":null,"key":"cd_auto_rollback","action":"remove"} -``` - -## `auth.log` - -> Introduced in GitLab 12.0. - -Depending on your installation method, this file is located at: - -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/auth.log` -- Installations from source: `/home/git/gitlab/log/auth.log` - -This log records: - -- Requests over the [Rate Limit](../user/admin_area/settings/rate_limits_on_raw_endpoints.md) on raw endpoints. -- [Protected paths](../user/admin_area/settings/protected_paths.md) abusive requests. -- In GitLab versions [12.3](https://gitlab.com/gitlab-org/gitlab/-/issues/29239) and later, - user ID and username, if available. - -## `graphql_json.log` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/59587) in GitLab 12.0. - -Depending on your installation method, this file is located at: - -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/graphql_json.log` -- Installations from source: `/home/git/gitlab/log/graphql_json.log` - -GraphQL queries are recorded in the file. For example: - -```json -{"query_string":"query IntrospectionQuery{__schema {queryType { name },mutationType { name }}}...(etc)","variables":{"a":1,"b":2},"complexity":181,"depth":1,"duration_s":7} -``` - -## `migrations.log` - -> Introduced in GitLab 12.3. - -Depending on your installation method, this file is located at: - -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/migrations.log` -- Installations from source: `/home/git/gitlab/log/migrations.log` - -## `mail_room_json.log` (default) - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/19186) in GitLab 12.6. - -Depending on your installation method, this file is located at: - -- Omnibus GitLab: `/var/log/gitlab/mailroom/current` -- Installations from source: `/home/git/gitlab/log/mail_room_json.log` - -This structured log file records internal activity in the `mail_room` gem. -Its name and path are configurable, so the name and path may not match the above. - -## Reconfigure logs - -Reconfigure log files are in `/var/log/gitlab/reconfigure` for Omnibus GitLab -packages. Installations from source don't have reconfigure logs. A reconfigure log -is populated whenever `gitlab-ctl reconfigure` is run manually or as part of an upgrade. - -Reconfigure logs files are named according to the UNIX timestamp of when the reconfigure -was initiated, such as `1509705644.log` - -## `sidekiq_exporter.log` and `web_exporter.log` - -If Prometheus metrics and the Sidekiq Exporter are both enabled, Sidekiq -starts a Web server and listen to the defined port (default: -`8082`). By default, Sidekiq Exporter access logs are disabled but can -be enabled based on your installation method: - -- Omnibus GitLab: Use the `sidekiq['exporter_log_enabled'] = true` - option in `/etc/gitlab/gitlab.rb` -- Installations from source: Use the `sidekiq_exporter.log_enabled` option - in `gitlab.yml` - -When enabled, depending on your installation method, this file is located at: - -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/sidekiq_exporter.log` -- Installations from source: `/home/git/gitlab/log/sidekiq_exporter.log` - -If Prometheus metrics and the Web Exporter are both enabled, Puma -starts a Web server and listen to the defined port (default: `8083`), and access logs -are generated in a location based on your installation method: - -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/web_exporter.log` -- Installations from source: `/home/git/gitlab/log/web_exporter.log` - -## `database_load_balancing.log` **(PREMIUM SELF)** - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/15442) in GitLab 12.3. - -Contains details of GitLab [Database Load Balancing](postgresql/database_load_balancing.md). -Depending on your installation method, this file is located at: - -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/database_load_balancing.log` -- Installations from source: `/home/git/gitlab/log/database_load_balancing.log` - -## `elasticsearch.log` **(PREMIUM SELF)** - -> Introduced in GitLab 12.6. - -This file logs information related to the Elasticsearch Integration, including -errors during indexing or searching Elasticsearch. Depending on your installation -method, this file is located at: - -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/elasticsearch.log` -- Installations from source: `/home/git/gitlab/log/elasticsearch.log` - -Each line contains JSON that can be ingested by services like Elasticsearch and Splunk. -Line breaks have been added to the following example line for clarity: - -```json -{ - "severity":"DEBUG", - "time":"2019-10-17T06:23:13.227Z", - "correlation_id":null, - "message":"redacted_search_result", - "class_name":"Milestone", - "id":2, - "ability":"read_milestone", - "current_user_id":2, - "query":"project" -} -``` - -## `exceptions_json.log` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17819) in GitLab 12.6. - -This file logs the information about exceptions being tracked by -`Gitlab::ErrorTracking`, which provides a standard and consistent way of -[processing rescued exceptions](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/logging.md#exception-handling). -Depending on your installation method, this file is located at: - -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/exceptions_json.log` -- Installations from source: `/home/git/gitlab/log/exceptions_json.log` - -Each line contains JSON that can be ingested by Elasticsearch. For example: - -```json -{ - "severity": "ERROR", - "time": "2019-12-17T11:49:29.485Z", - "correlation_id": "AbDVUrrTvM1", - "extra.project_id": 55, - "extra.relation_key": "milestones", - "extra.relation_index": 1, - "exception.class": "NoMethodError", - "exception.message": "undefined method `strong_memoize' for #<Gitlab::ImportExport::RelationFactory:0x00007fb5d917c4b0>", - "exception.backtrace": [ - "lib/gitlab/import_export/relation_factory.rb:329:in `unique_relation?'", - "lib/gitlab/import_export/relation_factory.rb:345:in `find_or_create_object!'" - ] -} -``` - -## `service_measurement.log` - -> Introduced in GitLab 13.0. - -Depending on your installation method, this file is located at: - -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/service_measurement.log` -- Installations from source: `/home/git/gitlab/log/service_measurement.log` - -It contains only a single structured log with measurements for each service execution. -It contains measurements such as the number of SQL calls, `execution_time`, `gc_stats`, and `memory usage`. - -For example: - -```json -{ "severity":"INFO", "time":"2020-04-22T16:04:50.691Z","correlation_id":"04f1366e-57a1-45b8-88c1-b00b23dc3616","class":"Projects::ImportExport::ExportService","current_user":"John Doe","project_full_path":"group1/test-export","file_path":"/path/to/archive","gc_stats":{"count":{"before":127,"after":127,"diff":0},"heap_allocated_pages":{"before":10369,"after":10369,"diff":0},"heap_sorted_length":{"before":10369,"after":10369,"diff":0},"heap_allocatable_pages":{"before":0,"after":0,"diff":0},"heap_available_slots":{"before":4226409,"after":4226409,"diff":0},"heap_live_slots":{"before":2542709,"after":2641420,"diff":98711},"heap_free_slots":{"before":1683700,"after":1584989,"diff":-98711},"heap_final_slots":{"before":0,"after":0,"diff":0},"heap_marked_slots":{"before":2542704,"after":2542704,"diff":0},"heap_eden_pages":{"before":10369,"after":10369,"diff":0},"heap_tomb_pages":{"before":0,"after":0,"diff":0},"total_allocated_pages":{"before":10369,"after":10369,"diff":0},"total_freed_pages":{"before":0,"after":0,"diff":0},"total_allocated_objects":{"before":24896308,"after":24995019,"diff":98711},"total_freed_objects":{"before":22353599,"after":22353599,"diff":0},"malloc_increase_bytes":{"before":140032,"after":6650240,"diff":6510208},"malloc_increase_bytes_limit":{"before":25804104,"after":25804104,"diff":0},"minor_gc_count":{"before":94,"after":94,"diff":0},"major_gc_count":{"before":33,"after":33,"diff":0},"remembered_wb_unprotected_objects":{"before":34284,"after":34284,"diff":0},"remembered_wb_unprotected_objects_limit":{"before":68568,"after":68568,"diff":0},"old_objects":{"before":2404725,"after":2404725,"diff":0},"old_objects_limit":{"before":4809450,"after":4809450,"diff":0},"oldmalloc_increase_bytes":{"before":140032,"after":6650240,"diff":6510208},"oldmalloc_increase_bytes_limit":{"before":68537556,"after":68537556,"diff":0}},"time_to_finish":0.12298400001600385,"number_of_sql_calls":70,"memory_usage":"0.0 MiB","label":"process_48616"} -``` - -## `geo.log` **(PREMIUM SELF)** - -Geo stores structured log messages in a `geo.log` file. For Omnibus GitLab -installations, this file is at `/var/log/gitlab/gitlab-rails/geo.log`. - -This file contains information about when Geo attempts to sync repositories -and files. Each line in the file contains a separate JSON entry that can be -ingested into (for example, Elasticsearch or Splunk). - -For example: - -```json -{"severity":"INFO","time":"2017-08-06T05:40:16.104Z","message":"Repository update","project_id":1,"source":"repository","resync_repository":true,"resync_wiki":true,"class":"Gitlab::Geo::LogCursor::Daemon","cursor_delay_s":0.038} -``` - -This message shows that Geo detected that a repository update was needed for project `1`. - -## `update_mirror_service_json.log` - -Depending on your installation method, this file is located at: - -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/update_mirror_service_json.log` -- Installations from source: `/home/git/gitlab/log/update_mirror_service_json.log` - -This file contains information about LFS errors that occurred during project mirroring. -While we work to move other project mirroring errors into this log, the [general log](#productionlog) -can be used. - -```json -{ - "severity":"ERROR", - "time":"2020-07-28T23:29:29.473Z", - "correlation_id":"5HgIkCJsO53", - "user_id":"x", - "project_id":"x", - "import_url":"https://mirror-source/group/project.git", - "error_message":"The LFS objects download list couldn't be imported. Error: Unauthorized" -} -``` - -## Registry Logs - -For Omnibus GitLab installations, Container Registry logs are in `/var/log/gitlab/registry/current`. - -## NGINX Logs - -For Omnibus GitLab installations, NGINX logs are in: - -- `/var/log/gitlab/nginx/gitlab_access.log`: A log of requests made to GitLab -- `/var/log/gitlab/nginx/gitlab_error.log`: A log of NGINX errors for GitLab -- `/var/log/gitlab/nginx/gitlab_pages_access.log`: A log of requests made to Pages static sites -- `/var/log/gitlab/nginx/gitlab_pages_error.log`: A log of NGINX errors for Pages static sites -- `/var/log/gitlab/nginx/gitlab_registry_access.log`: A log of requests made to the Container Registry -- `/var/log/gitlab/nginx/gitlab_registry_error.log`: A log of NGINX errors for the Container Registry -- `/var/log/gitlab/nginx/gitlab_mattermost_access.log`: A log of requests made to Mattermost -- `/var/log/gitlab/nginx/gitlab_mattermost_error.log`: A log of NGINX errors for Mattermost - -Below is the default GitLab NGINX access log format: - -```plaintext -$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" -``` - -## Pages Logs - -For Omnibus GitLab installations, Pages logs are in `/var/log/gitlab/gitlab-pages/current`. - -For example: - -```json -{ - "level": "info", - "msg": "GitLab Pages Daemon", - "revision": "52b2899", - "time": "2020-04-22T17:53:12Z", - "version": "1.17.0" -} -{ - "level": "info", - "msg": "URL: https://gitlab.com/gitlab-org/gitlab-pages", - "time": "2020-04-22T17:53:12Z" -} -{ - "gid": 998, - "in-place": false, - "level": "info", - "msg": "running the daemon as unprivileged user", - "time": "2020-04-22T17:53:12Z", - "uid": 998 -} -``` - -## Mattermost Logs - -For Omnibus GitLab installations, Mattermost logs are in these locations: - -- `/var/log/gitlab/mattermost/mattermost.log` -- `/var/log/gitlab/mattermost/current` - -## Workhorse Logs - -For Omnibus GitLab installations, Workhorse logs are in `/var/log/gitlab/gitlab-workhorse/current`. - -## PostgreSQL Logs - -For Omnibus GitLab installations, PostgreSQL logs are in `/var/log/gitlab/postgresql/current`. - -## Prometheus Logs - -For Omnibus GitLab installations, Prometheus logs are in `/var/log/gitlab/prometheus/current`. - -## Redis Logs - -For Omnibus GitLab installations, Redis logs are in `/var/log/gitlab/redis/current`. - -## Alertmanager Logs - -For Omnibus GitLab installations, Alertmanager logs are in `/var/log/gitlab/alertmanager/current`. - -<!-- vale gitlab.Spelling = NO --> - -## Crond Logs - -For Omnibus GitLab installations, crond logs are in `/var/log/gitlab/crond/`. - -<!-- vale gitlab.Spelling = YES --> - -## Grafana Logs - -For Omnibus GitLab installations, Grafana logs are in `/var/log/gitlab/grafana/current`. - -## LogRotate Logs - -For Omnibus GitLab installations, `logrotate` logs are in `/var/log/gitlab/logrotate/current`. - -## GitLab Monitor Logs - -For Omnibus GitLab installations, GitLab Monitor logs are in `/var/log/gitlab/gitlab-monitor/`. - -## GitLab Exporter - -For Omnibus GitLab installations, GitLab Exporter logs are in `/var/log/gitlab/gitlab-exporter/current`. - -## GitLab agent server - -For Omnibus GitLab installations, GitLab agent server logs are -in `/var/log/gitlab/gitlab-kas/current`. - -## Praefect Logs - -For Omnibus GitLab installations, Praefect logs are in `/var/log/gitlab/praefect/`. - -GitLab also tracks [Prometheus metrics for Praefect](gitaly/monitoring.md#monitor-gitaly-cluster). - -## Backup log - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63832) in GitLab 14.1. - -For Omnibus installations, the backup log is located at `/var/log/gitlab/gitlab-rails/backup_json.log`. - -This log is populated when a [GitLab backup is created](../raketasks/backup_restore.md). You can use this log to understand how the backup process performed. - -## Performance bar stats - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/48149) in GitLab 13.7. - -Depending on your installation method, this file is located at: - -- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/performance_bar_json.log` -- Installations from source: `/home/git/gitlab/log/performance_bar_json.log` - -Performance bar statistics (currently only duration of SQL queries) are recorded -in that file. For example: - -```json -{"severity":"INFO","time":"2020-12-04T09:29:44.592Z","correlation_id":"33680b1490ccd35981b03639c406a697","filename":"app/models/ci/pipeline.rb","method_path":"app/models/ci/pipeline.rb:each_with_object","request_id":"rYHomD0VJS4","duration_ms":26.889,"count":2,"query_type": "active-record"} -``` - -These statistics are logged on .com only, disabled on self-deployments. - -## Gathering logs - -When [troubleshooting](troubleshooting/index.md) issues that aren't localized to one of the -previously listed components, it's helpful to simultaneously gather multiple logs and statistics -from a GitLab instance. - -NOTE: -GitLab Support often asks for one of these, and maintains the required tools. - -### Briefly tail the main logs - -If the bug or error is readily reproducible, save the main GitLab logs -[to a file](troubleshooting/linux_cheat_sheet.md#files-and-directories) while reproducing the -problem a few times: - -```shell -sudo gitlab-ctl tail | tee /tmp/<case-ID-and-keywords>.log -``` - -Conclude the log gathering with <kbd>Control</kbd> + <kbd>C</kbd>. - -### GitLabSOS - -If performance degradations or cascading errors occur that can't readily be attributed to one -of the previously listed GitLab components, [GitLabSOS](https://gitlab.com/gitlab-com/support/toolbox/gitlabsos/) -can provide a broader perspective of the GitLab instance. For more details and instructions -to run it, read [the GitLabSOS documentation](https://gitlab.com/gitlab-com/support/toolbox/gitlabsos/#gitlabsos). - -### Fast-stats - -[Fast-stats](https://gitlab.com/gitlab-com/support/toolbox/fast-stats) is a tool -for creating and comparing performance statistics from GitLab logs. -For more details and instructions to run it, read the -[documentation for fast-stats](https://gitlab.com/gitlab-com/support/toolbox/fast-stats#usage). +<!-- This redirect file can be deleted after <2022-10-23>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/troubleshooting/img/database-query-dialog_v14_3.png b/doc/administration/logs/img/database-query-dialog_v14_3.png Binary files differindex 197cfa17da2..197cfa17da2 100644 --- a/doc/administration/troubleshooting/img/database-query-dialog_v14_3.png +++ b/doc/administration/logs/img/database-query-dialog_v14_3.png diff --git a/doc/administration/troubleshooting/img/network_monitor_xid.png b/doc/administration/logs/img/network_monitor_xid.png Binary files differindex fa64fd1509c..fa64fd1509c 100644 --- a/doc/administration/troubleshooting/img/network_monitor_xid.png +++ b/doc/administration/logs/img/network_monitor_xid.png diff --git a/doc/administration/troubleshooting/img/obtaining-a-session-cookie-for-request_v14_3.png b/doc/administration/logs/img/obtaining-a-session-cookie-for-request_v14_3.png Binary files differindex a63ebf9ecf2..a63ebf9ecf2 100644 --- a/doc/administration/troubleshooting/img/obtaining-a-session-cookie-for-request_v14_3.png +++ b/doc/administration/logs/img/obtaining-a-session-cookie-for-request_v14_3.png diff --git a/doc/administration/troubleshooting/img/paste-request-id-into-progress-bar_v14_3.png b/doc/administration/logs/img/paste-request-id-into-progress-bar_v14_3.png Binary files differindex a19585d7a89..a19585d7a89 100644 --- a/doc/administration/troubleshooting/img/paste-request-id-into-progress-bar_v14_3.png +++ b/doc/administration/logs/img/paste-request-id-into-progress-bar_v14_3.png diff --git a/doc/administration/troubleshooting/img/select-request-id-from-request-selector-drop-down-menu_v14_3.png b/doc/administration/logs/img/select-request-id-from-request-selector-drop-down-menu_v14_3.png Binary files differindex b8314056c9b..b8314056c9b 100644 --- a/doc/administration/troubleshooting/img/select-request-id-from-request-selector-drop-down-menu_v14_3.png +++ b/doc/administration/logs/img/select-request-id-from-request-selector-drop-down-menu_v14_3.png diff --git a/doc/administration/troubleshooting/img/view-pg-details_v14_3.png b/doc/administration/logs/img/view-pg-details_v14_3.png Binary files differindex 1fe12462e4e..1fe12462e4e 100644 --- a/doc/administration/troubleshooting/img/view-pg-details_v14_3.png +++ b/doc/administration/logs/img/view-pg-details_v14_3.png diff --git a/doc/administration/logs/index.md b/doc/administration/logs/index.md new file mode 100644 index 00000000000..d3529de08db --- /dev/null +++ b/doc/administration/logs/index.md @@ -0,0 +1,1182 @@ +--- +stage: Monitor +group: Respond +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Log system **(FREE SELF)** + +GitLab has an advanced log system where everything is logged, so you +can analyze your instance using various system log files. In addition to +system log files, GitLab Enterprise Edition provides [Audit Events](../audit_events.md). + +System log files are typically plain text in a standard log file format. +This guide talks about how to read and use these system log files. + +Read more about the log system and using the logs: + +- [Customize logging on Omnibus GitLab installations](https://docs.gitlab.com/omnibus/settings/logs.html) +including adjusting log retention, log forwarding, +switching logs from JSON to plain text logging, and more. +- [How to parse and analyze JSON logs](../logs/log_parsing.md). + +## Log Levels + +Each log message has an assigned log level that indicates its importance and verbosity. +Each logger has an assigned minimum log level. +A logger emits a log message only if its log level is equal to or above the minimum log level. + +The following log levels are supported: + +| Level | Name | +|:------|:----------| +| 0 | `DEBUG` | +| 1 | `INFO` | +| 2 | `WARN` | +| 3 | `ERROR` | +| 4 | `FATAL` | +| 5 | `UNKNOWN` | + +GitLab loggers emit all log messages because they are set to `DEBUG` by default. + +### Override default log level + +You can override the minimum log level for GitLab loggers using the `GITLAB_LOG_LEVEL` environment variable. +Valid values are either a value of `0` to `5`, or the name of the log level. + +Example: + +```shell +GITLAB_LOG_LEVEL=info +``` + +For some services, other log levels are in place that are not affected by this setting. +Some of these services have their own environment variables to override the log level. For example: + +| Service | Log level | Environment variable | +|:---------------------|:----------|:---------------------| +| GitLab API | `INFO` | | +| GitLab Cleanup | `INFO` | `DEBUG` | +| GitLab Doctor | `INFO` | `VERBOSE` | +| GitLab Export | `INFO` | `EXPORT_DEBUG` | +| GitLab Geo | `INFO` | | +| GitLab Import | `INFO` | `IMPORT_DEBUG` | +| GitLab QA Runtime | `INFO` | `QA_LOG_LEVEL` | +| Google APIs | `INFO` | | +| Rack Timeout | `ERROR` | | +| Sidekiq (server) | `INFO` | | +| Snowplow Tracker | `FATAL` | | +| gRPC Client (Gitaly) | `WARN` | `GRPC_LOG_LEVEL` | + +## Log Rotation + +The logs for a given service may be managed and rotated by: + +- `logrotate` +- `svlogd` (`runit`'s service logging daemon) +- `logrotate` and `svlogd` +- Or not at all + +The following table includes information about what's responsible for managing and rotating logs for +the included services. Logs +[managed by `svlogd`](https://docs.gitlab.com/omnibus/settings/logs.html#runit-logs) +are written to a file called `current`. The `logrotate` service built into GitLab +[manages all logs](https://docs.gitlab.com/omnibus/settings/logs.html#logrotate) +except those captured by `runit`. + +| Log type | Managed by logrotate | Managed by svlogd/runit | +|:------------------------------------------------|:------------------------|:------------------------| +| [Alertmanager Logs](#alertmanager-logs) | **{dotted-circle}** No | **{check-circle}** Yes | +| [Crond Logs](#crond-logs) | **{dotted-circle}** No | **{check-circle}** Yes | +| [Gitaly](#gitaly-logs) | **{check-circle}** Yes | **{check-circle}** Yes | +| [GitLab Exporter for Omnibus](#gitlab-exporter) | **{dotted-circle}** No | **{check-circle}** Yes | +| [GitLab Pages Logs](#pages-logs) | **{check-circle}** Yes | **{check-circle}** Yes | +| GitLab Rails | **{check-circle}** Yes | **{dotted-circle}** No | +| [GitLab Shell Logs](#gitlab-shelllog) | **{check-circle}** Yes | **{dotted-circle}** No | +| [Grafana Logs](#grafana-logs) | **{dotted-circle}** No | **{check-circle}** Yes | +| [LogRotate Logs](#logrotate-logs) | **{dotted-circle}** No | **{check-circle}** Yes | +| [Mailroom](#mail_room_jsonlog-default) | **{check-circle}** Yes | **{check-circle}** Yes | +| [NGINX](#nginx-logs) | **{check-circle}** Yes | **{check-circle}** Yes | +| [PostgreSQL Logs](#postgresql-logs) | **{dotted-circle}** No | **{check-circle}** Yes | +| [Praefect Logs](#praefect-logs) | **{dotted-circle}** Yes | **{check-circle}** Yes | +| [Prometheus Logs](#prometheus-logs) | **{dotted-circle}** No | **{check-circle}** Yes | +| [Puma](#puma-logs) | **{check-circle}** Yes | **{check-circle}** Yes | +| [Redis Logs](#redis-logs) | **{dotted-circle}** No | **{check-circle}** Yes | +| [Registry Logs](#registry-logs) | **{dotted-circle}** No | **{check-circle}** Yes | +| [Workhorse Logs](#workhorse-logs) | **{check-circle}** Yes | **{check-circle}** Yes | + +## `production_json.log` + +Depending on your installation method, this file is located at: + +- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/production_json.log` +- Installations from source: `/home/git/gitlab/log/production_json.log` + +It contains a structured log for Rails controller requests received from +GitLab, thanks to [Lograge](https://github.com/roidrage/lograge/). +Requests from the API are logged to a separate file in `api_json.log`. + +Each line contains JSON that can be ingested by services like Elasticsearch and Splunk. +Line breaks were added to examples for legibility: + +```json +{ + "method":"GET", + "path":"/gitlab/gitlab-foss/issues/1234", + "format":"html", + "controller":"Projects::IssuesController", + "action":"show", + "status":200, + "time":"2017-08-08T20:15:54.821Z", + "params":[{"key":"param_key","value":"param_value"}], + "remote_ip":"18.245.0.1", + "user_id":1, + "username":"admin", + "queue_duration_s":0.0, + "gitaly_calls":16, + "gitaly_duration_s":0.16, + "redis_calls":115, + "redis_duration_s":0.13, + "redis_read_bytes":1507378, + "redis_write_bytes":2920, + "correlation_id":"O1SdybnnIq7", + "cpu_s":17.50, + "db_duration_s":0.08, + "view_duration_s":2.39, + "duration_s":20.54, + "pid": 81836, + "worker_id":"puma_0" +} +``` + +This example was a GET request for a specific +issue. Each line also contains performance data, with times in +seconds: + +- `duration_s`: Total time to retrieve the request +- `queue_duration_s`: Total time the request was queued inside GitLab Workhorse +- `view_duration_s`: Total time inside the Rails views +- `db_duration_s`: Total time to retrieve data from PostgreSQL +- `cpu_s`: Total time spent on CPU +- `gitaly_duration_s`: Total time by Gitaly calls +- `gitaly_calls`: Total number of calls made to Gitaly +- `redis_calls`: Total number of calls made to Redis +- `redis_duration_s`: Total time to retrieve data from Redis +- `redis_read_bytes`: Total bytes read from Redis +- `redis_write_bytes`: Total bytes written to Redis +- `redis_<instance>_calls`: Total number of calls made to a Redis instance +- `redis_<instance>_duration_s`: Total time to retrieve data from a Redis instance +- `redis_<instance>_read_bytes`: Total bytes read from a Redis instance +- `redis_<instance>_write_bytes`: Total bytes written to a Redis instance +- `pid`: The worker's Linux process ID (changes when workers restart) +- `worker_id`: The worker's logical ID (does not change when workers restart) + +User clone and fetch activity using HTTP transport appears in the log as `action: git_upload_pack`. + +In addition, the log contains the originating IP address, +(`remote_ip`), the user's ID (`user_id`), and username (`username`). + +Some endpoints (such as `/search`) may make requests to Elasticsearch if using +[Advanced Search](../../user/search/advanced_search.md). These +additionally log `elasticsearch_calls` and `elasticsearch_call_duration_s`, +which correspond to: + +- `elasticsearch_calls`: Total number of calls to Elasticsearch +- `elasticsearch_duration_s`: Total time taken by Elasticsearch calls +- `elasticsearch_timed_out_count`: Total number of calls to Elasticsearch that + timed out and therefore returned partial results + +ActionCable connection and subscription events are also logged to this file and they follow the +previous format. The `method`, `path`, and `format` fields are not applicable, and are always empty. +The ActionCable connection or channel class is used as the `controller`. + +```json +{ + "method":null, + "path":null, + "format":null, + "controller":"IssuesChannel", + "action":"subscribe", + "status":200, + "time":"2020-05-14T19:46:22.008Z", + "params":[{"key":"project_path","value":"gitlab/gitlab-foss"},{"key":"iid","value":"1"}], + "remote_ip":"127.0.0.1", + "user_id":1, + "username":"admin", + "ua":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:76.0) Gecko/20100101 Firefox/76.0", + "correlation_id":"jSOIEynHCUa", + "duration_s":0.32566 +} +``` + +NOTE: +Starting with GitLab 12.5, if an error occurs, an +`exception` field is included with `class`, `message`, and +`backtrace`. Previous versions included an `error` field instead of +`exception.class` and `exception.message`. For example: + +```json +{ + "method": "GET", + "path": "/admin", + "format": "html", + "controller": "Admin::DashboardController", + "action": "index", + "status": 500, + "time": "2019-11-14T13:12:46.156Z", + "params": [], + "remote_ip": "127.0.0.1", + "user_id": 1, + "username": "root", + "ua": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:70.0) Gecko/20100101 Firefox/70.0", + "queue_duration": 274.35, + "correlation_id": "KjDVUhNvvV3", + "queue_duration_s":0.0, + "gitaly_calls":16, + "gitaly_duration_s":0.16, + "redis_calls":115, + "redis_duration_s":0.13, + "correlation_id":"O1SdybnnIq7", + "cpu_s":17.50, + "db_duration_s":0.08, + "view_duration_s":2.39, + "duration_s":20.54, + "pid": 81836, + "worker_id": "puma_0", + "exception.class": "NameError", + "exception.message": "undefined local variable or method `adsf' for #<Admin::DashboardController:0x00007ff3c9648588>", + "exception.backtrace": [ + "app/controllers/admin/dashboard_controller.rb:11:in `index'", + "ee/app/controllers/ee/admin/dashboard_controller.rb:14:in `index'", + "ee/lib/gitlab/ip_address_state.rb:10:in `with'", + "ee/app/controllers/ee/application_controller.rb:43:in `set_current_ip_address'", + "lib/gitlab/session.rb:11:in `with_session'", + "app/controllers/application_controller.rb:450:in `set_session_storage'", + "app/controllers/application_controller.rb:444:in `set_locale'", + "ee/lib/gitlab/jira/middleware.rb:19:in `call'" + ] +} +``` + +## `production.log` + +Depending on your installation method, this file is located at: + +- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/production.log` +- Installations from source: `/home/git/gitlab/log/production.log` + +It contains information about all performed requests. You can see the +URL and type of request, IP address, and what parts of code were +involved to service this particular request. Also, you can see all SQL +requests performed, and how much time each took. This task is +more useful for GitLab contributors and developers. Use part of this log +file when you're reporting bugs. For example: + +```plaintext +Started GET "/gitlabhq/yaml_db/tree/master" for 168.111.56.1 at 2015-02-12 19:34:53 +0200 +Processing by Projects::TreeController#show as HTML + Parameters: {"project_id"=>"gitlabhq/yaml_db", "id"=>"master"} + + ... [CUT OUT] + + Namespaces"."created_at" DESC, "namespaces"."id" DESC LIMIT 1 [["id", 26]] + CACHE (0.0ms) SELECT "members".* FROM "members" WHERE "members"."source_type" = 'Project' AND "members"."type" IN ('ProjectMember') AND "members"."source_id" = $1 AND "members"."source_type" = $2 AND "members"."user_id" = 1 ORDER BY "members"."created_at" DESC, "members"."id" DESC LIMIT 1 [["source_id", 18], ["source_type", "Project"]] + CACHE (0.0ms) SELECT "members".* FROM "members" WHERE "members"."source_type" = 'Project' AND "members". + (1.4ms) SELECT COUNT(*) FROM "merge_requests" WHERE "merge_requests"."target_project_id" = $1 AND ("merge_requests"."state" IN ('opened','reopened')) [["target_project_id", 18]] + Rendered layouts/nav/_project.html.haml (28.0ms) + Rendered layouts/_collapse_button.html.haml (0.2ms) + Rendered layouts/_flash.html.haml (0.1ms) + Rendered layouts/_page.html.haml (32.9ms) +Completed 200 OK in 166ms (Views: 117.4ms | ActiveRecord: 27.2ms) +``` + +In this example, the server processed an HTTP request with URL +`/gitlabhq/yaml_db/tree/master` from IP `168.111.56.1` at `2015-02-12 19:34:53 +0200`. +The request was processed by `Projects::TreeController`. + +## `api_json.log` + +Depending on your installation method, this file is located at: + +- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/api_json.log` +- Installations from source: `/home/git/gitlab/log/api_json.log` + +It helps you see requests made directly to the API. For example: + +```json +{ + "time":"2018-10-29T12:49:42.123Z", + "severity":"INFO", + "duration":709.08, + "db":14.59, + "view":694.49, + "status":200, + "method":"GET", + "path":"/api/v4/projects", + "params":[{"key":"action","value":"git-upload-pack"},{"key":"changes","value":"_any"},{"key":"key_id","value":"secret"},{"key":"secret_token","value":"[FILTERED]"}], + "host":"localhost", + "remote_ip":"::1", + "ua":"Ruby", + "route":"/api/:version/projects", + "user_id":1, + "username":"root", + "queue_duration":100.31, + "gitaly_calls":30, + "gitaly_duration":5.36, + "pid": 81836, + "worker_id": "puma_0", + ... +} +``` + +This entry shows an internal endpoint accessed to check whether an +associated SSH key can download the project in question by using a `git fetch` or +`git clone`. In this example, we see: + +- `duration`: Total time in milliseconds to retrieve the request +- `queue_duration`: Total time in milliseconds the request was queued inside GitLab Workhorse +- `method`: The HTTP method used to make the request +- `path`: The relative path of the query +- `params`: Key-value pairs passed in a query string or HTTP body (sensitive parameters, such as passwords and tokens, are filtered out) +- `ua`: The User-Agent of the requester + +## `application.log` + +Depending on your installation method, this file is located at: + +- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/application.log` +- Installations from source: `/home/git/gitlab/log/application.log` + +It helps you discover events happening in your instance such as user creation +and project deletion. For example: + +```plaintext +October 06, 2014 11:56: User "Administrator" (admin@example.com) was created +October 06, 2014 11:56: Documentcloud created a new project "Documentcloud / Underscore" +October 06, 2014 11:56: Gitlab Org created a new project "Gitlab Org / Gitlab Ce" +October 07, 2014 11:25: User "Claudie Hodkiewicz" (nasir_stehr@olson.co.uk) was removed +October 07, 2014 11:25: Project "project133" was removed +``` + +## `application_json.log` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22812) in GitLab 12.7. + +Depending on your installation method, this file is located at: + +- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/application_json.log` +- Installations from source: `/home/git/gitlab/log/application_json.log` + +It contains the JSON version of the logs in `application.log`, like this example: + +```json +{ + "severity":"INFO", + "time":"2020-01-14T13:35:15.466Z", + "correlation_id":"3823a1550b64417f9c9ed8ee0f48087e", + "message":"User \"Administrator\" (admin@example.com) was created" +} +{ + "severity":"INFO", + "time":"2020-01-14T13:35:15.466Z", + "correlation_id":"78e3df10c9a18745243d524540bd5be4", + "message":"Project \"project133\" was removed" +} +``` + +## `integrations_json.log` + +Depending on your installation method, this file is located at: + +- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/integrations_json.log` +- Installations from source: `/home/git/gitlab/log/integrations_json.log` + +It contains information about [integration](../../user/project/integrations/index.md) +activities, such as Jira, Asana, and irker services. It uses JSON format, +like this example: + +```json +{ + "severity":"ERROR", + "time":"2018-09-06T14:56:20.439Z", + "service_class":"Integrations::Jira", + "project_id":8, + "project_path":"h5bp/html5-boilerplate", + "message":"Error sending message", + "client_url":"http://jira.gitlap.com:8080", + "error":"execution expired" +} +{ + "severity":"INFO", + "time":"2018-09-06T17:15:16.365Z", + "service_class":"Integrations::Jira", + "project_id":3, + "project_path":"namespace2/project2", + "message":"Successfully posted", + "client_url":"http://jira.example.com" +} +``` + +## `kubernetes.log` + +Depending on your installation method, this file is located at: + +- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/kubernetes.log` +- Installations from source: `/home/git/gitlab/log/kubernetes.log` + +It logs information related to the Kubernetes Integration, including errors +during installing cluster applications on your managed Kubernetes +clusters. + +Each line contains JSON that can be ingested by services like Elasticsearch and Splunk. +Line breaks have been added to the following example for clarity: + +```json +{ + "severity":"ERROR", + "time":"2018-11-23T15:14:54.652Z", + "exception":"Kubeclient::HttpError", + "error_code":401, + "service":"Clusters::Applications::CheckInstallationProgressService", + "app_id":14, + "project_ids":[1], + "group_ids":[], + "message":"Unauthorized" +} +{ + "severity":"ERROR", + "time":"2018-11-23T15:42:11.647Z", + "exception":"Kubeclient::HttpError", + "error_code":null, + "service":"Clusters::Applications::InstallService", + "app_id":2, + "project_ids":[19], + "group_ids":[], + "message":"SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)" +} +``` + +## `git_json.log` + +Depending on your installation method, this file is located at: + +- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/git_json.log` +- Installations from source: `/home/git/gitlab/log/git_json.log` + +After GitLab version 12.2, this file was renamed from `githost.log` to +`git_json.log` and stored in JSON format. + +GitLab has to interact with Git repositories, but in some rare cases +something can go wrong. If this happens, you need to know exactly what +happened. This log file contains all failed requests from GitLab to Git +repositories. In the majority of cases this file is useful for developers +only. For example: + +```json +{ + "severity":"ERROR", + "time":"2019-07-19T22:16:12.528Z", + "correlation_id":"FeGxww5Hj64", + "message":"Command failed [1]: /usr/bin/git --git-dir=/Users/vsizov/gitlab-development-kit/gitlab/tmp/tests/gitlab-satellites/group184/gitlabhq/.git --work-tree=/Users/vsizov/gitlab-development-kit/gitlab/tmp/tests/gitlab-satellites/group184/gitlabhq merge --no-ff -mMerge branch 'feature_conflict' into 'feature' source/feature_conflict\n\nerror: failed to push some refs to '/Users/vsizov/gitlab-development-kit/repositories/gitlabhq/gitlab_git.git'" +} +``` + +## `audit_json.log` **(FREE)** + +NOTE: +GitLab Free tracks a small number of different audit events. +GitLab Premium tracks many more. + +Depending on your installation method, this file is located at: + +- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/audit_json.log` +- Installations from source: `/home/git/gitlab/log/audit_json.log` + +Changes to group or project settings and memberships (`target_details`) +are logged to this file. For example: + +```json +{ + "severity":"INFO", + "time":"2018-10-17T17:38:22.523Z", + "author_id":3, + "entity_id":2, + "entity_type":"Project", + "change":"visibility", + "from":"Private", + "to":"Public", + "author_name":"John Doe4", + "target_id":2, + "target_type":"Project", + "target_details":"namespace2/project2" +} +``` + +## Sidekiq Logs + +NOTE: +In Omnibus GitLab `12.10` or earlier, the Sidekiq log is at `/var/log/gitlab/gitlab-rails/sidekiq.log`. + +For Omnibus GitLab installations, some Sidekiq logs are in `/var/log/gitlab/sidekiq/current` +and as follows. + +### `sidekiq.log` + +Depending on your installation method, this file is located at: + +- Omnibus GitLab: `/var/log/gitlab/sidekiq/current` +- Installations from source: `/home/git/gitlab/log/sidekiq.log` + +GitLab uses background jobs for processing tasks which can take a long +time. All information about processing these jobs are written down to +this file. For example: + +```plaintext +2014-06-10T07:55:20Z 2037 TID-tm504 ERROR: /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/1.9.1/gems/redis-3.0.7/lib/redis/client.rb:228:in `read' +2014-06-10T18:18:26Z 14299 TID-55uqo INFO: Booting Sidekiq 3.0.0 with redis options {:url=>"redis://localhost:6379/0", :namespace=>"sidekiq"} +``` + +Instead of the previous format, you can opt to generate JSON logs for +Sidekiq. For example: + +```json +{ + "severity":"INFO", + "time":"2018-04-03T22:57:22.071Z", + "queue":"cronjob:update_all_mirrors", + "args":[], + "class":"UpdateAllMirrorsWorker", + "retry":false, + "queue_namespace":"cronjob", + "jid":"06aeaa3b0aadacf9981f368e", + "created_at":"2018-04-03T22:57:21.930Z", + "enqueued_at":"2018-04-03T22:57:21.931Z", + "pid":10077, + "worker_id":"sidekiq_0", + "message":"UpdateAllMirrorsWorker JID-06aeaa3b0aadacf9981f368e: done: 0.139 sec", + "job_status":"done", + "duration":0.139, + "completed_at":"2018-04-03T22:57:22.071Z", + "db_duration":0.05, + "db_duration_s":0.0005, + "gitaly_duration":0, + "gitaly_calls":0 +} +``` + +For Omnibus GitLab installations, add the configuration option: + +```ruby +sidekiq['log_format'] = 'json' +``` + +For installations from source, edit the `gitlab.yml` and set the Sidekiq +`log_format` configuration option: + +```yaml + ## Sidekiq + sidekiq: + log_format: json +``` + +### `sidekiq_client.log` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26586) in GitLab 12.9. + +Depending on your installation method, this file is located at: + +- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/sidekiq_client.log` +- Installations from source: `/home/git/gitlab/log/sidekiq_client.log` + +This file contains logging information about jobs before Sidekiq starts +processing them, such as before being enqueued. + +This log file follows the same structure as +[`sidekiq.log`](#sidekiqlog), so it is structured as JSON if +you've configured this for Sidekiq as mentioned above. + +## `gitlab-shell.log` + +GitLab Shell is used by GitLab for executing Git commands and provide SSH +access to Git repositories. + +### For GitLab versions 12.10 and up + +Information containing `git-{upload-pack,receive-pack}` requests is at +`/var/log/gitlab/gitlab-shell/gitlab-shell.log`. Information about hooks to +GitLab Shell from Gitaly is at `/var/log/gitlab/gitaly/current`. + +Example log entries for `/var/log/gitlab/gitlab-shell/gitlab-shell.log`: + +```json +{ + "duration_ms": 74.104, + "level": "info", + "method": "POST", + "msg": "Finished HTTP request", + "time": "2020-04-17T20:28:46Z", + "url": "http://127.0.0.1:8080/api/v4/internal/allowed" +} +{ + "command": "git-upload-pack", + "git_protocol": "", + "gl_project_path": "root/example", + "gl_repository": "project-1", + "level": "info", + "msg": "executing git command", + "time": "2020-04-17T20:28:46Z", + "user_id": "user-1", + "username": "root" +} +``` + +Example log entries for `/var/log/gitlab/gitaly/current`: + +```json +{ + "method": "POST", + "url": "http://127.0.0.1:8080/api/v4/internal/allowed", + "duration": 0.058012959, + "gitaly_embedded": true, + "pid": 16636, + "level": "info", + "msg": "finished HTTP request", + "time": "2020-04-17T20:29:08+00:00" +} +{ + "method": "POST", + "url": "http://127.0.0.1:8080/api/v4/internal/pre_receive", + "duration": 0.031022552, + "gitaly_embedded": true, + "pid": 16636, + "level": "info", + "msg": "finished HTTP request", + "time": "2020-04-17T20:29:08+00:00" +} +``` + +### For GitLab versions 12.5 through 12.9 + +For GitLab 12.5 to 12.9, depending on your installation method, this +file is located at: + +- Omnibus GitLab: `/var/log/gitlab/gitaly/gitlab-shell.log` +- Installation from source: `/home/git/gitaly/gitlab-shell.log` + +Example log entries: + +```json +{ + "method": "POST", + "url": "http://127.0.0.1:8080/api/v4/internal/post_receive", + "duration": 0.031809164, + "gitaly_embedded": true, + "pid": 27056, + "level": "info", + "msg": "finished HTTP request", + "time": "2020-04-17T16:24:38+00:00" +} +``` + +### For GitLab 12.5 and earlier + +For GitLab 12.5 and earlier, the file is at `/var/log/gitlab/gitlab-shell/gitlab-shell.log`. + +Example log entries: + +```plaintext +I, [2015-02-13T06:17:00.671315 #9291] INFO -- : Adding project root/example.git at </var/opt/gitlab/git-data/repositories/root/dcdcdcdcd.git>. +I, [2015-02-13T06:17:00.679433 #9291] INFO -- : Moving existing hooks directory and symlinking global hooks directory for /var/opt/gitlab/git-data/repositories/root/example.git. +``` + +User clone/fetch activity using SSH transport appears in this log as +`executing git command <gitaly-upload-pack...`. + +## Gitaly Logs + +This file is in `/var/log/gitlab/gitaly/current` and is produced by [runit](http://smarden.org/runit/). +`runit` is packaged with Omnibus GitLab and a brief explanation of its purpose +is available [in the Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/architecture/#runit). +[Log files are rotated](http://smarden.org/runit/svlogd.8.html), renamed in +Unix timestamp format, and `gzip`-compressed (like `@1584057562.s`). + +### `grpc.log` + +This file is at `/var/log/gitlab/gitlab-rails/grpc.log` for Omnibus GitLab +packages. Native [gRPC](https://grpc.io/) logging used by Gitaly. + +### `gitaly_ruby_json.log` + +> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/2678) in GitLab 13.6. + +This file is at `/var/log/gitlab/gitaly/gitaly_ruby_json.log` and is +produced by [`gitaly-ruby`](../gitaly/reference.md#gitaly-ruby). It contains an +access log of gRPC calls made by Gitaly to `gitaly-ruby`. + +### `gitaly_hooks.log` + +This file is at `/var/log/gitlab/gitaly/gitaly_hooks.log` and is +produced by `gitaly-hooks` command. It also contains records about +failures received during processing of the responses from GitLab API. + +## Puma Logs + +### `puma_stdout.log` + +Depending on your installation method, this file is located at: + +- Omnibus GitLab: `/var/log/gitlab/puma/puma_stdout.log` +- Installations from source: `/home/git/gitlab/log/puma_stdout.log` + +### `puma_stderr.log` + +Depending on your installation method, this file is located at: + +- Omnibus GitLab: `/var/log/gitlab/puma/puma_stderr.log` +- Installations from source: `/home/git/gitlab/log/puma_stderr.log` + +## `repocheck.log` + +Depending on your installation method, this file is located at: + +- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/repocheck.log` +- Installations from source: `/home/git/gitlab/log/repocheck.log` + +It logs information whenever a [repository check is run](../repository_checks.md) +on a project. + +## `importer.log` + +Depending on your installation method, this file is located at: + +- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/importer.log` +- Installations from source: `/home/git/gitlab/log/importer.log` + +It logs the progress of the import process. + +## `exporter.log` + +> Introduced in GitLab 13.1. + +Depending on your installation method, this file is located at: + +- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/exporter.log` +- Installations from source: `/home/git/gitlab/log/exporter.log` + +It logs the progress of the export process. + +## `features_json.log` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/59587) in GitLab 13.7. + +Depending on your installation method, this file is located at: + +- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/features_json.log` +- Installations from source: `/home/git/gitlab/log/features_json.log` + +The modification events from [Feature flags in development of GitLab](../../development/feature_flags/index.md) +are recorded in this file. For example: + +```json +{"severity":"INFO","time":"2020-11-24T02:30:59.860Z","correlation_id":null,"key":"cd_auto_rollback","action":"enable","extra.thing":"true"} +{"severity":"INFO","time":"2020-11-24T02:31:29.108Z","correlation_id":null,"key":"cd_auto_rollback","action":"enable","extra.thing":"true"} +{"severity":"INFO","time":"2020-11-24T02:31:29.129Z","correlation_id":null,"key":"cd_auto_rollback","action":"disable","extra.thing":"false"} +{"severity":"INFO","time":"2020-11-24T02:31:29.177Z","correlation_id":null,"key":"cd_auto_rollback","action":"enable","extra.thing":"Project:1"} +{"severity":"INFO","time":"2020-11-24T02:31:29.183Z","correlation_id":null,"key":"cd_auto_rollback","action":"disable","extra.thing":"Project:1"} +{"severity":"INFO","time":"2020-11-24T02:31:29.188Z","correlation_id":null,"key":"cd_auto_rollback","action":"enable_percentage_of_time","extra.percentage":"50"} +{"severity":"INFO","time":"2020-11-24T02:31:29.193Z","correlation_id":null,"key":"cd_auto_rollback","action":"disable_percentage_of_time"} +{"severity":"INFO","time":"2020-11-24T02:31:29.198Z","correlation_id":null,"key":"cd_auto_rollback","action":"enable_percentage_of_actors","extra.percentage":"50"} +{"severity":"INFO","time":"2020-11-24T02:31:29.203Z","correlation_id":null,"key":"cd_auto_rollback","action":"disable_percentage_of_actors"} +{"severity":"INFO","time":"2020-11-24T02:31:29.329Z","correlation_id":null,"key":"cd_auto_rollback","action":"remove"} +``` + +## `auth.log` + +> Introduced in GitLab 12.0. + +Depending on your installation method, this file is located at: + +- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/auth.log` +- Installations from source: `/home/git/gitlab/log/auth.log` + +This log records: + +- Requests over the [Rate Limit](../../user/admin_area/settings/rate_limits_on_raw_endpoints.md) on raw endpoints. +- [Protected paths](../../user/admin_area/settings/protected_paths.md) abusive requests. +- In GitLab versions [12.3](https://gitlab.com/gitlab-org/gitlab/-/issues/29239) and later, + user ID and username, if available. + +## `graphql_json.log` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/59587) in GitLab 12.0. + +Depending on your installation method, this file is located at: + +- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/graphql_json.log` +- Installations from source: `/home/git/gitlab/log/graphql_json.log` + +GraphQL queries are recorded in the file. For example: + +```json +{"query_string":"query IntrospectionQuery{__schema {queryType { name },mutationType { name }}}...(etc)","variables":{"a":1,"b":2},"complexity":181,"depth":1,"duration_s":7} +``` + +## `migrations.log` + +> Introduced in GitLab 12.3. + +Depending on your installation method, this file is located at: + +- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/migrations.log` +- Installations from source: `/home/git/gitlab/log/migrations.log` + +## `mail_room_json.log` (default) + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/19186) in GitLab 12.6. + +Depending on your installation method, this file is located at: + +- Omnibus GitLab: `/var/log/gitlab/mailroom/current` +- Installations from source: `/home/git/gitlab/log/mail_room_json.log` + +This structured log file records internal activity in the `mail_room` gem. +Its name and path are configurable, so the name and path may not match the above. + +## Reconfigure logs + +Reconfigure log files are in `/var/log/gitlab/reconfigure` for Omnibus GitLab +packages. Installations from source don't have reconfigure logs. A reconfigure log +is populated whenever `gitlab-ctl reconfigure` is run manually or as part of an upgrade. + +Reconfigure logs files are named according to the UNIX timestamp of when the reconfigure +was initiated, such as `1509705644.log` + +## `sidekiq_exporter.log` and `web_exporter.log` + +If Prometheus metrics and the Sidekiq Exporter are both enabled, Sidekiq +starts a Web server and listen to the defined port (default: +`8082`). By default, Sidekiq Exporter access logs are disabled but can +be enabled based on your installation method: + +- Omnibus GitLab: Use the `sidekiq['exporter_log_enabled'] = true` + option in `/etc/gitlab/gitlab.rb` +- Installations from source: Use the `sidekiq_exporter.log_enabled` option + in `gitlab.yml` + +When enabled, depending on your installation method, this file is located at: + +- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/sidekiq_exporter.log` +- Installations from source: `/home/git/gitlab/log/sidekiq_exporter.log` + +If Prometheus metrics and the Web Exporter are both enabled, Puma +starts a Web server and listen to the defined port (default: `8083`), and access logs +are generated in a location based on your installation method: + +- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/web_exporter.log` +- Installations from source: `/home/git/gitlab/log/web_exporter.log` + +## `database_load_balancing.log` **(PREMIUM SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/15442) in GitLab 12.3. + +Contains details of GitLab [Database Load Balancing](../postgresql/database_load_balancing.md). +Depending on your installation method, this file is located at: + +- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/database_load_balancing.log` +- Installations from source: `/home/git/gitlab/log/database_load_balancing.log` + +## `elasticsearch.log` **(PREMIUM SELF)** + +> Introduced in GitLab 12.6. + +This file logs information related to the Elasticsearch Integration, including +errors during indexing or searching Elasticsearch. Depending on your installation +method, this file is located at: + +- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/elasticsearch.log` +- Installations from source: `/home/git/gitlab/log/elasticsearch.log` + +Each line contains JSON that can be ingested by services like Elasticsearch and Splunk. +Line breaks have been added to the following example line for clarity: + +```json +{ + "severity":"DEBUG", + "time":"2019-10-17T06:23:13.227Z", + "correlation_id":null, + "message":"redacted_search_result", + "class_name":"Milestone", + "id":2, + "ability":"read_milestone", + "current_user_id":2, + "query":"project" +} +``` + +## `exceptions_json.log` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17819) in GitLab 12.6. + +This file logs the information about exceptions being tracked by +`Gitlab::ErrorTracking`, which provides a standard and consistent way of +[processing rescued exceptions](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/logging.md#exception-handling). +Depending on your installation method, this file is located at: + +- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/exceptions_json.log` +- Installations from source: `/home/git/gitlab/log/exceptions_json.log` + +Each line contains JSON that can be ingested by Elasticsearch. For example: + +```json +{ + "severity": "ERROR", + "time": "2019-12-17T11:49:29.485Z", + "correlation_id": "AbDVUrrTvM1", + "extra.project_id": 55, + "extra.relation_key": "milestones", + "extra.relation_index": 1, + "exception.class": "NoMethodError", + "exception.message": "undefined method `strong_memoize' for #<Gitlab::ImportExport::RelationFactory:0x00007fb5d917c4b0>", + "exception.backtrace": [ + "lib/gitlab/import_export/relation_factory.rb:329:in `unique_relation?'", + "lib/gitlab/import_export/relation_factory.rb:345:in `find_or_create_object!'" + ] +} +``` + +## `service_measurement.log` + +> Introduced in GitLab 13.0. + +Depending on your installation method, this file is located at: + +- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/service_measurement.log` +- Installations from source: `/home/git/gitlab/log/service_measurement.log` + +It contains only a single structured log with measurements for each service execution. +It contains measurements such as the number of SQL calls, `execution_time`, `gc_stats`, and `memory usage`. + +For example: + +```json +{ "severity":"INFO", "time":"2020-04-22T16:04:50.691Z","correlation_id":"04f1366e-57a1-45b8-88c1-b00b23dc3616","class":"Projects::ImportExport::ExportService","current_user":"John Doe","project_full_path":"group1/test-export","file_path":"/path/to/archive","gc_stats":{"count":{"before":127,"after":127,"diff":0},"heap_allocated_pages":{"before":10369,"after":10369,"diff":0},"heap_sorted_length":{"before":10369,"after":10369,"diff":0},"heap_allocatable_pages":{"before":0,"after":0,"diff":0},"heap_available_slots":{"before":4226409,"after":4226409,"diff":0},"heap_live_slots":{"before":2542709,"after":2641420,"diff":98711},"heap_free_slots":{"before":1683700,"after":1584989,"diff":-98711},"heap_final_slots":{"before":0,"after":0,"diff":0},"heap_marked_slots":{"before":2542704,"after":2542704,"diff":0},"heap_eden_pages":{"before":10369,"after":10369,"diff":0},"heap_tomb_pages":{"before":0,"after":0,"diff":0},"total_allocated_pages":{"before":10369,"after":10369,"diff":0},"total_freed_pages":{"before":0,"after":0,"diff":0},"total_allocated_objects":{"before":24896308,"after":24995019,"diff":98711},"total_freed_objects":{"before":22353599,"after":22353599,"diff":0},"malloc_increase_bytes":{"before":140032,"after":6650240,"diff":6510208},"malloc_increase_bytes_limit":{"before":25804104,"after":25804104,"diff":0},"minor_gc_count":{"before":94,"after":94,"diff":0},"major_gc_count":{"before":33,"after":33,"diff":0},"remembered_wb_unprotected_objects":{"before":34284,"after":34284,"diff":0},"remembered_wb_unprotected_objects_limit":{"before":68568,"after":68568,"diff":0},"old_objects":{"before":2404725,"after":2404725,"diff":0},"old_objects_limit":{"before":4809450,"after":4809450,"diff":0},"oldmalloc_increase_bytes":{"before":140032,"after":6650240,"diff":6510208},"oldmalloc_increase_bytes_limit":{"before":68537556,"after":68537556,"diff":0}},"time_to_finish":0.12298400001600385,"number_of_sql_calls":70,"memory_usage":"0.0 MiB","label":"process_48616"} +``` + +## `geo.log` **(PREMIUM SELF)** + +Geo stores structured log messages in a `geo.log` file. For Omnibus GitLab +installations, this file is at `/var/log/gitlab/gitlab-rails/geo.log`. + +This file contains information about when Geo attempts to sync repositories +and files. Each line in the file contains a separate JSON entry that can be +ingested into (for example, Elasticsearch or Splunk). + +For example: + +```json +{"severity":"INFO","time":"2017-08-06T05:40:16.104Z","message":"Repository update","project_id":1,"source":"repository","resync_repository":true,"resync_wiki":true,"class":"Gitlab::Geo::LogCursor::Daemon","cursor_delay_s":0.038} +``` + +This message shows that Geo detected that a repository update was needed for project `1`. + +## `update_mirror_service_json.log` + +Depending on your installation method, this file is located at: + +- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/update_mirror_service_json.log` +- Installations from source: `/home/git/gitlab/log/update_mirror_service_json.log` + +This file contains information about LFS errors that occurred during project mirroring. +While we work to move other project mirroring errors into this log, the [general log](#productionlog) +can be used. + +```json +{ + "severity":"ERROR", + "time":"2020-07-28T23:29:29.473Z", + "correlation_id":"5HgIkCJsO53", + "user_id":"x", + "project_id":"x", + "import_url":"https://mirror-source/group/project.git", + "error_message":"The LFS objects download list couldn't be imported. Error: Unauthorized" +} +``` + +## Registry Logs + +For Omnibus GitLab installations, Container Registry logs are in `/var/log/gitlab/registry/current`. + +## NGINX Logs + +For Omnibus GitLab installations, NGINX logs are in: + +- `/var/log/gitlab/nginx/gitlab_access.log`: A log of requests made to GitLab +- `/var/log/gitlab/nginx/gitlab_error.log`: A log of NGINX errors for GitLab +- `/var/log/gitlab/nginx/gitlab_pages_access.log`: A log of requests made to Pages static sites +- `/var/log/gitlab/nginx/gitlab_pages_error.log`: A log of NGINX errors for Pages static sites +- `/var/log/gitlab/nginx/gitlab_registry_access.log`: A log of requests made to the Container Registry +- `/var/log/gitlab/nginx/gitlab_registry_error.log`: A log of NGINX errors for the Container Registry +- `/var/log/gitlab/nginx/gitlab_mattermost_access.log`: A log of requests made to Mattermost +- `/var/log/gitlab/nginx/gitlab_mattermost_error.log`: A log of NGINX errors for Mattermost + +Below is the default GitLab NGINX access log format: + +```plaintext +$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" +``` + +## Pages Logs + +For Omnibus GitLab installations, Pages logs are in `/var/log/gitlab/gitlab-pages/current`. + +For example: + +```json +{ + "level": "info", + "msg": "GitLab Pages Daemon", + "revision": "52b2899", + "time": "2020-04-22T17:53:12Z", + "version": "1.17.0" +} +{ + "level": "info", + "msg": "URL: https://gitlab.com/gitlab-org/gitlab-pages", + "time": "2020-04-22T17:53:12Z" +} +{ + "gid": 998, + "in-place": false, + "level": "info", + "msg": "running the daemon as unprivileged user", + "time": "2020-04-22T17:53:12Z", + "uid": 998 +} +``` + +## Mattermost Logs + +For Omnibus GitLab installations, Mattermost logs are in these locations: + +- `/var/log/gitlab/mattermost/mattermost.log` +- `/var/log/gitlab/mattermost/current` + +## Workhorse Logs + +For Omnibus GitLab installations, Workhorse logs are in `/var/log/gitlab/gitlab-workhorse/current`. + +## PostgreSQL Logs + +For Omnibus GitLab installations, PostgreSQL logs are in `/var/log/gitlab/postgresql/current`. + +## Prometheus Logs + +For Omnibus GitLab installations, Prometheus logs are in `/var/log/gitlab/prometheus/current`. + +## Redis Logs + +For Omnibus GitLab installations, Redis logs are in `/var/log/gitlab/redis/current`. + +## Alertmanager Logs + +For Omnibus GitLab installations, Alertmanager logs are in `/var/log/gitlab/alertmanager/current`. + +<!-- vale gitlab.Spelling = NO --> + +## Crond Logs + +For Omnibus GitLab installations, crond logs are in `/var/log/gitlab/crond/`. + +<!-- vale gitlab.Spelling = YES --> + +## Grafana Logs + +For Omnibus GitLab installations, Grafana logs are in `/var/log/gitlab/grafana/current`. + +## LogRotate Logs + +For Omnibus GitLab installations, `logrotate` logs are in `/var/log/gitlab/logrotate/current`. + +## GitLab Monitor Logs + +For Omnibus GitLab installations, GitLab Monitor logs are in `/var/log/gitlab/gitlab-monitor/`. + +## GitLab Exporter + +For Omnibus GitLab installations, GitLab Exporter logs are in `/var/log/gitlab/gitlab-exporter/current`. + +## GitLab agent server + +For Omnibus GitLab installations, GitLab agent server logs are +in `/var/log/gitlab/gitlab-kas/current`. + +## Praefect Logs + +For Omnibus GitLab installations, Praefect logs are in `/var/log/gitlab/praefect/`. + +GitLab also tracks [Prometheus metrics for Praefect](../gitaly/monitoring.md#monitor-gitaly-cluster). + +## Backup log + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63832) in GitLab 14.1. + +For Omnibus installations, the backup log is located at `/var/log/gitlab/gitlab-rails/backup_json.log`. + +This log is populated when a [GitLab backup is created](../../raketasks/backup_restore.md). You can use this log to understand how the backup process performed. + +## Performance bar stats + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/48149) in GitLab 13.7. + +Depending on your installation method, this file is located at: + +- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/performance_bar_json.log` +- Installations from source: `/home/git/gitlab/log/performance_bar_json.log` + +Performance bar statistics (currently only duration of SQL queries) are recorded +in that file. For example: + +```json +{"severity":"INFO","time":"2020-12-04T09:29:44.592Z","correlation_id":"33680b1490ccd35981b03639c406a697","filename":"app/models/ci/pipeline.rb","method_path":"app/models/ci/pipeline.rb:each_with_object","request_id":"rYHomD0VJS4","duration_ms":26.889,"count":2,"query_type": "active-record"} +``` + +These statistics are logged on .com only, disabled on self-deployments. + +## Gathering logs + +When [troubleshooting](../troubleshooting/index.md) issues that aren't localized to one of the +previously listed components, it's helpful to simultaneously gather multiple logs and statistics +from a GitLab instance. + +NOTE: +GitLab Support often asks for one of these, and maintains the required tools. + +### Briefly tail the main logs + +If the bug or error is readily reproducible, save the main GitLab logs +[to a file](../troubleshooting/linux_cheat_sheet.md#files-and-directories) while reproducing the +problem a few times: + +```shell +sudo gitlab-ctl tail | tee /tmp/<case-ID-and-keywords>.log +``` + +Conclude the log gathering with <kbd>Control</kbd> + <kbd>C</kbd>. + +### GitLabSOS + +If performance degradations or cascading errors occur that can't readily be attributed to one +of the previously listed GitLab components, [GitLabSOS](https://gitlab.com/gitlab-com/support/toolbox/gitlabsos/) +can provide a broader perspective of the GitLab instance. For more details and instructions +to run it, read [the GitLabSOS documentation](https://gitlab.com/gitlab-com/support/toolbox/gitlabsos/#gitlabsos). + +### Fast-stats + +[Fast-stats](https://gitlab.com/gitlab-com/support/toolbox/fast-stats) is a tool +for creating and comparing performance statistics from GitLab logs. +For more details and instructions to run it, read the +[documentation for fast-stats](https://gitlab.com/gitlab-com/support/toolbox/fast-stats#usage). + +## Find relevant log entries with a correlation ID + +Most requests have a log ID that can be used to [find relevant log entries](tracing_correlation_id.md). diff --git a/doc/administration/logs/log_parsing.md b/doc/administration/logs/log_parsing.md new file mode 100644 index 00000000000..20b439af49b --- /dev/null +++ b/doc/administration/logs/log_parsing.md @@ -0,0 +1,316 @@ +--- +stage: Systems +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Parsing GitLab logs with `jq` **(FREE SELF)** + +We recommend using log aggregation and search tools like Kibana and Splunk whenever possible, +but if they are not available you can still quickly parse +[GitLab logs](../logs/index.md) in JSON format +(the default in GitLab 12.0 and later) using [`jq`](https://stedolan.github.io/jq/). + +NOTE: +Specifically for summarizing error events and basic usage statistics, +the GitLab Support Team provides the specialised +[`fast-stats` tool](https://gitlab.com/gitlab-com/support/toolbox/fast-stats/#when-to-use-it). + +## What is JQ? + +As noted in its [manual](https://stedolan.github.io/jq/manual/), `jq` is a command-line JSON processor. The following examples +include use cases targeted for parsing GitLab log files. + +## Parsing Logs + +The examples listed below address their respective log files by +their relative Omnibus paths and default filenames. +Find the respective full paths in the [GitLab logs sections](../logs/index.md#production_jsonlog). + +### General Commands + +#### Pipe colorized `jq` output into `less` + +```shell +jq . <FILE> -C | less -R +``` + +#### Search for a term and pretty-print all matching lines + +```shell +grep <TERM> <FILE> | jq . +``` + +#### Skip invalid lines of JSON + +```shell +jq -cR 'fromjson?' file.json | jq <COMMAND> +``` + +By default `jq` errors out when it encounters a line that is not valid JSON. +This skips over all invalid lines and parses the rest. + +#### Print a JSON log's time range + +```shell +cat log.json | (head -1; tail -1) | jq '.time' +``` + +Use `zcat` if the file has been rotated and compressed: + +```shell +zcat @400000006026b71d1a7af804.s | (head -1; tail -1) | jq '.time' + +zcat some_json.log.25.gz | (head -1; tail -1) | jq '.time' +``` + +#### Get activity for correlation ID across multiple JSON logs in chronological order + +```shell +grep -hR <correlationID> | jq -c -R 'fromjson?' | jq -C -s 'sort_by(.time)' | less -R +``` + +### Parsing `gitlab-rails/production_json.log` and `gitlab-rails/api_json.log` + +#### Find all requests with a 5XX status code + +```shell +jq 'select(.status >= 500)' <FILE> +``` + +#### Top 10 slowest requests + +```shell +jq -s 'sort_by(-.duration_s) | limit(10; .[])' <FILE> +``` + +#### Find and pretty print all requests related to a project + +```shell +grep <PROJECT_NAME> <FILE> | jq . +``` + +#### Find all requests with a total duration > 5 seconds + +```shell +jq 'select(.duration_s > 5000)' <FILE> +``` + +#### Find all project requests with more than 5 rugged calls + +```shell +grep <PROJECT_NAME> <FILE> | jq 'select(.rugged_calls > 5)' +``` + +#### Find all requests with a Gitaly duration > 10 seconds + +```shell +jq 'select(.gitaly_duration_s > 10000)' <FILE> +``` + +#### Find all requests with a queue duration > 10 seconds + +```shell +jq 'select(.queue_duration_s > 10000)' <FILE> +``` + +#### Top 10 requests by # of Gitaly calls + +```shell +jq -s 'map(select(.gitaly_calls != null)) | sort_by(-.gitaly_calls) | limit(10; .[])' <FILE> +``` + +### Parsing `gitlab-rails/production_json.log` + +#### Print the top three controller methods by request volume and their three longest durations + +```shell +jq -s -r 'group_by(.controller+.action) | sort_by(-length) | limit(3; .[]) | sort_by(-.duration_s) | "CT: \(length)\tMETHOD: \(.[0].controller)#\(.[0].action)\tDURS: \(.[0].duration_s), \(.[1].duration_s), \(.[2].duration_s)"' production_json.log +``` + +**Example output** + +```plaintext +CT: 2721 METHOD: SessionsController#new DURS: 844.06, 713.81, 704.66 +CT: 2435 METHOD: MetricsController#index DURS: 299.29, 284.01, 158.57 +CT: 1328 METHOD: Projects::NotesController#index DURS: 403.99, 386.29, 384.39 +``` + +### Parsing `gitlab-rails/api_json.log` + +#### Print top three routes with request count and their three longest durations + +```shell +jq -s -r 'group_by(.route) | sort_by(-length) | limit(3; .[]) | sort_by(-.duration_s) | "CT: \(length)\tROUTE: \(.[0].route)\tDURS: \(.[0].duration_s), \(.[1].duration_s), \(.[2].duration_s)"' api_json.log +``` + +**Example output** + +```plaintext +CT: 2472 ROUTE: /api/:version/internal/allowed DURS: 56402.65, 38411.43, 19500.41 +CT: 297 ROUTE: /api/:version/projects/:id/repository/tags DURS: 731.39, 685.57, 480.86 +CT: 190 ROUTE: /api/:version/projects/:id/repository/commits DURS: 1079.02, 979.68, 958.21 +``` + +### Print top API user agents + +```shell +jq --raw-output '[.route, .ua] | @tsv' api_json.log | sort | uniq -c | sort -n +``` + +**Example output**: + +```plaintext + 89 /api/:version/usage_data/increment_unique_users # plus browser details + 567 /api/:version/jobs/:id/trace gitlab-runner # plus version details +1234 /api/:version/internal/allowed GitLab-Shell +``` + +This sample response seems normal. A custom tool or script might be causing a high load +if the output contains many: + +- Third party libraries like `python-requests` or `curl`. +- [GitLab CLI clients](https://about.gitlab.com/partners/technology-partners/#cli-clients). + +You can also [use `fast-stats top`](#parsing-gitlab-logs-with-jq) to extract performance statistics. + +### Parsing `gitlab-workhorse/current` + +### Print top Workhorse user agents + +```shell +jq --raw-output '[.uri, .user_agent] | @tsv' current | sort | uniq -c | sort -n +``` + +**Example output**: + +```plaintext + 89 /api/graphql # plus browser details + 567 /api/v4/internal/allowed GitLab-Shell +1234 /api/v4/jobs/request gitlab-runner # plus version details +``` + +Similar to the [API `ua` data](#print-top-api-user-agents), +deviations from this common order might indicate scripts that could be optimized. + +The performance impact of runners checking for new jobs can be reduced by increasing +[the `check_interval` setting](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section), +for example. + +### Parsing `gitlab-rails/geo.log` + +#### Find most common Geo sync errors + +If [the `geo:status` Rake task](../geo/replication/troubleshooting.md#sync-status-rake-task) +repeatedly reports that some items never reach 100%, +the following command helps to focus on the most common errors. + +```shell +jq --raw-output 'select(.severity == "ERROR") | [.project_path, .message] | @tsv' geo.log | sort | uniq -c | sort | tail +``` + +### Parsing `gitaly/current` + +Use the following examples to [troubleshoot Gitaly](../gitaly/troubleshooting.md). + +#### Find all Gitaly requests sent from web UI + +```shell +jq 'select(."grpc.meta.client_name" == "gitlab-web")' current +``` + +#### Find all failed Gitaly requests + +```shell +jq 'select(."grpc.code" != null and ."grpc.code" != "OK")' current +``` + +#### Find all requests that took longer than 30 seconds + +```shell +jq 'select(."grpc.time_ms" > 30000)' current +``` + +#### Print top ten projects by request volume and their three longest durations + +```shell +jq --raw-output --slurp ' + map( + select( + ."grpc.request.glProjectPath" != null + and ."grpc.request.glProjectPath" != "" + and ."grpc.time_ms" != null + ) + ) + | group_by(."grpc.request.glProjectPath") + | sort_by(-length) + | limit(10; .[]) + | sort_by(-."grpc.time_ms") + | [ + length, + .[0]."grpc.time_ms", + .[1]."grpc.time_ms", + .[2]."grpc.time_ms", + .[0]."grpc.request.glProjectPath" + ] + | @sh' current \ +| awk 'BEGIN { printf "%7s %10s %10s %10s\t%s\n", "CT", "MAX DURS", "", "", "PROJECT" } + { printf "%7u %7u ms, %7u ms, %7u ms\t%s\n", $1, $2, $3, $4, $5 }' +``` + +**Example output** + +```plaintext + CT MAX DURS PROJECT + 206 4898 ms, 1101 ms, 1032 ms 'groupD/project4' + 109 1420 ms, 962 ms, 875 ms 'groupEF/project56' + 663 106 ms, 96 ms, 94 ms 'groupABC/project123' + ... +``` + +#### Find all projects affected by a fatal Git problem + +```shell +grep "fatal: " current | \ + jq '."grpc.request.glProjectPath"' | \ + sort | uniq +``` + +### Parsing `gitlab-shell/gitlab-shell.log` + +For investigating Git calls via SSH, from [GitLab 12.10](https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/367). + +Find the top 20 calls by project and user: + +```shell +jq --raw-output --slurp ' + map( + select( + .username != null and + .gl_project_path !=null + ) + ) + | group_by(.username+.gl_project_path) + | sort_by(-length) + | limit(20; .[]) + | "count: \(length)\tuser: \(.[0].username)\tproject: \(.[0].gl_project_path)" ' \ + gitlab-shell.log +``` + +Find the top 20 calls by project, user, and command: + +```shell +jq --raw-output --slurp ' + map( + select( + .command != null and + .username != null and + .gl_project_path !=null + ) + ) + | group_by(.username+.gl_project_path+.command) + | sort_by(-length) + | limit(20; .[]) + | "count: \(length)\tcommand: \(.[0].command)\tuser: \(.[0].username)\tproject: \(.[0].gl_project_path)" ' \ + gitlab-shell.log +``` diff --git a/doc/administration/logs/tracing_correlation_id.md b/doc/administration/logs/tracing_correlation_id.md new file mode 100644 index 00000000000..1d3e4955d3a --- /dev/null +++ b/doc/administration/logs/tracing_correlation_id.md @@ -0,0 +1,202 @@ +--- +stage: Systems +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Find relevant log entries with a correlation ID **(FREE SELF)** + +GitLab instances log a unique request tracking ID (known as the +"correlation ID") for most requests. Each individual request to GitLab gets +its own correlation ID, which then gets logged in each GitLab component's logs for that +request. This makes it easier to trace behavior in a +distributed system. Without this ID it can be difficult or +impossible to match correlating log entries. + +## Identify the correlation ID for a request + +The correlation ID is logged in structured logs under the key `correlation_id` +and in all response headers GitLab sends under the header `x-request-id`. +You can find your correlation ID by searching in either place. + +### Getting the correlation ID in your browser + +You can use your browser's developer tools to monitor and inspect network +activity with the site that you're visiting. See the links below for network monitoring +documentation for some popular browsers. + +- [Network Monitor - Firefox Developer Tools](https://developer.mozilla.org/en-US/docs/Tools/Network_Monitor) +- [Inspect Network Activity In Chrome DevTools](https://developer.chrome.com/docs/devtools/network/) +- [Safari Web Development Tools](https://developer.apple.com/safari/tools/) +- [Microsoft Edge Network panel](https://docs.microsoft.com/en-us/microsoft-edge/devtools-guide-chromium/network/) + +To locate a relevant request and view its correlation ID: + +1. Enable persistent logging in your network monitor. Some actions in GitLab redirect you quickly after you submit a form, so this helps capture all relevant activity. +1. To help isolate the requests you are looking for, you can filter for `document` requests. +1. Select the request of interest to view further detail. +1. Go to the **Headers** section and look for **Response Headers**. There you should find an `x-request-id` header with a +value that was randomly generated by GitLab for the request. + +See the following example: + + + +### Getting the correlation ID from your logs + +Another approach to finding the correct correlation ID is to search or watch +your logs and find the `correlation_id` value for the log entry that you're +watching for. + +For example, let's say that you want learn what's happening or breaking when +you reproduce an action in GitLab. You could tail the GitLab logs, filtering +to requests by your user, and then watch the requests until you see what you're +interested in. + +### Getting the correlation ID from curl + +If you're using `curl` then you can use the verbose option to show request and response headers, as well as other debug information. + +```shell +➜ ~ curl --verbose "https://gitlab.example.com/api/v4/projects" +# look for a line that looks like this +< x-request-id: 4rAMkV3gof4 +``` + +#### Using jq + +This example uses [jq](https://stedolan.github.io/jq/) to filter results and +display values we most likely care about. + +```shell +sudo gitlab-ctl tail gitlab-rails/production_json.log | jq 'select(.username == "bob") | "User: \(.username), \(.method) \(.path), \(.controller)#\(.action), ID: \(.correlation_id)"' +``` + +```plaintext +"User: bob, GET /root/linux, ProjectsController#show, ID: U7k7fh6NpW3" +"User: bob, GET /root/linux/commits/master/signatures, Projects::CommitsController#signatures, ID: XPIHpctzEg1" +"User: bob, GET /root/linux/blob/master/README, Projects::BlobController#show, ID: LOt9hgi1TV4" +``` + +#### Using grep + +This example uses only `grep` and `tr`, which are more likely to be installed than `jq`. + +```shell +sudo gitlab-ctl tail gitlab-rails/production_json.log | grep '"username":"bob"' | tr ',' '\n' | egrep 'method|path|correlation_id' +``` + +```plaintext +{"method":"GET" +"path":"/root/linux" +"username":"bob" +"correlation_id":"U7k7fh6NpW3"} +{"method":"GET" +"path":"/root/linux/commits/master/signatures" +"username":"bob" +"correlation_id":"XPIHpctzEg1"} +{"method":"GET" +"path":"/root/linux/blob/master/README" +"username":"bob" +"correlation_id":"LOt9hgi1TV4"} +``` + +## Searching your logs for the correlation ID + +Once you have the correlation ID you can start searching for relevant log +entries. You can filter the lines by the correlation ID itself. +Combining a `find` and `grep` should be sufficient to find the entries you are looking for. + +```shell +# find <gitlab log directory> -type f -mtime -0 exec grep '<correlation ID>' '{}' '+' +find /var/log/gitlab -type f -mtime 0 -exec grep 'LOt9hgi1TV4' '{}' '+' +``` + +```plaintext +/var/log/gitlab/gitlab-workhorse/current:{"correlation_id":"LOt9hgi1TV4","duration_ms":2478,"host":"gitlab.domain.tld","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://gitlab.domain.tld/root/linux","remote_addr":"68.0.116.160:0","remote_ip":"[filtered]","status":200,"system":"http","time":"2019-09-17T22:17:19Z","uri":"/root/linux/blob/master/README?format=json\u0026viewer=rich","user_agent":"Mozilla/5.0 (Mac) Gecko Firefox/69.0","written_bytes":1743} +/var/log/gitlab/gitaly/current:{"correlation_id":"LOt9hgi1TV4","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"FindCommits","grpc.request.deadline":"2019-09-17T22:17:47Z","grpc.request.fullMethod":"/gitaly.CommitService/FindCommits","grpc.request.glProjectPath":"root/linux","grpc.request.glRepository":"project-1","grpc.request.repoPath":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"@hashed","grpc.service":"gitaly.CommitService","grpc.start_time":"2019-09-17T22:17:17Z","grpc.time_ms":2319.161,"level":"info","msg":"finished streaming call with code OK","peer.address":"@","span.kind":"server","system":"grpc","time":"2019-09-17T22:17:19Z"} +/var/log/gitlab/gitlab-rails/production_json.log:{"method":"GET","path":"/root/linux/blob/master/README","format":"json","controller":"Projects::BlobController","action":"show","status":200,"duration":2448.77,"view":0.49,"db":21.63,"time":"2019-09-17T22:17:19.800Z","params":[{"key":"viewer","value":"rich"},{"key":"namespace_id","value":"root"},{"key":"project_id","value":"linux"},{"key":"id","value":"master/README"}],"remote_ip":"[filtered]","user_id":2,"username":"bob","ua":"Mozilla/5.0 (Mac) Gecko Firefox/69.0","queue_duration":3.38,"gitaly_calls":1,"gitaly_duration":0.77,"rugged_calls":4,"rugged_duration_ms":28.74,"correlation_id":"LOt9hgi1TV4"} +``` + +### Searching in distributed architectures + +If you have done some horizontal scaling in your GitLab infrastructure, then +you must search across _all_ of your GitLab nodes. You can do this with +some sort of log aggregation software like Loki, ELK, Splunk, or others. + +You can use a tool like Ansible or PSSH (parallel SSH) that can execute identical commands across your servers in +parallel, or craft your own solution. + +### Viewing the request in the Performance Bar + +You can use the [performance bar](../monitoring/performance/performance_bar.md) to view interesting data including calls made to SQL and Gitaly. + +To view the data, the correlation ID of the request must match the same session as the user +viewing the performance bar. For API requests, this means that you must perform the request +using the session cookie of the signed-in user. + +For example, if you want to view the database queries executed for the following API endpoint: + +```shell +https://gitlab.com/api/v4/groups/2564205/projects?with_security_reports=true&page=1&per_page=1 +``` + +First, enable the **Developer Tools** panel. See [Getting the correlation ID in your browser](#getting-the-correlation-id-in-your-browser) for details on how to do this. + +After developer tools have been enabled, obtain a session cookie as follows: + +1. Visit <https://gitlab.com> while logged in. +1. Optional. Select **Fetch/XHR** request filter in the **Developer Tools** panel. This step is described for Google Chrome developer tools and is not strictly necessary, it just makes it easier to find the correct request. +1. Select the `results?request_id=<some-request-id>` request on the left hand side. +1. The session cookie is displayed under the `Request Headers` section of the `Headers` panel. Right-click on the cookie value and select `Copy value`. + + + +You have the value of the session cookie copied to your clipboard, for example: + +```shell +experimentation_subject_id=<subject-id>; _gitlab_session=<session-id>; event_filter=all; visitor_id=<visitor-id>; perf_bar_enabled=true; sidebar_collapsed=true; diff_view=inline; sast_entry_point_dismissed=true; auto_devops_settings_dismissed=true; cf_clearance=<cf-clearance>; collapsed_gutter=false; frequently_used_emojis=clap,thumbsup,rofl,tada,eyes,bow +``` + +Use the value of the session cookie to craft an API request by pasting it into a custom header of a `curl` request: + +```shell +$ curl --include "https://gitlab.com/api/v4/groups/2564205/projects?with_security_reports=true&page=1&per_page=1" \ +--header 'cookie: experimentation_subject_id=<subject-id>; _gitlab_session=<session-id>; event_filter=all; visitor_id=<visitor-id>; perf_bar_enabled=true; sidebar_collapsed=true; diff_view=inline; sast_entry_point_dismissed=true; auto_devops_settings_dismissed=true; cf_clearance=<cf-clearance>; collapsed_gutter=false; frequently_used_emojis=clap,thumbsup,rofl,tada,eyes,bow' + + date: Tue, 28 Sep 2021 03:55:33 GMT + content-type: application/json + ... + x-request-id: 01FGN8P881GF2E5J91JYA338Y3 + ... + [ + { + "id":27497069, + "description":"Analyzer for images used on live K8S containers based on Starboard" + }, + "container_registry_image_prefix":"registry.gitlab.com/gitlab-org/security-products/analyzers/cluster-image-scanning", + "..." + ] +``` + +The response contains the data from the API endpoint, and a `correlation_id` value, returned in the `x-request-id` header, as described in the [Identify the correlation ID for a request](#identify-the-correlation-id-for-a-request) section. + +You can then view the database details for this request: + +1. Paste the `x-request-id` value into the `request details` field of the [performance bar](../monitoring/performance/performance_bar.md) and press <kbd>Enter/Return</kbd>. This example uses the `x-request-id` value `01FGN8P881GF2E5J91JYA338Y3`, returned by the above response: + +  + +1. A new request is inserted into the `Request Selector` dropdown on the right-hand side of the Performance Bar. Select the new request to view the metrics of the API request: + +  + + <!-- vale gitlab.Substitutions = NO --> +1. Select the `pg` link in the Progress Bar to view the database queries executed by the API request: + +  + <!-- vale gitlab.Substitutions = YES --> + + The database query dialog is displayed: + +  diff --git a/doc/administration/maintenance_mode/index.md b/doc/administration/maintenance_mode/index.md index 9128f978dd5..00e28e650ea 100644 --- a/doc/administration/maintenance_mode/index.md +++ b/doc/administration/maintenance_mode/index.md @@ -135,6 +135,8 @@ For most JSON requests, `POST`, `PUT`, `PATCH`, and `DELETE` are blocked, and th even if they finish running on the GitLab Runner. - Jobs in the `running` state for longer than the project's time limit do not time out. - Pipelines cannot be started, retried or canceled. No new jobs can be created either. +- The status of the runners in `/admin/runners` won't be updated. +- `gitlab-runner verify` will return the error `ERROR: Verifying runner... is removed`. After Maintenance Mode is disabled, new jobs are picked up again. Jobs that were in the `running` state before enabling Maintenance Mode resume and their logs start diff --git a/doc/administration/monitoring/ip_allowlist.md b/doc/administration/monitoring/ip_allowlist.md index adf9516733a..3151b696182 100644 --- a/doc/administration/monitoring/ip_allowlist.md +++ b/doc/administration/monitoring/ip_allowlist.md @@ -1,6 +1,6 @@ --- stage: Data Stores -group: Memory +group: Application Performance info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index d6575766b17..a2def8a9f64 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -111,6 +111,7 @@ The following metrics are available: | `failed_login_captcha_total` | Gauge | 11.0 | Counter of failed CAPTCHA attempts during login | | | `successful_login_captcha_total` | Gauge | 11.0 | Counter of successful CAPTCHA attempts during login | | | `auto_devops_pipelines_completed_total` | Counter | 12.7 | Counter of completed Auto DevOps pipelines, labeled by status | | +| `artifact_report_<report_type>_builds_completed_total` | Counter | 15.3 | Counter of completed CI Builds with report-type artifacts, grouped by report type and labeled by status | | | `gitlab_metrics_dashboard_processing_time_ms` | Summary | 12.10 | Metrics dashboard processing time in milliseconds | service, stages | | `action_cable_active_connections` | Gauge | 13.4 | Number of ActionCable WS clients currently connected | `server_mode` | | `action_cable_broadcasts_total` | Counter | 13.10 | The number of ActionCable broadcasts emitted | `server_mode` | @@ -184,6 +185,7 @@ configuration option in `gitlab.yml`. These metrics are served from the | `sidekiq_elasticsearch_requests_total` | Counter | 13.1 | Elasticsearch requests during a Sidekiq job execution | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency` | | `sidekiq_running_jobs` | Gauge | 12.2 | Number of Sidekiq jobs running | `queue`, `boundary`, `external_dependencies`, `feature_category`, `urgency` | | `sidekiq_concurrency` | Gauge | 12.5 | Maximum number of Sidekiq jobs | | +| `sidekiq_mem_total_bytes` | Gauge | 15.3 | Number of bytes allocated for both objects consuming an object slot and objects that required a malloc'| | | `geo_db_replication_lag_seconds` | Gauge | 10.2 | Database replication lag (seconds) | `url` | | `geo_repositories` | Gauge | 10.2 | Total number of repositories available on primary | `url` | | `geo_repositories_synced` | Gauge | 10.2 | Number of repositories synced on secondary | `url` | @@ -300,6 +302,16 @@ configuration option in `gitlab.yml`. These metrics are served from the | `geo_uploads_verification_failed` | Gauge | 14.6 | Number of uploads verifications failed on secondary | `url` | | `gitlab_sli:rails_request_apdex:total` | Counter | 14.4 | The number of request-apdex measurements, [more information the development documentation](../../../development/application_slis/rails_request_apdex.md) | `endpoint_id`, `feature_category`, `request_urgency` | | `gitlab_sli:rails_request_apdex:success_total` | Counter | 14.4 | The number of successful requests that met the target duration for their urgency. Divide by `gitlab_sli:rails_requests_apdex:total` to get a success ratio | `endpoint_id`, `feature_category`, `request_urgency` | +| `geo_ci_secure_files` | Gauge | 15.3 | Number of secure files on primary | `url` | +| `geo_ci_secure_files_checksum_total` | Gauge | 15.3 | Number of secure files tried to checksum on primary | `url` | +| `geo_ci_secure_files_checksummed` | Gauge | 15.3 | Number of secure files successfully checksummed on primary | `url` | +| `geo_ci_secure_files_checksum_failed` | Gauge | 15.3 | Number of secure files failed to calculate the checksum on primary | `url` | +| `geo_ci_secure_files_synced` | Gauge | 15.3 | Number of syncable secure files synced on secondary | `url` | +| `geo_ci_secure_files_failed` | Gauge | 15.3 | Number of syncable secure files failed to sync on secondary | `url` | +| `geo_ci_secure_files_registry` | Gauge | 15.3 | Number of secure files in the registry | `url` | +| `geo_ci_secure_files_verification_total` | Gauge | 15.3 | Number of secure files verifications tried on secondary | `url` | +| `geo_ci_secure_files_verified` | Gauge | 15.3 | Number of secure files verified on secondary | `url` | +| `geo_ci_secure_files_verification_failed` | Gauge | 15.3 | Number of secure files verifications failed on secondary | `url` | ## Database load balancing metrics **(PREMIUM SELF)** @@ -378,8 +390,8 @@ Some basic Ruby runtime metrics are available: ## Redis metrics These client metrics are meant to complement Redis server metrics. -These metrics are broken down per [Redis -instance](https://docs.gitlab.com/omnibus/settings/redis.html#running-with-multiple-redis-instances). +These metrics are broken down per +[Redis instance](https://docs.gitlab.com/omnibus/settings/redis.html#running-with-multiple-redis-instances). These metrics all have a `storage` label which indicates the Redis instance (`cache`, `shared_state`, and so on). diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index 51360800d66..6f6ac5c5d4b 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -465,3 +465,17 @@ If you are using Prometheus monitoring: ``` 1. Optional. [Configure the storage retention size](index.md#configure-the-storage-retention-size). + +### Monitoring node not receiving data + +If the monitoring node is not receiving any data, check that the exporters are capturing data: + +```shell +curl "http[s]://localhost:<EXPORTER LISTENING PORT>/metrics" +``` + +or + +```shell +curl "http[s]://localhost:<EXPORTER LISTENING PORT>/-/metrics" +``` diff --git a/doc/administration/monitoring/prometheus/web_exporter.md b/doc/administration/monitoring/prometheus/web_exporter.md index fe140b15ed2..45a8e8d5640 100644 --- a/doc/administration/monitoring/prometheus/web_exporter.md +++ b/doc/administration/monitoring/prometheus/web_exporter.md @@ -1,6 +1,6 @@ --- stage: Data Stores -group: Memory +group: Application Performance info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -14,7 +14,7 @@ Prometheus scraper understands. NOTE: This page is about web application metrics. -To export background job metrics, learn how to [configure the Sidekiq metrics server](../../sidekiq.md#configure-the-sidekiq-metrics-server). +To export background job metrics, learn how to [configure the Sidekiq metrics server](../../sidekiq/index.md#configure-the-sidekiq-metrics-server). We provide two mechanisms by which web application metrics can be exported: diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md index 51fa627b8d4..29f00fec3f7 100644 --- a/doc/administration/nfs.md +++ b/doc/administration/nfs.md @@ -65,8 +65,7 @@ Read: ## Known kernel version incompatibilities RedHat Enterprise Linux (RHEL) and CentOS v7.7 and v7.8 ship with kernel -version `3.10.0-1127`, which [contains a -bug](https://bugzilla.redhat.com/show_bug.cgi?id=1783554) that causes +version `3.10.0-1127`, which [contains a bug](https://bugzilla.redhat.com/show_bug.cgi?id=1783554) that causes [uploads to fail to copy over NFS](https://gitlab.com/gitlab-org/gitlab/-/issues/218999). The following GitLab versions include a fix to work properly with that kernel version: @@ -99,9 +98,20 @@ NFS performance with GitLab can in some cases be improved with [direct Git access](gitaly/index.md#direct-access-to-git-in-gitlab) using [Rugged](https://github.com/libgit2/rugged). -From GitLab 12.1, GitLab automatically detects if Rugged can and should be used per storage. -If you previously enabled Rugged using the feature flag and you want to use automatic detection instead, -you must unset the feature flag: +Versions of GitLab after 12.2 and prior to 15.3 automatically detect if +Rugged can and should be used per storage. + +NOTE: +GitLab 15.3 and later disables this automatic detection. Auto-detection can be enabled via the +`skip_rugged_auto_detect` feature flag: + +```ruby +Feature.disable(:skip_rugged_auto_detect) +``` + +In addition, if you previously enabled Rugged using the feature flag and +you want to use automatic detection instead, you must unset the feature +flag: ```shell sudo gitlab-rake gitlab:features:unset_rugged @@ -485,7 +495,7 @@ sudo perf trace --no-syscalls --event 'nfs4:*' -p $(pgrep -fd ',' puma) ### Warnings `garbage found: .../repositories/@hashed/...git/objects/pack/.nfs...` in Gitaly logs -If you find any warnings like `garbage found: .../repositories/@hashed/...git/objects/pack/.nfs...` in [Gitaly logs](logs.md#gitaly-logs), +If you find any warnings like `garbage found: .../repositories/@hashed/...git/objects/pack/.nfs...` in [Gitaly logs](logs/index.md#gitaly-logs), this problem occurs if `lookupcache=positive` is not set, which we recommend as an [NFS mount option](#mount-options). See [Gitaly issue #3175](https://gitlab.com/gitlab-org/gitaly/-/issues/3175) for more details. diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index ddeaf0280eb..0299d5f8b0c 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -26,8 +26,8 @@ GitLab has been tested by vendors and customers on a number of object storage pr ### Known compatibility issues -- Dell EMC ECS: Prior to GitLab 13.3, there is a [known bug in GitLab Workhorse that prevents - HTTP Range Requests from working with CI job artifacts](https://gitlab.com/gitlab-org/gitlab/-/issues/223806). +- Dell EMC ECS: Prior to GitLab 13.3, there is a + [known bug in GitLab Workhorse that prevents HTTP Range Requests from working with CI job artifacts](https://gitlab.com/gitlab-org/gitlab/-/issues/223806). Be sure to upgrade to GitLab 13.3.0 or above if you use S3 storage with this hardware. - Ceph S3 prior to [Kraken 11.0.2](https://ceph.com/releases/kraken-11-0-2-released/) does not support the [Upload Copy Part API](https://gitlab.com/gitlab-org/gitlab/-/issues/300604). You may need to [disable multi-threaded copying](#multi-threaded-copying). @@ -578,9 +578,8 @@ real bucket into multiple virtual buckets. If your object storage bucket is called `my-gitlab-objects` you can configure uploads to go into `my-gitlab-objects/uploads`, artifacts into `my-gitlab-objects/artifacts`, etc. The application will act as if -these are separate buckets. Note that use of bucket prefixes [may not -work correctly with Helm -backups](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/3376). +these are separate buckets. Note that use of bucket prefixes +[may not work correctly with Helm backups](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/3376). Helm-based installs require separate buckets to [handle backup restorations](https://docs.gitlab.com/charts/advanced/external-object-storage/#lfs-artifacts-uploads-packages-external-diffs-terraform-state-dependency-proxy). @@ -693,18 +692,17 @@ configuration. When configured either with an instance profile or with the consolidated object configuration, GitLab Workhorse properly uploads files to S3 -buckets that have [SSE-S3 or SSE-KMS encryption enabled by -default](https://docs.aws.amazon.com/kms/latest/developerguide/services-s3.html). -Customer master keys (CMKs) and SSE-C encryption are [not -supported since this requires sending the encryption keys in every request](https://gitlab.com/gitlab-org/gitlab/-/issues/226006). +buckets that have [SSE-S3 or SSE-KMS encryption enabled by default](https://docs.aws.amazon.com/kms/latest/developerguide/services-s3.html). +Customer master keys (CMKs) and SSE-C encryption are +[not supported since this requires sending the encryption keys in every request](https://gitlab.com/gitlab-org/gitlab/-/issues/226006). ##### Server-side encryption headers > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38240) in GitLab 13.3. Setting a default encryption on an S3 bucket is the easiest way to -enable encryption, but you may want to [set a bucket policy to ensure -only encrypted objects are uploaded](https://aws.amazon.com/premiumsupport/knowledge-center/s3-bucket-store-kms-encrypted-objects/). +enable encryption, but you may want to +[set a bucket policy to ensure only encrypted objects are uploaded](https://aws.amazon.com/premiumsupport/knowledge-center/s3-bucket-store-kms-encrypted-objects/). To do this, you must configure GitLab to send the proper encryption headers in the `storage_options` configuration section: diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md index 373017eefa7..58858c54843 100644 --- a/doc/administration/operations/extra_sidekiq_processes.md +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -1,362 +1,11 @@ --- -stage: Systems -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../sidekiq/extra_sidekiq_processes.md' +remove_date: '2022-11-11' --- -# Run multiple Sidekiq processes **(FREE SELF)** +This document was moved to [another location](../sidekiq/extra_sidekiq_processes.md). -GitLab allows you to start multiple Sidekiq processes. -These processes can be used to consume a dedicated set -of queues. This can be used to ensure certain queues always have dedicated -workers, no matter the number of jobs to be processed. - -NOTE: -The information in this page applies only to Omnibus GitLab. - -## Available Sidekiq queues - -For a list of the existing Sidekiq queues, check the following files: - -- [Queues for both GitLab Community and Enterprise Editions](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/all_queues.yml) -- [Queues for GitLab Enterprise Editions only](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/all_queues.yml) - -Each entry in the above files represents a queue on which Sidekiq processes -can be started. - -## Start multiple processes - -> - [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4006) in GitLab 12.10, starting multiple processes with Sidekiq cluster. -> - [Sidekiq cluster moved](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181) to GitLab Free in 12.10. -> - [Sidekiq cluster became default](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4140) in GitLab 13.0. - -When starting multiple processes, the number of processes should -equal (and **not** exceed) the number of CPU cores you want to -dedicate to Sidekiq. Each Sidekiq process can use only 1 CPU -core, subject to the available workload and concurrency settings. - -To start multiple processes: - -1. Using the `sidekiq['queue_groups']` array setting, specify how many processes to - create using `sidekiq-cluster` and which queue they should handle. - Each item in the array equates to one additional Sidekiq - process, and values in each item determine the queues it works on. - - For example, the following setting creates three Sidekiq processes, one to run on - `elastic_commit_indexer`, one to run on `mailers`, and one process running on all queues: - - ```ruby - sidekiq['queue_groups'] = [ - "elastic_commit_indexer", - "mailers", - "*" - ] - ``` - - To have an additional Sidekiq process handle multiple queues, add multiple - queue names to its item delimited by commas. For example: - - ```ruby - sidekiq['queue_groups'] = [ - "elastic_commit_indexer, elastic_association_indexer", - "mailers", - "*" - ] - ``` - - [In GitLab 12.9](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26594) and - later, the special queue name `*` means all queues. This starts two - processes, each handling all queues: - - ```ruby - sidekiq['queue_groups'] = [ - "*", - "*" - ] - ``` - - `*` cannot be combined with concrete queue names - `*, mailers` - just handles the `mailers` queue. - - When `sidekiq-cluster` is only running on a single node, make sure that at least - one process is running on all queues using `*`. This ensures a process - automatically picks up jobs in queues created in the future, - including queues that have dedicated processes. - - If `sidekiq-cluster` is running on more than one node, you can also use - [`--negate`](#negate-settings) and list all the queues that are already being - processed. - -1. Save the file and reconfigure GitLab for the changes to take effect: - - ```shell - sudo gitlab-ctl reconfigure - ``` - -To view the Sidekiq processes in GitLab: - -1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **Monitoring > Background Jobs**. - -## Negate settings - -To have the Sidekiq process work on every queue **except** the ones -you list. In this example, we exclude all import-related jobs from a Sidekiq node: - -1. Edit `/etc/gitlab/gitlab.rb` and add: - - ```ruby - sidekiq['negate'] = true - sidekiq['queue_selector'] = true - sidekiq['queue_groups'] = [ - "feature_category=importers" - ] - ``` - -1. Save the file and reconfigure GitLab for the changes to take effect: - - ```shell - sudo gitlab-ctl reconfigure - ``` - -## Queue selector - -> - [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/45) in GitLab 12.8. -> - [Sidekiq cluster, including queue selector, moved](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181) to GitLab Free in 12.10. -> - [Renamed from `experimental_queue_selector` to `queue_selector`](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/147) in GitLab 13.6. - -In addition to selecting queues by name, as above, the `queue_selector` option -allows queue groups to be selected in a more general way using a [worker matching -query](extra_sidekiq_routing.md#worker-matching-query). After `queue_selector` -is set, all `queue_groups` must follow the aforementioned syntax. - -In `/etc/gitlab/gitlab.rb`: - -```ruby -sidekiq['enable'] = true -sidekiq['queue_selector'] = true -sidekiq['queue_groups'] = [ - # Run all non-CPU-bound queues that are high urgency - 'resource_boundary!=cpu&urgency=high', - # Run all continuous integration and pages queues that are not high urgency - 'feature_category=continuous_integration,pages&urgency!=high', - # Run all queues - '*' -] -``` - -## Ignore all import queues - -When [importing from GitHub](../../user/project/import/github.md) or -other sources, Sidekiq might use all of its resources to perform those -operations. To set up two separate `sidekiq-cluster` processes, where -one only processes imports and the other processes all other queues: - -1. Edit `/etc/gitlab/gitlab.rb` and add: - - ```ruby - sidekiq['enable'] = true - sidekiq['queue_selector'] = true - sidekiq['queue_groups'] = [ - "feature_category=importers", - "feature_category!=importers" - ] - ``` - -1. Save the file and reconfigure GitLab for the changes to take effect: - - ```shell - sudo gitlab-ctl reconfigure - ``` - -## Number of threads - -By default each process defined under `sidekiq` starts with a -number of threads that equals the number of queues, plus one spare thread. -For example, a process that handles the `process_commit` and `post_receive` -queues uses three threads in total. - -These thread run inside a single Ruby process, and each process -can only use a single CPU core. The usefulness of threading depends -on the work having some external dependencies to wait on, like database queries or -HTTP requests. Most Sidekiq deployments benefit from this threading, and when -running fewer queues in a process, increasing the thread count might be -even more desirable to make the most effective use of CPU resources. - -### Manage thread counts explicitly - -The correct maximum thread count (also called concurrency) depends on the workload. -Typical values range from `1` for highly CPU-bound tasks to `15` or higher for mixed -low-priority work. A reasonable starting range is `15` to `25` for a non-specialized -deployment. - -You can find example values used by GitLab.com by searching for `concurrency:` in -[the Helm charts](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/blob/master/releases/gitlab/values/gprd.yaml.gotmpl). -The values vary according to the work each specific deployment of Sidekiq does. -Any other specialized deployments with processes dedicated to specific queues should -have the concurrency tuned according to: -have the concurrency tuned according to: - -- The CPU usage of each type of process. -- The throughput achieved. - -Each thread requires a Redis connection, so adding threads may increase Redis -latency and potentially cause client timeouts. See the [Sidekiq documentation -about Redis](https://github.com/mperham/sidekiq/wiki/Using-Redis) for more -details. - -#### When running Sidekiq cluster (default) - -Running Sidekiq cluster is the default in GitLab 13.0 and later. - -1. Edit `/etc/gitlab/gitlab.rb` and add: - - ```ruby - sidekiq['min_concurrency'] = 15 - sidekiq['max_concurrency'] = 25 - ``` - -1. Save the file and reconfigure GitLab for the changes to take effect: - - ```shell - sudo gitlab-ctl reconfigure - ``` - -`min_concurrency` and `max_concurrency` are independent; one can be set without -the other. Setting `min_concurrency` to `0` disables the limit. - -For each queue group, let `N` be one more than the number of queues. The -concurrency is set to: - -1. `N`, if it's between `min_concurrency` and `max_concurrency`. -1. `max_concurrency`, if `N` exceeds this value. -1. `min_concurrency`, if `N` is less than this value. - -If `min_concurrency` is equal to `max_concurrency`, then this value is used -regardless of the number of queues. - -When `min_concurrency` is greater than `max_concurrency`, it is treated as -being equal to `max_concurrency`. - -#### When running a single Sidekiq process - -Running a single Sidekiq process is the default in GitLab 12.10 and earlier. - -WARNING: -Running Sidekiq directly was removed in GitLab -[14.0](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/240). - -1. Edit `/etc/gitlab/gitlab.rb` and add: - - ```ruby - sidekiq['cluster'] = false - sidekiq['concurrency'] = 25 - ``` - -1. Save the file and reconfigure GitLab for the changes to take effect: - - ```shell - sudo gitlab-ctl reconfigure - ``` - -This sets the concurrency (number of threads) for the Sidekiq process. - -## Modify the check interval - -To modify `sidekiq-cluster`'s health check interval for the additional Sidekiq processes: - -1. Edit `/etc/gitlab/gitlab.rb` and add (the value can be any integer number of seconds): - - ```ruby - sidekiq['interval'] = 5 - ``` - -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -## Troubleshoot using the CLI - -WARNING: -It's recommended to use `/etc/gitlab/gitlab.rb` to configure the Sidekiq processes. -If you experience a problem, you should contact GitLab support. Use the command -line at your own risk. - -For debugging purposes, you can start extra Sidekiq processes by using the command -`/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster`. This command -takes arguments using the following syntax: - -```shell -/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster [QUEUE,QUEUE,...] [QUEUE, ...] -``` - -Each separate argument denotes a group of queues that have to be processed by a -Sidekiq process. Multiple queues can be processed by the same process by -separating them with a comma instead of a space. - -Instead of a queue, a queue namespace can also be provided, to have the process -automatically listen on all queues in that namespace without needing to -explicitly list all the queue names. For more information about queue namespaces, -see the relevant section in the -[Sidekiq development documentation](../../development/sidekiq/index.md#queue-namespaces). - -For example, say you want to start 2 extra processes: one to process the -`process_commit` queue, and one to process the `post_receive` queue. This can be -done as follows: - -```shell -/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster process_commit post_receive -``` - -If you instead want to start one process processing both queues, you'd use the -following syntax: - -```shell -/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster process_commit,post_receive -``` - -If you want to have one Sidekiq process dealing with the `process_commit` and -`post_receive` queues, and one process to process the `gitlab_shell` queue, -you'd use the following: - -```shell -/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster process_commit,post_receive gitlab_shell -``` - -### Monitor the `sidekiq-cluster` command - -The `sidekiq-cluster` command does not terminate once it has started the desired -amount of Sidekiq processes. Instead, the process continues running and -forwards any signals to the child processes. This allows you to stop all -Sidekiq processes as you send a signal to the `sidekiq-cluster` process, -instead of having to send it to the individual processes. - -If the `sidekiq-cluster` process crashes or receives a `SIGKILL`, the child -processes terminate themselves after a few seconds. This ensures you don't -end up with zombie Sidekiq processes. - -This allows you to monitor the processes by hooking up -`sidekiq-cluster` to your supervisor of choice (for example, runit). - -If a child process died the `sidekiq-cluster` command signals all remaining -process to terminate, then terminate itself. This removes the need for -`sidekiq-cluster` to re-implement complex process monitoring/restarting code. -Instead you should make sure your supervisor restarts the `sidekiq-cluster` -process whenever necessary. - -### PID files - -The `sidekiq-cluster` command can store its PID in a file. By default no PID -file is written, but this can be changed by passing the `--pidfile` option to -`sidekiq-cluster`. For example: - -```shell -/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster --pidfile /var/run/gitlab/sidekiq_cluster.pid process_commit -``` - -Keep in mind that the PID file contains the PID of the `sidekiq-cluster` -command and not the PIDs of the started Sidekiq processes. - -### Environment - -The Rails environment can be set by passing the `--environment` flag to the -`sidekiq-cluster` command, or by setting `RAILS_ENV` to a non-empty value. The -default value can be found in `/opt/gitlab/etc/gitlab-rails/env/RAILS_ENV`. +<!-- This redirect file can be deleted after <2022-11-11>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/operations/extra_sidekiq_routing.md b/doc/administration/operations/extra_sidekiq_routing.md index a6ad3e62bb7..072b6f63537 100644 --- a/doc/administration/operations/extra_sidekiq_routing.md +++ b/doc/administration/operations/extra_sidekiq_routing.md @@ -1,193 +1,11 @@ --- -stage: Systems -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../sidekiq/extra_sidekiq_routing.md' +remove_date: '2022-11-11' --- -# Queue routing rules **(FREE SELF)** +This document was moved to [another location](../sidekiq/extra_sidekiq_routing.md). -When the number of Sidekiq jobs increases to a certain scale, the system faces -some scalability issues. One of them is that the length of the queue tends to get -longer. High-urgency jobs have to wait longer until other less urgent jobs -finish. This head-of-line blocking situation may eventually affect the -responsiveness of the system, especially critical actions. In another scenario, -the performance of some jobs is degraded due to other long running or CPU-intensive jobs -(computing or rendering ones) in the same machine. - -To counter the aforementioned issues, one effective solution is to split -Sidekiq jobs into different queues and assign machines handling each queue -exclusively. For example, all CPU-intensive jobs could be routed to the -`cpu-bound` queue and handled by a fleet of CPU optimized instances. The queue -topology differs between companies depending on the workloads and usage -patterns. Therefore, GitLab supports a flexible mechanism for the -administrator to route the jobs based on their characteristics. - -As an alternative to [Queue selector](extra_sidekiq_processes.md#queue-selector), which -configures Sidekiq cluster to listen to a specific set of workers or queues, -GitLab also supports routing a job from a worker to the desired queue when it -is scheduled. Sidekiq clients try to match a job against a configured list of -routing rules. Rules are evaluated from first to last, and as soon as we find a -match for a given worker we stop processing for that worker (first match wins). -If the worker doesn't match any rule, it falls back to the queue name generated -from the worker name. - -By default, if the routing rules are not configured (or denoted with an empty -array), all the jobs are routed to the queue generated from the worker name. - -## Example configuration - -In `/etc/gitlab/gitlab.rb`: - -```ruby -sidekiq['routing_rules'] = [ - # Do not re-route workers that require their own queue - ['tags=needs_own_queue', nil], - # Route all non-CPU-bound workers that are high urgency to `high-urgency` queue - ['resource_boundary!=cpu&urgency=high', 'high-urgency'], - # Route all database, gitaly and global search workers that are throttled to `throttled` queue - ['feature_category=database,gitaly,global_search&urgency=throttled', 'throttled'], - # Route all workers having contact with outside work to a `network-intenstive` queue - ['has_external_dependencies=true|feature_category=hooks|tags=network', 'network-intensive'], - # Route all import workers to the queues generated by the worker name, for - # example, JiraImportWorker to `jira_import`, SVNWorker to `svn_worker` - ['feature_category=import', nil], - # Wildcard matching, route the rest to `default` queue - ['*', 'default'] -] -``` - -The routing rules list is an order-matter array of tuples of query and -corresponding queue: - -- The query is following a [worker matching query](#worker-matching-query) syntax. -- The `<queue_name>` must be a valid Sidekiq queue name. If the queue name - is `nil`, or an empty string, the worker is routed to the queue generated - by the name of the worker instead. - -The query supports wildcard matching `*`, which matches all workers. As a -result, the wildcard query must stay at the end of the list or the rules after it -are ignored. - -NOTE: -Mixing queue routing rules and queue selectors requires care to -ensure all jobs that are scheduled and picked up by appropriate Sidekiq -workers. - -## Worker matching query - -GitLab provides a query syntax to match a worker based on its -attributes. This query syntax is employed by both [Queue routing -rules](#queue-routing-rules) and [Queue -selector](extra_sidekiq_processes.md#queue-selector). A query includes two -components: - -- Attributes that can be selected. -- Operators used to construct a query. - -### Available attributes - -> [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/261) in GitLab 13.1 (`tags`). - -Queue matching query works upon the worker attributes, described in -[Sidekiq style guide](../../development/sidekiq/index.md). We support querying -based on a subset of worker attributes: - -- `feature_category` - the [GitLab feature - category](https://about.gitlab.com/direction/maturity/#category-maturity) the - queue belongs to. For example, the `merge` queue belongs to the - `source_code_management` category. -- `has_external_dependencies` - whether or not the queue connects to external - services. For example, all importers have this set to `true`. -- `urgency` - how important it is that this queue's jobs run - quickly. Can be `high`, `low`, or `throttled`. For example, the - `authorized_projects` queue is used to refresh user permissions, and - is `high` urgency. -- `worker_name` - the worker name. Use this attribute to select a specific worker. -- `name` - the queue name generated from the worker name. Use this attribute to select a specific queue. Because this is generated from - the worker name, it does not change based on the result of other routing - rules. -- `resource_boundary` - if the queue is bound by `cpu`, `memory`, or - `unknown`. For example, the `ProjectExportWorker` is memory bound as it has - to load data in memory before saving it for export. -- `tags` - short-lived annotations for queues. These are expected to frequently - change from release to release, and may be removed entirely. - -`has_external_dependencies` is a boolean attribute: only the exact -string `true` is considered true, and everything else is considered -false. - -`tags` is a set, which means that `=` checks for intersecting sets, and -`!=` checks for disjoint sets. For example, `tags=a,b` selects queues -that have tags `a`, `b`, or both. `tags!=a,b` selects queues that have -neither of those tags. - -The attributes of each worker are hard-coded in the source code. For -convenience, we generate a [list of all available attributes in -GitLab Community Edition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/all_queues.yml) -and a [list of all available attributes in -GitLab Enterprise Edition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/all_queues.yml). - -### Available operators - -`queue_selector` supports the following operators, listed from highest -to lowest precedence: - -- `|` - the logical `OR` operator. For example, `query_a|query_b` (where `query_a` - and `query_b` are queries made up of the other operators here) includes - queues that match either query. -- `&` - the logical `AND` operator. For example, `query_a&query_b` (where - `query_a` and `query_b` are queries made up of the other operators here) will - only include queues that match both queries. -- `!=` - the `NOT IN` operator. For example, `feature_category!=issue_tracking` - excludes all queues from the `issue_tracking` feature category. -- `=` - the `IN` operator. For example, `resource_boundary=cpu` includes all - queues that are CPU bound. -- `,` - the concatenate set operator. For example, - `feature_category=continuous_integration,pages` includes all queues from - either the `continuous_integration` category or the `pages` category. This - example is also possible using the OR operator, but allows greater brevity, as - well as being lower precedence. - -The operator precedence for this syntax is fixed: it's not possible to make `AND` -have higher precedence than `OR`. - -[In GitLab 12.9](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26594) and -later, as with the standard queue group syntax above, a single `*` as the -entire queue group selects all queues. - -### Migration - -After the Sidekiq routing rules are changed, administrators must take care -with the migration to avoid losing jobs entirely, especially in a system with -long queues of jobs. The migration can be done by following the migration steps -mentioned in [Sidekiq job -migration](../../raketasks/sidekiq_job_migration.md) - -### Workers that cannot be migrated - -Some workers cannot share a queue with other workers - typically because -they check the size of their own queue - and so must be excluded from -this process. We recommend excluding these from any further worker -routing by adding a rule to keep them in their own queue, for example: - -```ruby -sidekiq['routing_rules'] = [ - ['tags=needs_own_queue', nil], - # ... -] -``` - -These queues must also be included in at least one [Sidekiq -queue group](extra_sidekiq_processes.md#start-multiple-processes). - -The following table shows the workers that should have their own queue: - -| Worker name | Queue name | GitLab issue | -| --- | --- | --- | -| `EmailReceiverWorker` | `email_receiver` | [`gitlab-com/gl-infra/scalability#1263`](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1263) | -| `ServiceDeskEmailReceiverWorker` | `service_desk_email_receiver` | [`gitlab-com/gl-infra/scalability#1263`](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1263) | -| `ProjectImportScheduleWorker` | `project_import_schedule` | [`gitlab-org/gitlab#340630`](https://gitlab.com/gitlab-org/gitlab/-/issues/340630) | -| `HashedStorage::MigratorWorker` | `hashed_storage:hashed_storage_migrator` | [`gitlab-org/gitlab#340629`](https://gitlab.com/gitlab-org/gitlab/-/issues/340629) | -| `HashedStorage::ProjectMigrateWorker` | `hashed_storage:hashed_storage_project_migrate` | [`gitlab-org/gitlab#340629`](https://gitlab.com/gitlab-org/gitlab/-/issues/340629) | -| `HashedStorage::ProjectRollbackWorker` | `hashed_storage:hashed_storage_project_rollback` | [`gitlab-org/gitlab#340629`](https://gitlab.com/gitlab-org/gitlab/-/issues/340629) | -| `HashedStorage::RollbackerWorker` | `hashed_storage:hashed_storage_rollbacker` | [`gitlab-org/gitlab#340629`](https://gitlab.com/gitlab-org/gitlab/-/issues/340629) | +<!-- This redirect file can be deleted after <2022-11-11>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/operations/index.md b/doc/administration/operations/index.md index 41a9a0e6d67..a6e66abdbdb 100644 --- a/doc/administration/operations/index.md +++ b/doc/administration/operations/index.md @@ -18,10 +18,9 @@ Keep your GitLab instance up and running smoothly. - [Multiple Sidekiq processes](extra_sidekiq_processes.md): Configure multiple Sidekiq processes to ensure certain queues always have dedicated workers, no matter the number of jobs that must be processed. **(FREE SELF)** - [Sidekiq routing rules](extra_sidekiq_routing.md): Configure the routing rules to route a job from a worker to a desirable queue. **(FREE SELF)** - [Puma](puma.md): Understand Puma and puma-worker-killer. -- Speed up SSH operations by [Authorizing SSH users via a fast, - indexed lookup to the GitLab database](fast_ssh_key_lookup.md), and/or - by [doing away with user SSH keys stored on GitLab entirely in favor - of SSH certificates](ssh_certificates.md). +- Speed up SSH operations by + [Authorizing SSH users via a fast, indexed lookup to the GitLab database](fast_ssh_key_lookup.md), and/or + by [doing away with user SSH keys stored on GitLab entirely in favor of SSH certificates](ssh_certificates.md). - [File System Performance Benchmarking](filesystem_benchmarking.md): File system performance can have a big impact on GitLab performance, especially for actions that read or write Git repositories. This information helps benchmark diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md index bc95ee626ce..5ce469d3e63 100644 --- a/doc/administration/operations/puma.md +++ b/doc/administration/operations/puma.md @@ -54,7 +54,7 @@ A higher value of `1200` or more would be beneficial if the server has free memo The worker killer checks memory every 20 seconds. -To monitor the worker killer, use [the Puma log](../logs.md#puma_stdoutlog) `/var/log/gitlab/puma/puma_stdout.log`. +To monitor the worker killer, use [the Puma log](../logs/index.md#puma_stdoutlog) `/var/log/gitlab/puma/puma_stdout.log`. For example: ```plaintext @@ -246,7 +246,7 @@ automatically, due to differences between the two application servers. To switch from Unicorn to Puma: 1. Determine suitable Puma [worker and thread settings](../../install/requirements.md#puma-settings). -1. Convert any custom Unicorn settings to Puma. +1. Convert any custom Unicorn settings to Puma in `/etc/gitlab/gitlab.rb`. The table below summarizes which Unicorn configuration keys correspond to those in Puma when using the Linux package, and which ones have no corresponding counterpart. diff --git a/doc/administration/operations/rails_console.md b/doc/administration/operations/rails_console.md index 660a99faaf8..430dfbc637c 100644 --- a/doc/administration/operations/rails_console.md +++ b/doc/administration/operations/rails_console.md @@ -6,8 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Rails console **(FREE SELF)** -At the heart of GitLab is a web application [built using the Ruby on Rails -framework](https://about.gitlab.com/blog/2018/10/29/why-we-use-rails-to-build-gitlab/). +At the heart of GitLab is a web application +[built using the Ruby on Rails framework](https://about.gitlab.com/blog/2018/10/29/why-we-use-rails-to-build-gitlab/). The [Rails console](https://guides.rubyonrails.org/command_line.html#rails-console). provides a way to interact with your GitLab instance from the command line, and also grants access to the amazing tools built right into Rails. @@ -19,8 +19,8 @@ with no consequences, you are strongly advised to do so in a test environment. The Rails console is for GitLab system administrators who are troubleshooting a problem or need to retrieve some data that can only be done through direct -access of the GitLab application. Basic knowledge of Ruby is needed (try [this -30-minute tutorial](https://try.ruby-lang.org/) for a quick introduction). +access of the GitLab application. Basic knowledge of Ruby is needed (try +[this 30-minute tutorial](https://try.ruby-lang.org/) for a quick introduction). Rails experience is useful but not required. ## Starting a Rails console session @@ -136,8 +136,8 @@ root 1 ``` -Some basic knowledge of Ruby will be very useful. Try [this -30-minute tutorial](https://try.ruby-lang.org/) for a quick introduction. +Some basic knowledge of Ruby will be very useful. Try +[this 30-minute tutorial](https://try.ruby-lang.org/) for a quick introduction. Rails experience is helpful but not essential. ### Troubleshooting Rails Runner diff --git a/doc/administration/operations/sidekiq_memory_killer.md b/doc/administration/operations/sidekiq_memory_killer.md index d9558c3d7a6..b1977589fae 100644 --- a/doc/administration/operations/sidekiq_memory_killer.md +++ b/doc/administration/operations/sidekiq_memory_killer.md @@ -1,83 +1,11 @@ --- -stage: Data Stores -group: Memory -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../sidekiq/sidekiq_memory_killer.md' +remove_date: '2022-11-11' --- -# Sidekiq MemoryKiller **(FREE SELF)** +This document was moved to [another location](../sidekiq/sidekiq_memory_killer.md). -The GitLab Rails application code suffers from memory leaks. For web requests -this problem is made manageable using -[`puma-worker-killer`](https://github.com/schneems/puma_worker_killer) which -restarts Puma worker processes if it exceeds a memory limit. The Sidekiq -MemoryKiller applies the same approach to the Sidekiq processes used by GitLab -to process background jobs. - -Unlike puma-worker-killer, which is enabled by default for all GitLab -installations of GitLab 13.0 and later, the Sidekiq MemoryKiller is enabled by default -_only_ for Omnibus packages. The reason for this is that the MemoryKiller -relies on runit to restart Sidekiq after a memory-induced shutdown and GitLab -installations from source do not all use runit or an equivalent. - -With the default settings, the MemoryKiller causes a Sidekiq restart no -more often than once every 15 minutes, with the restart causing about one -minute of delay for incoming background jobs. - -Some background jobs rely on long-running external processes. To ensure these -are cleanly terminated when Sidekiq is restarted, each Sidekiq process should be -run as a process group leader (for example, using `chpst -P`). If using Omnibus or the -`bin/background_jobs` script with `runit` installed, this is handled for you. - -## Configuring the MemoryKiller - -The MemoryKiller is controlled using environment variables. - -- `SIDEKIQ_DAEMON_MEMORY_KILLER`: defaults to 1. When set to 0, the MemoryKiller - works in _legacy_ mode. Otherwise, the MemoryKiller works in _daemon_ mode. - - In _legacy_ mode, the MemoryKiller checks the Sidekiq process RSS - ([Resident Set Size](https://github.com/mperham/sidekiq/wiki/Memory#rss)) - after each job. - - In _daemon_ mode, the MemoryKiller checks the Sidekiq process RSS every 3 seconds - (defined by `SIDEKIQ_MEMORY_KILLER_CHECK_INTERVAL`). - -- `SIDEKIQ_MEMORY_KILLER_MAX_RSS` (KB): if this variable is set, and its value is greater - than 0, the MemoryKiller is enabled. Otherwise the MemoryKiller is disabled. - - `SIDEKIQ_MEMORY_KILLER_MAX_RSS` defines the Sidekiq process allowed RSS. - - In _legacy_ mode, if the Sidekiq process exceeds the allowed RSS then an irreversible - delayed graceful restart is triggered. The restart of Sidekiq happens - after `SIDEKIQ_MEMORY_KILLER_GRACE_TIME` seconds. - - In _daemon_ mode, if the Sidekiq process exceeds the allowed RSS for longer than - `SIDEKIQ_MEMORY_KILLER_GRACE_TIME` the graceful restart is triggered. If the - Sidekiq process go below the allowed RSS within `SIDEKIQ_MEMORY_KILLER_GRACE_TIME`, - the restart is aborted. - - The default value for Omnibus packages is set - [in the Omnibus GitLab - repository](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/attributes/default.rb). - -- `SIDEKIQ_MEMORY_KILLER_HARD_LIMIT_RSS` (KB): is used by _daemon_ mode. If the Sidekiq - process RSS (expressed in kilobytes) exceeds `SIDEKIQ_MEMORY_KILLER_HARD_LIMIT_RSS`, - an immediate graceful restart of Sidekiq is triggered. - -- `SIDEKIQ_MEMORY_KILLER_CHECK_INTERVAL`: used in _daemon_ mode to define how - often to check process RSS, default to 3 seconds. - -- `SIDEKIQ_MEMORY_KILLER_GRACE_TIME`: defaults to 900 seconds (15 minutes). - The usage of this variable is described as part of `SIDEKIQ_MEMORY_KILLER_MAX_RSS`. - -- `SIDEKIQ_MEMORY_KILLER_SHUTDOWN_WAIT`: defaults to 30 seconds. This defines the - maximum time allowed for all Sidekiq jobs to finish. No new jobs are accepted - during that time, and the process exits as soon as all jobs finish. - - If jobs do not finish during that time, the MemoryKiller interrupts all currently - running jobs by sending `SIGTERM` to the Sidekiq process. - - If the process hard shutdown/restart is not performed by Sidekiq, - the Sidekiq process is forcefully terminated after - `Sidekiq.options[:timeout] + 2` seconds. An external supervision mechanism - (for example, runit) must restart Sidekiq afterwards. +<!-- This redirect file can be deleted after <2022-11-11>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/operations/ssh_certificates.md b/doc/administration/operations/ssh_certificates.md index a777f5484fd..1e405189342 100644 --- a/doc/administration/operations/ssh_certificates.md +++ b/doc/administration/operations/ssh_certificates.md @@ -35,10 +35,10 @@ uploading user SSH keys to GitLab entirely. ## Setting up SSH certificate lookup via GitLab Shell How to fully set up SSH certificates is outside the scope of this -document. See [OpenSSH's -`PROTOCOL.certkeys`](https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/ssh/PROTOCOL.certkeys?annotate=HEAD) -for how it works, for example [RedHat's documentation about -it](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/sec-using_openssh_certificate_authentication). +document. See +[OpenSSH's`PROTOCOL.certkeys`](https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/ssh/PROTOCOL.certkeys?annotate=HEAD) +for how it works, for example +[RedHat's documentation about it](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/sec-using_openssh_certificate_authentication). We assume that you already have SSH certificates set up, and have added the `TrustedUserCAKeys` of your CA to your `sshd_config`, for example: @@ -143,8 +143,7 @@ This is because if the `AuthorizedPrincipalsCommand` can't authenticate the user, OpenSSH falls back on `~/.ssh/authorized_keys` (or the `AuthorizedKeysCommand`). -Therefore there may still be a reason to use the ["Fast lookup of -authorized SSH keys in the database"](fast_ssh_key_lookup.html) method +Therefore there may still be a reason to use the [Fast lookup of authorized SSH keys in the database](fast_ssh_key_lookup.md) method in conjunction with this. Since you are using SSH certificates for all your normal users, and relying on the `~/.ssh/authorized_keys` fallback for deploy keys, if you make use of those. @@ -160,8 +159,8 @@ users (especially if they're renewed) than you have deploy keys. Users can still bypass SSH certificate authentication by manually uploading an SSH public key to their profile, relying on the `~/.ssh/authorized_keys` fallback to authenticate it. There's -currently no feature to prevent this, [but there's an open request for -adding it](https://gitlab.com/gitlab-org/gitlab/-/issues/23260). +currently no feature to prevent this, +[but there's an open request for adding it](https://gitlab.com/gitlab-org/gitlab/-/issues/23260). Such a restriction can currently be hacked in by, for example, providing a custom `AuthorizedKeysCommand` which checks if the discovered key-ID diff --git a/doc/administration/package_information/deprecation_policy.md b/doc/administration/package_information/deprecation_policy.md index 422339c014c..73059514e1c 100644 --- a/doc/administration/package_information/deprecation_policy.md +++ b/doc/administration/package_information/deprecation_policy.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Deprecation policy **(FREE SELF)** +# Omnibus GitLab deprecation policy **(FREE SELF)** The Omnibus GitLab packages come with number of different libraries and services which offers users plethora of configuration options. diff --git a/doc/administration/package_information/index.md b/doc/administration/package_information/index.md index 746961e8613..d3a82b87116 100644 --- a/doc/administration/package_information/index.md +++ b/doc/administration/package_information/index.md @@ -34,8 +34,8 @@ These defaults are noted in the package [defaults document](defaults.md). ## Checking the versions of bundled software -After the Omnibus GitLab package is installed, all versions of the bundled -libraries are located in `/opt/gitlab/version-manifest.txt`. +After the Omnibus GitLab package is installed, you can find the version of +GitLab and all bundled libraries in `/opt/gitlab/version-manifest.txt`. If you don't have the package installed, you can always check the Omnibus GitLab [source repository](https://gitlab.com/gitlab-org/omnibus-gitlab/tree/master), specifically the diff --git a/doc/administration/package_information/postgresql_versions.md b/doc/administration/package_information/postgresql_versions.md index afdc63c4851..d831e8cea7f 100644 --- a/doc/administration/package_information/postgresql_versions.md +++ b/doc/administration/package_information/postgresql_versions.md @@ -11,15 +11,16 @@ This table lists only GitLab versions where a significant change happened in the package regarding PostgreSQL versions, not all. Usually, PostgreSQL versions change with major or minor GitLab releases. However, patch versions -of Omnibus GitLab sometimes update the patch level of PostgreSQL. +of Omnibus GitLab sometimes update the patch level of PostgreSQL. We've established a +[yearly cadence for PostgreSQL upgrades](https://about.gitlab.com/handbook/engineering/development/enablement/data_stores/database/postgresql-upgrade-cadence.html) +and trigger automatic database upgrades in the release before the new version is required. For example: - Omnibus 12.7.6 shipped with PostgreSQL 9.6.14 and 10.9. - Omnibus 12.7.7 shipped with PostgreSQL 9.6.17 and 10.12. -[Find out which versions of PostgreSQL (and other components) ship with -each Omnibus GitLab release](https://gitlab-org.gitlab.io/omnibus-gitlab/licenses.html). +[Find out which versions of PostgreSQL (and other components) ship with each Omnibus GitLab release](https://gitlab-org.gitlab.io/omnibus-gitlab/licenses.html). The lowest supported PostgreSQL versions are listed in the [installation requirements](../../install/requirements.md#postgresql-requirements). diff --git a/doc/administration/package_information/supported_os.md b/doc/administration/package_information/supported_os.md index 4f31981a05c..e4d7ea72cdc 100644 --- a/doc/administration/package_information/supported_os.md +++ b/doc/administration/package_information/supported_os.md @@ -10,8 +10,8 @@ GitLab officially supports LTS versions of operating systems. While OSs like Ubuntu have a clear distinction between LTS and non-LTS versions, there are other OSs, openSUSE for example, that don't follow the LTS concept. Hence to avoid confusion, the official policy is that at any point of time, all the -operating systems supported by GitLab are listed in the [installation -page](https://about.gitlab.com/install/). +operating systems supported by GitLab are listed in the +[installation page](https://about.gitlab.com/install/). The following lists the currently supported OSs and their possible EOL dates. @@ -19,11 +19,10 @@ The following lists the currently supported OSs and their possible EOL dates. | ------------------------------------------------------------ | ------------------------------ | --------------- | :----------------------------------------------------------: | ---------- | ------------------------------------------------------------ | | AlmaLinux 8 | GitLab CE / GitLab EE 14.5.0 | x86_64, aarch64 | [AlmaLinux Install Documentation](https://about.gitlab.com/install/#almalinux-8) | 2029 | <https://almalinux.org/> | | CentOS 7 | GitLab CE / GitLab EE 7.10.0 | x86_64 | [CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | June 2024 | <https://wiki.centos.org/About/Product> | -| Debian 9 | GitLab CE / GitLab EE 9.3.0 | amd64 | [Debian Install Documentation](https://about.gitlab.com/install/#debian) | 2022 | <https://wiki.debian.org/LTS> | | Debian 10 | GitLab CE / GitLab EE 12.2.0 | amd64, arm64 | [Debian Install Documentation](https://about.gitlab.com/install/#debian) | 2024 | <https://wiki.debian.org/LTS> | | Debian 11 | GitLab CE / GitLab EE 14.6.0 | amd64, arm64 | [Debian Install Documentation](https://about.gitlab.com/install/#debian) | 2026 | <https://wiki.debian.org/LTS> | | OpenSUSE 15.3 | GitLab CE / GitLab EE 14.5.0 | x86_64, aarch64 | [OpenSUSE Install Documentation](https://about.gitlab.com/install/#opensuse-leap-15-3) | Nov 2022 | <https://en.opensuse.org/Lifetime> | -| RHEL 8 | GitLab CE / GitLab EE 12.8.1 | x86_64, arm64 | [Use CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | May 2024 | [RHEL Details](https://access.redhat.com/support/policy/updates/errata/#Life_Cycle_Dates) | +| RHEL 8 | GitLab CE / GitLab EE 12.8.1 | x86_64, arm64 | [Use CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | May 2029 | [RHEL Details](https://access.redhat.com/support/policy/updates/errata/#Life_Cycle_Dates) | | SLES 12 | GitLab EE 9.0.0 | x86_64 | [Use OpenSUSE Install Documentation](https://about.gitlab.com/install/#opensuse-leap-15-3) | Oct 2027 | <https://www.suse.com/lifecycle/> | | Oracle Linux | GitLab CE / GitLab EE 8.14.0 | x86_64 | [Use CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | Jul 2024 | <https://www.oracle.com/a/ocom/docs/elsp-lifetime-069338.pdf> | | Scientific Linux | GitLab CE / GitLab EE 8.14.0 | x86_64 | [Use CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | June 2024 | <https://scientificlinux.org/downloads/sl-versions/sl7/> | @@ -87,6 +86,7 @@ release for them can be found below: | OpenSUSE 15.1 | [November 2020](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-13.12&dist=opensuse%2F15.1) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-13.12&dist=opensuse%2F15.1) 13.12 | | Ubuntu 16.04 | [April 2021](https://ubuntu.com/info/release-end-of-life) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_13.12&dist=ubuntu%2Fxenial) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_13.12&dist=ubuntu%2Fxenial) 13.12 | | OpenSUSE 15.2 | [December 2021](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-14.7&dist=opensuse%2F15.2) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-14.7&dist=opensuse%2F15.2) 14.7 | +| Debian 9 "Stretch" | [June 2022](https://lists.debian.org/debian-lts-announce/2022/07/msg00002.html) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_15.2&dist=debian%2Fstretch) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_15.2&dist=debian%2Fstretch) 15.2 | NOTE: An exception to this deprecation policy is when we are unable to provide diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index 3bfb3ceca86..8b115ca1af4 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -739,7 +739,7 @@ auth: Without these entries, the registry logins cannot authenticate with GitLab. GitLab also remains unaware of -[nested image names](../../user/packages/container_registry/#image-naming-convention) +[nested image names](../../user/packages/container_registry/index.md#image-naming-convention) under the project hierarchy, like `registry.example.com/group/project/image-name:tag` or `registry.example.com/group/project/my/image-name:tag`, and only recognizes @@ -1202,8 +1202,8 @@ Before diving in to the following sections, here's some basic troubleshooting: been synchronized (for example, via NTP). 1. If you are using an S3-backed Registry, double check that the IAM - permissions and the S3 credentials (including region) are correct. See [the - sample IAM policy](https://docs.docker.com/registry/storage-drivers/s3/) + permissions and the S3 credentials (including region) are correct. See + [the sample IAM policy](https://docs.docker.com/registry/storage-drivers/s3/) for more details. 1. Check the Registry logs (for example `/var/log/gitlab/registry/current`) and the GitLab production logs @@ -1388,7 +1388,7 @@ project or branch name. Special characters can include: - Trailing hyphen/dash - Double hyphen/dash -To get around this, you can [change the group path](../../user/group/index.md#change-a-groups-path), +To get around this, you can [change the group path](../../user/group/manage.md#change-a-groups-path), [change the project path](../../user/project/settings/index.md#rename-a-repository) or change the branch name. Another option is to create a [push rule](../../user/project/repository/push_rules.md) to prevent this at the instance level. @@ -1631,8 +1631,8 @@ wrong. However, since all communications between Docker clients and servers are done over HTTPS, it's a bit difficult to decrypt the traffic quickly even if you know the private key. What can we do instead? -One way would be to disable HTTPS by setting up an [insecure -Registry](https://docs.docker.com/registry/insecure/). This could introduce a +One way would be to disable HTTPS by setting up an +[insecure Registry](https://docs.docker.com/registry/insecure/). This could introduce a security hole and is only recommended for local testing. If you have a production system and can't or don't want to do this, there is another way: use mitmproxy, which stands for Man-in-the-Middle Proxy. diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index b58eeff48cf..e545fb3cd1b 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -78,7 +78,7 @@ among other things. Follow [these instructions](https://publicsuffix.org/submit/) to submit your GitLab Pages subdomain. For instance, if your domain is `example.io`, you should request that `example.io` is added to the Public Suffix List. GitLab.com -added `gitlab.io` [in 2016](https://gitlab.com/gitlab-com/infrastructure/-/issues/230). +added `gitlab.io` [in 2016](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/230). ### DNS configuration @@ -614,7 +614,7 @@ Follow the steps below to configure verbose logging of GitLab Pages daemon. Setting the `propagate_correlation_id` to true allows installations behind a reverse proxy to generate and set a correlation ID to requests sent to GitLab Pages. When a reverse proxy sets the header value `X-Request-ID`, the value propagates in the request chain. -Users [can find the correlation ID in the logs](../troubleshooting/tracing_correlation_id.md#identify-the-correlation-id-for-a-request). +Users [can find the correlation ID in the logs](../logs/tracing_correlation_id.md#identify-the-correlation-id-for-a-request). To enable the propagation of the correlation ID: @@ -933,8 +933,8 @@ The following settings are: | `connection` | Various connection options described below. | | NOTE: -If you want to stop using and disconnect the NFS server, you need to [explicitly disable -local storage](#disable-pages-local-storage), and it's only possible after upgrading to GitLab 13.11. +If you want to stop using and disconnect the NFS server, you need to +[explicitly disable local storage](#disable-pages-local-storage), and it's only possible after upgrading to GitLab 13.11. #### S3-compatible connection settings @@ -1319,25 +1319,6 @@ and in your Pages log shows this error: sudo gitlab-ctl restart gitlab-pages ``` -### 404 error after promoting a Geo secondary to a primary node - -Pages files are not among the -[supported data types](../geo/replication/datatypes.md#limitations-on-replicationverification) for replication in Geo. After a secondary node is promoted to a primary node, attempts to access a Pages site result in a `404 Not Found` error. - -It is possible to copy the subfolders and files in the [Pages path](#change-storage-path) -to the new primary node to resolve this. -For example, you can adapt the `rsync` strategy from the -[moving repositories documentation](../operations/moving_repositories.md). -Alternatively, run the CI pipelines of those projects that contain a `pages` job again. - -### 404 or 500 error when accessing GitLab Pages in a Geo setup - -Pages sites are only available on the primary Geo site, while the codebase of the project is available on all sites. - -If you try to access a Pages page on a secondary site, a 404 or 500 HTTP code is returned depending on the access control. - -Read more which [features don't support Geo replication/verification](../geo/replication/datatypes.md#limitations-on-replicationverification). - ### Failed to connect to the internal GitLab API If you see the following error: @@ -1495,7 +1476,7 @@ In GitLab 14.0-14.2 you can temporarily enable legacy storage and configuration To do that: -1. Please describe the issue you're seeing in the [migration feedback issue](https://gitlab.com/gitlab-org/gitlab/-/issues/331699). +1. Describe the issue you're seeing in the [migration feedback issue](https://gitlab.com/gitlab-org/gitlab/-/issues/331699). 1. Edit `/etc/gitlab/gitlab.rb`: diff --git a/doc/administration/pages/source.md b/doc/administration/pages/source.md index 4bb9fc56b6f..5ab508dbaca 100644 --- a/doc/administration/pages/source.md +++ b/doc/administration/pages/source.md @@ -14,8 +14,7 @@ This is the documentation for configuring a GitLab Pages when you have installed GitLab from source and not using the Omnibus packages. You are encouraged to read the [Omnibus documentation](index.md) as it provides -some invaluable information to the configuration of GitLab Pages. Please proceed -to read it before going forward with this guide. +invaluable information about the configuration of GitLab Pages. We also highly recommend that you use the Omnibus GitLab packages. We optimize them specifically for GitLab, and we take care of upgrading GitLab diff --git a/doc/administration/postgresql/database_load_balancing.md b/doc/administration/postgresql/database_load_balancing.md index b03abf9881c..f53b50192eb 100644 --- a/doc/administration/postgresql/database_load_balancing.md +++ b/doc/administration/postgresql/database_load_balancing.md @@ -180,7 +180,7 @@ gitlab_rails['db_load_balancing'] = { ## Logging The load balancer logs various events in -[`database_load_balancing.log`](../logs.md#database_load_balancinglog), such as +[`database_load_balancing.log`](../logs/index.md#database_load_balancinglog), such as - When a host is marked as offline - When a host comes back online diff --git a/doc/administration/pseudonymizer.md b/doc/administration/pseudonymizer.md index b14c1f64efe..8f27d41b75f 100644 --- a/doc/administration/pseudonymizer.md +++ b/doc/administration/pseudonymizer.md @@ -8,8 +8,7 @@ redirect_to: 'index.md' # Pseudonymizer (removed) **(ULTIMATE)** -> [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/219952) in -> GitLab 14.7 and removed in 15.0. +> [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/219952) in GitLab 14.7 and removed in 15.0. WARNING: This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/219952) in GitLab 14.7. diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md index 6de47c37957..a900da52aba 100644 --- a/doc/administration/raketasks/maintenance.md +++ b/doc/administration/raketasks/maintenance.md @@ -187,7 +187,7 @@ Checking GitLab ... Finished In some cases it is necessary to rebuild the `authorized_keys` file, for example, if after an upgrade you receive `Permission denied (publickey)` when pushing [via SSH](../../user/ssh.md) -and find `404 Key Not Found` errors in [the `gitlab-shell.log` file](../logs.md#gitlab-shelllog). +and find `404 Key Not Found` errors in [the `gitlab-shell.log` file](../logs/index.md#gitlab-shelllog). To rebuild `authorized_keys`, run: **Omnibus Installation** diff --git a/doc/administration/raketasks/project_import_export.md b/doc/administration/raketasks/project_import_export.md index f623d4e4526..f3a9845d129 100644 --- a/doc/administration/raketasks/project_import_export.md +++ b/doc/administration/raketasks/project_import_export.md @@ -17,11 +17,7 @@ GitLab provides Rake tasks relating to project import and export. For more infor You can query an import through the [Project import/export API](../../api/project_import_export.md#import-status). As described in the API documentation, the query may return an import error or exceptions. -## Import large projects - -If you have a larger project, consider using a Rake task, as described in our [developer documentation](../../development/import_project.md#importing-via-a-rake-task). - -## Import/export tasks +## Import/export Rake tasks The GitLab import/export version can be checked by using the following command: @@ -57,3 +53,38 @@ Note the following: - The exports are stored in a temporary directory and are deleted every 24 hours by a specific worker. + +### Import large projects using a Rake task + +If you have a larger project, consider using a Rake task as described in our [developer documentation](../../development/import_project.md#importing-via-a-rake-task). + +### Export using a Rake task + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25598) in GitLab 12.9. + +You can use a Rake task to export large project. + +Parameters: + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `username` | string | yes | User name | +| `namespace_path` | string | yes | Namespace path | +| `project_path` | string | yes | Project name | +| `archive_path` | string | yes | Path to file to store the export project tarball | + +```shell +gitlab-rake "gitlab:import_export:export[username, namespace_path, project_path, archive_path]" +``` + +## Troubleshooting + +If you are having trouble with import/export, you can enable debug mode using the same Rake task: + +```shell +# Import +IMPORT_DEBUG=true gitlab-rake "gitlab:import_export:import[root, group/subgroup, testingprojectimport, /path/to/file_to_import.tar.gz]" + +# Export +EXPORT_DEBUG=true gitlab-rake "gitlab:import_export:export[root, group/subgroup, projectnametoexport, /tmp/export_file.tar.gz]" +``` diff --git a/doc/administration/redis/replication_and_failover.md b/doc/administration/redis/replication_and_failover.md index 9b1a456835a..c4b83b66738 100644 --- a/doc/administration/redis/replication_and_failover.md +++ b/doc/administration/redis/replication_and_failover.md @@ -16,14 +16,14 @@ In Redis lingo, `primary` is called `master`. In this document, `primary` is use instead of `master`, except the settings where `master` is required. Using [Redis](https://redis.io/) in scalable environment is possible using a **Primary** x **Replica** -topology with a [Redis Sentinel](https://redis.io/topics/sentinel) service to watch and automatically +topology with a [Redis Sentinel](https://redis.io/docs/manual/sentinel/) service to watch and automatically start the failover procedure. Redis requires authentication if used with Sentinel. See -[Redis Security](https://redis.io/topics/security) documentation for more +[Redis Security](https://redis.io/docs/manual/security/) documentation for more information. We recommend using a combination of a Redis password and tight firewall rules to secure your Redis service. -You are highly encouraged to read the [Redis Sentinel](https://redis.io/topics/sentinel) documentation +You are highly encouraged to read the [Redis Sentinel](https://redis.io/docs/manual/sentinel/) documentation before configuring Redis with GitLab to fully understand the topology and architecture. @@ -67,7 +67,7 @@ When a **Primary** fails to respond, it's the application's responsibility for a new **Primary**). To get a better understanding on how to correctly set up Sentinel, please read -the [Redis Sentinel documentation](https://redis.io/topics/sentinel) first, as +the [Redis Sentinel](https://redis.io/docs/manual/sentinel/) documentation first, as failing to configure it correctly can lead to data loss or can bring your whole cluster down, invalidating the failover effort. @@ -343,8 +343,8 @@ NOTE: If you are using an external Redis Sentinel instance, be sure to exclude the `requirepass` parameter from the Sentinel configuration. This parameter causes clients to report `NOAUTH -Authentication required.`. [Redis Sentinel 3.2.x does not support -password authentication](https://github.com/antirez/redis/issues/3279). +Authentication required.`. +[Redis Sentinel 3.2.x does not support password authentication](https://github.com/antirez/redis/issues/3279). Now that the Redis servers are all set up, let's configure the Sentinel servers. diff --git a/doc/administration/redis/replication_and_failover_external.md b/doc/administration/redis/replication_and_failover_external.md index 998455e5621..d624fe28f80 100644 --- a/doc/administration/redis/replication_and_failover_external.md +++ b/doc/administration/redis/replication_and_failover_external.md @@ -148,7 +148,7 @@ starting with `sentinel` prefix. Assuming that the Redis Sentinel is installed on the same instance as Redis primary with IP `10.0.0.1` (some settings might overlap with the primary): -1. [Install Redis Sentinel](https://redis.io/topics/sentinel). +1. [Install Redis Sentinel](https://redis.io/docs/manual/sentinel/). 1. Edit `/etc/redis/sentinel.conf`: ```conf diff --git a/doc/administration/redis/troubleshooting.md b/doc/administration/redis/troubleshooting.md index ca52fe0a29a..f9e5390c227 100644 --- a/doc/administration/redis/troubleshooting.md +++ b/doc/administration/redis/troubleshooting.md @@ -167,4 +167,4 @@ production: port: 26379 # point to sentinel, not to redis port ``` -When in doubt, read the [Redis Sentinel documentation](https://redis.io/topics/sentinel). +When in doubt, read the [Redis Sentinel](https://redis.io/docs/manual/sentinel/) documentation. diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index a64828be4be..fcc32100bc8 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -693,7 +693,8 @@ SSH in to any of the Patroni nodes on the **primary site**: ``` If the 'State' column for any node doesn't say "running", check the -[Troubleshooting section](troubleshooting.md) before proceeding. +[PostgreSQL replication and failover troubleshooting section](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) +before proceeding. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -800,14 +801,14 @@ The following IPs will be used as an example: ## Configure Redis Using [Redis](https://redis.io/) in scalable environment is possible using a **Primary** x **Replica** -topology with a [Redis Sentinel](https://redis.io/topics/sentinel) service to watch and automatically +topology with a [Redis Sentinel](https://redis.io/docs/manual/sentinel/) service to watch and automatically start the failover procedure. Redis requires authentication if used with Sentinel. See -[Redis Security](https://redis.io/topics/security) documentation for more +[Redis Security](https://redis.io/docs/manual/security/) documentation for more information. We recommend using a combination of a Redis password and tight firewall rules to secure your Redis service. -You are highly encouraged to read the [Redis Sentinel](https://redis.io/topics/sentinel) documentation +You are highly encouraged to read the [Redis Sentinel](https://redis.io/docs/manual/sentinel/) documentation before configuring Redis with GitLab to fully understand the topology and architecture. @@ -2062,10 +2063,8 @@ On each node perform the following: ``` 1. Optionally, from the Gitaly servers, confirm that Gitaly can perform callbacks to the internal API: - - ```shell - sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml - ``` + - For GitLab 15.3 and later, run `sudo /opt/gitlab/embedded/bin/gitaly check /var/opt/gitlab/gitaly/config.toml`. + - For GitLab 15.2 and earlier, run `sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml`. When you specify `https` in the `external_url`, as in the previous example, GitLab expects that the SSL certificates are in `/etc/gitlab/ssl/`. If the @@ -2086,7 +2085,7 @@ the [NGINX documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#en failing to connect to PostgreSQL, it may be that your PgBouncer node's IP address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` on your database nodes. Before proceeding, see - [PgBouncer error `ERROR: pgbouncer cannot connect to server`](troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). + [PgBouncer error `ERROR: pgbouncer cannot connect to server`](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). 1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 5b1e8bfc16b..51d3e3d39a6 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -696,7 +696,8 @@ SSH in to any of the Patroni nodes on the **primary site**: ``` If the 'State' column for any node doesn't say "running", check the -[Troubleshooting section](troubleshooting.md) before proceeding. +[PostgreSQL replication and failover troubleshooting section](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) +before proceeding. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -803,14 +804,14 @@ The following IPs will be used as an example: ## Configure Redis Using [Redis](https://redis.io/) in scalable environment is possible using a **Primary** x **Replica** -topology with a [Redis Sentinel](https://redis.io/topics/sentinel) service to watch and automatically +topology with a [Redis Sentinel](https://redis.io/docs/manual/sentinel/) service to watch and automatically start the failover procedure. Redis requires authentication if used with Sentinel. See -[Redis Security](https://redis.io/topics/security) documentation for more +[Redis Security](https://redis.io/docs/manual/security/) documentation for more information. We recommend using a combination of a Redis password and tight firewall rules to secure your Redis service. -You are highly encouraged to read the [Redis Sentinel](https://redis.io/topics/sentinel) documentation +You are highly encouraged to read the [Redis Sentinel](https://redis.io/docs/manual/sentinel/) documentation before configuring Redis with GitLab to fully understand the topology and architecture. @@ -2067,10 +2068,8 @@ On each node perform the following: ``` 1. Optionally, from the Gitaly servers, confirm that Gitaly can perform callbacks to the internal API: - - ```shell - sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml - ``` + - For GitLab 15.3 and later, run `sudo /opt/gitlab/embedded/bin/gitaly check /var/opt/gitlab/gitaly/config.toml`. + - For GitLab 15.2 and earlier, run `sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml`. When you specify `https` in the `external_url`, as in the previous example, GitLab expects that the SSL certificates are in `/etc/gitlab/ssl/`. If the @@ -2091,7 +2090,7 @@ the [NGINX documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#en failing to connect to PostgreSQL, it may be that your PgBouncer node's IP address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` on your database nodes. Before proceeding, see - [PgBouncer error `ERROR: pgbouncer cannot connect to server`](troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). + [PgBouncer error `ERROR: pgbouncer cannot connect to server`](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). 1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index df84bf0402d..8ca58da55ca 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -512,10 +512,8 @@ Updates to example must be made at: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Confirm that Gitaly can perform callbacks to the internal API: - - ```shell - sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml - ``` + - For GitLab 15.3 and later, run `sudo /opt/gitlab/embedded/bin/gitaly check /var/opt/gitlab/gitaly/config.toml`. + - For GitLab 15.2 and earlier, run `sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml`. ### Gitaly TLS support @@ -658,10 +656,12 @@ On each node perform the following: # Set the network addresses that the exporters used for monitoring will listen on node_exporter['listen_address'] = '0.0.0.0:9100' gitlab_workhorse['prometheus_listen_addr'] = '0.0.0.0:9229' + puma['listen'] = '0.0.0.0' sidekiq['listen_address'] = "0.0.0.0" - # Set number of Sidekiq threads per queue process to the recommend number of 10 + + # Configure Sidekiq with 2 workers and 10 max concurrency sidekiq['max_concurrency'] = 10 - puma['listen'] = '0.0.0.0' + sidekiq['queue_groups'] = ['*'] * 2 # Add the monitoring node's IP address to the monitoring whitelist and allow it to # scrape the NGINX metrics. Replace placeholder `monitoring.gitlab.example.com` with @@ -766,7 +766,7 @@ the [NGINX documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#en failing to connect to PostgreSQL, it may be that your PgBouncer node's IP address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` on your database nodes. Before proceeding, see - [PgBouncer error `ERROR: pgbouncer cannot connect to server`](troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). + [PgBouncer error `ERROR: pgbouncer cannot connect to server`](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). 1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). @@ -1011,8 +1011,8 @@ future with further specific cloud provider details. | Service | Nodes | Configuration | GCP | AWS | Min Allocatable CPUs and Memory | |-----------------------------------------------|-------|------------------------|-----------------|--------------|---------------------------------| | Webservice | 3 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | 23.7 vCPU, 16.9 GB memory | -| Sidekiq | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | 3.9 vCPU, 11.8 GB memory | -| Supporting services such as NGINX, Prometheus | 2 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `m5.large` | 1.9 vCPU, 5.5 GB memory | +| Sidekiq | 1 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | 3.9 vCPU, 11.8 GB memory | +| Supporting services such as NGINX, Prometheus | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | 1.9 vCPU, 5.5 GB memory | - For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. @@ -1048,8 +1048,8 @@ card "Kubernetes via Helm Charts" as kubernetes { together { collections "**Webservice** x3" as gitlab #32CD32 - collections "**Sidekiq** x2" as sidekiq #ff8dd1 - card "**Supporting Services**" as support + card "**Sidekiq**" as sidekiq #ff8dd1 + collections "**Supporting Services** x2" as support } } @@ -1082,13 +1082,13 @@ documents how to apply the calculated configuration to the Helm Chart. #### Webservice Webservice pods typically need about 1 vCPU and 1.25 GB of memory _per worker_. -Each Webservice pod consumes roughly 2 vCPUs and 2.5 GB of memory using +Each Webservice pod consumes roughly 4 vCPUs and 5 GB of memory using the [recommended topology](#cluster-topology) because two worker processes are created by default and each pod has other small processes running. For 2,000 users we recommend a total Puma worker count of around 12. -With the [provided recommendations](#cluster-topology) this allows the deployment of up to 6 -Webservice pods with 2 workers per pod and 2 pods per node. Expand available resources using +With the [provided recommendations](#cluster-topology) this allows the deployment of up to 3 +Webservice pods with 4 workers per pod and 1 pod per node. Expand available resources using the ratio of 1 vCPU to 1.25 GB of memory _per each worker process_ for each additional Webservice pod. @@ -1099,7 +1099,7 @@ For further information on resource usage, see the [Webservice resources](https: Sidekiq pods should generally have 1 vCPU and 2 GB of memory. [The provided starting point](#cluster-topology) allows the deployment of up to -2 Sidekiq pods. Expand available resources using the 1 vCPU to 2GB memory +4 Sidekiq pods. Expand available resources using the 1 vCPU to 2 GB memory ratio for each additional pod. For further information on resource usage, see the [Sidekiq resources](https://docs.gitlab.com/charts/charts/gitlab/sidekiq/#resources). diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index 6a70739ca31..db81ae7ae3c 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -425,14 +425,14 @@ all nodes. ## Configure Redis Using [Redis](https://redis.io/) in scalable environment is possible using a **Primary** x **Replica** -topology with a [Redis Sentinel](https://redis.io/topics/sentinel) service to watch and automatically +topology with a [Redis Sentinel](https://redis.io/docs/manual/sentinel/) service to watch and automatically start the failover procedure. Redis requires authentication if used with Sentinel. See -[Redis Security](https://redis.io/topics/security) documentation for more +[Redis Security](https://redis.io/docs/manual/security/) documentation for more information. We recommend using a combination of a Redis password and tight firewall rules to secure your Redis service. -You are highly encouraged to read the [Redis Sentinel](https://redis.io/topics/sentinel) documentation +You are highly encouraged to read the [Redis Sentinel](https://redis.io/docs/manual/sentinel/) documentation before configuring Redis with GitLab to fully understand the topology and architecture. @@ -978,7 +978,8 @@ SSH in to any of the Patroni nodes on the **primary site**: ``` If the 'State' column for any node doesn't say "running", check the -[Troubleshooting section](troubleshooting.md) before proceeding. +[PostgreSQL replication and failover troubleshooting section](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) +before proceeding. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -2018,7 +2019,7 @@ the [NGINX documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#en failing to connect to PostgreSQL, it may be that your PgBouncer node's IP address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` on your database nodes. Before proceeding, see - [PgBouncer error `ERROR: pgbouncer cannot connect to server`](troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). + [PgBouncer error `ERROR: pgbouncer cannot connect to server`](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). 1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 0d0e44e2364..535e483cb7d 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -703,7 +703,8 @@ SSH in to any of the Patroni nodes on the **primary site**: ``` If the 'State' column for any node doesn't say "running", check the -[Troubleshooting section](troubleshooting.md) before proceeding. +[PostgreSQL replication and failover troubleshooting section](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) +before proceeding. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -810,14 +811,14 @@ The following IPs will be used as an example: ## Configure Redis Using [Redis](https://redis.io/) in scalable environment is possible using a **Primary** x **Replica** -topology with a [Redis Sentinel](https://redis.io/topics/sentinel) service to watch and automatically +topology with a [Redis Sentinel](https://redis.io/docs/manual/sentinel/) service to watch and automatically start the failover procedure. Redis requires authentication if used with Sentinel. See -[Redis Security](https://redis.io/topics/security) documentation for more +[Redis Security](https://redis.io/docs/manual/security/) documentation for more information. We recommend using a combination of a Redis password and tight firewall rules to secure your Redis service. -You are highly encouraged to read the [Redis Sentinel](https://redis.io/topics/sentinel) documentation +You are highly encouraged to read the [Redis Sentinel](https://redis.io/docs/manual/sentinel/) documentation before configuring Redis with GitLab to fully understand the topology and architecture. @@ -2083,10 +2084,8 @@ On each node perform the following: ``` 1. Optionally, from the Gitaly servers, confirm that Gitaly can perform callbacks to the internal API: - - ```shell - sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml - ``` + - For GitLab 15.3 and later, run `sudo /opt/gitlab/embedded/bin/gitaly check /var/opt/gitlab/gitaly/config.toml`. + - For GitLab 15.2 and earlier, run `sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml`. When you specify `https` in the `external_url`, as in the previous example, GitLab expects that the SSL certificates are in `/etc/gitlab/ssl/`. If the @@ -2107,7 +2106,7 @@ the [NGINX documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#en failing to connect to PostgreSQL, it may be that your PgBouncer node's IP address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` on your database nodes. Before proceeding, see - [PgBouncer error `ERROR: pgbouncer cannot connect to server`](troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). + [PgBouncer error `ERROR: pgbouncer cannot connect to server`](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). 1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index ef2e48baa33..e19440e7660 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -423,14 +423,14 @@ all nodes. ## Configure Redis Using [Redis](https://redis.io/) in scalable environment is possible using a **Primary** x **Replica** -topology with a [Redis Sentinel](https://redis.io/topics/sentinel) service to watch and automatically +topology with a [Redis Sentinel](https://redis.io/docs/manual/sentinel/) service to watch and automatically start the failover procedure. Redis requires authentication if used with Sentinel. See -[Redis Security](https://redis.io/topics/security) documentation for more +[Redis Security](https://redis.io/docs/manual/security/) documentation for more information. We recommend using a combination of a Redis password and tight firewall rules to secure your Redis service. -You are highly encouraged to read the [Redis Sentinel](https://redis.io/topics/sentinel) documentation +You are highly encouraged to read the [Redis Sentinel](https://redis.io/docs/manual/sentinel/) documentation before configuring Redis with GitLab to fully understand the topology and architecture. @@ -975,7 +975,8 @@ SSH in to any of the Patroni nodes on the **primary site**: ``` If the 'State' column for any node doesn't say "running", check the -[Troubleshooting section](troubleshooting.md) before proceeding. +[PostgreSQL replication and failover troubleshooting section](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) +before proceeding. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -2018,7 +2019,7 @@ the [NGINX documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#en failing to connect to PostgreSQL, it may be that your PgBouncer node's IP address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` on your database nodes. Before proceeding, see - [PgBouncer error `ERROR: pgbouncer cannot connect to server`](troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). + [PgBouncer error `ERROR: pgbouncer cannot connect to server`](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). 1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index e996acb1157..7be12e12386 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -240,7 +240,7 @@ The following table details the cost to run the different reference architecture <td></td> <td><a href="https://calculator.aws/#/estimate?id=b51f178f4403b69a63f6eb33ea425f82de3bf249">Calculated cost</a></td> <td></td> - <td><a href="https://azure.com/e/1adf30bef7e34ceba9efa97c4470417b">Calculated cost</a></td> + <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=1adf30bef7e34ceba9efa97c4470417b">Calculated cost</a></td> </tr> <tr> <th scope="row">2k</th> @@ -248,7 +248,7 @@ The following table details the cost to run the different reference architecture <td></td> <td><a href="https://calculator.aws/#/estimate?id=dce36b5cb6ab25211f74e47233d77f58fefb54e2">Calculated cost</a></td> <td></td> - <td><a href="https://azure.com/e/72764902f3854f798407fb03c3de4b6f">Calculated cost</a></td> + <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=72764902f3854f798407fb03c3de4b6f">Calculated cost</a></td> </tr> <tr> <th scope="row">3k</th> @@ -256,7 +256,7 @@ The following table details the cost to run the different reference architecture <td></td> <td><a href="https://calculator.aws/#/estimate?id=b1c5b4e32e990eaeb035a148255132bd28988760">Calculated cost</a></td> <td></td> - <td><a href="https://azure.com/e/0dbfc575051943b9970e5d8ace03680d">Calculated cost</a></td> + <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=0dbfc575051943b9970e5d8ace03680d">Calculated cost</a></td> </tr> <tr> <th scope="row">5k</th> @@ -264,7 +264,7 @@ The following table details the cost to run the different reference architecture <td></td> <td><a href="https://calculator.aws/#/estimate?id=2bf1af883096e6f4c6efddb4f3c35febead7fec2">Calculated cost</a></td> <td></td> - <td><a href="https://azure.com/e/8f618711ffec4b039f1581871ca6a7c9">Calculated cost</a></td> + <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=8f618711ffec4b039f1581871ca6a7c9">Calculated cost</a></td> </tr> <tr> <th scope="row">10k</th> @@ -272,7 +272,7 @@ The following table details the cost to run the different reference architecture <td></td> <td><a href="https://calculator.aws/#/estimate?id=1d374df13c0f2088d332ab0134f5b1d0f717259e">Calculated cost</a></td> <td></td> - <td><a href="https://azure.com/e/de3da8286dda4d4db1362932bc75410b">Calculated cost</a></td> + <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=de3da8286dda4d4db1362932bc75410b">Calculated cost</a></td> </tr> <tr> <th scope="row">25k</th> @@ -280,7 +280,7 @@ The following table details the cost to run the different reference architecture <td></td> <td><a href="https://calculator.aws/#/estimate?id=46fe6a6e9256d9b7779fae59fbbfa7e836942b7d">Calculated cost</a></td> <td></td> - <td><a href="https://azure.com/e/69724ebd82914a60857da6a3ace05a64">Calculate cost</a></td> + <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=69724ebd82914a60857da6a3ace05a64">Calculate cost</a></td> </tr> <tr> <th scope="row">50k</th> @@ -288,7 +288,7 @@ The following table details the cost to run the different reference architecture <td></td> <td><a href="https://calculator.aws/#/estimate?id=e15926b1a3c7139e4faf390a3875ff807d2ab91c">Calculated cost</a></td> <td></td> - <td><a href="https://azure.com/e/3f973040ebc14023933d35f576c89846">Calculated cost</a></td> + <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=3f973040ebc14023933d35f576c89846">Calculated cost</a></td> </tr> </table> diff --git a/doc/administration/reference_architectures/troubleshooting.md b/doc/administration/reference_architectures/troubleshooting.md index 6e0aaacd598..a9bf035a528 100644 --- a/doc/administration/reference_architectures/troubleshooting.md +++ b/doc/administration/reference_architectures/troubleshooting.md @@ -1,352 +1,11 @@ --- -stage: Systems -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../configure.md' +remove_date: '2022-10-24' --- -# Troubleshooting a reference architecture setup **(FREE SELF)** +This document was moved to [another location](../configure.md). -This page serves as the troubleshooting documentation if you followed one of -the [reference architectures](index.md#reference-architectures). - -## Troubleshooting object storage - -### S3 API compatibility issues - -Not all S3 providers [are fully compatible](../../raketasks/backup_gitlab.md#other-s3-providers) -with the Fog library that GitLab uses. Symptoms include: - -```plaintext -411 Length Required -``` - -### GitLab Pages can use object storage - -If you intend to use [GitLab Pages](../../user/project/pages/index.md), you can -[configure object storage](../pages/index.md#using-object-storage). -NFS is still available if you prefer. - -The [GitLab Pages Helm chart](https://docs.gitlab.com/charts/charts/gitlab/gitlab-pages/) is also available -for Kubernetes deployments. - -### Incremental logging is required for CI to use object storage - -If you configure GitLab to use object storage for CI logs and artifacts, -[you must also enable incremental logging](../job_logs.md#incremental-logging-architecture). - -### Proxy Download - -A number of the use cases for object storage allow client traffic to be redirected to the -object storage back end, like when Git clients request large files via LFS or when -downloading CI artifacts and logs. - -When the files are stored on local block storage or NFS, GitLab has to act as a proxy. -With object storage, the default behavior is for GitLab to redirect to the object -storage device rather than proxy the request. - -The `proxy_download` setting controls this behavior: the default is generally `false`. -Verify this in the documentation for each use case. Set it to `true` to make -GitLab proxy the files rather than redirect. - -When not proxying files, GitLab returns an -[HTTP 302 redirect with a pre-signed, time-limited object storage URL](https://gitlab.com/gitlab-org/gitlab/-/issues/32117#note_218532298). -This can result in some of the following problems: - -- If GitLab is using non-secure HTTP to access the object storage, clients may generate -`https->http` downgrade errors and refuse to process the redirect. The solution to this -is for GitLab to use HTTPS. LFS, for example, generates this error: - - ```plaintext - LFS: lfsapi/client: refusing insecure redirect, https->http - ``` - -- Clients must trust the certificate authority that issued the object storage -certificate, or may return common TLS errors such as: - - ```plaintext - x509: certificate signed by unknown authority - ``` - -- Clients need network access to the object storage. Errors that might result -if this access is not in place include: - - ```plaintext - Received status code 403 from server: Forbidden - ``` - -### ETag mismatch - -Using the default GitLab settings, some object storage back-ends such as -[MinIO](https://gitlab.com/gitlab-org/gitlab/-/issues/23188) -and [Alibaba](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/1564) -might generate `ETag mismatch` errors. - -When using GitLab direct upload, the -[workaround for MinIO](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/1564#note_244497658) -is to use the `--compat` parameter on the server. - -We are working on a fix to GitLab component Workhorse, and also -a workaround, in the mean time, to -[allow ETag verification to be disabled](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18175). - -## Troubleshooting Redis - -There are a lot of moving parts that needs to be taken care carefully -in order for the HA setup to work as expected. - -Before proceeding with the troubleshooting below, check your firewall rules: - -- Redis machines - - Accept TCP connection in `6379` - - Connect to the other Redis machines via TCP in `6379` -- Sentinel machines - - Accept TCP connection in `26379` - - Connect to other Sentinel machines via TCP in `26379` - - Connect to the Redis machines via TCP in `6379` - -### Troubleshooting Redis replication - -You can check if everything is correct by connecting to each server using -`redis-cli` application, and sending the `info replication` command as below. - -```shell -/opt/gitlab/embedded/bin/redis-cli -h <redis-host-or-ip> -a '<redis-password>' info replication -``` - -When connected to a `Primary` Redis, you see the number of connected -`replicas`, and a list of each with connection details: - -```plaintext -# Replication -role:master -connected_replicas:1 -replica0:ip=10.133.5.21,port=6379,state=online,offset=208037514,lag=1 -master_repl_offset:208037658 -repl_backlog_active:1 -repl_backlog_size:1048576 -repl_backlog_first_byte_offset:206989083 -repl_backlog_histlen:1048576 -``` - -When it's a `replica`, you see details of the primary connection and if -its `up` or `down`: - -```plaintext -# Replication -role:replica -master_host:10.133.1.58 -master_port:6379 -master_link_status:up -master_last_io_seconds_ago:1 -master_sync_in_progress:0 -replica_repl_offset:208096498 -replica_priority:100 -replica_read_only:1 -connected_replicas:0 -master_repl_offset:0 -repl_backlog_active:0 -repl_backlog_size:1048576 -repl_backlog_first_byte_offset:0 -repl_backlog_histlen:0 -``` - -### Troubleshooting Sentinel - -If you get an error like: `Redis::CannotConnectError: No sentinels available.`, -there may be something wrong with your configuration files or it can be related -to [this issue](https://github.com/redis/redis-rb/issues/531). - -You must make sure you are defining the same value in `redis['master_name']` -and `redis['master_password']` as you defined for your sentinel node. - -The way the Redis connector `redis-rb` works with sentinel is a bit -non-intuitive. We try to hide the complexity in omnibus, but it still requires -a few extra configurations. - ---- - -To make sure your configuration is correct: - -1. SSH into your GitLab application server -1. Enter the Rails console: - - ```shell - # For Omnibus installations - sudo gitlab-rails console - - # For source installations - sudo -u git rails console -e production - ``` - -1. Run in the console: - - ```ruby - redis = Redis.new(Gitlab::Redis::SharedState.params) - redis.info - ``` - - Keep this screen open and try to simulate a failover below. - -1. To simulate a failover on primary Redis, SSH into the Redis server and run: - - ```shell - # port must match your primary redis port, and the sleep time must be a few seconds bigger than defined one - redis-cli -h localhost -p 6379 DEBUG sleep 20 - ``` - -1. Then back in the Rails console from the first step, run: - - ```ruby - redis.info - ``` - - You should see a different port after a few seconds delay - (the failover/reconnect time). - -## Troubleshooting Gitaly - -For troubleshooting information, see Gitaly and Gitaly Cluster -[troubleshooting information](../gitaly/troubleshooting.md). - -## Troubleshooting the GitLab Rails application - -- `mount: wrong fs type, bad option, bad superblock on` - -You have not installed the necessary NFS client utilities. See step 1 above. - -- `mount: mount point /var/opt/gitlab/... does not exist` - -This particular directory does not exist on the NFS server. Ensure -the share is exported and exists on the NFS server and try to remount. - -## Troubleshooting Monitoring - -If the monitoring node is not receiving any data, check that the exporters are -capturing data. - -```shell -curl "http[s]://localhost:<EXPORTER LISTENING PORT>/metric" -``` - -or - -```shell -curl "http[s]://localhost:<EXPORTER LISTENING PORT>/-/metric" -``` - -## Troubleshooting PgBouncer - -In case you are experiencing any issues connecting through PgBouncer, the first place to check is always the logs: - -```shell -sudo gitlab-ctl tail pgbouncer -``` - -Additionally, you can check the output from `show databases` in the [administrative console](#pgbouncer-administrative-console). In the output, you would expect to see values in the `host` field for the `gitlabhq_production` database. Additionally, `current_connections` should be greater than 1. - -### PgBouncer administrative console - -As part of Omnibus GitLab, the `gitlab-ctl pgb-console` command is provided to automatically connect to the PgBouncer administrative console. See the [PgBouncer documentation](https://www.pgbouncer.org/usage.html#admin-console) for detailed instructions on how to interact with the console. - -To start a session: - -```shell -sudo gitlab-ctl pgb-console -``` - -The password you are prompted for is the `pgbouncer_user_password` - -To get some basic information about the instance, run - -```shell -pgbouncer=# show databases; show clients; show servers; - name | host | port | database | force_user | pool_size | reserve_pool | pool_mode | max_connections | current_connections ----------------------+-----------+------+---------------------+------------+-----------+--------------+-----------+-----------------+--------------------- - gitlabhq_production | 127.0.0.1 | 5432 | gitlabhq_production | | 100 | 5 | | 0 | 1 - pgbouncer | | 6432 | pgbouncer | pgbouncer | 2 | 0 | statement | 0 | 0 -(2 rows) - - type | user | database | state | addr | port | local_addr | local_port | connect_time | request_time | ptr | link -| remote_pid | tls -------+-----------+---------------------+--------+-----------+-------+------------+------------+---------------------+---------------------+-----------+------ -+------------+----- - C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44590 | 127.0.0.1 | 6432 | 2018-04-24 22:13:10 | 2018-04-24 22:17:10 | 0x12444c0 | -| 0 | - C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44592 | 127.0.0.1 | 6432 | 2018-04-24 22:13:10 | 2018-04-24 22:17:10 | 0x12447c0 | -| 0 | - C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44594 | 127.0.0.1 | 6432 | 2018-04-24 22:13:10 | 2018-04-24 22:17:10 | 0x1244940 | -| 0 | - C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44706 | 127.0.0.1 | 6432 | 2018-04-24 22:14:22 | 2018-04-24 22:16:31 | 0x1244ac0 | -| 0 | - C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44708 | 127.0.0.1 | 6432 | 2018-04-24 22:14:22 | 2018-04-24 22:15:15 | 0x1244c40 | -| 0 | - C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44794 | 127.0.0.1 | 6432 | 2018-04-24 22:15:15 | 2018-04-24 22:15:15 | 0x1244dc0 | -| 0 | - C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44798 | 127.0.0.1 | 6432 | 2018-04-24 22:15:15 | 2018-04-24 22:16:31 | 0x1244f40 | -| 0 | - C | pgbouncer | pgbouncer | active | 127.0.0.1 | 44660 | 127.0.0.1 | 6432 | 2018-04-24 22:13:51 | 2018-04-24 22:17:12 | 0x1244640 | -| 0 | -(8 rows) - - type | user | database | state | addr | port | local_addr | local_port | connect_time | request_time | ptr | link | rem -ote_pid | tls -------+--------+---------------------+-------+-----------+------+------------+------------+---------------------+---------------------+-----------+------+---- ---------+----- - S | gitlab | gitlabhq_production | idle | 127.0.0.1 | 5432 | 127.0.0.1 | 35646 | 2018-04-24 22:15:15 | 2018-04-24 22:17:10 | 0x124dca0 | | - 19980 | -(1 row) -``` - -### Message: `LOG: invalid CIDR mask in address` - -See the suggested fix [in Geo documentation](../geo/replication/troubleshooting.md#message-log--invalid-cidr-mask-in-address). - -### Message: `LOG: invalid IP mask "md5": Name or service not known` - -See the suggested fix [in Geo documentation](../geo/replication/troubleshooting.md#message-log--invalid-ip-mask-md5-name-or-service-not-known). - -## Troubleshooting PostgreSQL with Patroni - -In case you are experiencing any issues connecting through PgBouncer, the first place to check is always the logs for PostgreSQL (which is run through Patroni): - -```shell -sudo gitlab-ctl tail patroni -``` - -### Consul and PostgreSQL with Patroni changes not taking effect - -Due to the potential impacts, `gitlab-ctl reconfigure` only reloads Consul and PostgreSQL, it does not restart the services. However, not all changes can be activated by reloading. - -To restart either service, run `gitlab-ctl restart consul` or `gitlab-ctl restart patroni` respectively. - -For PostgreSQL with Patroni, to prevent the primary node from being failed over automatically, it's safest to stop all secondaries first, then restart the primary and finally restart the secondaries again. - -On the Consul server nodes, it is important to restart the Consul service in a controlled fashion. Read our [Consul documentation](../consul.md#restart-consul) for instructions on how to restart the service. - -### PgBouncer error `ERROR: pgbouncer cannot connect to server` - -You may get this error when running `gitlab-rake gitlab:db:configure` or you -may see the error in the PgBouncer log file. - -```plaintext -PG::ConnectionBad: ERROR: pgbouncer cannot connect to server -``` - -The problem may be that your PgBouncer node's IP address is not included in the -`trust_auth_cidr_addresses` setting in `/etc/gitlab/gitlab.rb` on the database nodes. - -You can confirm that this is the issue by checking the PostgreSQL log on the primary -database node. If you see the following error then `trust_auth_cidr_addresses` -is the problem. - -```plaintext -2018-03-29_13:59:12.11776 FATAL: no pg_hba.conf entry for host "123.123.123.123", user "pgbouncer", database "gitlabhq_production", SSL off -``` - -To fix the problem, add the IP address to `/etc/gitlab/gitlab.rb`. - -```ruby -postgresql['trust_auth_cidr_addresses'] = %w(123.123.123.123/32 <other_cidrs>) -``` - -[Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +<!-- This redirect file can be deleted after <2022-10-24>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
\ No newline at end of file diff --git a/doc/administration/repository_checks.md b/doc/administration/repository_checks.md index 8cfefdb9b56..a97c8611239 100644 --- a/doc/administration/repository_checks.md +++ b/doc/administration/repository_checks.md @@ -7,8 +7,18 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Repository checks **(FREE SELF)** You can use [`git fsck`](https://git-scm.com/docs/git-fsck) to verify the integrity of all data -committed to a repository. GitLab administrators can trigger this check for a project using the -GitLab UI: +committed to a repository. GitLab administrators can: + +- Manually trigger this check for a project, using the GitLab UI. +- Schedule this check to run automatically for all projects. +- Run this check from the command line. +- Run a [Rake task](raketasks/check.md#repository-integrity) for checking Git repositories, which can be used to run + `git fsck` against all repositories and generate repository checksums, as a way to compare repositories on different + servers. + +## Check a project's repository using GitLab UI + +To check a project's repository using GitLab UI: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Overview > Projects**. @@ -18,9 +28,7 @@ GitLab UI: The checks run asynchronously so it may take a few minutes before the check result is visible on the project page in the Admin Area. If the checks fail, see [what to do](#what-to-do-if-a-check-failed). -This setting is off by default, because it can cause many false alarms. - -## Enable periodic checks +## Enable repository checks for all projects Instead of checking repositories manually, GitLab can be configured to run the checks periodically: @@ -45,13 +53,31 @@ the start of Sunday. Repositories with known check failures can be found at `/admin/projects?last_repository_check_failed=1`. +## Run a check using the command line + +You can run [`git fsck`](https://git-scm.com/docs/git-fsck) using the command line on repositories on +[Gitaly servers](gitaly/index.md). To locate the repositories: + +1. Go to the storage location for repositories: + - For Omnibus GitLab installations, repositories are stored in the `/var/opt/gitlab/git-data/repositories` directory + by default. + - For GitLab Helm chart installations, repositories are stored in the `/home/git/repositories` directory inside the + Gitaly pod by default. +1. [Identify the subdirectory that contains the repository](repository_storage_types.md#from-project-name-to-hashed-path) + that you need to check. +1. Run the check. For example: + + ```shell + sudo /opt/gitlab/embedded/bin/git -C /var/opt/gitlab/git-data/repositories/@hashed/0b/91/0b91...f9.git fsck + ``` + ## What to do if a check failed -If a repository check fails, locate the error in the [`repocheck.log` file](logs.md#repochecklog) on -disk at: +If a repository check fails, locate the error in the [`repocheck.log` file](logs/index.md#repochecklog) on disk at: - `/var/log/gitlab/gitlab-rails` for Omnibus GitLab installations. - `/home/git/gitlab/log` for installations from source. +- `/var/log/gitlab` in the Sidekiq pod for GitLab Helm chart installations. If periodic repository checks cause false alarms, you can clear all repository check states: @@ -59,23 +85,3 @@ If periodic repository checks cause false alarms, you can clear all repository c 1. On the left sidebar, select **Settings > Repository** (`/admin/application_settings/repository`). 1. Expand the **Repository maintenance** section. 1. Select **Clear all repository checks**. - -## Run a check using the command line - -You can run [`git fsck`](https://git-scm.com/docs/git-fsck) using the command line on repositories -on [Gitaly servers](gitaly/index.md). To locate the repositories: - -1. Go to the storage location for repositories. For Omnibus GitLab installations, repositories are - stored by default in the `/var/opt/gitlab/git-data/repositories` directory. -1. [Identify the subdirectory that contains the repository](repository_storage_types.md#from-project-name-to-hashed-path) - that you need to check. - -To run a check (for example): - -```shell -sudo /opt/gitlab/embedded/bin/git -C /var/opt/gitlab/git-data/repositories/@hashed/0b/91/0b91...f9.git fsck -``` - -You can also run [Rake tasks](raketasks/check.md#repository-integrity) for checking Git -repositories, which can be used to run `git fsck` against all repositories and generate repository -checksums, as a way to compare repositories on different servers. diff --git a/doc/administration/restart_gitlab.md b/doc/administration/restart_gitlab.md index 6ac031ea6bd..6625039504a 100644 --- a/doc/administration/restart_gitlab.md +++ b/doc/administration/restart_gitlab.md @@ -102,8 +102,8 @@ depend on those files. ## Installations from source -If you have followed the official installation guide to [install GitLab from -source](../install/installation.md), run the following command to restart GitLab: +If you have followed the official installation guide to +[install GitLab from source](../install/installation.md), run the following command to restart GitLab: ```shell # For systems running systemd diff --git a/doc/administration/sidekiq.md b/doc/administration/sidekiq.md index fc24c764330..01f83f98607 100644 --- a/doc/administration/sidekiq.md +++ b/doc/administration/sidekiq.md @@ -1,385 +1,11 @@ --- -stage: Systems -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'sidekiq/index.md' +remove_date: '2022-11-11' --- -# Configure an external Sidekiq instance **(FREE SELF)** +This document was moved to [another location](sidekiq/index.md). -You can configure an external Sidekiq instance by using the Sidekiq that's bundled in the GitLab package. Sidekiq requires connection to the Redis, -PostgreSQL, and Gitaly instances. - -## Configure TCP access for PostgreSQL, Gitaly, and Redis - -By default, GitLab uses UNIX sockets and is not set up to communicate via TCP. To change this: - -1. Edit the `/etc/gitlab/gitlab.rb` file on your GitLab instance, and add the following: - - ```ruby - - ## PostgreSQL - - # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value - postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' - postgresql['listen_address'] = '0.0.0.0' - postgresql['port'] = 5432 - - # Add the Sidekiq nodes to PostgreSQL's trusted addresses. - # In the following example, 10.10.1.30/32 is the private IP - # of the Sidekiq server. - postgresql['md5_auth_cidr_addresses'] = %w(127.0.0.1/32 10.10.1.30/32) - postgresql['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.10.1.30/32) - - ## Gitaly - - # Make Gitaly accept connections on all network interfaces - gitaly['listen_addr'] = "0.0.0.0:8075" - ## Set up the Gitaly token as a form of authentication since you are accessing Gitaly over the network - ## https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#about-the-gitaly-token - gitaly['auth_token'] = 'abc123secret' - praefect['auth_token'] = 'abc123secret' - gitlab_rails['gitaly_token'] = 'abc123secret' - - ## Redis configuration - - redis['bind'] = '0.0.0.0' - redis['port'] = 6379 - # Password to Authenticate Redis - redis['password'] = 'redis-password-goes-here' - gitlab_rails['redis_password'] = 'redis-password-goes-here' - - gitlab_rails['auto_migrate'] = false - ``` - -1. Run `reconfigure`: - - ```shell - sudo gitlab-ctl reconfigure - ``` - -1. Restart the `PostgreSQL` server: - - ```shell - sudo gitlab-ctl restart postgresql - ``` - -1. After the restart, set `auto_migrate` to `true` or comment to use the default settings: - - ```ruby - gitlab_rails['auto_migrate'] = true - ``` - -1. Run `reconfigure` again: - - ```shell - sudo gitlab-ctl reconfigure - ``` - -## Set up Sidekiq instance - -1. SSH into the Sidekiq server. - -1. Confirm that you can access the PostgreSQL, Gitaly, and Redis ports: - - ```shell - telnet <GitLab host> 5432 # PostgreSQL - telnet <GitLab host> 8075 # Gitaly - telnet <GitLab host> 6379 # Redis - ``` - -1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package - using steps 1 and 2. **Do not complete any other steps.** - -1. Copy the `/etc/gitlab/gitlab.rb` file from the GitLab instance and add the following settings. Make sure - to replace them with your values: - -<!-- -Updates to example must be made at: -- https://gitlab.com/gitlab-org/gitlab/blob/master/doc/administration/sidekiq.md -- all reference architecture pages ---> - - ```ruby - ######################################## - ##### Services Disabled ### - ######################################## - # - # When running GitLab on just one server, you have a single `gitlab.rb` - # to enable all services you want to run. - # When running GitLab on N servers, you have N `gitlab.rb` files. - # Enable only the services you want to run on each - # specific server, while disabling all others. - # - gitaly['enable'] = false - postgresql['enable'] = false - redis['enable'] = false - nginx['enable'] = false - puma['enable'] = false - gitlab_workhorse['enable'] = false - prometheus['enable'] = false - alertmanager['enable'] = false - grafana['enable'] = false - gitlab_exporter['enable'] = false - gitlab_kas['enable'] = false - - ## - ## To maintain uniformity of links across nodes, the - ## `external_url` on the Sidekiq server should point to the external URL that users - ## use to access GitLab. This can be either: - ## - ## - The `external_url` set on your application server. - ## - The URL of a external load balancer, which routes traffic to the GitLab application server. - ## - external_url 'https://gitlab.example.com' - - # Configure the gitlab-shell API callback URL. Without this, `git push` will - # fail. This can be your 'front door' GitLab URL or an internal load - # balancer. - gitlab_rails['internal_api_url'] = 'GITLAB_URL' - gitlab_shell['secret_token'] = 'SHELL_TOKEN' - - ######################################## - #### Redis ### - ######################################## - - ## Must be the same in every sentinel node. - redis['master_name'] = 'gitlab-redis' # Required if you have setup redis cluster - ## The same password for Redis authentication you set up for the master node. - redis['master_password'] = '<redis_master_password>' - - ### If redis is running on the main Gitlab instance and you have opened the TCP port as above add the following - gitlab_rails['redis_host'] = '<gitlab_host>' - gitlab_rails['redis_port'] = 6379 - - ####################################### - ### Gitaly ### - ####################################### - - ## Replace <gitaly_token> with the one you set up, see - ## https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#about-the-gitaly-token - git_data_dirs({ - "default" => { - "gitaly_address" => "tcp://<gitlab_host>:8075", - "gitaly_token" => "<gitaly_token>" - } - }) - - ####################################### - ### Postgres ### - ####################################### - - # Replace <database_host> and <database_password> - gitlab_rails['db_host'] = '<database_host>' - gitlab_rails['db_port'] = '5432' - gitlab_rails['db_password'] = '<database_password>' - ## Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false - - ####################################### - ### Sidekiq configuration ### - ####################################### - sidekiq['enable'] = true - sidekiq['listen_address'] = "0.0.0.0" - - ## Set number of Sidekiq queue processes to the same number as available CPUs - sidekiq['queue_groups'] = ['*'] * 4 - - ## Set number of Sidekiq threads per queue process to the recommend number of 10 - sidekiq['max_concurrency'] = 10 - ``` - -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the GitLab instance and replace the file in the Sidekiq instance. - -1. Reconfigure GitLab: - - ```shell - sudo gitlab-ctl reconfigure - ``` - -1. Restart the Sidekiq instance after completing the process and finishing the database migrations. - -## Configure multiple Sidekiq nodes with shared storage - -If you run multiple Sidekiq nodes with a shared file storage, such as NFS, you must -specify the UIDs and GIDs to ensure they match between servers. Specifying the UIDs -and GIDs prevents permissions issues in the file system. This advice is similar to the -[advice for Geo setups](geo/replication/multiple_servers.md#step-4-configure-the-frontend-application-nodes-on-the-geo-secondary-site). - -To set up multiple Sidekiq nodes: - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - user['uid'] = 9000 - user['gid'] = 9000 - web_server['uid'] = 9001 - web_server['gid'] = 9001 - registry['uid'] = 9002 - registry['gid'] = 9002 - ``` - -1. Reconfigure GitLab: - - ```shell - sudo gitlab-ctl reconfigure - ``` - -## Configure the Container Registry when using an external Sidekiq - -If you're using the Container Registry and it's running on a different -node than Sidekiq, follow the steps below. - -1. Edit `/etc/gitlab/gitlab.rb`, and configure the registry URL: - - ```ruby - registry_external_url 'https://registry.example.com' - gitlab_rails['registry_api_url'] = "https://registry.example.com" - ``` - -1. Reconfigure GitLab: - - ```shell - sudo gitlab-ctl reconfigure - ``` - -1. In the instance where Container Registry is hosted, copy the `registry.key` - file to the Sidekiq node. - -## Configure the Sidekiq metrics server - -If you want to collect Sidekiq metrics, enable the Sidekiq metrics server. -To make metrics available from `localhost:8082/metrics`: - -To configure the metrics server: - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - sidekiq['metrics_enabled'] = true - sidekiq['listen_address'] = "localhost" - sidekiq['listen_port'] = "8082" - - # Optionally log all the metrics server logs to log/sidekiq_exporter.log - sidekiq['exporter_log_enabled'] = true - ``` - -1. Reconfigure GitLab: - - ```shell - sudo gitlab-ctl reconfigure - ``` - -### Enable HTTPS - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/364771) in GitLab 15.2. - -To serve metrics via HTTPS instead of HTTP, enable TLS in the exporter settings: - -1. Edit `/etc/gitlab/gitlab.rb` to add (or find and uncomment) the following lines: - - ```ruby - sidekiq['exporter_tls_enabled'] = true - sidekiq['exporter_tls_cert_path'] = "/path/to/certificate.pem" - sidekiq['exporter_tls_key_path'] = "/path/to/private-key.pem" - ``` - -1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) - for the changes to take effect. - -When TLS is enabled, the same `port` and `address` are used as described above. -The metrics server cannot serve both HTTP and HTTPS at the same time. - -## Configure health checks - -If you use health check probes to observe Sidekiq, enable the Sidekiq health check server. -To make health checks available from `localhost:8092`: - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - sidekiq['health_checks_enabled'] = true - sidekiq['health_checks_listen_address'] = "localhost" - sidekiq['health_checks_listen_port'] = "8092" - ``` - -1. Reconfigure GitLab: - - ```shell - sudo gitlab-ctl reconfigure - ``` - -For more information about health checks, see the [Sidekiq health check page](sidekiq_health_check.md). - -## Configure LDAP and user or group synchronization - -If you use LDAP for user and group management, you must add the LDAP configuration to your Sidekiq node as well as the LDAP -synchronization worker. If the LDAP configuration and LDAP synchronization worker are not applied to your Sidekiq node, -users and groups are not automatically synchronized. - -For more information about configuring LDAP for GitLab, see: - -- [GitLab LDAP configuration documentation](auth/ldap/index.md#configure-ldap) -- [LDAP synchronization documentation](auth/ldap/ldap_synchronization.md#adjust-ldap-user-sync-schedule) - -To enable LDAP with the synchronization worker for Sidekiq: - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_rails['ldap_enabled'] = true - gitlab_rails['prevent_ldap_sign_in'] = false - gitlab_rails['ldap_servers'] = { - 'main' => { - 'label' => 'LDAP', - 'host' => 'ldap.mydomain.com', - 'port' => 389, - 'uid' => 'sAMAccountName', - 'encryption' => 'simple_tls', - 'verify_certificates' => true, - 'bind_dn' => '_the_full_dn_of_the_user_you_will_bind_with', - 'password' => '_the_password_of_the_bind_user', - 'tls_options' => { - 'ca_file' => '', - 'ssl_version' => '', - 'ciphers' => '', - 'cert' => '', - 'key' => '' - }, - 'timeout' => 10, - 'active_directory' => true, - 'allow_username_or_email_login' => false, - 'block_auto_created_users' => false, - 'base' => 'dc=example,dc=com', - 'user_filter' => '', - 'attributes' => { - 'username' => ['uid', 'userid', 'sAMAccountName'], - 'email' => ['mail', 'email', 'userPrincipalName'], - 'name' => 'cn', - 'first_name' => 'givenName', - 'last_name' => 'sn' - }, - 'lowercase_usernames' => false, - - # Enterprise Edition only - # https://docs.gitlab.com/ee/administration/auth/ldap/ldap_synchronization.html - 'group_base' => '', - 'admin_group' => '', - 'external_groups' => [], - 'sync_ssh_keys' => false - } - } - gitlab_rails['ldap_sync_worker_cron'] = "0 */12 * * *" - ``` - -1. Reconfigure GitLab: - - ```shell - sudo gitlab-ctl reconfigure - ``` - -## Related topics - -- [Extra Sidekiq processes](operations/extra_sidekiq_processes.md) -- [Extra Sidekiq routing](operations/extra_sidekiq_routing.md) -- [Using the GitLab-Sidekiq chart](https://docs.gitlab.com/charts/charts/gitlab/sidekiq/) -- [Sidekiq health checks](sidekiq_health_check.md) +<!-- This redirect file can be deleted after <2022-11-11>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/sidekiq/extra_sidekiq_processes.md b/doc/administration/sidekiq/extra_sidekiq_processes.md new file mode 100644 index 00000000000..1cd3771c94d --- /dev/null +++ b/doc/administration/sidekiq/extra_sidekiq_processes.md @@ -0,0 +1,362 @@ +--- +stage: Systems +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Run multiple Sidekiq processes **(FREE SELF)** + +GitLab allows you to start multiple Sidekiq processes. +These processes can be used to consume a dedicated set +of queues. This can be used to ensure certain queues always have dedicated +workers, no matter the number of jobs to be processed. + +NOTE: +The information in this page applies only to Omnibus GitLab. + +## Available Sidekiq queues + +For a list of the existing Sidekiq queues, check the following files: + +- [Queues for both GitLab Community and Enterprise Editions](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/all_queues.yml) +- [Queues for GitLab Enterprise Editions only](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/all_queues.yml) + +Each entry in the above files represents a queue on which Sidekiq processes +can be started. + +## Start multiple processes + +> - [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4006) in GitLab 12.10, starting multiple processes with Sidekiq cluster. +> - [Sidekiq cluster moved](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181) to GitLab Free in 12.10. +> - [Sidekiq cluster became default](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4140) in GitLab 13.0. + +When starting multiple processes, the number of processes should +equal (and **not** exceed) the number of CPU cores you want to +dedicate to Sidekiq. Each Sidekiq process can use only 1 CPU +core, subject to the available workload and concurrency settings. + +To start multiple processes: + +1. Using the `sidekiq['queue_groups']` array setting, specify how many processes to + create using `sidekiq-cluster` and which queue they should handle. + Each item in the array equates to one additional Sidekiq + process, and values in each item determine the queues it works on. + + For example, the following setting creates three Sidekiq processes, one to run on + `elastic_commit_indexer`, one to run on `mailers`, and one process running on all queues: + + ```ruby + sidekiq['queue_groups'] = [ + "elastic_commit_indexer", + "mailers", + "*" + ] + ``` + + To have an additional Sidekiq process handle multiple queues, add multiple + queue names to its item delimited by commas. For example: + + ```ruby + sidekiq['queue_groups'] = [ + "elastic_commit_indexer, elastic_association_indexer", + "mailers", + "*" + ] + ``` + + [In GitLab 12.9](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26594) and + later, the special queue name `*` means all queues. This starts two + processes, each handling all queues: + + ```ruby + sidekiq['queue_groups'] = [ + "*", + "*" + ] + ``` + + `*` cannot be combined with concrete queue names - `*, mailers` + just handles the `mailers` queue. + + When `sidekiq-cluster` is only running on a single node, make sure that at least + one process is running on all queues using `*`. This ensures a process + automatically picks up jobs in queues created in the future, + including queues that have dedicated processes. + + If `sidekiq-cluster` is running on more than one node, you can also use + [`--negate`](#negate-settings) and list all the queues that are already being + processed. + +1. Save the file and reconfigure GitLab for the changes to take effect: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +To view the Sidekiq processes in GitLab: + +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Monitoring > Background Jobs**. + +## Negate settings + +To have the Sidekiq process work on every queue **except** the ones +you list. In this example, we exclude all import-related jobs from a Sidekiq node: + +1. Edit `/etc/gitlab/gitlab.rb` and add: + + ```ruby + sidekiq['negate'] = true + sidekiq['queue_selector'] = true + sidekiq['queue_groups'] = [ + "feature_category=importers" + ] + ``` + +1. Save the file and reconfigure GitLab for the changes to take effect: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +## Queue selector + +> - [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/45) in GitLab 12.8. +> - [Sidekiq cluster, including queue selector, moved](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181) to GitLab Free in 12.10. +> - [Renamed from `experimental_queue_selector` to `queue_selector`](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/147) in GitLab 13.6. + +In addition to selecting queues by name, as above, the `queue_selector` option +allows queue groups to be selected in a more general way using a +[worker matching query](extra_sidekiq_routing.md#worker-matching-query). After `queue_selector` +is set, all `queue_groups` must follow the aforementioned syntax. + +In `/etc/gitlab/gitlab.rb`: + +```ruby +sidekiq['enable'] = true +sidekiq['queue_selector'] = true +sidekiq['queue_groups'] = [ + # Run all non-CPU-bound queues that are high urgency + 'resource_boundary!=cpu&urgency=high', + # Run all continuous integration and pages queues that are not high urgency + 'feature_category=continuous_integration,pages&urgency!=high', + # Run all queues + '*' +] +``` + +## Ignore all import queues + +When [importing from GitHub](../../user/project/import/github.md) or +other sources, Sidekiq might use all of its resources to perform those +operations. To set up two separate `sidekiq-cluster` processes, where +one only processes imports and the other processes all other queues: + +1. Edit `/etc/gitlab/gitlab.rb` and add: + + ```ruby + sidekiq['enable'] = true + sidekiq['queue_selector'] = true + sidekiq['queue_groups'] = [ + "feature_category=importers", + "feature_category!=importers" + ] + ``` + +1. Save the file and reconfigure GitLab for the changes to take effect: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +## Number of threads + +By default each process defined under `sidekiq` starts with a +number of threads that equals the number of queues, plus one spare thread. +For example, a process that handles the `process_commit` and `post_receive` +queues uses three threads in total. + +These thread run inside a single Ruby process, and each process +can only use a single CPU core. The usefulness of threading depends +on the work having some external dependencies to wait on, like database queries or +HTTP requests. Most Sidekiq deployments benefit from this threading, and when +running fewer queues in a process, increasing the thread count might be +even more desirable to make the most effective use of CPU resources. + +### Manage thread counts explicitly + +The correct maximum thread count (also called concurrency) depends on the workload. +Typical values range from `1` for highly CPU-bound tasks to `15` or higher for mixed +low-priority work. A reasonable starting range is `15` to `25` for a non-specialized +deployment. + +You can find example values used by GitLab.com by searching for `concurrency:` in +[the Helm charts](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/blob/master/releases/gitlab/values/gprd.yaml.gotmpl). +The values vary according to the work each specific deployment of Sidekiq does. +Any other specialized deployments with processes dedicated to specific queues should +have the concurrency tuned according to: +have the concurrency tuned according to: + +- The CPU usage of each type of process. +- The throughput achieved. + +Each thread requires a Redis connection, so adding threads may increase Redis +latency and potentially cause client timeouts. See the +[Sidekiq documentation about Redis](https://github.com/mperham/sidekiq/wiki/Using-Redis) for more +details. + +#### When running Sidekiq cluster (default) + +Running Sidekiq cluster is the default in GitLab 13.0 and later. + +1. Edit `/etc/gitlab/gitlab.rb` and add: + + ```ruby + sidekiq['min_concurrency'] = 15 + sidekiq['max_concurrency'] = 25 + ``` + +1. Save the file and reconfigure GitLab for the changes to take effect: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +`min_concurrency` and `max_concurrency` are independent; one can be set without +the other. Setting `min_concurrency` to `0` disables the limit. + +For each queue group, let `N` be one more than the number of queues. The +concurrency is set to: + +1. `N`, if it's between `min_concurrency` and `max_concurrency`. +1. `max_concurrency`, if `N` exceeds this value. +1. `min_concurrency`, if `N` is less than this value. + +If `min_concurrency` is equal to `max_concurrency`, then this value is used +regardless of the number of queues. + +When `min_concurrency` is greater than `max_concurrency`, it is treated as +being equal to `max_concurrency`. + +#### When running a single Sidekiq process + +Running a single Sidekiq process is the default in GitLab 12.10 and earlier. + +WARNING: +Running Sidekiq directly was removed in GitLab +[14.0](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/240). + +1. Edit `/etc/gitlab/gitlab.rb` and add: + + ```ruby + sidekiq['cluster'] = false + sidekiq['concurrency'] = 25 + ``` + +1. Save the file and reconfigure GitLab for the changes to take effect: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +This sets the concurrency (number of threads) for the Sidekiq process. + +## Modify the check interval + +To modify `sidekiq-cluster`'s health check interval for the additional Sidekiq processes: + +1. Edit `/etc/gitlab/gitlab.rb` and add (the value can be any integer number of seconds): + + ```ruby + sidekiq['interval'] = 5 + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +## Troubleshoot using the CLI + +WARNING: +It's recommended to use `/etc/gitlab/gitlab.rb` to configure the Sidekiq processes. +If you experience a problem, you should contact GitLab support. Use the command +line at your own risk. + +For debugging purposes, you can start extra Sidekiq processes by using the command +`/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster`. This command +takes arguments using the following syntax: + +```shell +/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster [QUEUE,QUEUE,...] [QUEUE, ...] +``` + +Each separate argument denotes a group of queues that have to be processed by a +Sidekiq process. Multiple queues can be processed by the same process by +separating them with a comma instead of a space. + +Instead of a queue, a queue namespace can also be provided, to have the process +automatically listen on all queues in that namespace without needing to +explicitly list all the queue names. For more information about queue namespaces, +see the relevant section in the +[Sidekiq development documentation](../../development/sidekiq/index.md#queue-namespaces). + +For example, say you want to start 2 extra processes: one to process the +`process_commit` queue, and one to process the `post_receive` queue. This can be +done as follows: + +```shell +/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster process_commit post_receive +``` + +If you instead want to start one process processing both queues, you'd use the +following syntax: + +```shell +/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster process_commit,post_receive +``` + +If you want to have one Sidekiq process dealing with the `process_commit` and +`post_receive` queues, and one process to process the `gitlab_shell` queue, +you'd use the following: + +```shell +/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster process_commit,post_receive gitlab_shell +``` + +### Monitor the `sidekiq-cluster` command + +The `sidekiq-cluster` command does not terminate once it has started the desired +amount of Sidekiq processes. Instead, the process continues running and +forwards any signals to the child processes. This allows you to stop all +Sidekiq processes as you send a signal to the `sidekiq-cluster` process, +instead of having to send it to the individual processes. + +If the `sidekiq-cluster` process crashes or receives a `SIGKILL`, the child +processes terminate themselves after a few seconds. This ensures you don't +end up with zombie Sidekiq processes. + +This allows you to monitor the processes by hooking up +`sidekiq-cluster` to your supervisor of choice (for example, runit). + +If a child process died the `sidekiq-cluster` command signals all remaining +process to terminate, then terminate itself. This removes the need for +`sidekiq-cluster` to re-implement complex process monitoring/restarting code. +Instead you should make sure your supervisor restarts the `sidekiq-cluster` +process whenever necessary. + +### PID files + +The `sidekiq-cluster` command can store its PID in a file. By default no PID +file is written, but this can be changed by passing the `--pidfile` option to +`sidekiq-cluster`. For example: + +```shell +/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster --pidfile /var/run/gitlab/sidekiq_cluster.pid process_commit +``` + +Keep in mind that the PID file contains the PID of the `sidekiq-cluster` +command and not the PIDs of the started Sidekiq processes. + +### Environment + +The Rails environment can be set by passing the `--environment` flag to the +`sidekiq-cluster` command, or by setting `RAILS_ENV` to a non-empty value. The +default value can be found in `/opt/gitlab/etc/gitlab-rails/env/RAILS_ENV`. diff --git a/doc/administration/sidekiq/extra_sidekiq_routing.md b/doc/administration/sidekiq/extra_sidekiq_routing.md new file mode 100644 index 00000000000..ebbf4059290 --- /dev/null +++ b/doc/administration/sidekiq/extra_sidekiq_routing.md @@ -0,0 +1,180 @@ +--- +stage: Systems +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Queue routing rules **(FREE SELF)** + +When the number of Sidekiq jobs increases to a certain scale, the system faces +some scalability issues. One of them is that the length of the queue tends to get +longer. High-urgency jobs have to wait longer until other less urgent jobs +finish. This head-of-line blocking situation may eventually affect the +responsiveness of the system, especially critical actions. In another scenario, +the performance of some jobs is degraded due to other long running or CPU-intensive jobs +(computing or rendering ones) in the same machine. + +To counter the aforementioned issues, one effective solution is to split +Sidekiq jobs into different queues and assign machines handling each queue +exclusively. For example, all CPU-intensive jobs could be routed to the +`cpu-bound` queue and handled by a fleet of CPU optimized instances. The queue +topology differs between companies depending on the workloads and usage +patterns. Therefore, GitLab supports a flexible mechanism for the +administrator to route the jobs based on their characteristics. + +As an alternative to [Queue selector](extra_sidekiq_processes.md#queue-selector), which +configures Sidekiq cluster to listen to a specific set of workers or queues, +GitLab also supports routing a job from a worker to the desired queue when it +is scheduled. Sidekiq clients try to match a job against a configured list of +routing rules. Rules are evaluated from first to last, and as soon as we find a +match for a given worker we stop processing for that worker (first match wins). +If the worker doesn't match any rule, it falls back to the queue name generated +from the worker name. + +By default, if the routing rules are not configured (or denoted with an empty +array), all the jobs are routed to the queue generated from the worker name. + +## Example configuration + +In `/etc/gitlab/gitlab.rb`: + +```ruby +sidekiq['routing_rules'] = [ + # Do not re-route workers that require their own queue + ['tags=needs_own_queue', nil], + # Route all non-CPU-bound workers that are high urgency to `high-urgency` queue + ['resource_boundary!=cpu&urgency=high', 'high-urgency'], + # Route all database, gitaly and global search workers that are throttled to `throttled` queue + ['feature_category=database,gitaly,global_search&urgency=throttled', 'throttled'], + # Route all workers having contact with outside work to a `network-intenstive` queue + ['has_external_dependencies=true|feature_category=hooks|tags=network', 'network-intensive'], + # Route all import workers to the queues generated by the worker name, for + # example, JiraImportWorker to `jira_import`, SVNWorker to `svn_worker` + ['feature_category=import', nil], + # Wildcard matching, route the rest to `default` queue + ['*', 'default'] +] +``` + +The routing rules list is an order-matter array of tuples of query and +corresponding queue: + +- The query is following a [worker matching query](#worker-matching-query) syntax. +- The `<queue_name>` must be a valid Sidekiq queue name. If the queue name + is `nil`, or an empty string, the worker is routed to the queue generated + by the name of the worker instead. + +The query supports wildcard matching `*`, which matches all workers. As a +result, the wildcard query must stay at the end of the list or the rules after it +are ignored. + +NOTE: +Mixing queue routing rules and queue selectors requires care to +ensure all jobs that are scheduled and picked up by appropriate Sidekiq +workers. + +## Worker matching query + +GitLab provides a query syntax to match a worker based on its +attributes. This query syntax is employed by both +[Queue routing rules](#queue-routing-rules) and +[Queue selector](extra_sidekiq_processes.md#queue-selector). A query includes two +components: + +- Attributes that can be selected. +- Operators used to construct a query. + +### Available attributes + +> [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/261) in GitLab 13.1 (`tags`). + +Queue matching query works upon the worker attributes, described in +[Sidekiq style guide](../../development/sidekiq/index.md). We support querying +based on a subset of worker attributes: + +- `feature_category` - the + [GitLab feature category](https://about.gitlab.com/direction/maturity/#category-maturity) the + queue belongs to. For example, the `merge` queue belongs to the + `source_code_management` category. +- `has_external_dependencies` - whether or not the queue connects to external + services. For example, all importers have this set to `true`. +- `urgency` - how important it is that this queue's jobs run + quickly. Can be `high`, `low`, or `throttled`. For example, the + `authorized_projects` queue is used to refresh user permissions, and + is `high` urgency. +- `worker_name` - the worker name. Use this attribute to select a specific worker. +- `name` - the queue name generated from the worker name. Use this attribute to select a specific queue. Because this is generated from + the worker name, it does not change based on the result of other routing + rules. +- `resource_boundary` - if the queue is bound by `cpu`, `memory`, or + `unknown`. For example, the `ProjectExportWorker` is memory bound as it has + to load data in memory before saving it for export. +- `tags` - short-lived annotations for queues. These are expected to frequently + change from release to release, and may be removed entirely. + +`has_external_dependencies` is a boolean attribute: only the exact +string `true` is considered true, and everything else is considered +false. + +`tags` is a set, which means that `=` checks for intersecting sets, and +`!=` checks for disjoint sets. For example, `tags=a,b` selects queues +that have tags `a`, `b`, or both. `tags!=a,b` selects queues that have +neither of those tags. + +The attributes of each worker are hard-coded in the source code. For +convenience, we generate a +[list of all available attributes in GitLab Community Edition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/all_queues.yml) +and a +[list of all available attributes in GitLab Enterprise Edition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/all_queues.yml). + +### Available operators + +`queue_selector` supports the following operators, listed from highest +to lowest precedence: + +- `|` - the logical `OR` operator. For example, `query_a|query_b` (where `query_a` + and `query_b` are queries made up of the other operators here) includes + queues that match either query. +- `&` - the logical `AND` operator. For example, `query_a&query_b` (where + `query_a` and `query_b` are queries made up of the other operators here) will + only include queues that match both queries. +- `!=` - the `NOT IN` operator. For example, `feature_category!=issue_tracking` + excludes all queues from the `issue_tracking` feature category. +- `=` - the `IN` operator. For example, `resource_boundary=cpu` includes all + queues that are CPU bound. +- `,` - the concatenate set operator. For example, + `feature_category=continuous_integration,pages` includes all queues from + either the `continuous_integration` category or the `pages` category. This + example is also possible using the OR operator, but allows greater brevity, as + well as being lower precedence. + +The operator precedence for this syntax is fixed: it's not possible to make `AND` +have higher precedence than `OR`. + +[In GitLab 12.9](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26594) and +later, as with the standard queue group syntax above, a single `*` as the +entire queue group selects all queues. + +### Migration + +After the Sidekiq routing rules are changed, administrators must take care +with the migration to avoid losing jobs entirely, especially in a system with +long queues of jobs. The migration can be done by following the migration steps +mentioned in [Sidekiq job migration](sidekiq_job_migration.md) + +### Workers that cannot be migrated + +Some workers cannot share a queue with other workers - typically because +they check the size of their own queue - and so must be excluded from +this process. We recommend excluding these from any further worker +routing by adding a rule to keep them in their own queue, for example: + +```ruby +sidekiq['routing_rules'] = [ + ['tags=needs_own_queue', nil], + # ... +] +``` + +These queues must also be included in at least one +[Sidekiq queue group](extra_sidekiq_processes.md#start-multiple-processes). diff --git a/doc/administration/troubleshooting/img/sidekiq_flamegraph.png b/doc/administration/sidekiq/img/sidekiq_flamegraph.png Binary files differindex 89d6e8da3ce..89d6e8da3ce 100644 --- a/doc/administration/troubleshooting/img/sidekiq_flamegraph.png +++ b/doc/administration/sidekiq/img/sidekiq_flamegraph.png diff --git a/doc/administration/sidekiq/index.md b/doc/administration/sidekiq/index.md new file mode 100644 index 00000000000..f4e67fcb6dd --- /dev/null +++ b/doc/administration/sidekiq/index.md @@ -0,0 +1,403 @@ +--- +stage: Systems +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Configure an external Sidekiq instance **(FREE SELF)** + +You can configure an external Sidekiq instance by using the Sidekiq that's bundled in the GitLab package. Sidekiq requires connection to the Redis, +PostgreSQL, and Gitaly instances. + +## Configure TCP access for PostgreSQL, Gitaly, and Redis + +By default, GitLab uses UNIX sockets and is not set up to communicate via TCP. To change this: + +1. Edit the `/etc/gitlab/gitlab.rb` file on your GitLab instance, and add the following: + + ```ruby + + ## PostgreSQL + + # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value + postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' + postgresql['listen_address'] = '0.0.0.0' + postgresql['port'] = 5432 + + # Add the Sidekiq nodes to PostgreSQL's trusted addresses. + # In the following example, 10.10.1.30/32 is the private IP + # of the Sidekiq server. + postgresql['md5_auth_cidr_addresses'] = %w(127.0.0.1/32 10.10.1.30/32) + postgresql['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.10.1.30/32) + + ## Gitaly + + # Make Gitaly accept connections on all network interfaces + gitaly['listen_addr'] = "0.0.0.0:8075" + ## Set up the Gitaly token as a form of authentication since you are accessing Gitaly over the network + ## https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#about-the-gitaly-token + gitaly['auth_token'] = 'abc123secret' + praefect['auth_token'] = 'abc123secret' + gitlab_rails['gitaly_token'] = 'abc123secret' + + ## Redis configuration + + redis['bind'] = '0.0.0.0' + redis['port'] = 6379 + # Password to Authenticate Redis + redis['password'] = 'redis-password-goes-here' + gitlab_rails['redis_password'] = 'redis-password-goes-here' + + gitlab_rails['auto_migrate'] = false + ``` + +1. Run `reconfigure`: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. Restart the `PostgreSQL` server: + + ```shell + sudo gitlab-ctl restart postgresql + ``` + +1. After the restart, set `auto_migrate` to `true` or comment to use the default settings: + + ```ruby + gitlab_rails['auto_migrate'] = true + ``` + +1. Run `reconfigure` again: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +## Set up Sidekiq instance + +1. SSH into the Sidekiq server. + +1. Confirm that you can access the PostgreSQL, Gitaly, and Redis ports: + + ```shell + telnet <GitLab host> 5432 # PostgreSQL + telnet <GitLab host> 8075 # Gitaly + telnet <GitLab host> 6379 # Redis + ``` + +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package + using steps 1 and 2. **Do not complete any other steps.** + +1. Copy the `/etc/gitlab/gitlab.rb` file from the GitLab instance and add the following settings. Make sure + to replace them with your values: + +<!-- +Updates to example must be made at: +- https://gitlab.com/gitlab-org/gitlab/blob/master/doc/administration/sidekiq.md +- all reference architecture pages +--> + + ```ruby + ######################################## + ##### Services Disabled ### + ######################################## + # + # When running GitLab on just one server, you have a single `gitlab.rb` + # to enable all services you want to run. + # When running GitLab on N servers, you have N `gitlab.rb` files. + # Enable only the services you want to run on each + # specific server, while disabling all others. + # + gitaly['enable'] = false + postgresql['enable'] = false + redis['enable'] = false + nginx['enable'] = false + puma['enable'] = false + gitlab_workhorse['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false + gitlab_exporter['enable'] = false + gitlab_kas['enable'] = false + + ## + ## To maintain uniformity of links across nodes, the + ## `external_url` on the Sidekiq server should point to the external URL that users + ## use to access GitLab. This can be either: + ## + ## - The `external_url` set on your application server. + ## - The URL of a external load balancer, which routes traffic to the GitLab application server. + ## + external_url 'https://gitlab.example.com' + + # Configure the gitlab-shell API callback URL. Without this, `git push` will + # fail. This can be your 'front door' GitLab URL or an internal load + # balancer. + gitlab_rails['internal_api_url'] = 'GITLAB_URL' + gitlab_shell['secret_token'] = 'SHELL_TOKEN' + + ######################################## + #### Redis ### + ######################################## + + ## Must be the same in every sentinel node. + redis['master_name'] = 'gitlab-redis' # Required if you have setup redis cluster + ## The same password for Redis authentication you set up for the master node. + redis['master_password'] = '<redis_master_password>' + + ### If redis is running on the main Gitlab instance and you have opened the TCP port as above add the following + gitlab_rails['redis_host'] = '<gitlab_host>' + gitlab_rails['redis_port'] = 6379 + + ####################################### + ### Gitaly ### + ####################################### + + ## Replace <gitaly_token> with the one you set up, see + ## https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#about-the-gitaly-token + git_data_dirs({ + "default" => { + "gitaly_address" => "tcp://<gitlab_host>:8075", + "gitaly_token" => "<gitaly_token>" + } + }) + + ####################################### + ### Postgres ### + ####################################### + + # Replace <database_host> and <database_password> + gitlab_rails['db_host'] = '<database_host>' + gitlab_rails['db_port'] = '5432' + gitlab_rails['db_password'] = '<database_password>' + ## Prevent database migrations from running on upgrade automatically + gitlab_rails['auto_migrate'] = false + + ####################################### + ### Sidekiq configuration ### + ####################################### + sidekiq['enable'] = true + sidekiq['listen_address'] = "0.0.0.0" + + ## Set number of Sidekiq queue processes to the same number as available CPUs + sidekiq['queue_groups'] = ['*'] * 4 + + ## Set number of Sidekiq threads per queue process to the recommend number of 10 + sidekiq['max_concurrency'] = 10 + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the GitLab instance and replace the file in the Sidekiq instance. + +1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. Restart the Sidekiq instance after completing the process and finishing the database migrations. + +## Configure multiple Sidekiq nodes with shared storage + +If you run multiple Sidekiq nodes with a shared file storage, such as NFS, you must +specify the UIDs and GIDs to ensure they match between servers. Specifying the UIDs +and GIDs prevents permissions issues in the file system. This advice is similar to the +[advice for Geo setups](../geo/replication/multiple_servers.md#step-4-configure-the-frontend-application-nodes-on-the-geo-secondary-site). + +To set up multiple Sidekiq nodes: + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + user['uid'] = 9000 + user['gid'] = 9000 + web_server['uid'] = 9001 + web_server['gid'] = 9001 + registry['uid'] = 9002 + registry['gid'] = 9002 + ``` + +1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +## Configure the Container Registry when using an external Sidekiq + +If you're using the Container Registry and it's running on a different +node than Sidekiq, follow the steps below. + +1. Edit `/etc/gitlab/gitlab.rb`, and configure the registry URL: + + ```ruby + registry_external_url 'https://registry.example.com' + gitlab_rails['registry_api_url'] = "https://registry.example.com" + ``` + +1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. In the instance where Container Registry is hosted, copy the `registry.key` + file to the Sidekiq node. + +## Configure the Sidekiq metrics server + +If you want to collect Sidekiq metrics, enable the Sidekiq metrics server. +To make metrics available from `localhost:8082/metrics`: + +To configure the metrics server: + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + sidekiq['metrics_enabled'] = true + sidekiq['listen_address'] = "localhost" + sidekiq['listen_port'] = "8082" + + # Optionally log all the metrics server logs to log/sidekiq_exporter.log + sidekiq['exporter_log_enabled'] = true + ``` + +1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +### Enable HTTPS + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/364771) in GitLab 15.2. + +To serve metrics via HTTPS instead of HTTP, enable TLS in the exporter settings: + +1. Edit `/etc/gitlab/gitlab.rb` to add (or find and uncomment) the following lines: + + ```ruby + sidekiq['exporter_tls_enabled'] = true + sidekiq['exporter_tls_cert_path'] = "/path/to/certificate.pem" + sidekiq['exporter_tls_key_path'] = "/path/to/private-key.pem" + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect. + +When TLS is enabled, the same `port` and `address` are used as described above. +The metrics server cannot serve both HTTP and HTTPS at the same time. + +## Configure health checks + +If you use health check probes to observe Sidekiq, enable the Sidekiq health check server. +To make health checks available from `localhost:8092`: + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + sidekiq['health_checks_enabled'] = true + sidekiq['health_checks_listen_address'] = "localhost" + sidekiq['health_checks_listen_port'] = "8092" + ``` + +1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +For more information about health checks, see the [Sidekiq health check page](sidekiq_health_check.md). + +## Configure LDAP and user or group synchronization + +If you use LDAP for user and group management, you must add the LDAP configuration to your Sidekiq node as well as the LDAP +synchronization worker. If the LDAP configuration and LDAP synchronization worker are not applied to your Sidekiq node, +users and groups are not automatically synchronized. + +For more information about configuring LDAP for GitLab, see: + +- [GitLab LDAP configuration documentation](../auth/ldap/index.md#configure-ldap) +- [LDAP synchronization documentation](../auth/ldap/ldap_synchronization.md#adjust-ldap-user-sync-schedule) + +To enable LDAP with the synchronization worker for Sidekiq: + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_enabled'] = true + gitlab_rails['prevent_ldap_sign_in'] = false + gitlab_rails['ldap_servers'] = { + 'main' => { + 'label' => 'LDAP', + 'host' => 'ldap.mydomain.com', + 'port' => 389, + 'uid' => 'sAMAccountName', + 'encryption' => 'simple_tls', + 'verify_certificates' => true, + 'bind_dn' => '_the_full_dn_of_the_user_you_will_bind_with', + 'password' => '_the_password_of_the_bind_user', + 'tls_options' => { + 'ca_file' => '', + 'ssl_version' => '', + 'ciphers' => '', + 'cert' => '', + 'key' => '' + }, + 'timeout' => 10, + 'active_directory' => true, + 'allow_username_or_email_login' => false, + 'block_auto_created_users' => false, + 'base' => 'dc=example,dc=com', + 'user_filter' => '', + 'attributes' => { + 'username' => ['uid', 'userid', 'sAMAccountName'], + 'email' => ['mail', 'email', 'userPrincipalName'], + 'name' => 'cn', + 'first_name' => 'givenName', + 'last_name' => 'sn' + }, + 'lowercase_usernames' => false, + + # Enterprise Edition only + # https://docs.gitlab.com/ee/administration/auth/ldap/ldap_synchronization.html + 'group_base' => '', + 'admin_group' => '', + 'external_groups' => [], + 'sync_ssh_keys' => false + } + } + gitlab_rails['ldap_sync_worker_cron'] = "0 */12 * * *" + ``` + +1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +## Disable Rugged + +Calls into Rugged, Ruby bindings for `libgit2`, [lock the Sidekiq processes's GVL](https://silverhammermba.github.io/emberb/c/#c-in-ruby-threads), +blocking all jobs on that worker from proceeding. If Rugged calls performed by Sidekiq are slow, this can cause significant delays in +background task processing. + +By default, Rugged is used when Git repository data is stored on local storage or on an NFS mount. +[Using Rugged is recommended when using NFS](../nfs.md#improving-nfs-performance-with-gitlab), but if +you are using local storage, disabling Rugged can improve Sidekiq performance: + +```shell +sudo gitlab-rake gitlab:features:disable_rugged +``` + +## Related topics + +- [Extra Sidekiq processes](extra_sidekiq_processes.md) +- [Extra Sidekiq routing](extra_sidekiq_routing.md) +- [Sidekiq health checks](sidekiq_health_check.md) +- [Using the GitLab-Sidekiq chart](https://docs.gitlab.com/charts/charts/gitlab/sidekiq/) + +## Troubleshooting + +See our [administrator guide to troubleshooting Sidekiq](sidekiq_troubleshooting.md). diff --git a/doc/administration/sidekiq/sidekiq_health_check.md b/doc/administration/sidekiq/sidekiq_health_check.md new file mode 100644 index 00000000000..3477320a2ac --- /dev/null +++ b/doc/administration/sidekiq/sidekiq_health_check.md @@ -0,0 +1,58 @@ +--- +stage: Systems +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Sidekiq Health Check **(FREE SELF)** + +GitLab provides liveness and readiness probes to indicate service health and +reachability to the Sidekiq cluster. These endpoints +[can be provided to schedulers like Kubernetes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/) +to hold traffic until the system is ready or restart the container as needed. + +The health check server can be set up when [configuring Sidekiq](index.md). + +## Readiness + +The readiness probe checks whether the Sidekiq workers are ready to process jobs. + +```plaintext +GET /readiness +``` + +If the server is bound to `localhost:8092`, the process cluster can be probed for readiness as follows: + +```shell +curl "http://localhost:8092/readiness" +``` + +On success, the endpoint returns a `200` HTTP status code, and a response like the following: + +```json +{ + "status": "ok" +} +``` + +## Liveness + +Checks whether the Sidekiq cluster is running. + +```plaintext +GET /liveness +``` + +If the server is bound to `localhost:8092`, the process cluster can be probed for liveness as follows: + +```shell +curl "http://localhost:8092/liveness" +``` + +On success, the endpoint returns a `200` HTTP status code, and a response like the following: + +```json +{ + "status": "ok" +} +``` diff --git a/doc/administration/sidekiq/sidekiq_job_migration.md b/doc/administration/sidekiq/sidekiq_job_migration.md new file mode 100644 index 00000000000..a3bc8b2959a --- /dev/null +++ b/doc/administration/sidekiq/sidekiq_job_migration.md @@ -0,0 +1,40 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Sidekiq job migration **(FREE SELF)** + +WARNING: +This operation should be very uncommon. We do not recommend it for the vast majority of GitLab instances. + +Sidekiq routing rules allow administrators to re-route certain background jobs from their regular queue to an alternative queue. By default, GitLab uses one queue per background job type. GitLab has over 400 background job types, and so correspondingly it has over 400 queues. + +Most administrators do not need to change this setting. In some cases with particularly large background job processing workloads, Redis performance may suffer due to the number of queues that GitLab listens to. + +If the Sidekiq routing rules are changed, administrators need to take care with the migration to avoid losing jobs entirely. The basic migration steps are: + +1. Listen to both the old and new queues. +1. Update the routing rules. +1. Wait until there are no publishers dispatching jobs to the old queues. +1. Run the [Rake tasks for future jobs](#future-jobs). +1. Wait for the old queues to be empty. +1. Stop listening to the old queues. + +## Future jobs + +Step 4 involves rewriting some Sidekiq job data for jobs that are already stored in Redis, but due to run in future. There are two sets of jobs to run in future: scheduled jobs and jobs to be retried. We provide a separate Rake task to migrate each set: + +- `gitlab:sidekiq:migrate_jobs:retry` for jobs to be retried. +- `gitlab:sidekiq:migrate_jobs:scheduled` for scheduled jobs. + +Most of the time, running both at the same time is the correct choice. There are two separate tasks to allow for more fine-grained control where needed. To run both at once: + +```shell +# omnibus-gitlab +sudo gitlab-rake gitlab:sidekiq:migrate_jobs:retry gitlab:sidekiq:migrate_jobs:schedule + +# source installations +bundle exec rake gitlab:sidekiq:migrate_jobs:retry gitlab:sidekiq:migrate_jobs:schedule RAILS_ENV=production +``` diff --git a/doc/administration/sidekiq/sidekiq_memory_killer.md b/doc/administration/sidekiq/sidekiq_memory_killer.md new file mode 100644 index 00000000000..12381808523 --- /dev/null +++ b/doc/administration/sidekiq/sidekiq_memory_killer.md @@ -0,0 +1,82 @@ +--- +stage: Data Stores +group: Memory +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Sidekiq MemoryKiller **(FREE SELF)** + +The GitLab Rails application code suffers from memory leaks. For web requests +this problem is made manageable using +[`puma-worker-killer`](https://github.com/schneems/puma_worker_killer) which +restarts Puma worker processes if it exceeds a memory limit. The Sidekiq +MemoryKiller applies the same approach to the Sidekiq processes used by GitLab +to process background jobs. + +Unlike puma-worker-killer, which is enabled by default for all GitLab +installations of GitLab 13.0 and later, the Sidekiq MemoryKiller is enabled by default +_only_ for Omnibus packages. The reason for this is that the MemoryKiller +relies on runit to restart Sidekiq after a memory-induced shutdown and GitLab +installations from source do not all use runit or an equivalent. + +With the default settings, the MemoryKiller causes a Sidekiq restart no +more often than once every 15 minutes, with the restart causing about one +minute of delay for incoming background jobs. + +Some background jobs rely on long-running external processes. To ensure these +are cleanly terminated when Sidekiq is restarted, each Sidekiq process should be +run as a process group leader (for example, using `chpst -P`). If using Omnibus or the +`bin/background_jobs` script with `runit` installed, this is handled for you. + +## Configuring the MemoryKiller + +The MemoryKiller is controlled using environment variables. + +- `SIDEKIQ_DAEMON_MEMORY_KILLER`: defaults to 1. When set to 0, the MemoryKiller + works in _legacy_ mode. Otherwise, the MemoryKiller works in _daemon_ mode. + + In _legacy_ mode, the MemoryKiller checks the Sidekiq process RSS + ([Resident Set Size](https://github.com/mperham/sidekiq/wiki/Memory#rss)) + after each job. + + In _daemon_ mode, the MemoryKiller checks the Sidekiq process RSS every 3 seconds + (defined by `SIDEKIQ_MEMORY_KILLER_CHECK_INTERVAL`). + +- `SIDEKIQ_MEMORY_KILLER_MAX_RSS` (KB): if this variable is set, and its value is greater + than 0, the MemoryKiller is enabled. Otherwise the MemoryKiller is disabled. + + `SIDEKIQ_MEMORY_KILLER_MAX_RSS` defines the Sidekiq process allowed RSS. + + In _legacy_ mode, if the Sidekiq process exceeds the allowed RSS then an irreversible + delayed graceful restart is triggered. The restart of Sidekiq happens + after `SIDEKIQ_MEMORY_KILLER_GRACE_TIME` seconds. + + In _daemon_ mode, if the Sidekiq process exceeds the allowed RSS for longer than + `SIDEKIQ_MEMORY_KILLER_GRACE_TIME` the graceful restart is triggered. If the + Sidekiq process go below the allowed RSS within `SIDEKIQ_MEMORY_KILLER_GRACE_TIME`, + the restart is aborted. + + The default value for Omnibus packages is set + [in the Omnibus GitLab repository](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/attributes/default.rb). + +- `SIDEKIQ_MEMORY_KILLER_HARD_LIMIT_RSS` (KB): is used by _daemon_ mode. If the Sidekiq + process RSS (expressed in kilobytes) exceeds `SIDEKIQ_MEMORY_KILLER_HARD_LIMIT_RSS`, + an immediate graceful restart of Sidekiq is triggered. + +- `SIDEKIQ_MEMORY_KILLER_CHECK_INTERVAL`: used in _daemon_ mode to define how + often to check process RSS, default to 3 seconds. + +- `SIDEKIQ_MEMORY_KILLER_GRACE_TIME`: defaults to 900 seconds (15 minutes). + The usage of this variable is described as part of `SIDEKIQ_MEMORY_KILLER_MAX_RSS`. + +- `SIDEKIQ_MEMORY_KILLER_SHUTDOWN_WAIT`: defaults to 30 seconds. This defines the + maximum time allowed for all Sidekiq jobs to finish. No new jobs are accepted + during that time, and the process exits as soon as all jobs finish. + + If jobs do not finish during that time, the MemoryKiller interrupts all currently + running jobs by sending `SIGTERM` to the Sidekiq process. + + If the process hard shutdown/restart is not performed by Sidekiq, + the Sidekiq process is forcefully terminated after + `Sidekiq.options[:timeout] + 2` seconds. An external supervision mechanism + (for example, runit) must restart Sidekiq afterwards. diff --git a/doc/administration/sidekiq/sidekiq_troubleshooting.md b/doc/administration/sidekiq/sidekiq_troubleshooting.md new file mode 100644 index 00000000000..9bc8e473409 --- /dev/null +++ b/doc/administration/sidekiq/sidekiq_troubleshooting.md @@ -0,0 +1,381 @@ +--- +stage: Systems +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Troubleshooting Sidekiq **(FREE SELF)** + +Sidekiq is the background job processor GitLab uses to asynchronously run +tasks. When things go wrong it can be difficult to troubleshoot. These +situations also tend to be high-pressure because a production system job queue +may be filling up. Users will notice when this happens because new branches +may not show up and merge requests may not be updated. The following are some +troubleshooting steps to help you diagnose the bottleneck. + +GitLab administrators/users should consider working through these +debug steps with GitLab Support so the backtraces can be analyzed by our team. +It may reveal a bug or necessary improvement in GitLab. + +In any of the backtraces, be wary of suspecting cases where every +thread appears to be waiting in the database, Redis, or waiting to acquire +a mutex. This **may** mean there's contention in the database, for example, +but look for one thread that is different than the rest. This other thread +may be using all available CPU, or have a Ruby Global Interpreter Lock, +preventing other threads from continuing. + +## Log arguments to Sidekiq jobs + +[In GitLab 13.6 and later](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/44853) +some arguments passed to Sidekiq jobs are logged by default. +To avoid logging sensitive information (for instance, password reset tokens), +GitLab logs numeric arguments for all workers, with overrides for some specific +workers where their arguments are not sensitive. + +Example log output: + +```json +{"severity":"INFO","time":"2020-06-08T14:37:37.892Z","class":"AdminEmailsWorker","args":["[FILTERED]","[FILTERED]","[FILTERED]"],"retry":3,"queue":"admin_emails","backtrace":true,"jid":"9e35e2674ac7b12d123e13cc","created_at":"2020-06-08T14:37:37.373Z","meta.user":"root","meta.caller_id":"Admin::EmailsController#create","correlation_id":"37D3lArJmT1","uber-trace-id":"2d942cc98cc1b561:6dc94409cfdd4d77:9fbe19bdee865293:1","enqueued_at":"2020-06-08T14:37:37.410Z","pid":65011,"message":"AdminEmailsWorker JID-9e35e2674ac7b12d123e13cc: done: 0.48085 sec","job_status":"done","scheduling_latency_s":0.001012,"redis_calls":9,"redis_duration_s":0.004608,"redis_read_bytes":696,"redis_write_bytes":6141,"duration_s":0.48085,"cpu_s":0.308849,"completed_at":"2020-06-08T14:37:37.892Z","db_duration_s":0.010742} +{"severity":"INFO","time":"2020-06-08T14:37:37.894Z","class":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper","wrapped":"ActionMailer::MailDeliveryJob","queue":"mailers","args":["[FILTERED]"],"retry":3,"backtrace":true,"jid":"e47a4f6793d475378432e3c8","created_at":"2020-06-08T14:37:37.884Z","meta.user":"root","meta.caller_id":"AdminEmailsWorker","correlation_id":"37D3lArJmT1","uber-trace-id":"2d942cc98cc1b561:29344de0f966446d:5c3b0e0e1bef987b:1","enqueued_at":"2020-06-08T14:37:37.885Z","pid":65011,"message":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper JID-e47a4f6793d475378432e3c8: start","job_status":"start","scheduling_latency_s":0.009473} +{"severity":"INFO","time":"2020-06-08T14:39:50.648Z","class":"NewIssueWorker","args":["455","1"],"retry":3,"queue":"new_issue","backtrace":true,"jid":"a24af71f96fd129ec47f5d1e","created_at":"2020-06-08T14:39:50.643Z","meta.user":"root","meta.project":"h5bp/html5-boilerplate","meta.root_namespace":"h5bp","meta.caller_id":"Projects::IssuesController#create","correlation_id":"f9UCZHqhuP7","uber-trace-id":"28f65730f99f55a3:a5d2b62dec38dffc:48ddd092707fa1b7:1","enqueued_at":"2020-06-08T14:39:50.646Z","pid":65011,"message":"NewIssueWorker JID-a24af71f96fd129ec47f5d1e: start","job_status":"start","scheduling_latency_s":0.001144} +``` + +When using [Sidekiq JSON logging](../logs/index.md#sidekiqlog), +arguments logs are limited to a maximum size of 10 kilobytes of text; +any arguments after this limit are discarded and replaced with a +single argument containing the string `"..."`. + +You can set `SIDEKIQ_LOG_ARGUMENTS` [environment variable](https://docs.gitlab.com/omnibus/settings/environment-variables.html) +to `0` (false) to disable argument logging. + +Example: + +```ruby +gitlab_rails['env'] = {"SIDEKIQ_LOG_ARGUMENTS" => "0"} +``` + +In GitLab 13.5 and earlier, set `SIDEKIQ_LOG_ARGUMENTS` to `1` to start logging arguments passed to Sidekiq. + +## Thread dump + +Send the Sidekiq process ID the `TTIN` signal to output thread +backtraces in the log file. + +```shell +kill -TTIN <sidekiq_pid> +``` + +Check in `/var/log/gitlab/sidekiq/current` or `$GITLAB_HOME/log/sidekiq.log` for +the backtrace output. The backtraces are lengthy and generally start with +several `WARN` level messages. Here's an example of a single thread's backtrace: + +```plaintext +2016-04-13T06:21:20.022Z 31517 TID-orn4urby0 WARN: ActiveRecord::RecordNotFound: Couldn't find Note with 'id'=3375386 +2016-04-13T06:21:20.022Z 31517 TID-orn4urby0 WARN: /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.2/lib/active_record/core.rb:155:in `find' +/opt/gitlab/embedded/service/gitlab-rails/app/workers/new_note_worker.rb:7:in `perform' +/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:150:in `execute_job' +/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:132:in `block (2 levels) in process' +/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:127:in `block in invoke' +/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/memory_killer.rb:17:in `call' +/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke' +/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/arguments_logger.rb:6:in `call' +... +``` + +In some cases Sidekiq may be hung and unable to respond to the `TTIN` signal. +Move on to other troubleshooting methods if this happens. + +## Ruby profiling with `rbspy` + +[rbspy](https://rbspy.github.io) is an easy to use and low-overhead Ruby profiler that can be used to create +flamegraph-style diagrams of CPU usage by Ruby processes. + +No changes to GitLab are required to use it and it has no dependencies. To install it: + +1. Download the binary from the [`rbspy` releases page](https://github.com/rbspy/rbspy/releases). +1. Make the binary executable. + +To profile a Sidekiq worker for one minute, run: + +```shell +sudo ./rbspy record --pid <sidekiq_pid> --duration 60 --file /tmp/sidekiq_profile.svg +``` + + + +In this example of a flamegraph generated by `rbspy`, almost all of the Sidekiq process's time is spent in `rev_parse`, a native C +function in Rugged. In the stack, we can see `rev_parse` is being called by the `ExpirePipelineCacheWorker`. + +## Process profiling with `perf` + +Linux has a process profiling tool called `perf` that is helpful when a certain +process is eating up a lot of CPU. If you see high CPU usage and Sidekiq isn't +responding to the `TTIN` signal, this is a good next step. + +If `perf` is not installed on your system, install it with `apt-get` or `yum`: + +```shell +# Debian +sudo apt-get install linux-tools + +# Ubuntu (may require these additional Kernel packages) +sudo apt-get install linux-tools-common linux-tools-generic linux-tools-`uname -r` + +# Red Hat/CentOS +sudo yum install perf +``` + +Run `perf` against the Sidekiq PID: + +```shell +sudo perf record -p <sidekiq_pid> +``` + +Let this run for 30-60 seconds and then press Ctrl-C. Then view the `perf` report: + +```shell +$ sudo perf report + +# Sample output +Samples: 348K of event 'cycles', Event count (approx.): 280908431073 + 97.69% ruby nokogiri.so [.] xmlXPathNodeSetMergeAndClear + 0.18% ruby libruby.so.2.1.0 [.] objspace_malloc_increase + 0.12% ruby libc-2.12.so [.] _int_malloc + 0.10% ruby libc-2.12.so [.] _int_free +``` + +Above you see sample output from a `perf` report. It shows that 97% of the CPU is +being spent inside Nokogiri and `xmlXPathNodeSetMergeAndClear`. For something +this obvious you should then go investigate what job in GitLab would use +Nokogiri and XPath. Combine with `TTIN` or `gdb` output to show the +corresponding Ruby code where this is happening. + +## The GNU Project Debugger (`gdb`) + +`gdb` can be another effective tool for debugging Sidekiq. It gives you a little +more interactive way to look at each thread and see what's causing problems. + +Attaching to a process with `gdb` suspends the normal operation +of the process (Sidekiq does not process jobs while `gdb` is attached). + +Start by attaching to the Sidekiq PID: + +```shell +gdb -p <sidekiq_pid> +``` + +Then gather information on all the threads: + +```plaintext +info threads + +# Example output +30 Thread 0x7fe5fbd63700 (LWP 26060) 0x0000003f7cadf113 in poll () from /lib64/libc.so.6 +29 Thread 0x7fe5f2b3b700 (LWP 26533) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 +28 Thread 0x7fe5f2a3a700 (LWP 26534) 0x0000003f7ce0ba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 +27 Thread 0x7fe5f2939700 (LWP 26535) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 +26 Thread 0x7fe5f2838700 (LWP 26537) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 +25 Thread 0x7fe5f2737700 (LWP 26538) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 +24 Thread 0x7fe5f2535700 (LWP 26540) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 +23 Thread 0x7fe5f2434700 (LWP 26541) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 +22 Thread 0x7fe5f2232700 (LWP 26543) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 +21 Thread 0x7fe5f2131700 (LWP 26544) 0x00007fe5f7b570f0 in xmlXPathNodeSetMergeAndClear () +from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so +... +``` + +If you see a suspicious thread, like the Nokogiri one above, you may want +to get more information: + +```plaintext +thread 21 +bt + +# Example output +#0 0x00007ff0d6afe111 in xmlXPathNodeSetMergeAndClear () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so +#1 0x00007ff0d6b0b836 in xmlXPathNodeCollectAndTest () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so +#2 0x00007ff0d6b09037 in xmlXPathCompOpEval () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so +#3 0x00007ff0d6b09017 in xmlXPathCompOpEval () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so +#4 0x00007ff0d6b092e0 in xmlXPathCompOpEval () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so +#5 0x00007ff0d6b0bc37 in xmlXPathRunEval () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so +#6 0x00007ff0d6b0be5f in xmlXPathEvalExpression () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so +#7 0x00007ff0d6a97dc3 in evaluate (argc=2, argv=0x1022d058, self=<value optimized out>) at xml_xpath_context.c:221 +#8 0x00007ff0daeab0ea in vm_call_cfunc_with_frame (th=0x1022a4f0, reg_cfp=0x1032b810, ci=<value optimized out>) at vm_insnhelper.c:1510 +``` + +To output a backtrace from all threads at once: + +```plaintext +set pagination off +thread apply all bt +``` + +Once you're done debugging with `gdb`, be sure to detach from the process and +exit: + +```plaintext +detach +exit +``` + +## Sidekiq kill signals + +TTIN was described above as the signal to print backtraces for logging, however +Sidekiq responds to other signals as well. For example, TSTP and TERM can be used +to gracefully shut Sidekiq down, see +[the Sidekiq Signals docs](https://github.com/mperham/sidekiq/wiki/Signals#ttin). + +## Check for blocking queries + +Sometimes the speed at which Sidekiq processes jobs can be so fast that it can +cause database contention. Check for blocking queries when backtraces above +show that many threads are stuck in the database adapter. + +The PostgreSQL wiki has details on the query you can run to see blocking +queries. The query is different based on PostgreSQL version. See +[Lock Monitoring](https://wiki.postgresql.org/wiki/Lock_Monitoring) for +the query details. + +## Managing Sidekiq queues + +It is possible to use [Sidekiq API](https://github.com/mperham/sidekiq/wiki/API) +to perform a number of troubleshooting steps on Sidekiq. + +These are the administrative commands and it should only be used if currently +administration interface is not suitable due to scale of installation. + +All these commands should be run using `gitlab-rails console`. + +### View the queue size + +```ruby +Sidekiq::Queue.new("pipeline_processing:build_queue").size +``` + +### Enumerate all enqueued jobs + +```ruby +queue = Sidekiq::Queue.new("chaos:chaos_sleep") +queue.each do |job| + # job.klass # => 'MyWorker' + # job.args # => [1, 2, 3] + # job.jid # => jid + # job.queue # => chaos:chaos_sleep + # job["retry"] # => 3 + # job.item # => { + # "class"=>"Chaos::SleepWorker", + # "args"=>[1000], + # "retry"=>3, + # "queue"=>"chaos:chaos_sleep", + # "backtrace"=>true, + # "queue_namespace"=>"chaos", + # "jid"=>"39bc482b823cceaf07213523", + # "created_at"=>1566317076.266069, + # "correlation_id"=>"c323b832-a857-4858-b695-672de6f0e1af", + # "enqueued_at"=>1566317076.26761}, + # } + + # job.delete if job.jid == 'abcdef1234567890' +end +``` + +### Enumerate currently running jobs + +```ruby +workers = Sidekiq::Workers.new +workers.each do |process_id, thread_id, work| + # process_id is a unique identifier per Sidekiq process + # thread_id is a unique identifier per thread + # work is a Hash which looks like: + # {"queue"=>"chaos:chaos_sleep", + # "payload"=> + # { "class"=>"Chaos::SleepWorker", + # "args"=>[1000], + # "retry"=>3, + # "queue"=>"chaos:chaos_sleep", + # "backtrace"=>true, + # "queue_namespace"=>"chaos", + # "jid"=>"b2a31e3eac7b1a99ff235869", + # "created_at"=>1566316974.9215662, + # "correlation_id"=>"e484fb26-7576-45f9-bf21-b99389e1c53c", + # "enqueued_at"=>1566316974.9229589}, + # "run_at"=>1566316974}], +end +``` + +### Remove Sidekiq jobs for given parameters (destructive) + +The general method to kill jobs conditionally is the following command, which +removes jobs that are queued but not started. Running jobs can not be killed. + +```ruby +queue = Sidekiq::Queue.new('<queue name>') +queue.each { |job| job.delete if <condition>} +``` + +Have a look at the section below for cancelling running jobs. + +In the method above, `<queue-name>` is the name of the queue that contains the jobs you want to delete and `<condition>` decides which jobs get deleted. + +Commonly, `<condition>` references the job arguments, which depend on the type of job in question. To find the arguments for a specific queue, you can have a look at the `perform` function of the related worker file, commonly found at `/app/workers/<queue-name>_worker.rb`. + +For example, `repository_import` has `project_id` as the job argument, while `update_merge_requests` has `project_id, user_id, oldrev, newrev, ref`. + +Arguments need to be referenced by their sequence ID using `job.args[<id>]` because `job.args` is a list of all arguments provided to the Sidekiq job. + +Here are some examples: + +```ruby +queue = Sidekiq::Queue.new('update_merge_requests') +# In this example, we want to remove any update_merge_requests jobs +# for the Project with ID 125 and ref `ref/heads/my_branch` +queue.each { |job| job.delete if job.args[0] == 125 and job.args[4] == 'ref/heads/my_branch' } +``` + +```ruby +# Cancelling jobs like: `RepositoryImportWorker.new.perform_async(100)` +id_list = [100] + +queue = Sidekiq::Queue.new('repository_import') +queue.each do |job| + job.delete if id_list.include?(job.args[0]) +end +``` + +### Remove specific job ID (destructive) + +```ruby +queue = Sidekiq::Queue.new('repository_import') +queue.each do |job| + job.delete if job.jid == 'my-job-id' +end +``` + +## Canceling running jobs (destructive) + +> Introduced in GitLab 12.3. + +This is highly risky operation and use it as last resort. +Doing that might result in data corruption, as the job +is interrupted mid-execution and it is not guaranteed +that proper rollback of transactions is implemented. + +```ruby +Gitlab::SidekiqDaemon::Monitor.cancel_job('job-id') +``` + +> This requires the Sidekiq to be run with `SIDEKIQ_MONITOR_WORKER=1` +> environment variable. + +To perform of the interrupt we use `Thread.raise` which +has number of drawbacks, as mentioned in [Why Ruby's Timeout is dangerous (and Thread.raise is terrifying)](https://jvns.ca/blog/2015/11/27/why-rubys-timeout-is-dangerous-and-thread-dot-raise-is-terrifying/): + +> This is where the implications get interesting, and terrifying. This means that an exception can get raised: +> +> - during a network request (ok, as long as the surrounding code is prepared to catch Timeout::Error) +> - during the cleanup for the network request +> - during a rescue block +> - while creating an object to save to the database afterwards +> - in any of your code, regardless of whether it could have possibly raised an exception before +> +> Nobody writes code to defend against an exception being raised on literally any line. That's not even possible. So Thread.raise is basically like a sneak attack on your code that could result in almost anything. It would probably be okay if it were pure-functional code that did not modify any state. But this is Ruby, so that's unlikely :) diff --git a/doc/administration/sidekiq_health_check.md b/doc/administration/sidekiq_health_check.md index aaab2b7e086..3294eb663f2 100644 --- a/doc/administration/sidekiq_health_check.md +++ b/doc/administration/sidekiq_health_check.md @@ -1,58 +1,11 @@ --- -stage: Systems -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'sidekiq/sidekiq_health_check.md' +remove_date: '2022-11-11' --- -# Sidekiq Health Check **(FREE SELF)** +This document was moved to [another location](sidekiq/sidekiq_health_check.md). -GitLab provides liveness and readiness probes to indicate service health and -reachability to the Sidekiq cluster. These endpoints -[can be provided to schedulers like Kubernetes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/) -to hold traffic until the system is ready or restart the container as needed. - -The health check server can be set up when [configuring Sidekiq](sidekiq.md). - -## Readiness - -The readiness probe checks whether the Sidekiq workers are ready to process jobs. - -```plaintext -GET /readiness -``` - -If the server is bound to `localhost:8092`, the process cluster can be probed for readiness as follows: - -```shell -curl "http://localhost:8092/readiness" -``` - -On success, the endpoint returns a `200` HTTP status code, and a response like the following: - -```json -{ - "status": "ok" -} -``` - -## Liveness - -Checks whether the Sidekiq cluster is running. - -```plaintext -GET /liveness -``` - -If the server is bound to `localhost:8092`, the process cluster can be probed for liveness as follows: - -```shell -curl "http://localhost:8092/liveness" -``` - -On success, the endpoint returns a `200` HTTP status code, and a response like the following: - -```json -{ - "status": "ok" -} -``` +<!-- This redirect file can be deleted after <2022-11-11>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/terraform_state.md b/doc/administration/terraform_state.md index 14649b82ba8..7a8d7774948 100644 --- a/doc/administration/terraform_state.md +++ b/doc/administration/terraform_state.md @@ -78,9 +78,9 @@ Terraform state files are stored locally, follow the steps below. ## Using object storage **(FREE SELF)** -Instead of storing Terraform state files on disk, we recommend the use of [one of the supported object -storage options](object_storage.md#options). This configuration relies on valid credentials to -be configured already. +Instead of storing Terraform state files on disk, we recommend the use of +[one of the supported object storage options](object_storage.md#options). +This configuration relies on valid credentials to be configured already. [Read more about using object storage with GitLab](object_storage.md). diff --git a/doc/administration/troubleshooting/elasticsearch.md b/doc/administration/troubleshooting/elasticsearch.md index 7ce09252680..7390f4bc816 100644 --- a/doc/administration/troubleshooting/elasticsearch.md +++ b/doc/administration/troubleshooting/elasticsearch.md @@ -1,404 +1,11 @@ --- -stage: Data Stores -group: Global Search -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../../integration/advanced_search/elasticsearch_troubleshooting.md' +remove_date: '2022-11-02' --- -# Troubleshooting Elasticsearch **(PREMIUM SELF)** +This document was moved to [another location](../../integration/advanced_search/elasticsearch_troubleshooting.md). -To install and configure Elasticsearch, -visit the [administrator documentation](../../integration/advanced_search/elasticsearch.md). - -For troubleshooting, visit the -[administrator troubleshooting documentation](../../integration/advanced_search/elasticsearch_troubleshooting.md). - -Troubleshooting Elasticsearch requires: - -- Knowledge of common terms. -- Establishing within which category the problem fits. - -## Common terminology - -- **Lucene**: A full-text search library written in Java. -- **Near real time (NRT)**: Refers to the slight latency from the time to index a - document to the time when it becomes searchable. -- **Cluster**: A collection of one or more nodes that work together to hold all - the data, providing indexing and search capabilities. -- **Node**: A single server that works as part of a cluster. -- **Index**: A collection of documents that have somewhat similar characteristics. -- **Document**: A basic unit of information that can be indexed. -- **Shards**: Fully-functional and independent subdivisions of indices. Each shard is actually - a Lucene index. -- **Replicas**: Failover mechanisms that duplicate indices. - -## Troubleshooting workflows - -The type of problem will determine what steps to take. The possible troubleshooting workflows are for: - -- Search results. -- Indexing. -- Integration. -- Performance. -- Advanced Search Migrations. - -### Search Results workflow - -The following workflow is for Elasticsearch search results issues: - -```mermaid -graph TD; - B --> |No| B1 - B --> |Yes| B4 - B1 --> B2 - B2 --> B3 - B4 --> B5 - B5 --> |Yes| B6 - B5 --> |No| B7 - B7 --> B8 - B{Is GitLab using<br>Elasticsearch for<br>searching?} - B1[From the Admin Area, select<br>Integrations from the left<br>sidebar to ensure the settings<br>are correct.] - B2[Perform a search via<br>the rails console] - B3[If all settings are correct<br>and it still doesn't show Elasticsearch<br>doing the searches, escalate<br>to GitLab support.] - B4[Perform<br>the same search via the<br>Elasticsearch API] - B5{Are the results<br>the same?} - B6[This means it is working as intended.<br>Speak with GitLab support<br>to confirm if the issue lies with<br>the filters.] - B7[Check the index status of the project<br>containing the missing search<br>results.] - B8(Indexing Troubleshooting) -``` - -### Indexing workflow - -The following workflow is for Elasticsearch indexing issues: - -```mermaid -graph TD; - C --> |Yes| C1 - C1 --> |Yes| C2 - C1 --> |No| C3 - C3 --> |Yes| C4 - C3 --> |No| C5 - C --> |No| C6 - C6 --> |No| C10 - C7 --> |GitLab| C8 - C7 --> |Elasticsearch| C9 - C6 --> |Yes| C7 - C10 --> |No| C12 - C10 --> |Yes| C11 - C12 --> |Yes| C13 - C12 --> |No| C14 - C14 --> |Yes| C15 - C14 --> |No| C16 - C{Is the problem with<br>creating an empty<br>index?} - C1{Does the gitlab-production<br>index exist on the<br>Elasticsearch instance?} - C2(Try to manually<br>delete the index on the<br>Elasticsearch instance and<br>retry creating an empty index.) - C3{Can indices be made<br>manually on the Elasticsearch<br>instance?} - C4(Retry the creation of an empty index) - C5(It is best to speak with an<br>Elasticsearch admin concerning the<br>instance's inability to create indices.) - C6{Is the indexer presenting<br>errors during indexing?} - C7{Is the error a GitLab<br>error or an Elasticsearch<br>error?} - C8[Escalate to<br>GitLab support] - C9[You will want<br>to speak with an<br>Elasticsearch admin.] - C10{Does the index status<br>show 100%?} - C11[Escalate to<br>GitLab support] - C12{Does re-indexing the project<br> present any GitLab errors?} - C13[Rectify the GitLab errors and<br>restart troubleshooting, or<br>escalate to GitLab support.] - C14{Does re-indexing the project<br>present errors on the <br>Elasticsearch instance?} - C15[It would be best<br>to speak with an<br>Elasticsearch admin.] - C16[This is likely a bug/issue<br>in GitLab and will require<br>deeper investigation. Escalate<br>to GitLab support.] -``` - -### Integration workflow - -The following workflow is for Elasticsearch integration issues: - -```mermaid -graph TD; - D --> |No| D1 - D --> |Yes| D2 - D2 --> |No| D3 - D2 --> |Yes| D4 - D4 --> |No| D5 - D4 --> |Yes| D6 - D{Is the error concerning<br>the Go indexer?} - D1[It would be best<br>to speak with an<br>Elasticsearch admin.] - D2{Is the ICU development<br>package installed?} - D3>This package is required.<br>Install the package<br>and retry.] - D4{Is the error stemming<br>from the indexer?} - D5[This would indicate an OS level<br> issue. It would be best to<br>contact your sysadmin.] - D6[This is likely a bug/issue<br>in GitLab and will require<br>deeper investigation. Escalate<br>to GitLab support.] -``` - -### Performance workflow - -The following workflow is for Elasticsearch performance issues: - -```mermaid -graph TD; - F --> |Yes| F1 - F --> |No| F2 - F2 --> |No| F3 - F2 --> |Yes| F4 - F4 --> F5 - F5 --> |No| F6 - F5 --> |Yes| F7 - F{Is the Elasticsearch instance<br>running on the same server<br>as the GitLab instance?} - F1(This is not advised and will cause issues.<br>We recommend moving the Elasticsearch<br>instance to a different server.) - F2{Does the Elasticsearch<br>server have at least 8<br>GB of RAM and 2 CPU<br>cores?} - F3(According to Elasticsearch, a non-prod<br>server needs these as a base requirement.<br>Production often requires more. We recommend<br>you increase the server specifications.) - F4(Obtain the <br>cluster health information) - F5(Does it show the<br>status as green?) - F6(We recommend you speak with<br>an Elasticsearch admin<br>about implementing sharding.) - F7(Escalate to<br>GitLab support.) -``` - -### Advanced Search Migrations workflow - -```mermaid -graph TD; - D --> |No| D1 - D --> |Yes| D2 - D2 --> |No| D3 - D2 --> |Yes| D4 - D4 --> |No| D5 - D4 --> |Yes| D6 - D6 --> |No| D8 - D6 --> |Yes| D7 - - D{Is there a halted migration?} - D1[Migrations run in the<br>background and will<br>stop when completed.] - D2{Does the elasticsearch.log<br>file contain errors?} - D3[This is likely a bug/issue<br>in GitLab and will require<br>deeper investigation. Escalate<br>to GitLab support.] - D4{Have the errors<br>been addressed?} - D5[Have an Elasticsearch admin<br>review and address<br>the errors.] - D6{Has the migration<br>been retried?} - D7[This is likely a bug/issue<br>in GitLab and will require<br>deeper investigation. Escalate<br>to GitLab support.] - D8[Retry the migration from<br>the Admin > Settings ><br>Advanced Search UI.] -``` - -## Troubleshooting walkthrough - -Most Elasticsearch troubleshooting can be broken down into 4 categories: - -- [Troubleshooting search results](#troubleshooting-search-results) -- [Troubleshooting indexing](#troubleshooting-indexing) -- [Troubleshooting integration](#troubleshooting-integration) -- [Troubleshooting performance](#troubleshooting-performance) -- [Troubleshooting Advanced Search migrations](#troubleshooting-advanced-search-migrations) - -Generally speaking, if it does not fall into those four categories, it is either: - -- Something GitLab support needs to look into. -- Not a true Elasticsearch issue. - -Exercise caution. Issues that appear to be Elasticsearch problems can be OS-level issues. - -### Troubleshooting search results - -Troubleshooting search result issues is rather straight forward on Elasticsearch. - -The first step is to confirm GitLab is using Elasticsearch for the search function. -To do this: - -1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **Settings > General**, and then confirm the - integration is enabled. -1. Confirm searches use Elasticsearch by accessing the rails console - (`sudo gitlab-rails console`) and running the following commands: - - ```rails - u = User.find_by_email('email_of_user_doing_search') - s = SearchService.new(u, {:search => 'search_term'}) - pp s.search_objects.class - ``` - -The output from the last command is the key here. If it shows: - -- `ActiveRecord::Relation`, **it is not** using Elasticsearch. -- `Kaminari::PaginatableArray`, **it is** using Elasticsearch. - -| Not using Elasticsearch | Using Elasticsearch | -|--------------------------|------------------------------| -| `ActiveRecord::Relation` | `Kaminari::PaginatableArray` | - -If all the settings look correct and it is still not using Elasticsearch for the search function, it is best to escalate to GitLab support. This could be a bug/issue. - -Moving past that, it is best to attempt the same [search via the Rails console](../../integration/advanced_search/elasticsearch_troubleshooting.md#i-indexed-all-the-repositories-but-i-cant-get-any-hits-for-my-search-term-in-the-ui) -or the [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html), -and compare the results from what you see in GitLab. - -If the results: - -- Sync up, then there is not a technical "issue." Instead, it might be a problem - with the Elasticsearch filters we are using. This can be complicated, so it is best to - escalate to GitLab support to check these and guide you on the potential on whether or - not a feature request is needed. -- Do not match up, this indicates a problem with the documents generated from the - project. It is best to re-index that project and proceed with - [Troubleshooting indexing](#troubleshooting-indexing). - -### Troubleshooting indexing - -Troubleshooting indexing issues can be tricky. It can pretty quickly go to either GitLab -support or your Elasticsearch administrator. - -The best place to start is to determine if the issue is with creating an empty index. -If it is, check on the Elasticsearch side to determine if the `gitlab-production` (the -name for the GitLab index) exists. If it exists, manually delete it on the Elasticsearch -side and attempt to recreate it from the -[`recreate_index`](../../integration/advanced_search/elasticsearch.md#gitlab-advanced-search-rake-tasks) -Rake task. - -If you still encounter issues, try creating an index manually on the Elasticsearch -instance. The details of the index aren't important here, as we want to test if indices -can be made. If the indices: - -- Cannot be made, speak with your Elasticsearch administrator. -- Can be made, Escalate this to GitLab support. - -If the issue is not with creating an empty index, the next step is to check for errors -during the indexing of projects. If errors do occur, they stem from either the indexing: - -- On the GitLab side. You need to rectify those. If they are not - something you are familiar with, contact GitLab support for guidance. -- Within the Elasticsearch instance itself. See if the error is [documented and has a fix](../../integration/advanced_search/elasticsearch_troubleshooting.md). If not, speak with your Elasticsearch administrator. - -If the indexing process does not present errors, check the status of the indexed projects. You can do this via the following Rake tasks: - -- [`sudo gitlab-rake gitlab:elastic:index_projects_status`](../../integration/advanced_search/elasticsearch.md#gitlab-advanced-search-rake-tasks) (shows the overall status) -- [`sudo gitlab-rake gitlab:elastic:projects_not_indexed`](../../integration/advanced_search/elasticsearch.md#gitlab-advanced-search-rake-tasks) (shows specific projects that are not indexed) - -If: - -- Everything is showing at 100%, escalate to GitLab support. This could be a potential - bug/issue. -- You do see something not at 100%, attempt to reindex that project. To do this, - run `sudo gitlab-rake gitlab:elastic:index_projects ID_FROM=<project ID> ID_TO=<project ID>`. - -If reindexing the project shows: - -- Errors on the GitLab side, escalate those to GitLab support. -- Elasticsearch errors or doesn't present any errors at all, reach out to your - Elasticsearch administrator to check the instance. - -### Troubleshooting integration - -Troubleshooting integration tends to be pretty straight forward, as there really isn't -much to "integrate" here. - -If the issue is: - -- With the Go indexer, check if the ICU development package is installed. - This is a required package so make sure you install it. - Go indexer was a beta indexer which can be optionally turned on/off, but in 12.3 it reached stable status and is now the default. -- Not concerning the Go indexer, it is almost always an - Elasticsearch-side issue. This means you should reach out to your Elasticsearch administrator - regarding the errors you are seeing. If you are unsure here, it never hurts to reach - out to GitLab support. - -Beyond that, review the error. If it is: - -- Specifically from the indexer, this could be a bug/issue and should be escalated to - GitLab support. -- An OS issue, you should reach out to your systems administrator. -- A `Faraday::TimeoutError (execution expired)` error **and** you're using a proxy, - [set a custom `gitlab_rails['env']` environment variable, called `no_proxy`](https://docs.gitlab.com/omnibus/settings/environment-variables.html) - with the IP address of your Elasticsearch host. - -### Troubleshooting performance - -Troubleshooting performance can be difficult on Elasticsearch. There is a ton of tuning -that *can* be done, but the majority of this falls on shoulders of a skilled -Elasticsearch administrator. - -Generally speaking, ensure: - -- The Elasticsearch server **is not** running on the same node as GitLab. -- The Elasticsearch server have enough RAM and CPU cores. -- That sharding **is** being used. - -Going into some more detail here, if Elasticsearch is running on the same server as GitLab, resource contention is **very** likely to occur. Ideally, Elasticsearch, which requires ample resources, should be running on its own server (maybe coupled with Logstash and Kibana). - -When it comes to Elasticsearch, RAM is the key resource. Elasticsearch themselves recommend: - -- **At least** 8 GB of RAM for a non-production instance. -- **At least** 16 GB of RAM for a production instance. -- Ideally, 64 GB of RAM. - -For CPU, Elasticsearch recommends at least 2 CPU cores, but Elasticsearch states common -setups use up to 8 cores. For more details on server specs, check out -[Elasticsearch's hardware guide](https://www.elastic.co/guide/en/elasticsearch/guide/current/hardware.html). - -Beyond the obvious, sharding comes into play. Sharding is a core part of Elasticsearch. -It allows for horizontal scaling of indices, which is helpful when you are dealing with -a large amount of data. - -With the way GitLab does indexing, there is a **huge** amount of documents being -indexed. By utilizing sharding, you can speed up Elasticsearch's ability to locate -data, since each shard is a Lucene index. - -If you are not using sharding, you are likely to hit issues when you start using -Elasticsearch in a production environment. - -Keep in mind that an index with only one shard has **no scale factor** and will -likely encounter issues when called upon with some frequency. - -If you need to know how many shards, read -[Elasticsearch's documentation on capacity planning](https://www.elastic.co/guide/en/elasticsearch/guide/2.x/capacity-planning.html), -as the answer is not straight forward. - -The easiest way to determine if sharding is in use is to check the output of the -[Elasticsearch Health API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html): - -- Red means the cluster is down. -- Yellow means it is up with no sharding/replication. -- Green means it is healthy (up, sharding, replicating). - -For production use, it should always be green. - -Beyond these steps, you get into some of the more complicated things to check, -such as merges and caching. These can get complicated and it takes some time to -learn them, so it is best to escalate/pair with an Elasticsearch expert if you need to -dig further into these. - -Feel free to reach out to GitLab support, but this is likely to be something a skilled -Elasticsearch administrator has more experience with. - -### Troubleshooting Advanced Search migrations - -Troubleshooting Advanced Search migration failures can be difficult and may -require contacting an Elasticsearch administrator or GitLab Support. - -The best place to start while debugging issues with an Advanced Search -migration is the [`elasticsearch.log` file](../logs.md#elasticsearchlog). -Migrations log information while a migration is in progress and any -errors encountered. Apply fixes for any errors found in the log and retry -the migration. - -If you still encounter issues after retrying the migration, reach out to GitLab support. - -## Common issues - -All common issues [should be documented](../../integration/advanced_search/elasticsearch_troubleshooting.md). If not, -feel free to update that page with issues you encounter and solutions. - -## Replication - -Setting up Elasticsearch isn't too bad, but it can be a bit finicky and time consuming. - -The easiest method is to spin up a Docker container with the required version and -bind ports 9200/9300 so it can be used. - -The following is an example of running a Docker container of Elasticsearch v7.2.0: - -```shell -docker pull docker.elastic.co/elasticsearch/elasticsearch:7.2.0 -docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.2.0 -``` - -From here, you can: - -- Grab the IP of the Docker container (use `docker inspect <container_id>`) -- Use `<IP.add.re.ss:9200>` to communicate with it. - -This is a quick method to test out Elasticsearch, but by no means is this a -production solution. +<!-- This redirect file can be deleted after <2022-11-02>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 0ff1afa86ed..aa4dbec4f95 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -178,26 +178,6 @@ Feature.all Feature.all.map {|f| [f.name, f.state]} ``` -## Command Line - -### Check the GitLab version fast - -```shell -grep -m 1 gitlab /opt/gitlab/version-manifest.txt -``` - -### Debugging SSH - -```shell -GIT_SSH_COMMAND="ssh -vvv" git clone <repository> -``` - -### Debugging over HTTPS - -```shell -GIT_CURL_VERBOSE=1 GIT_TRACE=1 git clone <repository> -``` - ## Projects ### Clear a project's cache @@ -326,7 +306,7 @@ Project.find_each do |p| end ``` -## Bulk update to change all the Jira integrations to Jira instance-level values +### Bulk update to change all the Jira integrations to Jira instance-level values To change all Jira project to use the instance-level integration settings: @@ -341,6 +321,25 @@ To change all Jira project to use the instance-level integration settings: 1. Modify and save again the instance-level integration from the UI to propagate the changes to all the group-level and project-level integrations. +### Check if Jira Cloud is linked to a namespace + +```ruby +JiraConnectSubscription.where(namespace: Namespace.by_path('group/subgroup')) +``` + +### Check if Jira Cloud is linked to a project + +```ruby +Project.find_by_full_path('path/to/project').jira_subscription_exists? +``` + +### Check if Jira Cloud URL is linked to any namespace + +```ruby +installation = JiraConnectInstallation.find_by_base_url("https://customer_name.atlassian.net") +installation.subscriptions +``` + ### Bulk update to disable the Slack Notification service To disable notifications for all projects that have Slack service enabled, do: @@ -424,33 +423,6 @@ projects = Project.find_by_sql("SELECT * FROM projects WHERE name LIKE '%ject'") => [#<Project id:12 root/my-first-project>>, #<Project id:13 root/my-second-project>>] ``` -## Wikis - -### Recreate - -WARNING: -This is a destructive operation, the Wiki becomes empty. - -A Projects Wiki can be recreated by this command: - -```ruby -p = Project.find_by_full_path('<username-or-group>/<project-name>') ### enter your projects path - -GitlabShellWorker.perform_in(0, :remove_repository, p.repository_storage, p.wiki.disk_path) ### deletes the wiki project from the filesystem - -p.create_wiki ### creates the wiki project on the filesystem -``` - -## Issue boards - -### In case of issue boards not loading properly and it's getting time out. Call the Issue Rebalancing service to fix this - -```ruby -p = Project.find_by_full_path('<username-or-group>/<project-name>') - -Issues::RelativePositionRebalancingService.new(p.root_namespace.all_projects).execute -``` - ## Imports and exports ### Import a project @@ -855,52 +827,6 @@ Gitlab::CurrentSettings.update!(password_authentication_enabled_for_web: true) ## SCIM -### Fixing bad SCIM identities - -```ruby -def delete_bad_scim(email, group_path) - output = "" - u = User.find_by_email(email) - uid = u.id - g = Group.find_by_full_path(group_path) - saml_prov_id = SamlProvider.find_by(group_id: g.id).id - saml = Identity.where(user_id: uid, saml_provider_id: saml_prov_id) - scim = ScimIdentity.where(user_id: uid , group_id: g.id) - if saml[0] - saml_eid = saml[0].extern_uid - output += "%s," % [email] - output += "SAML: %s," % [saml_eid] - if scim[0] - scim_eid = scim[0].extern_uid - output += "SCIM: %s" % [scim_eid] - if saml_eid == scim_eid - output += " Identities matched, not deleted \n" - else - scim[0].destroy - output += " Deleted \n" - end - else - output = "ERROR No SCIM identify found for: [%s]\n" % [email] - puts output - return 1 - end - else - output = "ERROR No SAML identify found for: [%s]\n" % [email] - puts output - return 1 - end - puts output - return 0 -end - -# In case of multiple emails -emails = [email1, email2] - -emails.each do |e| - delete_bad_scim(e,'<group-path>') -end -``` - ### Find groups using an SQL query Find and store an array of groups based on an SQL query: @@ -927,13 +853,13 @@ conflicting_permanent_redirects.destroy_all ## Merge requests -### Close a merge request properly (if merged but still marked as open) +### Close a merge request ```ruby u = User.find_by_username('<username>') p = Project.find_by_full_path('<namespace/project>') m = p.merge_requests.find_by(iid: <iid>) -MergeRequests::PostMergeService.new(project: p, current_user: u).execute(m) +MergeRequests::CloseService.new(project: p, current_user: u).execute(m) ``` ### Delete a merge request @@ -954,6 +880,21 @@ m = p.merge_requests.find_by(iid: <iid>) MergeRequests::RebaseService.new(project: m.target_project, current_user: u).execute(m) ``` +### Set a merge request as merged + +Use when a merge request was accepted and the changes merged into the Git repository, +but the merge request still shows as open. + +If the changes are not merged yet, this action causes the merge request to +incorrectly show `merged into <branch-name>`. + +```ruby +u = User.find_by_username('<username>') +p = Project.find_by_full_path('<namespace/project>') +m = p.merge_requests.find_by(iid: <iid>) +MergeRequests::PostMergeService.new(project: p, current_user: u).execute(m) +``` + ## CI ### Cancel stuck pending pipelines @@ -1063,6 +1004,9 @@ License.current.trial? # License ID for lookup on CustomersDot License.current.license_id + +# License data in Base64-encoded ASCII format +License.current.data ``` ### Check if a project feature is available on the instance @@ -1378,16 +1322,6 @@ registry = Geo::SnippetRepositoryRegistry.find(registry_id) registry.replicator.send(:sync_repository) ``` -## Gitaly - -### Find available and used space - -A Gitaly storage resource can be polled through Rails to determine the available and used space. - -```ruby -Gitlab::GitalyClient::ServerService.new("default").storage_disk_statistics -``` - ## Generate Service Ping The [Service Ping Guide](../../development/service_ping/index.md) in our developer documentation @@ -1429,28 +1363,6 @@ Prints the metrics saved in `conversational_development_index_metrics`. rake gitlab:usage_data:generate_and_send ``` -## Kubernetes integration - -Find cluster: - -```ruby -cluster = Clusters::Cluster.find(1) -cluster = Clusters::Cluster.find_by(name: 'cluster_name') -``` - -Delete cluster without associated resources: - -```ruby -# Find users with the administrator access -user = User.find_by(username: 'admin_user') - -# Find the cluster with the ID -cluster = Clusters::Cluster.find(1) - -# Delete the cluster -Clusters::DestroyService.new(user).execute(cluster) -``` - ## Elasticsearch ### Configuration attributes diff --git a/doc/administration/troubleshooting/group_saml_scim.md b/doc/administration/troubleshooting/group_saml_scim.md index 145eb5f65ae..b5187504231 100644 --- a/doc/administration/troubleshooting/group_saml_scim.md +++ b/doc/administration/troubleshooting/group_saml_scim.md @@ -1,207 +1,11 @@ --- -stage: Manage -group: Authentication and Authorization -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference +redirect_to: '../../user/group/saml_sso/example_saml_config.md' +remove_date: '2022-10-29' --- -# Troubleshooting Group SAML and SCIM **(PREMIUM SAAS)** +This document was moved to [another location](../../user/group/saml_sso/example_saml_config.md). -These are notes and screenshots regarding Group SAML and SCIM that the GitLab Support Team sometimes uses while troubleshooting, but which do not fit into the official documentation. GitLab is making this public, so that anyone can make use of the Support team's collected knowledge. - -Please refer to the GitLab [Group SAML](../../user/group/saml_sso/index.md) docs for information on the feature and how to set it up. - -When troubleshooting a SAML configuration, GitLab team members will frequently start with the [SAML troubleshooting section](../../user/group/saml_sso/index.md#troubleshooting). - -They may then set up a test configuration of the desired identity provider. We include example screenshots in this section. - -## SAML and SCIM screenshots - -This section includes relevant screenshots of the following example configurations of [Group SAML](../../user/group/saml_sso/index.md) and [Group SCIM](../../user/group/saml_sso/scim_setup.md): - -- [Azure Active Directory](#azure-active-directory) -- [Google Workspace](#google-workspace) -- [Okta](#okta) -- [OneLogin](#onelogin) - -WARNING: -These screenshots are updated only as needed by GitLab Support. They are **not** official documentation. - -If you are currently having an issue with GitLab, you may want to check your [support options](https://about.gitlab.com/support/). - -## Azure Active Directory - -Basic SAML app configuration: - - - -User claims and attributes: - - - -SCIM mapping: - - - - -Group Sync: - - - -## Google Workspace - -Basic SAML app configuration: - - - -User claims and attributes: - - - -IdP links and certificate: - -NOTE: -Google Workspace displays a SHA256 fingerprint. To retrieve the SHA1 fingerprint required by GitLab for configuring SAML, download the certificate and calculate the SHA1 certificate -fingerprint. - - - -## Okta - -Basic SAML app configuration: - - - -User claims and attributes: - - - -Advanced SAML app settings (defaults): - - - -IdP Links and Certificate: - - - -Sign on settings: - - - -Self-managed instance example: - - - -Setting the username for the newly provisioned users when assigning them the SCIM app: - - - -## OneLogin - -Application details: - - - -Parameters: - - - -Adding a user: - - - -SSO settings: - - - -## SAML response example - -When a user signs in using SAML, GitLab receives a SAML response. The SAML response can be found in `production.log` logs as a base64-encoded message. Locate the response by -searching for `SAMLResponse`. The decoded SAML response is in XML format. For example: - -```xml -<?xml version="1.0" encoding="UTF-8"?> -<saml2p:Response xmlns:saml2p="urn:oasis:names:tc:SAML:2.0:protocol" xmlns:xs="http://www.w3.org/2001/XMLSchema" Destination="https://gitlabexample/-/saml/callback" ID="id4898983630840142426821432" InResponseTo="_c65e4c88-9425-4472-b42c-37f4186ac0ee" IssueInstant="2022-05-30T21:30:35.696Z" Version="2.0"> - <saml2:Issuer xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion" Format="urn:oasis:names:tc:SAML:2.0:nameid-format:entity">http://www.okta.com/exk2y6j57o1Pdr2lI8qh7</saml2:Issuer> - <ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#"> - <ds:SignedInfo> - <ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/> - <ds:SignatureMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"/> - <ds:Reference URI="#id4898983630840142426821432"> - <ds:Transforms> - <ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/> - <ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"> - <ec:InclusiveNamespaces xmlns:ec="http://www.w3.org/2001/10/xml-exc-c14n#" PrefixList="xs"/> - </ds:Transform> - </ds:Transforms> - <ds:DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/> - <ds:DigestValue>neiQvv9d3OgS4GZW8Nptp4JhjpKs3GCefibn+vmRgk4=</ds:DigestValue> - </ds:Reference> - </ds:SignedInfo> - <ds:SignatureValue>dMsQX8ivi...HMuKGhyLRvabGU6CuPrf7==</ds:SignatureValue> - <ds:KeyInfo> - <ds:X509Data> - <ds:X509Certificate>MIIDq...cptGr3vN9TQ==</ds:X509Certificate> - </ds:X509Data> - </ds:KeyInfo> - </ds:Signature> - <saml2p:Status xmlns:saml2p="urn:oasis:names:tc:SAML:2.0:protocol"> - <saml2p:StatusCode Value="urn:oasis:names:tc:SAML:2.0:status:Success"/> - </saml2p:Status> - <saml2:Assertion xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion" xmlns:xs="http://www.w3.org/2001/XMLSchema" ID="id489" IssueInstant="2022-05-30T21:30:35.696Z" Version="2.0"> - <saml2:Issuer xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion" Format="urn:oasis:names:tc:SAML:2.0:nameid-format:entity">http://www.okta.com/exk2y6j57o1Pdr2lI8qh7</saml2:Issuer> - <ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#"> - <ds:SignedInfo> - <ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/> - <ds:SignatureMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"/> - <ds:Reference URI="#id48989836309833801859473359"> - <ds:Transforms> - <ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/> - <ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"> - <ec:InclusiveNamespaces xmlns:ec="http://www.w3.org/2001/10/xml-exc-c14n#" PrefixList="xs"/> - </ds:Transform> - </ds:Transforms> - <ds:DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/> - <ds:DigestValue>MaIsoi8hbT9gsi/mNZsz449mUuAcuEWY0q3bc4asOQs=</ds:DigestValue> - </ds:Reference> - </ds:SignedInfo> - <ds:SignatureValue>dMsQX8ivi...HMuKGhyLRvabGU6CuPrf7==<</ds:SignatureValue> - <ds:KeyInfo> - <ds:X509Data> - <ds:X509Certificate>MIIDq...cptGr3vN9TQ==</ds:X509Certificate> - </ds:X509Data> - </ds:KeyInfo> - </ds:Signature> - <saml2:Subject xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion"> - <saml2:NameID Format="urn:oasis:names:tc:SAML:2.0:nameid-format:persistent">useremail@domain.com</saml2:NameID> - <saml2:SubjectConfirmation Method="urn:oasis:names:tc:SAML:2.0:cm:bearer"> - <saml2:SubjectConfirmationData InResponseTo="_c65e4c88-9425-4472-b42c-37f4186ac0ee" NotOnOrAfter="2022-05-30T21:35:35.696Z" Recipient="https://gitlab.example.com/-/saml/callback"/> - </saml2:SubjectConfirmation> - </saml2:Subject> - <saml2:Conditions xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion" NotBefore="2022-05-30T21:25:35.696Z" NotOnOrAfter="2022-05-30T21:35:35.696Z"> - <saml2:AudienceRestriction> - <saml2:Audience>https://gitlab.example.com/</saml2:Audience> - </saml2:AudienceRestriction> - </saml2:Conditions> - <saml2:AuthnStatement xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion" AuthnInstant="2022-05-30T21:30:35.696Z" SessionIndex="_c65e4c88-9425-4472-b42c-37f4186ac0ee"> - <saml2:AuthnContext> - <saml2:AuthnContextClassRef>urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport</saml2:AuthnContextClassRef> - </saml2:AuthnContext> - </saml2:AuthnStatement> - <saml2:AttributeStatement xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion"> - <saml2:Attribute Name="email" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified"> - <saml2:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">useremail@domain.com</saml2:AttributeValue> - </saml2:Attribute> - <saml2:Attribute Name="firtname" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified"> - <saml2:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">John</saml2:AttributeValue> - </saml2:Attribute> - <saml2:Attribute Name="lastname" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified"> - <saml2:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">Doe</saml2:AttributeValue> - </saml2:Attribute> - <saml2:Attribute Name="Groups" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified"> - <saml2:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">Super-awesome-group</saml2:AttributeValue> - </saml2:Attribute> - </saml2:AttributeStatement> - </saml2:Assertion> -</saml2p:Response> -``` +<!-- This redirect file can be deleted after <2022-10-29>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/troubleshooting/img/AzureAD-basic_SAML.png b/doc/administration/troubleshooting/img/AzureAD-basic_SAML.png Binary files differdeleted file mode 100644 index 7a0d83ab2dd..00000000000 --- a/doc/administration/troubleshooting/img/AzureAD-basic_SAML.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/AzureAD-claims.png b/doc/administration/troubleshooting/img/AzureAD-claims.png Binary files differdeleted file mode 100644 index 576040be337..00000000000 --- a/doc/administration/troubleshooting/img/AzureAD-claims.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/AzureAD-scim_attribute_mapping.png b/doc/administration/troubleshooting/img/AzureAD-scim_attribute_mapping.png Binary files differdeleted file mode 100644 index 5ad82003eed..00000000000 --- a/doc/administration/troubleshooting/img/AzureAD-scim_attribute_mapping.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/AzureAD-scim_provisioning.png b/doc/administration/troubleshooting/img/AzureAD-scim_provisioning.png Binary files differdeleted file mode 100644 index b2c385151a6..00000000000 --- a/doc/administration/troubleshooting/img/AzureAD-scim_provisioning.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/GoogleWorkspace-basic-SAML_v14_10.png b/doc/administration/troubleshooting/img/GoogleWorkspace-basic-SAML_v14_10.png Binary files differdeleted file mode 100644 index e002503ddc0..00000000000 --- a/doc/administration/troubleshooting/img/GoogleWorkspace-basic-SAML_v14_10.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/GoogleWorkspace-claims_v14_10.png b/doc/administration/troubleshooting/img/GoogleWorkspace-claims_v14_10.png Binary files differdeleted file mode 100644 index 2b827e122a8..00000000000 --- a/doc/administration/troubleshooting/img/GoogleWorkspace-claims_v14_10.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/GoogleWorkspace-linkscert_v14_10.png b/doc/administration/troubleshooting/img/GoogleWorkspace-linkscert_v14_10.png Binary files differdeleted file mode 100644 index 8d6c5010670..00000000000 --- a/doc/administration/troubleshooting/img/GoogleWorkspace-linkscert_v14_10.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/Okta-SAMLsetup.png b/doc/administration/troubleshooting/img/Okta-SAMLsetup.png Binary files differdeleted file mode 100644 index 1bd9bf4d7e9..00000000000 --- a/doc/administration/troubleshooting/img/Okta-SAMLsetup.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/Okta-advancedsettings.png b/doc/administration/troubleshooting/img/Okta-advancedsettings.png Binary files differdeleted file mode 100644 index 45e378d1d12..00000000000 --- a/doc/administration/troubleshooting/img/Okta-advancedsettings.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/Okta-attributes.png b/doc/administration/troubleshooting/img/Okta-attributes.png Binary files differdeleted file mode 100644 index a3405e4de9b..00000000000 --- a/doc/administration/troubleshooting/img/Okta-attributes.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/Okta-linkscert.png b/doc/administration/troubleshooting/img/Okta-linkscert.png Binary files differdeleted file mode 100644 index 38cae415f7e..00000000000 --- a/doc/administration/troubleshooting/img/Okta-linkscert.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/OneLogin-SSOsettings.png b/doc/administration/troubleshooting/img/OneLogin-SSOsettings.png Binary files differdeleted file mode 100644 index 58f936d8567..00000000000 --- a/doc/administration/troubleshooting/img/OneLogin-SSOsettings.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/OneLogin-app_details.png b/doc/administration/troubleshooting/img/OneLogin-app_details.png Binary files differdeleted file mode 100644 index 77618960897..00000000000 --- a/doc/administration/troubleshooting/img/OneLogin-app_details.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/OneLogin-parameters.png b/doc/administration/troubleshooting/img/OneLogin-parameters.png Binary files differdeleted file mode 100644 index a2fa734152c..00000000000 --- a/doc/administration/troubleshooting/img/OneLogin-parameters.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/OneLogin-userAdd.png b/doc/administration/troubleshooting/img/OneLogin-userAdd.png Binary files differdeleted file mode 100644 index 54c1ecd2e68..00000000000 --- a/doc/administration/troubleshooting/img/OneLogin-userAdd.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/azure_configure_group_claim.png b/doc/administration/troubleshooting/img/azure_configure_group_claim.png Binary files differdeleted file mode 100644 index 9d8c5348273..00000000000 --- a/doc/administration/troubleshooting/img/azure_configure_group_claim.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/okta_admin_panel_v13_9.png b/doc/administration/troubleshooting/img/okta_admin_panel_v13_9.png Binary files differdeleted file mode 100644 index 2ebb1f0112c..00000000000 --- a/doc/administration/troubleshooting/img/okta_admin_panel_v13_9.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/okta_saml_settings.png b/doc/administration/troubleshooting/img/okta_saml_settings.png Binary files differdeleted file mode 100644 index ee275ece369..00000000000 --- a/doc/administration/troubleshooting/img/okta_saml_settings.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/okta_setting_username.png b/doc/administration/troubleshooting/img/okta_setting_username.png Binary files differdeleted file mode 100644 index 5f87d14bb8b..00000000000 --- a/doc/administration/troubleshooting/img/okta_setting_username.png +++ /dev/null diff --git a/doc/administration/troubleshooting/index.md b/doc/administration/troubleshooting/index.md index 7fe731bda66..429dc79e95f 100644 --- a/doc/administration/troubleshooting/index.md +++ b/doc/administration/troubleshooting/index.md @@ -13,15 +13,12 @@ installation. - [SSL](ssl.md) - [Geo](../geo/replication/troubleshooting.md) -- [Elasticsearch](elasticsearch.md) -- [Sidekiq](sidekiq.md) - [GitLab Rails console cheat sheet](gitlab_rails_cheat_sheet.md) -- [Group SAML and SCIM troubleshooting](group_saml_scim.md) **(PREMIUM SAAS)** +- [Example group SAML and SCIM configurations](../../user/group/saml_sso/example_saml_config.md) - [Kubernetes cheat sheet](https://docs.gitlab.com/charts/troubleshooting/kubernetes_cheat_sheet.html) - [Linux cheat sheet](linux_cheat_sheet.md) -- [Parsing GitLab logs with `jq`](log_parsing.md) +- [Parsing GitLab logs with `jq`](../logs/log_parsing.md) - [Diagnostics tools](diagnostics_tools.md) -- [Tracing requests with correlation ID](tracing_correlation_id.md) Some feature documentation pages also have a troubleshooting section at the end that you can check for feature-specific help. diff --git a/doc/administration/troubleshooting/linux_cheat_sheet.md b/doc/administration/troubleshooting/linux_cheat_sheet.md index 66d5fb82936..6ff6e562a7d 100644 --- a/doc/administration/troubleshooting/linux_cheat_sheet.md +++ b/doc/administration/troubleshooting/linux_cheat_sheet.md @@ -14,7 +14,7 @@ having an issue with GitLab, you may want to check your [support options](https: first, before attempting to use this information. WARNING: -It is [beyond the scope of GitLab Support to assist in systems administration](https://about.gitlab.com/support/statement-of-support.html#training). GitLab administrators are expected to know these commands for their distribution +It is [beyond the scope of GitLab Support to assist in systems administration](https://about.gitlab.com/support/statement-of-support/#training). GitLab administrators are expected to know these commands for their distribution of choice. If you are a GitLab Support Engineer, consider this a cross-reference to translate `yum` -> `apt-get` and the like. @@ -204,24 +204,39 @@ or you can build it from source if you have the Rust compiler. #### How to use the tool -First run the tool with no arguments other than the strace output filename to get -a summary of the top processes sorted by time spent actively performing tasks. You -can also sort based on total time, # of system calls made, PID #, and # of child processes -using the `-S` or `--sort` flag. The number of results defaults to 25 processes, but +First run the tool with `summary` flag to get a summary of the top processes sorted by time spent actively performing tasks. +You can also sort based on total time, # of system calls made, PID #, and # of child processes +using the `-s` or `--sort` flag. The number of results defaults to 25 processes, but can be changed using the `-c`/`--count` option. See `--help` for full details. ```shell -$ ./strace-parser strace.txt +$ ./strace-parser sidekiq_trace.txt summary -c15 -s=pid -Top 25 PIDs +Top 15 PIDs by PID # ----------- - pid active (ms) wait (ms) total (ms) % active syscalls - ---------- ---------- --------- --------- --------- --------- - 8795 689.072 45773.832 46462.902 16.89% 23018 - 13408 679.432 55910.891 56590.320 16.65% 28593 - 6423 554.822 13175.485 13730.308 13.60% 13735 -... + pid actv (ms) wait (ms) user (ms) total (ms) % of actv syscalls children + ------- ---------- ---------- ---------- ---------- --------- --------- --------- + 16706 0.000 0.000 0.000 0.000 0.00% 0 0 + 16708 0.000 0.000 0.000 0.000 0.00% 0 0 + 16716 0.000 0.000 0.000 0.000 0.00% 0 0 + 16717 0.000 0.000 0.000 0.000 0.00% 0 0 + 16718 0.000 0.000 0.000 0.000 0.00% 0 0 + 16719 0.000 0.000 0.000 0.000 0.00% 0 0 + 16720 0.389 9796.434 1.090 9797.912 0.02% 16 0 + 16721 0.000 0.000 0.000 0.000 0.00% 0 0 + 16722 0.000 0.000 0.000 0.000 0.00% 0 0 + 16723 0.000 0.000 0.000 0.000 0.00% 0 0 + 16804 0.218 11099.535 1.881 11101.634 0.01% 36 0 + 16813 0.000 0.000 0.000 0.000 0.00% 0 0 + 16814 1.740 11825.640 4.616 11831.996 0.10% 57 0 + 16815 2.364 12039.993 7.669 12050.026 0.14% 80 0 + 16816 0.000 0.000 0.000 0.000 0.00% 0 0 + +PIDs 93 +real 0m12.287s +user 0m1.474s +sys 0m1.686s ``` Based on the summary, you can then view the details of system calls made by one or more @@ -229,36 +244,38 @@ processes using the `-p`/`--pid` for a specific process, or `-s`/`--stats` flags a sorted list. `--stats` takes the same sorting and count options as summary. ```shell -$ ./strace-parse strace.text -p 6423 - -PID 6423 -13735 syscalls, active time: 554.822ms, total time: 13730.308ms - - syscall count total max avg min errors - (ms) (ms) (ms) (ms) - --------------- -------- ---------- ---------- ---------- ---------- -------- - epoll_wait 628 13175.485 21.259 20.980 0.020 - clock_gettime 7326 199.500 0.249 0.027 0.013 - stat 2101 110.768 19.056 0.053 0.017 ENOENT: 2076 - ... +./strace-parser sidekiq_trace.txt p 16815 + +PID 16815 + + 80 syscalls, active time: 2.364ms, user time: 7.669ms, total time: 12050.026ms + start time: 22:46:14.830267 end time: 22:46:26.880293 + + syscall count total (ms) max (ms) avg (ms) min (ms) errors + ----------------- -------- ---------- ---------- ---------- ---------- -------- + futex 5 10100.229 5400.106 2020.046 0.022 ETIMEDOUT: 2 + restart_syscall 1 1939.764 1939.764 1939.764 1939.764 ETIMEDOUT: 1 + getpid 33 1.020 0.046 0.031 0.018 + clock_gettime 14 0.420 0.038 0.030 0.021 + stat 6 0.277 0.072 0.046 0.031 + read 6 0.170 0.036 0.028 0.020 + openat 3 0.126 0.045 0.042 0.038 + close 3 0.099 0.034 0.033 0.031 + lseek 3 0.089 0.035 0.030 0.021 + ioctl 3 0.082 0.033 0.027 0.023 ENOTTY: 3 + fstat 3 0.081 0.034 0.027 0.022 --------------- - Parent PID: 495 - Child PIDs: 8383, 8418, 8419, 8420, 8421 + Slowest file open times for PID 16815: - Slowest file access times for PID 6423: - - open (ms) timestamp error file name - ----------- --------------- --------------- ---------- - 29.818 10:53:11.528954 /srv/gitlab-data/builds/2018_08/6174/954448.log - 12.309 10:53:46.708274 /srv/gitlab-data/builds/2018_08/5342/954186.log - 0.039 10:53:49.222110 /opt/gitlab/embedded/service/gitlab-rails/app/views/events/event/_note.html.haml - 0.035 10:53:49.125115 /opt/gitlab/embedded/service/gitlab-rails/app/views/events/event/_push.html.haml - ... + dur (ms) timestamp error file name + ---------- --------------- --------------- --------- + 0.045 22:46:16.771318 - /opt/gitlab/embedded/service/gitlab-rails/config/database.yml + 0.043 22:46:26.877954 - /opt/gitlab/embedded/service/gitlab-rails/config/database.yml + 0.038 22:46:22.174610 - /opt/gitlab/embedded/service/gitlab-rails/config/database.yml ``` -In the example above, we can see that file opening times on `/srv/gitlab-data` are -extremely slow, about 100X slower than `/opt/gitlab`. +In the example above, we can see which files took longer to open for `PID 16815`. When nothing stands out in the results, a good way to get more context is to run `strace` on your own GitLab instance while performing the action performed by the customer, diff --git a/doc/administration/troubleshooting/log_parsing.md b/doc/administration/troubleshooting/log_parsing.md index 0320b2e52ce..929a49494be 100644 --- a/doc/administration/troubleshooting/log_parsing.md +++ b/doc/administration/troubleshooting/log_parsing.md @@ -1,316 +1,11 @@ --- -stage: Systems -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../logs/log_parsing.md' +remove_date: '2022-10-24' --- -# Parsing GitLab logs with `jq` **(FREE SELF)** +This document was moved to [another location](../logs/log_parsing.md). -We recommend using log aggregation and search tools like Kibana and Splunk whenever possible, -but if they are not available you can still quickly parse -[GitLab logs](../logs.md) in JSON format -(the default in GitLab 12.0 and later) using [`jq`](https://stedolan.github.io/jq/). - -NOTE: -Specifically for summarizing error events and basic usage statistics, -the GitLab Support Team provides the specialised -[`fast-stats` tool](https://gitlab.com/gitlab-com/support/toolbox/fast-stats/#when-to-use-it). - -## What is JQ? - -As noted in its [manual](https://stedolan.github.io/jq/manual/), `jq` is a command-line JSON processor. The following examples -include use cases targeted for parsing GitLab log files. - -## Parsing Logs - -The examples listed below address their respective log files by -their relative Omnibus paths and default filenames. -Find the respective full paths in the [GitLab logs sections](../logs.md#production_jsonlog). - -### General Commands - -#### Pipe colorized `jq` output into `less` - -```shell -jq . <FILE> -C | less -R -``` - -#### Search for a term and pretty-print all matching lines - -```shell -grep <TERM> <FILE> | jq . -``` - -#### Skip invalid lines of JSON - -```shell -jq -cR 'fromjson?' file.json | jq <COMMAND> -``` - -By default `jq` errors out when it encounters a line that is not valid JSON. -This skips over all invalid lines and parses the rest. - -#### Print a JSON log's time range - -```shell -cat log.json | (head -1; tail -1) | jq '.time' -``` - -Use `zcat` if the file has been rotated and compressed: - -```shell -zcat @400000006026b71d1a7af804.s | (head -1; tail -1) | jq '.time' - -zcat some_json.log.25.gz | (head -1; tail -1) | jq '.time' -``` - -#### Get activity for correlation ID across multiple JSON logs in chronological order - -```shell -grep -hR <correlationID> | jq -c -R 'fromjson?' | jq -C -s 'sort_by(.time)' | less -R -``` - -### Parsing `gitlab-rails/production_json.log` and `gitlab-rails/api_json.log` - -#### Find all requests with a 5XX status code - -```shell -jq 'select(.status >= 500)' <FILE> -``` - -#### Top 10 slowest requests - -```shell -jq -s 'sort_by(-.duration_s) | limit(10; .[])' <FILE> -``` - -#### Find and pretty print all requests related to a project - -```shell -grep <PROJECT_NAME> <FILE> | jq . -``` - -#### Find all requests with a total duration > 5 seconds - -```shell -jq 'select(.duration_s > 5000)' <FILE> -``` - -#### Find all project requests with more than 5 rugged calls - -```shell -grep <PROJECT_NAME> <FILE> | jq 'select(.rugged_calls > 5)' -``` - -#### Find all requests with a Gitaly duration > 10 seconds - -```shell -jq 'select(.gitaly_duration_s > 10000)' <FILE> -``` - -#### Find all requests with a queue duration > 10 seconds - -```shell -jq 'select(.queue_duration_s > 10000)' <FILE> -``` - -#### Top 10 requests by # of Gitaly calls - -```shell -jq -s 'map(select(.gitaly_calls != null)) | sort_by(-.gitaly_calls) | limit(10; .[])' <FILE> -``` - -### Parsing `gitlab-rails/production_json.log` - -#### Print the top three controller methods by request volume and their three longest durations - -```shell -jq -s -r 'group_by(.controller+.action) | sort_by(-length) | limit(3; .[]) | sort_by(-.duration_s) | "CT: \(length)\tMETHOD: \(.[0].controller)#\(.[0].action)\tDURS: \(.[0].duration_s), \(.[1].duration_s), \(.[2].duration_s)"' production_json.log -``` - -**Example output** - -```plaintext -CT: 2721 METHOD: SessionsController#new DURS: 844.06, 713.81, 704.66 -CT: 2435 METHOD: MetricsController#index DURS: 299.29, 284.01, 158.57 -CT: 1328 METHOD: Projects::NotesController#index DURS: 403.99, 386.29, 384.39 -``` - -### Parsing `gitlab-rails/api_json.log` - -#### Print top three routes with request count and their three longest durations - -```shell -jq -s -r 'group_by(.route) | sort_by(-length) | limit(3; .[]) | sort_by(-.duration_s) | "CT: \(length)\tROUTE: \(.[0].route)\tDURS: \(.[0].duration_s), \(.[1].duration_s), \(.[2].duration_s)"' api_json.log -``` - -**Example output** - -```plaintext -CT: 2472 ROUTE: /api/:version/internal/allowed DURS: 56402.65, 38411.43, 19500.41 -CT: 297 ROUTE: /api/:version/projects/:id/repository/tags DURS: 731.39, 685.57, 480.86 -CT: 190 ROUTE: /api/:version/projects/:id/repository/commits DURS: 1079.02, 979.68, 958.21 -``` - -### Print top API user agents - -```shell -jq --raw-output '[.route, .ua] | @tsv' api_json.log | sort | uniq -c | sort -n -``` - -**Example output**: - -```plaintext - 89 /api/:version/usage_data/increment_unique_users # plus browser details - 567 /api/:version/jobs/:id/trace gitlab-runner # plus version details -1234 /api/:version/internal/allowed GitLab-Shell -``` - -This sample response seems normal. A custom tool or script might be causing a high load -if the output contains many: - -- Third party libraries like `python-requests` or `curl`. -- [GitLab CLI clients](https://about.gitlab.com/partners/technology-partners/#cli-clients). - -You can also [use `fast-stats top`](#parsing-gitlab-logs-with-jq) to extract performance statistics. - -### Parsing `gitlab-workhorse/current` - -### Print top Workhorse user agents - -```shell -jq --raw-output '[.uri, .user_agent] | @tsv' current | sort | uniq -c | sort -n -``` - -**Example output**: - -```plaintext - 89 /api/graphql # plus browser details - 567 /api/v4/internal/allowed GitLab-Shell -1234 /api/v4/jobs/request gitlab-runner # plus version details -``` - -Similar to the [API `ua` data](#print-top-api-user-agents), -deviations from this common order might indicate scripts that could be optimized. - -The performance impact of runners checking for new jobs can be reduced by increasing -[the `check_interval` setting](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section), -for example. - -### Parsing `gitlab-rails/geo.log` - -#### Find most common Geo sync errors - -If [the `geo:status` Rake task](../geo/replication/troubleshooting.md#sync-status-rake-task) -repeatedly reports that some items never reach 100%, -the following command helps to focus on the most common errors. - -```shell -jq --raw-output 'select(.severity == "ERROR") | [.project_path, .message] | @tsv' geo.log | sort | uniq -c | sort | tail -``` - -### Parsing `gitaly/current` - -Use the following examples to [troubleshoot Gitaly](../gitaly/troubleshooting.md). - -#### Find all Gitaly requests sent from web UI - -```shell -jq 'select(."grpc.meta.client_name" == "gitlab-web")' current -``` - -#### Find all failed Gitaly requests - -```shell -jq 'select(."grpc.code" != null and ."grpc.code" != "OK")' current -``` - -#### Find all requests that took longer than 30 seconds - -```shell -jq 'select(."grpc.time_ms" > 30000)' current -``` - -#### Print top ten projects by request volume and their three longest durations - -```shell -jq --raw-output --slurp ' - map( - select( - ."grpc.request.glProjectPath" != null - and ."grpc.request.glProjectPath" != "" - and ."grpc.time_ms" != null - ) - ) - | group_by(."grpc.request.glProjectPath") - | sort_by(-length) - | limit(10; .[]) - | sort_by(-."grpc.time_ms") - | [ - length, - .[0]."grpc.time_ms", - .[1]."grpc.time_ms", - .[2]."grpc.time_ms", - .[0]."grpc.request.glProjectPath" - ] - | @sh' current \ -| awk 'BEGIN { printf "%7s %10s %10s %10s\t%s\n", "CT", "MAX DURS", "", "", "PROJECT" } - { printf "%7u %7u ms, %7u ms, %7u ms\t%s\n", $1, $2, $3, $4, $5 }' -``` - -**Example output** - -```plaintext - CT MAX DURS PROJECT - 206 4898 ms, 1101 ms, 1032 ms 'groupD/project4' - 109 1420 ms, 962 ms, 875 ms 'groupEF/project56' - 663 106 ms, 96 ms, 94 ms 'groupABC/project123' - ... -``` - -#### Find all projects affected by a fatal Git problem - -```shell -grep "fatal: " current | \ - jq '."grpc.request.glProjectPath"' | \ - sort | uniq -``` - -### Parsing `gitlab-shell/gitlab-shell.log` - -For investigating Git calls via SSH, from [GitLab 12.10](https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/367). - -Find the top 20 calls by project and user: - -```shell -jq --raw-output --slurp ' - map( - select( - .username != null and - .gl_project_path !=null - ) - ) - | group_by(.username+.gl_project_path) - | sort_by(-length) - | limit(20; .[]) - | "count: \(length)\tuser: \(.[0].username)\tproject: \(.[0].gl_project_path)" ' \ - gitlab-shell.log -``` - -Find the top 20 calls by project, user, and command: - -```shell -jq --raw-output --slurp ' - map( - select( - .command != null and - .username != null and - .gl_project_path !=null - ) - ) - | group_by(.username+.gl_project_path+.command) - | sort_by(-length) - | limit(20; .[]) - | "count: \(length)\tcommand: \(.[0].command)\tuser: \(.[0].username)\tproject: \(.[0].gl_project_path)" ' \ - gitlab-shell.log -``` +<!-- This redirect file can be deleted after <2022-10-24>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/troubleshooting/postgresql.md b/doc/administration/troubleshooting/postgresql.md index 61b661d45f8..9e3d3d47a10 100644 --- a/doc/administration/troubleshooting/postgresql.md +++ b/doc/administration/troubleshooting/postgresql.md @@ -216,7 +216,7 @@ because the statement timeout was too short: pg_dump: error: Error message from server: server closed the connection unexpectedly ``` -You may also see errors in the [PostgreSQL logs](../logs.md#postgresql-logs): +You may also see errors in the [PostgreSQL logs](../logs/index.md#postgresql-logs): ```plaintext canceling statement due to statement timeout diff --git a/doc/administration/troubleshooting/sidekiq.md b/doc/administration/troubleshooting/sidekiq.md index 40bfe22ac63..e49e0ed4f1c 100644 --- a/doc/administration/troubleshooting/sidekiq.md +++ b/doc/administration/troubleshooting/sidekiq.md @@ -1,395 +1,11 @@ --- -stage: Systems -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../sidekiq/sidekiq_troubleshooting.md' +remove_date: '2022-11-11' --- -# Troubleshooting Sidekiq **(FREE SELF)** +This document was moved to [another location](../sidekiq/sidekiq_troubleshooting.md). -Sidekiq is the background job processor GitLab uses to asynchronously run -tasks. When things go wrong it can be difficult to troubleshoot. These -situations also tend to be high-pressure because a production system job queue -may be filling up. Users will notice when this happens because new branches -may not show up and merge requests may not be updated. The following are some -troubleshooting steps to help you diagnose the bottleneck. - -GitLab administrators/users should consider working through these -debug steps with GitLab Support so the backtraces can be analyzed by our team. -It may reveal a bug or necessary improvement in GitLab. - -In any of the backtraces, be wary of suspecting cases where every -thread appears to be waiting in the database, Redis, or waiting to acquire -a mutex. This **may** mean there's contention in the database, for example, -but look for one thread that is different than the rest. This other thread -may be using all available CPU, or have a Ruby Global Interpreter Lock, -preventing other threads from continuing. - -## Log arguments to Sidekiq jobs - -[In GitLab 13.6 and later](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/44853) -some arguments passed to Sidekiq jobs are logged by default. -To avoid logging sensitive information (for instance, password reset tokens), -GitLab logs numeric arguments for all workers, with overrides for some specific -workers where their arguments are not sensitive. - -Example log output: - -```json -{"severity":"INFO","time":"2020-06-08T14:37:37.892Z","class":"AdminEmailsWorker","args":["[FILTERED]","[FILTERED]","[FILTERED]"],"retry":3,"queue":"admin_emails","backtrace":true,"jid":"9e35e2674ac7b12d123e13cc","created_at":"2020-06-08T14:37:37.373Z","meta.user":"root","meta.caller_id":"Admin::EmailsController#create","correlation_id":"37D3lArJmT1","uber-trace-id":"2d942cc98cc1b561:6dc94409cfdd4d77:9fbe19bdee865293:1","enqueued_at":"2020-06-08T14:37:37.410Z","pid":65011,"message":"AdminEmailsWorker JID-9e35e2674ac7b12d123e13cc: done: 0.48085 sec","job_status":"done","scheduling_latency_s":0.001012,"redis_calls":9,"redis_duration_s":0.004608,"redis_read_bytes":696,"redis_write_bytes":6141,"duration_s":0.48085,"cpu_s":0.308849,"completed_at":"2020-06-08T14:37:37.892Z","db_duration_s":0.010742} -{"severity":"INFO","time":"2020-06-08T14:37:37.894Z","class":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper","wrapped":"ActionMailer::MailDeliveryJob","queue":"mailers","args":["[FILTERED]"],"retry":3,"backtrace":true,"jid":"e47a4f6793d475378432e3c8","created_at":"2020-06-08T14:37:37.884Z","meta.user":"root","meta.caller_id":"AdminEmailsWorker","correlation_id":"37D3lArJmT1","uber-trace-id":"2d942cc98cc1b561:29344de0f966446d:5c3b0e0e1bef987b:1","enqueued_at":"2020-06-08T14:37:37.885Z","pid":65011,"message":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper JID-e47a4f6793d475378432e3c8: start","job_status":"start","scheduling_latency_s":0.009473} -{"severity":"INFO","time":"2020-06-08T14:39:50.648Z","class":"NewIssueWorker","args":["455","1"],"retry":3,"queue":"new_issue","backtrace":true,"jid":"a24af71f96fd129ec47f5d1e","created_at":"2020-06-08T14:39:50.643Z","meta.user":"root","meta.project":"h5bp/html5-boilerplate","meta.root_namespace":"h5bp","meta.caller_id":"Projects::IssuesController#create","correlation_id":"f9UCZHqhuP7","uber-trace-id":"28f65730f99f55a3:a5d2b62dec38dffc:48ddd092707fa1b7:1","enqueued_at":"2020-06-08T14:39:50.646Z","pid":65011,"message":"NewIssueWorker JID-a24af71f96fd129ec47f5d1e: start","job_status":"start","scheduling_latency_s":0.001144} -``` - -When using [Sidekiq JSON logging](../logs.md#sidekiqlog), -arguments logs are limited to a maximum size of 10 kilobytes of text; -any arguments after this limit are discarded and replaced with a -single argument containing the string `"..."`. - -You can set `SIDEKIQ_LOG_ARGUMENTS` [environment variable](https://docs.gitlab.com/omnibus/settings/environment-variables.html) -to `0` (false) to disable argument logging. - -Example: - -```ruby -gitlab_rails['env'] = {"SIDEKIQ_LOG_ARGUMENTS" => "0"} -``` - -In GitLab 13.5 and earlier, set `SIDEKIQ_LOG_ARGUMENTS` to `1` to start logging arguments passed to Sidekiq. - -## Thread dump - -Send the Sidekiq process ID the `TTIN` signal to output thread -backtraces in the log file. - -```shell -kill -TTIN <sidekiq_pid> -``` - -Check in `/var/log/gitlab/sidekiq/current` or `$GITLAB_HOME/log/sidekiq.log` for -the backtrace output. The backtraces are lengthy and generally start with -several `WARN` level messages. Here's an example of a single thread's backtrace: - -```plaintext -2016-04-13T06:21:20.022Z 31517 TID-orn4urby0 WARN: ActiveRecord::RecordNotFound: Couldn't find Note with 'id'=3375386 -2016-04-13T06:21:20.022Z 31517 TID-orn4urby0 WARN: /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.2/lib/active_record/core.rb:155:in `find' -/opt/gitlab/embedded/service/gitlab-rails/app/workers/new_note_worker.rb:7:in `perform' -/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:150:in `execute_job' -/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:132:in `block (2 levels) in process' -/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:127:in `block in invoke' -/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/memory_killer.rb:17:in `call' -/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke' -/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/arguments_logger.rb:6:in `call' -... -``` - -In some cases Sidekiq may be hung and unable to respond to the `TTIN` signal. -Move on to other troubleshooting methods if this happens. - -## Ruby profiling with `rbspy` - -[rbspy](https://rbspy.github.io) is an easy to use and low-overhead Ruby profiler that can be used to create -flamegraph-style diagrams of CPU usage by Ruby processes. - -No changes to GitLab are required to use it and it has no dependencies. To install it: - -1. Download the binary from the [`rbspy` releases page](https://github.com/rbspy/rbspy/releases). -1. Make the binary executable. - -To profile a Sidekiq worker for one minute, run: - -```shell -sudo ./rbspy record --pid <sidekiq_pid> --duration 60 --file /tmp/sidekiq_profile.svg -``` - - - -In this example of a flamegraph generated by `rbspy`, almost all of the Sidekiq process's time is spent in `rev_parse`, a native C -function in Rugged. In the stack, we can see `rev_parse` is being called by the `ExpirePipelineCacheWorker`. - -## Process profiling with `perf` - -Linux has a process profiling tool called `perf` that is helpful when a certain -process is eating up a lot of CPU. If you see high CPU usage and Sidekiq isn't -responding to the `TTIN` signal, this is a good next step. - -If `perf` is not installed on your system, install it with `apt-get` or `yum`: - -```shell -# Debian -sudo apt-get install linux-tools - -# Ubuntu (may require these additional Kernel packages) -sudo apt-get install linux-tools-common linux-tools-generic linux-tools-`uname -r` - -# Red Hat/CentOS -sudo yum install perf -``` - -Run `perf` against the Sidekiq PID: - -```shell -sudo perf record -p <sidekiq_pid> -``` - -Let this run for 30-60 seconds and then press Ctrl-C. Then view the `perf` report: - -```shell -$ sudo perf report - -# Sample output -Samples: 348K of event 'cycles', Event count (approx.): 280908431073 - 97.69% ruby nokogiri.so [.] xmlXPathNodeSetMergeAndClear - 0.18% ruby libruby.so.2.1.0 [.] objspace_malloc_increase - 0.12% ruby libc-2.12.so [.] _int_malloc - 0.10% ruby libc-2.12.so [.] _int_free -``` - -Above you see sample output from a `perf` report. It shows that 97% of the CPU is -being spent inside Nokogiri and `xmlXPathNodeSetMergeAndClear`. For something -this obvious you should then go investigate what job in GitLab would use -Nokogiri and XPath. Combine with `TTIN` or `gdb` output to show the -corresponding Ruby code where this is happening. - -## The GNU Project Debugger (`gdb`) - -`gdb` can be another effective tool for debugging Sidekiq. It gives you a little -more interactive way to look at each thread and see what's causing problems. - -Attaching to a process with `gdb` suspends the normal operation -of the process (Sidekiq does not process jobs while `gdb` is attached). - -Start by attaching to the Sidekiq PID: - -```shell -gdb -p <sidekiq_pid> -``` - -Then gather information on all the threads: - -```plaintext -info threads - -# Example output -30 Thread 0x7fe5fbd63700 (LWP 26060) 0x0000003f7cadf113 in poll () from /lib64/libc.so.6 -29 Thread 0x7fe5f2b3b700 (LWP 26533) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 -28 Thread 0x7fe5f2a3a700 (LWP 26534) 0x0000003f7ce0ba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 -27 Thread 0x7fe5f2939700 (LWP 26535) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 -26 Thread 0x7fe5f2838700 (LWP 26537) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 -25 Thread 0x7fe5f2737700 (LWP 26538) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 -24 Thread 0x7fe5f2535700 (LWP 26540) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 -23 Thread 0x7fe5f2434700 (LWP 26541) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 -22 Thread 0x7fe5f2232700 (LWP 26543) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 -21 Thread 0x7fe5f2131700 (LWP 26544) 0x00007fe5f7b570f0 in xmlXPathNodeSetMergeAndClear () -from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so -... -``` - -If you see a suspicious thread, like the Nokogiri one above, you may want -to get more information: - -```plaintext -thread 21 -bt - -# Example output -#0 0x00007ff0d6afe111 in xmlXPathNodeSetMergeAndClear () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so -#1 0x00007ff0d6b0b836 in xmlXPathNodeCollectAndTest () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so -#2 0x00007ff0d6b09037 in xmlXPathCompOpEval () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so -#3 0x00007ff0d6b09017 in xmlXPathCompOpEval () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so -#4 0x00007ff0d6b092e0 in xmlXPathCompOpEval () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so -#5 0x00007ff0d6b0bc37 in xmlXPathRunEval () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so -#6 0x00007ff0d6b0be5f in xmlXPathEvalExpression () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so -#7 0x00007ff0d6a97dc3 in evaluate (argc=2, argv=0x1022d058, self=<value optimized out>) at xml_xpath_context.c:221 -#8 0x00007ff0daeab0ea in vm_call_cfunc_with_frame (th=0x1022a4f0, reg_cfp=0x1032b810, ci=<value optimized out>) at vm_insnhelper.c:1510 -``` - -To output a backtrace from all threads at once: - -```plaintext -set pagination off -thread apply all bt -``` - -Once you're done debugging with `gdb`, be sure to detach from the process and -exit: - -```plaintext -detach -exit -``` - -## Sidekiq kill signals - -TTIN was described above as the signal to print backtraces for logging, however -Sidekiq responds to other signals as well. For example, TSTP and TERM can be used -to gracefully shut Sidekiq down, see -[the Sidekiq Signals docs](https://github.com/mperham/sidekiq/wiki/Signals#ttin). - -## Check for blocking queries - -Sometimes the speed at which Sidekiq processes jobs can be so fast that it can -cause database contention. Check for blocking queries when backtraces above -show that many threads are stuck in the database adapter. - -The PostgreSQL wiki has details on the query you can run to see blocking -queries. The query is different based on PostgreSQL version. See -[Lock Monitoring](https://wiki.postgresql.org/wiki/Lock_Monitoring) for -the query details. - -## Managing Sidekiq queues - -It is possible to use [Sidekiq API](https://github.com/mperham/sidekiq/wiki/API) -to perform a number of troubleshooting steps on Sidekiq. - -These are the administrative commands and it should only be used if currently -administration interface is not suitable due to scale of installation. - -All these commands should be run using `gitlab-rails console`. - -### View the queue size - -```ruby -Sidekiq::Queue.new("pipeline_processing:build_queue").size -``` - -### Enumerate all enqueued jobs - -```ruby -queue = Sidekiq::Queue.new("chaos:chaos_sleep") -queue.each do |job| - # job.klass # => 'MyWorker' - # job.args # => [1, 2, 3] - # job.jid # => jid - # job.queue # => chaos:chaos_sleep - # job["retry"] # => 3 - # job.item # => { - # "class"=>"Chaos::SleepWorker", - # "args"=>[1000], - # "retry"=>3, - # "queue"=>"chaos:chaos_sleep", - # "backtrace"=>true, - # "queue_namespace"=>"chaos", - # "jid"=>"39bc482b823cceaf07213523", - # "created_at"=>1566317076.266069, - # "correlation_id"=>"c323b832-a857-4858-b695-672de6f0e1af", - # "enqueued_at"=>1566317076.26761}, - # } - - # job.delete if job.jid == 'abcdef1234567890' -end -``` - -### Enumerate currently running jobs - -```ruby -workers = Sidekiq::Workers.new -workers.each do |process_id, thread_id, work| - # process_id is a unique identifier per Sidekiq process - # thread_id is a unique identifier per thread - # work is a Hash which looks like: - # {"queue"=>"chaos:chaos_sleep", - # "payload"=> - # { "class"=>"Chaos::SleepWorker", - # "args"=>[1000], - # "retry"=>3, - # "queue"=>"chaos:chaos_sleep", - # "backtrace"=>true, - # "queue_namespace"=>"chaos", - # "jid"=>"b2a31e3eac7b1a99ff235869", - # "created_at"=>1566316974.9215662, - # "correlation_id"=>"e484fb26-7576-45f9-bf21-b99389e1c53c", - # "enqueued_at"=>1566316974.9229589}, - # "run_at"=>1566316974}], -end -``` - -### Remove Sidekiq jobs for given parameters (destructive) - -The general method to kill jobs conditionally is the following command, which -removes jobs that are queued but not started. Running jobs can not be killed. - -```ruby -queue = Sidekiq::Queue.new('<queue name>') -queue.each { |job| job.delete if <condition>} -``` - -Have a look at the section below for cancelling running jobs. - -In the method above, `<queue-name>` is the name of the queue that contains the jobs you want to delete and `<condition>` decides which jobs get deleted. - -Commonly, `<condition>` references the job arguments, which depend on the type of job in question. To find the arguments for a specific queue, you can have a look at the `perform` function of the related worker file, commonly found at `/app/workers/<queue-name>_worker.rb`. - -For example, `repository_import` has `project_id` as the job argument, while `update_merge_requests` has `project_id, user_id, oldrev, newrev, ref`. - -Arguments need to be referenced by their sequence ID using `job.args[<id>]` because `job.args` is a list of all arguments provided to the Sidekiq job. - -Here are some examples: - -```ruby -queue = Sidekiq::Queue.new('update_merge_requests') -# In this example, we want to remove any update_merge_requests jobs -# for the Project with ID 125 and ref `ref/heads/my_branch` -queue.each { |job| job.delete if job.args[0] == 125 and job.args[4] == 'ref/heads/my_branch' } -``` - -```ruby -# Cancelling jobs like: `RepositoryImportWorker.new.perform_async(100)` -id_list = [100] - -queue = Sidekiq::Queue.new('repository_import') -queue.each do |job| - job.delete if id_list.include?(job.args[0]) -end -``` - -### Remove specific job ID (destructive) - -```ruby -queue = Sidekiq::Queue.new('repository_import') -queue.each do |job| - job.delete if job.jid == 'my-job-id' -end -``` - -## Canceling running jobs (destructive) - -> Introduced in GitLab 12.3. - -This is highly risky operation and use it as last resort. -Doing that might result in data corruption, as the job -is interrupted mid-execution and it is not guaranteed -that proper rollback of transactions is implemented. - -```ruby -Gitlab::SidekiqDaemon::Monitor.cancel_job('job-id') -``` - -> This requires the Sidekiq to be run with `SIDEKIQ_MONITOR_WORKER=1` -> environment variable. - -To perform of the interrupt we use `Thread.raise` which -has number of drawbacks, as mentioned in [Why Ruby's Timeout is dangerous (and Thread.raise is terrifying)](https://jvns.ca/blog/2015/11/27/why-rubys-timeout-is-dangerous-and-thread-dot-raise-is-terrifying/): - -> This is where the implications get interesting, and terrifying. This means that an exception can get raised: -> -> - during a network request (ok, as long as the surrounding code is prepared to catch Timeout::Error) -> - during the cleanup for the network request -> - during a rescue block -> - while creating an object to save to the database afterwards -> - in any of your code, regardless of whether it could have possibly raised an exception before -> -> Nobody writes code to defend against an exception being raised on literally any line. That's not even possible. So Thread.raise is basically like a sneak attack on your code that could result in almost anything. It would probably be okay if it were pure-functional code that did not modify any state. But this is Ruby, so that's unlikely :) - -## Disable Rugged - -Calls into Rugged, Ruby bindings for `libgit2`, [lock the Sidekiq processes's GVL](https://silverhammermba.github.io/emberb/c/#c-in-ruby-threads), -blocking all jobs on that worker from proceeding. If Rugged calls performed by Sidekiq are slow, this can cause significant delays in -background task processing. - -By default, Rugged is used when Git repository data is stored on local storage or on an NFS mount. -[Using Rugged is recommended when using NFS](../nfs.md#improving-nfs-performance-with-gitlab), but if -you are using local storage, disabling Rugged can improve Sidekiq performance: - -```shell -sudo gitlab-rake gitlab:features:disable_rugged -``` +<!-- This redirect file can be deleted after <2022-11-11>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/troubleshooting/tracing_correlation_id.md b/doc/administration/troubleshooting/tracing_correlation_id.md index 418dd729066..ee59b7c2504 100644 --- a/doc/administration/troubleshooting/tracing_correlation_id.md +++ b/doc/administration/troubleshooting/tracing_correlation_id.md @@ -1,202 +1,11 @@ --- -stage: Systems -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../logs/tracing_correlation_id.md' +remove_date: '2022-11-12' --- -# Finding relevant log entries with a correlation ID **(FREE SELF)** +This document was moved to [another location](../logs/tracing_correlation_id.md). -GitLab instances log a unique request tracking ID (known as the -"correlation ID") for most requests. Each individual request to GitLab gets -its own correlation ID, which then gets logged in each GitLab component's logs for that -request. This makes it easier to trace behavior in a -distributed system. Without this ID it can be difficult or -impossible to match correlating log entries. - -## Identify the correlation ID for a request - -The correlation ID is logged in structured logs under the key `correlation_id` -and in all response headers GitLab sends under the header `x-request-id`. -You can find your correlation ID by searching in either place. - -### Getting the correlation ID in your browser - -You can use your browser's developer tools to monitor and inspect network -activity with the site that you're visiting. See the links below for network monitoring -documentation for some popular browsers. - -- [Network Monitor - Firefox Developer Tools](https://developer.mozilla.org/en-US/docs/Tools/Network_Monitor) -- [Inspect Network Activity In Chrome DevTools](https://developer.chrome.com/docs/devtools/network/) -- [Safari Web Development Tools](https://developer.apple.com/safari/tools/) -- [Microsoft Edge Network panel](https://docs.microsoft.com/en-us/microsoft-edge/devtools-guide-chromium/network/) - -To locate a relevant request and view its correlation ID: - -1. Enable persistent logging in your network monitor. Some actions in GitLab redirect you quickly after you submit a form, so this helps capture all relevant activity. -1. To help isolate the requests you are looking for, you can filter for `document` requests. -1. Select the request of interest to view further detail. -1. Go to the **Headers** section and look for **Response Headers**. There you should find an `x-request-id` header with a -value that was randomly generated by GitLab for the request. - -See the following example: - - - -### Getting the correlation ID from your logs - -Another approach to finding the correct correlation ID is to search or watch -your logs and find the `correlation_id` value for the log entry that you're -watching for. - -For example, let's say that you want learn what's happening or breaking when -you reproduce an action in GitLab. You could tail the GitLab logs, filtering -to requests by your user, and then watch the requests until you see what you're -interested in. - -### Getting the correlation ID from curl - -If you're using `curl` then you can use the verbose option to show request and response headers, as well as other debug information. - -```shell -➜ ~ curl --verbose "https://gitlab.example.com/api/v4/projects" -# look for a line that looks like this -< x-request-id: 4rAMkV3gof4 -``` - -#### Using jq - -This example uses [jq](https://stedolan.github.io/jq/) to filter results and -display values we most likely care about. - -```shell -sudo gitlab-ctl tail gitlab-rails/production_json.log | jq 'select(.username == "bob") | "User: \(.username), \(.method) \(.path), \(.controller)#\(.action), ID: \(.correlation_id)"' -``` - -```plaintext -"User: bob, GET /root/linux, ProjectsController#show, ID: U7k7fh6NpW3" -"User: bob, GET /root/linux/commits/master/signatures, Projects::CommitsController#signatures, ID: XPIHpctzEg1" -"User: bob, GET /root/linux/blob/master/README, Projects::BlobController#show, ID: LOt9hgi1TV4" -``` - -#### Using grep - -This example uses only `grep` and `tr`, which are more likely to be installed than `jq`. - -```shell -sudo gitlab-ctl tail gitlab-rails/production_json.log | grep '"username":"bob"' | tr ',' '\n' | egrep 'method|path|correlation_id' -``` - -```plaintext -{"method":"GET" -"path":"/root/linux" -"username":"bob" -"correlation_id":"U7k7fh6NpW3"} -{"method":"GET" -"path":"/root/linux/commits/master/signatures" -"username":"bob" -"correlation_id":"XPIHpctzEg1"} -{"method":"GET" -"path":"/root/linux/blob/master/README" -"username":"bob" -"correlation_id":"LOt9hgi1TV4"} -``` - -## Searching your logs for the correlation ID - -Once you have the correlation ID you can start searching for relevant log -entries. You can filter the lines by the correlation ID itself. -Combining a `find` and `grep` should be sufficient to find the entries you are looking for. - -```shell -# find <gitlab log directory> -type f -mtime -0 exec grep '<correlation ID>' '{}' '+' -find /var/log/gitlab -type f -mtime 0 -exec grep 'LOt9hgi1TV4' '{}' '+' -``` - -```plaintext -/var/log/gitlab/gitlab-workhorse/current:{"correlation_id":"LOt9hgi1TV4","duration_ms":2478,"host":"gitlab.domain.tld","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://gitlab.domain.tld/root/linux","remote_addr":"68.0.116.160:0","remote_ip":"[filtered]","status":200,"system":"http","time":"2019-09-17T22:17:19Z","uri":"/root/linux/blob/master/README?format=json\u0026viewer=rich","user_agent":"Mozilla/5.0 (Mac) Gecko Firefox/69.0","written_bytes":1743} -/var/log/gitlab/gitaly/current:{"correlation_id":"LOt9hgi1TV4","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"FindCommits","grpc.request.deadline":"2019-09-17T22:17:47Z","grpc.request.fullMethod":"/gitaly.CommitService/FindCommits","grpc.request.glProjectPath":"root/linux","grpc.request.glRepository":"project-1","grpc.request.repoPath":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"@hashed","grpc.service":"gitaly.CommitService","grpc.start_time":"2019-09-17T22:17:17Z","grpc.time_ms":2319.161,"level":"info","msg":"finished streaming call with code OK","peer.address":"@","span.kind":"server","system":"grpc","time":"2019-09-17T22:17:19Z"} -/var/log/gitlab/gitlab-rails/production_json.log:{"method":"GET","path":"/root/linux/blob/master/README","format":"json","controller":"Projects::BlobController","action":"show","status":200,"duration":2448.77,"view":0.49,"db":21.63,"time":"2019-09-17T22:17:19.800Z","params":[{"key":"viewer","value":"rich"},{"key":"namespace_id","value":"root"},{"key":"project_id","value":"linux"},{"key":"id","value":"master/README"}],"remote_ip":"[filtered]","user_id":2,"username":"bob","ua":"Mozilla/5.0 (Mac) Gecko Firefox/69.0","queue_duration":3.38,"gitaly_calls":1,"gitaly_duration":0.77,"rugged_calls":4,"rugged_duration_ms":28.74,"correlation_id":"LOt9hgi1TV4"} -``` - -### Searching in distributed architectures - -If you have done some horizontal scaling in your GitLab infrastructure, then -you must search across _all_ of your GitLab nodes. You can do this with -some sort of log aggregation software like Loki, ELK, Splunk, or others. - -You can use a tool like Ansible or PSSH (parallel SSH) that can execute identical commands across your servers in -parallel, or craft your own solution. - -### Viewing the request in the Performance Bar - -You can use the [performance bar](../monitoring/performance/performance_bar.md) to view interesting data including calls made to SQL and Gitaly. - -To view the data, the correlation ID of the request must match the same session as the user -viewing the performance bar. For API requests, this means that you must perform the request -using the session cookie of the signed-in user. - -For example, if you want to view the database queries executed for the following API endpoint: - -```shell -https://gitlab.com/api/v4/groups/2564205/projects?with_security_reports=true&page=1&per_page=1 -``` - -First, enable the **Developer Tools** panel. See [Getting the correlation ID in your browser](#getting-the-correlation-id-in-your-browser) for details on how to do this. - -After developer tools have been enabled, obtain a session cookie as follows: - -1. Visit <https://gitlab.com> while logged in. -1. Optional. Select **Fetch/XHR** request filter in the **Developer Tools** panel. This step is described for Google Chrome developer tools and is not strictly necessary, it just makes it easier to find the correct request. -1. Select the `results?request_id=<some-request-id>` request on the left hand side. -1. The session cookie is displayed under the `Request Headers` section of the `Headers` panel. Right-click on the cookie value and select `Copy value`. - - - -You have the value of the session cookie copied to your clipboard, for example: - -```shell -experimentation_subject_id=<subject-id>; _gitlab_session=<session-id>; event_filter=all; visitor_id=<visitor-id>; perf_bar_enabled=true; sidebar_collapsed=true; diff_view=inline; sast_entry_point_dismissed=true; auto_devops_settings_dismissed=true; cf_clearance=<cf-clearance>; collapsed_gutter=false; frequently_used_emojis=clap,thumbsup,rofl,tada,eyes,bow -``` - -Use the value of the session cookie to craft an API request by pasting it into a custom header of a `curl` request: - -```shell -$ curl --include "https://gitlab.com/api/v4/groups/2564205/projects?with_security_reports=true&page=1&per_page=1" \ ---header 'cookie: experimentation_subject_id=<subject-id>; _gitlab_session=<session-id>; event_filter=all; visitor_id=<visitor-id>; perf_bar_enabled=true; sidebar_collapsed=true; diff_view=inline; sast_entry_point_dismissed=true; auto_devops_settings_dismissed=true; cf_clearance=<cf-clearance>; collapsed_gutter=false; frequently_used_emojis=clap,thumbsup,rofl,tada,eyes,bow' - - date: Tue, 28 Sep 2021 03:55:33 GMT - content-type: application/json - ... - x-request-id: 01FGN8P881GF2E5J91JYA338Y3 - ... - [ - { - "id":27497069, - "description":"Analyzer for images used on live K8S containers based on Starboard" - }, - "container_registry_image_prefix":"registry.gitlab.com/gitlab-org/security-products/analyzers/cluster-image-scanning", - "..." - ] -``` - -The response contains the data from the API endpoint, and a `correlation_id` value, returned in the `x-request-id` header, as described in the [Identify the correlation ID for a request](#identify-the-correlation-id-for-a-request) section. - -You can then view the database details for this request: - -1. Paste the `x-request-id` value into the `request details` field of the [performance bar](../monitoring/performance/performance_bar.md) and press <kbd>Enter/Return</kbd>. This example uses the `x-request-id` value `01FGN8P881GF2E5J91JYA338Y3`, returned by the above response: - -  - -1. A new request is inserted into the `Request Selector` dropdown on the right-hand side of the Performance Bar. Select the new request to view the metrics of the API request: - -  - - <!-- vale gitlab.Substitutions = NO --> -1. Select the `pg` link in the Progress Bar to view the database queries executed by the API request: - -  - <!-- vale gitlab.Substitutions = YES --> - - The database query dialog is displayed: - -  +<!-- This redirect file can be deleted after 2022-11-12. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
\ No newline at end of file diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index 0bd46193d10..2dd8f1ed819 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -112,15 +112,15 @@ _The uploads are stored by default in `/home/git/gitlab/public/uploads`._ 1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following - lines: + lines, making sure to use the [appropriate ones for your provider](object_storage.md#connection-settings): ```yaml uploads: object_store: enabled: true remote_directory: "uploads" # The bucket name - connection: - provider: AWS # Only AWS supported at the moment + connection: # The lines in this block depend on your provider + provider: AWS aws_access_key_id: AWS_ACCESS_KEY_ID aws_secret_access_key: AWS_SECRET_ACCESS_KEY region: eu-central-1 |
