diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2023-11-14 11:41:52 +0300 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2023-11-14 11:41:52 +0300 |
commit | 585826cb22ecea5998a2c2a4675735c94bdeedac (patch) | |
tree | 5b05f0b30d33cef48963609e8a18a4dff260eab3 /doc/administration | |
parent | df221d036e5d0c6c0ee4d55b9c97f481ee05dee8 (diff) |
Add latest changes from gitlab-org/gitlab@16-6-stable-eev16.6.0-rc42
Diffstat (limited to 'doc/administration')
63 files changed, 1454 insertions, 652 deletions
diff --git a/doc/administration/audit_event_streaming/audit_event_types.md b/doc/administration/audit_event_streaming/audit_event_types.md index 3b2ae098469..88212045d8e 100644 --- a/doc/administration/audit_event_streaming/audit_event_types.md +++ b/doc/administration/audit_event_streaming/audit_event_types.md @@ -37,6 +37,7 @@ Audit event types belong to the following product categories. | Name | Description | Saved to database | Streamed | Introduced in | |:-----|:------------|:------------------|:---------|:--------------| | [`amazon_s3_configuration_created`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/132443) | Triggered when Amazon S3 configuration for audit events streaming is created| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.5](https://gitlab.com/gitlab-org/gitlab/-/issues/423229) | +| [`amazon_s3_configuration_deleted`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/133695) | Triggered when Amazon S3 configuration for audit events streaming is deleted.| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.5](https://gitlab.com/gitlab-org/gitlab/-/issues/423229) | | [`amazon_s3_configuration_updated`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/133691) | Triggered when Amazon S3 configuration for audit events streaming is updated.| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.5](https://gitlab.com/gitlab-org/gitlab/-/issues/423229) | | [`audit_events_streaming_headers_create`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/92068) | Triggered when a streaming header for audit events is created| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [15.3](https://gitlab.com/gitlab-org/gitlab/-/issues/366350) | | [`audit_events_streaming_headers_destroy`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/92068) | Triggered when a streaming header for audit events is deleted| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [15.3](https://gitlab.com/gitlab-org/gitlab/-/issues/366350) | @@ -44,6 +45,7 @@ Audit event types belong to the following product categories. | [`audit_events_streaming_instance_headers_destroy`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/127228) | Triggered when a streaming header for instance level external audit event destination is deleted| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.3](https://gitlab.com/gitlab-org/gitlab/-/issues/417433) | | [`audit_events_streaming_instance_headers_update`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/127228) | Triggered when a streaming header for instance level external audit event destination is updated| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.3](https://gitlab.com/gitlab-org/gitlab/-/issues/417433) | | [`create_event_streaming_destination`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/74632) | Event triggered when an external audit event destination is created| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [14.6](https://gitlab.com/gitlab-org/gitlab/-/issues/344664) | +| [`create_http_namespace_filter`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/136047) | Event triggered when a namespace filter for an external audit event destination for a top-level group is created.| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.6](https://gitlab.com/gitlab-org/gitlab/-/issues/424176) | | [`create_instance_event_streaming_destination`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/123882) | Event triggered when an instance level external audit event destination is created| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.2](https://gitlab.com/gitlab-org/gitlab/-/issues/404730) | | [`destroy_event_streaming_destination`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/74632) | Event triggered when an external audit event destination is deleted| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [14.6](https://gitlab.com/gitlab-org/gitlab/-/issues/344664) | | [`destroy_instance_event_streaming_destination`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/125846) | Event triggered when an instance level external audit event destination is deleted| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.2](https://gitlab.com/gitlab-org/gitlab/-/issues/404730) | @@ -171,6 +173,7 @@ Audit event types belong to the following product categories. | [`ci_variable_created`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91983) | Triggered when a CI variable is created at a project level| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [15.2](https://gitlab.com/gitlab-org/gitlab/-/issues/363090) | | [`ci_variable_deleted`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91983) | Triggered when a project's CI variable is deleted| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [15.2](https://gitlab.com/gitlab-org/gitlab/-/issues/363090) | | [`ci_variable_updated`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91983) | Triggered when a project's CI variable is updated| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [15.2](https://gitlab.com/gitlab-org/gitlab/-/issues/363090) | +| [`destroy_pipeline`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/135255) | Event triggered when a pipeline is deleted| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.6](https://gitlab.com/gitlab-org/gitlab/-/issues/339041) | ### Deployment management @@ -227,6 +230,8 @@ Audit event types belong to the following product categories. | Name | Description | Saved to database | Streamed | Introduced in | |:-----|:------------|:------------------|:---------|:--------------| +| [`create_ssh_certificate`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/134556) | Event triggered when an SSH certificate is created.| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.6](https://gitlab.com/gitlab-org/gitlab/-/issues/427413) | +| [`delete_ssh_certificate`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/134556) | Event triggered when an SSH certificate is deleted.| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.6](https://gitlab.com/gitlab-org/gitlab/-/issues/427413) | | [`group_created`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121005) | Event triggered when a group is created.| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.3](https://gitlab.com/gitlab-org/gitlab/-/issues/411595) | | [`group_lfs_enabled_updated`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/106079) | Event triggered when a groups lfs enabled is updated.| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [15.7](https://gitlab.com/gitlab-org/gitlab/-/issues/369323) | | [`group_membership_lock_updated`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/106079) | Event triggered when a groups membership lock is updated.| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [15.7](https://gitlab.com/gitlab-org/gitlab/-/issues/369323) | @@ -306,7 +311,6 @@ Audit event types belong to the following product categories. | Name | Description | Saved to database | Streamed | Introduced in | |:-----|:------------|:------------------|:---------|:--------------| | [`experiment_features_enabled_updated`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/118222) | Event triggered on toggling setting for enabling experiment AI features| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.0](https://gitlab.com/gitlab-org/gitlab/-/issues/404856/) | -| [`third_party_ai_features_enabled_updated`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/118222) | Event triggered on toggling setting for enabling third-party AI features| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.0](https://gitlab.com/gitlab-org/gitlab/-/issues/404856/) | ### Portfolio management @@ -442,6 +446,7 @@ Audit event types belong to the following product categories. | [`email_confirmation_sent`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/129261) | Triggered when users add or change and email address and it needs to be confirmed.| **{dotted-circle}** No | **{check-circle}** Yes | GitLab [16.3](https://gitlab.com/gitlab-org/gitlab/-/issues/377625) | | [`remove_ssh_key`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65615) | Audit event triggered when a SSH key is removed| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [14.1](https://gitlab.com/gitlab-org/gitlab/-/issues/220127) | | [`user_admin_status_updated`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65168) | Adds an audit event when a user is either made an administrator, or removed as an administrator| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [14.1](https://gitlab.com/gitlab-org/gitlab/-/issues/323905) | +| [`user_auditor_status_updated`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/136456) | Adds an audit event when a user is either made an auditor, or removed as an auditor| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.6](https://gitlab.com/gitlab-org/gitlab/-/issues/430235) | | [`user_email_address_updated`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/2103) | Adds an audit event when a user updates their email address| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [10.1](https://gitlab.com/gitlab-org/gitlab-ee/issues/1370) | | [`user_profile_visiblity_updated`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/129149) | Triggered when user toggles private profile user setting| **{dotted-circle}** No | **{check-circle}** Yes | GitLab [16.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/129149) | | [`user_username_updated`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/106086) | Event triggered on updating a user's username| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [15.7](https://gitlab.com/gitlab-org/gitlab/-/issues/369329) | diff --git a/doc/administration/audit_event_streaming/graphql_api.md b/doc/administration/audit_event_streaming/graphql_api.md index 6e1a3424929..58668902b8e 100644 --- a/doc/administration/audit_event_streaming/graphql_api.md +++ b/doc/administration/audit_event_streaming/graphql_api.md @@ -177,9 +177,8 @@ Prerequisites: - Owner role for a top-level group. -Users with the Owner role for a group can update streaming destinations' custom HTTP headers using the -`auditEventsStreamingHeadersUpdate` mutation type. You can retrieve the custom HTTP headers ID -by [listing all the custom HTTP headers](#list-streaming-destinations) for the group. +To update streaming destinations for a group, use the `externalAuditEventDestinationUpdate` mutation type. You can retrieve the destinations ID +by [listing all the streaming destinations](#list-streaming-destinations) for the group. ```graphql mutation { @@ -206,6 +205,24 @@ Streaming destination is updated if: - The returned `errors` object is empty. - The API responds with `200 OK`. +Users with the Owner role for a group can update streaming destinations' custom HTTP headers using the +`auditEventsStreamingHeadersUpdate` mutation type. You can retrieve the custom HTTP headers ID +by [listing all the custom HTTP headers](#list-streaming-destinations) for the group. + +```graphql +mutation { + auditEventsStreamingHeadersUpdate(input: { headerId: "gid://gitlab/AuditEvents::Streaming::Header/2", key: "new-key", value: "new-value", active: false }) { + errors + header { + id + key + value + active + } + } +} +``` + Group owners can remove an HTTP header using the GraphQL `auditEventsStreamingHeadersDestroy` mutation. You can retrieve the header ID by [listing all the custom HTTP headers](#list-streaming-destinations) for the group. diff --git a/doc/administration/audit_event_streaming/index.md b/doc/administration/audit_event_streaming/index.md index 8f40dc6c34c..09474db1e08 100644 --- a/doc/administration/audit_event_streaming/index.md +++ b/doc/administration/audit_event_streaming/index.md @@ -206,7 +206,9 @@ To add Google Cloud Logging streaming destinations to a top-level group: 1. Select **Secure > Audit events**. 1. On the main area, select **Streams** tab. 1. Select **Add streaming destination** and select **Google Cloud Logging** to show the section for adding destinations. -1. Enter the Google project ID, Google client email, log ID, and Google private key to add. +1. Enter a random string to use as a name for the new destination. +1. Enter the Google project ID, Google client email, and Google private key from previously-created Google Cloud service account key to add to the new destination. +1. Enter a random string to use as a log ID for the new destination. You can use this later to filter log results in Google Cloud. 1. Select **Add** to add the new streaming destination. #### List Google Cloud Logging destinations @@ -236,7 +238,9 @@ To update Google Cloud Logging streaming destinations to a top-level group: 1. Select **Secure > Audit events**. 1. On the main area, select **Streams** tab. 1. Select the Google Cloud Logging stream to expand. -1. Enter the Google project ID, Google client email, and log ID to update. +1. Enter a random string to use as a name for the destination. +1. Enter the Google project ID and Google client email from previously-created Google Cloud service account key to update the destination. +1. Enter a random string to update the log ID for the destination. You can use this later to filter log results in Google Cloud. 1. Select **Add a new private key** and enter a Google private key to update the private key. 1. Select **Save** to update the streaming destination. @@ -255,6 +259,85 @@ To delete Google Cloud Logging streaming destinations to a top-level group: 1. Select **Delete destination**. 1. Confirm by selecting **Delete destination** in the dialog. +### AWS S3 destinations + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/132603) in GitLab 16.6 [with a flag](../feature_flags.md) named `allow_streaming_audit_events_to_amazon_s3`. Enabled by default. + +FLAG: +On self-managed GitLab, by default this feature is available. To hide the feature per group, an administrator can [disable the feature flag](../feature_flags.md) named `allow_streaming_audit_events_to_amazon_s3`. +On GitLab.com, this feature is available. + +Manage AWS S3 destinations for top-level groups. + +#### Prerequisites + +Before setting up AWS S3 streaming audit events, you must: + +1. Create a access key for AWS with the appropriate credentials and permissions. This account is used to configure audit log streaming authentication. + For more information, see [Managing access keys](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html?icmpid=docs_iam_console#Using_CreateAccessKey). +1. Create a AWS S3 bucket. This bucket is used to store audit log streaming data. For more information, see [Creating a bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html) + +#### Add a new AWS S3 destination + +Prerequisites: + +- Owner role for a top-level group. + +To add AWS S3 streaming destinations to a top-level group: + +1. On the left sidebar, select **Search or go to** and find your group. +1. Select **Secure > Audit events**. +1. On the main area, select **Streams** tab. +1. Select **Add streaming destination** and select **AWS S3** to show the section for adding destinations. +1. Enter a random string to use as a name for the new destination. +1. Enter the Access Key ID, Secret Access Key, Bucket Name, and AWS Region from previously-created AWS access key and bucket to add to the new destination. +1. Select **Add** to add the new streaming destination. + +#### List AWS S3 destinations + +Prerequisites: + +- Owner role for a top-level group. + +To list AWS S3 streaming destinations for a top-level group: + +1. On the left sidebar, select **Search or go to** and find your group. +1. Select **Secure > Audit events**. +1. On the main area, select **Streams** tab. +1. Select the AWS S3 stream to expand and see all the fields. + +#### Update a AWS S3 destination + +Prerequisites: + +- Owner role for a top-level group. + +To update AWS S3 streaming destinations to a top-level group: + +1. On the left sidebar, select **Search or go to** and find your group. +1. Select **Secure > Audit events**. +1. On the main area, select **Streams** tab. +1. Select the AWS S3 stream to expand. +1. Enter a random string to use as a name for the destination. +1. Enter the Access Key ID, Secret Access Key, Bucket Name, and AWS Region from previously-created AWS access key and bucket to update the destination. +1. Select **Add a new Secret Access Key** and enter a AWS Secret Access Key to update the Secret Access Key. +1. Select **Save** to update the streaming destination. + +#### Delete a AWS S3 streaming destination + +Prerequisites: + +- Owner role for a top-level group. + +To delete AWS S3 streaming destinations to a top-level group: + +1. On the left sidebar, select **Search or go to** and find your group. +1. Select **Secure > Audit events**. +1. On the main area, select the **Streams** tab. +1. Select the AWS S3 stream to expand. +1. Select **Delete destination**. +1. Confirm by selecting **Delete destination** in the dialog. + ## Instance streaming destinations **(ULTIMATE SELF)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/398107) in GitLab 16.1 [with a flag](../feature_flags.md) named `ff_external_audit_events`. Disabled by default. @@ -446,7 +529,9 @@ To add Google Cloud Logging streaming destinations to an instance: 1. On the left sidebar, select **Monitoring > Audit Events**. 1. On the main area, select **Streams** tab. 1. Select **Add streaming destination** and select **Google Cloud Logging** to show the section for adding destinations. -1. Enter the Google project ID, Google client email, log ID, and Google private key to add. +1. Enter a random string to use as a name for the new destination. +1. Enter the Google project ID, Google client email, and Google private key from previously-created Google Cloud service account key to add to the new destination. +1. Enter a random string to use as a log ID for the new destination. You can use this later to filter log results in Google Cloud. 1. Select **Add** to add the new streaming destination. #### List Google Cloud Logging destinations @@ -476,7 +561,9 @@ To update Google Cloud Logging streaming destinations to an instance: 1. On the left sidebar, select **Monitoring > Audit Events**. 1. On the main area, select **Streams** tab. 1. Select the Google Cloud Logging stream to expand. -1. Enter the Google project ID, Google client email, and log ID to update. +1. Enter a random string to use as a name for the destination. +1. Enter the Google project ID and Google client email from previously-created Google Cloud service account key to update the destination. +1. Enter a random string to update the log ID for the destination. You can use this later to filter log results in Google Cloud. 1. Select **Add a new private key** and enter a Google private key to update the private key. 1. Select **Save** to update the streaming destination. diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index 736f381e9d7..ba1a4ca05c4 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -6,171 +6,146 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Audit events **(PREMIUM ALL)** -Use audit events to track important events, including who performed the related action and when. -You can use audit events to track, for example: +A security audit is a in-depth analysis and review of your infrastructure, which is used to display +areas of concern and potentially hazardous practices. To assist with the audit process, GitLab provides +audit events which allow you to track a variety of different actions within GitLab. + +For example, you can use audit events to track: - Who changed the permission level of a particular user for a GitLab project, and when. - Who added a new user or removed a user, and when. -Audit events are similar to the [log system](logs/index.md). - -The GitLab API, database, and `audit_json.log` record many audit events. Some audit events are only available through -[streaming audit events](audit_event_streaming.md). +These events can be used to in an audit to assess risk, strengthen security measures, respond to incidents, and adhere to compliance. For a complete list the audit events GitLab provides, see [Audit event types](../administration/audit_event_streaming/audit_event_types.md). -You can also generate an [audit report](audit_reports.md) of audit events. +## Prerequisites -NOTE: -You can't configure a retention policy for audit events, but epic -[7917](https://gitlab.com/groups/gitlab-org/-/epics/7917) proposes to change this. +To view specific types of audit events, you need a minimum role. -## Time zones - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/242014) in GitLab 15.7, GitLab UI shows dates and times in the user's local time zone instead of UTC. +- To view the group audit events of all users in a group, you must have the [Owner role](../user/permissions.md#roles) for the group. +- To view the project audit events of all users in a project, you must have at least the [Maintainer role](../user/permissions.md#roles) for the project. +- To view the group and project audit events based on your own actions in a group or project, you must have at least the [Developer role](../user/permissions.md#roles) + for the group or project. -The time zone used for audit events depends on where you view them: +Users with the [Auditor access level](auditor_users.md) can see group and project events for all users. -- In GitLab UI, your local time zone (GitLab 15.7 and later) or UTC (GitLab 15.6 and earlier) is used. -- The [Audit Events API](../api/audit_events.md) returns dates and times in UTC by default, or the - [configured time zone](timezone.md) on a self-managed GitLab instance. -- In `audit_json.log`, UTC is used. -- In CSV exports, UTC is used. +## Viewing audit events -## View audit events +Audit events can be viewed at the group, project, instance, and sign-in level. Each level has different audit events which it logs. -Depending on the events you want to view, at a minimum you must have: - -- For group audit events of all users in the group, the Owner role for the group. -- For project audit events of all users in the project, the Maintainer role for the project. -- For group and project audit events based on your own actions, the Developer role for the group or project. -- [Auditor users](auditor_users.md) can see group and project events for all users. - -You can view audit events scoped to a group or project. +### Group audit events To view a group's audit events: -1. Go to the group. +1. On the left sidebar, select **Search or go to** and find your group. 1. On the left sidebar, select **Secure > Audit events**. +1. Filter the audit events by the member of the project (user) who performed the action and date range. -Group events do not include project audit events. Group events can also be accessed using the -[Group Audit Events API](../api/audit_events.md#group-audit-events). Group event queries are limited to a maximum of 30 -days. +Group audit events can also be accessed using the [Group Audit Events API](../api/audit_events.md#group-audit-events). Group audit event queries are limited to a maximum of 30 days. -To view a project's audit events: +### Project audit events -1. Go to the project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Secure > Audit events**. +1. Filter the audit events by the member of the project (user) who performed the action and date range. -Project events can also be accessed using the [Project Audit Events API](../api/audit_events.md#project-audit-events). -Project event queries are limited to a maximum of 30 days. +Project audit events can also be accessed using the [Project Audit Events API](../api/audit_events.md#project-audit-events). Project audit event queries are limited to a maximum of 30 days. -## View instance audit events **(PREMIUM SELF)** +### Instance audit events **(PREMIUM SAAS)** You can view audit events from user actions across an entire GitLab instance. - To view instance audit events: 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Audit Events**. +1. Filter by the following: + - Member of the project (user) who performed the action + - Group + - Project + - Date Range + +### Sign-in audit events **(FREE ALL)** + +Successful sign-in events are the only audit events available at all tiers. To see successful sign-in events: + +1. On the left sidebar, select your avatar. +1. Select **Edit profile > Authentication log**. -### Export to CSV +After upgrading to a paid tier, you can also see successful sign-in events on audit event pages. + +## Exporting audit events > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1449) in GitLab 13.4. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/285441) in GitLab 13.7. > - Entity type `Gitlab::Audit::InstanceScope` for instance audit events [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/418185) in GitLab 16.2. -You can export the current view (including filters) of your instance audit events as a CSV file. To export the instance -audit events to CSV: +You can export the current view (including filters) of your instance audit events as a +CSV(comma-separated values) file. To export the instance audit events to CSV: 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Audit Events**. -1. Select the available search [filters](#filter-audit-events). +1. Select the available search filters. 1. Select **Export as CSV**. -The exported file: - -- Is sorted by `created_at` in ascending order. -- Is limited to a maximum of 100 000 events. The remaining records are truncated when this limit is reached. - -Data is encoded with: - -- Comma as the column delimiter. -- `"` to quote fields if necessary. -- New lines separate rows. +A download confirmation dialog then appears for you to download the CSV file. The exported CSV is limited +to a maximum of 100000 events. The remaining records are truncated when this limit is reached. -The first row contains the headers, which are listed in the following table along with a description of the values: +### Audit event CSV encoding -| Column | Description | -|:---------------------|:-------------------------------------------------------------------| -| **ID** | Audit event `id`. | -| **Author ID** | ID of the author. | -| **Author Name** | Full name of the author. | -| **Entity ID** | ID of the scope. | -| **Entity Type** | Type of the scope (`Project`, `Group`, `User`, or `Gitlab::Audit::InstanceScope`). | -| **Entity Path** | Path of the scope. | -| **Target ID** | ID of the target. | -| **Target Type** | Type of the target. | -| **Target Details** | Details of the target. | -| **Action** | Description of the action. | -| **IP Address** | IP address of the author who performed the action. | -| **Created At (UTC)** | Formatted as `YYYY-MM-DD HH:MM:SS`. | +The exported CSV file is encoded as follows: -## View sign-in events **(FREE ALL)** +- `,` is used as the column delimiter +- `"` is used to quote fields if necessary. +- `\n` is used to separate rows. -Successful sign-in events are the only audit events available at all tiers. To see successful sign-in events: +The first row contains the headers, which are listed in the following table along +with a description of the values: -1. On the left sidebar, select your avatar. -1. Select **Edit profile > Authentication log**. - -After upgrading to a paid tier, you can also see successful sign-in events on audit event pages. +| Column | Description | +| --------------------- | ---------------------------------------------------------------------------------- | +| **ID** | Audit event `id`. | +| **Author ID** | ID of the author. | +| **Author Name** | Full name of the author. | +| **Entity ID** | ID of the scope. | +| **Entity Type** | Type of the scope (`Project`, `Group`, `User`, or `Gitlab::Audit::InstanceScope`). | +| **Entity Path** | Path of the scope. | +| **Target ID** | ID of the target. | +| **Target Type** | Type of the target. | +| **Target Details** | Details of the target. | +| **Action** | Description of the action. | +| **IP Address** | IP address of the author who performed the action. | +| **Created At (UTC)** | Formatted as `YYYY-MM-DD HH:MM:SS`. | -## Filter audit events - -From audit events pages, different filters are available depending on the page you're on. - -| Audit event page | Available filter | -|:-----------------|:-----------------------------------------------------------------------------------------------------------------------| -| Project | User (member of the project) who performed the action. | -| Group | User (member of the group) who performed the action. | -| Instance | Group, project, or user. | -| All | Date range buttons and pickers (maximum range of 31 days). Default is from the first day of the month to today's date. | +All items are sorted by `created_at` in ascending order. ## User impersonation > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/536) in GitLab 13.0. > - Impersonation session events included in group audit events in GitLab 14.8. -When a user is [impersonated](../administration/admin_area.md#user-impersonation), their actions are logged as audit events -with additional details: +When a user is [impersonated](../administration/admin_area.md#user-impersonation), their actions are logged as audit events with the following additional details: -- Audit events include information about the impersonating administrator. These audit events are visible in audit event - pages depending on the audit event type (group, project, or user). -- Extra audit events are recorded for the start and end of the administrator's impersonation session. These audit events - are visible as: - - Instance audit events. - - Group audit events for all groups the user belongs to. For performance reasons, group audit events are limited to - the oldest 20 groups you belong to. +- Audit events include information about the impersonating administrator. +- Extra audit events are recorded for the start and end of the administrator's impersonation session. ![Audit event with impersonated user](img/impersonated_audit_events_v15_7.png) -## Available audit events +## Time zones -For a list of available audit events, see [Audit event types](../administration/audit_event_streaming/audit_event_types.md). +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/242014) in GitLab 15.7, GitLab UI shows dates and times in the user's local time zone instead of UTC. -## Unsupported events +The time zone used for audit events depends on where you view them: -Some events are not tracked in audit events. The following epics and issues propose support for more events: +- In GitLab UI, your local time zone (GitLab 15.7 and later) or UTC (GitLab 15.6 and earlier) is used. +- The [Audit Events API](../api/audit_events.md) returns dates and times in UTC by default, or the + [configured time zone](timezone.md) on a self-managed GitLab instance. +- In CSV exports, UTC is used. -- [Project settings and activity](https://gitlab.com/groups/gitlab-org/-/epics/474). -- [Group settings and activity](https://gitlab.com/groups/gitlab-org/-/epics/475). -- [Instance-level settings and activity](https://gitlab.com/groups/gitlab-org/-/epics/476). -- [Deployment Approval activity](https://gitlab.com/gitlab-org/gitlab/-/issues/354782). -- [Approval rules processing by a non GitLab user](https://gitlab.com/gitlab-org/gitlab/-/issues/407384). +## Contribute to audit events If you don't see the event you want in any of the epics, you can either: - Use the **Audit Event Proposal** issue template to - [create an issue](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Audit%20Event%20Proposal) to - request it. + [create an issue](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Audit%20Event%20Proposal) to request it. - [Add it yourself](../development/audit_event_guide/index.md). diff --git a/doc/administration/auditor_users.md b/doc/administration/auditor_users.md index e9df9cc6e37..09d68e82782 100644 --- a/doc/administration/auditor_users.md +++ b/doc/administration/auditor_users.md @@ -25,6 +25,9 @@ Situations where auditor access for users could be helpful include: you can create an account with auditor access and then share the credentials with those users to which you want to grant access. +NOTE: +An auditor user counts as a billable user and consumes a license seat. + ## Add a user with auditor access To create a new user account with auditor access (or change an existing user): diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index bf2b3d7e53e..0c42ce90346 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -448,7 +448,7 @@ These LDAP sync configuration settings are available: | Setting | Description | Required | Examples | |-------------------|-------------|----------|----------| -| `group_base` | Base used to search for groups. | **{dotted-circle}** No (required when `external_groups` is configured) | `'ou=groups,dc=gitlab,dc=example'` | +| `group_base` | Base used to search for groups. All valid groups have this base as part of their DN. | **{dotted-circle}** No (required when `external_groups` is configured) | `'ou=groups,dc=gitlab,dc=example'` | | `admin_group` | The CN of a group containing GitLab administrators. Not `cn=administrators` or the full DN. | **{dotted-circle}** No | `'administrators'` | | `external_groups` | An array of CNs of groups containing users that should be considered external. Not `cn=interns` or the full DN. | **{dotted-circle}** No | `['interns', 'contractors']` | | `sync_ssh_keys` | The LDAP attribute containing a user's public SSH key. | **{dotted-circle}** No | `'sshPublicKey'` or false if not set | diff --git a/doc/administration/backup_restore/backup_gitlab.md b/doc/administration/backup_restore/backup_gitlab.md index 05a330bf3f5..5c0fcbbc4ef 100644 --- a/doc/administration/backup_restore/backup_gitlab.md +++ b/doc/administration/backup_restore/backup_gitlab.md @@ -437,7 +437,9 @@ sudo -u git -H bundle exec rake gitlab:backup:create SKIP=tar RAILS_ENV=producti #### Create server-side repository backups -> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4941) in GitLab 16.3. +> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4941) in GitLab 16.3. +> - Server-side support for restoring a specified backup instead of the latest backup [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/132188) in GitLab 16.6. +> - Server-side support for creating incremental backups [introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6475) in GitLab 16.6. Instead of storing large repository backups in the backup archive, repository backups can be configured so that the Gitaly node that hosts each repository is @@ -504,6 +506,7 @@ sudo -u git -H bundle exec rake gitlab:backup:create GITLAB_BACKUP_MAX_CONCURREN > - Introduced in GitLab 14.9 [with a flag](../feature_flags.md) named `incremental_repository_backup`. Disabled by default. > - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/355945) in GitLab 14.10. > - `PREVIOUS_BACKUP` option [introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4184) in GitLab 15.0. +> - Server-side support for creating incremental backups [introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6475) in GitLab 16.6. FLAG: On self-managed GitLab, by default this feature is available. To hide the feature, an administrator can [disable the feature flag](../feature_flags.md) named `incremental_repository_backup`. @@ -853,7 +856,7 @@ For the Linux package (Omnibus): ## If you have CNAME buckets (foo.example.com), you might run into SSL issues ## when uploading backups ("hostname foo.example.com.storage.googleapis.com - ## does not match the server certificate"). In that case, uncomnent the following + ## does not match the server certificate"). In that case, uncomment the following ## setting. See: https://github.com/fog/fog/issues/2834 #'path_style' => true } @@ -1272,7 +1275,7 @@ Gitaly Cluster [does not support snapshot backups](../gitaly/index.md#snapshot-b When considering using file system data transfer or snapshots: - Don't use these methods to migrate from one operating system to another. The operating systems of the source and destination should be as similar as possible. For example, - don't use these methods to migrate from Ubuntu to Fedora. + don't use these methods to migrate from Ubuntu to RHEL. - Data consistency is very important. You should stop GitLab with `sudo gitlab-ctl stop` before taking doing a file system transfer (with `rsync`, for example) or taking a snapshot. diff --git a/doc/administration/cicd.md b/doc/administration/cicd.md index 7a6316a1e50..10bc60fe399 100644 --- a/doc/administration/cicd.md +++ b/doc/administration/cicd.md @@ -18,7 +18,7 @@ CI/CD to be disabled by default in new projects by modifying the settings in: - `gitlab.rb` for Linux package installations. Existing projects that already had CI/CD enabled are unchanged. Also, this setting only changes -the project default, so project owners [can still enable CI/CD in the project settings](../ci/enable_or_disable_ci.md). +the project default, so project owners [can still enable CI/CD in the project settings](../ci/pipelines/settings.md#disable-gitlab-cicd-pipelines). For self-compiled installations: @@ -93,14 +93,96 @@ To change the frequency of the pipeline schedule worker: For example, to set the maximum frequency of pipelines to twice a day, set `pipeline_schedule_worker_cron` to a cron value of `0 */12 * * *` (`00:00` and `12:00` every day). -<!-- ## Troubleshooting +## Disaster recovery -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. +You can disable some important but computationally expensive parts of the application +to relieve stress on the database during ongoing downtime. -Each scenario can be a third-level heading, for example `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +### Disable fair scheduling on shared runners + +When clearing a large backlog of jobs, you can temporarily enable the `ci_queueing_disaster_recovery_disable_fair_scheduling` +[feature flag](../administration/feature_flags.md). This flag disables fair scheduling +on shared runners, which reduces system resource usage on the `jobs/request` endpoint. + +When enabled, jobs are processed in the order they were put in the system, instead of +balanced across many projects. + +### Disable compute quota enforcement + +To disable the enforcement of [compute quotas](../ci/pipelines/cicd_minutes.md) on shared runners, you can temporarily +enable the `ci_queueing_disaster_recovery_disable_quota` [feature flag](../administration/feature_flags.md). +This flag reduces system resource usage on the `jobs/request` endpoint. + +When enabled, jobs created in the last hour can run in projects which are out of quota. +Earlier jobs are already canceled by a periodic background worker (`StuckCiJobsWorker`). + +## CI/CD troubleshooting Rails console commands + +The following commands are run in the [Rails console](../administration/operations/rails_console.md#starting-a-rails-console-session). + +WARNING: +Any command that changes data directly could be damaging if not run correctly, or under the right conditions. +We highly recommend running them in a test environment with a backup of the instance ready to be restored, just in case. + +### Cancel stuck pending pipelines + +```ruby +project = Project.find_by_full_path('<project_path>') +Ci::Pipeline.where(project_id: project.id).where(status: 'pending').count +Ci::Pipeline.where(project_id: project.id).where(status: 'pending').each {|p| p.cancel if p.stuck?} +Ci::Pipeline.where(project_id: project.id).where(status: 'pending').count +``` + +### Try merge request integration + +```ruby +project = Project.find_by_full_path('<project_path>') +mr = project.merge_requests.find_by(iid: <merge_request_iid>) +mr.project.try(:ci_integration) +``` + +### Validate the `.gitlab-ci.yml` file + +```ruby +project = Project.find_by_full_path('<project_path>') +content = p.ci_config_for(project.repository.root_ref_sha) +Gitlab::Ci::Lint.new(project: project, current_user: User.first).validate(content) +``` + +### Disable AutoDevOps on Existing Projects + +```ruby +Project.all.each do |p| + p.auto_devops_attributes={"enabled"=>"0"} + p.save +end +``` + +### Obtain runners registration token + +```ruby +Gitlab::CurrentSettings.current_application_settings.runners_registration_token +``` + +### Seed runners registration token + +```ruby +appSetting = Gitlab::CurrentSettings.current_application_settings +appSetting.set_runners_registration_token('<new-runners-registration-token>') +appSetting.save! +``` + +### Run pipeline schedules manually + +You can run pipeline schedules manually through the Rails console to reveal any errors that are usually not visible. + +```ruby +# schedule_id can be obtained from Edit Pipeline Schedule page +schedule = Ci::PipelineSchedule.find_by(id: <schedule_id>) + +# Select the user that you want to run the schedule for +user = User.find_by_username('<username>') + +# Run the schedule +ps = Ci::CreatePipelineService.new(schedule.project, user, ref: schedule.ref).execute!(:schedule, ignore_skip_ci: true, save_on_errors: false, schedule: schedule) +``` diff --git a/doc/administration/dedicated/index.md b/doc/administration/dedicated/index.md index 2889fb9b389..16efc353c84 100644 --- a/doc/administration/dedicated/index.md +++ b/doc/administration/dedicated/index.md @@ -38,7 +38,7 @@ After you first sign in to Switchboard, you must update your password and set up The following stages guide you through a series of four steps to provide the information required to create your GitLab Dedicated tenant. 1. Confirm account details: Confirm key attributes of your GitLab Dedicated account: - - Reference architecture: Corresponds with the number of users you provided to your account team when beginning the onboarding process. For more information, see [reference architectures](../../administration/reference_architectures/index.md). + - Reference architecture: Corresponds with the number of users you provided to your account team when beginning the onboarding process. For more information, see [reference architectures](../../subscriptions/gitlab_dedicated/index.md#availability-and-scalability). - Total repository storage size: Corresponds with the storage size you provided to your account team when beginning the onboarding process. - If you need to make changes to these attributes, [submit a support ticket](https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=4414917877650). 1. Tenant configuration: Provides the minimum required information needed to create your GitLab Dedicated tenant: @@ -214,7 +214,9 @@ Make sure the AWS KMS keys are replicated to your desired primary, secondary and ## Configuration changes -To change or update the configuration for your GitLab Dedicated instance, open a [support ticket](https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=4414917877650) with your request. You can request configuration changes for the options originally specified during onboarding, or for any of the following optional features. +Switchboard empowers the user to make limited configuration changes to their Dedicated Tenant Instance. As Switchboard matures further configuration changes will be made available. + +To change or update the configuration of your GitLab Dedicated instance, use Switchboard following the instructions in the relevant section or open a [support ticket](https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=4414917877650) with your request. You can request configuration changes for the options originally specified during onboarding, or for any of the following optional features. The turnaround time to process configuration change requests is [documented in the GitLab handbook](https://about.gitlab.com/handbook/engineering/infrastructure/team/gitlab-dedicated/#handling-configuration-changes-for-tenant-environments). @@ -278,10 +280,22 @@ To enable an Outbound Private Link: GitLab then configures the tenant instance to create the necessary Endpoint Interfaces based on the service names you provided. Any matching outbound connections made from the tenant GitLab instance are directed through the PrivateLink into your VPC. -#### Custom certificates +### Custom certificates In some cases, the GitLab Dedicated instance can't reach an internal service you own because it exposes a certificate that can't be validated using a public Certification Authority (CA). In these cases, custom certificates are required. +#### Add a custom certificate with Switchboard + +1. Log in to [Switchboard](https://console.gitlab-dedicated.com/). +1. At the top of the page, select **Configuration**. +1. Expand **Custom Certificate Authorities**. +1. Select **+ Add Certificate**. +1. Paste the certificate into the text box. +1. Select **Save**. +1. Scroll up to the top of the page and select whether to apply the changes immediately or during the next maintenance window. + +#### Add a custom certificate with a Support Request + To request that GitLab add custom certificates when communicating with your services over PrivateLink, attach the custom public certificate files to your [support ticket](https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=4414917877650). #### Maximum number of reverse PrivateLink connections @@ -292,6 +306,19 @@ GitLab Dedicated limits the number of reverse PrivateLink connections to 10. GitLab Dedicated allows you to control which IP addresses can access your instance through an IP allowlist. +#### Add an IP to the allowlist with Switchboard + +1. Log in to [Switchboard](https://console.gitlab-dedicated.com/). +1. At the top of the page, select **Configuration**. +1. Expand **Allowed Source List Config / IP allowlist**. +1. Turn on the **Enable** toggle. +1. Select **Add Item**. +1. Enter the IP address and description. To add another IP address, repeat steps 5 and 6. +1. Select **Save**. +1. Scroll up to the top of the page and select whether to apply the changes immediately or during the next maintenance window. + +#### Add an IP to the allowlist with a Support Request + Specify a comma separated list of IP addresses that can access your GitLab Dedicated instance in your [support ticket](https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=4414917877650). After the configuration has been applied, when an IP not on the allowlist tries to access your instance, the connection is refused. ### SAML @@ -303,6 +330,23 @@ Prerequisites: - You must configure the identity provider before sending the required data to GitLab. +#### Activate SAML with Switchboard + +To activate SAML for your GitLab Dedicated instance: + +1. Log in to [Switchboard](https://console.gitlab-dedicated.com/). +1. At the top of the page, select **Configuration**. +1. Expand **SAML Config**. +1. Turn on the **Enable** toggle. +1. Complete the fields. +1. Select **Save**. +1. Scroll up to the top of the page and select whether to apply the changes immediately or during the next maintenance window. +1. To verify the SAML configuration is successful: + - Check that the SSO button description is displayed on your instance's sign-in page. + - Go to the metadata URL of your instance (`https://INSTANCE-URL/users/auth/saml/metadata`). This page can be used to simplify much of the configuration of the identity provider, and manually validate the settings. + +#### Activate SAML with a Support Request + To activate SAML for your GitLab Dedicated instance: 1. To make the necessary changes, include the desired [SAML configuration block](../../integration/saml.md#configure-saml-support-in-gitlab) for your GitLab application in your [support ticket](https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=4414917877650). At a minimum, GitLab needs the following information to enable SAML for your instance: diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index d6f6211ed4c..2f636dc6ba4 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -88,6 +88,7 @@ Note the following when promoting a secondary: - If you encounter an `ActiveRecord::RecordInvalid: Validation failed: Name has already been taken` error message during this process, for more information, see this [troubleshooting advice](../replication/troubleshooting.md#fixing-errors-during-a-failover-or-when-promoting-a-secondary-to-a-primary-site). +- You should [point the primary domain DNS at the newly promoted site](#step-4-optional-updating-the-primary-domain-dns-record). Otherwise, runners must be registered again with the newly promoted site, and all Git remotes, bookmarks, and external integrations must be updated. #### Promoting a **secondary** site running on a single node running GitLab 14.5 and later diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index 78bd685e06f..e8b2cb38563 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -19,8 +19,6 @@ Fetching large repositories can take a long time for teams located far from a si Geo provides local, read-only sites of your GitLab instances. This can reduce the time it takes to clone and fetch large repositories, speeding up development. -For a video introduction to Geo, see [Introduction to GitLab Geo - GitLab Features](https://www.youtube.com/watch?v=-HDLxSjEh6w). - To make sure you're using the right version of the documentation, go to [the Geo page on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/administration/geo/index.md) and choose the appropriate release from the **Switch branch/tag** dropdown list. For example, [`v13.7.6-ee`](https://gitlab.com/gitlab-org/gitlab/-/blob/v13.7.6-ee/doc/administration/geo/index.md). Geo uses a set of defined terms that are described in the [Geo Glossary](glossary.md). @@ -208,6 +206,7 @@ This list of limitations only reflects the latest version of GitLab. If you are - For Git over SSH, to make the project clone URL display correctly regardless of which site you are browsing, secondary sites must use the same port as the primary. [GitLab issue #339262](https://gitlab.com/gitlab-org/gitlab/-/issues/339262) proposes to remove this limitation. - Git push over SSH against a secondary site does not work for pushes over 1.86 GB. [GitLab issue #413109](https://gitlab.com/gitlab-org/gitlab/-/issues/413109) tracks this bug. - Backups [cannot be run on secondaries](replication/troubleshooting.md#message-error-canceling-statement-due-to-conflict-with-recovery). +- Git clone and fetch requests with option `--depth` over SSH against a secondary site does not work and hangs indefinitely if the secondary site is not up to date at the time the request is initiated. For more information, see [issue 391980](https://gitlab.com/gitlab-org/gitlab/-/issues/391980). ### Limitations on replication/verification diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 3c2d43d196a..dd021695800 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -1231,7 +1231,7 @@ status ### Failed verification of Uploads on the primary Geo site -If some Uploads verification is failing on the primary Geo site with the `verification_checksum: nil` and `verification_failure: Error during verification: undefined method 'underscore' for NilClass:Class` errros, this can be due to orphaned Uploads. The parent record owning the Upload (the Upload's `model`) has somehow been deleted, but the Upload record still exists. These verification failures are false. +If verification of some uploads is failing on the primary Geo site with `verification_checksum = nil` and with the ``verification_failure = Error during verification: undefined method `underscore' for NilClass:Class``, this can be due to orphaned Uploads. The parent record owning the Upload (the upload's model) has somehow been deleted, but the Upload record still exists. These verification failures are false. You can find these errors in the `geo.log` file on the primary Geo site. @@ -1249,7 +1249,7 @@ You can delete these Upload records on the primary Geo site to get rid of these uploads = Geo::UploadState.where( verification_checksum: nil, verification_state: 3, - verification_failure: "Error during verification: undefined method 'underscore' for NilClass:Class" + verification_failure: "Error during verification: undefined method `underscore' for NilClass:Class" ).pluck(:upload_id) uploads_deleted = 0 @@ -1434,8 +1434,8 @@ If you are using the Linux package installation, something might have failed dur ### GitLab indicates that more than 100% of repositories were synced -This can be caused by orphaned records in the project registry. You can clear them -[using the Rake task to remove orphaned project registries](../../../administration/raketasks/geo.md#remove-orphaned-project-registries). +This can be caused by orphaned records in the project registry. They are being cleaned +periodically using a registry worker, so give it some time to fix it itself. ### Secondary site shows "Unhealthy" in UI after changing the value of `external_url` for the primary site diff --git a/doc/administration/geo/setup/index.md b/doc/administration/geo/setup/index.md index ea3bb5afc24..f59dec17f8b 100644 --- a/doc/administration/geo/setup/index.md +++ b/doc/administration/geo/setup/index.md @@ -31,6 +31,8 @@ a single-node Geo site or a multi-node Geo site. If both Geo sites are based on the [1K reference architecture](../../reference_architectures/1k_users.md), follow [Set up Geo for two single-node sites](two_single_node_sites.md). +If using external PostgreSQL services, for example Amazon RDS, follow [Set up Geo for two single-node sites (with external PostgreSQL services)](two_single_node_external_services.md). + Depending on your GitLab deployment, [additional configuration](#additional-configuration) for LDAP, object storage, and the Container Registry might be required. ### Multi-node Geo sites diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md index f62f0a5a4e2..15ace9c4ed9 100644 --- a/doc/administration/gitaly/configure_gitaly.md +++ b/doc/administration/gitaly/configure_gitaly.md @@ -27,6 +27,7 @@ The following configuration options are also available: - Enabling [TLS support](#enable-tls-support). - Limiting [RPC concurrency](#limit-rpc-concurrency). +- Limiting [pack-objects concurrency](#limit-pack-objects-concurrency). ## About the Gitaly token @@ -361,7 +362,7 @@ Configure Gitaly server in one of two ways: WARNING: If directly copying repository data from a GitLab server to Gitaly, ensure that the metadata file, default path `/var/opt/gitlab/git-data/repositories/.gitaly-metadata`, is not included in the transfer. -Copying this file causes GitLab to use the [Rugged patches](index.md#direct-access-to-git-in-gitlab) for repositories hosted on the Gitaly server, +Copying this file causes GitLab to use the direct disk access to repositories hosted on the Gitaly server, leading to `Error creating pipeline` and `Commit not found` errors, or stale data. ### Configure Gitaly clients @@ -665,6 +666,8 @@ Configure Gitaly with TLS in one of two ways: ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). +1. Run `sudo gitlab-rake gitlab:gitaly:check` on the Gitaly client (for example, the + Rails application) to confirm it can connect to Gitaly servers. 1. Verify Gitaly traffic is being served over TLS by [observing the types of Gitaly connections](#observe-type-of-gitaly-connections). 1. Optional. Improve security by: @@ -751,6 +754,43 @@ Configure Gitaly with TLS in one of two ways: ::EndTabs +#### Update the certificates + +To update the Gitaly certificates after initial configuration: + +::Tabs + +:::TabTitle Linux package (Omnibus) + +If the content of your SSL certificates under the `/etc/gitlab/ssl` directory have been updated, but no configuration changes have been made to +`/etc/gitlab/gitlab.rb`, then reconfiguring GitLab doesn’t affect Gitaly. Instead, you must restart Gitaly manually for the certificates to be loaded +by the Gitaly process: + +```shell +sudo gitlab-ctl restart gitaly +``` + +If you change or update the certificates in `/etc/gitlab/trusted-certs` without making changes to the `/etc/gitlab/gitlab.rb` file, you must: + +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) so the symlinks for the trusted certificates are updated. +1. Restart Gitaly manually for the certificates to be loaded by the Gitaly process: + + ```shell + sudo gitlab-ctl restart gitaly + ``` + +:::TabTitle Self-compiled (source) + +If the content of your SSL certificates under the `/etc/gitlab/ssl` directory have been updated, you must +[restart GitLab](../restart_gitlab.md#self-compiled-installations) for the certificates to be loaded by the Gitaly process. + +If you change or update the certificates in `/usr/local/share/ca-certificates`, you must: + +1. Run `sudo update-ca-certificates` to update the system's trusted store. +1. [Restart GitLab](../restart_gitlab.md#self-compiled-installations) for the certificates to be loaded by the Gitaly process. + +::EndTabs + ### Observe type of Gitaly connections For information on observing the type of Gitaly connections being served, see the @@ -866,6 +906,126 @@ When the pack-object cache is enabled, pack-objects limiting kicks in only if th You can observe the behavior of this queue using Gitaly logs and Prometheus. For more information, see [Monitor Gitaly pack-objects concurrency limiting](monitoring.md#monitor-gitaly-pack-objects-concurrency-limiting). +## Adaptive concurrency limiting + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/10734) in GitLab 16.6. + +Gitaly supports two concurrency limits: + +- An [RPC concurrency limit](#limit-rpc-concurrency), which allow you to configure a maximum number of simultaneous in-flight requests for each + Gitaly RPC. The limit is scoped by RPC and repository. +- A [Pack-objects concurrency limit](#limit-pack-objects-concurrency), which restricts the number of concurrent Git data transfer request by IP. + +If this limit is exceeded, either: + +- The request is put in a queue. +- The request is rejected if the queue is full or if the request remains in the queue for too long. + +Both of these concurrency limits can be configured statically. Though static limits can yield good protection results, they have some drawbacks: + +- Static limits are not good for all usage patterns. There is no one-size-fits-all value. If the limit is too low, big repositories are + negatively impacted. If the limit is too high, the protection is essentially lost. +- It's tedious to maintain a sane value for the concurrency limit, especially when the workload of each repository changes over time. +- A request can be rejected even though the server is idle because the rate doesn't factor in the load on the server. + +You can overcome all of these drawbacks and keep the benefits of concurrency limiting by configuring adaptive concurrency limits. Adaptive +concurrency limits are optional and build on the two concurrency limiting types. It uses Additive Increase/Multiplicative Decrease (AIMD) +algorithm. Each adaptive limit: + +- Gradually increases up to a certain upper limit during typical process functioning. +- Quickly decreases when the host machine has a resource problem. + +This mechanism provides some headroom for the machine to "breathe" and speeds up current inflight requests. + +![Gitaly Adaptive Concurrency Limit](img/gitaly_adaptive_concurrency_limit.png) + +The adaptive limiter calibrates the limits every 30 seconds and: + +- Increases the limits by one until reaching the upper limit. +- Decreases the limits by half when the top-level cgroup has either memory usage that exceeds 90%, excluding highly-evictable page caches, + or CPU throttled for 50% or more of the observation time. + +Otherwise, the limits increase by one until reaching the upper bound. For more information about technical implementation +of this system, please refer to [this blueprint](../../architecture/blueprints/gitaly_adaptive_concurrency_limit/index.md). + +Adaptive limiting is enabled for each RPC or pack-objects cache individually. However, limits are calibrated at the same time. + +### Enable adaptiveness for RPC concurrency + +Prerequisites: + +- Because adaptive limiting depends on [control groups](#control-groups), control groups must be enabled before using adaptive limiting. + +The following is an example to configure an adaptive limit for RPC concurrency: + +```ruby +# in /etc/gitlab/gitlab.rb +gitaly['configuration'] = { + # ... + concurrency: [ + { + rpc: '/gitaly.SmartHTTPService/PostUploadPackWithSidechannel', + max_queue_wait: '1s', + max_queue_size: 10, + adaptive: true, + min_limit: 10, + initial_limit: 20, + max_limit: 40 + }, + { + rpc: '/gitaly.SSHService/SSHUploadPackWithSidechannel', + max_queue_wait: '10s', + max_queue_size: 20, + adaptive: true, + min_limit: 10, + initial_limit: 50, + max_limit: 100 + }, + ], +} +``` + +In this example: + +- `adaptive` sets whether the adaptiveness is enabled. If set, the `max_per_repo` value is ignored in favor of the following configuration. +- `initial_limit` is the per-repository concurrency limit to use when Gitaly starts. +- `max_limit` is the minimum per-repository concurrency limit of the configured RPC. Gitaly increases the current limit + until it reaches this number. +- `min_limit` is the is the minimum per-repository concurrency limit of the configured RPC. When the host machine has a resource problem, + Gitaly quickly reduces the limit until reaching this value. + +For more information, see [RPC concurrency](#limit-rpc-concurrency). + +### Enable adaptiveness for pack-objects concurrency + +Prerequisites: + +- Because adaptive limiting depends on [control groups](#control-groups), control groups must be enabled before using adaptive limiting. + +The following is an example to configure an adaptive limit for pack-objects concurrency: + +```ruby +# in /etc/gitlab/gitlab.rb +gitaly['pack_objects_limiting'] = { + 'max_queue_length' => 200, + 'max_queue_wait' => '60s', + 'adaptive' => true, + 'min_limit' => 10, + 'initial_limit' => 20, + 'max_limit' => 40 +} +``` + +In this example: + +- `adaptive` sets whether the adaptiveness is enabled. If set, the value of `max_concurrency` is ignored in favor of the following configuration. +- `initial_limit` is the per-IP concurrency limit to use when Gitaly starts. +- `max_limit` is the minimum per-IP concurrency limit for pack-objects. Gitaly increases the current limit until it reaches this number. +- `min_limit` is the is the minimum per-IP concurrency limit for pack-objects. When the host machine has a resources problem, Gitaly quickly + reduces the limit until it reaches this value. + +For more information, see [pack-objects concurrency](#limit-pack-objects-concurrency). + ## Control groups WARNING: @@ -1673,7 +1833,9 @@ Gitaly fails to start up if either: ## Configure server-side backups -> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4941) in GitLab 16.3. +> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4941) in GitLab 16.3. +> - Server-side support for restoring a specified backup instead of the latest backup [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/132188) in GitLab 16.6. +> - Server-side support for creating incremental backups [introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6475) in GitLab 16.6. Repository backups can be configured so that the Gitaly node that hosts each repository is responsible for creating the backup and streaming it to diff --git a/doc/administration/gitaly/img/gitaly_adaptive_concurrency_limit.png b/doc/administration/gitaly/img/gitaly_adaptive_concurrency_limit.png Binary files differnew file mode 100644 index 00000000000..ce6bb1a8dfc --- /dev/null +++ b/doc/administration/gitaly/img/gitaly_adaptive_concurrency_limit.png diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 46f6a5829c8..6784ff4d970 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -587,92 +587,6 @@ off Gitaly Cluster to a sharded Gitaly instance: 1. [Move the repositories](../operations/moving_repositories.md#moving-repositories) to the newly created storage. You can move them by shard or by group, which gives you the opportunity to spread them over multiple Gitaly servers. -## Direct access to Git in GitLab - -Direct access to Git uses code in GitLab known as the "Rugged patches". - -Before Gitaly existed, what are now Gitaly clients accessed Git repositories directly, either: - -- On a local disk in the case of a single-machine Linux package installation. -- Using NFS in the case of a horizontally-scaled GitLab installation. - -In addition to running plain `git` commands, GitLab used a Ruby library called -[Rugged](https://github.com/libgit2/rugged). Rugged is a wrapper around -[libgit2](https://libgit2.org/), a stand-alone implementation of Git in the form of a C library. - -Over time it became clear that Rugged, particularly in combination with -[Unicorn](https://yhbt.net/unicorn/), is extremely efficient. Because `libgit2` is a library and -not an external process, there was very little overhead between: - -- GitLab application code that tried to look up data in Git repositories. -- The Git implementation itself. - -Because the combination of Rugged and Unicorn was so efficient, the GitLab application code ended up -with lots of duplicate Git object lookups. For example, looking up the default branch commit a dozen -times in one request. We could write inefficient code without poor performance. - -When we migrated these Git lookups to Gitaly calls, we suddenly had a much higher fixed cost per Git -lookup. Even when Gitaly is able to re-use an already-running `git` process (for example, to look up -a commit), you still have: - -- The cost of a network roundtrip to Gitaly. -- Inside Gitaly, a write/read roundtrip on the Unix pipes that connect Gitaly to the `git` process. - -Using GitLab.com to measure, we reduced the number of Gitaly calls per request until we no longer felt -the efficiency loss of losing Rugged. It also helped that we run Gitaly itself directly on the Git -file servers, rather than by using NFS mounts. This gave us a speed boost that counteracted the -negative effect of not using Rugged anymore. - -Unfortunately, other deployments of GitLab could not remove NFS like we did on GitLab.com, and they -got the worst of both worlds: - -- The slowness of NFS. -- The increased inherent overhead of Gitaly. - -The code removed from GitLab during the Gitaly migration project affected these deployments. As a -performance workaround for these NFS-based deployments, we re-introduced some of the old Rugged -code. This re-introduced code is informally referred to as the "Rugged patches". - -### Automatic detection - -> Automatic detection for Rugged [disabled](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/95445) in GitLab 15.3. - -FLAG: -On self-managed GitLab, by default automatic detection of whether Rugged should be used (per storage) is not available. -To make it available, an administrator can [disable the feature flag](../../administration/feature_flags.md) named -`skip_rugged_auto_detect`. - -The Ruby methods that perform direct Git access are behind -[feature flags](../../development/gitaly.md#legacy-rugged-code), disabled by default. It wasn't -convenient to set feature flags to get the best performance, so we added an automatic mechanism that -enables direct Git access. - -When GitLab calls a function that has a "Rugged patch", it performs two checks: - -- Is the feature flag for this patch set in the database? If so, the feature flag setting controls - the GitLab use of "Rugged patch" code. -- If the feature flag is not set, GitLab tries accessing the file system underneath the - Gitaly server directly. If it can, it uses the "Rugged patch": - - If using Puma and [thread count](../../install/requirements.md#puma-threads) is set - to `1`. - -The result of these checks is cached. - -To see if GitLab can access the repository file system directly, we use the following heuristic: - -- Gitaly ensures that the file system has a metadata file in its root with a UUID in it. -- Gitaly reports this UUID to GitLab by using the `ServerInfo` RPC. -- GitLab Rails tries to read the metadata file directly. If it exists, and if the UUIDs match, - assume we have direct access. - -Direct Git access is: - -- [Disabled](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/95445) by default in GitLab 15.3 and later for - compatibility with [Praefect-generated replica paths](#praefect-generated-replica-paths-gitlab-150-and-later). It - can be enabled if Rugged [feature flags](../../development/gitaly.md#legacy-rugged-code) are enabled. -- Enabled by default in GitLab 15.2 and earlier because it fills in the correct repository paths in the GitLab - configuration file `config/gitlab.yml`. This satisfies the UUID check. - ### Transition to Gitaly Cluster For the sake of removing complexity, we must remove direct Git access in GitLab. However, we can't diff --git a/doc/administration/gitaly/monitoring.md b/doc/administration/gitaly/monitoring.md index cbf5722f2c5..5d8de42666b 100644 --- a/doc/administration/gitaly/monitoring.md +++ b/doc/administration/gitaly/monitoring.md @@ -90,6 +90,47 @@ In Prometheus, look for the following metrics: - `gitaly_pack_objects_queued` indicates how many requests for pack-objects processes are waiting due to the concurrency limit being reached. - `gitaly_pack_objects_acquiring_seconds` indicates how long a request for a pack-object process has to wait due to concurrency limits before being processed. +## Monitor Gitaly adaptive concurrency limiting + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/10734) in GitLab 16.6. + +You can observe specific behavior of [adaptive concurrency limiting](configure_gitaly.md#adaptive-concurrency-limiting) using Gitaly logs and Prometheus. + +In the [Gitaly logs](../logs/index.md#gitaly-logs), you can identify logs related to the adaptive concurrency limiting when the current limits are adjusted. +You can filter the content of the logs (`msg`) for "Multiplicative decrease" and "Additive increase" messages. + +| Log Field | Description | +|:---|:---| +| `limit` | The name of the limit being adjusted. | +| `previous_limit` | The previous limit before it was increased or decreased. | +| `new_limit` | The new limit after it was increased or decreased. | +| `watcher` | The resource watcher that decided the node is under pressure. For example: `CgroupCpu` or `CgroupMemory`. | +| `reason` | The reason behind limit adjustment. | +| `stats.*` | Some statistics behind an adjustment decision. They are for debugging purposes. | + +Example log: + +```json +{ + "msg": "Multiplicative decrease", + "limit": "pack-objects", + "new_limit": 14, + "previous_limit": 29, + "reason": "cgroup CPU throttled too much", + "watcher": "CgroupCpu", + "stats.time_diff": 15.0, + "stats.throttled_duration": 13.0, + "stat.sthrottled_threshold": 0.5 +} +``` + +In Prometheus, look for the following metrics: + +- `gitaly_concurrency_limiting_current_limit` The current limit value of an adaptive concurrency limit. +- `gitaly_concurrency_limiting_watcher_errors_total` indicates the total number of watcher errors while fetching resource metrics. +- `gitaly_concurrency_limiting_backoff_events_total` indicates the total number of backoff events, which are when the limits being + adjusted due to resource pressure. + ## Monitor Gitaly cgroups You can observe the status of [control groups (cgroups)](configure_gitaly.md#control-groups) using Prometheus: diff --git a/doc/administration/gitaly/recovery.md b/doc/administration/gitaly/recovery.md index 45bde083a1a..6779823c941 100644 --- a/doc/administration/gitaly/recovery.md +++ b/doc/administration/gitaly/recovery.md @@ -15,12 +15,17 @@ You can add and replace Gitaly nodes on a Gitaly Cluster. ### Add new Gitaly nodes -To add a new Gitaly node to a Gitaly Cluster that has [replication factor](praefect.md#configure-replication-factor): +The steps to add a new Gitaly node to a Gitaly Cluster depend on whether a [custom replication factor](praefect.md#configure-replication-factor) is set. -- Set, set the [replication factor](praefect.md#configure-replication-factor) for each repository using `set-replication-factor` Praefect command. New repositories are - replicated based on [replication factor](praefect.md#configure-replication-factor). Praefect doesn't automatically replicate existing repositories to the new Gitaly node. -- Not set, add the new node in your [Praefect configuration](praefect.md#praefect) under `praefect['virtual_storages']`. Praefect automatically replicates all data to any - new Gitaly node added to the configuration. +#### Custom replication factor + +If a custom replication factor is set, set the [replication factor](praefect.md#configure-replication-factor) for each repository using the +`set-replication-factor` Praefect command. New repositories are replicated based on the [replication factor](praefect.md#configure-replication-factor). Praefect doesn't automatically replicate existing repositories to the new Gitaly node. + +#### Default replication factor + +If the default replication factor is used, add the new node in your [Praefect configuration](praefect.md#praefect) under `praefect['virtual_storages']`. +Praefect automatically replicates all data to any new Gitaly node added to the configuration. ### Replace an existing Gitaly node @@ -33,32 +38,37 @@ To use the same name for the replacement node, use [repository verifier](praefec #### With a node with a different name -To use a different name for the replacement node for a Gitaly Cluster that has [replication factor](praefect.md#configure-replication-factor): +The steps use a different name for the replacement node for a Gitaly Cluster depend on if a [custom replication factor](praefect.md#configure-replication-factor) +is set. -- Set, use [`praefect set-replication-factor`](praefect.md#configure-replication-factor) to set the replication factor per repository again to get new storage assigned. - For example: +##### Custom replication factor set - ```shell - $ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml set-replication-factor -virtual-storage default -relative-path @hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.git -replication-factor 2 +If a custom replication factor is set, use [`praefect set-replication-factor`](praefect.md#configure-replication-factor) to set the replication factor per repository again to get new storage assigned. For example: - current assignments: gitaly-1, gitaly-2 - ``` +```shell +$ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml set-replication-factor -virtual-storage default -relative-path @hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.git -replication-factor 2 + +current assignments: gitaly-1, gitaly-2 +``` + +To reassign all repositories from the old storage to the new one, after configuring the new Gitaly node: - To reassign all repositories from the old storage to the new one, after configuring the new Gitaly node: +1. Connect to Praefect database: - 1. Connect to Praefect database: + ```shell + /opt/gitlab/embedded/bin/psql -h <psql host> -U <user> -d <database name> + ``` - ```shell - /opt/gitlab/embedded/bin/psql -h <psql host> -U <user> -d <database name> - ``` +1. Update the `repository_assignments` table to replace the old Gitaly node name (for example, `old-gitaly`) with the new Gitaly node name + (for example, `new-gitaly`): - 1. Update `repository_assignments` table to replace the old Gitaly node name (for example, `old-gitaly`) with the new Gitaly node name (for example, `new-gitaly`): + ```sql + UPDATE repository_assignments SET storage='new-gitaly' WHERE storage='old-gitaly'; + ``` - ```sql - UPDATE repository_assignments SET storage='new-gitaly' WHERE storage='old-gitaly'; - ``` +##### Default replication factor -- Not set, replace the node in the configuration. The old node's state remains in the Praefect database but it is ignored. +If the default replication factor is used, replace the node in the configuration. The old node's state remains in the Praefect database but it is ignored. ## Primary node failure diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index 556bc29b76f..17687cbb181 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -387,6 +387,43 @@ If Git pushes are too slow when Dynatrace is enabled, disable Dynatrace. One way to resolve this is to make sure the entry is correct for the GitLab internal API URL configured in `gitlab.rb` with `gitlab_rails['internal_api_url']`. +### Changes (diffs) don't load for new merge requests when using Gitaly TLS + +After enabling [Gitaly with TLS](configure_gitaly.md#enable-tls-support), changes (diffs) for new merge requests are not generated +and you see the following message in GitLab: + +```plaintext +Building your merge request... This page will update when the build is complete +``` + +Gitaly must be able to connect to itself to complete some operations. If the Gitaly certificate is not trusted by the Gitaly server, +merge request diffs can't be generated. + +If Gitaly can't connect to itself, you see messages in the [Gitaly logs](../../administration/logs/index.md#gitaly-logs) like the following messages: + +```json +{ + "level":"warning", + "msg":"[core] [Channel #16 SubChannel #17] grpc: addrConn.createTransport failed to connect to {Addr: \"ext-gitaly.example.com:9999\", ServerName: \"ext-gitaly.example.com:9999\", }. Err: connection error: desc = \"transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority\"", + "pid":820, + "system":"system", + "time":"2023-11-06T05:40:04.169Z" +} +{ + "level":"info", + "msg":"[core] [Server #3] grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"ServerHandshake(\\\"x.x.x.x:x\\\") failed: wrapped server handshake: remote error: tls: bad certificate\"", + "pid":820, + "system":"system", + "time":"2023-11-06T05:40:04.169Z" +} +``` + +To resolve the problem, ensure that you have added your Gitaly certificate to the `/etc/gitlab/trusted-certs` folder on the Gitaly server +and: + +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) so the certificates are symlinked +1. Restart Gitaly manually `sudo gitlab-ctl restart gitaly` for the certificates to be loaded by the Gitaly process. + ## Gitaly fails to fork processes stored on `noexec` file systems Because of changes [introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5999) in GitLab 14.10, applying the `noexec` option to a mount diff --git a/doc/administration/inactive_project_deletion.md b/doc/administration/inactive_project_deletion.md index b7f71505e70..7ccd3455011 100644 --- a/doc/administration/inactive_project_deletion.md +++ b/doc/administration/inactive_project_deletion.md @@ -34,10 +34,13 @@ To configure deletion of inactive projects: 1. Select **Save changes**. Inactive projects that meet the criteria are scheduled for deletion and a warning email is sent. If the -projects remain inactive, they are deleted after the specified duration. +projects remain inactive, they are deleted after the specified duration. These projects are deleted even if +[the project is archived](../user/project/settings/index.md#archive-a-project). ### Configuration example +#### Example 1 + If you use these settings: - **Delete inactive projects** enabled. @@ -52,6 +55,20 @@ If a project is more than 50 MB and it is inactive for: - More than 6 months: A deletion warning email is sent. This mail includes the date that the project will be deleted. - More than 12 months: The project is scheduled for deletion. +#### Example 2 + +If you use these settings: + +- **Delete inactive projects** enabled. +- **Delete inactive projects that exceed** set to `0`. +- **Delete project after** set to `12`. +- **Send warning email** set to `11`. + +If a project exists that has already been inactive for more than 12 months when you configure these settings: + +- A deletion warning email is sent immediately. This email includes the date that the project will be deleted. +- The project is scheduled for deletion 1 month (12 months - 11 months) after warning email. + ## Determine when a project was last active You can view a project's activities and determine when the project was last active in the following ways: diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index 6948009aab2..33afaf19220 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -68,7 +68,8 @@ this method only supports replies, and not the other features of [incoming email ## Accepted headers -> Accepting `Received` headers [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/81489) in GitLab 14.9. +> - Accepting `Received` headers [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/81489) in GitLab 14.9. +> - Accepting `Cc` headers [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/348572) in GitLab 16.5. Email is processed correctly when a configured email address is present in one of the following headers (sorted in the order they are checked): @@ -77,6 +78,7 @@ Email is processed correctly when a configured email address is present in one o - `Delivered-To` - `Envelope-To` or `X-Envelope-To` - `Received` +- `Cc` The `References` header is also accepted, however it is used specifically to relate email responses to existing discussion threads. It is not used for creating issues by email. @@ -86,8 +88,7 @@ also checks accepted headers. Usually, the "To" field contains the email address of the primary receiver. However, it might not include the configured GitLab email address if: -- The address is in the "CC" field. -- The address was included when using "Reply all". +- The address is in the "BCC" field. - The email was forwarded. The `Received` header can contain multiple email addresses. These are checked in the order that they appear. diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index 8f03a2224ec..d5855e3c832 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -309,7 +309,7 @@ The number of seconds GitLab waits for an HTTP response after sending a webhook. To change the webhook timeout value: -1. Edit `/etc/gitlab/gitlab.rb`: +1. Edit `/etc/gitlab/gitlab.rb` on all GitLab nodes that are running Sidekiq: ```ruby gitlab_rails['webhook_timeout'] = 60 @@ -992,18 +992,19 @@ Set the limit to `0` to disable it. ## Math rendering limits -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/132939) in GitLab 16.5. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/132939) in GitLab 16.5. +> - [Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/368009) the 50-node limit from Wiki and repository files. GitLab imposes default limits when rendering math in Markdown fields. These limits provide better security and performance. -The limits for issues, merge requests, wikis, and repositories: +The limits for issues, merge requests, epics, wikis, and repository files: -- Maximum number of nodes rendered: `50`. - Maximum number of macro expansions: `1000`. -- Maximum user-specified size in em: `20`. +- Maximum user-specified size in [em](https://en.wikipedia.org/wiki/Em_(typography)): `20`. -The limits for issues and merge requests: +The limits for issues, merge requests, and epics: +- Maximum number of nodes rendered: `50`. - Maximum number of characters in a math block: `1000`. - Maximum rendering time: `2000 ms`. diff --git a/doc/administration/integration/plantuml.md b/doc/administration/integration/plantuml.md index 0155f0300d4..dae400ff755 100644 --- a/doc/administration/integration/plantuml.md +++ b/doc/administration/integration/plantuml.md @@ -180,7 +180,7 @@ see the [Tomcat Documentation](https://tomcat.apache.org/tomcat-10.1-doc/index.h 1. Install and configure Tomcat 10: ```shell - wget https://dlcdn.apache.org/tomcat/tomcat-10/v10.1.9/bin/apache-tomcat-10.1.9.tar.gz -P /tmp + wget https://dlcdn.apache.org/tomcat/tomcat-10/v10.1.15/bin/apache-tomcat-10.1.15.tar.gz -P /tmp sudo tar xzvf /tmp/apache-tomcat-10*tar.gz -C /opt/tomcat --strip-components=1 sudo chown -R tomcat:tomcat /opt/tomcat/ sudo chmod -R u+x /opt/tomcat/bin @@ -266,12 +266,11 @@ see the [Tomcat Documentation](https://tomcat.apache.org/tomcat-10.1-doc/index.h 1. Install PlantUML and copy the `.war` file: - Use the [latest release](https://github.com/plantuml/plantuml-server/releases) of plantuml-jsp (example: plantuml-jsp-v1.2023.8.war). For context, see [this issue](https://github.com/plantuml/plantuml-server/issues/265). + Use the [latest release](https://github.com/plantuml/plantuml-server/releases) of plantuml-jsp (example: plantuml-jsp-v1.2023.12.war). For context, see [this issue](https://github.com/plantuml/plantuml-server/issues/265). ```shell - cd / - wget https://github.com/plantuml/plantuml-server/releases/download/v1.2023.8/plantuml-jsp-v1.2023.8.war - sudo cp plantuml-jsp-v1.2023.8.war /opt/tomcat/webapps/plantuml.war + wget -P /tmp https://github.com/plantuml/plantuml-server/releases/download/v1.2023.12/plantuml-jsp-v1.2023.12.war + sudo cp /tmp/plantuml-jsp-v1.2023.12.war /opt/tomcat/webapps/plantuml.war sudo chown tomcat:tomcat /opt/tomcat/webapps/plantuml.war sudo systemctl restart tomcat ``` diff --git a/doc/administration/logs/index.md b/doc/administration/logs/index.md index e7277ab3186..3bb26681fae 100644 --- a/doc/administration/logs/index.md +++ b/doc/administration/logs/index.md @@ -806,12 +806,12 @@ GraphQL queries are recorded in the file. For example: {"query_string":"query IntrospectionQuery{__schema {queryType { name },mutationType { name }}}...(etc)","variables":{"a":1,"b":2},"complexity":181,"depth":1,"duration_s":7} ``` -## `clickhouse.log` **(SAAS)** +## `clickhouse.log` > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/133371) in GitLab 16.5. -The `clickhouse.log` file logs information related to -Clickhouse database client within GitLab. +The `clickhouse.log` file logs information related to the +ClickHouse database client in GitLab. ## `migrations.log` diff --git a/doc/administration/logs/log_parsing.md b/doc/administration/logs/log_parsing.md index 21ce3d7f17f..b281620fcf3 100644 --- a/doc/administration/logs/log_parsing.md +++ b/doc/administration/logs/log_parsing.md @@ -96,10 +96,10 @@ grep <PROJECT_NAME> <FILE> | jq . jq 'select(.duration_s > 5000)' <FILE> ``` -#### Find all project requests with more than 5 rugged calls +#### Find all project requests with more than 5 Gitaly calls ```shell -grep <PROJECT_NAME> <FILE> | jq 'select(.rugged_calls > 5)' +grep <PROJECT_NAME> <FILE> | jq 'select(.gitaly_calls > 5)' ``` #### Find all requests with a Gitaly duration > 10 seconds @@ -273,8 +273,8 @@ jq --raw-output --slurp ' .[2]."grpc.time_ms", .[0]."grpc.request.glProjectPath" ] - | @sh' current \ -| awk 'BEGIN { printf "%7s %10s %10s %10s\t%s\n", "CT", "MAX DURS", "", "", "PROJECT" } + | @sh' current | + awk 'BEGIN { printf "%7s %10s %10s %10s\t%s\n", "CT", "MAX DURS", "", "", "PROJECT" } { printf "%7u %7u ms, %7u ms, %7u ms\t%s\n", $1, $2, $3, $4, $5 }' ``` @@ -288,12 +288,18 @@ jq --raw-output --slurp ' ... ``` +#### Types of user and project activity overview + +```shell +jq --raw-output '[.username, ."grpc.method", ."grpc.request.glProjectPath"] | @tsv' current | sort | uniq -c | sort -n +``` + #### Find all projects affected by a fatal Git problem ```shell -grep "fatal: " current | \ - jq '."grpc.request.glProjectPath"' | \ - sort | uniq +grep "fatal: " current | + jq '."grpc.request.glProjectPath"' | + sort | uniq ``` ### Parsing `gitlab-shell/gitlab-shell.log` diff --git a/doc/administration/merge_request_diffs.md b/doc/administration/merge_request_diffs.md index 746dccb99d6..9c4ddcdc094 100644 --- a/doc/administration/merge_request_diffs.md +++ b/doc/administration/merge_request_diffs.md @@ -21,7 +21,9 @@ that only [stores outdated diffs](#alternative-in-database-storage) outside of d ## Using external storage -For Linux package installations: +::Tabs + +:::TabTitle Linux package (Omnibus) 1. Edit `/etc/gitlab/gitlab.rb` and add the following line: @@ -41,7 +43,7 @@ For Linux package installations: 1. Save the file and [reconfigure GitLab](restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. GitLab then migrates your existing merge request diffs to external storage. -For self-compiled installations: +:::TabTitle Self-compiled (source) 1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: @@ -65,6 +67,8 @@ For self-compiled installations: 1. Save the file and [restart GitLab](restart_gitlab.md#self-compiled-installations) for the changes to take effect. GitLab then migrates your existing merge request diffs to external storage. +::EndTabs + ## Using object storage WARNING: @@ -74,7 +78,9 @@ Instead of storing the external diffs on disk, we recommended the use of an obje store like AWS S3 instead. This configuration relies on valid AWS credentials to be configured already. -For Linux package installations: +::Tabs + +:::TabTitle Linux package (Omnibus) 1. Edit `/etc/gitlab/gitlab.rb` and add the following line: @@ -86,7 +92,7 @@ For Linux package installations: 1. Save the file and [reconfigure GitLab](restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. GitLab then migrates your existing merge request diffs to external storage. -For self-compiled installations: +:::TabTitle Self-compiled (source) 1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: @@ -100,6 +106,8 @@ For self-compiled installations: 1. Save the file and [restart GitLab](restart_gitlab.md#self-compiled-installations) for the changes to take effect. GitLab then migrates your existing merge request diffs to external storage. +::EndTabs + [Read more about using object storage with GitLab](object_storage.md). ### Object Storage Settings @@ -123,7 +131,9 @@ then `object_store:`. On Linux package installations, they are prefixed by See [the available connection settings for different providers](object_storage.md#configure-the-connection-settings). -For Linux package installations: +::Tabs + +:::TabTitle Linux package (Omnibus) 1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with the values you want: @@ -153,7 +163,7 @@ For Linux package installations: 1. Save the file and [reconfigure GitLab](restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. -For self-compiled installations: +:::TabTitle Self-compiled (source) 1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: @@ -173,6 +183,8 @@ For self-compiled installations: 1. Save the file and [restart GitLab](restart_gitlab.md#self-compiled-installations) for the changes to take effect. +::EndTabs + ## Alternative in-database storage Enabling external diffs may reduce the performance of merge requests, as they @@ -182,7 +194,9 @@ in the database. To enable this feature, perform the following steps: -For Linux package installations: +::Tabs + +:::TabTitle Linux package (Omnibus) 1. Edit `/etc/gitlab/gitlab.rb` and add the following line: @@ -192,7 +206,7 @@ For Linux package installations: 1. Save the file and [reconfigure GitLab](restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. -For self-compiled installations: +:::TabTitle Self-compiled (source) 1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: @@ -205,6 +219,8 @@ For self-compiled installations: 1. Save the file and [restart GitLab](restart_gitlab.md#self-compiled-installations) for the changes to take effect. +::EndTabs + With this feature enabled, diffs are initially stored in the database, rather than externally. They are moved to external storage after any of these conditions become true: @@ -217,64 +233,45 @@ These rules strike a balance between space and performance by only storing frequently-accessed diffs in the database. Diffs that are less likely to be accessed are moved to external storage instead. -## Correcting incorrectly-migrated diffs - -Versions of GitLab earlier than `v13.0.0` would incorrectly record the location -of some merge request diffs when [external diffs in object storage](#object-storage-settings) -were enabled. This mainly affected imported merge requests, and was resolved -with [this merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/31005). - -If you are using object storage, or have never used on-disk storage for external -diffs, the **Changes** tab for some merge requests fails to load with a 500 error, -and the exception for that error is of this form: - -```plain -Errno::ENOENT (No such file or directory @ rb_sysopen - /var/opt/gitlab/gitlab-rails/shared/external-diffs/merge_request_diffs/mr-6167082/diff-8199789) -``` - -Then you are affected by this issue. Because it's not possible to safely determine -all these conditions automatically, we've provided a Rake task in GitLab v13.2.0 -that you can run manually to correct the data: - -For Linux package installations: - -```shell -sudo gitlab-rake gitlab:external_diffs:force_object_storage -``` - -For self-compiled installations: +## Switching from external storage to object storage -```shell -sudo -u git -H bundle exec rake gitlab:external_diffs:force_object_storage RAILS_ENV=production -``` +Automatic migration moves diffs stored in the database, but it does not move diffs between storage types. +To switch from external storage to object storage: -Environment variables can be provided to modify the behavior of the task. The -available variables are: +1. Move files stored on local or NFS storage to object storage manually. +1. Run this Rake task to change their location in the database. -| Name | Default value | Purpose | -| ---- | ------------- | ------- | -| `ANSI` | `true` | Use ANSI escape codes to make output more understandable | -| `BATCH_SIZE` | `1000` | Iterate through the table in batches of this size | -| `START_ID` | `nil` | If set, begin scanning at this ID | -| `END_ID` | `nil` | If set, stop scanning at this ID | -| `UPDATE_DELAY` | `1` | Number of seconds to sleep between updates | + For Linux package installations: -The `START_ID` and `END_ID` variables may be used to run the update in parallel, -by assigning different processes to different parts of the table. The `BATCH` -and `UPDATE_DELAY` parameters allow the speed of the migration to be traded off -against concurrent access to the table. The `ANSI` parameter should be set to -false if your terminal does not support ANSI escape codes. + ```shell + sudo gitlab-rake gitlab:external_diffs:force_object_storage + ``` -By default, `sudo` does not preserve existing environment variables. You should append them, rather than prefix them. + For self-compiled installations: -```shell -sudo gitlab-rake gitlab:external_diffs:force_object_storage START_ID=59946109 END_ID=59946109 UPDATE_DELAY=5 -``` + ```shell + sudo -u git -H bundle exec rake gitlab:external_diffs:force_object_storage RAILS_ENV=production + ``` -## Switching from external storage to object storage + By default, `sudo` does not preserve existing environment variables. You should + append them, rather than prefix them, like this: -Automatic migration moves diffs stored in the database, but it does not move diffs between storage types. -To switch from external storage to object storage: + ```shell + sudo gitlab-rake gitlab:external_diffs:force_object_storage START_ID=59946109 END_ID=59946109 UPDATE_DELAY=5 + ``` -1. Move files stored on local or NFS storage to object storage manually. -1. Run the Rake task in the [previous section](#correcting-incorrectly-migrated-diffs) to change their location in the database. +These environment variables modify the behavior of the Rake task: + +| Name | Default value | Purpose | +|----------------|---------------|---------| +| `ANSI` | `true` | Use ANSI escape codes to make output more understandable. | +| `BATCH_SIZE` | `1000` | Iterate through the table in batches of this size. | +| `START_ID` | `nil` | If set, begin scanning at this ID. | +| `END_ID` | `nil` | If set, stop scanning at this ID. | +| `UPDATE_DELAY` | `1` | Number of seconds to sleep between updates. | + +- `START_ID` and `END_ID` can be used to run the update in parallel, + by assigning different processes to different parts of the table. +- `BATCH` and `UPDATE_DELAY` enable the speed of the migration to be traded off + against concurrent access to the table. +- `ANSI` should be set to `false` if your terminal does not support ANSI escape codes. diff --git a/doc/administration/moderate_users.md b/doc/administration/moderate_users.md index b30294c5fe0..c12eb2b9a95 100644 --- a/doc/administration/moderate_users.md +++ b/doc/administration/moderate_users.md @@ -287,6 +287,45 @@ You can also delete a user and their contributions, such as merge requests, issu NOTE: Before 15.1, additionally groups of which deleted user were the only owner among direct members were deleted. +## Trust and untrust users + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/132402) in GitLab 16.5. + +You can trust and untrust users from the Admin Area. + +By default, a user is not trusted and is blocked from creating issues, notes, and snippets considered to be spam. When you trust a user, they can create issues, notes, and snippets without being blocked. + +Prerequisite: + +- You must be an administrator. + +::Tabs + +:::TabTitle Trust a user + +1. On the left sidebar, select **Search or go to**. +1. Select **Admin Area**. +1. Select **Overview > Users**. +1. Select a user. +1. From the **User administration** dropdown list, select **Trust user**. +1. On the confirmation dialog, select **Trust user**. + +The user is trusted. + +:::TabTitle Untrust a user + +1. On the left sidebar, select **Search or go to**. +1. Select **Admin Area**. +1. Select **Overview > Users**. +1. Select the **Trusted** tab. +1. Select a user. +1. From the **User administration** dropdown list, select **Untrust user**. +1. On the confirmation dialog, select **Untrust user**. + +The user is untrusted. + +::EndTabs + ## Troubleshooting When moderating users, you may need to perform bulk actions on them based on certain conditions. The following rails console scripts show some examples of this. You may [start a rails console session](../administration/operations/rails_console.md#starting-a-rails-console-session) and use scripts similar to the following: diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index 12fa79b3c13..95717f0c54f 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -17,6 +17,8 @@ For example: ## Available information +> Rugged calls [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/421591) in GitLab 16.6. + From left to right, the performance bar displays: - **Current Host**: the current host serving the page. @@ -37,8 +39,6 @@ From left to right, the performance bar displays: - **Gitaly calls**: the time taken (in milliseconds) and the total number of [Gitaly](../../gitaly/index.md) calls. Select to display a modal window with more details. -- **Rugged calls**: the time taken (in milliseconds) and the total number of - Rugged calls. Select to display a modal window with more details. - **Redis calls**: the time taken (in milliseconds) and the total number of Redis calls. Select to display a modal window with more details. - **Elasticsearch calls**: the time taken (in milliseconds) and the total number of diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index 9efe39b8d3a..2eb482cae69 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -48,8 +48,6 @@ The following metrics are available: | `gitlab_ci_runner_authentication_failure_total` | Counter | 15.2 | Total number of times that runner authentication has failed | `gitlab_ghost_user_migration_lag_seconds` | Gauge | 15.6 | The waiting time in seconds of the oldest scheduled record for ghost user migration | | | `gitlab_ghost_user_migration_scheduled_records_total` | Gauge | 15.6 | The total number of scheduled ghost user migrations | | -| `job_waiter_started_total` | Counter | 12.9 | Number of batches of jobs started where a web request is waiting for the jobs to complete | `worker` | -| `job_waiter_timeouts_total` | Counter | 12.9 | Number of batches of jobs that timed out where a web request is waiting for the jobs to complete | `worker` | | `gitlab_ci_active_jobs` | Histogram | 14.2 | Count of active jobs when pipeline is created | | | `gitlab_database_transaction_seconds` | Histogram | 12.1 | Time spent in database transactions, in seconds | | | `gitlab_method_call_duration_seconds` | Histogram | 10.2 | Method calls real duration | `controller`, `action`, `module`, `method` | @@ -245,7 +243,6 @@ configuration option in `gitlab.yml`. These metrics are served from the | `geo_cursor_last_event_timestamp` | Gauge | 10.2 | Last UNIX timestamp of the event log processed by the secondary | `url` | | `geo_status_failed_total` | Counter | 10.2 | Number of times retrieving the status from the Geo Node failed | `url` | | `geo_last_successful_status_check_timestamp` | Gauge | 10.2 | Last timestamp when the status was successfully updated | `url` | -| `geo_job_artifacts_synced_missing_on_primary` | Gauge | 10.7 | Number of job artifacts marked as synced due to the file missing on the primary | `url` | | `geo_package_files` | Gauge | 13.0 | Number of package files on primary | `url` | | `geo_package_files_checksummed` | Gauge | 13.0 | Number of package files checksummed on primary | `url` | | `geo_package_files_checksum_failed` | Gauge | 13.0 | Number of package files failed to calculate the checksum on primary | `url` | @@ -386,7 +383,12 @@ configuration option in `gitlab.yml`. These metrics are served from the | `geo_project_repositories_verification_total` | Gauge | 16.2 | Number of Project Repositories to attempt to verify on secondary | `url` | | `geo_project_repositories_verified` | Gauge | 16.2 | Number of Project Repositories successfully verified on secondary | `url` | | `geo_project_repositories_verification_failed` | Gauge | 16.2 | Number of Project Repositories that failed verification on secondary | `url` | - +| `geo_repositories_synced` | Gauge | 10.2 | Deprecated for removal in 17.0. Missing in 16.3 and 16.4. Replaced by `geo_project_repositories_synced`. Number of repositories synced on secondary | `url` | +| `geo_repositories_failed` | Gauge | 10.2 | Deprecated for removal in 17.0. Missing in 16.3 and 16.4. Replaced by `geo_project_repositories_failed`. Number of repositories failed to sync on secondary | `url` | +| `geo_repositories_checksummed` | Gauge | 10.7 | Deprecated for removal in 17.0. Missing in 16.3 and 16.4. Replaced by `geo_project_repositories_checksummed`. Number of repositories checksummed on primary | `url` | +| `geo_repositories_checksum_failed` | Gauge | 10.7 | Deprecated for removal in 17.0. Missing in 16.3 and 16.4. Replaced by `geo_project_repositories_checksum_failed`. Number of repositories failed to calculate the checksum on primary | `url` | +| `geo_repositories_verified` | Gauge | 10.7 | Deprecated for removal in 17.0. Missing in 16.3 and 16.4. Replaced by `geo_project_repositories_verified`. Number of repositories successfully verified on secondary | `url` | +| `geo_repositories_verification_failed` | Gauge | 10.7 | Deprecated for removal in 17.0. Missing in 16.3 and 16.4. Replaced by `geo_project_repositories_verification_failed`. Number of repositories that failed verification on secondary | `url` | | `gitlab_memwd_violations_total` | Counter | 15.9 | Total number of times a Sidekiq process violated a memory threshold | | | `gitlab_memwd_violations_handled_total` | Counter | 15.9 | Total number of times Sidekiq process memory violations were handled | | | `sidekiq_watchdog_running_jobs_total` | Counter | 15.9 | Current running jobs when RSS limit was reached | `worker_class` | diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index df6dd87c896..01b1851ab7f 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -302,6 +302,10 @@ update the firewall on the instance to only allow traffic from your Prometheus I static_configs: - targets: - 1.1.1.1:9236 + - job_name: registry + static_configs: + - targets: + - 1.1.1.1:5001 ``` WARNING: diff --git a/doc/administration/monitoring/prometheus/web_exporter.md b/doc/administration/monitoring/prometheus/web_exporter.md index a2dee80f6d4..fbf4a109813 100644 --- a/doc/administration/monitoring/prometheus/web_exporter.md +++ b/doc/administration/monitoring/prometheus/web_exporter.md @@ -71,3 +71,11 @@ To serve metrics via HTTPS instead of HTTP, enable TLS in the exporter settings: When TLS is enabled, the same `port` and `address` is used as described above. The metrics server cannot serve both HTTP and HTTPS at the same time. + +## Troubleshooting + +### Docker container runs out of space + +When running [GitLab in Docker](../../../install/docker.md), your container might run out of space. This can happen if you enable certain features which increase your space consumption, for example Web Exporter. + +To work around this issue, [update your `shm-size`](../../../install/docker.md#devshm-mount-not-having-enough-space-in-docker-container). diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md index f16f1ac46ae..89f1574697f 100644 --- a/doc/administration/operations/puma.md +++ b/doc/administration/operations/puma.md @@ -140,37 +140,6 @@ When running Puma in single mode, some features are not supported: For more information, see [epic 5303](https://gitlab.com/groups/gitlab-org/-/epics/5303). -## Performance caveat when using Puma with Rugged - -For deployments where NFS is used to store Git repositories, GitLab uses -[direct Git access](../gitaly/index.md#direct-access-to-git-in-gitlab) to improve performance by using -[Rugged](https://github.com/libgit2/rugged). - -Rugged usage is automatically enabled if direct Git access [is available](../gitaly/index.md#automatic-detection) and -Puma is running single threaded, unless it is disabled by a [feature flag](../../development/gitaly.md#legacy-rugged-code). - -MRI Ruby uses a Global VM Lock (GVL). GVL allows MRI Ruby to be multi-threaded, but running at -most on a single core. - -Git includes intensive I/O operations. When Rugged uses a thread for a long period of time, -other threads that might be processing requests can starve. Puma running in single thread mode -does not have this issue, because concurrently at most one request is being processed. - -GitLab is working to remove Rugged usage. Even though performance without Rugged -is acceptable today, in some cases it might be still beneficial to run with it. - -Given the caveat of running Rugged with multi-threaded Puma, and acceptable -performance of Gitaly, we disable Rugged usage if Puma multi-threaded is -used (when Puma is configured to run with more than one thread). - -This default behavior may not be the optimal configuration in some situations. If Rugged -plays an important role in your deployment, we suggest you benchmark to find the -optimal configuration: - -- The safest option is to start with single-threaded Puma. -- To force Rugged to be used with multi-threaded Puma, you can use a - [feature flag](../../development/gitaly.md#legacy-rugged-code). - ## Configuring Puma to listen over SSL Puma, when deployed with a Linux package installation, listens over a Unix socket by diff --git a/doc/administration/package_information/supported_os.md b/doc/administration/package_information/supported_os.md index 2064ee2a8e2..ab579ca93c6 100644 --- a/doc/administration/package_information/supported_os.md +++ b/doc/administration/package_information/supported_os.md @@ -24,7 +24,7 @@ architecture. | ------------------------------------------------------------ | ------------------------------ | --------------- | :----------------------------------------------------------: | ---------- | ------------------------------------------------------------ | | AlmaLinux 8 | GitLab CE / GitLab EE 14.5.0 | x86_64, aarch64 | [AlmaLinux Install Documentation](https://about.gitlab.com/install/#almalinux) | 2029 | <https://almalinux.org/> | | AlmaLinux 9 | GitLab CE / GitLab EE 16.0.0 | x86_64, aarch64 | [AlmaLinux Install Documentation](https://about.gitlab.com/install/#almalinux) | 2032 | <https://almalinux.org/> | -| CentOS 7 | GitLab CE / GitLab EE 7.10.0 | x86_64 | [CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | June 2024 | <https://wiki.centos.org/About/Product> | +| CentOS 7 | GitLab CE / GitLab EE 7.10.0 | x86_64 | [CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | June 2024 | <https://www.centos.org/about/> | | Debian 10 | GitLab CE / GitLab EE 12.2.0 | amd64, arm64 | [Debian Install Documentation](https://about.gitlab.com/install/#debian) | 2024 | <https://wiki.debian.org/LTS> | | Debian 11 | GitLab CE / GitLab EE 14.6.0 | amd64, arm64 | [Debian Install Documentation](https://about.gitlab.com/install/#debian) | 2026 | <https://wiki.debian.org/LTS> | | Debian 12 | GitLab CE / GitLab EE 16.1.0 | amd64, arm64 | [Debian Install Documentation](https://about.gitlab.com/install/#debian) | TBD | <https://wiki.debian.org/LTS> | diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index dcc6b768eed..74dd71c19bf 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -9,7 +9,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w With the GitLab Container Registry, every project can have its own space to store Docker images. -Read more about the Docker Registry in [the Docker documentation](https://docs.docker.com/registry/introduction/). +For more details about the Distribution Registry: + +- [Configuration](https://distribution.github.io/distribution/about/configuration/) +- [Storage drivers](https://distribution.github.io/distribution/storage-drivers/) +- [Deploy a registry server](https://distribution.github.io/distribution/about/deploying/) This document is the administrator's guide. To learn how to use the GitLab Container Registry, see the [user documentation](../../user/packages/container_registry/index.md). @@ -33,14 +37,12 @@ Otherwise, the Container Registry is not enabled. To enable it: The Container Registry works under HTTPS by default. You can use HTTP but it's not recommended and is beyond the scope of this document. -Read the [insecure Registry documentation](https://docs.docker.com/registry/insecure/) -if you want to implement this. ### Self-compiled installations If you self-compiled your GitLab installation: -1. You must [deploy a registry](https://docs.docker.com/registry/deploying/) using the image corresponding to the +1. You must deploy a registry using the image corresponding to the version of GitLab you are installing (for example: `registry.gitlab.com/gitlab-org/build/cng/gitlab-container-registry:v3.15.0-gitlab`) 1. After the installation is complete, to enable it, you must configure the Registry's @@ -70,15 +72,15 @@ Where: | `host` | The host URL under which the Registry runs and users can use. | | `port` | The port the external Registry domain listens on. | | `api_url` | The internal API URL under which the Registry is exposed. It defaults to `http://localhost:5000`. Do not change this unless you are setting up an [external Docker registry](#use-an-external-container-registry-with-gitlab-as-an-auth-endpoint). | -| `key` | The private key location that is a pair of Registry's `rootcertbundle`. Read the [token auth configuration documentation](https://docs.docker.com/registry/configuration/#token). | -| `path` | This should be the same directory like specified in Registry's `rootdirectory`. Read the [storage configuration documentation](https://docs.docker.com/registry/configuration/#storage). This path needs to be readable by the GitLab user, the web-server user and the Registry user. Read more in [#configure-storage-for-the-container-registry](#configure-storage-for-the-container-registry). | -| `issuer` | This should be the same value as configured in Registry's `issuer`. Read the [token auth configuration documentation](https://docs.docker.com/registry/configuration/#token). | +| `key` | The private key location that is a pair of Registry's `rootcertbundle`. | +| `path` | This should be the same directory like specified in Registry's `rootdirectory`. This path needs to be readable by the GitLab user, the web-server user and the Registry user. | +| `issuer` | This should be the same value as configured in Registry's `issuer`. | A Registry init file is not shipped with GitLab if you install it from source. Hence, [restarting GitLab](../restart_gitlab.md#self-compiled-installations) does not restart the Registry should you modify its settings. Read the upstream documentation on how to achieve that. -At the **absolute** minimum, make sure your [Registry configuration](https://docs.docker.com/registry/configuration/#auth) +At the **absolute** minimum, make sure your Registry configuration has `container_registry` as the service and `https://gitlab.example.com/jwt/auth` as the realm: @@ -383,9 +385,6 @@ The different supported drivers are: Although most S3 compatible services (like [MinIO](https://min.io/)) should work with the Container Registry, we only guarantee support for AWS S3. Because we cannot assert the correctness of third-party S3 implementations, we can debug issues, but we cannot patch the registry unless an issue is reproducible against an AWS S3 bucket. -Read more about the individual driver's configuration options in the -[Docker Registry docs](https://docs.docker.com/registry/configuration/#storage). - ### Use file system If you want to store your images on the file system, you can change the storage @@ -532,14 +531,14 @@ To configure the `gcs` storage driver for a Linux package installation: } ``` - GitLab supports all [available parameters](https://docs.docker.com/registry/storage-drivers/gcs/). + GitLab supports all available parameters. 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. #### Self-compiled installations Configuring the storage driver is done in the registry configuration YAML file created -when you [deployed your Docker registry](https://docs.docker.com/registry/deploying/). +when you deployed your Docker registry. `s3` storage driver example: @@ -638,11 +637,11 @@ you can pull from the Container Registry, but you cannot push. <!--- start_remove The following content will be removed on remove_date: '2023-10-22' --> WARNING: -The default configuration for the storage driver is scheduled to be [changed](https://gitlab.com/gitlab-org/container-registry/-/issues/854) in GitLab 16.0. The storage driver will use `/` as the default root directory. You can add `trimlegacyrootprefix: false` to your current configuration now to avoid any disruptions. For more information, see the [Container Registry configuration](https://gitlab.com/gitlab-org/container-registry/-/tree/master/docs-gitlab#azure-storage-driver) documentation. +The default configuration for the storage driver is scheduled to be [changed](https://gitlab.com/gitlab-org/container-registry/-/issues/854) in GitLab 16.0. The storage driver will use `/` as the default root directory. You can add `trimlegacyrootprefix: false` to your current configuration now to avoid any disruptions. For more information, see the [Container Registry configuration](https://gitlab.com/gitlab-org/container-registry/-/blob/master/docs/upstream-differences.md#azure-storage-driver) documentation. <!--- end_remove --> When moving from an existing file system or another object storage provider to Azure Object Storage, you must configure the registry to use the standard root directory. -Configure it by setting [`trimlegacyrootprefix: true`](https://gitlab.com/gitlab-org/container-registry/-/blob/a3f64464c3ec1c5a599c0a2daa99ebcbc0100b9a/docs-gitlab/README.md#azure-storage-driver) in the Azure storage driver section of the registry configuration. +Configure it by setting [`trimlegacyrootprefix: true`](https://gitlab.com/gitlab-org/container-registry/-/blob/master/docs/upstream-differences.md#azure-storage-driver) in the Azure storage driver section of the registry configuration. Without this configuration, the Azure storage driver uses `//` instead of `/` as the first section of the root path, rendering the migrated images inaccessible. ::Tabs @@ -675,7 +674,7 @@ storage: ::EndTabs -By default, Azure Storage Driver uses the `core.windows.net` realm. You can set another value for `realm` in the `azure` section (for example, `core.usgovcloudapi.net` for Azure Government Cloud). For more information, see the [Docker documentation](https://docs.docker.com/registry/storage-drivers/azure/). +By default, Azure Storage Driver uses the `core.windows.net` realm. You can set another value for `realm` in the `azure` section (for example, `core.usgovcloudapi.net` for Azure Government Cloud). ### Disable redirect for storage driver @@ -876,8 +875,7 @@ You can use GitLab as an auth endpoint with an external container registry. - `gitlab_rails['registry_api_url'] = "http://<external_registry_host>:5000"` must be changed to match the host where Registry is installed. It must also specify `https` if the external registry is - configured to use TLS. Read more on the - [Docker registry documentation](https://docs.docker.com/registry/deploying/). + configured to use TLS. 1. A certificate-key pair is required for GitLab and the external container registry to communicate securely. You need to create a certificate-key @@ -972,7 +970,7 @@ To configure a notification endpoint for a Linux package installation: :::TabTitle Self-compiled (source) Configuring the notification endpoint is done in your registry configuration YAML file created -when you [deployed your Docker registry](https://docs.docker.com/registry/deploying/). +when you deployed your Docker registry. Example: @@ -1028,7 +1026,7 @@ projects.each do |p| end if project_total_size > 0 - projects_and_size << [p.project_id, p.creator.id, project_total_size, p.full_path] + projects_and_size << [p.project_id, p.creator&.id, project_total_size, p.full_path] end end @@ -1374,7 +1372,7 @@ By default, the container registry uses object storage to persist metadata related to container images. This method to store metadata limits how efficiently the data can be accessed, especially data spanning multiple images, such as when listing tags. By using a database to store this data, many new features are possible, including -[online garbage collection](https://gitlab.com/gitlab-org/container-registry/-/blob/master/docs-gitlab/db/online-garbage-collection.md) +[online garbage collection](https://gitlab.com/gitlab-org/container-registry/-/blob/master/docs/spec/gitlab/online-garbage-collection.md) which removes old data automatically with zero downtime. This database works in conjunction with the object storage already used by the registry, but does not replace object storage. @@ -1580,7 +1578,7 @@ You can add a configuration option for backwards compatibility. :::TabTitle Self-compiled (source) -1. Edit the YAML configuration file you created when you [deployed the registry](https://docs.docker.com/registry/deploying/). Add the following snippet: +1. Edit the YAML configuration file you created when you deployed the registry. Add the following snippet: ```yaml compatibility: @@ -1632,7 +1630,7 @@ and a simple solution would be to enable relative URLs in the Registry. :::TabTitle Self-compiled (source) -1. Edit the YAML configuration file you created when you [deployed the registry](https://docs.docker.com/registry/deploying/). Add the following snippet: +1. Edit the YAML configuration file you created when you deployed the registry. Add the following snippet: ```yaml http: diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index f64c53e28a2..97acbf717fe 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -200,13 +200,13 @@ then run `gitlab-ctl reconfigure`. For more information, read **Requirements:** - [Wildcard DNS setup](#dns-configuration) -- [TLS-terminating load balancer](../../install/aws/manual_install_aws.md#load-balancer) +- [TLS-terminating load balancer](../../install/aws/index.md#load-balancer) --- URL scheme: `https://<namespace>.example.io/<project_slug>` -This setup is primarily intended to be used when [installing a GitLab POC on Amazon Web Services](../../install/aws/manual_install_aws.md). This includes a TLS-terminating [classic load balancer](../../install/aws/manual_install_aws.md#load-balancer) that listens for HTTPS connections, manages TLS certificates, and forwards HTTP traffic to the instance. +This setup is primarily intended to be used when [installing a GitLab POC on Amazon Web Services](../../install/aws/index.md). This includes a TLS-terminating [classic load balancer](../../install/aws/index.md#load-balancer) that listens for HTTPS connections, manages TLS certificates, and forwards HTTP traffic to the instance. 1. In `/etc/gitlab/gitlab.rb` specify the following configuration: diff --git a/doc/administration/postgresql/external.md b/doc/administration/postgresql/external.md index a9f857d8f00..b9bfda80b83 100644 --- a/doc/administration/postgresql/external.md +++ b/doc/administration/postgresql/external.md @@ -63,7 +63,6 @@ pg_dump: error: Error message from server: SSL SYSCALL error: EOF detected To resolve this error, ensure that you are meeting the [minimum PostgreSQL requirements](../../install/requirements.md#postgresql-requirements). After -upgrading your RDS instance to a suitable version, you should be able to perform a backup without -this error. Refer to issue #64763 -([Segmentation fault citing `LooseForeignKeys::CleanupWorker` causes complete database restart](https://gitlab.com/gitlab-org/gitlab/-/issues/364763)) -for more information. +upgrading your RDS instance to a [supported version](../../install/requirements.md#database), +you should be able to perform a backup without this error. +See [issue 64763](https://gitlab.com/gitlab-org/gitlab/-/issues/364763) for more information. diff --git a/doc/administration/postgresql/external_metrics.md b/doc/administration/postgresql/external_metrics.md new file mode 100644 index 00000000000..fc4c5652a18 --- /dev/null +++ b/doc/administration/postgresql/external_metrics.md @@ -0,0 +1,33 @@ +--- +stage: Data Stores +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Monitoring and logging setup for external databases + +External PostgreSQL database systems have different logging options for monitoring performance and troubleshooting, however they are not enabled by default. In this section we provide the recommendations for self-managed PostgreSQL, and recommendations for some major providers of PostgreSQL managed services. + +## Recommended PostgreSQL Logging settings + +You should enable the following logging settings: + +- `log_statement=ddl`: log changes of database model definition (DDL), such as `CREATE`, `ALTER` or `DROP` of objects. This helps track recent model changes that could be causing performance issues and identify security breaches and human errors. +- `log_lock_waits=on`: log of processes holding [locks](https://www.postgresql.org/docs/current/explicit-locking.html) for long periods, a common cause of poor query performance. +- `log_temp_files=0`: log usage of intense and unusual temporary files that can indicate poor query performance. +- `log_autovacuum_min_duration=0`: log all autovacuum executions. Autovacuum is a key component for overall PostgreSQL engine performance. Essential for troubleshooting and tuning if dead tuples are not being removed from tables. +- `log_min_duration_statement=1000`: log slow queries (slower than 1 second). + +The full description of the above parameter settings can be found in +[PostgreSQL error reporting and logging documentation](https://www.postgresql.org/docs/current/runtime-config-logging.html#RUNTIME-CONFIG-LOGGING-WHAT). + +## Amazon RDS + +The Amazon Relational Database Service (RDS) provides a large number of [monitoring metrics](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Monitoring.html) and [logging interfaces](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Monitor_Logs_Events.html). Here are a few you should configure: + +- Change all above [recommended PostgreSQL Logging settings](#recommended-postgresql-logging-settings) through [RDS Parameter Groups](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithDBInstanceParamGroups.html). + - As the recommended logging parameters are [dynamic in RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.Parameters.html) you don't require a reboot after changing these settings. + - The PostgreSQL logs can be observed through the [RDS console](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/logs-events-streams-console.html). +- Enable [RDS performance insight](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PerfInsights.html) allows you to visualise your database load with many important performance metrics of a PostgreSQL database engine. +- Enable [RDS Enhanced Monitoring](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_Monitoring.OS.html) to monitor the operating system metrics. These metrics can indicate bottlenecks in your underlying hardware and OS that are impacting your database performance. + - In production environments set the monitoring interval to 10 seconds (or less) to capture micro bursts of resource usage that can be the cause of many performance issues. Set `Granularity=10` in the console or `monitoring-interval=10` in the CLI. diff --git a/doc/administration/postgresql/external_upgrade.md b/doc/administration/postgresql/external_upgrade.md new file mode 100644 index 00000000000..3e2c3b09853 --- /dev/null +++ b/doc/administration/postgresql/external_upgrade.md @@ -0,0 +1,48 @@ +--- +stage: Data Stores +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Upgrading external PostgreSQL databases + +When upgrading your PostgreSQL database engine, it is important to follow all steps +recommended by the PostgreSQL community and your cloud provider. Two +kinds of upgrades exist for PostgreSQL databases: + +- **Minor version upgrades**: These include only bug and security fixes. They are + always backward-compatible with your existing application database model. + + The minor version upgrade process consists of replacing the PostgreSQL binaries + and restarting the database service. The data directory remains unchanged. + +- **Major version upgrades**: These change the internal storage format and the database + catalog. As a result, object statistics used by the query optimizer + [are not transferred to the new version](https://www.postgresql.org/docs/current/pgupgrade.html) + and must be rebuilt with `ANALYZE`. + + Not following the documented major version upgrade process often results in + poor database performance and high CPU use on the database server. + +All major cloud providers support in-place major version upgrades of database +instances, using the `pg_upgrade` utility. However you must follow the pre- and +post- upgrade steps to reduce the risk of performance degradation or database disruption. + +Read carefully the major version upgrade steps of your external database platform: + +- [Amazon RDS for PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_UpgradeDBInstance.PostgreSQL.html#USER_UpgradeDBInstance.PostgreSQL.MajorVersion.Process) +- [Azure Database for PostgreSQL Flexible Server](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-major-version-upgrade) +- [Google Cloud SQL for PostgreSQL](https://cloud.google.com/sql/docs/postgres/upgrade-major-db-version-inplace) +- [PostgreSQL community `pg_upgrade`](https://www.postgresql.org/docs/current/pgupgrade.html) + +## Always `ANALYZE` your database after a major version upgrade + +It is mandatory to run the [`ANALYZE` operation](https://www.postgresql.org/docs/current/sql-analyze.html) +to refresh the `pg_statistic` table after a major version upgrade, because optimizer statistics +[are not transferred by `pg_upgrade`](https://www.postgresql.org/docs/current/pgupgrade.html). +This should be done for all databases on the upgraded PostgreSQL service/instance/cluster. + +To speed up the `ANALYZE` operation, use the +[`vacuumdb` utility](https://www.postgresql.org/docs/current/app-vacuumdb.html), +with `--analyze-only --jobs=njobs` to execute the `ANALYZE` command in parallel by +running `njobs` commands simultaneously. diff --git a/doc/administration/postgresql/index.md b/doc/administration/postgresql/index.md index af0a86c3d72..4d73ba49846 100644 --- a/doc/administration/postgresql/index.md +++ b/doc/administration/postgresql/index.md @@ -30,6 +30,10 @@ your own external PostgreSQL server. Read how to [set up an external PostgreSQL instance](external.md). +When setting up an external database there are some metrics that are useful for monitoring and troubleshooting. +When setting up an external database there are monitoring and logging settings required for troubleshooting various database related issues. +Read more about [monitoring and logging setup for external Databases](external_metrics.md). + ### PostgreSQL replication and failover for Linux package installations **(PREMIUM SELF)** This setup is for when you have installed GitLab using the @@ -47,3 +51,4 @@ Read how to [set up PostgreSQL replication and failover](replication_and_failove - [Moving GitLab databases to a different PostgreSQL instance](moving.md) - [Multiple databases](multiple_databases.md) - [Database guides for GitLab development](../../development/database/index.md) +- [Upgrade external database](external_upgrade.md) diff --git a/doc/administration/raketasks/geo.md b/doc/administration/raketasks/geo.md index c6bc891f529..a4b14b132db 100644 --- a/doc/administration/raketasks/geo.md +++ b/doc/administration/raketasks/geo.md @@ -2,82 +2,14 @@ stage: Systems group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +remove_date: '2024-02-06' +redirect_to: '../../update/deprecations.md#geo-housekeeping-rake-tasks' --- -# Geo Rake tasks **(PREMIUM SELF)** +# Geo Rake tasks (removed) **(PREMIUM SELF)** -The following Rake tasks are for [Geo installations](../geo/index.md). -See also [troubleshooting Geo](../geo/replication/troubleshooting.md) for additional Geo Rake tasks. - -## Git housekeeping - -There are few tasks you can run to schedule a Git housekeeping to start at the -next repository sync in a **secondary** node: - -### Incremental Repack - -This is equivalent of running `git repack -d` on a _bare_ repository. - -- Linux package installations: - - ```shell - sudo gitlab-rake geo:git:housekeeping:incremental_repack - ``` - -- Self-compiled installations: - - ```shell - sudo -u git -H bundle exec rake geo:git:housekeeping:incremental_repack RAILS_ENV=production - ``` - -### Full Repack - -This is equivalent of running `git repack -d -A --pack-kept-objects` on a -_bare_ repository which optionally, writes a reachability bitmap index -when this is enabled in GitLab. - -- Linux package installations: - - ```shell - sudo gitlab-rake geo:git:housekeeping:full_repack - ``` - -- Self-compiled installations: - - ```shell - sudo -u git -H bundle exec rake geo:git:housekeeping:full_repack RAILS_ENV=production - ``` - -### GC - -This is equivalent of running `git gc` on a _bare_ repository, optionally writing -a reachability bitmap index when this is enabled in GitLab. - -- Linux package installations: - - ```shell - sudo gitlab-rake geo:git:housekeeping:gc - ``` - -- Self-compiled installations: - - ```shell - sudo -u git -H bundle exec rake geo:git:housekeeping:gc RAILS_ENV=production - ``` - -## Remove orphaned project registries - -Under certain conditions your project registry can contain obsolete records, you -can remove them using the Rake task `geo:run_orphaned_project_registry_cleaner`: - -- Linux package installations: - - ```shell - sudo gitlab-rake geo:run_orphaned_project_registry_cleaner - ``` - -- Self-compiled installations: - - ```shell - sudo -u git -H bundle exec rake geo:run_orphaned_project_registry_cleaner RAILS_ENV=production - ``` +The Geo housekeeping Rake tasks were +[deprecated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/125927) in +GitLab 16.3 and +[removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/130565) in +GitLab 16.5. diff --git a/doc/administration/raketasks/github_import.md b/doc/administration/raketasks/github_import.md index 82f3ffa2193..a4d52899f21 100644 --- a/doc/administration/raketasks/github_import.md +++ b/doc/administration/raketasks/github_import.md @@ -4,11 +4,15 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# GitHub import Rake task **(FREE SELF)** +# GitHub import Rake task (deprecated) **(FREE SELF)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/390690) in GitLab 15.9, Rake task no longer automatically creates namespaces or groups that don't exist. > - Requirement for Maintainer role instead of Developer role introduced in GitLab 16.0 and backported to GitLab 15.11.1 and GitLab 15.10.5. +WARNING: +This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/428225) in GitLab 16.6 and is planned for +removal in GitLab 17.0. Use the [GitHub import feature](../../user/project/import/github.md) instead. + To retrieve and import GitHub repositories, you need a [GitHub personal access token](https://github.com/settings/tokens). A username should be passed as the second argument to the Rake task, which becomes the owner of the project. You can resume an import diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 2e208c4eca1..2203f4b3a02 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -6,18 +6,21 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Reference architecture: up to 10,000 users **(PREMIUM SELF)** -This page describes GitLab reference architecture for up to 10,000 users. For a -full list of reference architectures, see +This page describes the GitLab reference architecture designed for the load of up to 10,000 users +with notable headroom. + +For a full list of reference architectures, see [Available reference architectures](index.md#available-reference-architectures). -> - **Supported users (approximate):** 10,000 +NOTE: +Before deploying this architecture it's recommended to read through the [main documentation](index.md) first, +specifically the [Before you start](index.md#before-you-start) and [Deciding which architecture to use](index.md#deciding-which-architecture-to-use) sections. + +> - **Target load:** API: 200 RPS, Web: 20 RPS, Git (Pull): 20 RPS, Git (Push): 4 RPS > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) > - **Estimated Costs:** [See cost table](index.md#cost-to-run) > - **Cloud Native Hybrid Alternative:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) -> - **Validation and test results:** The Quality Engineering team does [regular smoke and performance tests](index.md#validation-and-test-results) to ensure the reference architectures remain compliant -> - **Test requests per second (RPS) rates:** API: 200 RPS, Web: 20 RPS, Git (Pull): 20 RPS, Git (Push): 4 RPS -> - **[Latest Results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k)** -> - **Unsure which Reference Architecture to use?** [Go to this guide for more info](index.md#deciding-which-architecture-to-use). +> - **Unsure which Reference Architecture to use?** [Go to this guide for more info](index.md#deciding-which-architecture-to-use) | Service | Nodes | Configuration | GCP | AWS | |------------------------------------------|-------|-------------------------|------------------|----------------| @@ -144,6 +147,27 @@ monitor .[#7FFFD4,norank]u--> elb Before starting, see the [requirements](index.md#requirements) for reference architectures. +## Testing methodology + +The 10k architecture is designed to cover a large majority of workflows and is regularly +[smoke and performance tested](index.md#validation-and-test-results) by the Quality Engineering team +against the following endpoint throughput targets: + +- API: 200 RPS +- Web: 20 RPS +- Git (Pull): 20 RPS +- Git (Push): 4 RPS + +The above targets were selected based on real customer data of total environmental loads corresponding to the user count, +including CI and other workloads along with additional substantial headroom added. + +If you have metrics to suggest that you have regularly higher throughput against the above endpoint targets, [large monorepos](index.md#large-monorepos) +or notable [additional workloads](index.md#additional-workloads) these can notably impact the performance environment and [further adjustments may be required](index.md#scaling-an-environment). +If this applies to you, we strongly recommended referring to the linked documentation as well as reaching out to your [Customer Success Manager](https://handbook.gitlab.com/job-families/sales/customer-success-management/) or our [Support team](https://about.gitlab.com/support/) for further guidance. + +Testing is done regularly via our [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance) and its dataset, which is available for anyone to use. +The results of this testing are [available publicly on the GPT wiki](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest). For more information on our testing strategy [refer to this section of the documentation](index.md#validation-and-test-results). + ## Setup components To set up GitLab and its components to accommodate up to 10,000 users: @@ -1307,7 +1331,7 @@ This is how this would work with a Linux package PostgreSQL setup: 1. Create the new user `praefect`, replacing `<praefect_postgresql_password>`: ```shell - CREATE ROLE praefect WITH LOGIN CREATEDB PASSWORD <praefect_postgresql_password>; + CREATE ROLE praefect WITH LOGIN CREATEDB PASSWORD '<praefect_postgresql_password>'; ``` 1. Reconnect to the PostgreSQL server, this time as the `praefect` user: @@ -1763,7 +1787,8 @@ Updates to example must be made at: --> ```ruby - roles ["sidekiq_role"] + # https://docs.gitlab.com/omnibus/roles/#sidekiq-roles + roles(["sidekiq_role"]) # External URL ## This should match the URL of the external load balancer diff --git a/doc/administration/reference_architectures/1k_users.md b/doc/administration/reference_architectures/1k_users.md index 2f7c8209a44..362da0bd7c6 100644 --- a/doc/administration/reference_architectures/1k_users.md +++ b/doc/administration/reference_architectures/1k_users.md @@ -6,24 +6,18 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Reference architecture: up to 1,000 users **(FREE SELF)** -This page describes GitLab reference architecture for up to 1,000 users. For a -full list of reference architectures, see -[Available reference architectures](index.md#available-reference-architectures). +This page describes the GitLab reference architecture designed for the load of up to 1,000 users +with notable headroom (non-HA standalone). -If you are serving up to 1,000 users, and you don't have strict availability -requirements, a [standalone](index.md#standalone-non-ha) single-node solution with -frequent backups is appropriate for -many organizations. +For a full list of reference architectures, see +[Available reference architectures](index.md#available-reference-architectures). -> - **Supported users (approximate):** 1,000 +> - **Target Load:** API: 20 RPS, Web: 2 RPS, Git (Pull): 2 RPS, Git (Push): 1 RPS > - **High Availability:** No. For a highly-available environment, you can > follow a modified [3K reference architecture](3k_users.md#supported-modifications-for-lower-user-counts-ha). > - **Estimated Costs:** [See cost table](index.md#cost-to-run) > - **Cloud Native Hybrid:** No. For a cloud native hybrid environment, you > can follow a [modified hybrid reference architecture](#cloud-native-hybrid-reference-architecture-with-helm-charts). -> - **Validation and test results:** The Quality Engineering team does [regular smoke and performance tests](index.md#validation-and-test-results) to ensure the reference architectures remain compliant -> - **Test requests per second (RPS) rates:** API: 20 RPS, Web: 2 RPS, Git (Pull): 2 RPS, Git (Push): 1 RPS -> - **[Latest Results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/1k)** > - **Unsure which Reference Architecture to use?** [Go to this guide for more info](index.md#deciding-which-architecture-to-use). | Users | Configuration | GCP | AWS | Azure | @@ -73,6 +67,27 @@ WARNING: **However, if you have [large monorepos](index.md#large-monorepos) (larger than several gigabytes) or [additional workloads](index.md#additional-workloads) these can *significantly* impact the performance of the environment and further adjustments may be required.** If this applies to you, we strongly recommended referring to the linked documentation as well as reaching out to your [Customer Success Manager](https://handbook.gitlab.com/job-families/sales/customer-success-management/) or our [Support team](https://about.gitlab.com/support/) for further guidance. +## Testing methodology + +The 1k architecture is designed to cover a large majority of workflows and is regularly +[smoke and performance tested](index.md#validation-and-test-results) by the Quality Engineering team +against the following endpoint throughput targets: + +- API: 20 RPS +- Web: 2 RPS +- Git (Pull): 2 RPS +- Git (Push): 1 RPS + +The above targets were selected based on real customer data of total environmental loads corresponding to the user count, +including CI and other workloads along with additional substantial headroom added. + +If you have metrics to suggest that you have regularly higher throughput against the above endpoint targets, [large monorepos](index.md#large-monorepos) +or notable [additional workloads](index.md#additional-workloads) these can notably impact the performance environment and [further adjustments may be required](index.md#scaling-an-environment). +If this applies to you, we strongly recommended referring to the linked documentation as well as reaching out to your [Customer Success Manager](https://handbook.gitlab.com/job-families/sales/customer-success-management/) or our [Support team](https://about.gitlab.com/support/) for further guidance. + +Testing is done regularly via our [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance) and its dataset, which is available for anyone to use. +The results of this testing are [available publicly on the GPT wiki](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest). For more information on our testing strategy [refer to this section of the documentation](index.md#validation-and-test-results). + ## Setup instructions To install GitLab for this default reference architecture, use the standard diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 355fe45cc2f..a5d44edf877 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -6,18 +6,21 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Reference architecture: up to 25,000 users **(PREMIUM SELF)** -This page describes GitLab reference architecture for up to 25,000 users. For a -full list of reference architectures, see +This page describes the GitLab reference architecture designed for the load of up to 25,000 users +with notable headroom. + +For a full list of reference architectures, see [Available reference architectures](index.md#available-reference-architectures). -> - **Supported users (approximate):** 25,000 +NOTE: +Before deploying this architecture it's recommended to read through the [main documentation](index.md) first, +specifically the [Before you start](index.md#before-you-start) and [Deciding which architecture to use](index.md#deciding-which-architecture-to-use) sections. + +> - **Target load:** API: 500 RPS, Web: 50 RPS, Git (Pull): 50 RPS, Git (Push): 10 RPS > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) > - **Estimated Costs:** [See cost table](index.md#cost-to-run) > - **Cloud Native Hybrid Alternative:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) -> - **Validation and test results:** The Quality Engineering team does [regular smoke and performance tests](index.md#validation-and-test-results) to ensure the reference architectures remain compliant -> - **Test requests per second (RPS) rates:** API: 500 RPS, Web: 50 RPS, Git (Pull): 50 RPS, Git (Push): 10 RPS -> - **[Latest Results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/25k)** -> - **Unsure which Reference Architecture to use?** [Go to this guide for more info](index.md#deciding-which-architecture-to-use). +> - **Unsure which Reference Architecture to use?** [Go to this guide for more info](index.md#deciding-which-architecture-to-use) | Service | Nodes | Configuration | GCP | AWS | |------------------------------------------|-------|-------------------------|------------------|--------------| @@ -144,6 +147,27 @@ monitor .[#7FFFD4,norank]u--> elb Before starting, see the [requirements](index.md#requirements) for reference architectures. +## Testing methodology + +The 25k architecture is designed to cover a large majority of workflows and is regularly +[smoke and performance tested](index.md#validation-and-test-results) by the Quality Engineering team +against the following endpoint throughput targets: + +- API: 500 RPS +- Web: 50 RPS +- Git (Pull): 50 RPS +- Git (Push): 10 RPS + +The above targets were selected based on real customer data of total environmental loads corresponding to the user count, +including CI and other workloads along with additional substantial headroom added. + +If you have metrics to suggest that you have regularly higher throughput against the above endpoint targets, [large monorepos](index.md#large-monorepos) +or notable [additional workloads](index.md#additional-workloads) these can notably impact the performance environment and [further adjustments may be required](index.md#scaling-an-environment). +If this applies to you, we strongly recommended referring to the linked documentation as well as reaching out to your [Customer Success Manager](https://handbook.gitlab.com/job-families/sales/customer-success-management/) or our [Support team](https://about.gitlab.com/support/) for further guidance. + +Testing is done regularly via our [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance) and its dataset, which is available for anyone to use. +The results of this testing are [available publicly on the GPT wiki](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest). For more information on our testing strategy [refer to this section of the documentation](index.md#validation-and-test-results). + ## Setup components To set up GitLab and its components to accommodate up to 25,000 users: @@ -1324,7 +1348,7 @@ This is how this would work with a Linux package PostgreSQL setup: 1. Create the new user `praefect`, replacing `<praefect_postgresql_password>`: ```shell - CREATE ROLE praefect WITH LOGIN CREATEDB PASSWORD <praefect_postgresql_password>; + CREATE ROLE praefect WITH LOGIN CREATEDB PASSWORD '<praefect_postgresql_password>'; ``` 1. Reconnect to the PostgreSQL server, this time as the `praefect` user: @@ -1780,7 +1804,8 @@ Updates to example must be made at: --> ```ruby - roles ["sidekiq_role"] + # https://docs.gitlab.com/omnibus/roles/#sidekiq-roles + roles(["sidekiq_role"]) # External URL ## This should match the URL of the external load balancer diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index 5814d6c1e2d..fb8b9d8de45 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -6,18 +6,17 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Reference architecture: up to 2,000 users **(FREE SELF)** -This page describes GitLab reference architecture for up to 2,000 users. +This page describes the GitLab reference architecture designed for the load of up to 2,000 users +with notable headroom (non-HA). + For a full list of reference architectures, see [Available reference architectures](index.md#available-reference-architectures). -> - **Supported users (approximate):** 2,000 +> - **Target Load:** API: 40 RPS, Web: 4 RPS, Git (Pull): 4 RPS, Git (Push): 1 RPS > - **High Availability:** No. For a highly-available environment, you can > follow a modified [3K reference architecture](3k_users.md#supported-modifications-for-lower-user-counts-ha). > - **Estimated Costs:** [See cost table](index.md#cost-to-run) > - **Cloud Native Hybrid:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) -> - **Validation and test results:** The Quality Engineering team does [regular smoke and performance tests](index.md#validation-and-test-results) to ensure the reference architectures remain compliant -> - **Test requests per second (RPS) rates:** API: 40 RPS, Web: 4 RPS, Git (Pull): 4 RPS, Git (Push): 1 RPS -> - **[Latest Results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/2k)** > - **Unsure which Reference Architecture to use?** [Go to this guide for more info](index.md#deciding-which-architecture-to-use). | Service | Nodes | Configuration | GCP | AWS | Azure | @@ -81,6 +80,27 @@ monitor .[#7FFFD4,norank]u--> elb Before starting, see the [requirements](index.md#requirements) for reference architectures. +## Testing methodology + +The 2k architecture is designed to cover a large majority of workflows and is regularly +[smoke and performance tested](index.md#validation-and-test-results) by the Quality Engineering team +against the following endpoint throughput targets: + +- API: 40 RPS +- Web: 4 RPS +- Git (Pull): 4 RPS +- Git (Push): 1 RPS + +The above targets were selected based on real customer data of total environmental loads corresponding to the user count, +including CI and other workloads along with additional substantial headroom added. + +If you have metrics to suggest that you have regularly higher throughput against the above endpoint targets, [large monorepos](index.md#large-monorepos) +or notable [additional workloads](index.md#additional-workloads) these can notably impact the performance environment and [further adjustments may be required](index.md#scaling-an-environment). +If this applies to you, we strongly recommended referring to the linked documentation as well as reaching out to your [Customer Success Manager](https://handbook.gitlab.com/job-families/sales/customer-success-management/) or our [Support team](https://about.gitlab.com/support/) for further guidance. + +Testing is done regularly via our [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance) and its dataset, which is available for anyone to use. +The results of this testing are [available publicly on the GPT wiki](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest). For more information on our testing strategy [refer to this section of the documentation](index.md#validation-and-test-results). + ## Setup components To set up GitLab and its components to accommodate up to 2,000 users: @@ -609,7 +629,8 @@ Updates to example must be made at: --> ```ruby - roles ["sidekiq_role"] + # https://docs.gitlab.com/omnibus/roles/#sidekiq-roles + roles(["sidekiq_role"]) # External URL external_url 'https://gitlab.example.com' diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index 1fd8239c93f..73b0291ab95 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -6,27 +6,20 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Reference architecture: up to 3,000 users **(PREMIUM SELF)** -This GitLab reference architecture can help you deploy GitLab to up to 3,000 -users, and then maintain uptime and access for those users. You can also use -this architecture to provide improved GitLab uptime and availability for fewer -than 3,000 users. For fewer users, reduce the stated node sizes as needed. +This page describes the GitLab reference architecture designed for the load of up to 3,000 users +with notable headroom. -If maintaining a high level of uptime for your GitLab environment isn't a -requirement, or if you don't have the expertise to maintain this sort of -environment, we recommend using the non-HA [2,000-user reference architecture](2k_users.md) -for your GitLab installation. If HA is still a requirement, there's several supported -tweaks you can make to this architecture to reduce complexity as detailed here. +This architecture is the smallest one available with HA built in. If you require HA but +have a lower user count or total load the [Supported Modifications for lower user counts](#supported-modifications-for-lower-user-counts-ha) +section details how to reduce this architecture's size while maintaining HA. For a full list of reference architectures, see [Available reference architectures](index.md#available-reference-architectures). -> - **Supported users (approximate):** 3,000 +> - **Target Load:** 60 RPS, Web: 6 RPS, Git (Pull): 6 RPS, Git (Push): 1 RPS > - **High Availability:** Yes, although [Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution > - **Estimated Costs:** [See cost table](index.md#cost-to-run) > - **Cloud Native Hybrid Alternative:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) -> - **Validation and test results:** The Quality Engineering team does [regular smoke and performance tests](index.md#validation-and-test-results) to ensure the reference architectures remain compliant -> - **Test requests per second (RPS) rates:** API: 60 RPS, Web: 6 RPS, Git (Pull): 6 RPS, Git (Push): 1 RPS -> - **[Latest Results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/3k)** > - **Unsure which Reference Architecture to use?** [Go to this guide for more info](index.md#deciding-which-architecture-to-use). | Service | Nodes | Configuration | GCP | AWS | @@ -149,6 +142,27 @@ monitor .[#7FFFD4,norank]u--> elb Before starting, see the [requirements](index.md#requirements) for reference architectures. +## Testing methodology + +The 3k architecture is designed to cover a large majority of workflows and is regularly +[smoke and performance tested](index.md#validation-and-test-results) by the Quality Engineering team +against the following endpoint throughput targets: + +- API: 60 RPS +- Web: 6 RPS +- Git (Pull): 6 RPS +- Git (Push): 1 RPS + +The above targets were selected based on real customer data of total environmental loads corresponding to the user count, +including CI and other workloads along with additional substantial headroom added. + +If you have metrics to suggest that you have regularly higher throughput against the above endpoint targets, [large monorepos](index.md#large-monorepos) +or notable [additional workloads](index.md#additional-workloads) these can notably impact the performance environment and [further adjustments may be required](index.md#scaling-an-environment). +If this applies to you, we strongly recommended referring to the linked documentation as well as reaching out to your [Customer Success Manager](https://handbook.gitlab.com/job-families/sales/customer-success-management/) or our [Support team](https://about.gitlab.com/support/) for further guidance. + +Testing is done regularly via our [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance) and its dataset, which is available for anyone to use. +The results of this testing are [available publicly on the GPT wiki](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest). For more information on our testing strategy [refer to this section of the documentation](index.md#validation-and-test-results). + ## Setup components To set up GitLab and its components to accommodate up to 3,000 users: @@ -1248,7 +1262,7 @@ This is how this would work with a Linux package PostgreSQL setup: 1. Create the new user `praefect`, replacing `<praefect_postgresql_password>`: ```shell - CREATE ROLE praefect WITH LOGIN CREATEDB PASSWORD <praefect_postgresql_password>; + CREATE ROLE praefect WITH LOGIN CREATEDB PASSWORD '<praefect_postgresql_password>'; ``` 1. Reconnect to the PostgreSQL server, this time as the `praefect` user: @@ -1708,7 +1722,8 @@ Updates to example must be made at: --> ```ruby - roles ["sidekiq_role"] + # https://docs.gitlab.com/omnibus/roles/#sidekiq-roles + roles(["sidekiq_role"]) # External URL ## This should match the URL of the external load balancer diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 72ddd347856..ca39468a76e 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -6,18 +6,21 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Reference architecture: up to 50,000 users **(PREMIUM SELF)** -This page describes GitLab reference architecture for up to 50,000 users. For a -full list of reference architectures, see +This page describes the GitLab reference architecture designed for the load of up to 50,000 users +with notable headroom. + +For a full list of reference architectures, see [Available reference architectures](index.md#available-reference-architectures). -> - **Supported users (approximate):** 50,000 +NOTE: +Before deploying this architecture it's recommended to read through the [main documentation](index.md) first, +specifically the [Before you start](index.md#before-you-start) and [Deciding which architecture to use](index.md#deciding-which-architecture-to-use) sections. + +> - **Target load:** API: 1000 RPS, Web: 100 RPS, Git (Pull): 100 RPS, Git (Push): 20 RPS > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) > - **Estimated Costs:** [See cost table](index.md#cost-to-run) > - **Cloud Native Hybrid Alternative:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) -> - **Validation and test results:** The Quality Engineering team does [regular smoke and performance tests](index.md#validation-and-test-results) to ensure the reference architectures remain compliant -> - **Test requests per second (RPS) rates:** API: 1000 RPS, Web: 100 RPS, Git (Pull): 100 RPS, Git (Push): 20 RPS -> - **[Latest Results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/50k)** -> - **Unsure which Reference Architecture to use?** [Go to this guide for more info](index.md#deciding-which-architecture-to-use). +> - **Unsure which Reference Architecture to use?** [Go to this guide for more info](index.md#deciding-which-architecture-to-use) | Service | Nodes | Configuration | GCP | AWS | |------------------------------------------|-------|-------------------------|------------------|---------------| @@ -144,6 +147,27 @@ monitor .[#7FFFD4,norank]u--> elb Before starting, see the [requirements](index.md#requirements) for reference architectures. +## Testing methodology + +The 50k architecture is designed to cover a large majority of workflows and is regularly +[smoke and performance tested](index.md#validation-and-test-results) by the Quality Engineering team +against the following endpoint throughput targets: + +- API: 1000 RPS +- Web: 100 RPS +- Git (Pull): 100 RPS +- Git (Push): 20 RPS + +The above targets were selected based on real customer data of total environmental loads corresponding to the user count, +including CI and other workloads along with additional substantial headroom added. + +If you have metrics to suggest that you have regularly higher throughput against the above endpoint targets, [large monorepos](index.md#large-monorepos) +or notable [additional workloads](index.md#additional-workloads) these can notably impact the performance environment and [further adjustments may be required](index.md#scaling-an-environment). +If this applies to you, we strongly recommended referring to the linked documentation as well as reaching out to your [Customer Success Manager](https://handbook.gitlab.com/job-families/sales/customer-success-management/) or our [Support team](https://about.gitlab.com/support/) for further guidance. + +Testing is done regularly via our [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance) and its dataset, which is available for anyone to use. +The results of this testing are [available publicly on the GPT wiki](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest). For more information on our testing strategy [refer to this section of the documentation](index.md#validation-and-test-results). + ## Setup components To set up GitLab and its components to accommodate up to 50,000 users: @@ -1320,7 +1344,7 @@ This is how this would work with a Linux package PostgreSQL setup: 1. Create the new user `praefect`, replacing `<praefect_postgresql_password>`: ```shell - CREATE ROLE praefect WITH LOGIN CREATEDB PASSWORD <praefect_postgresql_password>; + CREATE ROLE praefect WITH LOGIN CREATEDB PASSWORD '<praefect_postgresql_password>'; ``` 1. Reconnect to the PostgreSQL server, this time as the `praefect` user: @@ -1776,7 +1800,8 @@ Updates to example must be made at: --> ```ruby - roles ["sidekiq_role"] + # https://docs.gitlab.com/omnibus/roles/#sidekiq-roles + roles(["sidekiq_role"]) # External URL ## This should match the URL of the external load balancer diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index e2bf0aa59f4..e908565e27e 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -6,25 +6,21 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Reference architecture: up to 5,000 users **(PREMIUM SELF)** -This page describes GitLab reference architecture for up to 5,000 users. For a -full list of reference architectures, see +This page describes the GitLab reference architecture designed for the load of up to 5,000 users +with notable headroom. + +For a full list of reference architectures, see [Available reference architectures](index.md#available-reference-architectures). NOTE: -This reference architecture is designed to help your organization achieve a -highly-available GitLab deployment. If you do not have the expertise or need to -maintain a highly-available environment, you can have a simpler and less -costly-to-operate environment by using the -[2,000-user reference architecture](2k_users.md). +Before deploying this architecture it's recommended to read through the [main documentation](index.md) first, +specifically the [Before you start](index.md#before-you-start) and [Deciding which architecture to use](index.md#deciding-which-architecture-to-use) sections. -> - **Supported users (approximate):** 5,000 +> - **Target load:** API: 100 RPS, Web: 10 RPS, Git (Pull): 10 RPS, Git (Push): 2 RPS > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) > - **Estimated Costs:** [See cost table](index.md#cost-to-run) > - **Cloud Native Hybrid Alternative:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) -> - **Validation and test results:** The Quality Engineering team does [regular smoke and performance tests](index.md#validation-and-test-results) to ensure the reference architectures remain compliant -> - **Test requests per second (RPS) rates:** API: 100 RPS, Web: 10 RPS, Git (Pull): 10 RPS, Git (Push): 2 RPS -> - **[Latest Results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/5k)** -> - **Unsure which Reference Architecture to use?** [Go to this guide for more info](index.md#deciding-which-architecture-to-use). +> - **Unsure which Reference Architecture to use?** [Go to this guide for more info](index.md#deciding-which-architecture-to-use) | Service | Nodes | Configuration | GCP | AWS | |-------------------------------------------|-------|-------------------------|-----------------|--------------| @@ -146,6 +142,27 @@ monitor .[#7FFFD4,norank]u--> elb Before starting, see the [requirements](index.md#requirements) for reference architectures. +## Testing methodology + +The 5k architecture is designed to cover a large majority of workflows and is regularly +[smoke and performance tested](index.md#validation-and-test-results) by the Quality Engineering team +against the following endpoint throughput targets: + +- API: 100 RPS +- Web: 10 RPS +- Git (Pull): 10 RPS +- Git (Push): 2 RPS + +The above targets were selected based on real customer data of total environmental loads corresponding to the user count, +including CI and other workloads along with additional substantial headroom added. + +If you have metrics to suggest that you have regularly higher throughput against the above endpoint targets, [large monorepos](index.md#large-monorepos) +or notable [additional workloads](index.md#additional-workloads) these can notably impact the performance environment and [further adjustments may be required](index.md#scaling-an-environment). +If this applies to you, we strongly recommended referring to the linked documentation as well as reaching out to your [Customer Success Manager](https://handbook.gitlab.com/job-families/sales/customer-success-management/) or our [Support team](https://about.gitlab.com/support/) for further guidance. + +Testing is done regularly via our [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance) and its dataset, which is available for anyone to use. +The results of this testing are [available publicly on the GPT wiki](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest). For more information on our testing strategy [refer to this section of the documentation](index.md#validation-and-test-results). + ## Setup components To set up GitLab and its components to accommodate up to 5,000 users: @@ -1242,7 +1259,7 @@ This is how this would work with a Linux package PostgreSQL setup: 1. Create the new user `praefect`, replacing `<praefect_postgresql_password>`: ```shell - CREATE ROLE praefect WITH LOGIN CREATEDB PASSWORD <praefect_postgresql_password>; + CREATE ROLE praefect WITH LOGIN CREATEDB PASSWORD '<praefect_postgresql_password>'; ``` 1. Reconnect to the PostgreSQL server, this time as the `praefect` user: @@ -1696,7 +1713,8 @@ Updates to example must be made at: --> ```ruby - roles ["sidekiq_role"] + # https://docs.gitlab.com/omnibus/roles/#sidekiq-roles + roles(["sidekiq_role"]) # External URL ## This should match the URL of the external load balancer diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index 44aa3d648ad..fcbfaf46009 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -12,36 +12,37 @@ GitLab Quality Engineering and Support teams to provide recommended deployments ## Available reference architectures -Depending on your workflow, the following recommended reference architectures -may need to be adapted accordingly. Your workload is influenced by factors -including how active your users are, how much automation you use, mirroring, -and repository/change size. Additionally, the displayed memory values are -provided by [GCP machine types](https://cloud.google.com/compute/docs/machine-resource). -For different cloud vendors, attempt to select options that best match the -provided architecture. +The following Reference Architectures are available as recommended starting points for your environment. + +The architectures are named in terms of user count, which in this case means the architecture is designed against +the _total_ load that comes with such a user count based on real data along with substantial headroom added to cover most scenarios such as CI or other automated workloads. + +However, it should be noted that in some cases, known heavy scenarios such as [large monorepos](#large-monorepos) or notable [additional workloads](#additional-workloads) may require adjustments to be made. + +For each Reference Architecture, the details of what they have been tested against can be found respectively in the "Testing Methodology" section of each page. ### GitLab package (Omnibus) -The following reference architectures, where the GitLab package is used, are available: +Below is a list of Linux package based architectures: -- [Up to 1,000 users](1k_users.md) -- [Up to 2,000 users](2k_users.md) -- [Up to 3,000 users](3k_users.md) -- [Up to 5,000 users](5k_users.md) -- [Up to 10,000 users](10k_users.md) -- [Up to 25,000 users](25k_users.md) -- [Up to 50,000 users](50k_users.md) +- [Up to 1,000 users](1k_users.md) <span style="color: darkgrey;">_API: 20 RPS, Web: 2 RPS, Git (Pull): 2 RPS, Git (Push): 1 RPS_</span> +- [Up to 2,000 users](2k_users.md) <span style="color: darkgrey;">_API: 40 RPS, Web: 4 RPS, Git (Pull): 4 RPS, Git (Push): 1 RPS_</span> +- [Up to 3,000 users](3k_users.md) <span style="color: darkgrey;">_API: 60 RPS, Web: 6 RPS, Git (Pull): 6 RPS, Git (Push): 1 RPS_</span> +- [Up to 5,000 users](5k_users.md) <span style="color: darkgrey;">_API: 100 RPS, Web: 10 RPS, Git (Pull): 10 RPS, Git (Push): 2 RPS_</span> +- [Up to 10,000 users](10k_users.md) <span style="color: darkgrey;">_API: 200 RPS, Web: 20 RPS, Git (Pull): 20 RPS, Git (Push): 4 RPS_</span> +- [Up to 25,000 users](25k_users.md) <span style="color: darkgrey;">_API: 500 RPS, Web: 50 RPS, Git (Pull): 50 RPS, Git (Push): 10 RPS_</span> +- [Up to 50,000 users](50k_users.md) <span style="color: darkgrey;">_API: 1000 RPS, Web: 100 RPS, Git (Pull): 100 RPS, Git (Push): 20 RPS_</span> ### Cloud native hybrid -The following Cloud Native Hybrid reference architectures, where select recommended components can be run in Kubernetes, are available: +Below is a list of Cloud Native Hybrid reference architectures, where select recommended components can be run in Kubernetes: -- [Up to 2,000 users](2k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) -- [Up to 3,000 users](3k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) -- [Up to 5,000 users](5k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) -- [Up to 10,000 users](10k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) -- [Up to 25,000 users](25k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) -- [Up to 50,000 users](50k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) +- [Up to 2,000 users](2k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) <span style="color: darkgrey;">_API: 40 RPS, Web: 4 RPS, Git (Pull): 4 RPS, Git (Push): 1 RPS_</span> +- [Up to 3,000 users](3k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) <span style="color: darkgrey;">_API: 60 RPS, Web: 6 RPS, Git (Pull): 6 RPS, Git (Push): 1 RPS_</span> +- [Up to 5,000 users](5k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) <span style="color: darkgrey;">_API: 100 RPS, Web: 10 RPS, Git (Pull): 10 RPS, Git (Push): 2 RPS_</span> +- [Up to 10,000 users](10k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) <span style="color: darkgrey;">_API: 200 RPS, Web: 20 RPS, Git (Pull): 20 RPS, Git (Push): 4 RPS_</span> +- [Up to 25,000 users](25k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) <span style="color: darkgrey;">_API: 500 RPS, Web: 50 RPS, Git (Pull): 50 RPS, Git (Push): 10 RPS_</span> +- [Up to 50,000 users](50k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) <span style="color: darkgrey;">_API: 1000 RPS, Web: 100 RPS, Git (Pull): 100 RPS, Git (Push): 20 RPS_</span> ## Before you start @@ -63,6 +64,19 @@ As a general guide, **the more performant and/or resilient you want your environ This section explains the designs you can choose from. It begins with the least complexity, goes to the most, and ends with a decision tree. +### Expected Load (RPS) + +The first thing to check is what the expected load is your environment would be expected to serve. + +The Reference Architectures have been designed with substantial headroom by default, but it's recommended to also check the +load of what each architecture has been tested against under the "Testing Methodology" section found on each page, +comparing those values with what load you are expecting against your existing GitLab environment to help select the right Reference Architecture +size. + +Load is given in terms of Requests per Section (RPS) for each endpoint type (API, Web, Git). This information on your existing infrastructure +can typically be surfaced by most reputable monitoring solutions or in some other ways such as load balancer metrics. For example, on existing GitLab environments, +[Prometheus metrics](../monitoring/prometheus/gitlab_metrics.md) such as `gitlab_transaction_duration_seconds` can be used to see this data. + ### Standalone (non-HA) For environments serving 2,000 or fewer users, we generally recommend a standalone approach by deploying a non-highly available single or multi-node environment. With this approach, you can employ strategies such as [automated backups](../../administration/backup_restore/backup_gitlab.md#configuring-cron-to-make-daily-backups) for recovery to provide a good level of RPO / RTO while avoiding the complexities that come with HA. @@ -144,10 +158,11 @@ Below you can find the above guidance in the form of a decision tree. It's recom ```mermaid %%{init: { 'theme': 'base' } }%% graph TD - L1A(<b>What Reference Architecture should I use?</b>) + L0A(<b>What Reference Architecture should I use?</b>) + L1A(<b>What is your <a href=#expected-load-rps>expected load</a>?</b>) - L2A(3,000 users or more?) - L2B(2,000 users or less?) + L2A("Equivalent to <a href=3k_users.md#testing-methodology>3,000 users</a> or more?") + L2B("Equivalent to <a href=2k_users.md#testing-methodology>2,000 users</a> or less?") L3A("<a href=#do-you-need-high-availability-ha>Do you need HA?</a><br>(or Zero-Downtime Upgrades)") L3B[Do you have experience with<br/>and want additional resilience<br/>with select components in Kubernetes?] @@ -157,6 +172,7 @@ graph TD L4C><b>Recommendation</b><br><br>Cloud Native Hybrid architecture<br>closest to user count] L4D>"<b>Recommendation</b><br><br>Standalone 1K or 2K<br/>architecture with Backups"] + L0A --> L1A L1A --> L2A L1A --> L2B L2A -->|Yes| L3B @@ -191,13 +207,22 @@ Before implementing a reference architecture, refer to the following requirement These reference architectures were built and tested on Google Cloud Platform (GCP) using the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) CPU platform as a lowest common denominator baseline ([Sysbench benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks)). +Newer, similarly-sized CPUs are supported and may have improved performance as a result. -Newer, similarly-sized CPUs are supported and may have improved performance as a result. For Linux package environments, -ARM-based equivalents are also supported. +ARM CPUs are supported for Linux package environments as well as for any [Cloud Provider services](#cloud-provider-services) where applicable. NOTE: Any "burstable" instance types are not recommended due to inconsistent performance. +### Supported disk types + +As a general guidance, most standard disk types are expected to work for GitLab, but be aware of the following specific call outs: + +- [Gitaly](../gitaly/index.md#disk-requirements) requires at least 8,000 input/output operations per second (IOPS) for read operations, and 2,000 IOPS for write operations. +- We don't recommend the use of any disk types that are "burstable" due to inconsistent performance. + +Outside the above standard, disk types are expected to work for GitLab and the choice of each depends on your specific requirements around areas, such as durability or costs. + ### Supported infrastructure As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP, Azure) and @@ -356,6 +381,12 @@ If you choose to use a third party external service: Redis is primarily single threaded. For the 10,000 user and above Reference Architectures, separate out the instances as specified into Cache and Persistent data to achieve optimum performance at this scale. +### Recommendation notes for Object Storage + +GitLab has been tested against [various Object Storage providers](../object_storage.md#supported-object-storage-providers) that are expected to work. + +As a general guidance, it's recommended to use a reputable solution that has full S3 compatibility. + #### Unsupported database services Several database cloud provider services are known not to support the above or have been found to have other issues and aren't recommended: @@ -649,22 +680,35 @@ You should upgrade a Reference Architecture in the same order as you created it. ### Scaling an environment -Scaling a GitLab environment is designed to be as seamless as possible. +Scaling a GitLab environment is designed to be as flexible and seamless as possible. + +This can be done iteratively or wholesale to the next size of architecture depending on your circumstances. +For example, if any of your GitLab Rails, Sidekiq, Gitaly, Redis or PostgreSQL nodes are consistently oversaturated, then increase their resources accordingly while leaving the rest of the environment as is. -In terms of the Reference Architectures, you would look to the next size and adjust accordingly. -Most setups would only need vertical scaling, but there are some specific areas that can be adjusted depending on the setup: +If expecting a large increase in users, you may elect to scale up the whole environment to the next +size of architecture. + +If the overall design is being followed, you can scale the environment vertically as required. + +If robust metrics are in place that show the environment is over-provisioned, you can apply the same process for +scaling downwards. You should take an iterative approach when scaling downwards to ensure there are no issues. + +#### Scaling from a non-HA to an HA architecture + +While in most cases vertical scaling is only required to increase an environment's resources, if you are moving to an HA environment, +there may be some additional steps required as shown below: - If you're scaling from a non-HA environment to an HA environment, various components are recommended to be deployed in their HA forms: - - Redis to multi-node Redis w/ Redis Sentinel - - Postgres to multi-node Postgres w/ Consul + PgBouncer - - Gitaly to Gitaly Cluster w/ Praefect + - [Redis to multi-node Redis w/ Redis Sentinel](../redis/replication_and_failover.md#switching-from-an-existing-single-machine-installation) + - [Postgres to multi-node Postgres w/ Consul + PgBouncer](../postgresql/moving.md) + - [Gitaly to Gitaly Cluster w/ Praefect](../gitaly/index.md#migrate-to-gitaly-cluster) - From 10k users and higher, Redis is recommended to be split into multiple HA servers as it's single threaded. Conversely, if you have robust metrics in place that show the environment is over-provisioned, you can apply the same process for scaling downwards. You should take an iterative approach when scaling downwards, however, to ensure there are no issues. -### How to monitor your environment +### Monitoring + +There are numerous options available to monitor your infrastructure, as well as [GitLab itself](../monitoring/index.md), and you should refer to your chosen monitoring solution's documentation for more information. -To monitor your GitLab environment, you can use the tools -[bundled with GitLab](../monitoring/index.md), but it's also possible to use third-party -options if desired. +Of note, the GitLab application is bundled with [Prometheus as well as various Prometheus compatible exporters](../monitoring/prometheus/index.md) that could be hooked into your solution. diff --git a/doc/administration/review_spam_logs.md b/doc/administration/review_spam_logs.md new file mode 100644 index 00000000000..e3b96cdae95 --- /dev/null +++ b/doc/administration/review_spam_logs.md @@ -0,0 +1,40 @@ +--- +stage: Govern +group: Anti-Abuse +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: reference, howto +--- + +# Review spam logs **(FREE SELF)** + +GitLab tracks user activity and flags certain behavior for potential spam. + +In the Admin Area, a GitLab administrator can view and resolve spam logs. + +## Manage spam logs + +> **Trust user** [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/131812) in GitLab 16.5. + +View and resolve spam logs to moderate user activity in your instance. + +To view spam logs: + +1. On the left sidebar, select **Search or go to**. +1. Select **Admin Area**. +1. Select **Spam Logs**. +1. Optional. To resolve a spam log, select a log and then select **Remove user**, **Block user**, **Remove log**, or **Trust user**. + +### Resolving spam logs + +You can resolve a spam log with one of the following effects: + +| Option | Description | +|---------|-------------| +| **Remove user** | The user is [deleted](../user/profile/account/delete_account.md) from the instance. | +| **Block user** | The user is blocked from the instance. The spam log remains in the list. | +| **Remove log** | The spam log is removed from the list. | +| **Trust user** | The user is trusted, and can create issues, notes, snippets, and merge requests without being blocked for spam. Spam logs are not created for trusted users. | + +NOTE: +Users can be [blocked](../api/users.md#block-user) and +[unblocked](../api/users.md#unblock-user) using the GitLab API. diff --git a/doc/administration/settings/continuous_integration.md b/doc/administration/settings/continuous_integration.md index 841b6e644eb..0e2a512302d 100644 --- a/doc/administration/settings/continuous_integration.md +++ b/doc/administration/settings/continuous_integration.md @@ -266,6 +266,22 @@ To enable or disable the banner: 1. Select or clear the **Enable pipeline suggestion banner** checkbox. 1. Select **Save changes**. +## Enable or disable the external redirect page for job artifacts + +By default, GitLab Pages shows an external redirect page when a user tries to view +a job artifact served by GitLab Pages. This page warns about the potential for +malicious user-generated content, as described in +[issue 352611](https://gitlab.com/gitlab-org/gitlab/-/issues/352611). + +Self-managed administrators can disable the external redirect warning page, +so you can view job artifact pages directly: + +1. On the left sidebar, select **Search or go to**. +1. Select **Admin Area**. +1. Select **Settings > CI/CD**. +1. Expand **Continuous Integration and Deployment**. +1. Deselect **Enable the external redirect page for job artifacts**. + ## Required pipeline configuration **(ULTIMATE SELF)** > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/352316) from GitLab Premium to GitLab Ultimate in 15.0. diff --git a/doc/administration/settings/gitaly_timeouts.md b/doc/administration/settings/gitaly_timeouts.md index 3304db3d148..1cab1e9fd01 100644 --- a/doc/administration/settings/gitaly_timeouts.md +++ b/doc/administration/settings/gitaly_timeouts.md @@ -20,8 +20,10 @@ To access Gitaly timeout settings: The following timeouts are available. -| Timeout | Default | Description | -|:--------|:-----------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Timeout | Default | Description | +|:--------|:-----------|:------------| | Default | 55 seconds | Timeout for most Gitaly calls (not enforced for `git` `fetch` and `push` operations, or Sidekiq jobs). For example, checking if a repository exists on disk. Makes sure that Gitaly calls made within a web request cannot exceed the entire request timeout. It should be shorter than the [worker timeout](../operations/puma.md#change-the-worker-timeout) that can be configured for [Puma](../../install/requirements.md#puma-settings). If a Gitaly call timeout exceeds the worker timeout, the remaining time from the worker timeout is used to avoid having to terminate the worker. | -| Fast | 10 seconds | Timeout for fast Gitaly operations used within requests, sometimes multiple times. For example, checking if a repository exists on disk. If fast operations exceed this threshold, there may be a problem with a storage shard. Failing fast can help maintain the stability of the GitLab instance. | -| Medium | 30 seconds | Timeout for Gitaly operations that should be fast (possibly within requests) but preferably not used multiple times within a request. For example, loading blobs. Timeout that should be set between Default and Fast. | +| Fast | 10 seconds | Timeout for fast Gitaly operations used within requests, sometimes multiple times. For example, checking if a repository exists on disk. If fast operations exceed this threshold, there may be a problem with a storage shard. Failing fast can help maintain the stability of the GitLab instance. | +| Medium | 30 seconds | Timeout for Gitaly operations that should be fast (possibly within requests) but preferably not used multiple times within a request. For example, loading blobs. Timeout that should be set between Default and Fast. | + +You can also [configure negotiation timeouts](../gitaly/configure_gitaly.md#configure-negotiation-timeouts). diff --git a/doc/administration/settings/jira_cloud_app.md b/doc/administration/settings/jira_cloud_app.md index f4f1db3617e..8ff2a9acdb8 100644 --- a/doc/administration/settings/jira_cloud_app.md +++ b/doc/administration/settings/jira_cloud_app.md @@ -37,6 +37,9 @@ To create an OAuth application on your self-managed instance: - If you're installing the app from the official marketplace listing, enter `https://gitlab.com/-/jira_connect/oauth_callbacks`. - If you're installing the app manually, enter `<instance_url>/-/jira_connect/oauth_callbacks` and replace `<instance_url>` with the URL of your instance. 1. Clear the **Trusted** and **Confidential** checkboxes. + + NOTE: + You must clear these checkboxes to avoid errors. 1. In **Scopes**, select the `api` checkbox only. 1. Select **Save application**. 1. Copy the **Application ID** value. @@ -45,6 +48,28 @@ To create an OAuth application on your self-managed instance: 1. Paste the **Application ID** value into **Jira Connect Application ID**. 1. Select **Save changes**. +## Jira user requirements + +> Support for the `org-admins` group [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/420687) in GitLab 16.6. + +In your [Atlassian organization](https://admin.atlassian.com), you must ensure that the Jira user that is used to set up the GitLab for Jira Cloud app is a member of +either: + +- The Organization Administrators (`org-admins`) group. Newer Atlassian organizations are using + [centralized user management](https://support.atlassian.com/user-management/docs/give-users-admin-permissions/#Centralized-user-management-content), + which contains the `org-admins` group. Existing Atlassian organizations are being migrated to centralized user management. + If available, you should use the `org-admins` group to indicate which Jira users can manage the GitLab for Jira app. Alternatively you can use the + `site-admins` group. +- The Site Administrators (`site-admins`) group. The `site-admins` group was used under + [original user management](https://support.atlassian.com/user-management/docs/give-users-admin-permissions/#Original-user-management-content). + +If necessary: + +1. [Create your preferred group](https://support.atlassian.com/user-management/docs/create-groups/). +1. [Edit the group](https://support.atlassian.com/user-management/docs/edit-a-group/) to add your Jira user as a member of it. +1. If you customized your global permissions in Jira, you might also need to grant the + [`Browse users and groups` permission](https://confluence.atlassian.com/jirakb/unable-to-browse-for-users-and-groups-120521888.html) to the Jira user. + ## Connect the GitLab for Jira Cloud app > Introduced in GitLab 15.7. @@ -76,6 +101,7 @@ With this method: - Set up an internet-facing reverse proxy in front of your self-managed instance. To secure this proxy further, only allow inbound traffic from [Atlassian IP addresses](https://support.atlassian.com/organization-administration/docs/ip-addresses-and-domains-for-atlassian-cloud-products/#Outgoing-Connections). - Add [GitLab IP addresses](../../user/gitlab_com/index.md#ip-range) to the allowlist of your firewall. +- The Jira user that installs and configures the GitLab for Jira Cloud app must meet certain [requirements](#jira-user-requirements). ### Set up your instance @@ -144,6 +170,7 @@ To support your self-managed instance with Jira Cloud, do one of the following: - The instance must be publicly available. - You must set up [OAuth authentication](#set-up-oauth-authentication). +- The Jira user that installs and configures the GitLab for Jira Cloud app must meet certain [requirements](#jira-user-requirements). ### Install the app in development mode @@ -314,6 +341,8 @@ To resolve this issue, ensure all prerequisites for your installation method hav - [Prerequisites for connecting the GitLab for Jira Cloud app](#prerequisites) - [Prerequisites for installing the GitLab for Jira Cloud app manually](#prerequisites-1) +If you have configured a Jira Connect Proxy URL and the problem persists after checking the prerequisites, review [Debugging Jira Connect Proxy issues](#debugging-jira-connect-proxy-issues). + If you're using GitLab 15.8 and earlier and have previously enabled both the `jira_connect_oauth_self_managed` and the `jira_connect_oauth` feature flags, you must disable the `jira_connect_oauth_self_managed` flag due to a [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/388943). To check for these flags: @@ -331,6 +360,46 @@ due to a [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/388943). To Feature.disable(:jira_connect_oauth_self_managed) ``` +#### Debugging Jira Connect Proxy issues + +If you are using a self-managed GitLab instance and you have configured `https://gitlab.com` for the Jira Connect Proxy URL when +[setting up the OAuth authentication](#set-up-oauth-authentication), you can inspect the network traffic in your browser's development +tools while reproducing the `Failed to update the GitLab instance` error to see a more precise error. + +You should see a `GET` request to `https://gitlab.com/-/jira_connect/installations`. + +This request should return a `200` status code, but it can return a `422` status code if there was a problem. The response body can be checked for the error. + +If you cannot resolve the problem and you are a GitLab customer, contact [GitLab Support](https://about.gitlab.com/support/) for assistance. Provide +GitLab Support with: + +1. Your GitLab self-managed instance URL. +1. Your GitLab.com username. +1. If possible, the `X-Request-Id` response header for the failed `GET` request to `https://gitlab.com/-/jira_connect/installations`. +1. Optional. [A HAR file that captured the problem](https://support.zendesk.com/hc/en-us/articles/4408828867098-Generating-a-HAR-file-for-troubleshooting). + +The GitLab Support team can then look up why this is failing in the GitLab.com server logs. + +##### Process for GitLab Support + +NOTE: +These steps can only be completed by GitLab Support. + +In Kibana, the logs should be filtered for `json.meta.caller_id: JiraConnect::InstallationsController#update` and `NOT json.status: 200`. +If you have been provided the `X-Request-Id` value, you can use that against `json.correlation_id` to narrow down the results. + +Each `GET` request to the Jira Connect Proxy URL `https://gitlab.com/-/jira_connect/installations` generates two log entries. + +For the first log: + +- `json.status` is `422`. +- `json.params.value` should match the GitLab self-managed URL `[[FILTERED], {"instance_url"=>"https://gitlab.example.com"}]`. + +For the second log: + +- `json.message` is `Proxy lifecycle event received error response` or similar. +- `json.jira_status_code` and `json.jira_body` might contain details on why GitLab.com wasn't able to connect back to the self-managed instance. + ### `Failed to link group` After you connect the GitLab for Jira Cloud app for self-managed instances, you might get one of these errors: @@ -349,9 +418,6 @@ When you check the browser console, you might see the following message: Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://gitlab.example.com/-/jira_connect/oauth_application_id. (Reason: CORS header 'Access-Control-Allow-Origin' missing). Status code: 403. ``` -`403` status code is returned if: - -- The user information cannot be fetched from Jira. -- The authenticated Jira user does not have [site administrator](https://support.atlassian.com/user-management/docs/give-users-admin-permissions/#Make-someone-a-site-admin) access. +`403` status code is returned if the user information cannot be fetched from Jira because of insufficient permissions. -To resolve this issue, ensure the authenticated user is a Jira site administrator and try again. +To resolve this issue, ensure that the Jira user that installs and configures the GitLab for Jira Cloud app meets certain [requirements](#jira-user-requirements). diff --git a/doc/administration/settings/rate_limits_on_git_ssh_operations.md b/doc/administration/settings/rate_limits_on_git_ssh_operations.md index 677d8fea195..4e60fd55b19 100644 --- a/doc/administration/settings/rate_limits_on_git_ssh_operations.md +++ b/doc/administration/settings/rate_limits_on_git_ssh_operations.md @@ -20,8 +20,6 @@ Each command has a rate limit of 600 per minute. For example: Because the same commands are shared by `git-upload-pack`, `git pull`, and `git clone`, they share a rate limit. -Users on self-managed GitLab can disable this rate limit. - ## Configure GitLab Shell operation limit > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/123761) in GitLab 16.2. @@ -33,4 +31,5 @@ Users on self-managed GitLab can disable this rate limit. 1. Select **Settings > Network**. 1. Expand **Git SSH operations rate limit**. 1. Enter a value for **Maximum number of Git operations per minute**. + - To disable the rate limit, set it to `0`. 1. Select **Save changes**. diff --git a/doc/administration/settings/scim_setup.md b/doc/administration/settings/scim_setup.md index 432c8598cf7..45020fdfb59 100644 --- a/doc/administration/settings/scim_setup.md +++ b/doc/administration/settings/scim_setup.md @@ -53,3 +53,7 @@ adding them to the SCIM identity provider. After the identity provider performs a sync based on its configured schedule, the user's SCIM identity is reactivated and their GitLab instance access is restored. + +## Troubleshooting + +See our [troubleshooting SCIM guide](../../user/group/saml_sso/troubleshooting_scim.md). diff --git a/doc/administration/settings/sign_in_restrictions.md b/doc/administration/settings/sign_in_restrictions.md index 6d38610192b..942b706b9a3 100644 --- a/doc/administration/settings/sign_in_restrictions.md +++ b/doc/administration/settings/sign_in_restrictions.md @@ -118,7 +118,7 @@ The following access methods are **not** protected by Admin Mode: In other words, administrators who are otherwise limited by Admin Mode can still use Git clients without additional authentication steps. -To use the GitLab REST- or GraphQL API, administrators must [create a personal access token](../../user/profile/personal_access_tokens.md#create-a-personal-access-token) with the [`admin_mode` scope](../../user/profile/personal_access_tokens.md#personal-access-token-scopes). +To use the GitLab REST- or GraphQL API, administrators must [create a personal access token](../../user/profile/personal_access_tokens.md#create-a-personal-access-token) or [OAuth token](../../api/oauth2.md) with the [`admin_mode` scope](../../user/profile/personal_access_tokens.md#personal-access-token-scopes). If an administrator with a personal access token with the `admin_mode` scope loses their administrator access, that user cannot access the API as an administrator even though they still have the token with the `admin_mode` scope. diff --git a/doc/administration/settings/slack_app.md b/doc/administration/settings/slack_app.md index ef756dfeff7..de11da281e4 100644 --- a/doc/administration/settings/slack_app.md +++ b/doc/administration/settings/slack_app.md @@ -105,9 +105,13 @@ To enable the GitLab for Slack app functionality, your network must allow inboun ## Troubleshooting -### Slash commands return `/gitlab failed with the error "dispatch_failed"` in Slack +When administering the GitLab for Slack app for self-managed instances, you might encounter the following issues. + +For GitLab.com, see [GitLab for Slack app](../../user/project/integrations/gitlab_slack_application.md#troubleshooting). + +### Slash commands return an error in Slack Slash commands might return `/gitlab failed with the error "dispatch_failed"` in Slack. To resolve this issue, ensure: -- The GitLab for Slack app is properly [configured](#configure-the-settings), and the **Enable GitLab for Slack app** checkbox is selected. +- The GitLab for Slack app is properly [configured](#configure-the-settings) and the **Enable GitLab for Slack app** checkbox is selected. - Your GitLab instance [allows requests to and from Slack](#connectivity-requirements). diff --git a/doc/administration/settings/usage_statistics.md b/doc/administration/settings/usage_statistics.md index 4887ebd8cfe..b9080f49f5d 100644 --- a/doc/administration/settings/usage_statistics.md +++ b/doc/administration/settings/usage_statistics.md @@ -9,7 +9,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w GitLab Inc. periodically collects information about your instance in order to perform various actions. -All usage statistics are [opt-out](#enable-or-disable-usage-statistics). +For free self-managed instances, all usage statistics are [opt-out](#enable-or-disable-service-ping). +For information about other tiers, see [Customer Product Usage Information](https://about.gitlab.com/handbook/legal/privacy/customer-product-usage-information/#service-ping-formerly-known-as-usage-ping). ## Service Ping @@ -63,6 +64,13 @@ In the following table, you can see: | [Issue analytics](../../user/group/issues_analytics/index.md) | GitLab 16.5 and later | | [Custom Text in Emails](../../administration/settings/email.md#custom-additional-text) | GitLab 16.5 and later | | [Contribution analytics](../../user/group/contribution_analytics/index.md) | GitLab 16.5 and later | +| [Group file templates](../../user/group/manage.md#group-file-templates) | GitLab 16.6 and later | +| [Group webhooks](../../user/project/integrations/webhooks.md#group-webhooks) | GitLab 16.6 and later | +| [Service Level Agreement countdown timer](../../operations/incident_management/incidents.md#service-level-agreement-countdown-timer) | GitLab 16.6 and later | +| [Lock project membership to group](../../user/group/access_and_permissions.md#prevent-members-from-being-added-to-projects-in-a-group) | GitLab 16.6 and later | +| [Users and permissions report](../../administration/admin_area.md#user-permission-export) | GitLab 16.6 and later | +| [Advanced search](../../user/search/advanced_search.md) | GitLab 16.6 and later | +| [DevOps Adoption](../../user/group/devops_adoption/index.md) | GitLab 16.6 and later | ### Enable registration features @@ -95,7 +103,16 @@ This information is used, among other things, to identify to which versions patches must be backported, making sure active GitLab instances remain secure. -If you [disable version check](#enable-or-disable-usage-statistics), this information isn't collected. +If you disable version check, this information isn't collected. + +### Enable or disable version check + +1. On the left sidebar, select **Search or go to**. +1. Select **Admin Area**. +1. Select **Settings > Metrics and profiling**. +1. Expand **Usage statistics**. +1. Select or clear the **Enable version check** checkbox. +1. Select **Save changes**. ### Request flow example @@ -121,23 +138,26 @@ GitLab instance to the host `version.gitlab.com` on port `443`. If your GitLab instance is behind a proxy, set the appropriate [proxy configuration variables](https://docs.gitlab.com/omnibus/settings/environment-variables.html). -## Enable or disable usage statistics +## Enable or disable Service Ping + +### Through the UI -To enable or disable Service Ping and version check: +To enable or disable Service Ping: 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Metrics and profiling**. 1. Expand **Usage statistics**. -1. Select or clear the **Enable version check** and **Enable Service Ping** checkboxes. +1. Select or clear the **Enable Service Ping** checkbox. 1. Select **Save changes**. NOTE: -Service Ping settings only control whether the data is being shared with GitLab, or used only internally. +The effect of disabling Service Ping depends on the instance's tier. For more information, see [Customer Product Usage Information](https://about.gitlab.com/handbook/legal/privacy/customer-product-usage-information/#service-ping-formerly-known-as-usage-ping). +Service Ping settings only control whether the data is being shared with GitLab, or limited to only internal use by the instance. Even if you disable Service Ping, the `gitlab_service_ping_worker` background job still periodically generates a Service Ping payload for your instance. -The payload is available in the [Service Usage data](#manually-upload-service-ping-payload) admin section. +The payload is available in the [Metrics and profiling](#manually-upload-service-ping-payload) admin section. -## Disable usage statistics with the configuration file +### Through the configuration file NOTE: The method to disable Service Ping in the GitLab configuration file does not work in @@ -189,7 +209,7 @@ You can view the exact JSON payload sent to GitLab Inc. in the Admin Area. To vi 1. Sign in as a user with administrator access. 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. -1. Select **Settings > Service usage data**. +1. Select **Settings > Metrics and profiling > Usage statistics**. 1. Select **Preview payload**. For an example payload, see [Example Service Ping payload](../../development/internal_analytics/service_ping/index.md#example-service-ping-payload). @@ -207,7 +227,7 @@ To upload the payload manually: 1. Sign in as a user with administrator access. 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. -1. Select **Settings > Service usage data**. +1. Select **Settings > Metrics and profiling > Usage statistics**. 1. Select **Download payload**. 1. Save the JSON file. 1. Visit [Service usage data center](https://version.gitlab.com/usage_data/new). diff --git a/doc/administration/sidekiq/index.md b/doc/administration/sidekiq/index.md index 10fadc40a82..0a7974c9622 100644 --- a/doc/administration/sidekiq/index.md +++ b/doc/administration/sidekiq/index.md @@ -95,27 +95,8 @@ Updates to example must be made at: --> ```ruby - ######################################## - ##### Services Disabled ### - ######################################## - # - # When running GitLab on just one server, you have a single `gitlab.rb` - # to enable all services you want to run. - # When running GitLab on N servers, you have N `gitlab.rb` files. - # Enable only the services you want to run on each - # specific server, while disabling all others. - # - gitaly['enable'] = false - postgresql['enable'] = false - redis['enable'] = false - nginx['enable'] = false - puma['enable'] = false - gitlab_workhorse['enable'] = false - prometheus['enable'] = false - alertmanager['enable'] = false - grafana['enable'] = false - gitlab_exporter['enable'] = false - gitlab_kas['enable'] = false + # https://docs.gitlab.com/omnibus/roles/#sidekiq-roles + roles(["sidekiq_role"]) ## ## To maintain uniformity of links across nodes, the @@ -375,20 +356,6 @@ To enable LDAP with the synchronization worker for Sidekiq: If you use [SAML Group Sync](../../user/group/saml_sso/group_sync.md), you must configure [SAML Groups](../../integration/saml.md#configure-users-based-on-saml-group-membership) on all your Sidekiq nodes. -## Disable Rugged - -Calls into Rugged, Ruby bindings for `libgit2`, [lock the Sidekiq processes (GVL)](https://silverhammermba.github.io/emberb/c/#c-in-ruby-threads), -blocking all jobs on that worker from proceeding. If Rugged calls performed by Sidekiq are slow, this can cause significant delays in -background task processing. - -By default, Rugged is used when Git repository data is stored on local storage or on an NFS mount. -Using Rugged is recommended when using NFS, but if -you are using local storage, disabling Rugged can improve Sidekiq performance: - -```shell -sudo gitlab-rake gitlab:features:disable_rugged -``` - ## Related topics - [Extra Sidekiq processes](extra_sidekiq_processes.md) diff --git a/doc/administration/sidekiq/processing_specific_job_classes.md b/doc/administration/sidekiq/processing_specific_job_classes.md index 696b0b9444c..74cbb6ca89b 100644 --- a/doc/administration/sidekiq/processing_specific_job_classes.md +++ b/doc/administration/sidekiq/processing_specific_job_classes.md @@ -179,14 +179,16 @@ nodes. In this example, we exclude all import-related jobs from a Sidekiq node. sudo gitlab-ctl reconfigure ``` -### Migrating from queue selectors to routing rules +## Migrating from queue selectors to routing rules We recommend GitLab deployments add more Sidekiq processes listening to all queues, as in the [Reference Architectures](../reference_architectures/index.md). For very large-scale deployments, we recommend [routing rules](#routing-rules) instead of [queue selectors](#queue-selectors-deprecated). We use routing rules on GitLab.com as it helps to lower the load on Redis. -To migrate from queue selectors to routing rules: +### Single node setup + +To migrate from queue selectors to routing rules in a [single node setup](../reference_architectures/index.md#standalone-non-ha): 1. Open `/etc/gitlab/gitlab.rb`. 1. Set `sidekiq['queue_selector']` to `false`. @@ -213,9 +215,11 @@ NOTE: It is important to run the Rake task immediately after reconfiguring GitLab. After reconfiguring GitLab, existing jobs are not processed until the Rake task starts to migrate the jobs. +#### Migration example + The following example better illustrates the migration process above: -1. Check the following content of `/etc/gitlab/gitlab.rb`: +1. In `/etc/gitlab/gitlab.rb`, check the `urgency` queries in the `sidekiq['queue_groups']`. For example: ```ruby sidekiq['routing_rules'] = [] @@ -228,7 +232,7 @@ The following example better illustrates the migration process above: ] ``` -1. Update `/etc/gitlab/gitlab.rb` to use routing rules: +1. Use these same `urgency` queries to update `/etc/gitlab/gitlab.rb` to use routing rules: ```ruby sidekiq['min_concurrency'] = 20 @@ -270,6 +274,31 @@ in a queue group entry is 1, while `min_concurrency` is set to `0`, and `max_con concurrency is set to `2` instead. A concurrency of `2` might be too low in most cases, except for very highly-CPU bound tasks. +### Multiple node setup + +For a multiple node setup: + +- Reconfigure all GitLab Rails and Sidekiq nodes with the same `sidekiq['routing_rules']` setting. +- Alternate between GitLab Rails and Sidekiq nodes as you update and reconfigure the nodes. This ensures the newly configured Sidekiq is ready to consume jobs from the new set of + queues during the migration. Otherwise, the new jobs hang until the end of the migration. + +Consider the following example of three GitLab Rails nodes and two Sidekiq nodes. To migrate from queue selectors to routing rules: + +1. In Sidekiq 1, follow all steps but one in [single node setup](#single-node-setup). + **Do not** run the Rake task to [migrate existing jobs](sidekiq_job_migration.md). +1. Configure the external load balancer to remove Rails 1 from accepting traffic. This step ensures Rails 1 is not serving any request while the Rails process is restarting. For more information, see [issue 428794](https://gitlab.com/gitlab-org/gitlab/-/issues/428794#note_1619505870). +1. In Rails 1, update `/etc/gitlab/gitlab.rb` to use the same `sidekiq['routing_rules']` setting as Sidekiq 1. + Only `sidekiq['routing_rules']` is required in Rails nodes. +1. Configure the external load balancer to register Rails 1 back. +1. Repeat steps 1 to 4 for Sidekiq 2 and Rails 2. +1. Repeat steps 2 to 4 for Rails 3. +1. If there are more Sidekiq nodes than Rails nodes, follow step 1 on the remaining Sidekiq nodes. +1. Run the Rake task to [migrate existing jobs](sidekiq_job_migration.md): + + ```shell + sudo gitlab-rake gitlab:sidekiq:migrate_jobs:retry gitlab:sidekiq:migrate_jobs:schedule gitlab:sidekiq:migrate_jobs:queued + ``` + <!--- end_remove --> ## Worker matching query diff --git a/doc/administration/sidekiq/sidekiq_troubleshooting.md b/doc/administration/sidekiq/sidekiq_troubleshooting.md index 9ae2a59251a..2990110150f 100644 --- a/doc/administration/sidekiq/sidekiq_troubleshooting.md +++ b/doc/administration/sidekiq/sidekiq_troubleshooting.md @@ -536,6 +536,28 @@ The list of available jobs can be found in the [workers](https://gitlab.com/gitl For more information about Sidekiq jobs, see the [Sidekiq-cron](https://github.com/sidekiq-cron/sidekiq-cron#work-with-job) documentation. +## Disabling cron jobs + +You can disable any Sidekiq cron jobs by visiting the [Monitoring section in the Admin area](../admin_area.md#monitoring-section). You can also perform the same action using the command line and [Rails Runner](../operations/rails_console.md#using-the-rails-runner). + +To disable all cron jobs: + +```shell +sudo gitlab-rails runner 'Sidekiq::Cron::Job.all.map(&:disable!)' +``` + +To enable all cron jobs: + +```shell +sudo gitlab-rails runner 'Sidekiq::Cron::Job.all.map(&:enable!)' +``` + +If you wish to enable only a subset of the jobs at a time you can use name matching. For example, to enable only jobs with `geo` in the name: + +```shell + sudo gitlab-rails runner 'Sidekiq::Cron::Job.all.select{ |j| j.name.match("geo") }.map(&:disable!)' +``` + ## Clearing a Sidekiq job deduplication idempotency key Occasionally, jobs that are expected to run (for example, cron jobs) are observed to not run at all. When checking the logs, there might be instances where jobs are seen to not run with a `"job_status": "deduplicated"`. diff --git a/doc/administration/silent_mode/index.md b/doc/administration/silent_mode/index.md index 379b00536f3..4f68a765585 100644 --- a/doc/administration/silent_mode/index.md +++ b/doc/administration/silent_mode/index.md @@ -4,10 +4,11 @@ group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# GitLab Silent Mode **(FREE SELF EXPERIMENT)** +# GitLab Silent Mode **(FREE SELF)** -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/9826) in GitLab 15.11. This feature is an [Experiment](../../policy/experiment-beta-support.md#experiment). -> - Enabling and disabling Silent Mode through the web UI was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/131090) in GitLab 16.4 +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/9826) in GitLab 15.11. This feature was an [Experiment](../../policy/experiment-beta-support.md#experiment). +> - Enabling and disabling Silent Mode through the web UI was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/131090) in GitLab 16.4. +> - Silent Mode was updated to [Generally Available (GA)](../../policy/experiment-beta-support.md#generally-available-ga) in GitLab 16.6. Silent Mode allows you to silence outbound communication, such as emails, from GitLab. Silent Mode is not intended to be used on environments which are in-use. Two use-cases are: @@ -76,7 +77,7 @@ It may take up to a minute to take effect. [Issue 405433](https://gitlab.com/git ## Behavior of GitLab features in Silent Mode -This section documents the current behavior of GitLab when Silent Mode is enabled. While Silent Mode is an Experiment, the behavior may change without notice. The work for the first iteration of Silent Mode is tracked by [Epic 9826](https://gitlab.com/groups/gitlab-org/-/epics/9826). +This section documents the current behavior of GitLab when Silent Mode is enabled. The work for the first iteration of Silent Mode is tracked by [Epic 9826](https://gitlab.com/groups/gitlab-org/-/epics/9826). When Silent Mode is enabled, a banner is displayed at the top of the page for all users stating the setting is enabled and **All outbound communications are blocked.**. diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 9432836c22b..01c75c32366 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -46,11 +46,11 @@ This content has been moved to [Troubleshooting Repository mirroring](../../user ## CI -This content has been moved to [Troubleshooting CI/CD](../../ci/troubleshooting.md). +This content has been moved to [Troubleshooting CI/CD](../cicd.md#cicd-troubleshooting-rails-console-commands). ## License -This content has been moved to [Activate GitLab EE with a license file or key](../../administration/license_file.md). +This content has been moved to [Activate GitLab EE with a license file or key](../license_file.md). ## Registry |